Researchers at the Innovation Cube at Fundação Getulio Vargas’ School of Communication, Media and Information (FGV ECMI) are conducting experiments with Stable Diffusion, a generative artificial intelligence model that produces images using ComfyUI. The experiment involves capturing images in real time, which are sent for processing in ComfyUI, which creates a new image, like a “rotoscope” animation, based not only on captured images, but also on the filter and instructions provided through a set of words called a prompt. 

The final goal is to create a multisensory work titled “SON(H)O PROFUNDO” (“DEEP SLEEP / DEEP DREAM”), to be displayed at the Digital Art Biennial exhibition, which will take place between 2024 and 2025 at the Futures Art and Technology Center in Rio de Janeiro. The project is the result of a partnership between FGV ECMI’s Innovation Cube and its Media Lab.

During the experiments, the researchers observed that even with a very detailed set of words to be excluded from the output (a negative prompt), many inappropriate images were generated, such as naked bodies or firearms. For this reason, after several tests, in order to reflect the institution’s policies, the age range of the target audience, the theme and the “dream goals,” a model based on crayon illustrations is being used. 

The Stable Diffusion model is free and can be used installed on a computer or via any browser. The coding is performed with the help of third parties, so this is an option for computers that don’t have much processing power.

As rendering depends a lot on the processing speed of the equipment, this process has a lag of approximately two seconds on the Innovation Cube’s machines and the resulting images can be viewed in a resolution of 1024x576 pixels.

ComfyUI is a graphical user interface based on points or nodes that generates image results through fine adjustments.