RESEARCH
3D to aI – maSKED Subgenerations
3D | AI | COMFYUI | 2024
This research explores a workflow to enhance control over AI-generated images by utilizing 3D block out techniques and region-specific subgenerations, defined by RGB AOVs. This approach utilizes ComfyUI and Stable Diffusion XL models.
3D Block Out and Passes
Cinema 4D was utilized to create a quick blockout of the scene. By rendering specific AOVs, we can use ControlNet to achieve greater control throughout all primary and secondary AI generations. Key AOVs for this pipeline include Z-Depth, CryptoMatte, and a line pass, which can be obtained by rendering the entire scene with Toon materials.
BLOCK OUT
Z-DEPTH
LINE PASS
RGB MASK
Comfyui Workflow Overview
Cinema 4D was utilized to create a quick blockout of the scene. By rendering specific AOVs, we can use ControlNet to achieve greater control throughout all primary and secondary AI generations. Key AOVs for this pipeline include Z-Depth, CryptoMatte, and a line pass, which can be obtained by rendering the entire scene with Toon materials.
Benefits to This Approach
Subgenerations provide an effective method to enhance art direction in the final output, preventing specific details from bleeding into other areas of the image. In the examples below, we successfully add distinct colors to the cars and fire details to the buildings without these elements inadvertently affecting other sections.
CAR SUBGEN
FIRE/EXPLOSION SUBGEN
In contrast, the examples below demonstrate how adding these details without properly segmenting them into subgenerations can lead to inconsistent outputs.
PROMPT:
detailed big combat robot, destroying a city, cars in the street,
BREAK
detailed buildings, buildings on fire, explosion, wet ground, reflections,
BREAK
tense atmosphere, cinematic shot, film grain, sunset time, detailed sky
In this output – “buildings on fire” ended up adding fire to the robot itself.
Conclusion
The overall results from this pipeline were highly satisfactory. Blocking out a scene in 3D and rendering it with AI is a fast process, making it ideal for the concepting stages of a project.
Subgenerations have proven to offer significantly greater control over specific details. Additionally, the use of individual masks for each subgeneration simplifies the workflow for Matte Painters and Concept Artists, allowing them to further refine the generated assets with ease.
