With ControlNet offering so many features, it’s natural to feel overwhelmed about which one to use. This article provides practical examples of ControlNet to address such concerns, with a focus on ControlNet NormalMap. By the end of this read, the following questions will be clarified:
- What exactly is ControlNet NormalMap?
- Can you provide tangible use-cases for ControlNet NormalMap?
- How does ControlNet NormalMap differ from similar techniques like ControlNet Depth and Segmentation?
What is ControlNet Normal Map?
Normal Map is also referred to as “normal map.” A normal map is an image that correlates the three-dimensional structure of XYZ with RGB (Red, Green, Blue). ControlNet Normal Map can infer and extract this normal map from an image, allowing you to generate pictures while retaining a detailed 3D structure.
Difference between generating images from prompts and using ControlNet NormalMap
Compared to generating images from prompts, using ControlNet NormalMap allows you to specify the 3D structure and details in the reference image. This method is particularly useful when you want to swap textures.
How ControlNet NormalMap differs from similar techniques
The following two are comparable techniques:
- Depth (measuring the depth or distance in the image)
- Segmentation
While Depth extracts the depth of an image, Normal Map, though similar in nature, can extract more detailed nuances and undulations. Thus, if you’re aiming to extract details from the reference image, Normal Map is superior. Segmentation only captures the flat composition, so it cannot extract 3D structures. If the 3D structure is crucial, you’d either use Normal Map or Depth. Conversely, for silhouettes where the 3D structure is irrelevant, Segmentation becomes the method of choice.
What can you do with ControlNet NormalMap?
- Change textures
- Generate images with the same composition
Texture Changes
On the left is the reference image, and on the right is the generated result. You can generate images that appear as real photos from sculptures.
Generating Images with the Same Composition
On the left is the reference image, and on the right is the generated result. As you can observe, it is possible to create images with different outfits or individuals while retaining the same composition.
How to Utilize ControlNet NormalMap
Preparation for ControlNet NormalMap
ControlNet NormalMap is an extension feature of the Stable Diffusion Web UI and falls under the ControlNet functionalities. Therefore, to employ ControlNet NormalMap, ControlNet needs to be installed. If not yet installed, consider this guide for reference:
Installing ControlNet NormalMap
To utilize ControlNet NormalMap, the ControlNet Model is essential. Download the two files below from the provided link and place them in stable-diffusion-webui/models/ControlNet:
- control_v11p_sd15_normalbae.pth
- control_v11p_sd15_normalbae.yaml
lllyasviel/ControlNet-v1-1 at main (huggingface.co)
Usage of ControlNet NormalMap
Set up the ControlNet menu following these steps:
- Input the prompt for the image you want to generate.
- Click on “▼” to open the ControlNet menu.
- Set the reference image on the ControlNet menu screen.
- Check “Enable” to activate ControlNet.
- Select “Normal Map” under Control Type. This action will set both the Preprocessor and the ControlNet Model.
- Click on the feature extraction button “💥” to perform the feature extraction. The results will be displayed after the preprocessing is applied.
- Now, with ControlNet NormalMap applied, click “Generate” to produce the image.
Specific Usage of ControlNet NormalMap
Changing Textures
- Drag and drop the image onto the ControlNet menu screen.
- Check “Enable”.
- Select “Normal Map” under Control Type.
- Click the feature extraction button “💥” to perform the feature extraction.
For this instance, input “Photo of blonde hair woman.” to generate the image. The results are as follows:
On the left is the reference image, and on the right is the generated result.
Generating Images with the Same Composition
- Drag and drop the image onto the ControlNet menu screen.
- Check “Enable”.
- Select “Normal Map” under Control Type.
- Click the feature extraction button “💥” to perform the feature extraction.
This time, input “Photo of beautiful Japanese woman on the beach. White bikini” to generate the image. The results are as follows:
On the left is the reference image, and on the right is the generated result.
About the Preprocessor of ControlNet NormalMap
- normal_bae is a preprocessor trained to estimate normal maps using a visualization method from NYU-V2, adhering to a relatively precise protocol.
- normal_midas is a preprocessor designed to infer normal maps from Midas depth maps.
In general, normal_bae is considered to have higher accuracy. Even when looking at the validation results, the images generated from normal_bae have better quality. If in doubt, it is recommended to use normal_bae.
On the left is the image generated from normal_bae, and on the right is the image produced from normal_midas.
コメント