In this article, we will provide a comprehensive guide on utilizing ControlNet Instruct Pix2Pix, a new feature introduced in ControlNet 1.1. By the end of this article, you will have answers to the following queries:
- What are the capabilities of ControlNet Instruct Pix2Pix?
- How can ControlNet Instruct Pix2Pix be effectively employed?
- What is the performance evaluation of Instruct Pix2Pix?
What is ControlNet Instruct Pix2Pix?
ControlNet Instruct Pix2Pix is a functionality that enables image rewriting based on given prompts. These prompts usually consist of instructional sentences like “make Y X” or “make Y into X”. A similar feature called img2img is offered by Stable Diffusion. As pix2pix is currently available in an experimental stage, it cannot be determined whether it possesses any specific advantages over img2img. Nonetheless, due to their distinct characteristics, it is recommended to experiment with both options to discover their effective uses.
ControlNet instruction for Pix2Pix:
This section showcases the various applications of Pix2Pix with ControlNet. Pix2Pix is most effective for making overall modifications to images rather than making partial corrections.
Expression Conversion:
Hairstyle Conversion:
Age Conversion:
Application of Concepts:
How to Use ControlNet to Instruct pix2pix
Installing ControlNet
To use ControlNet to instruct pix2pix, you first need to install ControlNet. ControlNet is an extension of the Stable Diffusion Web UI. If you haven’t installed ControlNet yet, please follow the instructions in the following article:
Downloading the ControlNet instruct pix2pix Model
To use ControlNet instruct pix2pix, you need to download the instruct pix2pix model files of ControlNet. Please download the following two files from the link below and place them in the stable-diffusion-webui/models/ControlNet directory:
- control_v11e_sd15_ip2p.pth
- control_v11e_sd15_ip2p.yaml
lllyasviel/ControlNet-v1-1 at main (huggingface.co)
Steps to Use ControlNet in the Web UI
- Enter the prompt you want to apply in pix2pix. Please input the prompt as an instructional sentence, such as “make her smile.”
- Open the ControlNet menu.
- Set the image in the ControlNet menu.
- Check the “Enable” option in the ControlNet menu.
- Select “IP2P” as the Control Type.
- When generating the image, a modified image using pix2pix will be generated.
Specific Usage of ControlNet Inpaint
Expression Conversion
Let’s explore expression conversion using ControlNet Inpaint.
In the prompt, enter “make her smile”.
The resulting images are displayed below. The image on the left is the reference image, while the image on the right is the output after applying pix2pix. As requested in the prompt, the person in the image is now smiling. However, the overall image has been greatly affected, and both the clothing and background have undergone significant changes.
Hair Style Change
Next, let’s try changing the hair style.
Enter “make her blonde hair” in the prompt.
The resulting images are shown below. The image on the left is the reference image, while the image on the right is the output after applying pix2pix. As specified in the prompt, the person in the image now has blonde hair. However, the overall image has been significantly altered, with everything appearing golden.
Age Change
Now, let’s try changing the age.
Enter “make her 10 years old” in the prompt.
The prompt has an effect on the person’s appearance, making them look younger. However, there are also many unnecessary modifications in the output.
Inpainting may not be suitable for color replacement. If you want to change the color of clothing, consider using ControlNet Tile or Reference Only.
Application of Concepts
An example introduced in the ControlNet 1.1 repository, “make he Iron Man”, yielded good results. The essence of Iron Man was captured well, so let’s see if specifying similar concepts can improve the results.
lllyasviel/ControlNet-v1-1-nightly: Nightly release of ControlNet 1.1 (github.com)
Let’s set the prompt as “make her Cinderella”.
As a result, the image obtains a Cinderella-like appearance, with changes in the background and other aspects, creating a harmonious overall composition.
Given that pix2pix has a substantial impact on the entire image, specifying a concept can be beneficial in achieving the desired results.
There are other functions available that are more user-friendly.
Based on the previous results, if you only want to convert specific parts of an image, there are alternative options that may be more suitable. These options include Tile, which allows for partial image conversion, reference only, and inpaint. The next section will introduce these functions in more detail.
For tutorials and examples of the Controlnet Tile feature, click here.
To learn about Controlnet reference only and its specific usage, click here.
For information on ControlNet inpaint, including its usage and a comparison of three processors, click here.
コメント