Using ControlNet 1.1: Guide & Examples (Anime to Live-Action, Corrections, Upscaling)

This article provides a comprehensive guide on using ControlNet Tile. While ControlNet is a valuable feature, its tile function may not appear as visually striking as OpenPose or canny. Consequently, users might be unsure about how to utilize it effectively.

By reading this article, you will gain insights into the following queries:

What exactly is ControlNet Tile?
When and where should ControlNet Tile be employed?
How can ControlNet Tile be used in various practical scenarios?

What is Controlnet Tile?
What can be achieved with ControlNet Tile
Using ControlNet Tile
ControlNet Tile Upscale: Image Upscaling and Detail Correction
How to Use 2: Changing Texture and Appearance
Explanation of different settings for ControlNet Tile

What is Controlnet Tile?

Controlnet Tile is an innovative technique that breaks down images into tile-like segments, enabling precise adjustments while preserving their characteristics with the help of ControlNet technology. Familiarizing yourself with the installation and utilization of this tool will expand the scope of image generation options. Now, let’s explore the different features and learn how to utilize Controlnet Tile.

What can be achieved with ControlNet Tile

ControlNet Tile offers several capabilities, including:

Fine adjustments of image details
Changing texture and style
Upscaling images

Fine adjustments of image details

Stable Diffusion may not effectively adjust image details, resulting in distortion or strange-looking features. To address this, adjustments can be made to the details after creating the overall framework. ControlNet Tile enables image generation followed by precise detail adjustments.

The left image shows the original image, while the right image is adjusted using ControlNet Tile.

Adjusting texture and style

ControlNet Tile is ideal for modifying texture and style. It allows you to adjust skin texture or achieve an anime-inspired look, granting you a high degree of freedom in transformations. You can also switch models, giving you the flexibility to alter the texture and style of the image.

The left image shows the original image, while the right image is transformed into tanned skin using ControlNet Tile.

The left image shows the original image, while the right image is transformed into a live-action style using ControlNet Tile.

Upscaling images

ControlNet enables image upscaling, offering the ability to adjust resolution and correct images.

Upscaling images with ControlNet provides advantages over simply generating high-resolution images with txt2img:

It offers greater ease in adjusting composition compared to generating high-resolution images.
Time is saved by experimenting with low-resolution images through trial and error.

Stable Diffusion proficiency varies across resolutions based on the training image data. For Stable Diffusion 1.5 or 2, 512✕512 resolution yields the best results.

By generating a basic composition at this resolution and then upscaling it, high-quality images can be produced.

This method not only enhances image quality but also saves time by allowing experimentation with small resolutions.

The difference between ControlNet Tile upscaling and Hires. fix

The Stable Diffusion Web UI includes an upscaling function called Hires. fix, which does not require the use of ControlNet Tile. Compared to this function, upscaling with ControlNet Tile is more convenient as it adjusts the details while preserving the original image state.

Using ControlNet Tile

Let’s learn how to use ControlNet Tile.

Installing ControlNet

ControlNet Tile is a feature of ControlNet, an extension of the Stable Diffusion Web UI. To use ControlNet Tile, you must have ControlNet installed. If you haven’t installed it yet, please refer to the following article for instructions on how to install ControlNet.

What is ControlNet? What Can It Do? A Comprehensive Guide to Installing ControlNet on Stable Diffusion Web UI (kindanai.com)

Downloading ControlNet Tile Model

To use ControlNet Tile, you need to download the ControlNet Tile Model for ControlNet Tile. Please download the following two files from the link below and place them in stable-diffusion-webui/models/ControlNet.

control_v11f1e_sd15_tile.pth
control_v11f1e_sd15_tile.yaml

lllyasviel/ControlNet-v1-1 at main (huggingface.co)

Steps for Using in WEB UI: txt2img(t2i)

Here, we will explain how to modify images using ControlNet Tile after generating images from prompts using txt2img. For now, we will confirm that the default values can be executed without changing them, and the detailed settings will be explained later.

Drag and drop the image onto the ControlNet menu screen. (Please note that the appearance may differ from the image below during drag and drop.)
Check the “Enable” checkbox.
Select “Tile” for Control Type.
Click the feature extraction button “💥”.
Generate the image again and see that Control Net Tile is applied.

Steps for Using in WEB UI: img2img(i2i)

Set the source image for conversion.
Set the image on the ControlNet menu screen. (Please note that the appearance may differ from the image below.)
Check the “Enable” checkbox.
Select “Tile” for Control Type.
Click the feature extraction button “💥”.
Generate the image again and see that Control Net Tile is applied.

ControlNet Tile Upscale: Image Upscaling and Detail Correction

Let’s discuss the process of image upscaling and detail correction using ControlNet tile.

Start by entering the prompt.
Adjust the resolution settings.
Choose the image you want to upscale.
Enable the feature.
Select the “Tile” option.
Extract the image features by clicking.
Generate the upscaled image by clicking “Generate.”

Results of ControlNet Tile Upscale

The image on the left is the original image, while the one on the right is the corrected image. Notice that the grainy texture in the left image has become smoother after applying the ControlNet tile upscale. This not only changes the resolution but also corrects the image details, enhancing the overall quality.

There might be slight changes in details, such as the tie design, but adjusting the settings can help maintain the original image’s design. For more information, please refer to the settings section later in the article.

Effects of Detail Correction in Images

By upscaling and dividing the image into tiles for detailed processing, ControlNet can achieve image detail correction. Note that while detail correction can be done without changing the resolution, it may result in some blurriness. For optimal image correction, it is recommended to choose a higher resolution setting, especially when dealing with low-resolution images.

How to Use 2: Changing Texture and Appearance

Usage 2-1: Changing Skin Texture (Try Tanning)

You can modify the texture of the skin in the generated image using ControlNet Tile. If the skin looks dirty, you have the option to correct it and make it appear beautiful or change its color. You can even make the skin look sweaty.

To demonstrate, let’s change the skin color to a tanned shade. Set the original image in the ControlNet menu and modify the prompt as follows (either txt2img or img2img will work):

Prompt: (brown skin, tanned skin: 2),1girl, a 20-year-old pretty Japanese girl in a classroom, wearing a school uniform, with a blackboard

By adjusting the Control Mode setting of ControlNet to “my prompt is more important,” you can have a stronger influence on the outcome.

Click “Generate” to create the image.

The image on the left is the original, and the one on the right is the image after the modification. You can observe that the skin has become tanned.

Usage 2-2: Converting Real Photos to Anime

By changing the model when altering the texture, you can achieve more flexibility in the conversion process. Here, we will explain the method of transforming real photos into anime-style images using an anime-style model. Additionally, we will explore the conversion from anime images to real photos.

Select an anime-style model.
Input the prompt.
Set the real photo.
Enable the option.
Choose “Tile.”
Click “Generate Image.”

Conversion from Real Photos to Anime Images

The image on the left is before the conversion, while the one on the right is after the conversion. You can observe that it has become an anime-style image.

If it appears unnatural, change the Control Mode to “My prompt is more important.”

Since it looks slightly unnatural, let’s adjust the settings. Switch the Control Mode to “My prompt is more important” and try again.

As a result of allowing the adjustments, the image now appears more natural as an anime-style picture.

Usage 2-3: Converting Anime Images to Real Photos

Conversely, let’s convert anime images into real photos.

Select a real photo model.
Input the prompt.
Set the anime image.
Enable the option.
Choose “Tile.”
Select “My prompt is more important.”
Generate the image.

Conversion from Anime Images to Real Photos

Explanation of different settings for ControlNet Tile

Preprocessor

resample

This is the default Preprocessor for sampling from images.

colorfix

Colorfix allows you to maintain the original image’s colors while making modifications. When using resample, colorfix can reduce the likelihood of changing the design of clothing details.

tile_colorfix+sharp

When using colorfix, the image edges can become blurry. By using sharpness, you can correct this issue. Generally, tile_colorfix+sharp produces higher quality images compared to colorfix.

Control Mode

There are three options for Control Mode. It adjusts the strength of preserving the features in Control Net.

Balanced is commonly used, but you can adjust it when the image appears unnatural while preserving the original information. For example, in the case of converting real-life images into an anime style, prioritizing the prompt can help resolve the unnaturalness. On the other hand, if you want to preserve the details of the original image design, choose “ControlNet is more important.”