5-1 Let's learn about ControlNet(Stable Diffusion Practical Guide Table of Contents)

>>>What is ControlNet?

ControlNet is a derivative extension of Stable Diffusion, which was published in the February 2023 paper, <Adding Conditional Control to Text-to-Image Diffusion Modes>. Using this, you can input an image or pose in addition to the prompt to generate complex compositions that cannot be indicated by the prompt alone, poses that are difficult to explain with text, and images that reproduce the characters in the original photo.

The person who created this technology is lllyasviel (zhang Lvmin), the main author of the paper, and as of March 2024, lllyasviel has released the details of the technology and the model on GitHub and Hugging Face.

Github - Lllyasviel / ControlNet-v1-1-nightly

https//github.com/lllyasviel/ControlNet-v1-1-nightly

Additionally, Mikubill has created an extension to allow ControlNet to be used in AUTOMATIC1111 and released it as open source.

Github - Mikubill / sd-webui-controlNet

https://github.com/Mikubill/sd-webui-controlnet

ControlNet is an artificial neural network technology that adds spatial condition control by the diffusion model of Stable Diffusion. It consists of several types of 'preprocessors' such as openpose, which extracts poses from images, and canny, which extracts outlines, and the extracted information is used as condition control for image creation using txt2img. If you use each preprocessor separately according to its purpose, you can control the composition or pose that was difficult to handle in the existing txt2img to create an image as you intended.

>>>Know the difference between img2img and ControlNet

If you think about generating a new image based on an image and a prompt, you might think of img2img, but img2img and ControlNet are completely different technologies.

While img2img identifies the features of the entire input image to generate an image, ControlNet pre-analyzes the information (image, border, pose, depth, etc.) on the input image using a pre-processor, and extracts only the features of the specific elements that each pre-processor is responsible for from the input image to generate an image. For example, you can reproduce only the pose from the input image as shown below.

Original image Image source pexels	Pose of a character extracted with the openpose preprocessor

4-7 Let's upscale with the extension function(Stable Diffusion Practical Guide Table of Contents)

>>>What is Ultimate SD Upscale?

In Section 4-6, we looked at upscaling using img2img. This time, we will explain another way of upscaling using an extension called Ultimate SD Upscale. First, let’s start by installing the extension.

About the Stable Division Extension

Stable Diffusion and AUTOMATIC1111 are open source licenses that disclose the source code of the program. This means that anyone can use the program and develop and disclose additional programs. By using these additional programs, you can add new functions to suit your usage environment or expand them to use them more freely, so let's actively use them. However, some extended functions have restricted licenses, so be sure to check before installing.

>>> Install the extension.

All extensions can be installed via the WebUI. First, launch the WebUI and open the Extentions tab. Select Install from URL, and then click and copy and paste the code from the GitHub page of the extension you want to install into the URL for extension's git repository.

GitHub - Coyote-A/ultimate-upscale-for-automatic1111

https://github.com/Coyote-A/ultimate-upscale-for-automatic1111

Once the installation is complete, click on the Installed tab and check if Ultimate SD Upscale is in the list of installed extensions. If it is not in the list, click on Apply and quit to restart the WebUI and check again. For Colab, you will need to run the last cell of the notepad again and then access the new Gradio URL.

>>>Let's try Ultimate SD Upscale

Restart the WebUI. Since Ultimate SD Upscale is an extension used by img2img, go to the img2img tab. Select Ultimate SD Upscale from the Script menu at the bottom of the Generation tab.

In the Ultimate SD Upscale menu, set five additional parameters.

• Target size type

Specifies the size of the upscaled image.

From img2img2 settings: Matches the settings in the img2img tab.

Custom size: A slider is displayed, specifying the width and height sizes respectively.

Scale from image size: Specifies how many times the size of the input image will be scaled.

• Upscaler

Select the upscaler algorithm to use.

• Type

When upscaling, AI divides the image and processes it. The divided parts are called 'tiles'. Here, we specify how to divide the tiles. For undulation, we can keep the initial state.

• Tile width/height

Specifies the size of the tiles.

• Mask blur

Specifies how much to blur the boundaries between tiles.

Okay, let's actually use it. First, prepare the image created with txt2img. Upload it to the img2img tab, and set img2img → Resize by: 2, Denosing strength: 0.5. Select Ultimate SD Upscale → Upscaler: R-ESRGAN 4x+, and create it while maintaining the default settings for other things.

Let's check the generated image

I just chose ESRGAN 4x+ as my upscaler, but I wonder what kind of images other upscalers will produce. Let's test it out.

Lanczos	Nearest	ESRGAN_4x	LDSR
R-ESRGAN 4x+	R-ESRGAN 4X+ Anime6B	ScuNET GAN	SwinlR 4x

4-6 Increase image resolution with im2img(Stable Diffusion Practical Guide Table of Contents)

>>>Why is upscaling necessary?

Upscaling refers to expanding an image to increase its resolution. The image generated using Stable Diffusion has an initial size of 512x512 (1024x1024 in SDXL), which is quite low in resolution. Upscaling is removing noise from an image generated with such low resolution and generating a high-resolution image.

Then why not generate a high-resolution image from the beginning? This is because inferring and generating a high-resolution image from the beginning requires a very large computational space. Generating a high-resolution image requires a huge VRAM, that is, an expensive GPU with high computational power, and it costs much more. Also, there is a limit to the size of the learned image, so if you try to generate a high-resolution image, the image itself may be damaged.

Hires. Fix, which was explained in Chapter 3, is also one of the various upscaling methods, and here we will explain upscaling using img2img.

>>>Let's try upscaling using img2img

Upscale the generated image with img2img. First, create an image in the txt2img tab. Send the generated image to the img2img tab by clicking Send to img to img in the viewer.

Prompt

score_9, score_8_up, score_7_up,  realistic, realism, raw photo of 20 years old woman in off-shoulder, close up portrait, deep blue sky, cloudy sky, outdoor, high key light, soft shadow, Fiery clouds

Negative Prompt

score_6, score_5, score_4, source_pony, (worst quality:1.2), (low quality:1.2), (normal quality:1.2), lowres, bad anatomy, bad hands, signature, watermarks, ugly, imperfect eyes, skewed eyes, unnatural face, unnatural body, error, extra limb, missing limbs, painting by bad-artist, 3d

In the img2img tab, specify the size after enlarging with the Resize to parameter. This time, specify it as 1024x1024, which is twice as much. The same thing happens if you specify the enlargement ratio with Resize by.

Since we want to generate an image that is as similar to the input image as possible, leave the prompt unchanged and set Resize mode: Just resize, Denoising strength: 0.5. The smaller the Denoising strength value, the more faithfully the input image is reproduced before enlargement, but if it is too small, the output image will be unclear like a low-resolution image even if the size is large. After completing the settings, click Generate to generate the image.

4-5 Expand the image with outpainting(Stable Diffusion Practical Guide Table of Contents)

>>>What is Outpainting?

In the previous section, we learned that using Inpaint, we can select a part of the input image and create it. Using Outpainting, we can create a new image that continues around the input image. This is useful when you find out later that there is a broken part in the subject of the image or that the image size is small.

Here, we will proceed with outpainting using the most practical outpainting tool, Poor man's outpainting. First, open the img2img →img2img tab and upload the input image.

>>>Let's try Outpainting

Here, we will proceed with outpainting using the most practical outpainting tool, Poor man's outpainting. First, open the img2img →img2img tab and upload the input image.

Prompt

masterpiece, best quality, ultra detailed, fantasy,colorful, portrait, 1 girl, straw hat, white shirt,white background

Negative Prompt

worst quality, low quality, normal quality, easynegative, multiple girls,

From the Script menu, select Poor man's outpainting. The Poor man's outpainting menu will appear, and you can set four additional parameters.

• Pixels to expand: Specifies the range to expand in pixels.

• Mask blur: Specifies how to blur the border between the input image and the newly created area.

• Masked content: Specifies the method to supplement the range to expand. There are four types of supplementation methods, just like Inpaint.

• Outpainting direction: Specifies the direction to expand in: up, down, left, or right. Multiple directions can be selected at the same time.

This time, let's expand the image at the bottom by setting Pixels to expand: 128, Mask blur: 4, Masked content: fill, Outpainting direction: down without changing the prompt.

If you repeatedly run Outpainting on an image that has been Outpainted, you can further expand the image. If the boundary with the expanded part is noticeable, try adjusting the Mask blur or Pixels to expand values.

4-4 Modify images using Inpaint(Stable Diffusion Practical Guide Table of Contents)

>>>Change expression

In the previous section, we explained how to mask with Inpaint and add a new prompt to reflect the content in the generated image. You can also use this feature to modify only the expression. For example, let's change the shape of the mouth of the input image to make it a smiling face. Upload the image to Inpaint, mask around the mouth, modify the prompt, and proceed with the image generation

.

Prompt

masterpiece, best quality, ultra detailed, fantasy,colorful, portrait, 1 girl, straw hat, white shirt,white background

Prompt

masterpiece, best quality, ultra detailed, fantasy,colorful, portrait, 1 girl, straw hat, white shirt,white background,smile

>>>Add accessories

If you want to add hair decorations, accessories, ribbons, etc., just mask the area you want to create and give it a prompt. This time, let's add an earring to the right ear. Mask the area below the earlobe where you want to add the earring, and input the prompt 'star-shaped earring' to create the image.

Prompt

masterpiece, best quality, ultra detailed, fantasy,colorful, portrait, 1 girl, straw hat, white shirt,white background,star earring,

4-3 Let's edit the image with Inpaint(Stable Diffusion Practical Guide Table of Contents)

>>>What is Inpaint?

Inpaint is a function that specifies a certain area (masking) of an input image, creates only that area, and then changes it. You can later add or delete the background and details of the input image.

>>>Let's try Inpaint

Now let's actually use Inpaint. Open the img2img tab, and select Generation → Inpaint tab. Drag and drop the input image into the area, or click inside the area to upload the file directly.

Prompt 

masterpiece, best quality, ultra detailed, 1girl, upper body

Negative Prompt

worst quality, low quality, normal quality, easynegative,

When you upload the input image, a toolbar will appear at the top right of the canvas, just like in Sketch. The operation method is the same as in Sketch. Color and mask all the parts you want to modify in Illustrator. This time, let's mask them by surrounding the background part.

• Mask blur

Specifies the range to blur the boundary with the masked area in pixels.

• Mask mode

Sets how to handle the masked area.

impaint masked: Creates the masked area.

Inpaint not masked: Creates the area outside the masked area.

• Masked content

Selects how to handle the masked area in advance.

fill: Creates an image by filling the masked area with a color that is the same as the surrounding pixels.

original: Creates an image by leaving the masked area as the input image.

latent noise: Creates an image by filling the masked area with random noise.

latent nothing: Creates an image without giving any features to the masked area.

• Inpaint area

Sets the range to be the input target.

whole picture: Extracts features from the entire input image and creates an image.

Only masked: Creates an image by extracting features only from the masked area.

• Only masked padding, pixels

When Inpaint area: Only masked is selected above, specify how much to expand the masked area in pixels.

This time, I set Mask blur: 4, Mask mode: Inpaint masked, Masked content: original, Inpaint area: Whole picture.

First, let's click the Generate button without entering the prompt. Then, the image is generated by only taking advantage of the features of the input image without being instructed by the prompt, so there is no significant change in the masked range.

Next, let's change the background by writing a prompt this time. Let's change the background around the character by adding Prompt blue sky. The blue sky appears in the masked area as the prompt instruction is reflected.

Prompt 

masterpiece, best quality, ultra detailed, 1girl, upper body, blue sky,

Negative Prompt

worst quality, low quality, normal quality, easynegative,