5-1 Let's learn about ControlNet(Stable Diffusion Practical Guide Table of Contents)

List

Chapter 1 Let's learn about image generation AI

• 1-1 Let's create images with AI
• 1-2 The birth and evolution of image generation AI
• 1-3 Let's think about the 'definition of AI' at the present time
• 1-4 Let's learn about artificial neural networks
• 1-5 Let's learn about the principle of image generation by the diffusion model

Chapter 2 Let's start building the environment

• 2-1 Let's prepare the environment to use Stable Diffusion
• 2-2 Let's build the environment using Google Colab
• 2-3 Let's build the Stability Matrix in the local environment
• 2-4 Let's create images with simple words
• 2-5 Download the model
• 2-6 Download the VAE

Chapter 3 Let's create images with prompts

• 3-1 Create images as you think using prompts
• 3-2 Build negative prompts
• 3-3 Let's create images 'as you think'
• 3-4 Let's increase the resolution of the image
• 3-5 Let's adjust various parameters
• 3-6 Let's test various prompts

Chapter 4 Let's create images using images

• 4-1 You can do it with img2img Let's figure out what's there
• 4-2 Let's create an image using Sketch
• 4-3 Let's edit an image using Inpaint
• 4-4 Apply Inpaint to modify an image
• 4-5 Extend an image using Outpainting
• 4-6 Increase the resolution of an image using img2img
• 4-7 Let's upscale with the extension function

Chapter 5 Let's use ControlNet

• 5-1 Let's learn about ControlNet
• 5-2 Download and prepare ControlNet
• 5-3 Create an image using ControlNet
• 5-4 Understand the role of the preprocessor

Chapter 6 Let's create and use LoRA

• 6-1 Let's learn what we can do with additional learning
• 6-2 Let's create an image using LoRA
• 6-3 Create your own dedicated painting style LoRA
• 6-4 Let's create various types of LoRA
• 6-5 Let's evaluate the learning content

Chapter 7 Let's use the image generation AI more

>>>What is ControlNet?

ControlNet is a derivative extension of Stable Diffusion, which was published in the February 2023 paper, <Adding Conditional Control to Text-to-Image Diffusion Modes>. Using this, you can input an image or pose in addition to the prompt to generate complex compositions that cannot be indicated by the prompt alone, poses that are difficult to explain with text, and images that reproduce the characters in the original photo.

The person who created this technology is lllyasviel (zhang Lvmin), the main author of the paper, and as of March 2024, lllyasviel has released the details of the technology and the model on GitHub and Hugging Face.

Github - Lllyasviel / ControlNet-v1-1-nightly

https//github.com/lllyasviel/ControlNet-v1-1-nightly

Additionally, Mikubill has created an extension to allow ControlNet to be used in AUTOMATIC1111 and released it as open source.

Github - Mikubill / sd-webui-controlNet

https://github.com/Mikubill/sd-webui-controlnet

ControlNet is an artificial neural network technology that adds spatial condition control by the diffusion model of Stable Diffusion. It consists of several types of 'preprocessors' such as openpose, which extracts poses from images, and canny, which extracts outlines, and the extracted information is used as condition control for image creation using txt2img. If you use each preprocessor separately according to its purpose, you can control the composition or pose that was difficult to handle in the existing txt2img to create an image as you intended.

>>>Know the difference between img2img and ControlNet

If you think about generating a new image based on an image and a prompt, you might think of img2img, but img2img and ControlNet are completely different technologies.

While img2img identifies the features of the entire input image to generate an image, ControlNet pre-analyzes the information (image, border, pose, depth, etc.) on the input image using a pre-processor, and extracts only the features of the specific elements that each pre-processor is responsible for from the input image to generate an image. For example, you can reproduce only the pose from the input image as shown below.

Original image Image source pexels	Pose of a character extracted with the openpose preprocessor