How to use the powerful Flux.1 Tools to modify images in ComfyUI
Since we introduced the Flux.1 ControlNet in our last article, we will now introduce the 🔗Flux.1 Tools Tools available from Black Forest Labs. Flux.1 Tools allows you to use Flux.1 models for “Inpainting/Outpainting“, ”Redux“ which is similar to the IP Adapter, and ”Canny or Depth” to reference input images. Flux.1 Tools requires a PC with high specs, but it is a very powerful tool, so read the article to learn how to use it.
Download the Flux.1 Tools model
Models for Flux.1 Tools are available on Black Forest Lab’s HuggingFace. Download the model you want to use. To download an official model, you need to accept the terms and conditions of the Black Forest Labs license. If you don’t want to accept the terms and conditions of the license, you can download the same model from 🔗theally of Cvitai.
Fill (Inpainting)
Fill (Inpainting) is a model for inpainting and outpainting using Flux.1 Download it at /ComfyUI/models/diffusion_models. An unofficial lightweight version is also available if you are using a low spec PC.
Redux
Redux is a model with features similar to IP Adapter. To use it, you will need a clip vision model as well as a diffusion model. Download them in /ComfyUI/models/diffusion_models. Download the following clip vision model in /ComfyUI/models/clip_vision.
Canny
Canny is a model that is generated by reference to a line drawing extracted from an image; download it to /ComfyUI/models/diffusion_models.
Depth
Depth is a model that is generated by reference to a depth map extracted from an image; download it to /ComfyUI/models/diffusion_models.
Canny (LoRA)
The Canny and Depth models in Flux.1 Tools can be applied using the LoRA model in addition to the diffusion model. This model will be very versatile since it is used as a LoRA.
Download Canny (LoRA) in /ComfyUI/models/loras.
Depth (LoRA)
Download Depth (LoRA) in /ComfyUI/models/loras.
Review the official ComfyUI sample.
For official ComfyUI examples, download the images shown on the linked page and load them into ComfyUI’s Load or drag-and-drop them into the UI. Input images need to be downloaded from other examples in ComfyUI_examples, so put them all together on the DCAI’s drive.
Fill (Inpainting)
The Fill (Inpainting) model generates a mask range for the input image according to the prompt. In this example, the input image has already been masked. To edit the mask, right-click on the “Load Image” node to open the “Open in MaskEditor” menu.
Please refer to the following article for a detailed explanation of ComfyUI inpainting.
This workflow features the following nodes in addition to the normal Flux.1 workflow
- Differential Diffusion：This node improves the inpainting quality. By passing this node, the quality of the input image and the inpainting can be matched.
- InpaintModelConditioning：Apply prompts, input images and masks.
Fill (Outpainting)
This is a workflow for outpainting that also uses the Fill model.
In addition to the two nodes introduced earlier, the following nodes are used in this workflow.
- Pad Image for Outpainting：Extend the input image according to the settings. In this example, it adds 400 pixels to the left, right and bottom. feathering allows you to blur the borders, so set the value to make the borders less noticeable.
Redux
Redux is a model that functions like an IP Adapter. It is used when you want to generate different versions of an input image or generate different images in the same style.
This workflow features the usual Flux.1 workflow plus applying styles at nodes enclosed in Redux Model groups.
- Load CLIP Vision：Loading the downloaded
sigclip_vision_384.safetensors. - CLIP Vision Encode：The input image is converted to conditioning (prompt data) using sigclip_vision_384 described earlier.
- Load Style Model：Load the downloaded
flux1-redux-dev.safetensors. - Apply Style Model：Apply the data converted by CLIP Vision Encode using the Style Model. Adjust the strength of the application using the strength value.
Canny and Depth
This workflow is an example for Canny, using the flux1-canny-dev.safetensors model to reference line drawings in the input image.
If you want to use flux1-depth-dev.safetensors, you need to install a custom node “ComfyUI’s ControlNet Auxiliary Preprocessors” to extract the depth image from the input image. Please refer to the following article for details on how to install the custom node.
![Featured image of [AI Illustration] Introduction of Recommended Custom Nodes for ComfyUI](https://dca.data-hub-center.com/content/uploads/2024/08/eye_catch_comfyui-recommended-custom-nodes_en.jpg)
Once the custom node is installed, the depth map is extracted from the input image using a preprocessor of your choice such as the “Depth Anything” node and sent to “InstructPixToPixConditioning”. Then load flux1-depth-dev.safetensors into the “Load Diffusion Model” node and you are ready to use it.
Canny and Depth (LoRA)
The Flux.1 Tools models all have a capacity of 22.1 GB, as does the regular Flux.1 Dev, making it difficult to generate on a low-spec PC. However, the “Canny” and “Depth” LoRA models can be used with Flux.1 [schnell], FP8 (Dev/Schnell), GGUF, etc., which are easy to handle even with low specs, so they are recommended.
If you want to use canny, change the lora_name of the “LoraLoaderModelOnly” node to flux1-canny-dev-lora.safetensors and use image > preprocessors > canny for the input image to extract the line drawing.
This workflow features the following nodes in addition to the normal Flux.1 workflow
- InstructPixToPixConditioning：Apply canny and depth data to conditioning data.
Introduction to practical workflows using Flux.1 Tools
Here is a workflow using “Flux.1 Tools”. Normally, we try to keep the workflow as simple as possible with as few custom nodes as possible, but this time we have made it a bit more complex with 5 custom nodes. The workflow uses Flux.1 Fill to refine and extend the input image. Use the techniques described to generate high quality illustrations with Flux.1 detail using “Animagine XL”, “Pony Diffusion V6 XL” and “Illustrious-XL” images.
Expansion direction is supported in vertical and horizontal direction; not in 3-direction or all directions. Also, the expansion range can only extend the same pixel in each direction.
- Detailing the background of input images using inpaint and outpaint
- Use flux.1-fill-dev-gguf to reduce load on VRAM
- Protecting the face of the input image with the implementation of SAMDetector
The workflow is available on Patreon, but only paid supporters can view and download it.
Even if you cannot download the workflow, you can construct it yourself by looking at the explanation, so please continue on to the explanation.
Installation of custom nodes
The following custom nodes are required and should be installed. If you have already installed it, update it to the latest version to avoid errors.
- 🔗ComfyUI-Impact-Pack: Used to create facial mask.
- 🔗ComfyUI-Impact-Subpack: Since V8.0, the UltrayticsDetectorProvider has been separated.
- 🔗ComfyUI-Custom-Scripts: Used in calculations to center the original image on the expanded image.
- 🔗ComfyUI-Crystool: Used to control workflow, extract image resolution, and switching between horizontal and vertical.
- 🔗ComfyUI-GGUF：Uses to load Unet and CLIP of GGUF format.
About the models in use
The following model is used: GGUF model is Q8_0, but please change it according to your machine spec.
- 🔗second-state/FLUX.1-Fill-dev-GGUF/flux1-fill-dev-Q8_0.gguf
- 🔗city96/t5-v1_1-xxl-encoder-gguf/t5-v1_1-xxl-encoder-Q8_0.gguf
- 🔗comfyanonymous/flux_text_encoders/clip_l.safetensors
- 🔗UltraSharp
About the input image
Input images are generated at 1024×1024 using Animagine XL and scaled up to 2x2048x2048 pixels using HiresFix. Download the images from DCAI’s Drive. *The above images are scaled down, so we recommend using the Drive images if possible.
Explanation of Graphs
Input Image
The Input Image group loads the input image and reduces or enlarges it to 1024px. A square image is preferred, but a rectangle is also acceptable. The height of “Upscale Image” is set to 0 so that the ratio does not change when a rectangle is input.
Basic Nodes
Basic shared nodes are grouped together. This workflow uses a regular KSampler, so a negative prompt is required. So the prompt is not reflected, but it is necessary, so therefore it is included in this group.
The GGUF model uses Q8_0. However, 24GB of VRAM is recommended, so use Q6_K or Q5_K_S if your specs are not met. The details are explained in the following article.
![Featured image of How to use high performance GGUF with ComfyUI Flux.1 [schnell]#To use GGUF with ComfyUI Flux.1 [schnell]-Quantization](https://dca.data-hub-center.com/content/uploads/2024/11/eye_catch_use-high-performance-gguf-comfyui-flux-1-schnell_en.jpg)
Face Detect 1
This is a group of nodes around SAMDetector in the custom node “Impact-Pack”. Normally, this node is used to repair the face of a Face detailer, but in this case, it is used to protect the face by creating a face mask.
If the mask cannot be successfully extracted, use the “SAMDetector (combined)” detection_hint or threshold to make adjustments.
The “Mask Preview” node is a group node that combines the “Convert Mask to Image” and “Preview Image” nodes. We will explain how to create a group node later.
InPaint
The InPaint group inverts the mask created in the Face Detect 1 group and inpaints the non-face areas.
The prompt should include a brief prompt for the image you wish to generate and a quality tag, as shown below.
Beautiful scenery of a medieval village.
detailed illustration, highly detailed, very detailed, intricately detailed, hyper detailed, finest details, intricate, wide angled, a rich and complex image of nature, realistic, captivating, fantastical, maximalist style, sharp focus, harmony, serenity, dynamic lighting, superb composition, highest aesthetic, stylized, clear, colorful, ultra quality, 8k, best quality, a breathtaking masterpiece.For “FluxGuidance” we use a larger value of 20. The node default value of 3.5 is also usable, but increasing this value will increase the level of detail.
The denoise value of “KSampler” sets how much of the original image is retained. 0 for the original image is retained, 1 for a completely different image except for the protected faces. The quality did not change much when the steps were increased. cfg should be used with 1 for Flux.1.
OutPaint
The OutPaint group allows the image generated by the InPaint group to be extended by a specified number of pixels in any direction. By default, the image is extended horizontally by 400 px in each direction.
The prompt applies only to the extended area. The default is to use the same prompt as inpainting, but you may want to use a simple prompt.
The “H/V Switcher True = Horizontal” node specifies the direction of extension; if the value is true, the extension is horizontal; if the value is false, the extension is vertical.
The “H/V Param Switcher (Group)” is a group node that brings together “Primitive integer [Crystools]” and “Switch any [Crystools]”. It extends the pixels of value according to the value of “H/V Switcher True = Horizontal” mentioned earlier.
The “Pad Image for Outpainting” node specifies pixels to be outpainted. This time, left/top/right/bottom are externalized using Convert Widget to Input from the node’s right-click menu, and input from the H/V Param Switcher (Group).
Upscale
This is the upscaled node group that we always introduce at DCAI: using 4x-UltraSharp to enlarge the image by 4 times, then reducing it to 3 megapixels.
Finishing
In the Finishing group, the entire image is generated one last time to blend in the extended areas and upscale from the outpainting. The inpainting is also used here to protect the face.
This group should also adjust the finish with “FluxGuidance” and “KSampler” denoise.
Face Detect 2
Creates a mask to protect a face in upscaled resolution. bbox_detector and segs are shared from Face Detect 1. The created masks are used for inpainting in the Finishing group and face replacement in the Face Swap group.
Face Swap
In the Face Swap group, the input image is made the same size as the image generated in the Finishing group and the face part is replaced. This is a bit complicated, so we use colors to categorize by type.
- Cyan: Nodes related to Images generated by Finishing group
- Red: Nodes related to input image
- Yellow: Nodes related to calculation
Basically, by placing the input image, adjusted to match the resolution, at the center of an EmptyImage that corresponds to the final size, the input image aligns with the mask position. Using the “ImageCompositeMasked” node, the mask created with the Face Detect 2 group is applied to replace the face.
The “🪛 Switch any” node is used to switch between vertical and horizontal orientation depending on the value of “H/V Switcher True = Horizontal” in the OutPaint group.
The “Math Expression 🐍 – Centering” node calculates the offset value to center the input image. The formula is simple: (width of final image - width of input image) / 2 for a landscape image.
The connection of the nodes is complicated, so let’s look at the detailed image to see how to connect them.
Preview
A preview of each graph is grouped together. If you want to save the image automatically, replace it with a “Save Image” node.
About Group Nodes
A group node is a feature of ComfyUI that allows multiple nodes to be combined into one node. In this case, the “Mask Preview” node and the “H/V Param Switcher (Group)” node will become such nodes, so let’s take a look at how to create them.
How to create a Mask Preview node
This is a simple group node, but the group node basics are easy to understand.
- Place “Convert Mask to Image” and “Preview Image” to connect IMAGE out and images in.
- With two nodes selected, right-click to display the menu.
- Select “Convert to Group Node” from the menu and enter
Mask Previewon the node name input screen.
This completes the process. The nodes you have created can be used anytime within the same workflow, as long as they are in Add Node > group nodes > workflow.
How to create a H/V Param Switcher (Group) node
This one is a little more complicated, but the basics remain the same.
- Place 2 “🪛 primitive integer” and 4 “🪛 switch any” as shown in the image below. The “🪛 Primitive integer” should have two values,
400and0, and the 400 value should be connected to true/false/true/false from the top, referring to the image. 0 should be connected to the opposite side. This is because the out of “🪛 Switch any” connects to the left/top/right/bottom of the “Pad Image for Outpainting” node, so that the value is applied to left and right when True. - We want to externalize the boolean of the “🪛 Switch any” node, so we change it from the node’s right-click menu.
- Double-click the display name of each node and change it as shown in the image for easy editing later.
- With all nodes selected, right-click to display the menu.
- Select “Convert to Group Node” from the menu and enter
H/V Param Switcher (Group)on the node name entry screen. - The node display is confusing, so right-click on the canvas or node and select Manage Group Nodes from the menu.
- You can call the Manage Group Nodes as shown in the image to make it easier to work with the nodes, and check the display of each node with the Save button after each operation.
- First of all, remove the “int 0” in the Primitive integer [Crystools] that does not need to be changed. Select “int 0” from the list on the left and uncheck the Visible checkbox on the Widgets tab.
- Change the display name of “int 400” from int to
value. - Switch any [Crystools] from Left to Bottom like
boolean left. - Switch tabs from Widgets to Outputs and change the display of outputs from Left to Bottom like
LEFT, which will be displayed according to the ComfyUI node format where inputs are lower case and outputs are upper case. - When you are satisfied that the display is correct, press the Save button and close Manage Group Nodes with the Close button.
The H/V Param Switcher (Group) node is now complete.
Instructions for use
When using workflow, you can stop the process with “Control Bridge (InPaint)” and “Control Bridge (OutPaint),” so first switch mode from Active to Stop/Mute/Bypass for both.
- Load the input image into “Load Image” in the Input Image group.
- Switch the contral_after_generate of “KSampler” in the InPaint group to
randomize. Enter the prompt and adjust the guidance of “FluxGuidance” and denoise etc. of “KSampler” and repeat until a satisfactory image is generated. - Once a satisfactory image is generated, right-click on the resulting image you like from the Queue list in the side menu and call it up from “Load Workflow”.
- A new workflow will open, copy the seed value of “KSampler” and close the workflow.
- Switch the “KSampler” contral_after_generate to
fixedand paste the seed value you just copied. - Switch the mode of “Control Bridge (InPaint)” to
Activeand move to the OutPaint group. - Set the direction you want to extend with the “H/V Switcher True = Horizontal” value.
trueis horizontal,falseis vertical. - Enter the size you want to expand in the “H/V Param Switcher (Group)” value. By default, the left and right sides are expanded by 400px each.
- Feathering in “Pad Image for Outpainting” blurs the boundary between the original image and the generated image to create a smooth connection.
- Enter the prompt and adjust the “FluxGuidance” guidance.
- When you are satisfied with the image, switch the mode of “Control Bridge (OutPaint)” to
Activeand move to the Upscale/Finishing group. - The Finishing group is similar to the InPaint group, with prompts to adjust the guidance of “FluxGuidance” and the denoise of “KSampler”, etc.
The above produces an image that has been detailed with Flux.1 while maintaining the face of the input image.
Final Results
Flux.1 quality images were generated while retaining the image taste of Animagine XL.
A sample generated vertically is also shown. The parameters have been changed to 725205388587662 in the OutPaint group seed, and it resulted in a beautiful vertical image.
Conclusion
Flux.1 Tools gives you much more control over your production, like ControlNet, and allows you to modify images more efficiently. However, the Flux.1 model is still very slow and will be difficult to run on a PC with less than 24 GB of VRAM unless you install the GGUF or FP8 models.
Also, we think it is a good use of Flux.1 Tools to improve quality, as there are methods that make the most of the SD1.5 and SDXL models, such as the DCAI workflow we have introduced.
