Image Generation Workflows with ComfyUI

ComfyUI is a very powerful user interface to run open source diffusion model  image generation locally on your computer. It uses an extensible node based framework based on Python classes. This allows a lot of control over image generation. It works with a wide range of different diffusion model checkpoints and LoRAs (Low Rank Adaptions) allowing a lot of different effects. Lets look at some hints and tips. I will also provide a good general purpose workflow for image generation in ComfyUI.

Getting Started

There are numerous handy guides to getting started with ComfyUI. These include articles ,  video tutorials and the comfyUI documentation itself.  Here are some important points to note:

• ComfyUI is available in a portable version which is very convenient since it means you can easily move your whole ComfyUI setup about by just moving its containing folder about. This is useful because model checkpoint files tend to be large so you may want to be able to move comfyUI to a spare drive
• ComfyUI will use GPU acceleration if available. Note that there is a distinct hierarchy to the useability of GPU acceleration. NVIDIA CUDA cards are at the top of the tree.
• workflows can be saved to and read from JSON using the Save and Load buttons. Note that the save button does not save images you a use a Save Image node for that
• When you save a .png image output from ComfyUI, the structure of the node based workflow used to generate it is saved as metatadata in the image. Placing the image into the comfyUI/input folder will then allow the relevant workflow to be loaded and examined using the Load button in the comfyUI interface.
• One of the best sources for checkpoint models and LoRAs is the CIVITAI website, there are also often demonstration .png images with loadable workflows for the model in question on the model’s page.

Workflow Basics

A basic workflow will load a model checkpoint, take a prompt input, feed both of these items and an empty latent image into a K-Sampler, decode the resulting output and save it to a file. An example of this kind of extremely basic workflow can be found on my Github here. By allowing the seed of the KSampler to increment, each time you click Generate, a new image will be created in accordance with the prompt. Thing to remember for basic workflows include:

• Nodes can only be linked to nodes of the same type. Even though Nodes are defined in Python under the hood, this does not mean that types are mutable. You may need to use converter nodes where types do not match. If two nodes will not join this may be a sign you are missing an element from you workflow or that your workflow is otherwise incorrect.
• enclosing something in brackets can increase the strength of that part of the prompt. the default is a factor of 1.1 but other factors can be specified. E.g. (delapidated:1.3)
• curly braces and pipes can be used to specify a random choice between elements e.g.{red|green|blue} this can be useful where you want some random but limited variations to your generations
• Right clicking on the canvas lets you select nodes by menu. Double clicking on the canvas lets you select nodes by search
• Right clicking on a node gives you a menu to modify the node. This allows you to colour a node. It also lets you convert the widgets on some nodes into node inputs so the value can be fed from another node rather than defined on the node
• A node’s output can lead to multiple other nodes inputs, but an input can only lead back to one output

Flux Schnell

Flux Schnell is a powerful new open weights model which gives good general purpose results with ComfyUI. It does not have negative prompts so it needs a rather different workflow to most other models. A good guide can be found here.  A good workflow can be derived by loading it from the example picture here. The nice thing about Flux Schnell is that it gives good results without too much prompt engineering or use of LORAs, however even the Schnell version can be more resource intensive than some less capable models. There are also Flux Dev and Flux Pro versions , but these are not open licensed and require serious hardware to run at reasonable speeds.

Installing Custom Nodes

For more advanced workflows you will need to install some custom node packages. To allow this you need to install the ComfyUI-Manager to give you the Manager button. You can find the instructions and package to do this here.

Once this is installed you can search for the following packages in the package manager (amongst others) and install them:

• ComfyRoll Studio: useful nodes for working with LoRAs amongst other things
• ComfyUI Inspire Pack: contains nodes for processing lists of prompts rather than only being able to schedule one prompt at a time
• WAS Node Suite: Lots of useful nodes including text concatenation
• pythongosssss/ComfyUI-Custom-Scripts: A wide variety of useful helpers including a maths node and a node for displaying the text output from other nodes (useful where you have compiled a prompt form several source nodes

Note that you will need to both restart comfyUI and refresh the comfyUI interface browser page before you will be able to use the new nodes. With all this installed (and a suitable checkpoint and LoRAs) you should now be ready to use the more advanced workflow presented below. Notice how the text_data_opt input has been used to override the details which would be read from file. The format for prompt alterations can be found here or by looking at the example.

More Advanced Workflows

A more advanced workflow can allow you to use a series of LoRAs. It could also allow you to run a series of slightly modified prompts automatically rather than having to run each one by hand. An example workflow of this type can be found on my Github here.

Here are a couple of things to bear in mind with more complex workflows:

• Checkpoint models may have a prefered generation size and aspect ratio. Be sure to read the documentation about the model to get the most out of it
• LoRAs (Low Rank Adaptions) tend to be model specific, or at least model family specific. If you use a LoRA with a model and get nonsense output check compatibility on CIVITAI
• LoRAs generally need to be applied at a CLIP (Contrastive Language-Image Pretraining) layer of -1 or -2.
• Likewise VAEs (variational autoencoders) for latent space decoding are model specific so either connect the VAE output of your loaded checkpoint to the VAE input of your VAE Decode node or ensure the VAE you are using is compatible
• It can be worth playing about with the weights to see if that gives you better results. Experimentation is key to good results.
• You can always recover an old workflow by putting the output PNG into the input folder and using the Load button to load its workflow

Here is an output from this more advanced workflow, its not perfect, but its pretty close.

And of course this png has the creation metadata included. You could place this image in the input folder of your comfyUI folder and Load it to get a copy of the workflow used to create it. The picture at the top of the article has the basic workflow embedded.

ComfyUI API

Additionally ComfyUI includes an API. Therefore you can use Python via its API to control ComfyUI . This allows you  even more complex control of ComfyUI. Also you can use this to integrate it with other AI tools. Note that when working via the API you need to enable the dev mode options and  export the workflow JSON in a different form, instructions are here. You can find more information on working with the API in this tutorial and a basic example script can be found here.