World of LoRAs
The Good, the Bad and the Downright Infuriating
If you have started to use ComfyUI the chances are you will have come across LoRAs or Low-Rank Adaption models. These marvellous little creations, which are a bit like a filter on top of the base model, can guide and direct a diffusion model using additional trainable parameters, resulting in output tailored to particular styles or content.
One reason for their popularity is that they can be trained relatively easily (more on training in a later post), compared to a full model refinement, which is very difficult, time-consuming, and computationally expensive.
This relative generation ease means there are many LoRAs available, good sources include Hugging Face and Civitai Models with LoRAs covering every conceivable need. Be warned Civitai in particular has a wide range of LoRAs with many NSFW. LoRA selection can be a lottery as some are excellent, others however can be very poor. It’s important to check reviews and just do some testing.
Many LoRAs are trained for artistic styles, types of clothing, realism and skin detail, etc. but others change the way the base model works such as increasing depth of field or speeding up the sampling process reducing the number of required steps.
LoRAs are not perfect though, aside from the variable quality of some, all LoRAs tend to alter the composition of the output to some degree, and not in a predictable way.
Using LoRAs within ComfyUI is straightforward as there are many nodes which can load a LoRA model (or multiple LoRAs). In terms of workflow these typically sit after the base model and clip loaders but before clip encoding and samplers. There are some more advanced techniques when it comes to LoRA scheduling and LoRA masking both of which are very powerful but a lot more complicated.
Settings
When loading a LoRA there are two settings - strength_model and strength_clip, however, many loaders just present this as one option of ‘strength’ and set both of the settings to this value.
The logic behind this is that in the vast majority of cases you will want to send the same strength of the LoRA to both the diffusion model and the CLIP model. The exact way these settings work is a little unclear but the basic principle is the strength for the model changes how much of the LoRA is applied and the strength for the CLIP controls the strength of the activation. Sometimes it is interesting to play with varying the settings separately but on the whole it doesn’t seem to make much difference.
Although the strength settings can range from -100 to +100, anything less than 0 turns the LoRA off, and the typical range is 0-2 with 1 as the default.
The second part of using a LoRA is activating it - many LoRAs do not have any affect until you use a trigger word in the prompt. This may just be the name of the LoRA but some LoRAs have a number of different trigger words, each triggering the LoRA to apply a different affect. Some LoRAs do not require a trigger word and always take affect no matter what the text prompt is.
Can you use more than one LoRA? Yes absolutely, but the way they interact with each other can get very messy. Not only do you have the relative strengths of the LoRAs but the order in which the LoRAs are applied also has an impact. Mixing LoRAs which overlap in what they are trying to do can also result in undesirable and unexpected output.
This interaction aspect between the base model and then between the different LoRAs can be very infuriating as the output can change considerably from the original output without the LoRAs applied and deviate considerably from the prompt.
These interactions are unpredictable, with trial and error being the only approach, tweaking strengths, adjusting Flux Guidance and even just changing the order of the LoRAs. My experience so far is to try and minimise the number of LoRAs used, rarely using more than two, and in most cases these will be aimed at very different aspects. There can be a tendency to reach too quickly for a LoRA when actually tweaking some other settings can achieve the desired result.
Examples
After all of this out of the many hundreds of LoRAs available which are some of my current favourites in my toolbox? (note this is ever changing!)
Shakker-Labs FLUX.1-dev-LoRA-AntiBlur
One of Flux’s traits is to create a small depth of field leading to blurry backgrounds. Sometimes a shallow depth of field is desirable but often it is difficult to persuade Flux to use a larger depth of field when you want it to, that’s where the Anti-Blur LoRA comes in. It cannot completely solve the issue but it does help considerably, I often use it with a strength of 1 but it can go up to 2.
Byte Dance Hyper-FLUX.1-dev-16steps-lora
The Hyper Flux LoRA is something a little different, its aim is not to change the output but simply reduce the number of sampler steps required - in this case to 16 from a more typical 25-30 - without impacting quality. Not only does this work but I think it can actually improve the quality and detail. For those desperate to save a few more seconds there is even an 8 step version for quick testing.
AIDMA Realistic People Photograph-FLUX-V0.2
This is one I use sparingly, typically only with the base Flux.1 Dev model and not when using refined models. It can help give a more natural look to photographs. There are many ‘realism’ LoRAs around with revisions and new ones appearing all the time, it’s a case of experimenting and seeing which ones work for you.
AIDMA also has a number of other LoRAs including Hyper Realism, Realistic Skin and Flux Pro Ultra, all of which are worth experimenting with.
Comic/Anime/Cartoon LoRAs
Anime/Comic style imaging is a new area for me and I include it here more to show the variety of LoRAs available. I have been experimenting with a couple - Real Comic and Cyberpunk Anime Style - so I have called them out but there are plenty more to experiment with. As well as trigger words many of them can be guided by using specific style references of known artists. You can create anime/comic styles without LoRAs but I found the results hit and miss.
The image at the top of this piece was created with Flux.1 Dev using the Real Comic LoRA.
Character and Face LoRAs
Not really in the same category as the other LoRAs, character and face LoRAs are a whole subject in themselves with the specific intent of creating a repeatable character or face. I raise it because I have been amazed at the quality of LoRAs you can create yourself for this purpose and I will write a separate piece on how to generate LoRAs.
Summary
Having a few LoRAs available in your workflows is highly worthwhile but with a mental note that ‘less is more’ in most cases. The base models often do an amazing job on their own but when you can’t quite get what you want then adding in a LoRA can make all the difference, especially if you are working on a niche style or composition.