LORA collection that I use in my practice
https://civitai.com/collections/4905737
#comfyui

I've updated the link, yesterday it didn't work

I've been doing a lot of model training for FLUX in the last couple of weeks, and there are some interesting findings there. i'll try to summarize the information and make posts soon. for now, here's a spreadsheet i'm creating to track progress during training.
#comfyui

I promised to share a workflow for changing the seasons on an image today in a chat connected to this channel, so here it is. I only used Control Net, but if you want to completely change some parts of the image, you can combine this technique with inpaint, or, which is probably even better, at least for some cases, just save and edit the image from preprocessor. For example, you can erase grass or leaves for winter images, or replace some elements with other elements

I promised to post information on how to train LORA on FLUX this week, but there is a new version of the model, which I also need to try in tests, if it is dramatically better and there is no point in posting old information, so I need some more time.

a bit delayed, but it's time to talk about training LORA models for FLUX. i think i'm going to split this material into a few days. let's start with the training technique itself: we'll use this tool.
https://github.com/ostris/ai-toolkit

If you have 24Gb+ graphics card, you can run it locally. Ostris has links to tutorials on github page I mentioned above. but I used Google Colab.

Artem Svetozarov adapted the original colab by adding separate windows with parameters, and I, in turn, slightly tweaked the code to allow different models to be used. but this link is specifically for Flux Dev only (others in the next posts):
https://colab.research.google.com/drive/1xWiIQFpCx7aEkgEd_aBmrHm9hH5iBMIb?usp=sharing

#comfyui #lora #flux

In order to start training a model, we need a dataset. proper preparation and processing of the dataset is the key to success. the tests for this material I made on the basis of photos of Coop Himmelb(l)au projects, which I downloaded from the company's website, where they are in good enough quality. The dataset contains 57 photos of about 20 different projects. in the folder it looks like on the screenshot. don't pay attention to the json file and the _latent_cache folder, they are created automatically when you start the training. we only need images and text files with descriptions of these images, the names of png and txt files must match.

Image descriptions can of course be done yourself manually, but I've trained GPT for this task, here's a link to it:
https://chatgpt.com/g/g-wvc9iwYuc-image-captioner

just upload images without asking anything, it knows what to do. the output will be a neat table.

If you want to know what criteria are used to make descriptions, there is a button “Show the list of parameters...” on the start page. The only point is that ChatGPT does not work well with large amounts of uploaded materials, so it is better to upload images in portions, 5 at a time for example.

tomorrow we will analyze the parameters in Google Colab in detail. if someone has time to build his dataset, you can test it right away. an important point is that we will need a paid Colab, because we need a graphics card A100. in the free subscription at least it is not always possible to connect to it, at most it will not be available at all. for one session of training I spent about 30 compute units, which is about 3 eu.

Google Colab, what we need:
- to start somewhere around 100 compute units for a few tests (you can buy them here)
- google drive for 100 GB (link).

Next, let's open Colab, to which I already gave you a link. and go to setting up the parameters, located in the second section Step 2. Setup Config

#comfyui #lora #flux

The first three parameters are simple:
- choose a name for your model (literally any name, but I prefer to use a prefix with a reference to the base model, in this case FLUX_)
- specify the path to the folder in Google Drive where your dataset is located
- specify the path to the folder in Google Drive where LORA models and images from test renders will be saved.

next we have sample prompts. these prompts are used to generate images after a certain number of training steps (in our case, the default is that every 250 steps test renders will be run. Of course, it is better to write prompts that suit your purposes. I usually use one of them exactly as for one of the images in the dataset, i.e. copy the description, and the other two are random, but also on the topic of architecture.

The three parameters that most affect the learning process:

- network_rank is responsible for how many different details in the dataset images LORA will be able to memorize and then use. the higher the value of the parameter, the more attention to details, but also the weight of LORA in the end will be greater. thus, if the overall style of the image is important to you, rather than individual elements, then set something around 16 or 32. if details are important, then 64 or even 128.

- learning_rate is responsible for how quickly the model will be trained. the range of values here is from 0.0001 to 0.0015. the higher the value, the faster you will reach the state of the overtrained model. usually, the smaller your dataset is, the higher the learning_rate should be set, otherwise there is a risk that you will waste a lot of time and money. the risk is minor, because we have test renders and we can draw conclusions from them and stop the process. well, and through steps_number you can set the limit when the process will stop itself

- steps_number - the number of training steps. most often the optimal parameter value here for a dataset of 50 images is between 2000 and 4000 steps.