A Primer on the Most Important Concepts to Train a LoRA - part 3: Hyperparameters
# A Primer on the Most Important Concepts to Train a LoRA - part 3: Hyperparameters
*Tutorial - Guide — Version 2*
This is the revised version of my LoRA guide, the original version can be found here: [version 1](https://www.reddit.com/r/StableDiffusion/comments/1qqqstw/a_primer_on_the_most_important_concepts_to_train) NOTE: English is my 2nd language. Bare with me for possible mistakes.
[Part 1: Some definitions, FAQ, and Dataset Preparation](https://www.reddit.com/r/StableDiffusion/comments/1svsa4g/a_primer_on_the_most_important_concepts_to_train)
[Part 2: Captioning guide](https://www.reddit.com/r/StableDiffusion/comments/1svsea1/a_primer_on_the_most_important_concepts_to_train)
Part 3: Hyperparameter guide and regularization <-- you are here
# PART 3 ==== HYPERPARAMETERS AND REGULARIZATION ====
# Hyperparameters: Caption dropout and Token shuffling
Some training software offers options to randomly drop captions for a percentage of images during training, or to shuffle the order of words in captions. These are worth knowing about so you can make an informed decision.
* **Caption dropout** exists because it trains the model to respond to unconditioned or weakly conditioned generation, which was useful for large finetune training on millions of images. For a small character LoRA dataset of 15 to 30 images, every dropped caption is a wasted step where the trigger word association is not being reinforced. Keep caption dropout at zero or very close to zero for character LoRAs.
* **Token shuffling** is a legacy feature from the era of CLIP-based models like SD1.5 and SDXL, where word order carried less semantic weight. Modern T5-conditioned models (Flux, Chroma, and most current architectures) are deeply order-sensitive because it understands natural language. "a woman wearing a red dress" and "a red dress wearing a woman" are not the same thing to T5. Token shuffling on modern models is at best useless and at worst actively poisoning your LoRA. Turn it off.
# Hyperparameter : Rank (Network Dim) and Alpha
The rank of a LoRA represents the number of independent dimensions available to express the concept being learned. Think of it as the number of instruments in an orchestra — more instruments means more independent musical lines you can play simultaneously.
* Use high rank when you have a lot of things to learn.
* Use low rank when you have something simple to learn.
This is important because:
* If you use too high a rank, your LoRA will start learning additional details from your dataset that may clutter or even make it rigid and bleed during generation as it tries to learn too much
* If you use too low a rank, your LoRA will stop learning after a certain number of steps
Character LoRA that only learns a face: use a small rank like 16. It's enough. Full body LoRA: you need at least 32, perhaps 64. Otherwise it will have a hard time learning the body. Any LoRA that adds a NEW concept (not just refine an existing one) needs extra room, so use a higher rank than default. Multi-concept LoRA also needs more rank.
If you are not sure, a rank of 32 is enough for most tasks.
# Alpha
There is a secondary parameters that goes hand in hand with the rank parameter: it's called Alpha. It is used to scale the strength of the LoRA. For most LoRAs, it has to be set to :
* Alpha = Rank : Default set-up
* Alpha = Half the Rank : Your LoRA will be more flexible and less rigid but you may need more steps to get it to converge
In AI-Toolkit you can set alpha independently of rank in your YAML config:
network:
type: lora
linear: 32
linear_alpha: 16
# Hyperparameter: Repeats (per dataset)
To learn, the LoRA training will try to noise and de-noise your dataset hundreds of times, comparing the result and learning from it. The "repeats" parameter is only useful when you are using a dataset containing images that must be "seen" by the trainer at a different frequency. Consider this:
1. The training will reinforce the
# A Primer on the Most Important Concepts to Train a LoRA - part 3: Hyperparameters
*Tutorial - Guide — Version 2*
This is the revised version of my LoRA guide, the original version can be found here: [version 1](https://www.reddit.com/r/StableDiffusion/comments/1qqqstw/a_primer_on_the_most_important_concepts_to_train) NOTE: English is my 2nd language. Bare with me for possible mistakes.
[Part 1: Some definitions, FAQ, and Dataset Preparation](https://www.reddit.com/r/StableDiffusion/comments/1svsa4g/a_primer_on_the_most_important_concepts_to_train)
[Part 2: Captioning guide](https://www.reddit.com/r/StableDiffusion/comments/1svsea1/a_primer_on_the_most_important_concepts_to_train)
Part 3: Hyperparameter guide and regularization <-- you are here
# PART 3 ==== HYPERPARAMETERS AND REGULARIZATION ====
# Hyperparameters: Caption dropout and Token shuffling
Some training software offers options to randomly drop captions for a percentage of images during training, or to shuffle the order of words in captions. These are worth knowing about so you can make an informed decision.
* **Caption dropout** exists because it trains the model to respond to unconditioned or weakly conditioned generation, which was useful for large finetune training on millions of images. For a small character LoRA dataset of 15 to 30 images, every dropped caption is a wasted step where the trigger word association is not being reinforced. Keep caption dropout at zero or very close to zero for character LoRAs.
* **Token shuffling** is a legacy feature from the era of CLIP-based models like SD1.5 and SDXL, where word order carried less semantic weight. Modern T5-conditioned models (Flux, Chroma, and most current architectures) are deeply order-sensitive because it understands natural language. "a woman wearing a red dress" and "a red dress wearing a woman" are not the same thing to T5. Token shuffling on modern models is at best useless and at worst actively poisoning your LoRA. Turn it off.
# Hyperparameter : Rank (Network Dim) and Alpha
The rank of a LoRA represents the number of independent dimensions available to express the concept being learned. Think of it as the number of instruments in an orchestra — more instruments means more independent musical lines you can play simultaneously.
* Use high rank when you have a lot of things to learn.
* Use low rank when you have something simple to learn.
This is important because:
* If you use too high a rank, your LoRA will start learning additional details from your dataset that may clutter or even make it rigid and bleed during generation as it tries to learn too much
* If you use too low a rank, your LoRA will stop learning after a certain number of steps
Character LoRA that only learns a face: use a small rank like 16. It's enough. Full body LoRA: you need at least 32, perhaps 64. Otherwise it will have a hard time learning the body. Any LoRA that adds a NEW concept (not just refine an existing one) needs extra room, so use a higher rank than default. Multi-concept LoRA also needs more rank.
If you are not sure, a rank of 32 is enough for most tasks.
# Alpha
There is a secondary parameters that goes hand in hand with the rank parameter: it's called Alpha. It is used to scale the strength of the LoRA. For most LoRAs, it has to be set to :
* Alpha = Rank : Default set-up
* Alpha = Half the Rank : Your LoRA will be more flexible and less rigid but you may need more steps to get it to converge
In AI-Toolkit you can set alpha independently of rank in your YAML config:
network:
type: lora
linear: 32
linear_alpha: 16
# Hyperparameter: Repeats (per dataset)
To learn, the LoRA training will try to noise and de-noise your dataset hundreds of times, comparing the result and learning from it. The "repeats" parameter is only useful when you are using a dataset containing images that must be "seen" by the trainer at a different frequency. Consider this:
1. The training will reinforce the
Reddit
From the StableDiffusion community on Reddit
Explore this post and more from the StableDiffusion community
signal learned from each image into the LoRA each time it is processing that image. If it's not processed enough times, (under-training), the model still doesn't fully know how to draw it. If it is processed too many times (over-training) it will become rigid and will forget how to draw everything else. The key is to find the sweet spot.
2. You are training a model that already knows a lot because it has already been trained on million of images. The LoRA is trying to "adjust" it to generate specific things you trained it for. So when you train something it already knows, you don't need a lot of steps to reach the sweet spot. But if you train it on something that is NOT known to it, then it needs a lot more steps to reach that same sweet spot.
This is where the "repeat" parameter associated with each dataset is used. There are two major situations in which you want to carefully use the repeat parameter.
a) To balance a dataset that lacks variety
* The dataset should contain an equal amount of each camera angle, zoom level, etc.
* If your dataset only has a few profile images but a ton of font facing images, you risk overtraining the front angle and under-training the profile angle.
* You can set your "unique" angles in a separate dataset and set it to repeat 2x or 3x more than the front facing dataset, for instance, which will rebalance your dataset.
b) To balance known items with unknown items
* The mode should process 5x more the images of thing it doesn't know vs the things it knows
* If your dataset contains uncensored images on a censored model, for instance, you are going to need a lot more exposure to teach those new concepts
* Use more repeats on the unknown elements to avoid undertraining those elements or overtraining the regular ones.
# Hyperparameter: Batch or Gradient Accumulation
To learn, the LoRA trainer takes your dataset image, adds noise to it, and learns how to find back the image from the noise. When you use batch 2, it does the job for 2 images, then the learning is averaged between the two. On the long run, it means the quality is higher as it helps the model avoid learning "extreme" outliers.
* **Batch** means it's processing those images in parallel — which requires a lot more VRAM and GPU power. It doesn't require more steps, but each step will be that much longer. In theory it learns faster, so you can use fewer total steps.
* **Gradient accumulation** means it's processing those images in series, one by one — doesn't take more VRAM but each step will be proportionally longer.
For most consumer GPU setups where VRAM is the main constraint, gradient accumulation of 2 to 4 is the practical recommendation. It gives you the averaging benefit without the VRAM cost.
# Hyperparameter: LR (Learning Rate)
LR stands for "Learning Rate" and it is the #1 most important parameter of all your LoRA training.
Imagine you are trying to copy a drawing by dividing the image into small squares and copying one square at a time. This is what LR means: how small or big a "chunk" it is taking at a time to learn from it.
* If the chunk is huge, it means you will make great strides in learning (fewer steps)... but you will learn coarse things. Small details may be lost.
* If the chunk is small, it means it will be much more effective at learning some small delicate details... but it might take a very long time (more steps).
Some models are more sensitive to high LR than others. On Qwen-Image, you can use LR 0.0003 and it works fairly well. Use that same LR on Chroma and you will destroy your LoRA within 1000 steps.
Too high LR is the #1 cause for a LoRA not converging to your target. However, each time you lower your LR by half, you'd need twice as many steps to compensate.
So if LR 0.0001 requires 3000 steps on a given model, a more sensitive model might need LR 0.00005 but may need 6000 steps to get there.
Try LR 0.0001 at first — it's a fairly safe starting point.
# LR Scheduler
One of the best way to get good results without worries is to use an LR scheduler. This nifty parameter will
2. You are training a model that already knows a lot because it has already been trained on million of images. The LoRA is trying to "adjust" it to generate specific things you trained it for. So when you train something it already knows, you don't need a lot of steps to reach the sweet spot. But if you train it on something that is NOT known to it, then it needs a lot more steps to reach that same sweet spot.
This is where the "repeat" parameter associated with each dataset is used. There are two major situations in which you want to carefully use the repeat parameter.
a) To balance a dataset that lacks variety
* The dataset should contain an equal amount of each camera angle, zoom level, etc.
* If your dataset only has a few profile images but a ton of font facing images, you risk overtraining the front angle and under-training the profile angle.
* You can set your "unique" angles in a separate dataset and set it to repeat 2x or 3x more than the front facing dataset, for instance, which will rebalance your dataset.
b) To balance known items with unknown items
* The mode should process 5x more the images of thing it doesn't know vs the things it knows
* If your dataset contains uncensored images on a censored model, for instance, you are going to need a lot more exposure to teach those new concepts
* Use more repeats on the unknown elements to avoid undertraining those elements or overtraining the regular ones.
# Hyperparameter: Batch or Gradient Accumulation
To learn, the LoRA trainer takes your dataset image, adds noise to it, and learns how to find back the image from the noise. When you use batch 2, it does the job for 2 images, then the learning is averaged between the two. On the long run, it means the quality is higher as it helps the model avoid learning "extreme" outliers.
* **Batch** means it's processing those images in parallel — which requires a lot more VRAM and GPU power. It doesn't require more steps, but each step will be that much longer. In theory it learns faster, so you can use fewer total steps.
* **Gradient accumulation** means it's processing those images in series, one by one — doesn't take more VRAM but each step will be proportionally longer.
For most consumer GPU setups where VRAM is the main constraint, gradient accumulation of 2 to 4 is the practical recommendation. It gives you the averaging benefit without the VRAM cost.
# Hyperparameter: LR (Learning Rate)
LR stands for "Learning Rate" and it is the #1 most important parameter of all your LoRA training.
Imagine you are trying to copy a drawing by dividing the image into small squares and copying one square at a time. This is what LR means: how small or big a "chunk" it is taking at a time to learn from it.
* If the chunk is huge, it means you will make great strides in learning (fewer steps)... but you will learn coarse things. Small details may be lost.
* If the chunk is small, it means it will be much more effective at learning some small delicate details... but it might take a very long time (more steps).
Some models are more sensitive to high LR than others. On Qwen-Image, you can use LR 0.0003 and it works fairly well. Use that same LR on Chroma and you will destroy your LoRA within 1000 steps.
Too high LR is the #1 cause for a LoRA not converging to your target. However, each time you lower your LR by half, you'd need twice as many steps to compensate.
So if LR 0.0001 requires 3000 steps on a given model, a more sensitive model might need LR 0.00005 but may need 6000 steps to get there.
Try LR 0.0001 at first — it's a fairly safe starting point.
# LR Scheduler
One of the best way to get good results without worries is to use an LR scheduler. This nifty parameter will
automatically decay the LR across your training progress. Think of it like sculpting a piece of marble: at first you want to BIG chisel with a big hammer to take away the rough chunks quickly. However the closer you get to your target, the more precise you need to be. At some point you have to use smaller chisel and be very careful not to ruin your art piece. The LR scheduler will make sure you change to a lower LR (smaller chisel) as you progress into LoRA learning.
On AI-Toolkit, you have to activate the LR scheduling in the advanced properties in the YAML config file directly, under the training section :
train:
lr_scheduler: "cosine"
# Hyperparameter: Timestep
During diffusion training, the model learns to denoise images at varying levels of noise — from nearly clean images to pure noise. Each noise level (called a timestep) teaches the model something different:
* **High timesteps (heavy noise):** The model learns global structure and broad composition — "is this a face or a landscape?"
* **Middle timesteps:** The model learns semantic identity and specific features — "whose face is this? what are the specific proportions?"
* **Low timesteps (light noise):** The model learns fine details and textures — "how sharp are these edges? what does this skin texture look like?"
By default, training samples all timesteps equally. But you can change this - this is what the Timestep parameter is all about. For character LoRAs, the middle range is where identity lives, so we want to spent most of the training effort there.
In AI-Toolkit, the recommended setting for character LoRAs is the **sigmoid** timestep distribution. This concentrates training probability around the middle timesteps in a smooth bell-curve shape, naturally de-emphasizing both extremes. Other distributions exist for other use cases: biasing toward high timesteps is useful for style LoRAs that need to affect global composition; biasing toward low timesteps is useful for texture or fine detail work.
# Hyperparameter: Optimizer
The optimizer is the algorithm that decides how to adjust the LoRA's weights in response to the training loss at each step. It's the heart of the training software.
* \***AdamW** is the most widely used optimizer for LoRA training. AdamW8bit is a memory-efficient version that uses less VRAM with minimal quality impact. For most consumer GPU setups, AdamW8bit is the practical default and the right place to start. I get excellent result with AdamW, as long as I use an LR scheduler to make sure LR properly decays across time.
* **Prodigy** is an optimizer that attempts to manage LR automatically It starts at LR 1.0 (it's just a placeholder) and then it gets adjusted dynamically. If you don't know what to do with LR or if you are working with very sensitive models that reacts badly to LR, it can be an interesting choice.
Most LoRA failures are not optimizer failures — they are dataset, caption, or LR failures. If something isn't working, changing the optimizer is usually the last thing to try, not the first.
# How to Monitor the Training
Many people disable sampling because it makes the training much longer. However, unless you exactly know what you are doing, it's a bad idea. Sampling help you understand what's going on and if the training is working or not.
When planning your sampling prompts, try to use:
* One basic prompt to test if your model has learned the trigger word in a basic situation
* One prompt from another angle and with a different zoom level - helps verify if all angles and zoom levels are being learned properly - if face drifts under unusual angles, it's undertrained or perhaps your dataset doesn't have enough repeats for that angle
* One prompt showing specifically the body parts or elements the model didn't know (like censored elements) - as long as you see body horror, it's undertrained
* One prompt with a variation not present in any of your dataset image. For instance: blue hair. If it starts becoming the same color as your main dataset, you know it's overfitting
* One prompt with a full
On AI-Toolkit, you have to activate the LR scheduling in the advanced properties in the YAML config file directly, under the training section :
train:
lr_scheduler: "cosine"
# Hyperparameter: Timestep
During diffusion training, the model learns to denoise images at varying levels of noise — from nearly clean images to pure noise. Each noise level (called a timestep) teaches the model something different:
* **High timesteps (heavy noise):** The model learns global structure and broad composition — "is this a face or a landscape?"
* **Middle timesteps:** The model learns semantic identity and specific features — "whose face is this? what are the specific proportions?"
* **Low timesteps (light noise):** The model learns fine details and textures — "how sharp are these edges? what does this skin texture look like?"
By default, training samples all timesteps equally. But you can change this - this is what the Timestep parameter is all about. For character LoRAs, the middle range is where identity lives, so we want to spent most of the training effort there.
In AI-Toolkit, the recommended setting for character LoRAs is the **sigmoid** timestep distribution. This concentrates training probability around the middle timesteps in a smooth bell-curve shape, naturally de-emphasizing both extremes. Other distributions exist for other use cases: biasing toward high timesteps is useful for style LoRAs that need to affect global composition; biasing toward low timesteps is useful for texture or fine detail work.
# Hyperparameter: Optimizer
The optimizer is the algorithm that decides how to adjust the LoRA's weights in response to the training loss at each step. It's the heart of the training software.
* \***AdamW** is the most widely used optimizer for LoRA training. AdamW8bit is a memory-efficient version that uses less VRAM with minimal quality impact. For most consumer GPU setups, AdamW8bit is the practical default and the right place to start. I get excellent result with AdamW, as long as I use an LR scheduler to make sure LR properly decays across time.
* **Prodigy** is an optimizer that attempts to manage LR automatically It starts at LR 1.0 (it's just a placeholder) and then it gets adjusted dynamically. If you don't know what to do with LR or if you are working with very sensitive models that reacts badly to LR, it can be an interesting choice.
Most LoRA failures are not optimizer failures — they are dataset, caption, or LR failures. If something isn't working, changing the optimizer is usually the last thing to try, not the first.
# How to Monitor the Training
Many people disable sampling because it makes the training much longer. However, unless you exactly know what you are doing, it's a bad idea. Sampling help you understand what's going on and if the training is working or not.
When planning your sampling prompts, try to use:
* One basic prompt to test if your model has learned the trigger word in a basic situation
* One prompt from another angle and with a different zoom level - helps verify if all angles and zoom levels are being learned properly - if face drifts under unusual angles, it's undertrained or perhaps your dataset doesn't have enough repeats for that angle
* One prompt showing specifically the body parts or elements the model didn't know (like censored elements) - as long as you see body horror, it's undertrained
* One prompt with a variation not present in any of your dataset image. For instance: blue hair. If it starts becoming the same color as your main dataset, you know it's overfitting
* One prompt with a full
body shot to verify proportions are being learned
* One prompt with a wide shot to verify it hasn't unlearned different composition and can draw your subject from afar
You get the gist: test test test so you can see if it works and where you will have to act to arrange the problem. Generally speaking, if you see the samples suddenly stop converging, or even start diverging, stop the training immediately : the LR is too high and it is probably ruining the LoRA.
# When to Stop Training to Avoid Overtraining
Look at the samples. If you feel like you have reached a point where the consistency is good and looks close to the target, and you see no real improvement after the next sample batch, it's time to stop. Most trainers will produce a LoRA after each epoch, so you can let it run past that point and then look back on all your samples to decide at which point it looks best without losing its flexibility.
If you have body horror mixed with perfect faces, that's a sign that your dataset proportions are off and some images are undertrained while others are overtrained.
The full overtraining progression typically looks like this:
* LoRA starts improving
* Reaches a good balance of consistency and flexibility
* Begins to look overly sharp or "crispy"
* Starts losing prompt flexibility, resisting creative prompts
* Eventually degrades in quality
# Using a Regularization Dataset
When you are training a LoRA, one possible danger is that you may get the base model to "unlearn" the concepts it already knows. For instance, if you train on images of a woman, it may unlearn what other women look like.
This is also a problem when training multi-concept LoRAs. The LoRA has to understand what looks like triggerA, what looks like triggerB, and what's neither A nor B.
This is what the regularization dataset is for. Most training software supports this feature. You add a dataset containing other images showing the same generic class (like "woman") but that are NOT your target. This dataset allows the model to refresh its memory, so to speak, so it doesn't unlearn the rest of its base training.
You need at least 1 regularization image for every 2 image *processed* by the training, taking repeats into account. If your trained LoRA is noticeably corrupting other women in generated scenes, increase regularization exposure. If your character is coming out weak or inconsistent, reduce it.
If you have further questions, post them below, or send me a chat request.
[Previous part <== Part 1: Dataset](https://www.reddit.com/r/StableDiffusion/comments/1svsa4g/a_primer_on_the_most_important_concepts_to_train)
[Previous part <== Part 2: Captioning](https://www.reddit.com/r/StableDiffusion/comments/1svsea1/a_primer_on_the_most_important_concepts_to_train)
https://redd.it/1svsk08
@rStableDiffusion
* One prompt with a wide shot to verify it hasn't unlearned different composition and can draw your subject from afar
You get the gist: test test test so you can see if it works and where you will have to act to arrange the problem. Generally speaking, if you see the samples suddenly stop converging, or even start diverging, stop the training immediately : the LR is too high and it is probably ruining the LoRA.
# When to Stop Training to Avoid Overtraining
Look at the samples. If you feel like you have reached a point where the consistency is good and looks close to the target, and you see no real improvement after the next sample batch, it's time to stop. Most trainers will produce a LoRA after each epoch, so you can let it run past that point and then look back on all your samples to decide at which point it looks best without losing its flexibility.
If you have body horror mixed with perfect faces, that's a sign that your dataset proportions are off and some images are undertrained while others are overtrained.
The full overtraining progression typically looks like this:
* LoRA starts improving
* Reaches a good balance of consistency and flexibility
* Begins to look overly sharp or "crispy"
* Starts losing prompt flexibility, resisting creative prompts
* Eventually degrades in quality
# Using a Regularization Dataset
When you are training a LoRA, one possible danger is that you may get the base model to "unlearn" the concepts it already knows. For instance, if you train on images of a woman, it may unlearn what other women look like.
This is also a problem when training multi-concept LoRAs. The LoRA has to understand what looks like triggerA, what looks like triggerB, and what's neither A nor B.
This is what the regularization dataset is for. Most training software supports this feature. You add a dataset containing other images showing the same generic class (like "woman") but that are NOT your target. This dataset allows the model to refresh its memory, so to speak, so it doesn't unlearn the rest of its base training.
You need at least 1 regularization image for every 2 image *processed* by the training, taking repeats into account. If your trained LoRA is noticeably corrupting other women in generated scenes, increase regularization exposure. If your character is coming out weak or inconsistent, reduce it.
If you have further questions, post them below, or send me a chat request.
[Previous part <== Part 1: Dataset](https://www.reddit.com/r/StableDiffusion/comments/1svsa4g/a_primer_on_the_most_important_concepts_to_train)
[Previous part <== Part 2: Captioning](https://www.reddit.com/r/StableDiffusion/comments/1svsea1/a_primer_on_the_most_important_concepts_to_train)
https://redd.it/1svsk08
@rStableDiffusion
Reddit
From the StableDiffusion community on Reddit
Explore this post and more from the StableDiffusion community
Why do people release models on Huggingface that have no explanation on how to use it?
So this is really frustrating. When a developer releases a model, they won't just have the model, vae, clip, ect. as regular files that you can drop into the ComfuUI directory. Instead it will be the type of installation where you have to do some sort of git pull. And the files are generically named.
Why do some of these developers not make it easier for users? Does this upset you that Huggingface users do not make it easy to just download the file and drop it into the models directory?
There are newer types of models that have no explanation at all on what they do or how to use them. You would think if someone spent hundreds of hours making a model they would have a simple summary of what the hell it does and how to use it other than "here's the Git file, good luck!"
https://redd.it/1sw7sp4
@rStableDiffusion
So this is really frustrating. When a developer releases a model, they won't just have the model, vae, clip, ect. as regular files that you can drop into the ComfuUI directory. Instead it will be the type of installation where you have to do some sort of git pull. And the files are generically named.
Why do some of these developers not make it easier for users? Does this upset you that Huggingface users do not make it easy to just download the file and drop it into the models directory?
There are newer types of models that have no explanation at all on what they do or how to use them. You would think if someone spent hundreds of hours making a model they would have a simple summary of what the hell it does and how to use it other than "here's the Git file, good luck!"
https://redd.it/1sw7sp4
@rStableDiffusion
Reddit
From the StableDiffusion community on Reddit
Explore this post and more from the StableDiffusion community
huggingface/ml-intern: 🤗 ml-intern: an open-source ML engineer that reads papers, trains models, and ships ML models
https://github.com/huggingface/ml-intern/tree/main
https://redd.it/1swctgr
@rStableDiffusion
https://github.com/huggingface/ml-intern/tree/main
https://redd.it/1swctgr
@rStableDiffusion
GitHub
GitHub - huggingface/ml-intern: 🤗 ml-intern: an open-source ML engineer that reads papers, trains models, and ships ML models
🤗 ml-intern: an open-source ML engineer that reads papers, trains models, and ships ML models - huggingface/ml-intern
This media is not supported in your browser
VIEW IN TELEGRAM
Most image managers suck for AI. I built AURA: A local-first Vault for 'AI Hoarders' with Civitai integration, Vision Tagging, Aesthetic Grading, and more!
https://redd.it/1swck8j
@rStableDiffusion
https://redd.it/1swck8j
@rStableDiffusion
NaughtyAmerica is looking for AI Video Creators to contract
Naughty America is looking to pay professional AI video creators/studios to produce short videos from approved user pitches.
We launched PRODUCERS MARKETPLACE (not linking on purpose,) where users submit pitches for scenes or fantasies they want created. Models can audition for those pitches, and when a model is approved, she is compensated for participating.
A lot of these pitches are short fantasies. They are not always big enough to justify a full filmed scene, VR shoot, or mixed-reality production. In many cases, they would make more sense as a short AI-generated vignette.
What we are looking for:
A user submits a pitch.
A model auditions and approves participation.
We hire a professional AI creator/studio to turn that approved pitch into a short video.
This is paid vendor work through the company. It is not a tool for users to generate content of models directly.
If you are an AI video creator, studio, or production company that can do this professionally, please reach out, reply.
Also open to suggestions on better subreddits for finding this kind of vendor.
https://redd.it/1swmb9f
@rStableDiffusion
Naughty America is looking to pay professional AI video creators/studios to produce short videos from approved user pitches.
We launched PRODUCERS MARKETPLACE (not linking on purpose,) where users submit pitches for scenes or fantasies they want created. Models can audition for those pitches, and when a model is approved, she is compensated for participating.
A lot of these pitches are short fantasies. They are not always big enough to justify a full filmed scene, VR shoot, or mixed-reality production. In many cases, they would make more sense as a short AI-generated vignette.
What we are looking for:
A user submits a pitch.
A model auditions and approves participation.
We hire a professional AI creator/studio to turn that approved pitch into a short video.
This is paid vendor work through the company. It is not a tool for users to generate content of models directly.
If you are an AI video creator, studio, or production company that can do this professionally, please reach out, reply.
Also open to suggestions on better subreddits for finding this kind of vendor.
https://redd.it/1swmb9f
@rStableDiffusion
Reddit
From the StableDiffusion community on Reddit
Explore this post and more from the StableDiffusion community
If Wan made an image editor, wouldn't character consistency be solved?
I've been messing with Wan 2.2 a lot lately. It's a year old, but gets good character consistency at higher resolution. People also use the low-noise model for image generation, something I've never actually got to work right, but will be trying again at some point.
The point is, we're still bound to creating LoRAs for true character consistency. The only game in town that more or less has the single image style/likeness transfer down is Midjourney. Qwen IE, Flux Klein, Kontext...these are all noble attempts, but they aren't Nano Banana, and not as flexible as we need them to be, even with loras on top.
But if Wan were to make an image editor, wouldn't this issue essentially be solved?
For example - FFGO. You can just put a bunch of ref images, different styles, and it can "animate" those images with near perfect likeness. Why not just create a image editor? The community would make custom loras for style transfer overnight.
I guess the only caveat being since Wan isn't really doing open source anymore, they probably aren't interested?
https://redd.it/1swp0qc
@rStableDiffusion
I've been messing with Wan 2.2 a lot lately. It's a year old, but gets good character consistency at higher resolution. People also use the low-noise model for image generation, something I've never actually got to work right, but will be trying again at some point.
The point is, we're still bound to creating LoRAs for true character consistency. The only game in town that more or less has the single image style/likeness transfer down is Midjourney. Qwen IE, Flux Klein, Kontext...these are all noble attempts, but they aren't Nano Banana, and not as flexible as we need them to be, even with loras on top.
But if Wan were to make an image editor, wouldn't this issue essentially be solved?
For example - FFGO. You can just put a bunch of ref images, different styles, and it can "animate" those images with near perfect likeness. Why not just create a image editor? The community would make custom loras for style transfer overnight.
I guess the only caveat being since Wan isn't really doing open source anymore, they probably aren't interested?
https://redd.it/1swp0qc
@rStableDiffusion
Reddit
From the StableDiffusion community on Reddit
Explore this post and more from the StableDiffusion community
This media is not supported in your browser
VIEW IN TELEGRAM
LTX2.3 in Ostris Ai toolkit on a 5090 Training done in 7 hours ... I went Thanos way and I said fine ... I'll do it myself
https://redd.it/1swrs76
@rStableDiffusion
https://redd.it/1swrs76
@rStableDiffusion
Open Source UniGeo: Unifying Geometric Guidance for Camera-Controllable Image Editing via Video Models (Powered by Wan2.2 & VGGT)
Hey everyone! 👋
I'm excited to share our latest open-source research: UniGeo. It's a framework that leverages video models (Wan2.2) and unified geometric guidance to achieve precise, camera-controllable image editing.
🧠 The Pipeline (How to actually use it): We wanted to avoid the "black-box" prompting experience where you just type and hope for the best. Here is the step-by-step workflow:
Prompt to Physics: You provide a source image and a natural language command. You can chain multiple movements (e.g., "Camera pans left by 15 degrees; Camera moves left by 0.27"). The system parses this into explicit physical camera parameters.
Point Cloud Generation (The Preview): Using VGGT, we translate those parameters into a guiding Point Cloud. You can iterate and tweak your camera parameters at this stage until the geometric trajectory looks perfect, saving you from wasting heavy compute on a bad render.
Video Model Rendering: Once you are satisfied with the point cloud, it gets fed into our fine-tuned Wan2.2-5B model along with the source image to render the final fluid sequence.
✨ Some results generated by our model. You can check out more examples on our project page
🔍 Why we built this (Observations vs. Current Models):
Recently, Qwen-Image-Edit-2511-Multiple-Angles-LoRA has been getting a lot of well-deserved attention. It's fantastic, but during our research, we wanted to solve a few specific pain points we noticed in current methodologies:
Continuous Motion vs. Discrete Angles: Unlike methods that switch between fixed viewpoints, UniGeo enables continuous, physically fluid camera trajectories on images, offering much broader generalization.
Real-World Robustness: On "in-the-wild" images, our geometric guidance forces the model to maintain strict spatial consistency, effectively eliminating background distortion and structural collapse.
✨ A side-by-side comparison with the Qwen mode
All code, weights, and demos are completely open-source. We’d love for the community to try running the pipeline locally with your own images, break it, and give us feedback on the methodology!
https://redd.it/1swriv3
@rStableDiffusion
Hey everyone! 👋
I'm excited to share our latest open-source research: UniGeo. It's a framework that leverages video models (Wan2.2) and unified geometric guidance to achieve precise, camera-controllable image editing.
🧠 The Pipeline (How to actually use it): We wanted to avoid the "black-box" prompting experience where you just type and hope for the best. Here is the step-by-step workflow:
Prompt to Physics: You provide a source image and a natural language command. You can chain multiple movements (e.g., "Camera pans left by 15 degrees; Camera moves left by 0.27"). The system parses this into explicit physical camera parameters.
Point Cloud Generation (The Preview): Using VGGT, we translate those parameters into a guiding Point Cloud. You can iterate and tweak your camera parameters at this stage until the geometric trajectory looks perfect, saving you from wasting heavy compute on a bad render.
Video Model Rendering: Once you are satisfied with the point cloud, it gets fed into our fine-tuned Wan2.2-5B model along with the source image to render the final fluid sequence.
✨ Some results generated by our model. You can check out more examples on our project page
🔍 Why we built this (Observations vs. Current Models):
Recently, Qwen-Image-Edit-2511-Multiple-Angles-LoRA has been getting a lot of well-deserved attention. It's fantastic, but during our research, we wanted to solve a few specific pain points we noticed in current methodologies:
Continuous Motion vs. Discrete Angles: Unlike methods that switch between fixed viewpoints, UniGeo enables continuous, physically fluid camera trajectories on images, offering much broader generalization.
Real-World Robustness: On "in-the-wild" images, our geometric guidance forces the model to maintain strict spatial consistency, effectively eliminating background distortion and structural collapse.
✨ A side-by-side comparison with the Qwen mode
All code, weights, and demos are completely open-source. We’d love for the community to try running the pipeline locally with your own images, break it, and give us feedback on the methodology!
https://redd.it/1swriv3
@rStableDiffusion
Built a open-source local music video generator using SDXL + AnimateDiff + audio-reactive GLSL shaders
I needed visuals for AI-generated tracks, so I built Glitchframe, a pipeline that takes an audio file and produces a full music video using SDXL keyframe stills or AnimateDiff motion, with GLSL shaders that react to beat/onset/spectrum data in real time.
Stack: SDXL for backgrounds, optional AnimateDiff (fair warning: \~20 GB VRAM), Skia for kinetic typography, WhisperX for word-level lyric sync, FFmpeg NVENC for encode. UI runs in Gradio locally.
AnimateDiff integration was the most painful part — VRAM requirements are brutal so Ken Burns is the default fallback for most people.
Examples of what it currently produces: https://www.youtube.com/@voidcatalog
GitHub (MIT): https://github.com/OlaProeis/Glitchframe
Interested in any feedback on the diffusion/motion side especially.
https://redd.it/1swx934
@rStableDiffusion
I needed visuals for AI-generated tracks, so I built Glitchframe, a pipeline that takes an audio file and produces a full music video using SDXL keyframe stills or AnimateDiff motion, with GLSL shaders that react to beat/onset/spectrum data in real time.
Stack: SDXL for backgrounds, optional AnimateDiff (fair warning: \~20 GB VRAM), Skia for kinetic typography, WhisperX for word-level lyric sync, FFmpeg NVENC for encode. UI runs in Gradio locally.
AnimateDiff integration was the most painful part — VRAM requirements are brutal so Ken Burns is the default fallback for most people.
Examples of what it currently produces: https://www.youtube.com/@voidcatalog
GitHub (MIT): https://github.com/OlaProeis/Glitchframe
Interested in any feedback on the diffusion/motion side especially.
https://redd.it/1swx934
@rStableDiffusion
GitHub
GitHub - OlaProeis/Glitchframe: Local, GPU-accelerated music video generator: upload a track, analyze it, align lyrics, generate…
Local, GPU-accelerated music video generator: upload a track, analyze it, align lyrics, generate stylized backgrounds, composite reactive shaders and kinetic type, and encode with ffmpeg (NVENC on ...
Trying to create simple SDXL+ZIT Refiner workflow (failing...)
Hello :>
I'm failing at creating a simple SDXL+ZIT Refiner workflow. I didn't think it is that tricky.... I'm getting this Error: RuntimeError: mat1 and mat2 shapes cannot be multiplied (4096x16 and 64x3840)
Here is the workflow: https://drive.proton.me/urls/X3JKS6CBBR#gcpsRbszKbct
Would be awesome if someone could step in and help out :>
https://redd.it/1swziwq
@rStableDiffusion
Hello :>
I'm failing at creating a simple SDXL+ZIT Refiner workflow. I didn't think it is that tricky.... I'm getting this Error: RuntimeError: mat1 and mat2 shapes cannot be multiplied (4096x16 and 64x3840)
Here is the workflow: https://drive.proton.me/urls/X3JKS6CBBR#gcpsRbszKbct
Would be awesome if someone could step in and help out :>
https://redd.it/1swziwq
@rStableDiffusion
drive.proton.me
Proton Drive
Securely store, share, and access your important files and photos. Anytime, anywhere.
Trying to enhance some old hentai mangas(image to image enhancement)
Recently I tried enhancing some old hentai mangas with very poor quality(some pretty much sketches) and got outstanding results with ChatGPT and Grok.
The problem is, they don't do explicit content, so I tried using other online sites(like Civtai, Mage space) with different models, but none got even close to the results I got with the two aforementioned.
The conclusion I ended up with is that I would have to resort to offline AI, but I know very little about the subject. So I come here to ask for tips and what would be the most beginner friendly way to do it. Good tutorials and tips are welcome.
Thanks in advance.
https://redd.it/1sx02hp
@rStableDiffusion
Recently I tried enhancing some old hentai mangas with very poor quality(some pretty much sketches) and got outstanding results with ChatGPT and Grok.
The problem is, they don't do explicit content, so I tried using other online sites(like Civtai, Mage space) with different models, but none got even close to the results I got with the two aforementioned.
The conclusion I ended up with is that I would have to resort to offline AI, but I know very little about the subject. So I come here to ask for tips and what would be the most beginner friendly way to do it. Good tutorials and tips are welcome.
Thanks in advance.
https://redd.it/1sx02hp
@rStableDiffusion
Reddit
From the StableDiffusion community on Reddit
Explore this post and more from the StableDiffusion community