Ernie and a Complex Composition in one Run (guest ZIT, Details and Prompt Included)
https://redd.it/1sof8hx
@rStableDiffusion
https://redd.it/1sof8hx
@rStableDiffusion
Reddit
From the StableDiffusion community on Reddit: Ernie and a Complex Composition in one Run (guest ZIT, Details and Prompt Included)
Explore this post and more from the StableDiffusion community
Pantomime | Facial expression sprite generator using Flux2.Klein and SDXL
https://redd.it/1sobvch
@rStableDiffusion
https://redd.it/1sobvch
@rStableDiffusion
Reddit
From the StableDiffusion community on Reddit: Pantomime | Facial expression sprite generator using Flux2.Klein and SDXL
Explore this post and more from the StableDiffusion community
New Optimizer 🌹 Rose: low VRAM, easy to use, great results, Apache 2.0
Hello, World! I have finally publicly released a new PyTorch optimizer I've been researching and developing for the last couple of years. It's named "Rose" in memory of my mother, who loved to hear about my discoveries and progress with AI.
Without going into the technical details (which you can read about in the GitHub repo), here are some of its benefits:
- It's stateless, which means it uses less memory than even AdamW8bit. If it weren't for working memory, its memory use would be as low as plain vanilla SGD (without momentum).
- Fast convergence, low VRAM, and excellent generalization, along with overfitting resistance. Yeah, I know... sounds too good to be true. Try it for yourself and tell me what you think, I'd really love to hear everyone's experiences, good or bad.
- Apache 2.0 license
You can find the code and more information at: https://github.com/MatthewK78/Rose
Benchmarks can sometimes be misleading, which is why I haven't included any. For example, sometimes training loss is higher in Rose than in Adam but validation loss is lower in Rose. The actual output of the trained model is what really matters in the end, and even that can be subjective. I'd prefer to let the community decide.
Here's some quickstart help for getting it up and running in
Install with:
Add this alongside other optimizers in the
Here's a config file example:
It may also initially be helpful to assess what it's doing by setting
If you try it, please let me know your thoughts and share your results. 😊
https://redd.it/1sokmqw
@rStableDiffusion
Hello, World! I have finally publicly released a new PyTorch optimizer I've been researching and developing for the last couple of years. It's named "Rose" in memory of my mother, who loved to hear about my discoveries and progress with AI.
Without going into the technical details (which you can read about in the GitHub repo), here are some of its benefits:
- It's stateless, which means it uses less memory than even AdamW8bit. If it weren't for working memory, its memory use would be as low as plain vanilla SGD (without momentum).
- Fast convergence, low VRAM, and excellent generalization, along with overfitting resistance. Yeah, I know... sounds too good to be true. Try it for yourself and tell me what you think, I'd really love to hear everyone's experiences, good or bad.
- Apache 2.0 license
You can find the code and more information at: https://github.com/MatthewK78/Rose
Benchmarks can sometimes be misleading, which is why I haven't included any. For example, sometimes training loss is higher in Rose than in Adam but validation loss is lower in Rose. The actual output of the trained model is what really matters in the end, and even that can be subjective. I'd prefer to let the community decide.
Here's some quickstart help for getting it up and running in
ostris/ai-toolkit.Install with:
pip install git+https://github.com/MatthewK78/Rose
Add this alongside other optimizers in the
toolkit/optimizer.py file:elif lower_type.startswith("rose"):
from rose import Rose
print(f"Using Rose optimizer, lr: {learning_rate:.2e}")
optimizer = Rose(params, lr=learning_rate, **optimizer_params)
Here's a config file example:
optimizer: Rose
lr: 8e-4
lr_scheduler: cosine
lr_scheduler_params:
eta_min: 1e-4
# all are default settings except `wd_schedule`
optimizer_params:
weight_decay: 1e-4 # adamw-style decoupled weight decay
wd_schedule: true # helps when using wd + lr_scheduler
centralize: true # gradient centralization
stabilize: true # disable for more aggressive training
bf16_sr: true # bf16 stochastic rounding
compute_dtype: fp64 # use fp32 only if you really need it
max_grad_norm: 65504 # effectively disables gradient clipping
ema_config:
use_ema: false
timestep_type: weighted
It may also initially be helpful to assess what it's doing by setting
sample_every to something low like 128 steps.If you try it, please let me know your thoughts and share your results. 😊
https://redd.it/1sokmqw
@rStableDiffusion
GitHub
GitHub - MatthewK78/Rose: 🌹 Rose: Range-Of-Slice Equilibration PyTorch optimizer. Stateless optimization through range-normalized…
🌹 Rose: Range-Of-Slice Equilibration PyTorch optimizer. Stateless optimization through range-normalized gradient updates. - MatthewK78/Rose