PyTorch profiling tutorial k2 edition

https://github.com/k2-fsa/icefall/issues/610

That model warmup trick is also known as ReZero when you gradually change alpha from 0 to 1

https://proceedings.mlr.press/v161/bachlechner21a/bachlechner21a.pdf

https://twitter.com/92hschoi/status/1593467656379990016

Disentangled self-supervised representations for timbre, language and pitch using contrastive and reconstruction objectives.

DEMAND noise is used for linguistic contrast, CQT cropping is used for pitch. No extra augmentation seems to be used for timbre. Global speaker encoder tokens are turned into time-dependent using cross-attention in the synthesizer.

10k hours of pretraining data on 10x 3090. Beats supervised YourTTS on some tasks.

Rotate the phase of each frequency bin to simulate one-to-many mapping. Useful augmentation for GAN vocoder training. Differentiable.

https://mindslab-ai.github.io/phaseaug/

On augmentations for voice conversion:

> On the other hand, the use of more carefully designed augmentation techniques (such as timbre transformation with VoTrans and modifying prosody with NoisyF0) may be helpful in achieving more realistic target speaker identity and better audio quality

https://arxiv.org/abs/2212.13581

New scaling laws paper for combining different modalities (e.g. speech and text) using Chinchilla style notation.

https://twitter.com/ArmenAgha/status/1613192899646324736

I've been thinking about Transformers a lot — I've trained 5 GPT-2 models of different sizes, my first decoder-only Transformers over the past few weeks and trained my first encoder-decoder transformer on Math tasks from DeepMind.

It's magical how much faster these models converge if you care to spend engineering effort to scale them up. Recent advancements with Flash Attention and Triton make them go faster, 8-bit Adam saves memory, FP8 training is around the block.

I found a few training tricks people use in various combinations: gradient clipping, LR warmup + cooldown, model warmup (stochastic depth)

Going to share this important paper that moves Layer Norm to the top of the block (norm_first=True in PyTorch)
https://arxiv.org/abs/2002.04745

Sometimes, you can only train once. No time to tune learning rates.

Zero-shot Hyperparameter transfer: https://www.microsoft.com/en-us/research/blog/µtransfer-a-technique-for-hyperparameter-tuning-of-enormous-neural-networks/

Causal attention is all you need https://arxiv.org/abs/2204.05832

Geoffrey Hinton tweeted a guide to tuning deep learning models. Definitely reading this one tomorrow:

https://twitter.com/geoffreyhinton/status/1616154554768162818

1 GPU = 1.6B tokens per day
https://arxiv.org/abs/2206.07682

S4 ready to prod? Colab with the author of Flash Attention


https://twitter.com/realdanfu/status/1617605971395891201

Transformers post by Lilian Weng got an update

Bidirectional attention and unidirectional attention in one model can be achieved by changing the attention mask. This is called "Prefix LM"

https://www.semanticscholar.org/paper/Generating-Wikipedia-by-Summarizing-Long-Sequences-Liu-Saleh/7570afa31c68e24fce1342b7d67c591787219bc1
The image is from the T5 paper.