I've been thinking about Transformers a lot — I've trained 5 GPT-2 models of different sizes, my first decoder-only Transformers over the past few weeks and trained my first encoder-decoder transformer on Math tasks from DeepMind.

It's magical how much faster these models converge if you care to spend engineering effort to scale them up. Recent advancements with Flash Attention and Triton make them go faster, 8-bit Adam saves memory, FP8 training is around the block.

I found a few training tricks people use in various combinations: gradient clipping, LR warmup + cooldown, model warmup (stochastic depth)

Going to share this important paper that moves Layer Norm to the top of the block (norm_first=True in PyTorch)
https://arxiv.org/abs/2002.04745

Sometimes, you can only train once. No time to tune learning rates.

Zero-shot Hyperparameter transfer: https://www.microsoft.com/en-us/research/blog/µtransfer-a-technique-for-hyperparameter-tuning-of-enormous-neural-networks/

Causal attention is all you need https://arxiv.org/abs/2204.05832

Geoffrey Hinton tweeted a guide to tuning deep learning models. Definitely reading this one tomorrow:

https://twitter.com/geoffreyhinton/status/1616154554768162818

1 GPU = 1.6B tokens per day
https://arxiv.org/abs/2206.07682

S4 ready to prod? Colab with the author of Flash Attention


https://twitter.com/realdanfu/status/1617605971395891201

Transformers post by Lilian Weng got an update

Bidirectional attention and unidirectional attention in one model can be achieved by changing the attention mask. This is called "Prefix LM"

https://www.semanticscholar.org/paper/Generating-Wikipedia-by-Summarizing-Long-Sequences-Liu-Saleh/7570afa31c68e24fce1342b7d67c591787219bc1
The image is from the T5 paper.

Your MLM loss is lying to you https://openreview.net/pdf?id=F5uYcwABMu

https://twitter.com/30SecToMercury/status/1626265029346566145

HMM efficient forward-backward kernel + representation learning method

> However, running dynamic programming in Pytorch is computationally expensive especially for long sequences such as speech.
> This repo provides a C++ kernel of the HMM layer, gradients are computed with batched message passing algorithm (also known as forward-backward).

All you need is a good init, again

1. Model warmup (zipformer) is not only a soft way to speed up training (in contrast to train small, double, continue on medium) IS ALSO a way to mitigate attention rank collapse.

2. Deep attention loses rank very quickly as depth grows — attention can collapse when training large models. Probably hints at an explanation of why my causal -> bidirectional GPT adaptation failed.

https://twitter.com/bobby_he/status/1628388400142204928

when you forget to unscale gradients before clipping your gradient norms look like this (this is LSTM)

Ok you might have caught the wind that I like dabbing with LSTM now. In my second paper on GPT-2 (details later!) reviewers noticed that the LSTM baseline performed surprisingly well and encouraged me to dig deeper. Here's a slide on LSTM limitations from the man himself: https://people.idsia.ch/~juergen/lstm/sld025.htm

In the modern world shooting sparrows with cannons means using context length 1024 and larger when you can fit your data into 128 time steps.

Yes, you can stack 5 MFCC time steps with no overlap no problem

> We note that the models trained on uniformly distributed samples seem to perform the best amongst all the four distributions in all the three languages.

https://aclanthology.org/W19-0128.pdf

Let's find out if resampling my dataset is going to help with the data distributed like on the attached figure.

This video is definitely validating my anxieties when training models https://www.youtube.com/watch?v=p9IxoSkvZ-M

also, training fp16-only, wow