Bidirectional attention and unidirectional attention in one model can be achieved by changing the attention mask. This is called "Prefix LM"

https://www.semanticscholar.org/paper/Generating-Wikipedia-by-Summarizing-Long-Sequences-Liu-Saleh/7570afa31c68e24fce1342b7d67c591787219bc1
The image is from the T5 paper.

Your MLM loss is lying to you https://openreview.net/pdf?id=F5uYcwABMu

https://twitter.com/30SecToMercury/status/1626265029346566145

HMM efficient forward-backward kernel + representation learning method

> However, running dynamic programming in Pytorch is computationally expensive especially for long sequences such as speech.
> This repo provides a C++ kernel of the HMM layer, gradients are computed with batched message passing algorithm (also known as forward-backward).

All you need is a good init, again

1. Model warmup (zipformer) is not only a soft way to speed up training (in contrast to train small, double, continue on medium) IS ALSO a way to mitigate attention rank collapse.

2. Deep attention loses rank very quickly as depth grows — attention can collapse when training large models. Probably hints at an explanation of why my causal -> bidirectional GPT adaptation failed.

https://twitter.com/bobby_he/status/1628388400142204928

when you forget to unscale gradients before clipping your gradient norms look like this (this is LSTM)

Ok you might have caught the wind that I like dabbing with LSTM now. In my second paper on GPT-2 (details later!) reviewers noticed that the LSTM baseline performed surprisingly well and encouraged me to dig deeper. Here's a slide on LSTM limitations from the man himself: https://people.idsia.ch/~juergen/lstm/sld025.htm

In the modern world shooting sparrows with cannons means using context length 1024 and larger when you can fit your data into 128 time steps.

Yes, you can stack 5 MFCC time steps with no overlap no problem

> We note that the models trained on uniformly distributed samples seem to perform the best amongst all the four distributions in all the three languages.

https://aclanthology.org/W19-0128.pdf

Let's find out if resampling my dataset is going to help with the data distributed like on the attached figure.

This video is definitely validating my anxieties when training models https://www.youtube.com/watch?v=p9IxoSkvZ-M

also, training fp16-only, wow

Proud to say I have a first paper. Will be sharing soon.

Long time no see! It seems that a transformer learns a separate algorithm (e.g. alignment) for each input length. I started observing this behaviour in speech transformers. Here's a gist of what I'm working on. Stay tuned

In our previous work we trained several GPT-style models on UberText 2.0 and found that

1. the bigger transformer the faster to train (applying Chinchilla scaling laws to Ukrainian datasets)
2. small transformers seem to struggle with context lookup, which we fix by constraining decoding
3. small multi-task decoder-only models lose to small encoder-decoder models pretrained more (possibly out of domain) data and finetuned on one task (in retrospect really hoped that one would not be true)

Please enjoy:

https://aclanthology.org/2023.unlp-1.4/


You can play with the models by doing pip install haloop, see instructions in https://github.com/proger/haloop

Since then we learned that context lookup (aka induction heads) requires 2^14 *tasks* in decoder-only models, and it's feasible using synthetic pretraining (Kirsch on GPICL 2022). We also learned a thing or two about learning rate schedules, like pretend convergences happen faster if you decay early (search for Vaswani on arxiv), Dmytro is currently playing with that on TPUs.

Now it's time to take all that to speech!

Oh, and metadata pretraining is awesome. Stop cleaning your data, let the transformer slurp it all in.

So far I'm struggling with preventing forgetting when shoving the model with real speech data after synthetic pretraining.

I decided to park that for now and come back to tuning the model. I have started multiplicative relative position encodings in encoder and decoder attention and left cross attention on its own.

Things I'd like to try to prevent cross attention drift:

1. use relative position encodings but interpolate token positions between speech frame positions (i must have seen this somewhere already)

2. put a gaussian window around every token with a learnable sigma. This idea is one year yonger than Alibi and uses a different additive attention bias: http://www.interspeech2020.org/uploadfile/pdf/Thu-3-10-8.pdf

So the gaussian window around every token for speech paper does not mention how to compute the center of the gaussian.

I found an even older paper (Alex Graves 2013) for a different task (handwriting synthesis) that suggests to learn the centers and enforce their monotonicity. Which I am about to do!

The simple trick worked! It fixed the drift problem with long sentences. In the mean time I started using my third block implementation for the encoder and found this bug. This bug let the decoder (with cross attention) learn just fine!

New paper by Alex Graves et al, Bayesian diffusion for language modeling.

https://arxiv.org/abs/2308.07037