No reason to use transformer decoders any more for LLMs :)

RNNs are faster to train, faster in inference and are more data efficient.

Arpa count tables? RNN weight matrices? Decision trees? Suffix arrays!

https://arxiv.org/abs/2401.17377

wandb is in a good mood today:

https://twitter.com/DlCountdown/status/1764278990011813975

NeurIPS conference submission deadline is in late May, workshops deadlines will probably be August

https://x.com/mlstreettalk/status/1765701266221522986

This is what you learn as a side note in our Machine Learning course at USI. Glad Yann communicates this message to a large audience. Recurrent neural nets can do anything, but gradient descent won’t find everything.

Математика — це наука трансмісії простих ідей про регулярність світу між людьми. Це мова програмування, на якій ви стисло описуєте вашу думку, щоб завантажити її у свідомість ваших колег з абсолютною точністю.

Єгор зробив канал, де ми вчимось покращити навичку точної комунікації бібліотеки математичних ідей серед розробників штучного інтелекту.

Доєднуйтесь: https://t.me/applied_math_uk

Перший реліз Hippogriff: моєї імплементації архітектури Griffin, гібрид локального трансформера з sliding multi query attention (як mistral) та лінійної рекурентності (як mamba/rwkv)

В середині пакету ви також знайдете мій крафтовий трейнлуп з діагностиками активацій та стану вагів.

https://github.com/proger/hippogriff

I love MATLAB/Octave. It's plotting experience is so smooth compared to matplotlib! Numpy/torch have their array APIs copied from MATLAB, so the amount of things you need to remember to move from Python is very small.

To train transformers, you need a lot of diverse data. Let's use online RL to generate data!

Check out my new repo, control: Soft Actor Critic to produce experience trajectories

https://github.com/proger/control

Bayesian Flow Networks (BFNs) link iterative denoising diffusion and recursive estimation of distribution parameters.

In my new post, I constrast autoregressive generative modeling (prevalent in language) and recursive Bayesian estimation of all parameters jointly.

https://proger.github.io/posts/bfn/normal.html

Excited to see the first book on differentiable programming. It explicitly talks about how to encode regular programs into structures that have gradient flow. https://arxiv.org/abs/2403.14606

Discrete Bayesian Flow Networks teaser

I stumbled on this paper on Efficient Backprop from LeCun et al when discussing the differences between internal covariate shift and input whitening.

This work provides a comprehensive overview of tricks that are necessary succeessfully train deep models — why and how to initialize weights, choose nonlinearities (to some extent), how to choose and preprocess training data, how to choose learning rates, what is the basic optimization dynamics behavior and how to use the Hessian to diagnose it: https://cseweb.ucsd.edu/classes/wi08/cse253/Handouts/lecun-98b.pdf

Balancing sequence lengths in your dataset is the best augmentation you can do to successfully train a Transformer

https://aclanthology.org/2021.emnlp-main.650/

The same principle (sequence length distribution needs to be uniform) actually applies to RNNs too. I trained a SHA-RNN on byte-level ukpron (grapheme to phoneme task) and making sequence lengths uniform was key to get the model to work: https://huggingface.co/darkproger/ukpron

When we were already training uk4b, Karpathy posted a note that padding the number of rows in the tied input-output embedding table to a multiple of 8 (from 50257 to 50304) gave a significant training speedup. This thread from Horace He characterizes the space of phenomena that are related to this https://twitter.com/cHHillee/status/1630274804795445248

In short, GEMM kernels access memory in tile blocks, and when the blocks are aligned to the cache line the SM spends the least amount of memory access operations to feed the kernel.

In Accelerated Scan (it is my high performance training kernel for linear RNNs — it's responsible for computing the recurrence for all tokens when training the network or reading the prompt at inference), the backward kernel would read the memory in reverse. Alex Nichol found that loading the memory in reverse in chunks of 4 would make the access aligned — and speed up the loads!

That change in turn allowed fusing the reverse scan with a few other kernels, resulting in 30-50% training speedups. Coming in Accelerated Scan 0.2.