4o has become so good at math that it can analyze recurrences for me (and tolerate my typos)

Doing symbolic differentiation with loops is a piece of cake, I don't have to explain what a "backwards pass" is: https://chatgpt.com/share/858b2882-9d29-442e-a0cb-7e3afb24abab

reproducing gpt-2 small is now about 10x cheaper than last year:

https://x.com/karpathy/status/1795484547267834137/photo/2

OMNI reboot as OMNI-EPIC: ask ChatGPT to generate environments for your agent that are progressively harder to solve!

https://x.com/jeffclune/status/1795787632435212732

The recent CoPE paper made it all click. Gated Linear RNNs are transformers with token-dependent masks without softmax. If you would like to know how to calculate the attention mask, read my upcoming thesis :D

Spoiler: it’s dynamic programming!

Dynamic programming is also used to find optimal tensor contraction paths, expoiting associativity of tensor products

The screenshot uses opt_einsum

Тимофій Милованов дає приклади байесівського інференсу:
1. дати prior через промпт моделі (спитати що таке organizational culture principles by Ed Schade)
2. дати likelihood моделі (зкинути датасет та спитати identify cultural misalignments)
3. отримати posterior — обговорити результат

https://youtu.be/LTpWpadoT_U

A few of recipes for cleaning pretraining data are popping up:

- FineWeb-Edu, maximizes scores on downstream educational benchmarks

https://x.com/gui_penedo/status/1797173053123916036

The ablations are 1.82B training runs for 30B tokens (almost same as uk4b-large in data, >2x the size) — ~1 GPU-month per one ablation!

> Our ablation models were trained using nanotron. Our "ablation models" have 1.82B parameters (including embeddings), used the Llama architecture with a 2048 sequence length, a global batch size of ~2 million tokens, and the GPT2 tokenizer. For most ablations we trained on ~28B tokens (roughly the Chinchilla
optimal training size for this model size). To confirm relative performance improvements after each step of filtering we conducted longer training runs on 350 billion tokens as mentioned further below


And similar work from the FLAN collection author:

https://arxiv.org/abs/2305.13169

https://www.youtube.com/watch?v=i2Mp_Bc14WI

Since FPV pretraining is actually mainstream (= best ICLR honorable mention paper) I'm now obsessed with looking up animal camera videos on YouTube. Do you have a favorite of your own?

GH200 consume 900W per GPU

RNNs = orders of magnitude compute and memory savings. Latest results from my thesis.

baby reproduction of chinchilla scaling laws, i want to reuse this https://x.com/Locchiu/status/1797751414548246624

я: радію що код працює
код:

OpenAI has released the simplest sparse autoencoder using top-k activations instead of L1 regularization to convert dense feature superpositions generated by neural networks to sparse interpretable features

Fun fact: L1 regularization helps sparsity for the same reason why the sample estimator for mean absolute error is the median

https://x.com/norabelrose/status/1798766340427403472?

While I am trying to use more matmuls, people are getting rid of them

https://arxiv.org/abs/2406.02528

Good news for pretraining: using cosine schedule is unnecessary.

Relying on cosine schedules has been popularized in Chinchilla and nanoGPT and I'm happy to see multiple confirations that it's not necessary: you can just train with constant LR for most of the time and use 20% to cooldown linearly. You can even to better it you use Noam cooldown (1-sqrt(t)). As a bonus, constant + linear is more forgiving to not searching for a good learning rate.

https://arxiv.org/abs/2405.18392

Fun fact: uk4b large was trained with a constant learning rate without full decay

Remember Monarch Mixer? Dense linear layers soon to be replaced with more structured counterparts

https://x.com/andrewgwils/status/1800532157406011523?

I construct a linear RNN that receives ones as inputs and produces increasingly better approximations to pi as outputs.

The width of the network grows logarithmically wrt the maximum sequence length.

https://gist.github.com/proger/ba147e3953a155d833aae084c1f0cd12