https://www.youtube.com/watch?v=i2Mp_Bc14WI

Since FPV pretraining is actually mainstream (= best ICLR honorable mention paper) I'm now obsessed with looking up animal camera videos on YouTube. Do you have a favorite of your own?

GH200 consume 900W per GPU

RNNs = orders of magnitude compute and memory savings. Latest results from my thesis.

baby reproduction of chinchilla scaling laws, i want to reuse this https://x.com/Locchiu/status/1797751414548246624

я: радію що код працює
код:

OpenAI has released the simplest sparse autoencoder using top-k activations instead of L1 regularization to convert dense feature superpositions generated by neural networks to sparse interpretable features

Fun fact: L1 regularization helps sparsity for the same reason why the sample estimator for mean absolute error is the median

https://x.com/norabelrose/status/1798766340427403472?

While I am trying to use more matmuls, people are getting rid of them

https://arxiv.org/abs/2406.02528

Good news for pretraining: using cosine schedule is unnecessary.

Relying on cosine schedules has been popularized in Chinchilla and nanoGPT and I'm happy to see multiple confirations that it's not necessary: you can just train with constant LR for most of the time and use 20% to cooldown linearly. You can even to better it you use Noam cooldown (1-sqrt(t)). As a bonus, constant + linear is more forgiving to not searching for a good learning rate.

https://arxiv.org/abs/2405.18392

Fun fact: uk4b large was trained with a constant learning rate without full decay

Remember Monarch Mixer? Dense linear layers soon to be replaced with more structured counterparts

https://x.com/andrewgwils/status/1800532157406011523?

I construct a linear RNN that receives ones as inputs and produces increasingly better approximations to pi as outputs.

The width of the network grows logarithmically wrt the maximum sequence length.

https://gist.github.com/proger/ba147e3953a155d833aae084c1f0cd12

God gave us natural numbers. Humans invented real numbers for oppression (at ETH by the way)

Ready for flow matching? https://bm371613.github.io/conditional-flow-matching/

Very cool speech recognition work: FastConformer + SC-CTC ablations wrt sequence length. 6 layer models can properly use 21.8 minutes of context.

https://x.com/RobFlynnHere/status/1803004030576152944

Noam Shazeer has shared techniques used to scale Transformer inference to 1/5 of Google search traffic

https://research.character.ai/optimizing-inference/

Will Merill's work helped me gain intuition about what classess of tasks are solvable by what kinds of sequence modeling architectures, notably that constant-depth Transformers are not universal (i.e. not even P). The ability to execute any algorithm in P can be achieved by appying chains of thought, scratchpads, pause tokens or infinite depth (see "Universal Transformer"). Chain of thought can be seen as a way to gain infinite depth from a finite network. Check out his recent video

https://www.youtube.com/watch?v=30MhUdapqc8

emotional computing

How to do vector calculus with ChatGPT: ask it to put indices on vectors explicitly, then there is no confusion about what dimensions need to be used when outer products arise.

Example: https://chatgpt.com/share/588e5ca1-b286-4f74-b2ef-bdbb402e3c83

https://report2023.cscs.ch/user-lab

Public projects on the Swiss supercomputer