I construct a linear RNN that receives ones as inputs and produces increasingly better approximations to pi as outputs.

The width of the network grows logarithmically wrt the maximum sequence length.

https://gist.github.com/proger/ba147e3953a155d833aae084c1f0cd12

God gave us natural numbers. Humans invented real numbers for oppression (at ETH by the way)

Ready for flow matching? https://bm371613.github.io/conditional-flow-matching/

Very cool speech recognition work: FastConformer + SC-CTC ablations wrt sequence length. 6 layer models can properly use 21.8 minutes of context.

https://x.com/RobFlynnHere/status/1803004030576152944

Noam Shazeer has shared techniques used to scale Transformer inference to 1/5 of Google search traffic

https://research.character.ai/optimizing-inference/

Will Merill's work helped me gain intuition about what classess of tasks are solvable by what kinds of sequence modeling architectures, notably that constant-depth Transformers are not universal (i.e. not even P). The ability to execute any algorithm in P can be achieved by appying chains of thought, scratchpads, pause tokens or infinite depth (see "Universal Transformer"). Chain of thought can be seen as a way to gain infinite depth from a finite network. Check out his recent video

https://www.youtube.com/watch?v=30MhUdapqc8

emotional computing

How to do vector calculus with ChatGPT: ask it to put indices on vectors explicitly, then there is no confusion about what dimensions need to be used when outer products arise.

Example: https://chatgpt.com/share/588e5ca1-b286-4f74-b2ef-bdbb402e3c83

https://report2023.cscs.ch/user-lab

Public projects on the Swiss supercomputer

When making matplotlib figures, it's important to match the font to the rest of the paper: https://x.com/giffmana/status/1632506730897653761

В мене була можливість поспілкуватися з дослідниками robotics + computer vision — їх фокус розробки завжди на обчислення в реальному часі. Мапи і моделі місцевості будуються в реальному часі. Може так є сенс думати і про тренування мультимодальних діалогових систем? Працювати з чатботами натренованими на випадкових текстах зібраних хто зна ким хто зна коли не настільки цікаво.

https://www.youtube.com/watch?v=1t7AWa4SMlo

I got excited and built a fast weight programmer loop into the script demo I shared before. It's using a read-forget-update loop to remember patterns that are coming from the microphone.

Check it out — it learns music and speech on the fly.

How 32 GPU threads cooperate in storing a single 16x16 tile for maximal memory coalescing. This is a swizzled layout from ThunderKittens. The labels on the right image should be transposed for the row-major format.

The format above is required for using the mma.m16n8k16 instruction in PTX 8.5: https://docs.nvidia.com/cuda/parallel-thread-execution/#warp-level-matrix-fragment-mma-16816-float

Turing complete programs in transformers by construction give length generalization out of the box


https://arxiv.org/abs/2407.03310

Cool example of deep learning research going in circles: short conv1d has been reintroduced with QRNN, reintroduced to linear transformers in H3 and made mainstream in Mamba. This convolution makes the network learn faster in the beginning, improves recall and allows a single-layer linear RNN solve associative retrieval — I spent about a month figuring that property out.

This convolution block has made its way into Noam Shazeer's last transformer variant, and people are now writing papers making statements about its expressive power.

The conclusion authors make:

> An important parameterization to explore is replacing the short convolutions within CAT with SSMs

We've been there! Next thing you know you get Conv-SSM-Attention, and that is called Recurrent Gemma 😄

https://arxiv.org/abs/2407.05591

Compiler Explorer supports CUDA C++. See how nvcc emits SASS for thread barriers: https://godbolt.org/z/613zseoGW

Hello from ICML. I am at the tutorial on Data Attribution at Scale. We are studying how to relate model outputs to training inputs. Here are the notes:

https://ml-data-tutorial.org/

The next tutorial by Zeyuan Allen-Zhu is on Physics of Language Models. We study how to apply scientific method to the study of language models with examples. The examples include curation of synthetic data, mechanistic probing, and more.

Tutorial website: https://physics.allen-zhu.com/

Announcement: https://x.com/zeyuanallenzhu/status/1813150298363601102