Ha, selective scan is not using CUTLASS, it's written using cub! https://github.com/state-spaces/mamba/blob/main/csrc/selective_scan/selective_scan_fwd_kernel.cuh

But what about BERT? Monarch matrices have you covered https://www.youtube.com/live/IS59IwGLvVs?si=3yvBYGsOSx3jU2tE

Got a poster

Word2vec received a test of time award at NeurIPS https://papers.nips.cc/paper_files/paper/2013/hash/9aa42b31882ec039965f3c4923ce901b-Abstract.html

Flash attention was inspired by a block nested loop join from databases, and online softmax is inspired by algebraic aggregations (Jim Gray 2007)

Sepp is about to drop xLSTM

Happy New Year!I have started curating a collection of new year speeches. Try talking to one!

https://wilab.org.ua/watch/

Mamba is the first large scale-trained RNN that speaks Ukrainian. It's a large architectural shift from transformers: RNNs use constant memory and have linear complexity (down from quadratic!) when generating sequences.

RNNs is a recipe to get efficient machine learning systems on the end-user device.

I've forked it so you can run it on CPU (tested on Mac) without cuda dependencies:

git clone https://github.com/proger/mamba-cpu
cd mamba-cpu
pip install -e .
python benchmarks/benchmark_generation_mamba_simple.py --model-name "state-spaces/mamba-130m" --prompt "Мій пес написав цей код на Python і вийшло таке:" --topp 0.9 --temperature 0.7 --repetition-penalty 1.2

Вийшла нова версія mlx — 0.0.7, лінк на реліз.

Для тих, хто не знає, що це таке — це як torch, але для процесорів Apple.

На фотографіях все, що було зроблено.

Підтримка формату моделей safetensor — великий лайк 👍🏻, бо, знаєте самі, безпека понад усе.

Практичне використання просте:

— ML інженерія на макбуках;
— Інференс моделей;
— Маємо змогу на макбуках офлайново файн-тюнити моделі у клієнта (100% on-device ML).

Awni Hannun (author of mlx) is also known as the first author of Deep Speech — the first MLP-LSTM large scale speech recognizer trained with CTC on many GPUs on 10k+ hours of data. The data was not that abundant so the team had to hire people to read books.

My best memory of references to his work was my first year at KPI where I took a course from Prof Marchenko. He mentioned Algorithms + Data Structures = Programs book as recommended reading. I hated the course at the time — we were taught to describe algorithms in an arcane graphical syntax and then code it in Turbo Pascal. I rejected Pascal because I thought nobody wrote serious programs in it — I had never been exposed to high quality Pascal code at the time, all interesting code I’ve seen was written in C for various Unix operating systems. I dropped out of school entirely within a year.

I took Master’s level courses last year and experienced pressure to stay less and less grounded in the machine codes. On a piece of paper changing one symbol changes turns the entire computation upside down, while doing so with a computer requires rewriting all of your code. When working through an idea you first imagine it, lay it down in natural language and then slowly formalize.

Turns out this method is called step-wise refinement — a program development technique that Niklas Wirth, an ETH Professor, has published in 1971. That’s the paper I’ll be reading today.

http://pascal.hansotten.com/uploads/wirth/Program%20development%20by%20step-wise%20refinement%20jan%201971%20002.pdf

For every GPU, a group of 32 threads is called a *warp*.
Threads in a warp have an efficient lock-free synchronous communication method called a *shuffle*.

On this screenshot a shfl_up_sync intrinsic is used to simultaneously send a value of the register file (`acc`) up to the 2**e-th neighbour five times, simulating a propagation down a binary tree.

The next figure (from Using CUDA Warp-Level Primitives by NVIDIA) illustrates the same concept using a down-shuffle on a mini-warp of 8 threads. A thread inside a warp is also called a "lane".

Optimizing parallel deep learning systems is a bit like navigating Tokyo by public transit

RWKV scaled to 1T tokens seems to beat Mistral trained on 8 on some multilingual benchmarks

Zero shot translation to Ukrainian in Eagle is about the same as Mistral in 2-shot setting and fine tuned llama2 with 10k examples.


https://twitter.com/RWKV_AI/status/1751797147492888651

No reason to use transformer decoders any more for LLMs :)

RNNs are faster to train, faster in inference and are more data efficient.

Arpa count tables? RNN weight matrices? Decision trees? Suffix arrays!

https://arxiv.org/abs/2401.17377

wandb is in a good mood today: