Don’t regulate AI, regulate uses of AI

https://laion.ai/blog/transparent-ai/

torch.export can unroll for loops that depend on the time dimension of the input. Sequential RNNs go brrrr!

New work from IDSIA. Transformer's largest bottleneck is a feedforward MLP block that expands hidden dimension four times and shrinks it back.

Instead of running all parts of this network on every request, sparse gating decides what subnetwork to run depending on the input. This is straightforward at inference time but hard to backpropagate at training time.

Sparsely-gated Mixtures of Experts date back to Ivakhnenko and Lapa (see Section 4) , and now feature an open source implementation from Robert.

https://arxiv.org/abs/2310.10837

Btw does anybody know how to use Triton kernels with torch.compile without graph breaks?

ICLR 2024 has a track that accepts anonymized blog posts. Submissions are now open:

https://x.com/fpedregosa/status/1722197379418665247?

It’s time to build backprop over HTTP:

https://x.com/wightmanr/status/1724841649233350659

Alec Radford on language models. He gave this lecture right after he completed his work on GPT-2.

https://www.youtube.com/watch?v=BnpB3GrpsfM

Meta learning has its own *class* at Stanford

https://x.com/_jasonwei/status/1729585618311950445?

https://nips.cc/virtual/2023/events/journal_track_2023


My VAE paper is on NeurIPS Journal Track. See you in New Orleans!

Ha, selective scan is not using CUTLASS, it's written using cub! https://github.com/state-spaces/mamba/blob/main/csrc/selective_scan/selective_scan_fwd_kernel.cuh

But what about BERT? Monarch matrices have you covered https://www.youtube.com/live/IS59IwGLvVs?si=3yvBYGsOSx3jU2tE

Got a poster

Word2vec received a test of time award at NeurIPS https://papers.nips.cc/paper_files/paper/2013/hash/9aa42b31882ec039965f3c4923ce901b-Abstract.html

Flash attention was inspired by a block nested loop join from databases, and online softmax is inspired by algebraic aggregations (Jim Gray 2007)

Sepp is about to drop xLSTM

Happy New Year!I have started curating a collection of new year speeches. Try talking to one!

https://wilab.org.ua/watch/

Mamba is the first large scale-trained RNN that speaks Ukrainian. It's a large architectural shift from transformers: RNNs use constant memory and have linear complexity (down from quadratic!) when generating sequences.

RNNs is a recipe to get efficient machine learning systems on the end-user device.

I've forked it so you can run it on CPU (tested on Mac) without cuda dependencies:

git clone https://github.com/proger/mamba-cpu
cd mamba-cpu
pip install -e .
python benchmarks/benchmark_generation_mamba_simple.py --model-name "state-spaces/mamba-130m" --prompt "Мій пес написав цей код на Python і вийшло таке:" --topp 0.9 --temperature 0.7 --repetition-penalty 1.2

Вийшла нова версія mlx — 0.0.7, лінк на реліз.

Для тих, хто не знає, що це таке — це як torch, але для процесорів Apple.

На фотографіях все, що було зроблено.

Підтримка формату моделей safetensor — великий лайк 👍🏻, бо, знаєте самі, безпека понад усе.

Практичне використання просте:

— ML інженерія на макбуках;
— Інференс моделей;
— Маємо змогу на макбуках офлайново файн-тюнити моделі у клієнта (100% on-device ML).

Awni Hannun (author of mlx) is also known as the first author of Deep Speech — the first MLP-LSTM large scale speech recognizer trained with CTC on many GPUs on 10k+ hours of data. The data was not that abundant so the team had to hire people to read books.