When making matplotlib figures, it's important to match the font to the rest of the paper: https://x.com/giffmana/status/1632506730897653761

В мене була можливість поспілкуватися з дослідниками robotics + computer vision — їх фокус розробки завжди на обчислення в реальному часі. Мапи і моделі місцевості будуються в реальному часі. Може так є сенс думати і про тренування мультимодальних діалогових систем? Працювати з чатботами натренованими на випадкових текстах зібраних хто зна ким хто зна коли не настільки цікаво.

https://www.youtube.com/watch?v=1t7AWa4SMlo

I got excited and built a fast weight programmer loop into the script demo I shared before. It's using a read-forget-update loop to remember patterns that are coming from the microphone.

Check it out — it learns music and speech on the fly.

How 32 GPU threads cooperate in storing a single 16x16 tile for maximal memory coalescing. This is a swizzled layout from ThunderKittens. The labels on the right image should be transposed for the row-major format.

The format above is required for using the mma.m16n8k16 instruction in PTX 8.5: https://docs.nvidia.com/cuda/parallel-thread-execution/#warp-level-matrix-fragment-mma-16816-float

Turing complete programs in transformers by construction give length generalization out of the box


https://arxiv.org/abs/2407.03310

Cool example of deep learning research going in circles: short conv1d has been reintroduced with QRNN, reintroduced to linear transformers in H3 and made mainstream in Mamba. This convolution makes the network learn faster in the beginning, improves recall and allows a single-layer linear RNN solve associative retrieval — I spent about a month figuring that property out.

This convolution block has made its way into Noam Shazeer's last transformer variant, and people are now writing papers making statements about its expressive power.

The conclusion authors make:

> An important parameterization to explore is replacing the short convolutions within CAT with SSMs

We've been there! Next thing you know you get Conv-SSM-Attention, and that is called Recurrent Gemma 😄

https://arxiv.org/abs/2407.05591

Compiler Explorer supports CUDA C++. See how nvcc emits SASS for thread barriers: https://godbolt.org/z/613zseoGW

Hello from ICML. I am at the tutorial on Data Attribution at Scale. We are studying how to relate model outputs to training inputs. Here are the notes:

https://ml-data-tutorial.org/

The next tutorial by Zeyuan Allen-Zhu is on Physics of Language Models. We study how to apply scientific method to the study of language models with examples. The examples include curation of synthetic data, mechanistic probing, and more.

Tutorial website: https://physics.allen-zhu.com/

Announcement: https://x.com/zeyuanallenzhu/status/1813150298363601102

Use a world model to interpolate between two learning algorithms

Nando de Freitas giving love to OpenAI https://x.com/nandodf/status/1816530449830936805

The tutorial video is now up https://youtu.be/yBL7J0kgldU?si=II4_C2fCaUQnkNyS

Юрій рухає локомотив

https://x.com/HumansNoContext/status/1821925436512665639

How to tune hyperparameters. Notice how smoothly the behavior changes when the speed is varied.

Do better science by watching the computer do it with you:

https://github.com/SakanaAI/AI-Scientist

https://x.com/hamelhusain/status/1824452022890119658?

Programming languages are not the problem, time to check your writing skills.