Вийшла нова версія mlx — 0.0.7, лінк на реліз.

Для тих, хто не знає, що це таке — це як torch, але для процесорів Apple.

На фотографіях все, що було зроблено.

Підтримка формату моделей safetensor — великий лайк 👍🏻, бо, знаєте самі, безпека понад усе.

Практичне використання просте:

— ML інженерія на макбуках;
— Інференс моделей;
— Маємо змогу на макбуках офлайново файн-тюнити моделі у клієнта (100% on-device ML).

Awni Hannun (author of mlx) is also known as the first author of Deep Speech — the first MLP-LSTM large scale speech recognizer trained with CTC on many GPUs on 10k+ hours of data. The data was not that abundant so the team had to hire people to read books.

My best memory of references to his work was my first year at KPI where I took a course from Prof Marchenko. He mentioned Algorithms + Data Structures = Programs book as recommended reading. I hated the course at the time — we were taught to describe algorithms in an arcane graphical syntax and then code it in Turbo Pascal. I rejected Pascal because I thought nobody wrote serious programs in it — I had never been exposed to high quality Pascal code at the time, all interesting code I’ve seen was written in C for various Unix operating systems. I dropped out of school entirely within a year.

I took Master’s level courses last year and experienced pressure to stay less and less grounded in the machine codes. On a piece of paper changing one symbol changes turns the entire computation upside down, while doing so with a computer requires rewriting all of your code. When working through an idea you first imagine it, lay it down in natural language and then slowly formalize.

Turns out this method is called step-wise refinement — a program development technique that Niklas Wirth, an ETH Professor, has published in 1971. That’s the paper I’ll be reading today.

http://pascal.hansotten.com/uploads/wirth/Program%20development%20by%20step-wise%20refinement%20jan%201971%20002.pdf

For every GPU, a group of 32 threads is called a *warp*.
Threads in a warp have an efficient lock-free synchronous communication method called a *shuffle*.

On this screenshot a shfl_up_sync intrinsic is used to simultaneously send a value of the register file (`acc`) up to the 2**e-th neighbour five times, simulating a propagation down a binary tree.

The next figure (from Using CUDA Warp-Level Primitives by NVIDIA) illustrates the same concept using a down-shuffle on a mini-warp of 8 threads. A thread inside a warp is also called a "lane".

Optimizing parallel deep learning systems is a bit like navigating Tokyo by public transit

RWKV scaled to 1T tokens seems to beat Mistral trained on 8 on some multilingual benchmarks

Zero shot translation to Ukrainian in Eagle is about the same as Mistral in 2-shot setting and fine tuned llama2 with 10k examples.


https://twitter.com/RWKV_AI/status/1751797147492888651

No reason to use transformer decoders any more for LLMs :)

RNNs are faster to train, faster in inference and are more data efficient.

Arpa count tables? RNN weight matrices? Decision trees? Suffix arrays!

https://arxiv.org/abs/2401.17377

wandb is in a good mood today:

https://twitter.com/DlCountdown/status/1764278990011813975

NeurIPS conference submission deadline is in late May, workshops deadlines will probably be August

https://x.com/mlstreettalk/status/1765701266221522986

This is what you learn as a side note in our Machine Learning course at USI. Glad Yann communicates this message to a large audience. Recurrent neural nets can do anything, but gradient descent won’t find everything.

Математика — це наука трансмісії простих ідей про регулярність світу між людьми. Це мова програмування, на якій ви стисло описуєте вашу думку, щоб завантажити її у свідомість ваших колег з абсолютною точністю.

Єгор зробив канал, де ми вчимось покращити навичку точної комунікації бібліотеки математичних ідей серед розробників штучного інтелекту.

Доєднуйтесь: https://t.me/applied_math_uk

Перший реліз Hippogriff: моєї імплементації архітектури Griffin, гібрид локального трансформера з sliding multi query attention (як mistral) та лінійної рекурентності (як mamba/rwkv)

В середині пакету ви також знайдете мій крафтовий трейнлуп з діагностиками активацій та стану вагів.

https://github.com/proger/hippogriff

I love MATLAB/Octave. It's plotting experience is so smooth compared to matplotlib! Numpy/torch have their array APIs copied from MATLAB, so the amount of things you need to remember to move from Python is very small.

To train transformers, you need a lot of diverse data. Let's use online RL to generate data!

Check out my new repo, control: Soft Actor Critic to produce experience trajectories

https://github.com/proger/control

Bayesian Flow Networks (BFNs) link iterative denoising diffusion and recursive estimation of distribution parameters.

In my new post, I constrast autoregressive generative modeling (prevalent in language) and recursive Bayesian estimation of all parameters jointly.

https://proger.github.io/posts/bfn/normal.html