Pretty cool that in a progressive superposition of MLPs scaling activations for older MLPs just by a single large value is enough to restore access to the old task knowledge written in the "old" MLP.

SGD seems to be incentivized to find larger weights for the second MLP in the continual training scenario so its activations always "beat" the activations of the first network.

I am referring to this experiment: https://github.com/kazuki-irie/kv-memory-brain/blob/master/Forgetting_and_recovery.ipynb

When training neural networks with SGD, the learning rate critically depends on the batch size. If you change one parameter you usually need to sweep the other. This a heat map that measures log(1-accuracy) on the test set as a function of width, batch size and learning rate.

Decided to collect a few tokens from telegram for the UNLP shared task pretraining. Will 1B be enough? At 350M words now.

I ran a baseline for UNLP2025 shared task using gpt-4o-mini-2024-07-18 and gpt-4o-2024-08-06 with a basic prompt and structured outputs asking for reasoning and labels as binary flags. The macro f1 scores for technique detection are 0.32 and 0.34, mini model tends to trade precision for extra recall.

> Так, всі вже зареєструвалися на спільну задачу UNLP, а ти чого чекаєш?

> Зареєструйся на спільну задачу UNLP, інакше будеш лузером

> Так а чого ти бертом тегатимеш слова в реченні, це ж просто енкодер. Енкодери то обмежена архітектура, нею AGI не зробиш

> Ллама то класна ллмка, і все тут сказано.

> Кажеш, BERT — то просто енкодер, але а як же тоді його топові результати у всяких бенчмарках? (this one is especially weird to me — it feels like it's addressing the point but it's actually diverting to another one!)

Leo has sent me a paper that talks about how neural networks learn fourier features to perform addition — a recurrent topic in NN literature. It's story time.

In a bar after my master's thesis defense, Professor Stefan Wolf wrote an equation on a napkin:

$ \pi/4 = \sum_n^\infty (-1)^n / (2n + 1) $

This was the shortest program to compute pi I've ever seen yet, so I was incredibly excited. If I spend more test time compute on it, I get a better approximation of pi. It's not the fastest program in terms of convergence speed, but definitely a one short enough that I can remember it. It was an approximation due to Leibniz.

Next morning I got interested in finding a linear RNN curcuit that encodes the Leibniz approximation. A linear RNN uses a cumulative sum operation at its heart, so given a stream of input ones it outputs a count of ones so far "for free" (by construction / inductive bias).

I can express the division as a nonlinearity, but how to express sign flipping (-1)^n as n goes up?

A natural way of encoding a flipping sign is using base 2 representation: if you encode n using binary then the least significant bit will be alternating.
In a neural network we can express each bit position using a separate dimension. Consider this linear feature map:

def binary(digits: int):
"Make a basis of powers of two of dimension digits, lowest bits first"
return 1 << np.arange(digits)

After mapping a sum into this high dimensional space you only need to read off the leading dimension, aka the highest frequency component.

What is a natural learnable approximation of representing numbers in base K? Well, it seems that it is somewhere in the basis of sinusoids of different frequencies.

If you're curious about what the final construction looks like, check out this program: https://gist.github.com/proger/ba147e3953a155d833aae084c1f0cd12

Behind the scenes of early versions of ChatGPT by John Schulman and Barret Zoph: https://x.com/johnschulman2/status/1891539960743743756?

https://youtu.be/9_PepvnqIfU

There are no authorities in science

science says we will remain smart as long as we want
https://www.science.org/doi/full/10.1126/sciadv.ads1560?af=R

Incompleteness? Just avoid it

https://arxiv.org/abs/1408.3821

Швайнокарась Петриківкою на екрані центру керування космічними місіями

Археологічний артефакт — Джеремі Говард розповідає, що його загальна мовна лстмка ULMFiT (Universal Language Model Fine-tuning for Text Classification) була мотивацією будувати GPT-1. Під час адаптацііі на фінальну задачу замість лори тренували всю мережу з різними льорнінг рейтами на різні шари.

https://x.com/jeremyphoward/status/1906478657100755011

Що додати в ллмку? https://t.co/XKB4XxjREV