Teaser: maximizing data selection with higher error rates than random using expected gradient length and monte carlo dropout.

This is a prototype I'm developing to clean uk1e2

Surprisingly, low resolution (10fps) discrete features work for synthesis

https://arxiv.org/abs/2306.01084

На інтерспічі я бачив декілька постерів які презентували *останні* автори. Постер по 100fps hubert презентував Watanabe, постер по Neural HMM презентував Gustav Eje Henter. Це були одні з найкращих презентацій постерів, максимально детальні та повні несподіваних інсайтів.

З цього я вивів таке правило: якщо ваш керівник не може презентувати вашу роботу краще за вас, вам не потрібен цей керівник.

On the difference between supervised and reinforcement learning.


Most supervised learning reduces to performing a softmax over evidence (aka "hidden activations") and then maximizing likelihood (minimizing cross entropy). When backpropagating, softmax ensures you get a very high error signal for the wrong answer towards the right answer (or multiple answers if you don't have a spiky distribution, imagine cases like label smoothing or distribution matching).


The softmax+cross entropy let your learning process consider *all outcomes at once*. This is better visible when you compute gradients of likelihood and softmax. Something you don’t get to see if you use autodiff all the time.

This is not the case with reinforcement learning. Consider policy gradient: you sample an action, record its likelihood and then when you get the reward you scale the likelihood of a *single action* by that reward. You get unbiased and high variance learning signal estimates because you don't have updates for all outputs in a softmax! You don’t have it in reinforcement learning because you are in the decision *process* framework: there is no way to evaluate all alternative actions and choose the best, you only get to evaluate one.


The morale is: use supervised learning when you can.

IMPALA is a reinforcement learning architecture where distributed agents collect experience using latest available policy and submit it to the experience buffer. The training loop uses batches from the buffer to make updates.

This way the learner can pick up data from a buffer that’s generated by an outdated policy. When computing state values, likelihoods of old policies are weighted against the current policy and clipped to avoid variance in updates.

This importance weighting allows the architecture to work *off-policy*, incorporating old data into the training.

If you keep clipping and remove importance weighting you get PPO. PPO is a method used for RLHF in GPT. It trades off massive data requirements for extra stabilty by being an *on-policy* method: you can’t reuse any old data after a policy update is done.



Check out my fork of IMPALA at https://github.com/proger/curiosity. I use it to research learning in procedurally generated environments with sparse rewards.

Learning to learn with gradient descent (Hochreiter 2001) is all you need. "Emergent abilities" of LLMs are shown to be instances of in-context learning.

In-context learning is the ability of transformers and RNNs to apply learned learning algorithms on data present in its prompt.

Work by group of Iryna Gurevych, of sentence-transformers fame.

https://twitter.com/UKPLab/status/1699348822609060158

"Our choice of T5 models, of which the largest (T5-Large) has 770M parameters, enables us to evaluate models at a scale where instruction tuning proves effective. Our experiments involving T5-Large also show that there is no difference between the zero-shot and few-shot settings. This suggests that the model’s scale is insufficient to support explicit in-context learning effectively"

Scaling laws say that it’s not only about the size, but also about the data. It just needs more relevant data to get ICL.

I’m going to post the key paper on learning to learn (aka ICL) here.

ICANN 2024 is happening in Lugano in September next year. If you DM me I'll tell you who the likely keynote is :). They are not listed on the page below but paper rules are, check it out and come see me in my school, I'd love to hang out with you:

http://www.wikicfp.com/cfp/servlet/event.showcfp?eventid=175770&copyownerid=138026


- Opening for submissions: February 15, 2023
- Deadline for full paper submission: March 15, 2023
- Notication of acceptance: May 20, 2023

Smaller LSTMs outperform larger Transformers on datasets sampled from arbitrary HMMs designed to trigger in-context learning

https://openreview.net/forum?id=RdJVFCHjUMI

https://x.com/jonathanleroux/status/1702298961212576134

What happens when you wait until the last hour to update your paper

Running artificial curiosity sweeps on CSCS

Do not throw away your value network, use it to score paths:

https://arxiv.org/abs/2309.15028