Learning to learn with gradient descent (Hochreiter 2001) is all you need. "Emergent abilities" of LLMs are shown to be instances of in-context learning.

In-context learning is the ability of transformers and RNNs to apply learned learning algorithms on data present in its prompt.

Work by group of Iryna Gurevych, of sentence-transformers fame.

https://twitter.com/UKPLab/status/1699348822609060158

"Our choice of T5 models, of which the largest (T5-Large) has 770M parameters, enables us to evaluate models at a scale where instruction tuning proves effective. Our experiments involving T5-Large also show that there is no difference between the zero-shot and few-shot settings. This suggests that the model’s scale is insufficient to support explicit in-context learning effectively"

Scaling laws say that it’s not only about the size, but also about the data. It just needs more relevant data to get ICL.

I’m going to post the key paper on learning to learn (aka ICL) here.

ICANN 2024 is happening in Lugano in September next year. If you DM me I'll tell you who the likely keynote is :). They are not listed on the page below but paper rules are, check it out and come see me in my school, I'd love to hang out with you:

http://www.wikicfp.com/cfp/servlet/event.showcfp?eventid=175770&copyownerid=138026


- Opening for submissions: February 15, 2023
- Deadline for full paper submission: March 15, 2023
- Notication of acceptance: May 20, 2023

Smaller LSTMs outperform larger Transformers on datasets sampled from arbitrary HMMs designed to trigger in-context learning

https://openreview.net/forum?id=RdJVFCHjUMI

https://x.com/jonathanleroux/status/1702298961212576134

What happens when you wait until the last hour to update your paper

Running artificial curiosity sweeps on CSCS

Do not throw away your value network, use it to score paths:

https://arxiv.org/abs/2309.15028

ACL 2023 rolling review: https://www.aclweb.org/portal/content/submission-dates-and-process-eaclnaacl-and-acl-2024

You can submit to one conference (EACL or NAACL), but commit to another (e.g. ACL) after receiving reviews with an option to resubmit.

Dates:

15 Oct 2023: October ARR Cycle - EACL submission deadline
15 Dec 2023: ARR reviews & meta-reviews available to authors of October cycle
15 Dec 2023: December ARR Cycle - NAACL submission deadline
20 Dec 2023: EACL commitment deadline
15 Jan 2024: EACL decisions available
15 Feb 2024: ARR reviews & meta-reviews available to authors of December cycle
15 Feb 2024: February ARR Cycle - ACL submission deadline
20 Feb 2024: NAACL commitment deadline
15 Mar 2024: NAACL decisions available
15 Apr 2024: ARR reviews & meta-reviews available to authors of February cycle
20 Apr 2024: ACL commitment deadline
15 May 2024: ACL decisions available

Bayesian Flow Networks official code has been released. https://github.com/nnaisense/bayesian-flow-networks

How to count experience? This post introduces a counter for embeddings: https://proger.github.io/posts/counter/counter.html

I made a post on neural circuits, in the context of the gradient vanishing problem, linearity and autoregression:

https://proger.github.io/posts/neural-circuits/recurrent.html

Deep Reinforcement Learning has been used to develop champion-level FPV drone racers, with a paper in Nature appearing 10 years after DQN was developed to beat humans in Atari. Key success lies in development of high quality simulations and sim2real policy transfer. RL is no longer about games.

One of the authors, Vladlen Koltun was giving a seminar in MIT: https://www.youtube.com/watch?v=vNFTcD3QMn0

Here’s the system demonstration of Swift https://www.youtube.com/watch?v=fBiataDpGIo

Meme moment, my pure torch linearized LSTM implementation just beat GPT in throughput in the small (110M) setting. This is the fastest LSTM implementation to date. More experiments soon.

zero-shot learning by searching for support vectors for generative tasks

https://openreview.net/forum?id=QBlegfNZNE

irfft(rfft(ones(4), n=4*2) * rfft(arange(1,5), n=4*2))[:4]
= cumsum(arange(1,5),-1)

Raining tokens 2 https://x.com/statmlpapers/status/1717353357772300342

Scalable instance-level meta-learning by looking at every token having its own reconstruction task

https://arxiv.org/abs/2310.13807

https://twitter.com/srush_nlp/status/1720113524121235577

Fine grained control of KV caching is one of the reasons I work on haloop.

Install today: https://www.youtube.com/watch?v=2-G5bomAkfs