Bayesian Flow Networks official code has been released. https://github.com/nnaisense/bayesian-flow-networks

How to count experience? This post introduces a counter for embeddings: https://proger.github.io/posts/counter/counter.html

I made a post on neural circuits, in the context of the gradient vanishing problem, linearity and autoregression:

https://proger.github.io/posts/neural-circuits/recurrent.html

Deep Reinforcement Learning has been used to develop champion-level FPV drone racers, with a paper in Nature appearing 10 years after DQN was developed to beat humans in Atari. Key success lies in development of high quality simulations and sim2real policy transfer. RL is no longer about games.

One of the authors, Vladlen Koltun was giving a seminar in MIT: https://www.youtube.com/watch?v=vNFTcD3QMn0

Here’s the system demonstration of Swift https://www.youtube.com/watch?v=fBiataDpGIo

Meme moment, my pure torch linearized LSTM implementation just beat GPT in throughput in the small (110M) setting. This is the fastest LSTM implementation to date. More experiments soon.

zero-shot learning by searching for support vectors for generative tasks

https://openreview.net/forum?id=QBlegfNZNE

irfft(rfft(ones(4), n=4*2) * rfft(arange(1,5), n=4*2))[:4]
= cumsum(arange(1,5),-1)

Raining tokens 2 https://x.com/statmlpapers/status/1717353357772300342

Scalable instance-level meta-learning by looking at every token having its own reconstruction task

https://arxiv.org/abs/2310.13807

https://twitter.com/srush_nlp/status/1720113524121235577

Fine grained control of KV caching is one of the reasons I work on haloop.

Install today: https://www.youtube.com/watch?v=2-G5bomAkfs

Prof reminds us to add gradient noise and decay learning rate. Probabilistic AI 2023 is deep learning lab in math speak

Don’t regulate AI, regulate uses of AI

https://laion.ai/blog/transparent-ai/

torch.export can unroll for loops that depend on the time dimension of the input. Sequential RNNs go brrrr!

New work from IDSIA. Transformer's largest bottleneck is a feedforward MLP block that expands hidden dimension four times and shrinks it back.

Instead of running all parts of this network on every request, sparse gating decides what subnetwork to run depending on the input. This is straightforward at inference time but hard to backpropagate at training time.

Sparsely-gated Mixtures of Experts date back to Ivakhnenko and Lapa (see Section 4) , and now feature an open source implementation from Robert.

https://arxiv.org/abs/2310.10837

Btw does anybody know how to use Triton kernels with torch.compile without graph breaks?

ICLR 2024 has a track that accepts anonymized blog posts. Submissions are now open:

https://x.com/fpedregosa/status/1722197379418665247?

It’s time to build backprop over HTTP:

https://x.com/wightmanr/status/1724841649233350659

Alec Radford on language models. He gave this lecture right after he completed his work on GPT-2.

https://www.youtube.com/watch?v=BnpB3GrpsfM

Meta learning has its own *class* at Stanford

https://x.com/_jasonwei/status/1729585618311950445?

https://nips.cc/virtual/2023/events/journal_track_2023


My VAE paper is on NeurIPS Journal Track. See you in New Orleans!