zero-shot learning by searching for support vectors for generative tasks
https://openreview.net/forum?id=QBlegfNZNE
https://openreview.net/forum?id=QBlegfNZNE
OpenReview
Language as Kernels
In the realm of natural language understanding, the synergy between large language models (LLMs) and prompt engineering has unfurled an impressive tapestry of performance. Nonetheless, this prowess...
Vol Building AGI
Meme moment, my pure torch linearized LSTM implementation just beat GPT in throughput in the small (110M) setting. This is the fastest LSTM implementation to date. More experiments soon.
irfft(rfft(ones(4), n=4*2) * rfft(arange(1,5), n=4*2))[:4]
= cumsum(arange(1,5),-1)
= cumsum(arange(1,5),-1)
Scalable instance-level meta-learning by looking at every token having its own reconstruction task
https://arxiv.org/abs/2310.13807
https://arxiv.org/abs/2310.13807
https://twitter.com/srush_nlp/status/1720113524121235577
Fine grained control of KV caching is one of the reasons I work on haloop.
Install today: https://www.youtube.com/watch?v=2-G5bomAkfs
Fine grained control of KV caching is one of the reasons I work on haloop.
Install today: https://www.youtube.com/watch?v=2-G5bomAkfs
X (formerly Twitter)
Sasha Rush (@srush_nlp) on X
I got excited about a bunch of fast LLM generators (vLLM, MLC, etc) but none of them implement prefix caching / kv storage? This seems like a benchmark failure where everyone optimized Tok/Sec, and use cases all have massive prompts. Find myself back using…
New work from IDSIA. Transformer's largest bottleneck is a feedforward MLP block that expands hidden dimension four times and shrinks it back.
Instead of running all parts of this network on every request, sparse gating decides what subnetwork to run depending on the input. This is straightforward at inference time but hard to backpropagate at training time.
Sparsely-gated Mixtures of Experts date back to Ivakhnenko and Lapa (see Section 4) , and now feature an open source implementation from Robert.
https://arxiv.org/abs/2310.10837
Instead of running all parts of this network on every request, sparse gating decides what subnetwork to run depending on the input. This is straightforward at inference time but hard to backpropagate at training time.
Sparsely-gated Mixtures of Experts date back to Ivakhnenko and Lapa (see Section 4) , and now feature an open source implementation from Robert.
https://arxiv.org/abs/2310.10837
🔥1
Btw does anybody know how to use Triton kernels with torch.compile without graph breaks?
ICLR 2024 has a track that accepts anonymized blog posts. Submissions are now open:
https://x.com/fpedregosa/status/1722197379418665247?
https://x.com/fpedregosa/status/1722197379418665247?
X (formerly Twitter)
Fabian Pedregosa on X
📢 Attention machine learning enthusiasts!📢
The #ICLR2024 blog post track is now accepting submissions!. This is a great opportunity to share your insights on the latest #MachineLearning research and present it at one of the main ML conferences.
https:/…
The #ICLR2024 blog post track is now accepting submissions!. This is a great opportunity to share your insights on the latest #MachineLearning research and present it at one of the main ML conferences.
https:/…
Alec Radford on language models. He gave this lecture right after he completed his work on GPT-2.
https://www.youtube.com/watch?v=BnpB3GrpsfM
https://www.youtube.com/watch?v=BnpB3GrpsfM
YouTube
L11 Language Models -- guest instructor: Alec Radford (OpenAI) --- Deep Unsupervised Learning SP20
Course homepage:
https://sites.google.com/view/berkeley-cs294-158-sp20/home
Lecture Instructor: Alec Radford (OpenAI)
Course Instructors: Pieter Abbeel, Aravind Srinivas, Peter Chen, Jonathan Ho, Alex Li, Wilson Yan
CS294-158-SP20: Deep Unsupervised Learning…
https://sites.google.com/view/berkeley-cs294-158-sp20/home
Lecture Instructor: Alec Radford (OpenAI)
Course Instructors: Pieter Abbeel, Aravind Srinivas, Peter Chen, Jonathan Ho, Alex Li, Wilson Yan
CS294-158-SP20: Deep Unsupervised Learning…
https://nips.cc/virtual/2023/events/journal_track_2023
My VAE paper is on NeurIPS Journal Track. See you in New Orleans!
My VAE paper is on NeurIPS Journal Track. See you in New Orleans!
👍2🔥1
RNNs are officially back. This paper is such a good read and the experiments are actually serious, we've been doing 350M models so far with our fast LSTM on 6B token runs.
Recipe for success: use VERY large hidden state, do not materialize activations early, use CUTLASS, do final runs on 300B tokens.
https://arxiv.org/abs/2312.00752
Recipe for success: use VERY large hidden state, do not materialize activations early, use CUTLASS, do final runs on 300B tokens.
https://arxiv.org/abs/2312.00752
Ha, selective scan is not using CUTLASS, it's written using cub! https://github.com/state-spaces/mamba/blob/main/csrc/selective_scan/selective_scan_fwd_kernel.cuh
GitHub
mamba/csrc/selective_scan/selective_scan_fwd_kernel.cuh at main · state-spaces/mamba
Mamba SSM architecture. Contribute to state-spaces/mamba development by creating an account on GitHub.
But what about BERT? Monarch matrices have you covered https://www.youtube.com/live/IS59IwGLvVs?si=3yvBYGsOSx3jU2tE
YouTube
Monarch Mixer: Making Foundation Models More Efficient - Dan Fu | Stanford MLSys #86
Episode 86 of the Stanford MLSys Seminar Series!
Monarch Mixer: Making Foundation Models More Efficient
Speaker: Dan Fu
Abstract:
Machine learning models are increasingly being scaled in both sequence length and model dimension to reach longer contexts…
Monarch Mixer: Making Foundation Models More Efficient
Speaker: Dan Fu
Abstract:
Machine learning models are increasingly being scaled in both sequence length and model dimension to reach longer contexts…