So the gaussian window around every token for speech paper does not mention how to compute the center of the gaussian.

I found an even older paper (Alex Graves 2013) for a different task (handwriting synthesis) that suggests to learn the centers and enforce their monotonicity. Which I am about to do!

The simple trick worked! It fixed the drift problem with long sentences. In the mean time I started using my third block implementation for the encoder and found this bug. This bug let the decoder (with cross attention) learn just fine!

New paper by Alex Graves et al, Bayesian diffusion for language modeling.

https://arxiv.org/abs/2308.07037

The uglier the code, the faster it runs. This KV caching optimization (no tensor reallocation, alive used to shut off completed parts of the batch) allows me to decode train-clean-100 in 5 minutes. I started at 16 hours.

Rupesh Srivastava with a thread https://x.com/rupspace/status/1691584987148218841?

My paper on reproducing approximation errors in VAEs has been published in ReScience C. We see how choosing a factorized latent distributions causes perfectly good decoders to run their reconstruction quality adapting to the encoder.

https://x.com/darkproger/status/1691755047145673029

https://drive.google.com/file/d/15TmDQ8lONE66_hc_me7ML1Xv6a4pup59/edit

Tutorial 2 at Interspeech:

Recent Advances in Speech Frontend, Diarization and ASR for Multi-talker Cocktail Party Problem

Austin Zhang / Yong Xu / Dong Yu / Shinji Watanabe

Teaser: maximizing data selection with higher error rates than random using expected gradient length and monte carlo dropout.

This is a prototype I'm developing to clean uk1e2

Surprisingly, low resolution (10fps) discrete features work for synthesis

https://arxiv.org/abs/2306.01084

На інтерспічі я бачив декілька постерів які презентували *останні* автори. Постер по 100fps hubert презентував Watanabe, постер по Neural HMM презентував Gustav Eje Henter. Це були одні з найкращих презентацій постерів, максимально детальні та повні несподіваних інсайтів.

З цього я вивів таке правило: якщо ваш керівник не може презентувати вашу роботу краще за вас, вам не потрібен цей керівник.

On the difference between supervised and reinforcement learning.


Most supervised learning reduces to performing a softmax over evidence (aka "hidden activations") and then maximizing likelihood (minimizing cross entropy). When backpropagating, softmax ensures you get a very high error signal for the wrong answer towards the right answer (or multiple answers if you don't have a spiky distribution, imagine cases like label smoothing or distribution matching).


The softmax+cross entropy let your learning process consider *all outcomes at once*. This is better visible when you compute gradients of likelihood and softmax. Something you don’t get to see if you use autodiff all the time.

This is not the case with reinforcement learning. Consider policy gradient: you sample an action, record its likelihood and then when you get the reward you scale the likelihood of a *single action* by that reward. You get unbiased and high variance learning signal estimates because you don't have updates for all outputs in a softmax! You don’t have it in reinforcement learning because you are in the decision *process* framework: there is no way to evaluate all alternative actions and choose the best, you only get to evaluate one.


The morale is: use supervised learning when you can.