Neural Phonetic Alignment with pretrained models for English:
https://github.com/lingjzhu/charsiu/

Lightweight speech encoder

https://github.com/yl4579/AuxiliaryASR

StyleGAN3 antialiasing generator meets vocoder. Trained on all of LibriTTS. Generalizes to laughter and music.

https://arxiv.org/abs/2206.04658

https://github.com/NVIDIA/BigVGAN

https://bigvgan-demo.github.io

Try StarGAN-VC and ACVAE-VC to speak like a dog. ACVAE sounds more like a dog while StarGAN has better speech clarity.

https://arxiv.org/abs/2206.04780

https://github.com/suzuki256/dog-dataset

ACL 2022: Direct speech-to-speech translation with discrete units, Lee at al

https://ai.facebook.com/blog/advancing-direct-speech-to-speech-modeling-with-discrete-units/
Meta does speech translation by feeding discrete units from a transformer encoder-decoder block to a vocoder. I noted how they don’t use pitch information as a HiFi-GAN input and use a mini duration prediction block from FastSpeech 2.

https://twitter.com/ysaito_human/status/1536521048568438785

日本語を学びましょう

Very neat TTS composer from Sonatic https://www.youtube.com/watch?v=fNtwg-lXie8

Turns out the 80fps vs 200fps frame rate issue was addressed in the original Tacotron 2 paper.

https://arxiv.org/abs/1712.05884

LJSpeech is a noisy dataset! Compare a single utterance from LJ and HiFi-TTS speaker 92_clean

BigVGAN implementation by sh-lee-prml https://github.com/sh-lee-prml/BigVGAN

The sample above was actually preprocessed (at least downsampled to 16khz), here’s the original one, the noise at the silence interval is audible

Thanks Taras for sharing the Deep Creativity course, I’ve been stuck watching a lecture on music synthesis.

The topic of using proper strong inductive biases to achieve realistic output seems to be much more explored there: it seems like vocoder community has just started using DWT (FreGAN), PQMF (Avocodo, RAVE) and antialiasing and quality for parameter efficiency while DDSP has a much larger pool of building blocks for upsampling.

https://youtu.be/oiPWOTr44qQ

https://github.com/TariqAHassan/HiFiHybrid

Anti-aliased multi-periodicity composition backported to HiFi-GAN

The Snake1d activation replaces more common Leaky ReLU (snake a x = x + sin^2(ax)/a where a is trainable) to arbitrarily change the frequency of the input. It is anti-aliased by applying low pass filtering (blurring) after upsampling and downsampling operations.

However trying to replace transposed convoluions with their antialiased counterparts causes mode collapse in the BigVGAN. Maybe it won’t in Diff? :)

A map of vocoders

Inside one of the slide decks of NSF https://nii-yamagishilab.github.io/samples-nsf/index.html

https://www.youtube.com/watch?v=Q3gNj7XlArs

Daniel Povey describes how to get Conformer to converge faster, watch after 16th minute if you don’t care about intro to K2 and RNN-T. This is hands down the most down to earth hacker talk on neural nets I’ve seen in a long time.

Key takeaways:

- [fill me in when you stop rewatching]

kaldi nnet3 suprised me with some pretty detailed training diagnostics and sophisticated tricks early on, like weight averaging, LR scheduling, per-layer gradient statistics, etc. Icefall (recipes for k2) now has a similar diagnostics toolkit ported over using PyTorch hooks, see

https://github.com/k2-fsa/icefall/blob/c0101185d7be5e353db01dad326d530faa4ea718/icefall/diagnostics.py

The talk above describes what you can achieve if you get your diagnostics together