Thanks Taras for sharing the Deep Creativity course, I’ve been stuck watching a lecture on music synthesis.

The topic of using proper strong inductive biases to achieve realistic output seems to be much more explored there: it seems like vocoder community has just started using DWT (FreGAN), PQMF (Avocodo, RAVE) and antialiasing and quality for parameter efficiency while DDSP has a much larger pool of building blocks for upsampling.

https://youtu.be/oiPWOTr44qQ

https://github.com/TariqAHassan/HiFiHybrid

Anti-aliased multi-periodicity composition backported to HiFi-GAN

The Snake1d activation replaces more common Leaky ReLU (snake a x = x + sin^2(ax)/a where a is trainable) to arbitrarily change the frequency of the input. It is anti-aliased by applying low pass filtering (blurring) after upsampling and downsampling operations.

However trying to replace transposed convoluions with their antialiased counterparts causes mode collapse in the BigVGAN. Maybe it won’t in Diff? :)

A map of vocoders

Inside one of the slide decks of NSF https://nii-yamagishilab.github.io/samples-nsf/index.html

https://www.youtube.com/watch?v=Q3gNj7XlArs

Daniel Povey describes how to get Conformer to converge faster, watch after 16th minute if you don’t care about intro to K2 and RNN-T. This is hands down the most down to earth hacker talk on neural nets I’ve seen in a long time.

Key takeaways:

- [fill me in when you stop rewatching]

kaldi nnet3 suprised me with some pretty detailed training diagnostics and sophisticated tricks early on, like weight averaging, LR scheduling, per-layer gradient statistics, etc. Icefall (recipes for k2) now has a similar diagnostics toolkit ported over using PyTorch hooks, see

https://github.com/k2-fsa/icefall/blob/c0101185d7be5e353db01dad326d530faa4ea718/icefall/diagnostics.py

The talk above describes what you can achieve if you get your diagnostics together

There is a blog post now https://medium.com/@nadirapovey/next-gen-kaldi-reworked-conformer-model-8a3828f364af

G2P for 8000 languages: https://github.com/xinjli/transphone

Happy Interspeech 2022 Tutorial Day!

First up, Learning from Weak Labels by Bhiksha Raj, Anurag Kumar, Ankit Shah

Slides and Code (MATLAB) in https://github.com/cmu-mlsp/Interspeech-Tutorial-2022-Learning_from_weak_labels

An afternoon tutorial is a by Xu Tan and Hung-yi Lee on Neural Speech Synthesis

Xu Tan has a massive survey paper on a Speech Synthesis, check out tts-tutorial on GitHub for more.

Hung-yi Lee has a very broad Machine Learning lectures channel in Mandarin with slides in English. A lot of lectures are about speech. https://www.youtube.com/c/HungyiLeeNTU

As an upgrade from this year’s ICASSP, it features a whole part devoted to Voice Conversion.

https://github.com/tts-tutorial/interspeech2022

Self-supervised Representation Learning for Speech Processing by Hung-yi Lee, Abdelrahman Mohamed, Shinji Watanabe, Tara Sainath, Karen Livescu, Shang-Wen Li, Shu-wen Yang, Katrin Kirchhoff

This tutorial looks like an upgrade from a preceding one at NAACL 2022. There is an abstract paper inside (13 pages) and a link to s3prl on GitHub.

https://aclanthology.org/2022.naacl-tutorials.2/

SSL tutorial slides have been published:

https://docs.google.com/presentation/d/12W-L8EPQ3SfCPmgNcSm4B-_Psksqmvw_O6YD8CULelY/edit?resourcekey=0-ZBTF2RG_dHWNqq9q_w-eeg

source: https://twitter.com/HungyiLee2/status/1571511025475948549

PyTorch profiling tutorial k2 edition

https://github.com/k2-fsa/icefall/issues/610

That model warmup trick is also known as ReZero when you gradually change alpha from 0 to 1

https://proceedings.mlr.press/v161/bachlechner21a/bachlechner21a.pdf

https://twitter.com/92hschoi/status/1593467656379990016

Disentangled self-supervised representations for timbre, language and pitch using contrastive and reconstruction objectives.

DEMAND noise is used for linguistic contrast, CQT cropping is used for pitch. No extra augmentation seems to be used for timbre. Global speaker encoder tokens are turned into time-dependent using cross-attention in the synthesizer.

10k hours of pretraining data on 10x 3090. Beats supervised YourTTS on some tasks.

Rotate the phase of each frequency bin to simulate one-to-many mapping. Useful augmentation for GAN vocoder training. Differentiable.

https://mindslab-ai.github.io/phaseaug/

On augmentations for voice conversion:

> On the other hand, the use of more carefully designed augmentation techniques (such as timbre transformation with VoTrans and modifying prosody with NoisyF0) may be helpful in achieving more realistic target speaker identity and better audio quality

https://arxiv.org/abs/2212.13581

New scaling laws paper for combining different modalities (e.g. speech and text) using Chinchilla style notation.

https://twitter.com/ArmenAgha/status/1613192899646324736