MMS: Massively Multilingual Speech.
- Can do speech2text and text speech in 1100 languages.
- Can recognize 4000 spoken languages.
- Code and models available under the CC-BY-NC 4.0 license.
- half the word error rate of Whisper.

Code+Models: https://github.com/facebookresearch/fairseq/tree/main/examples/mms
Paper: https://scontent-lga3-2.xx.fbcdn.net/v/t39.8562-6/348836647_265923086001014_6878005808275791319_n.pdf
Blog: https://ai.facebook.com/blog/multilingual-model-speech-recognition/

In case you want to play with laughter

https://twitter.com/forthshinji/status/1660990946606219266

CALLS: Japanese Empathetic Dialogue Speech Corpus of Complaint Handling and Attentive Listening in Customer Center
Yuki Saito, Eiji Iimori, Shinnosuke Takamichi, Kentaro Tachibana, Hiroshi Saruwatari
We present CALLS, a Japanese speech corpus that considers phone calls in a customer center as a new domain of empathetic spoken dialogue. The existing STUDIES corpus covers only empathetic dialogue between a teacher and student in a school. To extend the application range of empathetic dialogue speech synthesis (EDSS), we designed our corpus to include the same female speaker as the STUDIES teacher, acting as an operator in simulated phone calls. We describe a corpus construction methodology and analyze the recorded speech. We also conduct EDSS experiments using the CALLS and STUDIES corpora to investigate the effect of domain differences. The results show that mixing the two corpora during training causes biased improvements in the quality of synthetic speech due to the different degrees of expressiveness. Our project page of the corpus is this http URL.

https://arxiv.org/abs/2305.13713

https://sython.org/Corpus/STUDIES-2/

Announcing the VoiceMOS Challenge 2023!
Challenge website: https://voicemos-challenge-2023.github.io
Register to participate: https://forms.gle/kcLc69Wa4Q97rSNq7

This edition of the challenge will focus on real-world and challenging zero-shot out-of-domain mean opinion score prediction!
https://twitter.com/yamagishilab/status/1643788523886235648

https://pages.cs.huji.ac.il/adiyoss-lab/twist/

Textually Pretrained Speech Language Models

https://arxiv.org/pdf/2305.13009.pdf

Speech language models (SpeechLMs) process and generate acoustic data only, without textual supervision. In this work, we propose TWIST, a method for training SpeechLMs using a warm-start from a pretrained textual language model. We show using both automatic and human evaluation that TWIST outperforms a cold-start SpeechLM across the board. We empirically analyze the effect of different model design choices such as the speech tokenizer, the pretrained textual model, and the dataset size. We find that model and dataset scale both play an important role in constructing better-performing SpeechLMs. Based on our observation, we present the largest (to the best of our knowledge) SpeechLM both in terms of number of parameters and training data. We additionally introduce two spoken versions of the StoryCloze textual benchmark to further improve model evaluation and advance future research in the field.

Another audio LM from Google Research

LMs with a Voice: Spoken Language Modeling beyond Speech Tokens

- Presents Spectron, a novel approach to adapting pre-trained LMs to perform speech continuation.

- Surpasses existing spoken LMs both in semantic content and speaker preservation

proj: https://michelleramanovich.github.io/spectron/spectron/
abs: https://arxiv.org/abs/2305.15255

VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation

Is one decoder-only generative model all you need for speech recognition, synthesis, and translation?

https://arxiv.org/abs/2305.16107

https://github.com/Takaaki-Saeki/zm-text-tts

https://arxiv.org/abs/2301.12596

While neural text-to-speech (TTS) has achieved human-like natural synthetic speech, multilingual TTS systems are limited to resource-rich languages due to the need for paired text and studio-quality audio data. This paper proposes a method for zero-shot multilingual TTS using text-only data for the target language. The use of text-only data allows the development of TTS systems for low-resource languages for which only textual resources are available, making TTS accessible to thousands of languages. Inspired by the strong cross-lingual transferability of multilingual language models, our framework first performs masked language model pretraining with multilingual text-only data. Then we train this model with a paired data in a supervised manner, while freezing a language-aware embedding layer. This allows inference even for languages not included in the paired data but present in the text-only data. Evaluation results demonstrate highly intelligible zero-shot TTS with a character error rate of less than 12% for an unseen language. All experiments were conducted using public datasets and the implementation will be made available for reproducibility.

New paper from @GoogleResearch & @GoogleDeepMind

Translatotron 3: Unsupervised Speech-to-Speech Translation

Paper: https://arxiv.org/abs/2305.17547
Audio Samples: https://google-research.github.io/lingvo-lab/translatotron3

Text-to-speech synthesis from dark data with evaluation-in-the-loop data selection

Kentaro Seki, Shinnosuke Takamichi, Takaaki Saeki, Hiroshi Saruwatari
This paper proposes a method for selecting training data for text-to-speech (TTS) synthesis from dark data. TTS models are typically trained on high-quality speech corpora that cost much time and money for data collection, which makes it very challenging to increase speaker variation. In contrast, there is a large amount of data whose availability is unknown (a.k.a, "dark data"), such as YouTube videos. To utilize data other than TTS corpora, previous studies have selected speech data from the corpora on the basis of acoustic quality. However, considering that TTS models robust to data noise have been proposed, we should select data on the basis of its importance as training data to the given TTS model, not the quality of speech itself. Our method with a loop of training and evaluation selects training data on the basis of the automatically predicted quality of synthetic speech of a given TTS model. Results of evaluations using YouTube data reveal that our method outperforms the conventional acoustic-quality-based method.

https://arxiv.org/abs/2210.14850

New studio-quality & large-scale speech dataset🎙️
LibriTTS-R is a sound quality improved LibriTTS.

Dataset is freely available: http://openslr.org/141/
Speech samples and TTS outputs in our demo page: https://google.github.io/df-conformer/librittsr/index.html
Paper: https://arxiv.org/abs/2305.18802

Spoken dataset of books read in French, initially collected from audiocite.net by the GETALP team for the LeBenchmark project.

http://openslr.org/139/

Audiocite.net is a corpus of read French speech downloaded in November 2021 from the Audiocite.net website.

With a total duration of 6682 hours of audio recording, this corpus is the result of the voluntary work of 130 speakers. The metadata is divided into 4 .jsons files (all(100%), train(80%), dev(10%) and test(10%)) to be used in NLP models.

The corpus and its metadata were uploaded through a script distributing the information in a .csv file. The use of these audio and metadata files is intended for pre-trained speech models.

ICASSP begins today

https://twitter.com/ieeeICASSP/status/1665942845147029506

Reenforcement learning in speech from Google

Edit Distance based RL for RNNT decoding
https://arxiv.org/abs/2306.01789

Dongseong Hwang, Changwan Ryu, Khe Chai Sim

RNN-T is currently considered the industry standard in ASR due to its exceptional WERs in various benchmark tests and its ability to support seamless streaming and longform transcription. However, its biggest drawback lies in the significant discrepancy between its training and inference objectives. During training, RNN-T maximizes all alignment probabilities by teacher forcing, while during inference, it uses beam search which may not necessarily find the maximum probable alignment. Additionally, RNN-T's inability to experience mistakes during teacher forcing training makes it more problematic when a mistake occurs in inference. To address this issue, this paper proposes a Reinforcement Learning method that minimizes the gap between training and inference time. Our Edit Distance based RL (EDRL) approach computes rewards based on the edit distance, and trains the network at every action level. The proposed approach yielded SoTA WERs on LibriSpeech for the 600M Conformer RNN-T model.

Nice paper on Whisper adaptation to word lists

Code: https://github.com/BriansIDP/WhisperBiasing

https://arxiv.org/abs/2306.01942

Can Contextual Biasing Remain Effective with Whisper and GPT-2?
Guangzhi Sun, Xianrui Zheng, Chao Zhang, Philip C. Woodland
End-to-end automatic speech recognition (ASR) and large language models, such as Whisper and GPT-2, have recently been scaled to use vast amounts of training data. Despite the large amount of training data, infrequent content words that occur in a particular task may still exhibit poor ASR performance, with contextual biasing a possible remedy. This paper investigates the effectiveness of neural contextual biasing for Whisper combined with GPT-2. Specifically, this paper proposes integrating an adapted tree-constrained pointer generator (TCPGen) component for Whisper and a dedicated training scheme to dynamically adjust the final output without modifying any Whisper model parameters. Experiments across three datasets show a considerable reduction in errors on biasing words with a biasing list of 1000 words. Contextual biasing was more effective when applied to domain-specific data and can boost the performance of Whisper and GPT-2 without losing their generality.

Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understanding

paper page: https://huggingface.co/papers/2306.07944

Large Language Models (LLMs) have been applied in the speech domain, often incurring a performance drop due to misaligned between speech and language representations. To bridge this gap, we propose a joint speech and language model (SLM) using a Speech2Text adapter, which maps speech into text token embedding space without speech information loss. Additionally, using a CTC-based blank-filtering, we can reduce the speech sequence length to that of text. In speech MultiWoz dataset (DSTC11 challenge), SLM largely improves the dialog state tracking (DST) performance (24.7% to 28.4% accuracy). Further to address errors on rare entities, we augment SLM with a Speech2Entity retriever, which uses speech to retrieve relevant entities, and then adds them to the original SLM input as a prefix. With this retrieval-augmented SLM (ReSLM), the DST performance jumps to 34.6% accuracy. Moreover, augmenting the ASR task with the dialog understanding task improves the ASR performance from 9.4% to 8.5% WER.

https://github.com/gweltou/vosk-br

Implemented nice Breton model for Vosk. Very valuable contribution! Please don't hesitate to add a star to that project!

https://arxiv.org/abs/2306.07691

https://styletts2.github.io/

StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models
Yinghao Aaron Li, Cong Han, Vinay S. Raghavan, Gavin Mischler, Nima Mesgarani
In this paper, we present StyleTTS 2, a text-to-speech (TTS) model that leverages style diffusion and adversarial training with large speech language models (SLMs) to achieve human-level TTS synthesis. StyleTTS 2 differs from its predecessor by modeling styles as a latent random variable through diffusion models to generate the most suitable style for the text without requiring reference speech, achieving efficient latent diffusion while benefiting from the diverse speech synthesis offered by diffusion models. Furthermore, we employ large pre-trained SLMs, such as WavLM, as discriminators with our novel differentiable duration modeling for end-to-end training, resulting in improved speech naturalness. StyleTTS 2 surpasses human recordings on the single-speaker LJSpeech dataset and matches it on the multispeaker VCTK dataset as judged by native English speakers. Moreover, when trained on the LibriTTS dataset, our model outperforms previous publicly available models for zero-shot speaker adaptation. This work achieves the first human-level TTS on both single and multispeaker datasets, showcasing the potential of style diffusion and adversarial training with large SLMs. The audio demos and source code are available at this https URL.

Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale

https://ai.facebook.com/blog/voicebox-generative-ai-model-speech/

A device to track human activity from Meta/Facebook

https://ariatutorial2023.github.io/