Speech Technology
1.72K subscribers
140 photos
4 videos
1 file
2.24K links
Download Telegram
https://github.com/anyreachai/dualturn

DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining
Shangeth Rajaa
Speech-to-speech models handle turn-taking naturally but offer limited support for tool-calling or complex reasoning, while production ASR-LLM-TTS voice pipelines offer these capabilities but rely on silence timeouts, which lead to unnatural turn-taking. We present DualTurn, which narrows this gap through generative pretraining on dual-channel conversational audio. The model generates both speakers' future audio autoregressively, implicitly learning conversational dynamics without any labels, and is then fine-tuned to predict interpretable turn-taking signals that map directly to agent actions. DualTurn monitors both channels continuously, anticipating turn boundaries and producing five agent actions. On standard benchmarks, DualTurn (0.5B) outperforms both VAP on agent action prediction (wF1 0.633 vs. 0.389) and a 3.1B audio-text model on word-level turn prediction (AUC 0.930 vs. 0.880), while anticipating turn boundaries earlier with fewer interruptions.
When full to end doesn't really work this thing has potential

https://arxiv.org/abs/2608.13831

VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents
Edresson Casanova, Jaehyeon Kim, Mariana Graterol Fuenmayor, Shehzeen Hussain, Viacheslav Klimkov, Valentin Mendelev, Mikyas Desta, Paarth Neekhara, Piotr Zelasko, Chen Chen, Elena Rastorgueva, Ke Hu, Ankita Pasad, Xuesong Yang, Aya Alja'fari, Rajarshi Roy, Rohan Badlani, Jason Roche, Jason Li, Zhehuai Chen
Spoken dialogue is a natural form of human--computer interaction, yet most speech language models remain limited to turn-based operation and lack real-time adaptability, such as user barge-in. Recent duplex speech-to-speech and speech-to-text models reduce latency by replacing multi-stage pipelines, but often compromise speech quality because accurate ASR, interruption handling, and high-fidelity synthesis must be optimized jointly. We propose VoiceChat-TTS, a low-latency, continuous, and streamable text-to-speech model for interactive agents. VoiceChat-TTS is driven directly by LLM text-token streams, supports explicit interruption via control tokens, and produces silence when no textual input is available. The model enables always-on, responsive speech generation while preserving modularity and high speech quality, and it supports mid-utterance interruptions without resetting the KV cache.
Alex Smola leading a great research at Boson.AI (Higgs authors). Couple of recent selected papers:

https://arxiv.org/search/cs?searchtype=author&query=Smola,+A

https://arxiv.org/abs/2603.25727

Back to Basics: Revisiting ASR in the Age of Voice Agents
Geeyang Tay, Wentao Ma, Jaewon Lee, Yuzhi Tang, Daniel Lee, Weisu Yin, Dongming Shen, Silin Meng, Yi Zhu, Mu Li, Alex Smola
Automatic speech recognition (ASR) systems have achieved near-human accuracy on curated benchmarks, yet still fail in real-world voice agents under conditions that current evaluations do not systematically cover. Without diagnostic tools that isolate specific failure factors, practitioners cannot anticipate which conditions, in which languages, will cause what degree of degradation. We introduce WildASR, a multilingual (four-language) diagnostic benchmark sourced entirely from real human speech that factorizes ASR robustness along three axes: environmental degradation, demographic shift, and linguistic diversity. Evaluating seven widely used ASR systems, we find severe and uneven performance degradation, and model robustness does not transfer across languages or conditions. Critically, models often hallucinate plausible but unspoken content under partial or degraded inputs, creating concrete safety risks for downstream agent behavior. Our results demonstrate that targeted, factor-isolated evaluation is essential for understanding and improving ASR reliability in production systems. Besides the benchmark itself, we also present three analytical tools that practitioners can use to guide deployment decisions.


https://arxiv.org/abs/2607.20460

Instruct-FD: Can Your Full-Duplex Speech System Follow Turn-Taking Instructions?
Yuzhi Tang, Wentao Ma, Xiling Zhao, Ahmad Salimi, Sepehr Harfi Moridani, Dongming Shen, Jixuan Wang, Abdulrahman Abdulrazzag, Murdock Aubry, Yu-Hua Chen, Daniel Lee, Jaewon Lee, Jonah Mackey, Silin Meng, Nicholas Stranges, Chenxu Xiong, Hao Yu, Yi Zhu, Mu Li, Alex Smola
Current full-duplex (FD) spoken dialogue systems can produce fluid interactions, yet it remains unclear whether they can adapt their turn-taking behavior when explicitly instructed. This is critical for real-world deployment, where conversational policies vary across applications (e.g., proactive tutoring vs. passive counseling). We introduce Instruct-FD, an instruction-conditioned benchmark for evaluating controllable turn management in FD systems. To enable this, we develop a human-validated, scalable synthetic pipeline that generates instruction-conditioned conversations, along with a deployment-agnostic multi-turn evaluation protocol and an LLM-based judge. Benchmarking six state-of-the-art full-duplex systems reveals a substantial gap in instruction-following turn management: the best model achieves only 64.4% adherence. Performance is highly uneven across behaviors and scenarios, with proactive behaviors such as model backchanneling and interruption remaining particularly challenging. These findings establish instruction-following turn management as a crucial direction for building adaptable and deployable full-duplex dialogue systems.
Audio in the wild is still very complex

https://aslp-lab.github.io/SmartGlasses/

https://x.com/MosiAI_Official/status/2090775763666812997

MOSS takes 1st place across all four tasks in the IEEE SLT 2026 SmartGlasses Challenge. The challenge tests whether AI can truly hear and understand real conversations through smart glasses — handling noise, overlapping speech, multiple speakers, and long-context interactions. Among 77 participating teams, MOSS achieved 1st place across both tracks and all four tasks, covering speech recognition and spoken language understanding in real-world smart glasses scenarios.

Results: Two-person TSA-ASR — 5.23% tcpCER

Two-person SLU — 88.8% Accuracy

Multi-party TSA-ASR — 27.95% tcpCER

Multi-party SLU — 93.0% Accuracy

The models behind these results: • MOSS-Transcribe-Diarize for TSA-ASR • MOSS-Audio for SLU
You can download really huge datasets these days

https://x.com/ahochlehnert/status/2092648676829413778

LAION-BVD: a 10-million-hour open video dataset for multimodal pre-training. - 1.3B video URLs from CommonCrawl - 80M downloaded videos - 10M video hours - 55M captioned clips - 300M frame-caption pairs

This repository contains 1.7 million audio clips taken from BVD-V-55M and sampled for uniqueness of the source video, so that the subset maximises source diversity rather than clip count. Each clip comes with a caption, its language, and the timestamps locating it in the source video.

https://huggingface.co/datasets/laion/BVD-A-1.7M
https://turnbench.sesame.com/

A multi-domain benchmark for evaluating conversational turn-taking. We hand-annotate end-of-turn and interruption events in dual-channel human conversations, and measure how accurately and how quickly models detect them.
We know human scores are useless but anyway

https://x.com/datapointai/status/2094829412625654141

today, we're releasing the largest open-source human audio preferences dataset, focused on the customer support use-case

- 300K+ annotations by real people
- 15 SOTA TTS models ranked (Sonic 3.6, Grok TTS, Simba 3.2, Eleven Labs v3)
- 8 categories (IVR menus, empathy, escalations, refunds etc)

dataset + benchmark + frontier plot below:
Quite an obvious but so ignored by industry before. There is certainly no need to clone from 3 seconds

You can now create an instant voice clone from up to 2 minutes of reference audio, compared with 20 seconds before.

The longer sample gives the model much more information about the voice, including phonemes, intonation, pauses, rhythm, pacing, emphasis, emotion, and speaking style.

The result:
• Higher voice similarity
• Greater consistency, especially across longer sentences
• Better pronunciation coverage
• More natural prosody
• Better preservation of speaking style

https://www.linkedin.com/posts/soniox_soniox-texttospeech-voiceai-activity-7501594413660401664-p9Cq
https://arxiv.org/abs/2609.01246v1

Ready to Speak: Aligning LLMs for TTS-Friendly Text Generation

Thibaut Thonet, Jos Rozen, Laurent Besacier

Current Large Language Models (LLMs) are primarily optimized for written text, often producing outputs that are grammatically correct and helpful yet poorly suited for spoken delivery via Text-to-Speech (TTS). In this work, we study how to make LLMs natively generate TTS-friendly text, which we frame as a preference alignment problem: instead of relying on downstream rewriting modules, we directly align LLMs to generate text optimized for spoken delivery. We introduce two preference datasets spanning different target domains, CORA and Recipe, which contain paired TTS-friendly and TTS-unfriendly responses. We further propose an evaluation suite combining a pattern-based heuristic metric, a TTS→ASR evaluation pipeline, and a MUSHRA listening study with human judges. Our experiments compare the recently proposed Feature-aware Sampling and Tuning (FaST) framework -- leveraging interpretable features instead of a black-box reward model -- against an array of alignment baselines on the TTS-friendly generation task. Notably, we found that FaST achieves the best overall tradeoff between TTS-friendliness and helpfulness across various settings. We also identified a strong correlation between our different metrics, highlighting the ability to reliably assess TTS-friendliness via an efficient heuristic.
Bodhan AI together with AI4Bharat recently released a great update on Indic ASR

https://bodhan.ai/research/blogs/indic-transcribe
https://huggingface.co/tencent/AuK

AuK is a 1.5B foundation model for speech generation and editing. Trained on millions of hours of diverse audio data, AuK supports zero-shot and instruction-based TTS, content and acoustic editing, paralinguistic editing, speech enhancement, and source separation through a unified natural-language instruction interface.

https://arxiv.org/abs/2609.08936
Suplime results on Russian telephony data, good results, first place, large model is better than Diarizen

A bit slow though