https://github.com/anyreachai/dualturn
DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining
Shangeth Rajaa
DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining
Shangeth Rajaa
Speech-to-speech models handle turn-taking naturally but offer limited support for tool-calling or complex reasoning, while production ASR-LLM-TTS voice pipelines offer these capabilities but rely on silence timeouts, which lead to unnatural turn-taking. We present DualTurn, which narrows this gap through generative pretraining on dual-channel conversational audio. The model generates both speakers' future audio autoregressively, implicitly learning conversational dynamics without any labels, and is then fine-tuned to predict interpretable turn-taking signals that map directly to agent actions. DualTurn monitors both channels continuously, anticipating turn boundaries and producing five agent actions. On standard benchmarks, DualTurn (0.5B) outperforms both VAP on agent action prediction (wF1 0.633 vs. 0.389) and a 3.1B audio-text model on word-level turn prediction (AUC 0.930 vs. 0.880), while anticipating turn boundaries earlier with fewer interruptions.
GitHub
GitHub - anyreachai/dualturn: Official implementation of "DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining"…
Official implementation of "DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining" (Interspeech 2026) - anyreachai/dualturn
When full to end doesn't really work this thing has potential
https://arxiv.org/abs/2608.13831
VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents
Edresson Casanova, Jaehyeon Kim, Mariana Graterol Fuenmayor, Shehzeen Hussain, Viacheslav Klimkov, Valentin Mendelev, Mikyas Desta, Paarth Neekhara, Piotr Zelasko, Chen Chen, Elena Rastorgueva, Ke Hu, Ankita Pasad, Xuesong Yang, Aya Alja'fari, Rajarshi Roy, Rohan Badlani, Jason Roche, Jason Li, Zhehuai Chen
https://arxiv.org/abs/2608.13831
VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents
Edresson Casanova, Jaehyeon Kim, Mariana Graterol Fuenmayor, Shehzeen Hussain, Viacheslav Klimkov, Valentin Mendelev, Mikyas Desta, Paarth Neekhara, Piotr Zelasko, Chen Chen, Elena Rastorgueva, Ke Hu, Ankita Pasad, Xuesong Yang, Aya Alja'fari, Rajarshi Roy, Rohan Badlani, Jason Roche, Jason Li, Zhehuai Chen
Spoken dialogue is a natural form of human--computer interaction, yet most speech language models remain limited to turn-based operation and lack real-time adaptability, such as user barge-in. Recent duplex speech-to-speech and speech-to-text models reduce latency by replacing multi-stage pipelines, but often compromise speech quality because accurate ASR, interruption handling, and high-fidelity synthesis must be optimized jointly. We propose VoiceChat-TTS, a low-latency, continuous, and streamable text-to-speech model for interactive agents. VoiceChat-TTS is driven directly by LLM text-token streams, supports explicit interruption via control tokens, and produces silence when no textual input is available. The model enables always-on, responsive speech generation while preserving modularity and high speech quality, and it supports mid-utterance interruptions without resetting the KV cache.
arXiv.org
VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for...
Spoken dialogue is a natural form of human--computer interaction, yet most speech language models remain limited to turn-based operation and lack real-time adaptability, such as user barge-in....
Alex Smola leading a great research at Boson.AI (Higgs authors). Couple of recent selected papers:
https://arxiv.org/search/cs?searchtype=author&query=Smola,+A
https://arxiv.org/abs/2603.25727
Back to Basics: Revisiting ASR in the Age of Voice Agents
Geeyang Tay, Wentao Ma, Jaewon Lee, Yuzhi Tang, Daniel Lee, Weisu Yin, Dongming Shen, Silin Meng, Yi Zhu, Mu Li, Alex Smola
https://arxiv.org/abs/2607.20460
Instruct-FD: Can Your Full-Duplex Speech System Follow Turn-Taking Instructions?
Yuzhi Tang, Wentao Ma, Xiling Zhao, Ahmad Salimi, Sepehr Harfi Moridani, Dongming Shen, Jixuan Wang, Abdulrahman Abdulrazzag, Murdock Aubry, Yu-Hua Chen, Daniel Lee, Jaewon Lee, Jonah Mackey, Silin Meng, Nicholas Stranges, Chenxu Xiong, Hao Yu, Yi Zhu, Mu Li, Alex Smola
https://arxiv.org/search/cs?searchtype=author&query=Smola,+A
https://arxiv.org/abs/2603.25727
Back to Basics: Revisiting ASR in the Age of Voice Agents
Geeyang Tay, Wentao Ma, Jaewon Lee, Yuzhi Tang, Daniel Lee, Weisu Yin, Dongming Shen, Silin Meng, Yi Zhu, Mu Li, Alex Smola
Automatic speech recognition (ASR) systems have achieved near-human accuracy on curated benchmarks, yet still fail in real-world voice agents under conditions that current evaluations do not systematically cover. Without diagnostic tools that isolate specific failure factors, practitioners cannot anticipate which conditions, in which languages, will cause what degree of degradation. We introduce WildASR, a multilingual (four-language) diagnostic benchmark sourced entirely from real human speech that factorizes ASR robustness along three axes: environmental degradation, demographic shift, and linguistic diversity. Evaluating seven widely used ASR systems, we find severe and uneven performance degradation, and model robustness does not transfer across languages or conditions. Critically, models often hallucinate plausible but unspoken content under partial or degraded inputs, creating concrete safety risks for downstream agent behavior. Our results demonstrate that targeted, factor-isolated evaluation is essential for understanding and improving ASR reliability in production systems. Besides the benchmark itself, we also present three analytical tools that practitioners can use to guide deployment decisions.
https://arxiv.org/abs/2607.20460
Instruct-FD: Can Your Full-Duplex Speech System Follow Turn-Taking Instructions?
Yuzhi Tang, Wentao Ma, Xiling Zhao, Ahmad Salimi, Sepehr Harfi Moridani, Dongming Shen, Jixuan Wang, Abdulrahman Abdulrazzag, Murdock Aubry, Yu-Hua Chen, Daniel Lee, Jaewon Lee, Jonah Mackey, Silin Meng, Nicholas Stranges, Chenxu Xiong, Hao Yu, Yi Zhu, Mu Li, Alex Smola
Current full-duplex (FD) spoken dialogue systems can produce fluid interactions, yet it remains unclear whether they can adapt their turn-taking behavior when explicitly instructed. This is critical for real-world deployment, where conversational policies vary across applications (e.g., proactive tutoring vs. passive counseling). We introduce Instruct-FD, an instruction-conditioned benchmark for evaluating controllable turn management in FD systems. To enable this, we develop a human-validated, scalable synthetic pipeline that generates instruction-conditioned conversations, along with a deployment-agnostic multi-turn evaluation protocol and an LLM-based judge. Benchmarking six state-of-the-art full-duplex systems reveals a substantial gap in instruction-following turn management: the best model achieves only 64.4% adherence. Performance is highly uneven across behaviors and scenarios, with proactive behaviors such as model backchanneling and interruption remaining particularly challenging. These findings establish instruction-following turn management as a crucial direction for building adaptable and deployable full-duplex dialogue systems.
arXiv.org
Back to Basics: Revisiting ASR in the Age of Voice Agents
Automatic speech recognition (ASR) systems have achieved near-human accuracy on curated benchmarks, yet still fail in real-world voice agents under conditions that current evaluations do not...
Some modern tech for speech processing on earbuds, interesting talk overall
https://www.youtube.com/watch?v=zeDkT8EuKao
https://www.youtube.com/watch?v=zeDkT8EuKao
YouTube
CLSP Summer Program: Plenary Speaker and Weekly Progress Report
Enjoy the videos and music you love, upload original content, and share it all with friends, family, and the world on YouTube.
Audio in the wild is still very complex
https://aslp-lab.github.io/SmartGlasses/
https://x.com/MosiAI_Official/status/2090775763666812997
MOSS takes 1st place across all four tasks in the IEEE SLT 2026 SmartGlasses Challenge. The challenge tests whether AI can truly hear and understand real conversations through smart glasses — handling noise, overlapping speech, multiple speakers, and long-context interactions. Among 77 participating teams, MOSS achieved 1st place across both tracks and all four tasks, covering speech recognition and spoken language understanding in real-world smart glasses scenarios.
Results: Two-person TSA-ASR — 5.23% tcpCER
Two-person SLU — 88.8% Accuracy
Multi-party TSA-ASR — 27.95% tcpCER
Multi-party SLU — 93.0% Accuracy
The models behind these results: • MOSS-Transcribe-Diarize for TSA-ASR • MOSS-Audio for SLU
https://aslp-lab.github.io/SmartGlasses/
https://x.com/MosiAI_Official/status/2090775763666812997
MOSS takes 1st place across all four tasks in the IEEE SLT 2026 SmartGlasses Challenge. The challenge tests whether AI can truly hear and understand real conversations through smart glasses — handling noise, overlapping speech, multiple speakers, and long-context interactions. Among 77 participating teams, MOSS achieved 1st place across both tracks and all four tasks, covering speech recognition and spoken language understanding in real-world smart glasses scenarios.
Results: Two-person TSA-ASR — 5.23% tcpCER
Two-person SLU — 88.8% Accuracy
Multi-party TSA-ASR — 27.95% tcpCER
Multi-party SLU — 93.0% Accuracy
The models behind these results: • MOSS-Transcribe-Diarize for TSA-ASR • MOSS-Audio for SLU
Speech Technology
Audio in the wild is still very complex https://aslp-lab.github.io/SmartGlasses/ https://x.com/MosiAI_Official/status/2090775763666812997 MOSS takes 1st place across all four tasks in the IEEE SLT 2026 SmartGlasses Challenge. The challenge tests whether…
Somehow they forgot about leaderboard
https://www.hume.ai/blog/measuring-benchmark-optimization-in-speech-recognition
https://x.com/hume_ai/status/2090813428793319933
https://x.com/hume_ai/status/2090813428793319933
www.hume.ai
Measuring benchmark optimization in speech recognition
New research introduces three tests to quantify benchmark optimization in speech recognition, finding top open-source ASR models reproduce benchmark transcripts even when the audio contradicts them.
Kytai released pockettts training code
https://github.com/kyutai-labs/pocket-tts/tree/main/training
https://x.com/kyutai_labs/status/2092254286772080768
https://github.com/kyutai-labs/pocket-tts/tree/main/training
https://x.com/kyutai_labs/status/2092254286772080768
https://huggingface.co/BreezeBlue/Breeze-TTS-2
https://breezeblue.ai/breeze-tts-2
English/Chinese only but really good quality. 1st place on ArtificalAnalysis leaderboard
https://breezeblue.ai/breeze-tts-2
English/Chinese only but really good quality. 1st place on ArtificalAnalysis leaderboard
huggingface.co
BreezeBlue/Breeze-TTS-2 · Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
You can download really huge datasets these days
https://x.com/ahochlehnert/status/2092648676829413778
LAION-BVD: a 10-million-hour open video dataset for multimodal pre-training. - 1.3B video URLs from CommonCrawl - 80M downloaded videos - 10M video hours - 55M captioned clips - 300M frame-caption pairs
This repository contains 1.7 million audio clips taken from BVD-V-55M and sampled for uniqueness of the source video, so that the subset maximises source diversity rather than clip count. Each clip comes with a caption, its language, and the timestamps locating it in the source video.
https://huggingface.co/datasets/laion/BVD-A-1.7M
https://x.com/ahochlehnert/status/2092648676829413778
LAION-BVD: a 10-million-hour open video dataset for multimodal pre-training. - 1.3B video URLs from CommonCrawl - 80M downloaded videos - 10M video hours - 55M captioned clips - 300M frame-caption pairs
This repository contains 1.7 million audio clips taken from BVD-V-55M and sampled for uniqueness of the source video, so that the subset maximises source diversity rather than clip count. Each clip comes with a caption, its language, and the timestamps locating it in the source video.
https://huggingface.co/datasets/laion/BVD-A-1.7M
X (formerly Twitter)
Andreas Hochlehnert (@ahochlehnert) on X
[1/7] 🚨 We're releasing LAION-BVD: a 10-million-hour open video dataset for multimodal pre-training.
- 1.3B video URLs from CommonCrawl
- 80M downloaded videos
- 10M video hours
- 55M captioned …
- 1.3B video URLs from CommonCrawl
- 80M downloaded videos
- 10M video hours
- 55M captioned …
https://turnbench.sesame.com/
A multi-domain benchmark for evaluating conversational turn-taking. We hand-annotate end-of-turn and interruption events in dual-channel human conversations, and measure how accurately and how quickly models detect them.
A multi-domain benchmark for evaluating conversational turn-taking. We hand-annotate end-of-turn and interruption events in dual-channel human conversations, and measure how accurately and how quickly models detect them.
We know human scores are useless but anyway
https://x.com/datapointai/status/2094829412625654141
today, we're releasing the largest open-source human audio preferences dataset, focused on the customer support use-case
- 300K+ annotations by real people
- 15 SOTA TTS models ranked (Sonic 3.6, Grok TTS, Simba 3.2, Eleven Labs v3)
- 8 categories (IVR menus, empathy, escalations, refunds etc)
dataset + benchmark + frontier plot below:
https://x.com/datapointai/status/2094829412625654141
today, we're releasing the largest open-source human audio preferences dataset, focused on the customer support use-case
- 300K+ annotations by real people
- 15 SOTA TTS models ranked (Sonic 3.6, Grok TTS, Simba 3.2, Eleven Labs v3)
- 8 categories (IVR menus, empathy, escalations, refunds etc)
dataset + benchmark + frontier plot below:
X (formerly Twitter)
Datapoint AI (@datapointai) on X
today, we're releasing the largest open-source human audio preferences dataset, focused on the customer support use-case
- 300K+ annotations by real people
- 15 SOTA TTS models ranked (Sonic 3.6,…
- 300K+ annotations by real people
- 15 SOTA TTS models ranked (Sonic 3.6,…
Quite an obvious but so ignored by industry before. There is certainly no need to clone from 3 seconds
You can now create an instant voice clone from up to 2 minutes of reference audio, compared with 20 seconds before.
The longer sample gives the model much more information about the voice, including phonemes, intonation, pauses, rhythm, pacing, emphasis, emotion, and speaking style.
The result:
• Higher voice similarity
• Greater consistency, especially across longer sentences
• Better pronunciation coverage
• More natural prosody
• Better preservation of speaking style
https://www.linkedin.com/posts/soniox_soniox-texttospeech-voiceai-activity-7501594413660401664-p9Cq
You can now create an instant voice clone from up to 2 minutes of reference audio, compared with 20 seconds before.
The longer sample gives the model much more information about the voice, including phonemes, intonation, pauses, rhythm, pacing, emphasis, emotion, and speaking style.
The result:
• Higher voice similarity
• Greater consistency, especially across longer sentences
• Better pronunciation coverage
• More natural prosody
• Better preservation of speaking style
https://www.linkedin.com/posts/soniox_soniox-texttospeech-voiceai-activity-7501594413660401664-p9Cq
LinkedIn
Soniox TTS v2: Improved Voice Cloning with Longer Audio Samples | Soniox posted on the topic | LinkedIn
We’ve shipped a major improvement to voice cloning in Soniox TTS v2.
You can now create an instant voice clone from up to 2 minutes of reference audio, compared with 20 seconds before.
The longer sample gives the model much more information about the voice…
You can now create an instant voice clone from up to 2 minutes of reference audio, compared with 20 seconds before.
The longer sample gives the model much more information about the voice…
https://arxiv.org/abs/2609.01246v1
Ready to Speak: Aligning LLMs for TTS-Friendly Text Generation
Thibaut Thonet, Jos Rozen, Laurent Besacier
Current Large Language Models (LLMs) are primarily optimized for written text, often producing outputs that are grammatically correct and helpful yet poorly suited for spoken delivery via Text-to-Speech (TTS). In this work, we study how to make LLMs natively generate TTS-friendly text, which we frame as a preference alignment problem: instead of relying on downstream rewriting modules, we directly align LLMs to generate text optimized for spoken delivery. We introduce two preference datasets spanning different target domains, CORA and Recipe, which contain paired TTS-friendly and TTS-unfriendly responses. We further propose an evaluation suite combining a pattern-based heuristic metric, a TTS→ASR evaluation pipeline, and a MUSHRA listening study with human judges. Our experiments compare the recently proposed Feature-aware Sampling and Tuning (FaST) framework -- leveraging interpretable features instead of a black-box reward model -- against an array of alignment baselines on the TTS-friendly generation task. Notably, we found that FaST achieves the best overall tradeoff between TTS-friendliness and helpfulness across various settings. We also identified a strong correlation between our different metrics, highlighting the ability to reliably assess TTS-friendliness via an efficient heuristic.
Ready to Speak: Aligning LLMs for TTS-Friendly Text Generation
Thibaut Thonet, Jos Rozen, Laurent Besacier
Current Large Language Models (LLMs) are primarily optimized for written text, often producing outputs that are grammatically correct and helpful yet poorly suited for spoken delivery via Text-to-Speech (TTS). In this work, we study how to make LLMs natively generate TTS-friendly text, which we frame as a preference alignment problem: instead of relying on downstream rewriting modules, we directly align LLMs to generate text optimized for spoken delivery. We introduce two preference datasets spanning different target domains, CORA and Recipe, which contain paired TTS-friendly and TTS-unfriendly responses. We further propose an evaluation suite combining a pattern-based heuristic metric, a TTS→ASR evaluation pipeline, and a MUSHRA listening study with human judges. Our experiments compare the recently proposed Feature-aware Sampling and Tuning (FaST) framework -- leveraging interpretable features instead of a black-box reward model -- against an array of alignment baselines on the TTS-friendly generation task. Notably, we found that FaST achieves the best overall tradeoff between TTS-friendliness and helpfulness across various settings. We also identified a strong correlation between our different metrics, highlighting the ability to reliably assess TTS-friendliness via an efficient heuristic.
arXiv.org
Ready to Speak: Aligning LLMs for TTS-Friendly Text Generation
Current Large Language Models (LLMs) are primarily optimized for written text, often producing outputs that are grammatically correct and helpful yet poorly suited for spoken delivery via...
Scicom from Malaysia tries Ascend 910B3
https://github.com/Scicom-AI-Enterprise-Organization/TTS-API-Neucodec/blob/main/ASCEND_910B3_PRECISION_REPORT.md
https://github.com/Scicom-AI-Enterprise-Organization/TTS-API-Neucodec/blob/main/ASCEND_910B3_PRECISION_REPORT.md
GitHub
TTS-API-Neucodec/ASCEND_910B3_PRECISION_REPORT.md at main · Scicom-AI-Enterprise-Organization/TTS-API-Neucodec
TTS API OpenAI compatible on top of Neucodec Speech Tokenizer LLM - Scicom-AI-Enterprise-Organization/TTS-API-Neucodec
Bodhan AI together with AI4Bharat recently released a great update on Indic ASR
https://bodhan.ai/research/blogs/indic-transcribe
https://bodhan.ai/research/blogs/indic-transcribe
ParsVoice, the largest open-source Persian speech dataset, along with a TTS model and an open-source processing pipeline is released.
The paper has also been accepted as a main conference paper at EMNLP 2026.
Paper: https://arxiv.org/abs/2510.10774
Dataset: https://huggingface.co/datasets/MohammadJRanjbar/ParsVoice
TTS model: https://huggingface.co/MohammadJRanjbar/ParsVoice-XTTS
Code & pipeline: https://github.com/MohammadJRanjbar/ParsVoice
The paper has also been accepted as a main conference paper at EMNLP 2026.
Paper: https://arxiv.org/abs/2510.10774
Dataset: https://huggingface.co/datasets/MohammadJRanjbar/ParsVoice
TTS model: https://huggingface.co/MohammadJRanjbar/ParsVoice-XTTS
Code & pipeline: https://github.com/MohammadJRanjbar/ParsVoice
arXiv.org
ParsVoice: A Large-Scale Multi-Speaker Persian Speech Corpus for...
Persian remains substantially underrepresented in open speech-text resources, limiting progress in multi-speaker text-to-speech (TTS), speech-language modelling, and low-resource speech...
https://huggingface.co/tencent/AuK
AuK is a 1.5B foundation model for speech generation and editing. Trained on millions of hours of diverse audio data, AuK supports zero-shot and instruction-based TTS, content and acoustic editing, paralinguistic editing, speech enhancement, and source separation through a unified natural-language instruction interface.
https://arxiv.org/abs/2609.08936
AuK is a 1.5B foundation model for speech generation and editing. Trained on millions of hours of diverse audio data, AuK supports zero-shot and instruction-based TTS, content and acoustic editing, paralinguistic editing, speech enhancement, and source separation through a unified natural-language instruction interface.
https://arxiv.org/abs/2609.08936
huggingface.co
tencent/AuK · Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.