Time to go multimodal, great dataset for audio/video things

https://github.com/FreedomIntelligence/TalkVid

https://github.com/higurashi-ai/FastDialogueSidon

FastDialogueSidon is a fast one-step model for full-duplex two-speaker dialogue separation and joint restoration: a raw 16 kHz mono mixture (telephone / conversational audio, up to 120 s per window) goes in, two separated 24 kHz speaker tracks come out. Every module — encoder, DiT head and decoder — is trained for FastDialogueSidon. A single DiT step per window replaces the 30 diffusion steps of DialogueSidon, and a lightweight decoder (a frame-rate SAN-WaveNeXt2 (WaveFitNeXt) latent decoder, 30 M parameters) replaces its DAC decoder. Running on fused GPU kernels, it is ~100x faster than the official DialogueSidon release run as-is, at matching intelligibility.

Deepgram is acquiring Daily (company behind Pipecat)

https://x.com/kwindla/status/2115791491586986432

https://huggingface.co/spaces/Edge0/Audio8-TTS-Realistic-0.6b

Introducing Audio8-TTS-Realistic, @Edge0_AI’s new open-weight speech model: realistic, zero-shot voice cloning in just 0.6B parameters.

• Human-like prosody and expressive delivery
• Clones a voice from one short clip, in 23 languages
• 601M params: small enough to fine-tune and serve affordably

Currently #1 on the Hugging Face TTS Arena, where people vote blind on which voice sounds more human.

Open weights are available now on @huggingface. Try it free.

1. How it learns to sound human:

TTS is usually trained on two things: the right words (WER) and the right voice (similarity).

But neither captures rhythm, pauses, or timing.

So we trained a realism critic that scores each output against a real person saying the same sentence in the same voice—ranked via a blind listening study.

Then: SFT → GRPO on accuracy + similarity → GRPO on the critic.

That last stage only saw Chinese and English prompts, yet it improved all 13 lower-resource languages.

2. How it stacks up at 0.6B:

• CV3 (9 languages): 4.86% error — ahead of CosyVoice3-1.5B (6.89%), Qwen3-TTS-1.7B (7.73%), and VoxCPM2 (5.35%)
• MiniMax (24 languages): 5.89% error — vs. 27.41% for Qwen3-TTS-1.7B
• Runs in ~1.1 GB RAM on an Apple M2 CPU (INT4)

Larger 4–8B models still lead on some sets, but we’re within reach at a fraction of the size.

3. Audio8-TTS-Realistic is now open source.

Quick adaptation to a domain is still a big problem, an interesting concept from Whissle from our Reddit:

https://huggingface.co/WhissleAI/arc-v1.1

Given the list of names it biases output to them and returns them already annotated in speech. However,the algorithm itself is a simple trie biasing during beam search and it doesn't always work as required tokens disappear in lower layers.

A good approach is to inject bias deeper in the layers, however, it is not very standard yet. An interesting recent work on optimizing NanoGPT training with ngrams might be the right direction

https://brunborg.com/n-gram-hinted-lm

Overall there is an interesting trend in improvement of CTC architectures with some LLM features. A good Turkish model
released recently:

https://huggingface.co/erayyapagci/balbal-v2-manas
https://erayy.com/balbal/

No publication for that, so quite unclear what the features bring but the features look interesting. Claude lists
the following related publications:

Lample, Sablayrolles, Ranzato, Denoyer, Jégou. Large Memory Layers with Product Keys. NeurIPS 2019.
https://arxiv.org/abs/1907.05242
The original PKM: two sub-key banks, top-k over k*k pairs, query BatchNorm, multi-head reads of one shared value table.

Kim, Jung. Large Product Key Memory for Pretrained Language Models. Findings of EMNLP 2020.
https://arxiv.org/abs/2010.03881
Memory slots going stale during training; adding PKM to a pretrained model, alongside the FFN rather than replacing it.

Berges, Oğuz, Haziza, Yih, Zettlemoyer, Ghosh. Memory Layers at Scale. ICML 2025 (PMLR 267).
https://arxiv.org/abs/2412.09764
Memory layers up to 128B memory parameters; largest gains on factual tasks.

Huang, Min, et al. (ByteDance Seed). Ultra-Sparse Memory Network (UltraMem). ICLR 2025.
https://arxiv.org/abs/2411.12364
PKM extended to 20M slots; faster inference than MoE of the same size.

DeepSeek-AI, Peking University. Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models (Engram). 2026.
https://arxiv.org/abs/2601.07372
Hashed N-gram lookup. Gate sigmoid(RMSNorm(h) . RMSNorm(W_K e) / sqrt(d)) on W_V e: the same structure as balbal's gate
(balbal uses LayerNorm and a learned bias starting at -2).

Nozaki, Komatsu. Relaxing the Conditional Independence Assumption of CTC-Based ASR by Conditioning on Intermediate Predictions
(Self-Conditioned CTC). Interspeech 2021.
https://www.isca-archive.org/interspeech_2021/nozaki21_interspeech.html
Intermediate CTC predictions fed back into the next layer's input; compare balbal's soft_feedback.

Chi, Salazar, Kirchhoff. Align-Refine: Non-Autoregressive Speech Recognition via Iterative Realignment. NAACL 2021.
https://arxiv.org/abs/2010.14233
Iterative Transformer refinement of CTC alignments in three passes.

Another model is Japanese

https://github.com/third-intelligence/Haas-ASR

Its prediction network uses Kimi Delta Attention from Kimi Linear, run on the kernels of flash-linear-attention

https://arxiv.org/abs/2510.26692
Kimi Linear: An Expressive, Efficient Attention Architecture

Massive effort. Datasets are not complete but often interesting finds

https://huggingface.co/spaces/kadirnar/stt-arena

More complete information about memory experiments from Erayy

https://erayy.com/balbal-blog/

Honestly, my impression is that architecture is not well tested, a dataset and tests could be better. Maybe if he trained English model first it would be easier to evaluate it. Optimizing from 38% to 37.5% on Turkish doesn't make much sense.

LFM3 Omni is cool but I don't understand something. It claims 140ms TTFA, but the actual delay in the video between input end and the response is 1.3 seconds.

https://x.com/xenovacom/status/2115923280079286543

Arabic gonna have some boost soon

https://huggingface.co/datasets/speechcolab/gigaspeech3

https://arxiv.org/abs/2611.02476

GigaSpeech 3: Scaling Multi-Dialect Arabic Speech Recognition with Weakly Labeled Data
Junzhe Liu, Yujie Tu, Yifan Yang, Peng Wang, Haoran Wang, Guodong Lin, Leyan Liao, Chengquan Zhang, Zhifu Gao, Xiangang Li, Qinyuan Cheng, Xipeng Qiu, Wei-Qiang Zhang, Guoguo Chen, Kai Yu, Xie Chen

Dialectal Arabic speech recognition remains constrained by the scarcity of dialect-resolved transcribed speech despite the abundance of web audio. We introduce GigaSpeech 3, a corpus of 39,000 hours and 10 million weakly transcribed speech segments covering Algerian, Egyptian, Iraqi, Saudi, and Syrian Arabic. Starting from 60 hours of manual transcription per dialect, we train labeling models using speech-recognition-biased self-supervised learning and apply them to dialect-matched web audio. Data selection and four rounds of noisy-student refinement for Saudi and Syrian Arabic yield the final corpus. We validate its utility through dialect-specific and multi-dialect speech recognition training and cross-domain evaluation. On GigaSpeechBench, the final dialect-specific models outperform their labeling models across all five dialects and achieve a macro-average word error rate of 23.86%, a 16.6% relative reduction from the strongest third-party system covering all five dialects, while outperforming the best third-party result on three dialects. A multi-dialect model using self-predicted dialect identities achieves a macro-average word error rate of 25.51% while also supporting Modern Standard Arabic. Across eight conditions from three external corpora, a GigaSpeech 3-trained Zipformer achieves the lowest word error rate among six evaluated models in every condition, while adapting the Qwen3 speech recognizer on GigaSpeech 3 improves performance across all eight conditions. We release the corpus and recipes as a foundation for dialect-aware Arabic speech processing.

Beside GPRO

A recursive self-improvement loop
We also use a recursive self-improvement (RSI) loop: our agent system interacts with simulated environments, reviews its failures, synthesizes targeted data, and trains the model on it before repeating the cycle. The simulation must reproduce the timing and variety of real interaction—interruptions, overlapping speech, side conversations, background noise—realistic enough to teach natural behavior and hard enough to expose weaknesses. The engineering target is the whole harness of simulators, tools, and judges, not an isolated model. For a small team, the loop turns edge cases into training data and lets us iterate with far less manual data work.

https://www.boson.ai/blog/higgs-realtime-duplex

160M TTS, singing feature is nice

https://github.com/lowkeytea/scyllasband

Hugging Face: https://huggingface.co/spybyscript/scyllasband

Try it in your browser: https://lowkeytea.dev/scyllasband/demo/

https://www.youtube.com/watch?v=GHkj44MaB9Y

https://github.com/lilt/tau-bench-lilt

https://arxiv.org/abs/2611.07203

τ-bench-LILT extends τ³-bench to German, Korean, Turkish and Farsi in the airline, retail, telecom and banking domains. We find that even frontier models can be tripped up by simple differences in language and locale. Claude Opus 5, for example, has a pass^4 rate of 81.6 in the English retail domain, but only 26.3 in Korean, because it insists on romanizing Korean customers' names (among other failure modes) before looking them up in the database.

Voice AI rarely works, guy tested many of them in real situation

https://x.com/huskirl/status/2116941001335947635

https://www.youtube.com/@HuskIRL