Hybrid AR + MoG head for RVQ-based streaming TTS

1. Introduction

The AR head, also called Depth Transformer, was introduced in Moshi [1] for 8 codebooks (following the RQ-Transformer [2]), also adopted in DSM [3] and Qwen3-TTS [4] for 16 codebooks, and in CSM [5] for 32 codebooks of the Mimi [1] audio codec. The Depth Transformer provides stable generation and has the same optimization objective as the main backbone (Temporal Transformer), but utilizing more codebooks negatively affects latency, that is why Moshi is using only 8 codebooks, which in turn limits its speech quality. For example, VoXtream2 [6, 7] is using the Depth Transformer from the CSM model and operates on 16 codebooks of Mimi, where the generation of acoustic codebooks takes 62% of the total inference generation time (Figure 2).

The MoG head was proposed in VoiceChat-TTS [8] to accelerate the generation of deep RVQ tokens (see also ResGen [9] and GIVT [10]). Unlike the Depth Transformer, which predicts a single codebook at each generation step, the MoG head predicts the sum of multiple codebooks (the sum of their codebook vectors in the latent space of the codec, modelled as a mixture of Gaussians) with a consequent residual quantization step, which allows it to predict the same number of codebooks with fewer generation steps (Figure 3). VoiceChat-TTS uses 31 codebooks and predicts them in 4–8 generation steps, while a Depth Transformer would require 30 steps for the same setup.

2. Experiment setup

In our experiments we used the VoXtream2 model based on the Mimi audio codec (16 codebooks) with the AR Depth Transformer from the CSM model. The baseline is VoXtream2.1, evaluated on SEED-TTS test-en [11]: WER 1.32%, SPK-SIM 0.682, UTMOS 3.78 (Table 1).

We first replaced the Depth Transformer with the MoG head following the released code of VoiceChat-TTS. Unlike VoiceChat-TTS, our model did not improve from more generation steps (either due to a wrong implementation or because we used a different audio codec), and the best quality was achieved at 2 steps (Figure 4), with a slight improvement over single-step generation, while 4 or 8 steps introduced noise and worsened the quality. The obtained model got a significant speedup from the 15 → 2 generation steps reduction, but its quality also dropped (WER 1.86%, SPK-SIM 0.629, UTMOS 3.67).

We then analyzed the distribution of energy among codebooks (Figure 5) and found that most energy is concentrated within the first 4 codebooks: they hold 92.0% of the acoustic energy of the 15 acoustic codebooks (82.5% in the first codebook alone), and 89.1% of the 31 acoustic codebooks of the 32-codebook model. Then we decided to keep the Depth Transformer, but use it only for the first 4 coarse codebooks, and predict the remaining 11 codebooks in a single step of the MoG head. This gave us evaluation metrics similar to the baseline (WER 1.56%, SPK-SIM 0.671, UTMOS 3.77) but at 5 generation steps instead of 15 before (Table 1). We also conducted an ablation study with the reduced number of generation steps applied to the baseline model without the MoG head (Figure 6) in order to check whether we need the MoG head or not: grouping the codebooks of the AR head into 4 steps collapses the quality as soon as the groups get large, while the hybrid head stays at the baseline.

Then we checked if we can improve the quality by using more codebooks, as the number of steps is not capped at 5 (4 AR + MoG) anymore. From this point we used updated 54k hours training dataset (based on Emilia-EN [12] and HiFiTTS2 [13]) with improved 2-stage phoneme alignment and a new speaker embedding model ReDimNet2 [14], but here we just mention that the new baseline for further comparison is the model trained with 16 codebooks and 5 (4 AR + MoG) generation steps: WER 1.44%, SPK-SIM 0.702, UTMOS 3.77. With 32 codebooks we got improvements over all objective metrics (Table 1). At this point we improved the quality over VoXtream2 and also made the model faster (FPL 74 → 34 ms, RTF 0.256 → 0.174 on an RTX 3090). We also quantized the model (including the Mimi decoder) to the GGUF INT8 format [15] and tested it on Apple M4 (single core) and Intel i9-10920X (single core) CPUs. The RTF on M4 is 0.490, which means that the model works 2 times faster than real time, but on a single core of the Intel CPU it has an RTF of 1.3, limiting its deployment on less powerful hardware.

To address the latency issue we reduced the Depth Transformer size. The original Depth Transformer used in VoXtream2 is adopted from CSM, which is using a single model to generate 32 codebooks, but in our new setup we only need to generate 3 codebooks. DSM is also using a smaller separate Depth Transformer for the first 8 codebooks, confirming the relevance of a smaller model. We reduced the number of layers from 4 → 2, and the intermediate dimension from 8192 → 4096 (111M → 30M parameters). Unlike the previous experiments, we trained the Depth Transformer from scratch rather than initializing the weights from CSM and then fine-tuning them, but we did not notice a significant difference in objective metrics (except for SPK-SIM) for the new model (Table 1). As a result, we got a noticeable speedup: RTF on Intel dropped from 1.3 to 0.755, which paves the way for CPU deployment.

3. Evaluation

Table 2 shows results on common short-form zero-shot TTS benchmarks: SEED-TTS test-en [11] and LibriSpeech-PC [16]. We compare different versions of VoXtream2 to state-of-the-art full-stream TTS: Kyutai-TTS (DSM) [3] and CosyVoice3 [17]. VoXtream2.1 is trained on the same dataset as VoXtream2 and shares the same architecture, but its Depth Transformer was unfrozen during training, while VoXtream2 does not update the weights of the Depth Transformer borrowed from CSM, and VoXtream2.5 trains the Depth Transformer and MoG head from scratch. VoXtream2.5 achieves the best voice cloning, one of the best intelligibility and competitive UTMOS among all streaming systems, while using fewer trainable parameters and training data compared to DSM and CosyVoice3. We used Whisper-Large-v3 [18] to calculate WER (with Whisper text normalization applied as in [3]), WavLM-based ECAPA-TDNN [19, 20] for SPK-SIM calculation, and the UTMOS model [21].

Table 3 measures streaming performance: full-stream (input text and output audio streams) on the GPU, and output audio stream on CPU. FPL denotes First Packet Latency, the time from the first word on the input to the first audio frame on the output, and RTF stands for Real Time Factor, the time spent on generation divided by the duration of the generated audio. RTF < 1 means that the model is capable of real-time processing.

VoXtream2.5 runs much faster compared to CosyVoice3 and DSM and brings a significant improvement over VoXtream2.1 with 30 ms of FPL and almost 10 times faster than real-time generation. CosyVoice3 is using a chunked-streaming flow matching model and needs to accumulate some initial context before decoding, which negatively affects its FPL. DSM has a similar architecture to VoXtream2 and is using the same audio codec, although it delays the output stream by 1.2 seconds on the architecture level and is also using 15 AR steps in the Depth Transformer for each temporal generation step.

We also measured the performance of quantized VoXtream2.5 on different CPUs. On the Apple M4 CPU the model is running almost 4 times faster than real time with 218 ms of FPL on a single core, and 111 ms on 2 cores. On a single core of the Intel CPU VoXtream2.5 runs in real time, and on 2 cores it achieves 210 ms FPL and 2 times faster than real-time generation. We also compared objective metrics between the BF16 GPU version and the INT8 CPU version and did not notice any significant difference.

We also evaluated long-form TTS on the LibriSpeech-Long benchmark [22] (Figure 7). Lim et al. [23] proposed inference-time optimization which improves VoXtream2 long-form capabilities. Following their findings, we applied an attention bias of 1 to VoXtream2.5 to enable long-form generation. We borrowed the evaluation results from their paper and ran our model on the same dataset split and evaluated it with the same models (Whisper-Large-v3 and ECAPA-TDNN [20]). VoXtream2.5 achieves the strongest voice cloning in long-form generation, and its intelligibility is stable regardless of the audio length with an average WER of 1.65%.

4. Conclusions

The proposed hybrid AR + MoG head for RVQ-based TTS allows for high quality and low latency speech generation. The same head can be applied to other models such as Moshi [1], PersonaPlex [24], DSM [3], or Qwen3-TTS [4], to make them faster and improve speech quality.

Acknowledgements

Computational resources were provided on the Berzelius system funded by the Knut and Alice Wallenberg foundation and operated by NAISS.

5. Links

• Code: https://github.com/herimor/voxtream
• Model: https://huggingface.co/herimor/voxtream2.5
• Demo (CPU): https://huggingface.co/spaces/herimor/voxtream2.5-q8
• Demo (GPU): https://huggingface.co/spaces/herimor/voxtream2.5

Citation

If you find this work useful, please cite us.

@inproceedings{torgashov2026voxtream,
  title={Vo{X}tream: Full-Stream Text-to-Speech with Extremely Low Latency},
  author={Torgashov, Nikita and Henter, Gustav Eje and Skantze, Gabriel},
  booktitle={Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  year={2026},
  url={https://arxiv.org/abs/2509.15969}
}

@article{torgashov2026voxtream2,
  author={Torgashov, Nikita and Henter, Gustav Eje and Skantze, Gabriel},
  title={Vo{X}tream2: Full-stream TTS with dynamic speaking rate control},
  journal={arXiv:2603.13518},
  year={2026}
}

6. References

1. A. Défossez, L. Mazaré, M. Orsini, A. Royer, P. Pérez, H. Jégou, E. Grave, N. Zeghidour. Moshi: a speech-text foundation model for real-time dialogue. arXiv:2410.00037, 2024. https://arxiv.org/abs/2410.00037
2. D. Lee, C. Kim, S. Kim, M. Cho, W.-S. Han. Autoregressive Image Generation using Residual Quantization. CVPR 2022, arXiv:2203.01941. https://arxiv.org/abs/2203.01941
3. N. Zeghidour, E. Kharitonov, M. Orsini, V. Volhejn, G. de Marmiesse, E. Grave, P. Pérez, L. Mazaré, A. Défossez. Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling. arXiv:2509.08753, 2025. Code: kyutai-labs/delayed-streams-modeling. https://arxiv.org/abs/2509.08753
4. Qwen Team. Qwen3-TTS Technical Report. arXiv:2601.15621, 2026. Code: QwenLM/Qwen3-TTS. https://arxiv.org/abs/2601.15621
5. Sesame. Crossing the uncanny valley of conversational voice. Blog post, 2025. Code: SesameAILabs/csm. https://www.sesame.com/journal/crossing-the-uncanny-valley-of-voice
6. N. Torgashov, G. E. Henter, G. Skantze. VoXtream: Full-Stream Text-to-Speech with Extremely Low Latency. ICASSP 2026, arXiv:2509.15969. https://arxiv.org/abs/2509.15969
7. N. Torgashov, G. E. Henter, G. Skantze. VoXtream2: Full-stream TTS with dynamic speaking rate control. arXiv:2603.13518, 2026. Code: herimor/voxtream. https://arxiv.org/abs/2603.13518
8. E. Casanova, J. Kim, M. Graterol Fuenmayor, S. Hussain, V. Klimkov, V. Mendelev, M. Desta, P. Neekhara, P. Zelasko, C. Chen, E. Rastorgueva, K. Hu, A. Pasad, X. Yang, A. Alja'fari, R. Roy, R. Badlani, J. Roche, J. Li, Z. Chen. VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents. arXiv:2608.13831, 2026. https://arxiv.org/abs/2608.13831
9. J. Kim, T. Moon, K. Lee, J. Cho. Efficient Generative Modeling with Residual Vector Quantization-Based Tokens (ResGen). arXiv:2412.10208, 2024. https://arxiv.org/abs/2412.10208
10. M. Tschannen, C. Eastwood, F. Mentzer. GIVT: Generative Infinite-Vocabulary Transformers. ECCV 2024, arXiv:2312.02116. https://arxiv.org/abs/2312.02116
11. P. Anastassiou et al. Seed-TTS: A Family of High-Quality Versatile Speech Generation Models. arXiv:2406.02430, 2024. Test set: BytedanceSpeech/seed-tts-eval. https://arxiv.org/abs/2406.02430
12. H. He et al. Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation. arXiv:2407.05361, 2024. https://arxiv.org/abs/2407.05361
13. R. Langman et al. HiFiTTS-2: A Large-Scale High Bandwidth Speech Dataset. Interspeech 2025, arXiv:2506.04152. https://arxiv.org/abs/2506.04152
14. I. Yakovlev, A. Okhotnikov. ReDimNet2: Scaling Speaker Verification via Time-Pooled Dimension Reshaping. Interspeech 2026, arXiv:2603.11841. https://arxiv.org/abs/2603.11841
15. ggml-org. llama.cpp and the GGUF file format. GitHub. https://github.com/ggml-org/llama.cpp
16. A. Meister, M. Novikov, N. Karpov, E. Bakhturina, V. Lavrukhin, B. Ginsburg. LibriSpeech-PC: Benchmark for Evaluation of Punctuation and Capitalization Capabilities of End-to-end ASR Models. ASRU 2023, arXiv:2310.02943. https://arxiv.org/abs/2310.02943
17. Z. Du et al. CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training. arXiv:2505.17589, 2025. https://arxiv.org/abs/2505.17589
18. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, I. Sutskever. Robust Speech Recognition via Large-Scale Weak Supervision (Whisper). arXiv:2212.04356, 2022. https://arxiv.org/abs/2212.04356
19. S. Chen et al. WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing. arXiv:2110.13900, 2021. https://arxiv.org/abs/2110.13900
20. B. Desplanques, J. Thienpondt, K. Demuynck. ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification. Interspeech 2020, arXiv:2005.07143. https://arxiv.org/abs/2005.07143
21. T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, H. Saruwatari. UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022. arXiv:2204.02152. Implementation: tarepan/SpeechMOS. https://arxiv.org/abs/2204.02152
22. S. J. Park, J. Salazar, A. Jansen, K. Kinoshita, Y. M. Ro, R. J. Skerry-Ryan. Long-Form Speech Generation with Spoken Language Models (LibriSpeech-Long). ICML 2025, arXiv:2412.18603. https://arxiv.org/abs/2412.18603
23. J. Lim, J. Woo, H. Kim. Speaking Longer Than Learned: Training-Free Long-Form Generation for Full-Stream Text-to-Speech. arXiv:2610.19553, 2026. https://arxiv.org/abs/2610.19553
24. R. Roy, J. Raiman, B. Catanzaro et al. PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models. arXiv:2602.06053, 2026. Code: NVIDIA/PersonaPlex. https://arxiv.org/abs/2602.06053