Models

TML-Interaction-Small

276B-A12B mixture-of-experts interaction model with VAD-free, time-aligned 200 ms micro-turns across audio, video, and text — Thinking Machines Lab's first public voice release.

Announced 2026-05-12 as Thinking Machines Lab's first public model. Encoder-free early-fusion design (dMel audio embeddings, hMLP video patches, flow-head audio decoder) ingests audio, video, and text as concurrent streams and decodes in 200 ms time-aligned micro-turns — replacing standard voice-activity detection with model-internal yield / self-correct / invite signals. Reports 0.40 s turn-taking latency on FD-Bench v1 and 77.8 / 100 interaction quality on FD-Bench v1.5, ahead of GPT-Realtime-2 and Gemini 3.1 Flash Live on conversational dynamics; trails GPT-Realtime-2 xhigh on pure intelligence (43.4 % vs 48.5 % on Audio MultiChallenge APR). Introduces four TML-internal benchmarks — TimeSpeak, CueSpeak, RepCount-A, ProactiveVideoQA — alongside published scores. Limited research preview in the coming months; wider release later in 2026. License and weights posture undisclosed.

Covo-Audio-Chat / Covo-Audio-Chat-FD

7B end-to-end audio LLM with a native full-duplex variant — tri-modal speech-text interleaving and dual-stream listen-and-speak.

Paper: 'Covo-Audio Technical Report' (arXiv 2602.09823, Feb 2026). Three releases share the same 7B backbone (Qwen2.5-7B + Whisper): Covo-Audio (base), Covo-Audio-Chat (spoken-dialogue variant, open-sourced), and Covo-Audio-Chat-FD (native low-latency full-duplex variant, benchmarked against Freeze-Omni / MiMo-Audio / Step-Audio 2 on URO-Bench). Model card labels weights as 'research and experimental purposes only,' so treat as non-commercial until a formal license is published.

MiniCPM-o 4.5

9B on-device omni-modal LLM with full-duplex speech + vision streaming, targeting Gemini 2.5 Flash quality on a laptop.

End-to-end multimodal model that processes continuous video + audio streams while generating text + speech in parallel — none of the four streams block each other. Uses a time-division-multiplex (TDM) full-duplex mechanism with an interleaved speech-token decoder. Ships with llama.cpp-omni / WebRTC demos for local deployment on a MacBook (M4 Max / 24 GB) or a small GPU. Covers 30+ languages via Qwen backbone.

PersonaPlex

7B full-duplex speech LM with hybrid voice + text prompts — any role, any voice, on top of Moshi.

Fine-tunes Kyutai's Moshi stack (Mimi codec @ 24 kHz, Helium LM backbone) with a hybrid conditioning path: a voice prompt captures timbre/style and a text prompt pins role, facts, and scenario. Trained on 1,217 h of Fisher English dyads plus ~2,250 h of synthetic assistant / customer-service dialogues rendered with Chatterbox TTS. NVIDIA reports ~170 ms average latency vs cascaded baselines on FullDuplexBench-style evaluations. Weights are hosted as `nvidia/personaplex-7b-v1` on Hugging Face; code is MIT, weights use the NVIDIA Open Model License.

SALMONN-omni

First standalone codec-free full-duplex speech LLM — no audio tokens in the vocabulary, RL-trained for barge-in.

Codec-free architecture: streaming speech encoder → LLM backbone (no codec tokens in its vocab) → streaming speech synthesizer, synchronised through embeddings and a dynamic 'thinking' mechanism. Reports a 30–36% relative improvement over Moshi, Freeze-Omni, Qwen2.5-Omni, GLM-4-Voice, VITA-1.5, MiniCPM-o, Kimi-Audio, Baichuan-Audio on open-domain spoken QA with substantially less training data. First FD speech LLM to apply RL to turn-taking / context-dependent barge-in. Published as NeurIPS 2025; weights released on the SALMONN org.

Step-Audio 2 mini

End-to-end multimodal audio LLM for speech conversation with expressive TTS and tool / search integration.

Open-weights release of the mini and mini-Base checkpoints (initialized from Qwen2-Audio / Qwen2.5-7B). Step-Audio 2 handles paralinguistic cues (laughter, sighs, accent, dialect), supports dynamic voice switching, and ships with a vLLM backend plus a hosted StepFun realtime console. The `mini Think` variant released in 2025-09 adds chain-of-thought reasoning for speech. Covered as a full-duplex baseline in the Covo-Audio and MiMo-Audio technical reports.

Kimi-Audio

A 7B audio foundation model covering ASR, AQA, audio QA, and end-to-end voice chat in one net.

Hybrid audio input (continuous acoustic vectors + discrete semantic tokens) feeds an LLM with parallel text and audio output heads. Pre-trained on 13M hours of audio. Ships with an evaluation toolkit, Kimi-Audio-Evalkit.

LLaMA-Omni 2

Qwen2.5-based real-time spoken chatbot series (0.5B–14B) with autoregressive streaming speech synthesis.

ACL 2025 main-conference paper. Speech encoder → Qwen2.5-0.5B/1.5B/3B/7B/14B/32B-Instruct backbone → autoregressive streaming speech decoder (CosyVoice 2 tokenizer + flow-matching vocoder). 7B variant reports total first-audio latency ≈580 ms with chunk size 3 / 10. Trained on only 200K multi-turn speech dialogues yet matches GLM-4-Voice for spoken QA. Turn-based rather than native full-duplex; interrupt / barge-in is not part of the released model. **Signals W25 positioning note:** [BayLing-Duplex (arXiv 2606.14528)](https://arxiv.org/abs/2606.14528) explicitly cites LLaMA-Omni and GLM-4-Voice as the two reference points for SpeechLMs that are still turn-based because they rely on an external VAD module. The `fdMode: "turn"` flag here was already correct; BayLing-Duplex is now the canonical citation for that classification.

Sesame CSM

The conversational speech model everyone called "uncanny-valley-free."

A Llama-backbone model with a smaller audio decoder that emits Mimi codec frames. The 1B checkpoint is public; larger 8B variant powers the hosted demo. Strong at laughs, sighs, and non-verbal cues.

GLM-4-Voice

An end-to-end Chinese/English speech LM that exposes tone, dialect, and emotion as instructions.

A three-part stack: a Whisper-based VQ tokenizer (12.5 tok/s), a 9B LM fine-tuned on speech tokens, and a Flow-Matching decoder that starts streaming audio after as few as 10 tokens. One of the few open speech-LMs with strong Chinese coverage. **Signals W25 positioning note:** [BayLing-Duplex (arXiv 2606.14528)](https://arxiv.org/abs/2606.14528) names GLM-4-Voice as still **turn-based** because it relies on an external VAD module to mark the end of the user turn. The `fdMode: "native"` tag here reflects the original GLM-4-Voice claim of full-duplex token streaming; for taxonomy purposes treat it as VAD-dependent rather than VAD-free.

Moshi

A 7B speech-text foundation model that generates user and assistant audio streams in parallel.

Moshi ingests user audio and emits its own audio simultaneously through an RQ-Transformer stacked on the Mimi streaming codec. It can think, back-channel, and interrupt while the other side is still talking. Theoretical latency is 160 ms, ≈200 ms on an L4 GPU. **Signals W25 (Jun 2026):** the Kyutai team (Ohashi, Zeghidour, Défossez) published [Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models](https://arxiv.org/abs/2606.11167), arguing that token-level likelihood maximisation does not optimise interaction-level behaviours — i.e., Moshi-style pretraining alone produces excessive silence and ill-timed turn-taking, and a post-training alignment pass is the missing axis. Read as a forward signal on where the next Moshi iteration is likely to land.

OpenAI Realtime (gpt-realtime-2)

GA Realtime API flagship — gpt-realtime-2 is the first voice model with GPT-5-class reasoning, a 128k-token context, and five selectable reasoning levels (minimal → xhigh).

Announced 2026-05-07 alongside the Realtime API moving to GA. `gpt-realtime-2` succeeds the 2025-08-28-pinned `gpt-realtime` snapshot, expands the context window from 32k to 128k tokens (131,072), and exposes a `reasoning_effort` parameter with five tiers (minimal / low / medium / high / xhigh, default low). OpenAI's own scoreboard reports a 15.2 pp lift on Big Bench Audio over Realtime-1.5 and 13.8 pp on Audio MultiChallenge at the higher reasoning tier. Audio tokens are billed at $32 / 1M input, $64 / 1M output, with cached input at $0.40 / 1M. Sibling models `gpt-realtime-translate` (live S2ST, billed by the minute) and `gpt-realtime-whisper` (streaming STT) ship the same week. Transports: WebRTC, WebSocket, SIP; features include Cedar & Marin voices, remote MCP servers, image input, async function calling, parallel tool calls, and spoken preambles.

Gemini 3.1 Flash Live Preview

Current Google Live API target — preview snapshot with proactive-audio, native thinking, and tighter tool-call latency on top of 3.1 Flash.

Snapshot name: `gemini-3.1-flash-live-preview`. Preview-tier successor to Gemini 2.5 Live, rolled out as the recommended Live API target alongside the Gemini 3.1 Flash base model. Reported improvements include tighter first-response latency, proactive-audio triggering, and more stable tool / function-calling through long sessions. Still labelled preview, so naming and pricing may move before GA.

Gemini 2.5 Live

Legacy native-audio live API — still available but Google has announced gemini-3.1-flash-live-preview as the successor.

Legacy — vendor recommends migrating to Gemini 3.1 Flash Live Preview.

Gemini 2.5 Flash with native audio output went GA on Vertex AI in December 2025 (snapshot: `gemini-2.5-flash-native-audio-preview-12-2025`). A single model replaces the STT→LLM→TTS pipeline, with improved barge-in in noisy rooms and affective dialog that reacts to the speaker's emotion. Superseded by `gemini-3.1-flash-live-preview` in March 2026 — 2.5 remains available for backward compatibility but new integrations should target 3.1.

Amazon Nova 2 Sonic

Bedrock-hosted GA speech-to-speech model — polyglot voices, 1M-token context, async tool calls, telephony-ready.

Successor to Nova Sonic (Apr 2025); GA on Amazon Bedrock in Dec 2025 via model ID `amazon.nova-2-sonic-v1:0`. Adds Portuguese and Hindi, polyglot voices that keep identity across languages, configurable turn-taking (low / medium / high pause sensitivity), cross-modal input, asynchronous tool calling, and a 1M-token context window. Integrates with Amazon Connect, Vonage, Twilio, AudioCodes, LiveKit, and Pipecat. Artificial Analysis reports median TTFA ≈0.4 s and higher speech-reasoning accuracy than GPT-Realtime and Gemini 2.5 Flash Live on common benchmarks.

Grok Voice Agent API

OpenAI-Realtime-compatible voice agent API — 20+ languages, flat $0.05/min, ships with web/iOS testers and LiveKit plugin.

Launched 2025-12-17 as the developer-facing spin-out of the Grok Voice stack used in the xAI mobile apps and Tesla vehicles. xAI trained its own VAD, tokenizer, and audio models in-house, and exposes them via a WebSocket endpoint at `wss://api.x.ai/v1/realtime` that is wire-compatible with the OpenAI Realtime API. Reports ≈1 s average time-to-first-audio and #1 on Big Bench Audio as of launch. Supports web search / x_search / file_search / custom tools, LiveKit plugin, and ephemeral browser-side client secrets.

Hume EVI 3

An empathic speech-to-speech foundation model that infers user affect and tunes its tone to match.

First speech-LM to speak expressively with any voice, real or designed, without fine-tuning. Supports 200K+ custom voices, 30-second voice clones, and pluggable LLM backends (Claude 4, Gemini 2.5, Kimi K2, custom).

ElevenLabs Agents

The TTS incumbent's turnkey agent platform — turn-taking, RAG, and telephony pre-wired.

Conversational AI 2.0 (rebranded as ElevenLabs Agents) ships SOTA turn-taking that reads backchannels like "um" / "ah," multi-character voices in a single agent, RAG, HIPAA mode, and SIP trunking. 5,000+ voices across 31 languages.

Deepgram Voice Agent

STT+TTS incumbent's agent API using Nova-3 / Flux for transcription and Aura-2 for voice.

Pairs Deepgram's Nova-3 / Flux STT (with explicit turn detection) and Aura-2 TTS with pluggable LLMs including GPT-5.4 and Gemini. Flux's end-of-turn model is aimed squarely at sub-second back-and-forth latency.

Retell AI

Low-latency voice agent API built for call-centre workloads and SIP carriers.

Tight SIP and Twilio integration, with batteries-included call recording, post-call analytics, and outbound campaigns. One of the more battle-tested commercial stacks for production voice agents.

Bland

Turnkey voice-agent platform for inbound and outbound telephony with web agents and an API.

Phone-native voice-agent platform with inbound / outbound orchestration, web-embedded agents, pathways (node-based conversation logic), and a documented API. Often compared directly against Vapi, Retell, and Deepgram Voice Agent for production call deployments.

Vapi

Developer-first orchestration layer for inbound/outbound voice agents with BYO models.

Billed at $0.05/min of orchestration on top of your own STT/LLM/TTS/telephony keys. Two-tier building model — Assistants (single prompt) and Squads (multi-agent). Claims sub-500 ms infra latency at 99.99% uptime.

Kyutai Unmute

Open reference stack that gives any text LLM ears and a voice via delayed streams modelling.

Wraps a text LLM with Kyutai STT (1B EN/FR with 0.5 s delay or 2.6B EN with 2.5 s delay, both with semantic VAD) and Kyutai TTS 1.6B. Rust and Python implementations, Docker Compose deploy.

LiveKit Agents

Open agent framework on top of the LiveKit WebRTC mesh — 50+ provider plugins, first-class FD.

Since 1.5 ships an adaptive interruption ML model (86% precision vs VAD baselines), dynamic endpointing, preemptive generation, and per-turn latency metrics out of the box. Integrates with 50+ STT/LLM/TTS providers including gpt-realtime.

HF speech-to-speech

Modular reference pipeline that cascades VAD → ASR → LLM → TTS for local voice agents.

Hugging Face's reference cascaded stack (Silero VAD → Whisper / Lightning-Whisper-MLX → any HF LLM → Parler / MeloTTS / ChatTTS / Kokoro). Not full-duplex out of the box — it uses a turn-taking loop — but often the fastest way to prototype an open-source voice agent end to end. Apple-silicon optimised.

Pipecat

Python framework for voice agents with swappable ASR / LLM / TTS and sane defaults for turn-taking.

Wires a WebRTC transport (typically Daily) to a chain of ASR / LLM / TTS blocks. Increasingly the default pipeline for teams who want full control over every hop without writing their own media server.

Qwen3-Omni

Natively end-to-end omni-LLM (text / image / audio / video in, text + speech out).

MoE Thinker–Talker architecture with an Audio Transformer encoder pre-trained on 20M hours of speech. Claims SOTA on 22/36 audio/video benchmarks; ASR and voice chat on par with Gemini 2.5 Pro. Speech output in 10 languages.

Qwen2.5-Omni

End-to-end multimodal Qwen (text / image / audio / video in, text + speech out) with streaming response.

Legacy — vendor recommends migrating to Qwen3-Omni.

The 2.5-generation omni model, predecessor to Qwen3-Omni. Thinker–Talker split that interleaves text reasoning with streaming speech output. Listed here because it is still the reference for any paper comparing against the Qwen omni line.

Ultravox

Lightweight audio adapter that bolts onto Llama / Qwen to give an LLM ears without retraining.

Cross-attends a Whisper-class speech encoder into an existing text LLM. Popular when you want to bolt voice onto an in-house base model without running a separate ASR stage.

Qwen3-TTS

Open streaming TTS successor to CosyVoice with voice cloning, voice design, and 10-language coverage.

Part of the 2026 wave of open expressive TTS, alongside MOSS-TTS. Streams tokens to audio with voice cloning, prompt-based voice design, and cross-lingual speaker transfer. Positioned as the replacement for CosyVoice 2 in the Qwen ecosystem.

MOSS-TTS

Real-time streaming TTS family with voice cloning, multi-speaker dialogue, and sound generation.

Open TTS family from the OpenMOSS group at Fudan. Covers real-time streaming synthesis, zero-shot voice cloning, multi-speaker dialogue generation, and ambient sound generation in one codebase — modernises the open TTS drawer alongside Qwen3-TTS.

Cartesia Sonic 3

State-space TTS with ~90 ms time-to-first-audio and realistic non-verbal sounds.

WebSocket TTS with bidirectional streaming, context continuations for prosody, and laughter / breathing synthesis. Shipped by the Mamba / SSM team; the TTS of choice for sub-second voice agents.

CosyVoice 2

Open expressive TTS with strong zero-shot voice cloning across Japanese, Chinese, and English.

Half of Alibaba's open voice stack, paired with SenseVoice for ASR. The canonical pipeline is SenseVoice → LLM → CosyVoice. Strong zero-shot cloning with ~3 s of reference audio.

GPT-Realtime-Translate

Closed-API live speech-to-speech translation alongside the Realtime API GA — 70+ input languages into 13 output languages at $0.034 / minute.

Announced 2026-05-07 as part of the Realtime API GA wave. Live S2ST that keeps pace with the speaker; 70+ input languages to 13 output languages. Per-minute billing at $0.034 / min (translation) makes it the closed-commercial counterpart to Kyutai Hibiki on the simultaneous-interpretation axis flagged in Article 08 of the STS series. Sibling `gpt-realtime-whisper` covers streaming STT at $0.017 / min.

Hibiki

On-device high-fidelity simultaneous speech-to-speech translation — Moshi-style dual-stream for FR → EN, with voice transfer.

Hibiki reuses Moshi's multistream architecture to jointly model source and target audio, producing text + audio tokens at 12.5 Hz while the user is still speaking. Two checkpoints: Hibiki 2B (≈2.7B with the depth transformer, 16 RVQ per stream) and Hibiki-M 1B for on-device smartphone inference. Currently French → English only. Inference code for PyTorch (CUDA), Rust (CUDA), MLX (macOS), and MLX-Swift (iOS). Open-sourced under CC-BY-4.0 for weights.

SeamlessM4T v2

Direct speech-to-speech translation across 100+ languages, with weights in the open.

Meta shipped ExpressiveSeamless and SeamlessStreaming as companion drops for prosody transfer and streaming translation. Commercial use requires either a derivative model or a separate arrangement.