Models

Realtime-Venus

Two open 9B full-duplex models, one audio-visual and one speech-only, that keep talking while a background harness runs reasoning and tools.

Released 2026-09-16 under Apache-2.0: Realtime-Venus-Omni for audio-visual interaction and Realtime-Venus-Audio for spoken interaction, both 9B and adapted from MiniCPM-o 4.5. User inputs, model outputs, and delegation events share one causal timeline, and a dual-loop runtime lets Realtime-Venus-Harness run tasks asynchronously while the conversation continues. On Full-Duplex-Bench v1.5 the audio model responds to 75% of user interruptions and keeps going through 97%, 88%, and 86% of backchannels, other-directed speech, and background speech. The Omni model leads the compared online models on six of eight video benchmarks, including StreamingBench (70.2%).

NVIDIA NemotronLabs VoiceChat 11B

Open 11B full-duplex speech model on a Nemotron Nano v2 backbone, with a separate output channel for tool calls so it keeps talking while tools run.

The model card gives a release date of 2026-08-03 under the OpenMDW 1.1 licence. A FastConformer streaming encoder feeds a Nemotron Nano v2 9B backbone that predicts text tokens, a TTS decoder turns them into speech, and a separate output channel carries tool-call scripts, so a per-tool on-hold message can be spoken while the call runs. NVIDIA calls it the first open full-duplex model with tool calling and reports 448 ms smooth turn-taking latency on Full-Duplex-Bench v1. The card ranks it second among open models on Full-Duplex-Bench v1 and second among open full-duplex models on VoiceBench. Training used about 550k hours of real and synthetic speech; serving runs on vLLM and NVIDIA GPUs, English only. The September system report (arXiv 2609.21967) documents the design.

DuplexOmni

Open full-duplex omni model that listens and watches in 480 ms slices, answers in speech in real time, and hands hard questions to a pluggable thinking layer.

The DuplexOmni model is the interaction layer: it takes streaming audio and video in fixed 480 ms slices and produces text and speech every slice, while a pluggable thinking layer (Gemini-3.1-Flash-Lite in the paper's default setup) returns reasoning and tool results asynchronously. The paper's comparison table reports 0.506 s response latency (MiniCPM-o 4.5: 0.502 s; Gemini-3.1-Flash-Live: 2.57 s) and 77.2% on Big Bench Audio. Training dialogues are about 70% Chinese and 30% English, and the authors list limited video ability and weaker English speech as known gaps. Weights followed the June paper on 2026-07-24; the model card says it is still under active optimization and recommends at least 8 NVIDIA H20 GPUs for low latency.

TML-Interaction-Small

276B-A12B mixture-of-experts interaction model with VAD-free, time-aligned 200 ms micro-turns across audio, video, and text — Thinking Machines Lab's first public voice release.

Announced 2026-05-12 as Thinking Machines Lab's first public model. Encoder-free early-fusion design (dMel audio embeddings, hMLP video patches, flow-head audio decoder) ingests audio, video, and text as concurrent streams and decodes in 200 ms time-aligned micro-turns — replacing standard voice-activity detection with model-internal yield / self-correct / invite signals. Reports 0.40 s turn-taking latency on FD-Bench v1 and 77.8 / 100 interaction quality on FD-Bench v1.5, ahead of GPT-Realtime-2 and Gemini 3.1 Flash Live on conversational dynamics; trails GPT-Realtime-2 xhigh on pure intelligence (43.4 % vs 48.5 % on Audio MultiChallenge APR). Introduces four TML-internal benchmarks — TimeSpeak, CueSpeak, RepCount-A, ProactiveVideoQA — alongside published scores. Limited research preview in the coming months; wider release later in 2026. License and weights posture undisclosed. As of 2026-10, Thinking Machines' news page lists no wider release or public API for the interaction model; its July posts cover Inkling, an open-weights model that takes audio in but does not speak.

Raon-SpeechChat-9B

Open 9B English full-duplex speech model with a dedicated backchannel token, speak-first or listen-first modes, and voice conditioning.

Built on KRAFTON's Raon-Speech-9B and extended with full-duplex decoding, so it listens and speaks at the same time and handles barge-in. The stack combines a Qwen3 backbone, the Voxtral-Mini-4B-Realtime audio encoder, the Mimi codec, an ECAPA-TDNN speaker encoder for optional voice conditioning, and a Qwen3-based talker. A dedicated backchannel token produces short listener responses, with a penalty to tune how often. Runtime token forcing switches between speak-first and listen-first, and 17 built-in personas ship with it. Loads through Hugging Face Transformers with remote code; the self-hosted demo needs a 16 GB+ NVIDIA GPU. Weights are CC BY-NC 4.0.

Covo-Audio-Chat / Covo-Audio-Chat-FD

7B end-to-end audio LLM with a native full-duplex variant — tri-modal speech-text interleaving and dual-stream listen-and-speak.

Paper: 'Covo-Audio Technical Report' (arXiv 2602.09823, Feb 2026). Three releases share the same 7B backbone (Qwen2.5-7B + Whisper): Covo-Audio (base), Covo-Audio-Chat (spoken-dialogue variant, open-sourced), and Covo-Audio-Chat-FD (native low-latency full-duplex variant, benchmarked against Freeze-Omni / MiMo-Audio / Step-Audio 2 on URO-Bench). Model card labels weights as 'research and experimental purposes only,' so treat as non-commercial until a formal license is published.

MiniCPM-o 4.5

9B on-device omni-modal LLM with full-duplex speech + vision streaming, targeting Gemini 2.5 Flash quality on a laptop.

End-to-end multimodal model that processes continuous video + audio streams while generating text + speech in parallel — none of the four streams block each other. Uses a time-division-multiplex (TDM) full-duplex mechanism with an interleaved speech-token decoder. Ships with llama.cpp-omni / WebRTC demos for local deployment on a MacBook (M4 Max / 24 GB) or a small GPU. Covers 30+ languages via Qwen backbone.

PersonaPlex

7B full-duplex speech LM with hybrid voice + text prompts — any role, any voice, on top of Moshi.

Fine-tunes Kyutai's Moshi stack (Mimi codec @ 24 kHz, Helium LM backbone) with a hybrid conditioning path: a voice prompt captures timbre/style and a text prompt pins role, facts, and scenario. Trained on 1,217 h of Fisher English dyads plus ~2,250 h of synthetic assistant / customer-service dialogues rendered with Chatterbox TTS. NVIDIA reports ~170 ms average latency vs cascaded baselines on FullDuplexBench-style evaluations. Weights are hosted as `nvidia/personaplex-7b-v1` on Hugging Face; code is MIT, weights use the NVIDIA Open Model License.

SALMONN-omni

First standalone codec-free full-duplex speech LLM — no audio tokens in the vocabulary, RL-trained for barge-in.

Codec-free architecture: streaming speech encoder → LLM backbone (no codec tokens in its vocab) → streaming speech synthesizer, synchronised through embeddings and a dynamic 'thinking' mechanism. Reports a 30–36% relative improvement over Moshi, Freeze-Omni, Qwen2.5-Omni, GLM-4-Voice, VITA-1.5, MiniCPM-o, Kimi-Audio, Baichuan-Audio on open-domain spoken QA with substantially less training data. First FD speech LLM to apply RL to turn-taking / context-dependent barge-in. Published as NeurIPS 2025; weights released on the SALMONN org.

Step-Audio 2 mini

End-to-end multimodal audio LLM for speech conversation with expressive TTS and tool / search integration.

Open-weights release of the mini and mini-Base checkpoints (initialized from Qwen2-Audio / Qwen2.5-7B). Step-Audio 2 handles paralinguistic cues (laughter, sighs, accent, dialect), supports dynamic voice switching, and ships with a vLLM backend plus a hosted StepFun realtime console. The `mini Think` variant released in 2025-09 adds chain-of-thought reasoning for speech. Covered as a full-duplex baseline in the Covo-Audio and MiMo-Audio technical reports. StepFun's hosted realtime voice model is now StepAudio 3 Realtime, which is API-only and has its own card; no newer open checkpoint has followed the 2.x mini weights.

Kimi-Audio

A 7B audio foundation model covering ASR, AQA, audio QA, and end-to-end voice chat in one net.

Hybrid audio input (continuous acoustic vectors + discrete semantic tokens) feeds an LLM with parallel text and audio output heads. Pre-trained on 13M hours of audio. Ships with an evaluation toolkit, Kimi-Audio-Evalkit.

LLaMA-Omni 2

Qwen2.5-based real-time spoken chatbot series (0.5B–14B) with autoregressive streaming speech synthesis.

ACL 2025 main-conference paper. Speech encoder → Qwen2.5-0.5B/1.5B/3B/7B/14B/32B-Instruct backbone → autoregressive streaming speech decoder (CosyVoice 2 tokenizer + flow-matching vocoder). 7B variant reports total first-audio latency ≈580 ms with chunk size 3 / 10. Trained on only 200K multi-turn speech dialogues yet matches GLM-4-Voice for spoken QA. Turn-based rather than native full-duplex; interrupt / barge-in is not part of the released model. **Signals W25 positioning note:** [BayLing-Duplex (arXiv 2606.14528)](https://arxiv.org/abs/2606.14528) explicitly cites LLaMA-Omni and GLM-4-Voice as the two reference points for SpeechLMs that are still turn-based because they rely on an external VAD module. The `fdMode: "turn"` flag here was already correct; BayLing-Duplex is now the canonical citation for that classification.

Sesame CSM

The conversational speech model everyone called "uncanny-valley-free."

A Llama-backbone model with a smaller audio decoder that emits Mimi codec frames. The 1B checkpoint is public; larger 8B variant powers the hosted demo. Strong at laughs, sighs, and non-verbal cues.

GLM-4-Voice

An end-to-end Chinese/English speech LM that exposes tone, dialect, and emotion as instructions.

A three-part stack: a Whisper-based VQ tokenizer (12.5 tok/s), a 9B LM fine-tuned on speech tokens, and a Flow-Matching decoder that starts streaming audio after as few as 10 tokens. One of the few open speech-LMs with strong Chinese coverage. **Signals W25 positioning note:** [BayLing-Duplex (arXiv 2606.14528)](https://arxiv.org/abs/2606.14528) names GLM-4-Voice as still **turn-based** because it relies on an external VAD module to mark the end of the user turn. The `fdMode: "native"` tag here reflects the original GLM-4-Voice claim of full-duplex token streaming; for taxonomy purposes treat it as VAD-dependent rather than VAD-free.

Moshi

A 7B speech-text foundation model that generates user and assistant audio streams in parallel.

Moshi ingests user audio and emits its own audio simultaneously through an RQ-Transformer stacked on the Mimi streaming codec. It can think, back-channel, and interrupt while the other side is still talking. Theoretical latency is 160 ms, ≈200 ms on an L4 GPU. **Signals W25 (Jun 2026):** the Kyutai team (Ohashi, Zeghidour, Défossez) published [Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models](https://arxiv.org/abs/2606.11167), arguing that token-level likelihood maximisation does not optimise interaction-level behaviours — i.e., Moshi-style pretraining alone produces excessive silence and ill-timed turn-taking, and a post-training alignment pass is the missing axis. Read as a forward signal on where the next Moshi iteration is likely to land.

Amazon Nova 2.5 Sonic

Bedrock speech-to-speech model that succeeds Nova 2 Sonic at the same price, with claimed gains in reasoning, instruction following, and tool-calling accuracy.

Generally available in Amazon Bedrock on 2026-10-05 in US East (N. Virginia), US West (Oregon), Europe (Stockholm), and Asia Pacific (Tokyo), priced the same as Nova 2 Sonic. AWS claims better reasoning, instruction following, and tool-calling accuracy and "lower latency", but publishes no latency figure. It keeps expressive voices in seven languages (the announcement does not list them), controllable turn-taking, and asynchronous tool calling, with a 256K-token context window.

Grok Voice 3.0 (xAI Speech to Speech API)

xAI's speech-to-speech API, now on Grok Voice 3.0: 20+ languages, $0.08 per minute, wire-compatible with the OpenAI Realtime API.

Launched 2025-12-17 as the developer-facing spin-out of the Grok Voice stack used in the xAI mobile apps and Tesla vehicles. xAI trained its own VAD, tokenizer, and audio models in-house, and exposes them via a WebSocket endpoint at `wss://api.x.ai/v1/realtime` that is wire-compatible with the OpenAI Realtime API. Reports ≈1 s average time-to-first-audio and #1 on Big Bench Audio as of launch. Supports web search / x_search / file_search / custom tools, LiveKit plugin, and ephemeral browser-side client secrets. Grok Voice 2.0 (`grok-voice-2.0`, 2026-07-29) priced audio at $0.08 per minute, and Artificial Analysis measured 0.70 s time to first audio for it. Grok Voice 3.0 followed on 2026-10-14 at the same price. xAI describes its latency only as "under one second" and publishes no millisecond figure, so this card leaves the number blank. xAI reports 86.9 on the Artificial Analysis Speech-to-Speech Index, 98.5% on Big Bench Audio, 97.0% on Full Duplex Bench, and 71.6% on tau-Voice. The docs now list `grok-voice-3.0` as the flagship, point the `grok-voice-latest` alias at it, and mark `grok-voice-2.0` as deprecated; the endpoint, session parameters, and events are unchanged.

Eleven Realtime (v1 Mini / v1 Max)

ElevenLabs' speech-to-speech models, each a complete voice agent: Mini for low-cost conversation, Max for reasoning and multi-step tool use, behind an OpenAI-Realtime-compatible API.

Launched 2026-10-20 as `eleven_realtime_v1_mini` and `eleven_realtime_v1_max`. ElevenLabs calls them "a new class of Speech to Speech models" in which each model is a complete voice agent, but it has not disclosed whether the internals are a single model or a cascade, so the native fdMode here follows the vendor's framing. Both support more than 90 languages and any of the 17,000+ library voices. In Artificial Analysis testing cited by ElevenLabs, Max starts speaking in 0.57 s on average and Mini in 0.62 s on Big Bench Audio (98.8% and 75.9% correct), Mini scores 95.8% and Max 94.1% on Full-Duplex-Bench, and Max completes 87.1% of tool-use tasks against 60.3% for Mini. The models run inside ElevenAgents or through a Realtime API that follows the OpenAI Realtime spec over WebSocket only, separate from the ElevenAgents platform features.

Gemini 3.8 Live

Google's GA Live API voice model: native audio in and out, camera and screen frames as input, and tool calls that run in the background while it keeps talking.

Generally available in the Gemini API's Live API on 2026-09-15, together with Gemini 3.8 Live Extended Thinking, which adds configurable background reasoning; Google now recommends it over Gemini 3.1 Flash Live. Function calls are non-blocking by default, so the model keeps talking while tools run, and input covers text, images, audio, and video sent as frames at up to 1 fps. Audio costs $0.005 per minute in and $0.018 per minute out on the paid tier. Google's launch post claims 97+ languages, though its own docs give other counts, and it publishes no latency figure. The Live Avatar video output in Gemini Enterprise has its own card under full-duplex video.

StepAudio 3 Realtime

StepFun's full-duplex realtime voice model on its open platform: the user and the model can speak at once, and it thinks while it speaks.

StepFun's docs describe it as "a full-duplex voice model for real-time interaction" in which "the user and the model can speak at the same time", with interruption handling, adaptive reasoning, thinking while speaking, and Voice Agent tool calling. It is served through the Realtime API over a persistent WebSocket as `stepaudio-3-realtime-preview`, a free-trial name that StepFun says will be retired for a paid version when the trial ends. The technical report (arXiv 2609.14005, submitted 2026-09-12) describes it as an audio-language foundation model with an integrated voice agent. StepFun publishes no latency figure, language list, or weights.

SeedRealtime

ByteDance Seed's audio-visual full-duplex model: it watches a live camera feed, decides when to speak without an external VAD, and runs video calls in the Doubao app.

Announced 2026-08-05 as a native audio-visual full-duplex model that fuses audio, video, and text in one model and decides turn-taking itself rather than through an external VAD. It watches a continuous camera stream and can speak up unprompted when something changes in view. In ByteDance's human evaluation against cascaded systems it halved audio-visual pacing problems such as cutting the user off, slow replies, and false triggers from background chatter. The Chinese announcement says it is fully live in the Doubao app's video-call mode; there are no weights, technical report, latency figures, language list, or developer API.

Qwen-Audio Realtime (3.0 / 3.1 Plus)

Alibaba Cloud Model Studio's duplex speech models for voice conversation, served over WebSocket, AOQ, or WebRTC with system voices and voice cloning.

Model Studio added `qwen-audio-3.0-realtime-plus` and `qwen-audio-3.0-realtime-flash` on 2026-08-10, and `qwen-audio-3.1-realtime-plus` on 2026-09-20 with the same integration protocol, eight system voices, and voice cloning. Alibaba's release notes call 3.0 Plus a "next-generation duplex speech model" and Flash a low-latency variant; the user guide says the service converts real-time audio to speech and text "over a full-duplex connection". Alibaba has not disclosed whether the internals are a single model or a cascade, so the native fdMode here follows the vendor's framing. The guide lists 11 languages plus 21 Chinese varieties and gives no latency figure. A separate `qwen3.8-omni-flash-realtime` (2026-09-21) adds audio and video interaction. The open-weights Qwen line is covered by the Qwen3-Omni card.

OpenAI Realtime (gpt-realtime-2.1)

Realtime API reasoning voice model: gpt-realtime-2.1 updates gpt-realtime-2 with better alphanumeric recognition and steadier handling of silence, noise, and interruptions, plus a cheaper mini sibling.

Announced 2026-05-07 alongside the Realtime API moving to GA. `gpt-realtime-2` succeeds the 2025-08-28-pinned `gpt-realtime` snapshot, expands the context window from 32k to 128k tokens (131,072), and exposes a `reasoning_effort` parameter with five tiers (minimal / low / medium / high / xhigh, default low). OpenAI's own scoreboard reports a 15.2 pp lift on Big Bench Audio over Realtime-1.5 and 13.8 pp on Audio MultiChallenge at the higher reasoning tier. Audio tokens are billed at $32 / 1M input, $64 / 1M output, with cached input at $0.40 / 1M. Sibling models `gpt-realtime-translate` (live S2ST, billed by the minute) and `gpt-realtime-whisper` (streaming STT) ship the same week. Transports: WebRTC, WebSocket, SIP; features include Cedar & Marin voices, remote MCP servers, image input, async function calling, parallel tool calls, and spoken preambles. On 2026-07-06 OpenAI released `gpt-realtime-2.1` and `gpt-realtime-2.1-mini` on the Realtime API. The full model is an updated realtime reasoning model with better alphanumeric recognition and improved handling of silence, noise, and interruptions; the mini is a faster, lower-cost distilled version. Context stays at 128k tokens (32k max output) and audio pricing is unchanged at $32 / 1M input, $64 / 1M output, $0.40 / 1M cached input. As of 2026-10 the changelog lists no deprecation of `gpt-realtime-2`. OpenAI's separate full-duplex model is GPT-Live 1, which has its own card. The 310 ms first-response figure was measured on gpt-realtime-2; OpenAI has not published a comparable number for gpt-realtime-2.1.

GPT-Live 1

OpenAI's full-duplex voice model: it listens and talks at the same time and hands reasoning and tool calls to a backend model or agent.

Launched 2026-07-08 as the new ChatGPT Voice and generally available in the API on 2026-09-10 through the v1/live/sessions endpoint. One model listens and speaks at once, while reasoning and tool calls go to a backend: an OpenAI Responses model, or under client delegation any model or agent harness the developer runs; interrupting the speech leaves backend work running. Voice sessions cost $0.05 per minute billed per second, backend usage is billed separately, and concurrency runs from 25 sessions (Tier 1) to 500 (Tier 5). OpenAI reports 0.798 s turn-taking latency on Full-Duplex-Bench v1 (gpt-realtime-2.1: 1.41 s); the model takes no image or video input, so visual context has to go through the backend. On 2026-10-15 OpenAI added native Decisions integration in GPT-Live, and on 2026-10-16 it opened prompt-to-voice to all API customers (`v1/audio/voices`, used from `v1/live/sessions`) and added nine GPT-Live voices across German, French, Hindi, Japanese, and Korean. OpenAI still publishes no supported-language list for the model.

Gemini 3.1 Flash Live Preview

Preview Live API snapshot on 3.1 Flash with proactive audio, native thinking, and tighter tool-call latency; Google now recommends Gemini 3.8 Live instead.

Legacy. The vendor recommends migrating to Gemini 3.8 Live.

Snapshot name: `gemini-3.1-flash-live-preview`. Preview-tier successor to Gemini 2.5 Live, rolled out as the recommended Live API target alongside the Gemini 3.1 Flash base model. Reported improvements include tighter first-response latency, proactive-audio triggering, and more stable tool / function-calling through long sessions. As of 2026-10, Google labels it a legacy preview model and recommends Gemini 3.8 Live; no shutdown date has been announced.

Gemini 2.5 Live

Legacy native-audio Live API model, still available; Google now recommends Gemini 3.8 Live as the replacement.

Legacy. The vendor recommends migrating to Gemini 3.8 Live.

Gemini 2.5 Flash with native audio output went GA on Vertex AI in December 2025 (snapshot: `gemini-2.5-flash-native-audio-preview-12-2025`). A single model replaces the STT→LLM→TTS pipeline, with improved barge-in in noisy rooms and affective dialog that reacts to the speaker's emotion. Superseded by `gemini-3.1-flash-live-preview` in March 2026, and Google's deprecations table now lists Gemini 3.8 Live (GA 2026-09-15) as its replacement. 2.5 remains available for backward compatibility.

Amazon Nova 2 Sonic

Bedrock-hosted GA speech-to-speech model — polyglot voices, 1M-token context, async tool calls, telephony-ready.

Successor to Nova Sonic (Apr 2025); GA on Amazon Bedrock in Dec 2025 via model ID `amazon.nova-2-sonic-v1:0`. Adds Portuguese and Hindi, polyglot voices that keep identity across languages, configurable turn-taking (low / medium / high pause sensitivity), cross-modal input, asynchronous tool calling, and a 1M-token context window. Integrates with Amazon Connect, Vonage, Twilio, AudioCodes, LiveKit, and Pipecat. Artificial Analysis reports median TTFA ≈0.4 s and higher speech-reasoning accuracy than GPT-Realtime and Gemini 2.5 Flash Live on common benchmarks. Amazon Nova 2.5 Sonic became generally available in Bedrock on 2026-10-05 at the same price and has its own card; AWS has not announced a deprecation of Nova 2 Sonic.

Hume EVI 3

An empathic speech-to-speech foundation model that infers user affect and tunes its tone to match.

First speech-LM to speak expressively with any voice, real or designed, without fine-tuning. Supports 200K+ custom voices, 30-second voice clones, and pluggable LLM backends (Claude 4, Gemini 2.5, Kimi K2, custom). Hume's developer changelog (2026-10-02, "TTS and EVI API sunset notice") says access to the TTS and EVI APIs ends on 2026-11-13 at 12:01 a.m. EST. Both stay fully supported until then, and account data is permanently deleted afterwards. Hume gives no reason and no migration path; it points developers to its Real World VoiceEQ comparison for alternatives.

PolyAI Dialog-RSN-1

Audio-in, text-out dialog model that folds turn-taking, speech recognition, reasoning, and tool calls into one model and leaves speech output to a separate TTS.

Announced 2026-07-30. Dialog-RSN-1 perceives the user's audio directly and produces turn-taking decisions, reasoning, text replies, tool calls, and transcriptions in one model; speech is produced by a separate TTS so enterprises keep control of the voice, which is why this card files it as a stack. PolyAI built it by post-training open-weight multimodal models with supervised and reinforcement fine-tuning, and the first release focuses on English. PolyAI says it responds in under 300 ms in live deployments (vendor-reported, no independent measurement). Existing PolyAI customers can enable it now; the post also advertises the API with free credit while asking new customers to request early access.

AssemblyAI Voice Agent API

AssemblyAI's own speech understanding, LLM reasoning, and voice generation behind a single WebSocket, billed at $4.50 per hour.

Introduced 2026-04-29. Audio goes in and audio comes back over one streaming connection that AssemblyAI says is "built on our own models top to bottom", bundling speech understanding (Universal-3 Pro Streaming), LLM reasoning, and voice generation with turn detection and interruption handling. The product page calls it a "production speech-to-speech stack"; this card treats it as a managed stack because the components are separate stages behind one socket. Pricing is $4.50 per hour ($0.075 per minute) for the whole pipeline. AssemblyAI publishes no latency figure or language list for it; teams on LiveKit or Pipecat can still use its streaming STT on its own.

ElevenLabs Agents (ElevenAgents)

The TTS incumbent's turnkey agent platform — turn-taking, RAG, and telephony pre-wired.

Conversational AI 2.0 (rebranded as ElevenLabs Agents) ships SOTA turn-taking that reads backchannels like "um" / "ah," multi-character voices in a single agent, RAG, HIPAA mode, and SIP trunking. At that 2025 launch it offered 5,000+ voices across 31 languages. ElevenLabs now brands the platform ElevenAgents. Eleven v4 and Eleven v4 Turbo (2026-09-28) are available in it; v4 Turbo was optimized together with ElevenAgents, with about 100 ms median inference latency and about 150 ms median time to first speech, and both v4 models support more than 90 languages. From 2026-10-20 an agent can also run on the Eleven Realtime speech-to-speech models (separate card), which ElevenLabs recommends using through ElevenAgents for phone, web, WhatsApp, and CCaaS channels, workflows, a knowledge base, guardrails, testing, and monitoring.

Deepgram Voice Agent

STT+TTS incumbent's agent API using Nova-3 / Flux for transcription and Aura-2 for voice.

Pairs Deepgram's Nova-3 / Flux STT (with explicit turn detection) and Aura-2 TTS with pluggable LLMs including GPT-5.4 and Gemini. Flux's end-of-turn model is aimed squarely at sub-second back-and-forth latency.

Retell AI

Low-latency voice agent API built for call-centre workloads and SIP carriers.

Tight SIP and Twilio integration, with batteries-included call recording, post-call analytics, and outbound campaigns. One of the more battle-tested commercial stacks for production voice agents. Retell's 2026-08-24 changelog adds Retell Workflows (orchestration for work before a call connects and after it ends), makes GPT-5.6 and GPT Realtime 2.1 available in the agent builder, and extends Soniox Spanish speech recognition to every call.

Bland

Turnkey voice-agent platform for inbound and outbound telephony with web agents and an API.

Phone-native voice-agent platform with inbound / outbound orchestration, web-embedded agents, pathways (node-based conversation logic), and a documented API. Often compared directly against Vapi, Retell, and Deepgram Voice Agent for production call deployments.

Vapi

Developer-first orchestration layer for inbound/outbound voice agents with BYO models.

Billed at $0.05/min of orchestration on top of your own STT/LLM/TTS/telephony keys. Two-tier building model — Assistants (single prompt) and Squads (multi-agent). Claims sub-500 ms infra latency at 99.99% uptime. Vapi's release notes for the week of 2026-08-31 add outbound calling campaigns in the dashboard, a prompt editor, named transfer destinations, and tunable Soniox endpoint sensitivity and endpoint latency for finer turn-taking control.

Qwen Audio Agent

Apache-2.0 realtime voice runtime that keeps a conversation going while a backend agent runs tools and longer tasks.

Repository created 2026-07-27, with v2.0.0 released on 2026-09-23. A realtime voice frontend holds the conversation while a backend agent handles tool use and longer tasks in the background. Supported frontends include Qwen-Audio 3.0 and 3.1 Realtime, Qwen3.5-Omni Realtime, Qwen3.8 Omni Flash Realtime, GPT-Live and OpenAI Realtime, Gemini Live, Doubao Seeduplex 3.0, and StepAudio 3 Realtime in the cloud, plus Hugging Face speech-to-speech and MiniCPM-o 4.5 for local or self-hosted use; other services plug in through a Realtime Provider interface. Technical report: arXiv 2609.25195.

Kyutai Unmute

Open reference stack that gives any text LLM ears and a voice via delayed streams modelling.

Wraps a text LLM with Kyutai STT (1B EN/FR with 0.5 s delay or 2.6B EN with 2.5 s delay, both with semantic VAD) and Kyutai TTS 1.6B. Rust and Python implementations, Docker Compose deploy.

LiveKit Agents

Open agent framework on top of the LiveKit WebRTC mesh — 50+ provider plugins, first-class FD.

Since 1.5 ships an adaptive interruption ML model (86% precision vs VAD baselines), dynamic endpointing, preemptive generation, and per-turn latency metrics out of the box. Integrates with 50+ STT/LLM/TTS providers including gpt-realtime. The 1.8 line began with 1.8.0 (2026-09-05), which moved telemetry onto the OpenTelemetry GenAI semantic conventions and added in-process PII filtering. 1.8.8 (2026-10-15) runs GPT-Live through LiveKit Inference as `inference.DuplexModel` with LiveKit credentials instead of a separate OpenAI key, adds `google/gemini-3.8-live` and its Extended Thinking variant to Inference, and swaps Grok Voice Think Fast 2.0 for Grok Voice 3.0 in the model lists. Separately, LiveKit Agent Simulations became generally available for LiveKit Cloud projects on 2026-10-08: an LLM-driven caller plays scenarios and an LLM judge grades each transcript pass or fail, and audio mode adds background noise, interruptions, and backchannels for cascaded and full-duplex agents.

HF speech-to-speech

Modular reference pipeline that cascades VAD → ASR → LLM → TTS for local voice agents.

Hugging Face's reference cascaded stack (Silero VAD → Whisper / Lightning-Whisper-MLX → any HF LLM → Parler / MeloTTS / ChatTTS / Kokoro). Not full-duplex out of the box — it uses a turn-taking loop — but often the fastest way to prototype an open-source voice agent end to end. Apple-silicon optimised.

Pipecat

Python framework for voice agents with swappable ASR / LLM / TTS and sane defaults for turn-taking.

Wires a WebRTC transport (typically Daily) to a chain of ASR / LLM / TTS blocks. Increasingly the default pipeline for teams who want full control over every hop without writing their own media server. As of v1.13.0 (2026-10-19), `GeminiLiveLLMService` defaults to Gemini 3.8 Live, the Nova Sonic service defaults to Nova 2.5 Sonic, a new `AzureVoiceLiveLLMService` covers Azure's Voice Live speech-to-speech API, speech-to-speech services receive a tool's intermediate results, and an alpha `LLMWithBackend` lets a text or speech-to-speech frontend delegate deeper work to a backend LLM. The same release adds user and bot backchannel frames that stay out of the LLM context. v1.10.0 (2026-09-12) let Gradium's server-side end-of-turn signal replace pipeline VAD.

Qwen3-Omni

Natively end-to-end omni-LLM (text / image / audio / video in, text + speech out).

MoE Thinker–Talker architecture with an Audio Transformer encoder pre-trained on 20M hours of speech. Claims SOTA on 22/36 audio/video benchmarks; ASR and voice chat on par with Gemini 2.5 Pro. Speech output in 10 languages.

Qwen2.5-Omni

End-to-end multimodal Qwen (text / image / audio / video in, text + speech out) with streaming response.

Legacy. The vendor recommends migrating to Qwen3-Omni.

The 2.5-generation omni model, predecessor to Qwen3-Omni. Thinker–Talker split that interleaves text reasoning with streaming speech output. Listed here because it is still the reference for any paper comparing against the Qwen omni line.

Ultravox

Lightweight audio adapter that bolts onto Llama / Qwen to give an LLM ears without retraining.

Cross-attends a Whisper-class speech encoder into an existing text LLM. Popular when you want to bolt voice onto an in-house base model without running a separate ASR stage. Ultravox, formerly Fixie, now belongs to Inworld: Inworld's blog announced the acquisition on 2026-09-30 to advance realtime voice AI.

Gemini 3.8 Flash TTS / Flash-Lite TTS

Google's GA text-to-speech pair in the Gemini API: Flash TTS for studio-grade expressive voice, Flash-Lite TTS for high-volume and real-time voice-agent cascades.

Generally available in the Gemini API on 2026-09-22 as `gemini-3.8-flash-tts` and `gemini-3.8-flash-lite-tts`, together with the Voices endpoint (`/v1beta/voices`). Flash TTS is the flagship creative model, aimed at studio-grade fidelity, expressive acting, and long multi-turn stability. Flash-Lite TTS is the faster, lower-cost option for high-volume and real-time voice-agent use and is built to replace `gemini-3.1-flash-tts-preview`. The Voices endpoint adds voice design from text prompts, voice replication with consent verification, and a library of 150+ prebuilt and custom voices. The changelog gives no language count or latency figure.

Cartesia Sonic-3.6

State-space TTS with ~90 ms time-to-first-audio and realistic non-verbal sounds.

WebSocket TTS with bidirectional streaming, context continuations for prosody, and laughter / breathing synthesis. Shipped by the Mamba / SSM team; the TTS of choice for sub-second voice agents. Sonic-3.6, published 2026-08-27, is Cartesia's step up from Sonic-3.5 in naturalness and quality. Cartesia says first audio bytes arrive in about 80 ms (under 90 ms on the public API) and lists 44 languages.

Qwen3-TTS

Open streaming TTS successor to CosyVoice with voice cloning, voice design, and 10-language coverage.

Part of the 2026 wave of open expressive TTS, alongside MOSS-TTS. Streams tokens to audio with voice cloning, prompt-based voice design, and cross-lingual speaker transfer. Positioned as the replacement for CosyVoice 2 in the Qwen ecosystem.

MOSS-TTS

Real-time streaming TTS family with voice cloning, multi-speaker dialogue, and sound generation.

Open TTS family from the OpenMOSS group at Fudan. Covers real-time streaming synthesis, zero-shot voice cloning, multi-speaker dialogue generation, and ambient sound generation in one codebase — modernises the open TTS drawer alongside Qwen3-TTS.

CosyVoice 2

Open expressive TTS with strong zero-shot voice cloning across Japanese, Chinese, and English.

Half of Alibaba's open voice stack, paired with SenseVoice for ASR. The canonical pipeline is SenseVoice → LLM → CosyVoice. Strong zero-shot cloning with ~3 s of reference audio.

GPT-Realtime-Translate

Closed-API live speech-to-speech translation alongside the Realtime API GA — 70+ input languages into 13 output languages at $0.034 / minute.

Announced 2026-05-07 as part of the Realtime API GA wave. Live S2ST that keeps pace with the speaker; 70+ input languages to 13 output languages. Per-minute billing at $0.034 / min (translation) makes it the closed-commercial counterpart to Kyutai Hibiki on the simultaneous-interpretation axis flagged in Article 08 of the STS series. Sibling `gpt-realtime-whisper` covers streaming STT at $0.017 / min.

Hibiki

On-device high-fidelity simultaneous speech-to-speech translation — Moshi-style dual-stream for FR → EN, with voice transfer.

Hibiki reuses Moshi's multistream architecture to jointly model source and target audio, producing text + audio tokens at 12.5 Hz while the user is still speaking. Two checkpoints: Hibiki 2B (≈2.7B with the depth transformer, 16 RVQ per stream) and Hibiki-M 1B for on-device smartphone inference. Currently French → English only. Inference code for PyTorch (CUDA), Rust (CUDA), MLX (macOS), and MLX-Swift (iOS). Open-sourced under CC-BY-4.0 for weights.

SeamlessM4T v2

Direct speech-to-speech translation across 100+ languages, with weights in the open.

Meta shipped ExpressiveSeamless and SeamlessStreaming as companion drops for prosody transfer and streaming translation. Commercial use requires either a derivative model or a separate arrangement.

Griffin-Lite

Research preview of Tavus Griffin: one real-time model that watches and hears the user, decides every sub-second whether to speak, react, or wait, and renders its own voice and 720p face.

Announced 2026-10-01 as Tavus's first Human Interaction Model. A conversational engine reads the user's audio and camera and, every sub-second mini-turn, decides when and how to respond; streaming speech and video generators follow its signals, the video at 720p in 320 ms chunks with 0.43 s average audio-to-video latency on H100s (vendor-reported). On NVIDIA's VideoFDB leaderboard it is the top non-human system on both tracks, 3.83 / 5 on generation (human reference 3.92) and 3.73 / 5 on perception (human 4.20), but its median response latency there is 1,892 ms and 2,232 ms, against 900 ms and 1,400 ms for the human references. Tavus reports that 26 of 54 people (48%) took it for a real person after a one-minute call, against 1 of 41 for its previous stack. Selected testers only; no weights, paper, pricing, or language list.

Gemini 3.8 Live with Live Avatar

Gemini 3.8 Live sessions that return a 24 FPS avatar video stream alongside the model's own speech, generally available in Gemini Enterprise.

Available in Gemini Enterprise from 2026-09-24. Setting the Live API response modality to VIDEO with a prebuilt avatar makes the model stream 24 FPS video whose facial expressions and lip movements follow its own synthesized speech (Google Cloud docs). Google's model card lists the avatar as an optional native output of the same model and says sessions support "a few minutes of continuous interaction, rather than extended hours". Google documents no listening behaviour while the user talks, and the avatar has not been scored on VideoFDB. Audio and video carry SynthID watermarks.

Tavus CVI (Phoenix-4.5 / Sparrow-2 / Raven-1)

Tavus's production video-agent stack: camera perception, its own turn-taking model, STT, LLM, TTS, and a real-time face that keeps reacting while the user talks.

Tavus's Conversational Video Interface runs Raven (visual perception), Sparrow (turn-taking and interruptions), STT, an LLM, TTS, and Phoenix (the real-time face) over WebRTC; this entry reflects the stack as of Phoenix-4.5 (2026-09-10). Phoenix-4.5 animates from two audio streams, the agent's own speech and the user's, so the face and upper body keep moving while the user talks; Tavus reports 134 ms from audio to video. Sparrow-2 sits third on TurnBench's end-of-turn board at 0.930 recall, 0.084 false-positive rate, and 615 ms. In Tavus's one-minute study, 1 of 41 people took this stack for a real person, against 26 of 54 for Griffin-Lite.

AVTR-1

Open stack for real-time interactive avatars: a 153M-parameter motion model conditioned on both speakers' audio, so the face listens back while an external voice agent runs the turns.

An autoregressive flow-matching motion generator conditioned on both participants' audio, plus a renderer and a live-session streamer that keeps the video in sync with an external voice agent and handles interruptions. It renders at 25 fps in 5-frame (200 ms) chunks; the README lists 91 ms per chunk on an A100 and 232 ms on an RTX 4060, which falls short of real time. Turn-taking belongs to the voice agent. Licences are split: the weights allow commercial use for entities under USD 10M annual revenue, while the renderer and streamer are noncommercial only.

Runway Characters

Real-time video agent API on Runway's GWM-1 world model: one image becomes a conversational character that keeps expressing while it speaks and listens.

Launched 2026-03-09 on the avatar variant of GWM-1. Runway says characters keep "natural facial expressions, eye movements, lip-syncing and gestures during both speaking and listening", with a hosted voice agent running the conversation. Its engineering post reports 37 ms of model time per frame at more than 24 fps and 1.75 s of server-side turnaround from the end of the user's speech to the character's reply, voice agent plus video pipeline. Interruption handling is not documented.

Avatar Forcing

Research head avatar that takes the user's audio and head motion as live inputs and reacts to speech, nods, and laughter while it listens.

Uses diffusion forcing so the avatar consumes the user's audio and motion causally and reacts to verbal and non-verbal cues. The authors report about 500 ms interaction latency, a 6.8x speedup over their baseline, and over 80% preference against that baseline; expressive listening is learned with preference optimization whose losing samples drop the user's inputs, so no extra labels are needed. The public repo is a minimal inference pipeline (reference image, avatar audio, user video); real-time conversation demos are not included, and the avatar's own speech has to come from a separate voice agent.

HeyGen LiveAvatar (FULL mode)

HeyGen's real-time avatar service; in FULL mode HeyGen runs speech detection, the LLM, TTS, and turn management, and switches the avatar into a listening state.

Introduced in HeyGen's October 2025 product release. In FULL mode HeyGen runs the pipeline itself, with voice activity detection and turn management that "manages conversation flow automatically based on speech pauses and user interruptions". The avatar moves between idle, listening, and talking states through events such as avatar.start_listening, so its listening pose is a state change rather than a reaction to what the user says. LITE mode drops the pipeline and only renders a face for your own agent's audio, which puts it outside this section.