Benchmarks

• Sesame AI Labs / Mundo AI / CMU / otoupdated 2026-10verified 2026-10

  nativelab

  TurnBench

  Scores end-of-turn and interruption detection on 30 hours of hand-labelled dyadic conversation, with conversation type as a controlled variable.

  full-duplex

  Released 2026-08. The evaluation corpus is 154 studio-recorded dual-channel dialogues across 106 actors, balanced over six conversation types, triple-annotated at Fleiss's kappa 0.78 with ground truth taken from 2/3 consensus. Systems output timed end-of-turn and interruption events and are ranked on recall, with a false-positive budget of 0.15 and median latency reported alongside. Of the 14 systems benchmarked at release, none was simultaneously fast, high-recall and low on false positives; end-of-turn recall held steady across conversation types while interruption false positives concentrated in backchannel-dense talk. External submissions have since grown the board to 25 rows as of October 2026, led by OpenYap Turn. The paper was accepted to IEEE SLT 2026. For reference, humans in the corpus begin turn transfers a median 151 ms before the previous turn ends. Disclosure: Fullduplex is published by oto, which collected the training split and has two co-authors on the paper; the evaluation corpus is Mundo AI's and the scoring is Sesame's.

  top entry

  OpenYap Turn (external submission)0.978 recall, 0.077 FPR, 298 ms
  models scoredTavus CVI (Phoenix-4.5 / Sparrow-2 / Raven-1)
  end-of-turn recallinterruption recallfalse-positive ratemedian detection latency

  sitepapercodeleaderboard
• NVIDIA / David AIupdated 2026-10verified 2026-10

  nativelive

  VideoFDB

  Full-duplex benchmark for video calls: does the agent read the user's nonverbal cues, and do its own voice and face respond the way a person's would?

  full-duplex

  Released 2026-05 (arXiv v2 2026-07). 237 clips from real two-person video calls cover 11 nonverbal dynamics such as gaze avoidance, laughter, and backchannels, split into a perception track (reading the user) and a generation track (the agent's own audio and video); a language-model judge scores each on a 0 to 5 rubric, with TOR-Alignment and median latency for timing. The paper finds that speech-vision agents use video for explicit visual questions but not for continuous grounding, and that cascaded speech-to-avatar systems cannot produce full-duplex nonverbal cues. The data is gated under a non-commercial NVIDIA evaluation licence, the evaluation code is not yet released, and NVIDIA scores submissions sent to it.

  top entry

  Tavus Griffin Lite3.73 perception · 3.83 generation
  models scoredGemini 2.5 LiveGemini 3.1 Flash Live PreviewGriffin-LiteMiniCPM-o 4.5
  perception rubric (0 to 5)generation rubric (0 to 5)TOR-Alignmentmedian response latency

  sitepaperleaderboard
• Dong et al.updated 2026-10verified 2026-10

  nativelab

  DuplexAct

  Scores whether full-duplex speech models take the right interaction action when a conversation calls for it, and whether they follow through.

  full-duplex

  Released 2026-10. A three-level taxonomy covers eight interaction actions under 24 interaction conditions, and each action is scored twice: on immediate execution and on follow-through. The 4,691 test instances span casual conversation, meetings, customer service and interviews, and 2,531 of them are derived from real recordings. Evaluation code is on GitHub and the packaged test audio is on Hugging Face behind an access form. Not to be confused with DuplexAct-Bench (arXiv 2609.39446), a separate bilingual benchmark that is not listed here.

  immediate execution successfollow-through success

  sitepapercode
• Mathur and Manochaupdated 2026-10verified 2026-10

  nativelab

  MultiPartyTurnBench

  Floor control with two to five speakers: can a detector or full-duplex agent tell a real turn change from a hold, a backchannel, or speech that does not take the floor?

  full-duplex

  Released 2026-10. 240 synthetically generated, channel-separated conversations with two to five speakers across six interaction types, scored on end-of-turn and interruption detection with false-positive-constrained recall and signed latency, using TurnBench's scoring rule. Learned detectors reach 82.5 to 92.6% end-of-turn recall at 4.3 to 11.8% false-positive rates, while interruption detection either misses takeovers or fires too often. Five full-duplex agents reach 86.1 to 95.8% balanced accuracy on turn entry but only 46.9 to 54.7% on selective interruption handling. Code is MIT and the data is CC BY 4.0 on Hugging Face; TurnBench's dev split and AMI serve as real-audio checks. Disclosure: Fullduplex is published by oto, which supplied TurnBench's training split and has co-authors on the TurnBench paper; this benchmark builds on TurnBench.

  end-of-turn recall at fixed FPRinterruption recall at fixed FPRsigned latencyselective interruption balanced accuracy

  sitepapercode
• Sierra Researchupdated 2026-10verified 2026-10

  nativevertical

  τ-bench 2.0 (τ-Voice)

  The first agent benchmark with a live, full-duplex voice track, now in six languages with version 2.0.

  vertical-agentfull-duplexspeech-lm

  Extends τ-bench and τ²-bench into banking (698 docs, ~195 k tokens of policy) and knowledge retrieval, then adds τ-Voice: live full-duplex conversations with OpenAI / Gemini / xAI voice models under simulated noisy, accented conditions. Text-reasoning agents score ~85 %; the same models drop to 26–38 % once voice and real-time are introduced. Version 2.0 (released 2026-10-19) adds language packs for English, Spanish, Portuguese, Hindi, Chinese and Korean, a frozen 900-call voice-v2 benchmark, and τ-entity outbound calls that test exact entity capture; the multilingual extension is described in the τ-Multilingual paper (arXiv 2609.35820). Scores are not comparable with 1.x, which stays available under the v1.0.1 tag.

  top entry

  GPT-Live-1 (Astra backend, medium)79.9% on voice-v2
  models scoredGPT-Live 1
  pass^ktool-use successvoice task successper-language task completion

  sitepapercodeleaderboard
• Scale AI (Scale Labs)updated 2026-10verified 2026-10

  nativelab

  Audio MultiChallenge

  Multi-turn evaluation of spoken dialogue systems on natural human interaction — scores Inference Memory, Instruction Retention, Self Coherence, Voice Editing, Audio-Cue handling.

  speech-lmfull-duplex

  Covers the long-range context and self-coherence axes that are currently thin on the STS measurement layer. Strong complement to Talking Turns (turn-taking) and MTR-DuplexBench (multi-turn dynamics). 452 conversations from 47 speakers with 1,712 rubrics, built from human speech that keeps disfluencies. Scale Labs runs a live leaderboard with separate text-output and audio-output boards; as of 2026-10 the audio-output board is led by gpt-realtime-2 (xHigh) at 48.45%.

  top entry

  gemini-3.8-flash (high)60.4%
  models scoredOpenAI Realtime (gpt-realtime-2.1)TML-Interaction-Small
  Inference MemoryInstruction RetentionSelf CoherenceVoice EditingAudio-Cue

  sitepaperleaderboard
• Wang et al. (MultiTalk)updated 2026-09verified 2026-10

  nativelab

  MultiTalkBench

  Long, multi-party, English and Chinese full-duplex dialogue, with conversations that average 32.6 minutes.

  full-duplex

  Released 2026-09 with the MultiTalk paper (NeurIPS 2026). Built from real human recordings: 104 test samples (60 English, 44 Chinese) totalling about 56.5 hours, with probes for long-range entity tracking, topic coherence and addressee selection plus turn-taking metrics. The authors' bilingual Moshi-style model substantially outperforms Moshi, MiniCPM-o-4.5 and Qwen3-Omni-30B-A3B-Instruct on it. The test set is CC BY-SA 4.0 on Hugging Face and the evaluation pipeline (LLM judge, rubric, model adapters) is Apache-2.0 on GitHub; the paper also releases 57.6k hours of synthetic training data.

  long-range entity trackingtopic coherenceaddressee selectionturn-taking

  sitepapercode
• NTU (NVIDIA advisory)updated 2026-05verified 2026-10

  nativelive

  Full-Duplex-Bench v3

  Multi-step tool calling for voice agents, driven by real human speech with natural disfluencies.

  full-duplexvertical-agent

  Paper 2026-04, code and data 2026-05. 100 human-recorded spoken requests (79 scenarios, 12 speakers) across four domains, annotated for five disfluency categories such as self-corrections, ask the agent for one to three chained tool calls against mock APIs. Agents are streamed the audio through LiveKit and scored on tool-call accuracy, Pass@1, latency and turn-taking. In the paper, GPT-Realtime led Pass@1 at 0.600 and Gemini Live 3.1 was fastest at 4.25 s, while self-correction handling and multi-step reasoning on hard scenarios failed across all systems.

  models scoredGPT-Live 1
  Pass@1tool-call F1response latencyturn-take rate

  sitepapercode
• NTU / UC Berkeley / UW / MITupdated 2026-02verified 2026-10

  nativelive

  Full-Duplex-Bench v2

  Late-2025 WebRTC-based live harness — first FDB release that runs dynamic evaluation against streaming endpoints, plus multi-turn stimuli.

  full-duplexspeech-lm

  v2 moves the FDB family from fixed-file scoring to real-time streaming against the model under test. Adds multi-turn dialogue stimuli and a (subjective) first probe of instruction-following inside the FDB harness. Now the most-cited public full-duplex eval outside arenas. The paper appeared in 2025-10 and the WebRTC framework code was released in 2026-02.

  streaming latencymulti-turn coherenceoverlap handling

  sitepapercode
• ICASSP 2026 Spoken Dialogue Challengeupdated 2026-01verified 2026-04

  nativelab

  HumDial (ICASSP '26)

  Dual-track human-evaluated spoken dialogue challenge — first public benchmark to explicitly score paralinguistic input alongside turn-taking dynamics.

  full-duplexspeech-lmemotion

  Two tracks (Track I English; Track II Chinese + English) with a held-out human evaluation protocol. Sits at the intersection of full-duplex dynamics and paralinguistic understanding.

  turn-takingparalinguistic groundinghuman preference

  site
• Academic (FLEXI authors)updated 2025-11verified 2026-04

  nativelab

  FLEXI

  Full-duplex safety benchmark — scores whether a model correctly barges in on a user in safety-critical moments (medical, self-harm, imminent danger).

  full-duplex

  The only public eval targeted at proactive safety interrupts. Thin coverage across other axes, but unique: it forces full-duplex systems to demonstrate that they will speak up on their own when needed.

  safety-interrupt successfalse-alarm rate

  sitepaper
• Academic (MTR-DuplexBench authors)updated 2025-10verified 2026-04

  nativelab

  MTR-DuplexBench

  Multi-turn, multi-topic full-duplex benchmark built from recorded call-center-style dialogues.

  full-duplexspeech-lm

  Probes how full-duplex dynamics degrade across longer conversations and topic shifts — the regime where most turn-taking benchmarks stop. Currently the main public signal on long-range full-duplex behaviour.

  multi-turn coherencetopic-shift recoverydynamics retention

  sitepaper
• Academic (SID-Bench authors)updated 2025-09verified 2026-04

  nativelab

  SID-Bench

  Spoken Interaction Dynamics benchmark — millisecond-resolution ground truth for end-of-turn detection and interruption recovery.

  full-duplex

  Zooms in where FDB stays macro: SID-Bench scores reaction-time dynamics in full-duplex speech with millisecond-level labels for turn boundaries, barge-in timing, and interruption recovery. Useful as a paired eval alongside FDB v1 on the reaction-time axis.

  EOT latency (ms)barge-in timingrecovery time

  sitepaper
• Academic (MMedFD authors)updated 2025-09verified 2026-04

  adjacentvertical

  MMedFD

  Real-world healthcare benchmark for multi-turn full-duplex automatic speech recognition — adjacent to STS, crucial for duplex deployment.

  asrvertical-agentfull-duplex

  Not STS in the strict sense — it scores ASR under multi-turn full-duplex healthcare conversations — but the failure modes it exposes (overlap, disfluency, medical-term recall) are exactly the ones that break vertical voice deployments. Listed as `adjacent` because the STS community will quote it whenever duplex systems hit healthcare production.

  duplex WERmedical-term recalloverlap-region WER

  sitepaper
• NTU / UC Berkeley / UW / MITupdated 2025-08verified 2026-10

  nativelab

  Full-Duplex-Bench v1.5

  Mid-2025 refresh of FDB v1 — adds the simultaneous-speech / overlap axis (user interruption, listener back-channel, side conversation, ambient speech).

  full-duplex

  Same harness as v1, same automatic scoring, one new axis: what happens when both sides are producing audio at once. v1.5 is what most 2025 papers now cite when they claim 'full-duplex evaluated.' Artificial Analysis charts a subset of v1 and v1.5 as Conversational Dynamics on its speech-to-speech leaderboard; it dropped that score from its Speech to Speech Index in August 2026 but still publishes it.

  top entry

  StepAudio 3 Realtime98.9% (AA Conversational Dynamics, v1 and v1.5 subset)
  models scoredGPT-Live 1Realtime-VenusTML-Interaction-Small
  overlap durationmutual-silence rateinterruption recovery

  sitepapercodeleaderboard
• Academic (FD-Bench authors)updated 2025-07verified 2026-04

  nativelab

  FD-Bench

  Independent full-duplex benchmark focused on natural-pause / short-utterance regimes — a cross-check against FDB.

  full-duplex

  Different stimulus sources to FDB but overlapping metrics, which is the point: agreement / disagreement between FDB and FD-Bench has become a quick sanity signal for whether a model is overfit to one harness.

  pause handlingturn-takingoverlap

  sitepaper
• Apple / CMUupdated 2025-04verified 2026-10

  nativelab

  Talking Turns

  A supervised turn-taking judge trained on human-human conversations to score audio foundation models.

  full-duplexspeech-lm

  Trains a classifier to predict turn-taking events (end-of-turn, backchannel, interruption) in Switchboard and uses it as an automatic judge for spoken dialogue systems. The first study to systematically show that existing speech-LMs fail to understand when to speak and rarely back-channel. Published at ICLR 2025.

  models scoredGemini 2.5 LiveGemini 3.1 Flash Live PreviewMoshi
  turn-event F1backchannel rateinterruption rate

  sitepaper
• NTU / UC Berkeley / UW / MITupdated 2025-03verified 2026-10

  nativelab

  Full-Duplex-Bench v1

  Original Full-Duplex-Bench — automatic scoring of the four canonical turn-taking events: when-to-speak, back-channel, interruption success, pause handling.

  full-duplex

  Released 2025-03. Fixed audio stimuli + reproducible automatic metrics, establishing the first public benchmark dedicated to full-duplex interactive behaviour. Everything after (v1.5 / v2 / v3) is additive on top of this harness.

  models scoredGPT-Live 1MoshiNVIDIA NemotronLabs VoiceChat 11BOpenAI Realtime (gpt-realtime-2.1)
  turn-taking latencyback-channel timinginterruption successpause handling

  sitepapercode
• Japanese academic communityupdated 2024-11verified 2026-04

  nativelab

  J-Moshi (subjective)

  Japanese open-weights full-duplex model paper whose subjective MOS listening tests are the main public Japanese STS measurement layer today.

  full-duplexspeech-lm

  Not a held-out test set — just MOS scores on human-evaluated samples. Listed here because right now there is no shared automatic Japanese full-duplex benchmark, and J-Moshi's listener ratings are what the JA community points at.

  MOSlistener-preference

  sitepaper

• Boson AIupdated 2026-09verified 2026-10

  nativelab

  MSI-Bench

  Multi-speaker voice-agent benchmark: short multi-party audio scenes with participant context, expected tool calls and atomic rubrics, in English and Mandarin.

  speech-lmvertical-agent

  Released 2026-09. 1,152 test cases split evenly between Mandarin and English, each a short multi-party, multi-turn audio scene, covering multi-speaker memory, instruction following and reasoning. The strongest configuration passes every rubric on only 66.8% of English and 54.5% of Mandarin cases, and the strongest open-weight one on 34.0% and 19.3%. Open-weight models are held back by the multi-speaker audio front end, frontier systems still fail speaker-scoped decisions on clean transcripts, and models across the board often respond when nobody addressed them. Code is MIT and the data is CC BY 4.0 on Hugging Face.

  all-rubric pass ratemulti-speaker memorymulti-speaker instruction followingmulti-speaker reasoning

  sitepapercode
• Artificial Analysisupdated 2026-08verified 2026-10

  nativelab

  Artificial Analysis Speech to Speech Index

  Composite score for native speech-to-speech models: speech reasoning, agentic task completion, human preference and task success, weighted equally.

  speech-lmvertical-agent

  Launched 2026-06 and revised twice in 2026-08. Version 2.0 weights four components at 25% each: Speech Reasoning (Big Bench Audio), Agentic Performance (τ-Voice), Arena Preference (from the Speech Agent Arena) and Task Success Rate (also measured in the arena). Conversational Dynamics, a Full-Duplex-Bench v1 and v1.5 subset, was part of the index until version 2.0 and is still charted separately. Only models with results on all four components receive a score, and since version 2.0 no component scores turn-taking timing directly.

  top entry

  Grok Voice 3.086.9
  index scoreBig Bench Audio accuracyτ-Voice pass rateArena Scoretask success rate

  siteleaderboard
• Artificial Analysisupdated 2026-05verified 2026-10

  nativelab

  Big Bench Audio

  1 000 audio-question reasoning benchmark — the dataset behind the Artificial Analysis S2S speech-reasoning leaderboard.

  speech-lminstruction-following

  Adapts four Big Bench Hard categories (Formal Fallacies, Navigate, Object Counting, Web of Lies) into 250 spoken questions each, rendered with 23 top TTS voices. Used by Artificial Analysis to publish the public `Speech Reasoning` score across S2S providers and to compute Time-to-First-Audio. v1.1 (2026-03) switched to a Claude Sonnet 4.6 judge and includes unanswered questions in the score. v1.2 (2026-05) updated the judge model and grader harness to recognise correct final answers in verbose responses. Big Bench Audio is also one of four equal components of the Artificial Analysis Speech to Speech Index.

  top entry

  StepAudio 3 Realtime99.7%
  models scoredAmazon Nova 2 SonicDuplexOmniEleven Realtime (v1 Mini / v1 Max)Gemini 2.5 LiveGemini 3.1 Flash Live PreviewGrok Voice 3.0 (xAI Speech to Speech API)MoshiOpenAI Realtime (gpt-realtime-2.1)Qwen3-OmniStep-Audio 2 mini
  accuracyTTFA (s)

  siteleaderboard
• SJTU (OmniAgent)updated 2026-01verified 2026-10

  nativelab

  VocalBench

  A four-dimensional stress test for speech-interaction models — semantic, acoustic, conversational, and robust.

  speech-lmaudio-understanding

  ~24 k EN + ZH instances evaluating knowledge / reasoning (ACC), open-ended chat (LLM-judge), acoustic quality (UTMOS, WER), emotional empathy (EER), and robustness to noise, reverb, far-field, packet loss, and clipping. Tested on 27 mainstream speech-interaction models including GPT-4o Voice and Qwen-Audio.

  ACCLLM-judgeUTMOSWEREER

  sitepaper
• SJTU (OmniAgent)updated 2025-11verified 2026-04

  nativelab

  VocalBench-zh

  Mandarin version of VocalBench — 10 subsets, ~10k instances, 14 models evaluated at the Nov 2025 launch.

  speech-lmaudio-understanding

  The default Mandarin STS scoring target and a common co-citation with VoiceBench. Same four-dimensional design as the original VocalBench adapted to Chinese data.

  ACCLLM-judgeUTMOSWEREER

  sitepaper
• Academic (CS3-Bench authors)updated 2025-10verified 2026-04

  nativelab

  CS3-Bench

  Mandarin-English code-switching benchmark — headline finding: S2S models drop ~66% relative on code-switched inputs vs monolingual ones.

  speech-lm

  Closest thing the field has to a real code-switch axis for speech-LMs. Probes where most current systems silently fail — language mixing within an utterance — which the multilingual benchmarks above currently underweight.

  monolingual-vs-codeswitch deltaper-pair accuracy

  sitepaper
• Ruiqi Yan et al.updated 2025-08verified 2026-10

  nativelab

  URO-Bench

  The first end-to-end S2S benchmark with multilingual, multi-turn, and paralinguistic coverage.

  speech-lminstruction-followingemotion

  Forty datasets across twenty tasks in full Chinese-English pairings, split into a basic track and a pro track. Each track scores Understanding / Reasoning / Oral-conversation (URO) axes, exposing how open-source spoken dialogue models lag their backbone LLMs in instruction following, paralinguistics, and audio understanding. Findings of EMNLP 2025.

  models scoredCovo-Audio-Chat / Covo-Audio-Chat-FDStep-Audio 2 mini
  U/R/O compositeparalinguistic score

  sitepaper
• NUSupdated 2025-06verified 2026-10

  nativelab

  VoiceBench

  Evaluates LLM-based voice assistants across knowledge, reasoning, safety, and instruction-following — judged by a GPT-4-class model.

  speech-lminstruction-following

  Wraps nine text/spoken subtasks (AlpacaEval, CommonEval, WildVoice, OpenBookQA, MMSU, SD-QA, IFEval, BBH, AdvBench) around speech-in / text-out assistants, injecting speaker, environment, and content variations. The public leaderboard tracks 39+ systems split across cascaded, audio-LLM, omni, and S2S / full-duplex categories.

  models scoredCovo-Audio-Chat / Covo-Audio-Chat-FDKimi-AudioLLaMA-Omni 2NVIDIA NemotronLabs VoiceChat 11BQwen2.5-OmniQwen3-OmniStep-Audio 2 mini
  overallQAreasoning (BBH)IFEvalsafety

  sitepaper
• AudioLLMs (I2R Singapore)updated 2025-05verified 2026-10

  nativelab

  IFEval-Audio

  Format-constrained spoken instructions, verified programmatically — an audio port of text IFEval.

  instruction-followingspeech-lm

  280 audio-instruction-answer triples across six dimensions (content, capitalization, symbol, list, length, format), drawn from Spoken SQuAD, TED-LIUM 3, MuchoMusic, and others. Each answer is programmatically checked against its constraint, giving IFR (format), SCR (semantics), and OSR (both) scores. Published at IJCNLP-AACL 2025.

  IFRSCROSR

  sitepaper
• Alibaba DAMO (OFA-Sys)updated 2024-12verified 2026-10

  nativelab

  AIR-Bench

  A two-tier stress test for audio-LLMs covering speech, natural sound, and music — MCQ plus chat.

  audio-understandinginstruction-following

  Foundation benchmark runs ~19 k single-choice questions across 19 audio-understanding tasks; the chat benchmark scores ~2 k open-ended audio-instruction dialogues with a GPT-4 judge. Supports English and Chinese; published at ACL 2024.

  models scoredKimi-AudioQwen2.5-OmniQwen3-Omni
  MCQ accGPT-4 judge score

  sitepaper

• Ceccatelli, Jeon, Adelani et al.updated 2026-06verified 2026-06

  nativelab

  SpeechJBB

  Audio jailbreak benchmark for LALMs under multilingual and code-switched speech, with phonological perturbations as an augmented evaluation regime.

  audio-safetyspeech-lm

  Addresses the gap that LALM safety has been primarily evaluated on monolingual text-based harmful prompts. SpeechJBB benchmarks state-of-the-art LALMs on code-switched speech specifically, and introduces an augmented setting with phonological perturbations that probes whether safety alignment generalises across language mixing. Complements Acoustic Interference (paralinguistic priors) by isolating the code-switching attack vector.

  code-switched ASRphonological-perturbation successmultilingual alignment gap

  sitepaper
• Feng et al.updated 2026-05verified 2026-06

  nativelab

  LALM Jailbreak Taxonomy & Eval

  Unified taxonomy of LALM jailbreak attacks (semantic, acoustic, signal, internal-representation) with controlled empirical evaluation under a single threat model and cost-aware protocol.

  audio-safetyspeech-lm

  Organises prior LALM jailbreak work — including Acoustic Interference and SpeechJBB — into a single comparison grid, then runs controlled evaluation under a unified threat model and cost-aware metric. The cost-aware framing matters: jailbreaks succeeding at near-zero compute cost are a different production-risk object from ones that need thousands of dollars of optimisation. This is the consolidation paper that turns scattered audio-safety attacks into a comparable benchmark family.

  attack success ratecost-aware ASRdefense utilitycross-attack generalisation

  sitepaper
• Wang, Huang, Liang, Wu (CUHK · HKUST)updated 2026-05verified 2026-06

  nativelab

  AIA — Acoustic Interference Attack

  Universal-jailbreak benchmark for LALMs using Acoustic Latent Semantics — paralinguistic features intrinsic to the priors of audio generative models — that decouples attack payload from audio content.

  audio-safetyspeech-lm

  Demonstrates that LALM safety alignment can be compromised purely by Acoustic Latent Semantics, the paralinguistic features intrinsic to audio generative-model priors. The Acoustic Interference Attack uses a set of universal, instruction-neutral interference audio so that standard malicious text queries become jailbreak triggers without any content injection. The accompanying evaluation establishes the first published benchmark for this paralinguistic-prior attack class.

  universal jailbreak successalignment-evasion ratetransferability across LALMs

  sitepaper

• Artificial Analysisupdated 2026-10verified 2026-10

  adjacentarena

  Artificial Analysis Speech Arena

  tts-onlynot-full-duplex

  Blind user votes turn TTS models into a live Elo leaderboard — the de-facto preference number for TTS. Despite the 'Speech Arena' label, this is a TTS-only (text → speech) arena, not an STS ranking.

  tts

  TTS models ranked by Elo from blind A/B listening comparisons on the same prompt. Filterable by category (assistants, entertainment, customer service, knowledge sharing) and accent. As of October 2026 it is led by ElevenLabs' Eleven v4 Turbo and Eleven v4. Useful as a voice-quality proxy, but it does not evaluate conversation, turn-taking, or dialogue behaviour — do not read it as an STS benchmark.

  top entry

  Eleven v4 Turbo1335 Elo
  Elo

  siteleaderboard
• Artificial Analysisupdated 2026-10verified 2026-10

  nativearena

  Artificial Analysis Speech Agent Arena

  Human participants complete the same task with two hidden speech-to-speech systems over live calls, then vote for the conversation they preferred.

  speech-lmvertical-agent

  Launched 2026-08-24. Paid, screened participants receive one scenario, hold separate live voice calls with two hidden models, and record a blind overall preference plus diagnostic answers. There are 15 agentic scenarios that need tool calls, such as booking a dental check-up, and 20 non-agentic ones, such as asking about pool hours. Votes are fitted to a Bradley-Terry Elo with GPT Realtime 1.5 anchored at 1000, and a separate pipeline checks whether the required tool calls were made, reported as Task Success Rate. Providers' default cascaded voice-agent stacks are ranked alongside native models. Unlike the Artificial Analysis TTS arena, this one scores whole task conversations.

  top entry

  Gemini 3.1 Flash Live (Minimal)1095 Elo overall
  preference Eloagentic Elonon-agentic Elotask success rate

  siteleaderboard
• Hume AIupdated 2026-07verified 2026-10

  componentarena

  Real World Voice EQ Bench

  Human-rated leaderboards across TTS, speech-to-speech, speech understanding and ASR, built to test whether systems use what is in the audio and not only the words.

  ttsspeech-lmasremotion

  Released 2026-07. Raters score providers per factor on a 1 to 5 scale, with one leaderboard per modality; the ASR board uses WER. The paper finds that TTS naturalness, expressiveness, identity stability and reliability vary largely independently, that some speech-to-speech agents stay largely transcript-driven even with audio access, and that accented, emotional, noisy and conversational speech exposes ASR failures that clean-speech benchmarks miss. The leaderboard is a public Hugging Face Space, but the ratings dataset behind it is private and no evaluation code is released. Maintenance risk: Hume is sunsetting its TTS and EVI APIs on 2026-11-13, so whether the board keeps updating is an open question.

  rater score (1 to 5) per factorWER (ASR board)

  sitepaperleaderboard
• Scale AIupdated 2026-03verified 2026-10

  nativearena

  Scale Voice Showdown

  not-full-duplex

  First in-the-wild preference arena for voice AI, launched March 2026 with 11 frontier models, 60+ languages and real user speech. The public leaderboard is offline as of October 2026.

  speech-lmtts

  Intercepts <5% of real ChatLab conversations for blind head-to-head battles between two voice models, then continues the session with the winner. Eighty-one percent of prompts are open-ended conversational, and a third of battles happen in non-English (ES, AR, JA, PT, HI, FR). Modes currently evaluated are Dictate (speech-in / text-out) and S2S — *not* yet full-duplex live streaming, despite the arena framing. Treat scores as 'best voice assistant under turn-based play,' not as a full-duplex ranking. Offline as of 2026-10: the launch post and the leaderboard URL both return 404, scale.com/showdown redirects to the Scale Labs homepage, and the Scale Labs leaderboard index no longer lists the board.

  models scoredOpenAI Realtime (gpt-realtime-2.1)
  preference win-rateper-language score

  site
• Hugging Faceupdated 2026-03verified 2026-10

  adjacentarena

  TTS Arena

  tts-onlynot-full-duplex

  The original blind A/B voting space for TTS, feeding an Elo ladder on Hugging Face.

  tts

  Users blind-compare outputs from two random TTS systems reading the same prompt; votes feed an Elo score. Predates Artificial Analysis' arena and remains the open-research reference point for human-preference TTS scoring.

  Elo

  siteleaderboard
• FreedomIntelligence / CUHK-SZupdated 2025-09verified 2026-04

  nativearena

  MTalk-Bench

  First multi-turn S2S benchmark with both arena-style (pairwise) and rubric-based (absolute) protocols for dialogue evaluation.

  speech-lminstruction-followingemotion

  Covers three dimensions — Semantic, Paralinguistic, and Ambient Sound — over nine scenarios each, judged by humans and by LLM-as-a-judge. Unlike VoiceBench (single-turn) and S2S-Arena (no rubric), MTalk-Bench evaluates holistic conversational behaviour in audio-grounded multi-turn contexts. Code released under Apache-2.0; dataset under research license. Updated through 2025-09 with an expanded model pool (GPT-4o Voice, Kimi-Audio, Step-Audio, Qwen2.5-Omni, Moshi, and more).

  arena win-raterubric scoreper-dimension score

  sitepaper

• Meituan / Xbench / Agoraupdated 2025-10verified 2026-10

  nativevertical

  VoiceAgentEval

  A dual-dimensional benchmark for expert-level outbound-calling agents — interaction fluency vs task flow compliance.

  vertical-agentspeech-lm

  Covers six business domains and thirty sub-scenarios (recruitment, sales, CS, financial risk control, market research, proactive care). A large-model User Simulator combines five personalities × thirty scenarios to yield 150 evaluation dialogues. Separates General Interaction Capability (GIC) from Task Flow Compliance (TFC), exposing 'thoughtful listener' vs 'strict executor' trade-offs.

  GICTFCtask success

  sitepaper
• Academic (VoiceAgentBench authors)updated 2025-10verified 2026-04

  nativevertical

  VoiceAgentBench

  Spoken agentic benchmark — multi-tool workflows, multi-turn interactions, and safety under realistic voice-agent conditions; covers English and Hindi.

  vertical-agentspeech-lminstruction-following

  Complements τ³-bench's voice track: VoiceAgentBench focuses on ~tool-rich multi-step agent tasks in the voice-first setting, with explicit safety and multilingual (including Hindi) subsets. Fills a gap that VoiceBench / VocalBench leave at the 'does the assistant actually execute the task' end.

  task successtool-selection accuracymulti-turn stabilitysafety

  sitepaper
• OpenAIupdated 2025-05verified 2026-10

  adjacentvertical

  HealthBench

  text-only

  5,000 multi-turn health conversations × 48k physician-written rubric criteria — text-only, listed as a transferable foundation rubric rather than a voice benchmark.

  vertical-agentinstruction-following

  The most rigorous vertical dialogue eval to date, but *text-only*: conversations and rubrics are strings, so it scores an LLM's clinical reasoning rather than any speech or full-duplex behaviour. 262 physicians practicing in 60 countries wrote bespoke rubric criteria, weighted by clinical importance; responses are graded by GPT-4.1. Included here as a **transferable text-only benchmark** that any serious medical voice agent will ultimately need to pass on top of its speech stack — not as a native STS benchmark.

  rubric scoreper-theme breakdown

  sitepapercode

• SJTU / MBZUAI · NeurIPS 2025updated 2026-02verified 2026-04

  componentlab

  MMAR

  1 000 human-curated deep-reasoning audio QA items spanning speech, sound, music, and their mix — four hierarchical reasoning layers.

  audio-understandingspeech-lm

  Covers Signal / Perception / Semantic / Cultural reasoning layers, each annotated with chain-of-thought rationales. Harder than AudioBench and MMAU: no evaluated model approaches human performance as of publication. MMAR-Rubrics (2026-02) adds instance-level rubric scoring, and the Interspeech 2026 Audio Reasoning Challenge uses MMAR as its primary evaluation set. Dataset is CC-BY-NC-4.0; code is on GitHub under Apache-2.0.

  models scoredKimi-AudioQwen3-Omni
  overall accuracyper-layer accuracyrubric score

  sitepaper
• NVSpeech authorsupdated 2025-08verified 2026-10

  componentlab

  NV-Bench

  Standardised test for generating laughs, sighs, fillers, and back-channels in TTS.

  ttsemotion

  1,651 multilingual (English + Mandarin) utterances across 14 non-verbal vocalization categories — vegetative sounds, affect bursts, conversational grunts — each paired with human reference audio. Scores instruction alignment via PCER (paralinguistic character error rate) and acoustic fidelity via FAD, WavLM speaker similarity, and DNSMOS.

  PCERFADSIMDNSMOS

  sitepaper
• ByteDanceupdated 2024-11verified 2026-10

  componentlab

  Seed-TTS Eval

  The de-facto evaluation set for zero-shot voice cloning — WER, speaker similarity, and CMOS in one recipe.

  tts

  Defined in the Seed-TTS tech report and reused by most zero-shot TTS papers since. Combines automatic ASR-based word error, SECS speaker similarity against a reference clip, and CMOS listening tests to triangulate intelligibility, identity, and naturalness.

  WERSECSCMOS

  sitepaper
• NTU et al.updated 2024-11verified 2026-04

  componentlab

  Dynamic-SUPERB

  Instruction-tuned successor to SUPERB — ~55 tasks across speech understanding and light paralinguistic input.

  speech-lminstruction-following

  The baseline every new speech-LM paper still reports. Covers instruction-following over speech across classification, generation, and open-ended tasks; Phase-2 (2024-11) doubled the task pool.

  per-task accuracycomposite score

  sitepapercode
• Academic (VoiceAssistant-Eval authors)updated 2024-09verified 2026-04

  componentlab

  VoiceAssistant-Eval

  End-to-end voice assistant eval — instruction-following, multi-turn dialogue, and naturalness in one suite with GPT-4-class judges.

  speech-lminstruction-following

  Bundles open-ended task prompts with GPT-4-judge scoring on content, style, and multi-turn coherence. Useful as a second opinion alongside VoiceBench on the assistant-quality axis.

  judge scoremulti-turn coherencenaturalness

  sitepaper
• AudioLLMs (I2R Singapore)updated 2024-07verified 2026-04

  componentlab

  AudioBench

  Broad audio-understanding benchmark covering speech, music, and ambient sound reasoning.

  audio-understanding

  Good signal on audio-grounded reasoning across speech / sound / music, but does not score conversational dynamics. Complements AIR-Bench as a second audio-LLM reasoning reference.

  per-track accuracyoverall

  sitepapercode
• Academic (SD-Eval authors)updated 2024-06verified 2026-04

  componentlab

  SD-Eval

  Baseline probe on paralinguistic input — does a speech-LM use tone, emotion, and speaker traits at all?

  speech-lmemotion

  Four subsets (emotion, accent, age, environment) check whether the model's response is conditioned on paralinguistic features rather than transcript-only. A precondition every serious paralinguistic-output claim should first beat.

  paralinguistic-grounded accuracydelta over ASR baseline

  sitepaper
• Meta AIupdated 2023-12verified 2026-10

  componentlab

  SeamlessExpressive (mExpresso)

  Expressive speech-to-speech translation judged on translation, prosody, and vocal-style preservation.

  s2st

  Bundles the mExpresso benchmark (EN → FR / DE / ES / IT / ZH, seven styles including happy, sad, whisper, and laughing) with metrics that go beyond ASR-BLEU. AutoPCP scores phrase-level prosody correspondence and Vocal Style Similarity measures voice carry-over across languages.

  ASR-BLEUAutoPCPVocal Style Sim

  sitepaper
• Academic (ProsAudit authors)updated 2023-02verified 2026-04

  componentlab

  ProsAudit

  Prosodic boundary detection benchmark — can the model perceive phrase / sentence boundaries from audio alone?

  emotionaudio-understanding

  Old (2023) but still cited as a prerequisite: if the model cannot hear prosodic boundaries, downstream paralinguistic-output claims do not hold up.

  boundary F1delta over text-only

  sitepaper
• Meta AIupdated 2020-07verified 2026-10

  componentlab

  CoVoST-2 BLEU

  21 language pairs of speech-to-text translation, long the S2T standard.

  s2stasr

  Built on Common Voice, CoVoST-2 pairs read speech with translated text across 21 language pairs (EN⇄X and X⇄EN). It is the reference workload for cross-model S2T benchmarking used by SeamlessM4T, Whisper, USM, and friends.

  BLEUchrF

  sitepaper

Nothing here yet.

• NTU et al. (s3prl)updated 2024-06verified 2026-10

  legacylab

  SUPERB

  The classic: general-purpose speech representations scored on ten frozen-encoder downstream tasks.

  speech-lmasraudio-understanding

  Ten-task evaluation suite (ASR, PR, KS, SID, IC, SF, ER, QbE, SD, VC) for frozen self-supervised speech encoders, run through the s3prl toolkit introduced at Interspeech 2021. The Dynamic-SUPERB fork extends the protocol to instruction-following for speech-LMs. Listed here for context — modern speech-LMs are no longer ranked on SUPERB.

  composite scoreper-task scores

  sitepaper
• OpenSLRupdated 2024-01verified 2026-10

  legacylab

  LibriSpeech WER

  The coordinate system of English ASR — test-clean and test-other WER.

  asr

  Evaluation splits of the 960-hour LibriSpeech read-speech corpus distributed through OpenSLR. Not conversational and long-since saturated, but still the most cited fitness test for English ASR and the reference number every new speech encoder reports.

  WER (test-clean)WER (test-other)

  site
• Academicupdated 2023-06verified 2026-10

  legacylab

  MELD / IEMOCAP

  The classic emotion-recognition pair, evaluated with full dialogue context.

  emotionaudio-understanding

  MELD is built from Friends with 7-way utterance-level emotion labels; IEMOCAP uses scripted-plus-improvised dyadic sessions with 6 emotion classes. Both remain standard baselines for emotional ASR and conversational emotion recognition, though modern speech-LMs are rarely ranked on them.

  Weighted F1Accuracy

  site