Full-Duplex-Bench, the NTU-led turn-taking benchmark that voice AI vendors now cite

Full-Duplex-Bench started in March 2025 as a paper from National Taiwan University and collaborators, with test clips and scripts that score when a voice model speaks. Eighteen months later, OpenAI reported results on it in a launch post. There are now four versions, and the same model can score very differently depending on who runs the test.

Nov 202614 min readMarkdown

1. The benchmark in vendor launch posts

On September 10, 2026, OpenAI made GPT-Live-1, its full-duplex voice model, available in its API. The launch post carried charts from three versions of a benchmark OpenAI did not build. On Full-Duplex-Bench v1, OpenAI reported a turn-taking latency of 0.798 seconds for GPT-Live-1, against 1.41 seconds for gpt-realtime-2.1 and 1.63 seconds for gpt-realtime-2. The same post put GPT-Live-1 at 80.1% on a chart it titled “Full Duplex Bench v1.5 Interactivity” and at 87.0% Pass@1 on “Full Duplex Bench v3 Tool calling”. OpenAI ran these evaluations on its own models.

Two other vendors used the name at second hand. On July 29, 2026, xAI’s Grok Voice 2.0 launch post listed a “Full Duplex Bench” row under “Conversational Dynamics” in a comparison it credited to Artificial Analysis. On October 20, 2026, ElevenLabs’ Eleven Realtime launch named “Full-Duplex-Bench for conversational dynamics” among three public benchmarks Artificial Analysis ran on its models. Both results came through Artificial Analysis, whose Conversational Dynamics measure is built on a subset of Full-Duplex-Bench v1 and v1.5.

We did not expect a test from four academic papers to travel this far in eighteen months. Behind the name are four arXiv papers, a GitHub repository opened on February 17, 2025, and a set of audio clips with scripts that score what a model does with them. No company runs it, and there is no hosted leaderboard. Because the clips and scripts are public and small, labs run the test themselves, and many have. Guan-Ting Lin of National Taiwan University (NTU) is the first author on all four papers (on version 2 he shares first authorship with Shih-Yun Shan Kuan), and Hung-yi Lee, also of NTU, is the last author on all four.

2. The authors and their institutions

In its current version, the version 1 paper (posted March 6, 2025, and accepted at ASRU 2025) lists seven authors: Guan-Ting Lin of NTU’s Graduate Institute of Communication Engineering; Jiachen Lian, Tingle Li and Gopala Anumanchipalli of UC Berkeley; Qirui Wang of the University of Washington; Alexander H. Liu of MIT CSAIL; and Hung-yi Lee of NTU. Lian, Li and Wang are marked as equal contributors.

The version 1.5 paper (posted July 30, 2025, and accepted at ICASSP 2026) kept Lin, Wang, Lian, Li and Lee and added Shih-Yun Shan Kuan of NTU; Shinji Watanabe of Carnegie Mellon University joined in a September 2025 revision. The version 2 paper (posted October 9, 2025, and accepted at ACL 2026) brought in Jiatong Shi and Siddhant Arora of Carnegie Mellon and Kai-Wei Chang of MIT alongside Lin, Kuan, Watanabe and Lee. Lee’s second affiliation, the NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), first appears in the April 2026 revisions of versions 1.5 and 2, where the version 2 acknowledgments also credit support from Taiwan’s Ministry of Education through it.

The version 3 paper (posted April 6, 2026, thirteen months after the first) has the smallest team: Lin and Lee of NTU, with Chen Chen and Zhehuai Chen of NVIDIA, the only industry co-authors in the series. Its arXiv comment still reads “Work in progress”.

Version 1 set itself against Talking Turns, an earlier turn-taking benchmark that relies on a judge model trained on one dialogue dataset and on user studies, which the version 1 authors said makes results hard to reproduce. Two of the Talking Turns authors later joined the series: Shinji Watanabe from the September 2025 revision of version 1.5, and Siddhant Arora on version 2.

3. What version 1 measures

The four versions ask different questions of a model that listens while it speaks.

Version 1 tests four moments in a conversation. When the user pauses mid-turn to think, the model should stay quiet (pause handling). While the user holds the floor, it should offer short acknowledgements such as “mm-hmm” where people would (backchanneling). When the user finishes, it should answer promptly (smooth turn-taking), and when the user cuts in, it should stop and take up the new request (user interruption).

Most of the scoring rests on one yes-or-no event, the takeover. The paper counts any model speech that is neither silence nor a backchannel as a takeover, and treats a segment as a backchannel if it lasts under one second and has fewer than two words. The takeover rate (TOR) is the share of samples with a takeover. Lower is better for pauses and backchannels, where the model should hold back, and higher is better for turn-taking and interruptions, where it should speak. Latency is averaged only over samples where the model did take over, so silent samples do not distort it.

We suspect the takeover rate is a large part of why version 1 caught on. One yes-or-no event per clip is cheap to compute and easy to explain. Its limit is that it cannot say whether a takeover was well judged; the interruption test needs a GPT-4o rating on top for that.

In the current version the clips come from three places. Pause and turn-taking samples are cut from Candor, an 850-hour corpus of two-channel spontaneous conversation: pauses of 0.4 to 1.0 seconds inside a turn, and turn changes with gaps under 0.4 seconds, checked by hand after automatic filtering. Backchannel timing comes from the In Conversation Corpus, where 118 native speakers responded wherever they found it appropriate; the paper turns their responses into a timing distribution over 200 ms windows and compares the model’s backchannels with it using Jensen-Shannon divergence. The interruptions are synthetic dialogues written with GPT-4o and voiced with ChatTTS in ten voices, and a second set of pauses is inserted with a ChatTTS pause tag. NVIDIA’s Parakeet TDT 0.6B v2 transcribes the model’s output with word timings, and GPT-4o rates each post-interruption answer from 0 to 5.

The March 2025 version tested only three open systems (dGSLM, Moshi and Freeze-Omni), on a larger Candor sample of 897 pause and 539 turn-taking clips, with pauses of 0.6 to 1.5 seconds and CrisperWhisper for transcription. The June 2025 revision moved to the current clip set, switched to Parakeet (the repository logs the change on June 5) and added Google’s Gemini Live (gemini-2.0-flash-live-001).

In the revised paper, the two end-to-end research models took over at nearly every pause. Moshi’s takeover rate on synthetic pauses was 0.985 and dGSLM’s 0.934. Gemini Live had the lowest, 0.255, yet took its turn in only 65.5% of the real turn-taking samples; the authors read that as a sign that even commercial models sometimes fail to take the turn. Moshi was quickest to answer once a turn ended, at 0.265 seconds on average, against 1.301 seconds for Gemini Live.

The August 2025 revision calls the metrics “intentionally descriptive rather than prescriptive” and leaves developers to decide which behaviours matter for their application, although its results table still marks which direction is better. It names two limits: the analysis covers English only, and the metrics are not yet linked to human preferences.

4. Overlap in version 1.5, conversations in version 2

Version 1.5

Version 1.5 tests what a model does when the user’s audio arrives while it is already talking. There are four scenarios: a user interruption (200 clips), a user backchannel such as “uh-huh” (99), the user talking to someone else (100) and background speech in another voice (100). The last two are processed to sound away from the microphone. Speech aimed at someone else is turned down 8 dB with early reflections; background speech is turned down 15 dB, low-pass filtered at 3 kHz and given an echo.

GPT-4o reads the transcripts and labels what the model did after the overlap: respond to it, resume what it was saying, express uncertainty, or something else. Silero VAD supplies two times. Stop latency runs from the start of the overlap until the model falls silent, and response latency from the end of the overlap until it speaks again. The paper also tracks pitch, loudness, speaking rate and a predicted quality score (UTMOSv2) before and after. A model should respond to an interruption and keep talking through the other three.

The paper tested five systems, Freeze-Omni, Moshi, Gemini (gemini-2.0-flash-live-001), Amazon Nova Sonic and GPT-4o Realtime (the December 17, 2024 preview), and found two strategies. GPT-4o Realtime was the most responsive. It responded to 78% of interruptions and stopped within 0.23 seconds on average, but it also responded to 91% of speech aimed at someone else and 93% of background speech. Nova Sonic and Gemini held the floor. Sonic resumed after 98% of backchannels and 98% of background speech, yet on real interruptions Gemini kept going 55% of the time and Sonic 71%, with stop latencies above two seconds.

Neither strategy wins outright, and we think that is the most useful result in the paper. GPT-4o Realtime handles real interruptions well and misfires on side talk; Sonic and Gemini do the reverse. Which failure costs more depends on where the model is deployed.

Version 2

Version 2 replaces pre-recorded clips with live conversation. An automated examiner, itself a speech model, talks to the system under test through an orchestrator that streams audio both ways over WebRTC as 48 kHz, 16-bit mono PCM in 10 ms frames. Each scenario has staged goals, and the examiner moves on only when the current one is met. With fast pacing it may interrupt or backchannel; with slow pacing it waits for the model to finish. The four task families are daily tasks, corrections (the user changes an earlier request), entity tracking (the user refers back to options by their attributes) and safety across 11 policy classes. Gemini 2.5 Flash scores the transcripts from 1 to 5 for turn-taking fluency, instruction following and a task-specific score.

The paper tested GPT-Realtime, Moshi and Freeze-Omni, and all three got worse as conversations went on. Under fast pacing GPT-Realtime scored above 4.0 on correction, entity tracking and safety, while Moshi and Freeze-Omni stayed below 3.0 on correction and entity tracking. Slow pacing helped the open models most; Moshi’s entity-tracking score rose by about 1.1. The April 2026 revision added a check on the judge: people scored 120 sessions, and their agreement with the LLM judge was moderate to strong, with Pearson’s r between 0.59 and 0.69.

The examiner is OpenAI’s gpt-realtime, so in this paper an OpenAI model both takes the test and administers it. The authors say they picked it for stability and instruction following, and that any model can replace it through the same adapter interface. We would like to see the results rerun with a different examiner. Every score in the paper comes from conversations an OpenAI model steered, and GPT-Realtime had the best task-specific score in every case the paper reports.

5. Tool calls from real, disfluent speech in version 3

Version 3 asks whether an agent acts correctly on what it hears. The paper’s example is a user who asks for a flight to New York and then corrects it to Boston; the agent has to drop the first destination before it calls the booking API. The paper went up on April 6, 2026, and the code and data followed in the repository on May 20, 2026.

It is the first version recorded entirely from people: 100 recordings from 12 speakers, native and non-native English speakers with Korean and Russian backgrounds, made mostly on everyday built-in microphones (11 of 12 setups) in rooms that ranged from quiet to mildly noisy. Each recording is annotated for five kinds of disfluency (fillers, pauses, hesitations, false starts and self-corrections), and 21 scenarios turn on a self-correction. The tasks cover four domains: travel and identity, finance and billing, housing and location, and e-commerce support. They call 11 mock functions that return fixed answers instantly, so that, the authors say, measured latency reflects the model’s processing rather than the tools.

Every agent runs through LiveKit. The scores are tool-selection F1, argument accuracy judged by GPT-4o, Pass@1 (the exact set of tools, with every argument right), response quality, a turn-take rate, an interruption rate (the agent starts before the user has finished) and latency, measured as the time until the agent delivers the answer.

GPT-Realtime was the most accurate, with Pass@1 of 0.600, and had the lowest interruption rate, 13.5%. Gemini Live 3.1 delivered answers fastest, in 4.25 seconds, but gave no response in 22 scenarios, a turn-take rate of 78.0%. The cascade of Whisper, GPT-4o and OpenAI TTS answered every time and was the slowest, at 10.12 seconds. Ultravox spoke over users in 47.9% of cases and opened 88.0% of its answers with a content-free filler sentence. The paper names self-correction handling as one of the most consistent failures: GPT-Realtime passed 58.8% of those scenarios, more than any other system, and the cascade 17.6%.

Of the four versions, this is the one we would watch most closely. A voice agent that books the wrong flight does more harm than one that pauses awkwardly, and version 3 is the only version that checks the booking. It is also the least settled, with 100 recordings and a paper still marked work in progress.

OpenAI’s September chart for tool calling also used version 3. With GPT-Live-1 running a Terra backend at low reasoning effort, it put gpt-realtime-2.1 at 60.0% Pass@1. That matches the paper’s figure for its GPT-Realtime entry, but the two are different models measured in different runs.

6. Who runs it, and why their numbers differ

Model papers

Within a year, model papers were reporting FDB results. NVIDIA’s PersonaPlex paper extended it beyond a single assistant role to multi-role customer service. SB Intuitions and the University of Tokyo reported DuplexCascade on it in March 2026. An interactivity alignment paper from researchers at Kyutai and Gradium (June 2026) trains Moshi and PersonaPlex with rewards for the four behaviours version 1 defined and evaluates on versions 1 and 2. NVIDIA’s NemotronLabs VoiceChat report (September 2026) gives results on versions 1, 1.5 and 3, and Alibaba’s Qwen-Audio-3.1-Realtime report (September 2026) says its response rate to background speech on version 1.5 fell from 73.0% to 13.0% against the previous release.

IndicFDB, from NVIDIA and VoiceArena (September 25, 2026), extends the test itself. It keeps the four version 1 categories and covers ten languages spoken in India with 12,350 samples, which its authors describe as nearly 17 times as many as the original.

Single scores

Artificial Analysis built its Conversational Dynamics measure on a subset of versions 1 and 1.5. It scores pause handling, turn-taking, interruption handling and backchannel handling as the percentage of samples handled correctly. The measure made up 33.3% of version 1.0 of its Speech to Speech Index (June to August 2026) and 25% of version 1.1; version 2.0, in August 2026, replaced it with Task Success Rate.

Turning the metrics into one percentage goes further than the papers do. The version 1 authors call their metrics descriptive, though their table marks which direction is better, and the version 1.5 paper names an expected behaviour for each scenario. A percentage correct fixes one right outcome per sample and drops the spread of behaviours the papers report. OpenAI’s “Interactivity” chart does the same for version 1.5, whose paper gives a distribution of behaviours for each scenario rather than a single score. The launch post describes that chart in one line, as testing reactions to background speech, speech aimed at others, backchannels and interruptions.

Differences between runs

Because anyone can run the scripts, the same model gets measured again and again. Moshi’s takeover rate on version 1’s synthetic pauses was 1.000 in the benchmark’s first posting and is 0.985 in the revised paper, 0.445 in the Kyutai and Gradium paper and 0.299 in the Raon-Speech technical report.

The papers record several differences between these runs. The authors revised the benchmark in June 2025, moving to a new clip set and to Parakeet TDT 0.6B v2 for transcription, and their own Moshi figure went from 1.000 to 0.985. Other teams change the protocol. For Moshi, the Kyutai and Gradium paper prepends 3 seconds of silence to the input so the model can say its opening phrase before the user speaks, following earlier work, and Raon’s team reproduced the version 1 and 1.5 scores with its own evaluator.

The scoring code has quirks of its own. The paper counts a backchannel as under one second and fewer than two words, while the public scoring scripts allow up to three words. The Kyutai and Gradium paper also reports that the official backchannel script did not resample generated audio to the 16 kHz its voice detector expects, and it leaves out another paper’s backchannel scores that predate the fix. The repository pinned the sampling rate in a commit on May 10, 2026. That fix touches backchannel scoring only, so it does not explain the pause numbers above. Commercial endpoints change too: the version 1 README now marks its Gemini Live 2.0 row as deprecated because the API changed, and adds rows for GPT-Realtime, Gemini 3.1 Flash Live at two thinking levels, and PersonaPlex.

No leaderboard accepts submissions. The version 1 README keeps a results table that the authors update themselves, and neither the version 2 nor the version 3 project page runs a leaderboard or describes a way to submit results. The numbers live in papers, in the README and in other people’s reports. When an FDB number appears in a launch post, our advice is to find out who ran it, with which scripts and on which model snapshot before comparing it with anything else.

7. Limits and related benchmarks

The authors list their own limits. Version 1 covers English only and does not link its metrics to human preference. Version 2 is also English only. It scores the meaning of what is said and does not reward expressive prosody or active-listening cues, and it depends on an automated examiner and an LLM judge, which the authors say bring prompt sensitivity, bias and calibration drift. Versions 1.5 and 2 use synthesized speech, which the version 2 authors defend as giving every model identical acoustic conditions. Version 3 switches to real human speech but has 100 recordings, and its paper is still labelled work in progress. The repository’s LICENSE file is Creative Commons Attribution-NonCommercial 4.0.

The biggest open question, to us, is whether doing well on these clips predicts what users prefer. The version 1 authors say their metrics are not yet linked to human preference, and version 2’s human check measures agreement with its LLM judge, not what users prefer.

Other benchmarks cover some of these gaps, several with FDB in the name. IndicFDB adds languages. HumDial-FDBench, from the ICASSP 2026 HumDial Challenge, uses conversations that actors perform from scripts and record on two channels, with a protocol built on version 1.5 and a public leaderboard. VideoFDB adds video.

DyaFDB (October 2026) takes issue with the designs Full-Duplex-Bench has used so far. It argues that pre-recorded audio and a fixed automated examiner test only one side of what it calls a two-body problem, because turn-taking and interruption come from two speakers reacting to each other. It has two full-duplex models talk to each other under assigned roles and scores both sides.

Sources

• Lin et al., Full-Duplex-Bench (v1), arXiv 2503.04721, ASRU 2025
• Lin et al., Full-Duplex-Bench v1.5, arXiv 2507.23159, ICASSP 2026
• Lin et al., Full-Duplex-Bench-v2, arXiv 2510.07838, ACL 2026
• Lin et al., Full-Duplex-Bench-v3, arXiv 2604.04847
• Full-Duplex-Bench repository, README, news log and LICENSE
• OpenAI, Introducing GPT-Live-1 in the API, September 10, 2026
• xAI, Introducing Grok Voice 2.0, July 29, 2026; ElevenLabs, Eleven Realtime, October 20, 2026
• Artificial Analysis, speech-to-speech benchmarking methodology
• Model and benchmark papers that run FDB: PersonaPlex, DuplexCascade, Kyutai interactivity alignment, Raon-Speech, NemotronLabs VoiceChat, Qwen-Audio-3.1-Realtime, IndicFDB

Next: oto: open full-duplex conversation data that voice AI is building on