oto: open full-duplex conversation data that voice AI is building on

oto has become one of the main sources of open full-duplex conversation data. Four of its Hugging Face sets record each speaker on a separate channel and carry CC BY 4.0, so a model trained on them can ship commercially. Its turn-taking set, labeled with 17 event types, is the training split of TurnBench, a benchmark oto co-developed with Sesame and others, and thirteen outside organizations have already submitted results to its leaderboard. By November 2026, ten months after the first release, seventeen outside papers, models and tools named otoSpeech in their own sources, and five more built on models trained on it.

Nov 2026Updated Nov 202615 min readMarkdown

The author of fullduplex.ai founded oto; the site is a personal project.

1. The case for oto

A full-duplex model learns when to speak from recordings in which each person has a track of their own. The usual source of that audio has been Fisher, the telephone corpus LDC collected in 2003: Kyutai’s Moshi paper fine-tunes on Fisher to gain its full-duplex ability and notes that each side of a Fisher call sits on its own channel. oto (otoearth, Inc.) records the same kind of data at 44.1 or 48 kHz, in English, Japanese and Korean, and its catalog says every set is full-duplex and recorded with speaker consent. In oto’s site summary, full-duplex means each speaker is recorded on a separate channel at the source, so overlap, backchannels and interruptions stay on their own tracks.

oto is now a leading supplier of this data. Four of its sets are open for commercial training under CC BY 4.0. The catalog runs from two-person English chat to Japanese role-play, Korean conversation, rooms of up to five people, video and conversation tied to a task. The turn-taking set carries 17 event labels per speaker and is the training split of TurnBench, which is becoming the standard benchmark for turn-taking. And the data spread quickly: the earliest outside use found for this article, a data recipe in the Lhotse toolkit, was merged two months after the first release, and by November 10, 2026 the list in section 5 runs to twenty-two outside projects.

Fisher is still larger than anything oto has released, Open Yap 1K offers more open hours with transcripts, and the open otoSpeech conversation sets ship without transcripts; section 6 weighs these against oto’s strengths.

2. Terms a model builder can ship with

Four of the five sets under the otoearth account on Hugging Face are CC BY 4.0: the two conversation sets, otoSpeech-full-duplex-280h and otoSpeech-full-duplex-processed-141h, and the two task-oriented sets. A team may train on them and sell the result, provided it credits the source. Access is requested on Hugging Face. The 280-hour card and the two task cards add conditions that users agree to (the 141-hour card points to the 280-hour one): no attempts to identify speakers, de-anonymize participants or link them to outside identities, and no attempts to reconstruct muted content.

That is a shorter path than the alternatives. Fisher comes through LDC’s user agreements, for members or non-members. The full Open Yap 1K corpus is free for commercial use but is granted per request under its own data use agreement; only its 8.92-hour sample is CC BY 4.0.

The turn-taking set is the exception. It uses oto’s own oto Speech License v1.0, which its card calls a non-commercial research license and not an open-source one. It prohibits voice cloning, speaker identification or verification, biometric matching and re-identification, and it restricts corporate research, redistribution, derived data and model artifacts. Production and other commercial use need a separate written license, and for-profit organizations need written approval for a defined research project. The card says access is granted by manual review, and the license text sits behind the same gate.

Everything off the Hub is licensed per buyer: the 1,125-hour English set under a commercial license, the other catalog sets “on request.” oto’s buyer guide (November 2026) puts its licensed datasets at $10 to $100 per hour of audio, with the license and labels such as turn-taking annotation adding to the price.

The cards also protect the people recorded. Speaker profile fields are opt-in, the name field is empty, the task cards call speaker IDs anonymized, muted intervals are listed with start and end times so users can exclude them, and the turn-taking license bans cloning a participant’s voice. For otoSpeech Video, faces are mosaicked in the public sample, and the page says the licensed data has no mosaic. Beyond the catalog’s one-line statement, the cards do not reproduce the consent terms participants agreed to, so a buyer who needs their scope has to ask oto.

3. One channel per speaker, in many settings

Put two people talking at once on a single mixed track and their voices become one signal. A model that has to decide when to speak, or whether a short “mm-hm” means “go on” or “stop,” needs to know which voice did what. Every otoSpeech set keeps that information.

English two-person conversation comes in three sizes. The 280-hour set is raw 48 kHz audio whose card says acoustic conditions vary from session to session and that overlaps, interruptions and laughter are kept. The 141-hour set is, as its name says, processed: 44.1 kHz, drawn partly from the 280-hour release and partly from conversations recorded after it, so the two do not add up to one total. The commercial set, according to oto’s English datasets page, has 1,125 hours across 3,878 sessions at 48 kHz. The open sets ship as WebDataset shards of stereo FLAC, channel 0 for one speaker and channel 1 for the other, each with a JSON record of session metadata, speaker profiles, muted intervals and post-session ratings from each participant.

Beyond English, the Japanese page offers free conversation, unscripted role-play and dialogue read from scripts, all at 48 kHz, and otoSpeech Korean is free conversation at 44.1 kHz with ASR transcripts for each speaker. For models that see as well as hear, otoSpeech Video gives each speaker a camera synced to the two-channel audio. The multi-speaker page lists a three-person set with one mono file per speaker and Face-to-Face, two to five people in one room on 16 channels of 48 kHz, 24-bit audio; because they share a room, each microphone also picks up some of the other voices. The egocentric page describes two to four people folding towels, tidying everyday objects or playing Jenga, with a close microphone for every person, head-mounted cameras on two of them and 18 recording conditions in all.

The task-oriented sets record pairs working toward a goal and ship the task with the audio. otoSpeech-full-duplex-task-oriented-20h has 58 sessions, 20.0 hours, across seven tasks that oto’s write-up names, from Spot the Difference to Voice-to-Form, with 11,025 timestamped interface events, the task material, reference answers where a task has one, and artifacts the participants created. The v2 release adds 47 later sessions and 17.4 hours across eleven task types, including two that bring an AI assistant into the conversation, with 8,301 synchronized events. That is material for testing whether a voice agent can hold a natural conversation and still get the task done. oto’s catalog lists transcripts as not included in the 20-hour set, and the 20-hour card notes that its 82 Photo Talk photographs, from YFCC100M on Flickr, are indexed under their own licenses (40 CC BY, 25 CC BY-SA, 17 CC BY-ND) rather than included.

4. Turn-taking labels and TurnBench

otoSpeech-full-duplex-turn-104h holds 420 two-person conversations, 104.94 hours in all, each about 15 minutes long (the shortest about 10, the longest about 18.5), across six conversation types from narrative to instructional. The audio is 48 kHz, 32-bit float WAV, one file per speaker plus a time-aligned stereo version, about 290 GB in all. Each speaker channel comes with a subtitle (SRT) file in which every entry has a millisecond time range, a bracketed event label and the words spoken, if there are any.

The 17 labels go well beyond the two events TurnBench scores. Backchannels come in three kinds (acknowledgement, continuer and reaction) and interruptions in four, crossing floor-taking against non-floor-taking with competitive against cooperative. The rest cover strong floor holds, awkward silence, fillers, laughter, overlap, normal turns, bounded responses, non-speech noise, non-linguistic speech and channel bleed. As oto’s page on the set puts it, work that needs fewer classes can merge labels, and a team studying backchannel or interruption types does not have to invent a taxonomy first. The card warns that speaker, accent, topic, device and acoustic conditions may be unevenly distributed, that gender metadata is missing for some sessions, and that transcripts and labels may contain errors and judgment calls.

TurnBench, posted to arXiv on August 25, 2026, is a turn-taking benchmark that oto co-developed with Sesame AI, Mundo AI, Carnegie Mellon, National Taiwan University, Academia Sinica and Brno University of Technology; two of its authors are oto researchers. Its benchmark corpus is 30 hours of hand-labeled conversation, split into a labeled dev set and an audio-only test set hosted under Mundo AI’s Hugging Face account. The paper also releases a training set of about 104 hours, annotated under the same protocol and speaker-disjoint from the benchmark, and links it to otoSpeech-full-duplex-turn-104h. A system can therefore train and be scored on labels made the same way.

TurnBench is becoming the standard way to score turn-taking. Its public leaderboard listed fifteen external test submissions from thirteen organizations on November 10, 2026, all submitted between September 1 and November 3: Tavus, Ooma, Neosapience, AI Rudder, Zero Runtime, Joveo, Vox, SPX-AI, p99lab, ZanAudio, the University of Melbourne, the University of Maryland and The Agentic Data Company, which makes Open Yap 1K. Two of those entries come with papers, Melbourne’s VoxTurn and Maryland’s FD-VAD, and Maryland’s MultiPartyTurnBench (October 14, 2026) sets its multi-party results beside the TurnBench leaderboard. Beyond the leaderboard, Soniqo’s VAP-OTO card includes a TurnBench development pilot, Induction Labs built the end-of-turn task in its FrontierML benchmark on the TurnBench dev set, and LiveKit added the TurnBench VAP baseline checkpoint to its eot-bench, with results in 14 languages.

Vox and Joveo trained their entries on otoSpeech, by their own accounts. Vox’s Maru v1 write-up (September 11, 2026) says its onset classifier was trained only on the otoSpeech turn corpus, and Joveo’s BarnOwl card (November 2, 2026) lists the 280-hour and Task v2 sets in its training data, reports dev results with the official scorer, and describes a check that its otoSpeech audio does not overlap the TurnBench dev and test conversations.

oto also describes an end-of-turn detector of its own, otoTurn, made by fine-tuning the open SmartTurn 3.2 model on its turn-labeled data. At a 300 ms operating point, oto reports false interruptions falling from 63.0% to 33.0%, a 47.6% relative reduction, on its own otoSpeech evaluation. Those figures are oto’s, not an outside benchmark’s, and the page lists no public weights.

5. Who builds on otoSpeech

Adoption is the strongest part of the case. The table lists every use this article could confirm in the project’s own paper, model card, write-up or code on November 10, 2026: seventeen outside projects that name otoSpeech, TurnBench itself, and five more that build on a model trained on it. They come from universities (Brno, Carnegie Mellon, Johns Hopkins, Maryland), large companies (NVIDIA, Amazon), voice-AI companies and projects including Pipecat, LiveKit, Soniqo, Vox and Anyreach AI, and individual developers who publish open-source turn detectors. oto keeps a similar list on its use-cases page (last checked November 4, 2026), which says the logos on it do not mean product adoption, production use or endorsement.

Some of these uses are training at scale. Anyreach AI’s DualTurn (March 9, 2026) pretrains a turn-taking model on two-channel audio drawn from 289 hours of otoSpeech at 24 kHz and 220 hours of Switchboard telephone speech, holds out 113 otoSpeech sessions as a second test set, and released open weights trained on an otoSpeech-derived turn-taking set. Those weights became the scoring backbone of TurnNat from Johns Hopkins and Amazon, and the endpoint predictor in the OpenRealtime project. People Make Things published three papers on September 23, 2026 that draw on the open sets; one of them, PALS, post-trains a full-duplex model on 3,118 hours that include 141 hours of otoSpeech.

Others are models for voice agents. Pipecat’s backchannel detector, published with open weights on October 28, 2026, trains and validates on Open Yap 1K and on 1,081 conversations from the 280-hour set. On held-out conversations its card reports 81.2% of interruptions caught within 700 ms on otoSpeech (40 conversations) against 82.9% on Open Yap (60); the bot talked through 75.2% of backchannels on otoSpeech and 86.5% on Open Yap. Adventists.ai trains the end-of-turn stage of DuplexJev on the 141-hour set.

One study measured otoSpeech against other sources and found it behind. In Mind the Gap (May 14, 2026), researchers at Brno University of Technology, Carnegie Mellon and NVIDIA used the 141-hour set as one of five sources for simulated multi-talker training data. With 500 hours simulated from each, the recognizer’s macro-average error (tcpWER) was 12.4% from close-talk AMI meetings, 13.6% from close-talk NOTSOFAR-1, 14.0% from LibriSpeech read speech, 17.6% from otoSpeech and 17.7% from VoxPopuli. The sources differ in size, with 141 hours of otoSpeech against 960 of LibriSpeech and 543 of VoxPopuli, and the authors put the weaker results of the conversational and parliamentary sources partly down to domain mismatch, accented speech and less accurate forced alignment.

Download counts point the same way as the table. Hugging Face counts a download as all the files one IP address fetches from a repository within five minutes. On November 10, 2026, the five sets showed 28,171 downloads between them, 17,637 of them for the turn-taking set alone, which drew 6,592 in the previous 30 days.

6. Against Fisher, Common Voice and Open Yap 1K

Two earlier profiles in this series cover the reference points. LDC distributes Fisher English, two-person telephone calls collected in 2003: 984 hours in Part 1 and 975 hours in Part 2, 11,699 calls in total, as two-channel 8 kHz mu-law audio under LDC’s user agreements. The main dataset of Mozilla Common Voice, Scripted Speech, is single speakers reading written sentences. Its 27.0 release of September 16, 2026 has 42,593 hours, 29,295 of them validated, in 295 languages, and the project releases its datasets into the public domain. The closest rival is Open Yap 1K from The Agentic Data Company: 1,000 hours of two-channel English conversation at 48 kHz, the same audio format as otoSpeech.

Fisher is still the largest: about 1,960 hours, against 280 for the biggest open otoSpeech set and 1,125 for the commercial one. On bandwidth, terms and labels otoSpeech is ahead. Its sets are recorded at 44.1 or 48 kHz against Fisher’s 8 kHz telephone audio from the early 2000s, four of them are CC BY 4.0 against LDC’s user agreements, and it ships turn-taking event labels and task logs, which Fisher’s catalog entries do not list. Common Voice is far wider, with 295 languages in the public domain, but one person reads per clip, so it holds none of the overlap or turn-taking that two-channel conversation records.

Open Yap 1K is the fairest comparison, and on size it wins. All 1,000 hours are free for commercial use under its data use agreement and come with word-level transcripts for both speakers, while otoSpeech’s largest open set is 280 hours without transcripts and its 1,125-hour set is sold. Fisher ships with transcripts in companion corpora, and TurnBench’s survey of corpora records the 280-hour otoSpeech set as having no annotations. otoSpeech’s advantage is breadth and labeling. It covers Japanese, Korean, video, groups of three to five and task-oriented talk, its CC BY sets need no agreement beyond the card terms, and it carries opt-in speaker profiles, post-session surveys, 17 turn-taking labels per speaker on one set and task state, reference answers and interface events on two. Pipecat’s detector trained on both.

7. What to expect

We expect TurnBench to settle in as the reference for turn-taking evaluation. Thirteen organizations outside its author list put systems on its leaderboard within ten weeks of the paper, from Tavus and Ooma to two universities and the maker of a rival conversation dataset, and Induction Labs and LiveKit built on its data and checkpoints. It ships a training split annotated under the same protocol as its test set, a public leaderboard and its scoring code. Each new system scored on it gives the next team a reason to train on the matched otoSpeech split.

A team building a full-duplex system can pretrain and fine-tune on the CC BY sets with no agreement beyond the card terms, train and score end-of-turn and interruption behavior on the matched pair of the turn-taking set and TurnBench, and use the task-oriented sets to check that an agent can talk naturally and still finish the job. When the product needs Japanese, Korean, video or a room of up to five people, the licensed sets keep the same channel layout.

Next: Tavus: what changes when a full-duplex agent can see and be seen