A full-duplex model learns turn-taking from conversations recorded with each speaker on their own channel. Separate channels keep overlaps, interruptions and backchannels intact, while a single mixed track blurs them. Before training on a dataset, check two things: whether its licence allows your use, and whether its speakers consented to the recording.
Why each speaker needs a channel
The overlap blurs into one sound, the backchannel disappears into the main voice, and turn ends have to be inferred.
The overlap, the backchannel and every turn end keep their own exact timing.
When two speakers share one track, a 200 ms overlap that was two clearly attributed events becomes one blur, backchannels disappear into the main speaker's voice, and turn ends have to be inferred instead of read off. Separation models can split a mixed track by speaker afterwards, but they lose much of their accuracy on real overlapping speech.
With a channel per speaker, every turn end and every takeover has an exact timestamp. That is also why a benchmark such as MultiPartyTurnBench keeps each of its synthetic speakers on a separate channel.
How much two-channel conversation a model needs depends on its design. Dual-stream models such as Moshi need the most. Designs that interleave text and speech in one stream have been trained largely on synthetic dialogue.
Read more: The data ceiling
Channel-separated datasets
31 of the 107 datasets in the database record each speaker on their own channel. The chart shows their hours on a log scale, coloured by whether commercial use is allowed. Select a bar to open the dataset's page.
No published hours: otoSpeech Video, otoSpeech Face-to-Face, otoSpeech 3-Speaker, otoSpeech Korean, otoSpeech Japanese, otoSpeech Role-Play, otoSpeech Scripted, otoSpeech Egocentric.
9 allow commercial use, 14 restrict it and 8 do not allow it. 8 publish no hour count. Browse and filter them all on Datasets.
Reading the licence
The database sorts each licence into one of five classes. The counts are for the channel-separated datasets.
| Class | What it means | Datasets |
|---|---|---|
| Permissive | CC0, CC BY, Apache or MIT. Commercial use is allowed. | 6 |
| Share-alike | CC BY-SA and similar. Work built on the data must be shared under the same terms. | 1 |
| Non-commercial | CC BY-NC or research-only terms. | 7 |
| Gated | Access is paid, needs a membership or needs an application, as with the LDC catalogue. | 7 |
| Custom | Terms written by the provider that fit none of the above. | 10 |
The class describes the licence text, and the commercial-use flag describes what it allows. They usually agree but not always, so check both on each dataset page and read the original terms before you train.
Reading consent
The database records where each dataset's speech comes from. Of the channel-separated datasets, 28 use speech given with consent, 1 comes from public media and 2 do not say.
Consent to a recording is not the same as consent to training. One two-speaker recording can fall under several separate rules: consent to record the call, the terms of the platform it was uploaded to, disclosure rules for AI training data, biometric rules if the voice is used to identify someone, and likeness rights if a synthetic voice resembles a speaker.
Two things are worth looking for in a dataset's documentation: whether each speaker agreed, not only the person who uploaded the file, and whether that agreement covers model training. Platform terms usually attach to the uploader, which leaves the other voices in a conversation uncovered.
Read more: Consent, licensing, and the opt-in economy for conversational data
Real and synthetic conversation
Synthetic dialogue is easier to produce at scale, and some designs train largely on it. OmniFlatten reached full-duplex behaviour with 0.5 billion parameters on 2,000 hours of synthetic dialogue. The limits come from the generator: prosody drifts toward the text-to-speech model's habits, backchannels appear only where a rule places them, and overlaps follow the script.
Self-play has the same problem. When two full-duplex models generate conversations for each other, their late turn-taking goes into the data. See Signals 2026-W42 and the Backchannels guide.
FAQ
- What makes a dataset full-duplex?
- On this site, a dataset counts as full-duplex when each speaker is recorded on their own channel. 31 of the 107 datasets in the database qualify.
- Can I use these datasets commercially?
- 9 of the 31 channel-separated datasets allow commercial use. 14 restrict it and 8 do not allow it. Use the Commercial OK filter, then read each dataset's original terms before you train.
- Can I split a mixed recording into channels afterwards?
- Only partly. Separation models that are close to clean on studio mixes lose much of that on real overlapping speech, and diarization still assigns part of the audio to the wrong speaker: pyannote 3.1 reports about 22% diarization error on AMI meeting audio.
- Are there Japanese channel-separated datasets?
- 8 in the database include Japanese: CALLHOME (JA / ZH / ES / DE / AR …), Japanese Duplex Conversation Dataset, DuplexChat, otoSpeech Japanese, otoSpeech Role-Play, otoSpeech Scripted, Basis Conversations 1500, DAI-ASR-I18N.
Updated