What is speaker identification in captions, and how accurate is it?

Speaker identification in captions means the text tells you who said what. It comes in two levels. The first, speaker diarization, only tells voices apart: "Speaker 1" says this, "Speaker 2" says that. The second, recognizing a known voice, puts a name on the voice. Both are less reliable when people talk at the same time.

What is speaker diarization?

Speaker diarization is the step that splits a recording into turns and groups the turns by voice. It does not know who anyone is, so the labels are generic. Captioning standards expect this: the Captioning Key says to identify a speaker with the same information a hearing viewer has, and gives "(speaker #1)" as a correct label for an unnamed speaker¹.

For a deaf or hard of hearing person at a group dinner or in a meeting, the label does real work. Without it, a stream of words has no owner, and you cannot tell whether the question was aimed at you.

How is the accuracy of speaker identification measured?

Diarization is scored with the diarization error rate (DER). The pyannoteAI benchmark defines it as the aggregate duration of all three error types divided by the total duration of speech in the recording³. The three errors are:

A lower DER is better. The number says nothing about names: it measures whether the voices were separated correctly, not whether a person was recognized.

Why is overlapping speech the hard part?

Overlapping speech is hard because the system has to place two voices in the same moment, and it often detects only one. In the DIHARD II challenge, a research test built from difficult recordings, one team's system report gave an overall overlapped error of 10.76% in the development set, and "in specific domains such as meeting, restaurant and webvideo, the error is much higher"². The same team's overlap detection recovered less than a tenth of the overlapped speech and cut the DER by only 0.38% to 0.69%².

One more team, working on the telephone-conversation part of the DIHARD III data, where each call has 2 speakers, found that missed speech was the largest part of its first system's 16.22% DER, and that this suggested a traditional method could not handle overlapped speech well. Adding speech separation steps brought its DER down to 8.31%⁴. That is a large gain on one test, but overlap remains a known weak point.

In pyannoteAI's own benchmark data, overlap speech makes up 9.3% of its 259 recordings³; that figure describes that set, not speech in general.

How accurate is speaker diarization today?

Accuracy depends heavily on the setting. In October 2026, pyannoteAI, a speaker diarization vendor, published a test of streaming diarization on DIHARD III, a research set of 11 types of recording. Its own system scored a DER of 19.8% across all languages, ahead of three other commercial systems tested, at 31.3% to 39.2%⁵. The vendor ran the test itself, so read it as one company's measurement, not an independent ranking. These are figures for streaming systems from pyannoteAI and other vendors, not for Captio AI. The results by recording type show the spread:

For a deaf or hard of hearing reader, the pattern is the point. DIHARD's clinical interviews, which are not the same as a real doctor's appointment, scored 19.8%. Meetings and restaurants, the settings of a team meeting and a dinner out, scored 44.6% and 54.4%. Even the best system in the test had a DER above 50% for restaurants. The challenge organisers say web video "remains far from solved"⁶.

How does recognizing a known voice differ?

Recognizing a known voice gives the caption a name instead of a number. The app needs a sample of the person talking, compares each new turn with it, and uses the saved name when the voice matches.

Captio AI, a live captioning app for deaf and hard of hearing people, does this on iPhone since version 1.1.3 with its Recognize voices feature. You save a voice by recording about 20 seconds of the person talking, or by choosing a file, and captions then show their name instead of "Speaker 2", live and in saved recordings. Voices are made and kept on the phone, and the feature works with Separate Speakers turned on. The free plan recognizes your own voice plus one other person, and Captio AI Plus allows more.

If a voice is not saved, it stays "Speaker N". The app can also ask who an unknown speaker is, and every answer improves recognition. In a saved recording of a doctor's appointment or a lecture, you tap a speaker's tag and say who it is. For more on how apps compare here, see how to see who is talking in live captions.

Which situations should you trust speaker labels in?

The figures above suggest speaker labels are more reliable for an interview than in meetings and restaurants, though a real appointment may differ. Be careful wherever voices overlap, as in a group setting at a dinner or a meeting.

In every case the words matter more than the label. If a label looks wrong, read the sentence for who it makes sense for, and correct the name afterward if the app lets you.

Where can you learn more?

The Captioning Key from the Described and Captioned Media Program sets out how speakers should be identified in captions¹. For the research side, the DIHARD challenge papers (individual teams' system reports, not the organisers' overview) show where diarization stands and where it still fails²⁴.