Glossary

What Is Speaker Diarization? A Plain-English Explanation

Updated · 6 min read

Speaker diarization is the process of splitting an audio recording into segments by speaker, answering the question “who spoke when?”. A diarization system does not know anyone’s name: it groups stretches of speech that sound like the same voice and labels them Speaker 1, Speaker 2, Speaker 3. Combined with speech-to-text, it turns one block of text into a transcript where every line is attributed to a speaker, which you can then rename to real people.

Speaker diarization meaning, in one example

Without diarization, a transcript of a three-person meeting is a single wall of text. With it, the same text is split into turns: “Speaker 1: Can we ship on the 15th? Speaker 2: QA still has 42 open tickets. Speaker 1: Then let’s push a week.” The word comes from “diary”: the system writes a log of who held the floor at each moment.

Diarization timeline: one mixed audio waveform split into blue, purple and amber segments for Speakers 1 to 3, with an overlap marked, then renamed to Priya, Marcus and Lena
One recording in, a timeline of speaker segments out. Names are added afterwards.

How speaker diarization works

Systems differ in detail, and many modern ones run several of these stages inside one neural network, but almost all do the same four jobs:

  1. Find the speech

    Voice activity detection separates speech from silence, music, typing and room noise, so only spoken audio is analyzed.

  2. Cut it into short pieces

    The speech is split at points where the voice seems to change, or simply into windows of a second or two.

  3. Describe each voice

    Each piece is turned into a “voiceprint”: a list of numbers (an embedding) that captures pitch, timbre and speaking style, not the words.

  4. Group similar voices

    Pieces with similar voiceprints are clustered together, and each cluster gets a label. The system also has to guess how many speakers there are, which is where many errors start.

The labels are then lined up with the timestamps from speech-to-text, so each sentence inherits the speaker who was talking at that moment. Some newer systems, including large AI models that take audio directly, transcribe and attribute speakers in one pass, and can also pick up names from context, such as “Thanks, Priya”.

Diarization vs speaker identification vs transcription

TermQuestion it answersOutputNeeds known voices?
Transcription (speech-to-text)What was said?TextNo
Speaker diarizationWho spoke when?Speaker 1, Speaker 2 … with time rangesNo
Speaker identificationWhich known person is this?A name from an enrolled listYes, voice samples in advance
Speaker verificationIs this the person they claim to be?Yes or noYes, one enrolled voice

Meeting tools that join a call as a bot can often skip much of the guesswork by reading names from the video platform’s participant list. Tools that record from a device microphone, including botless note takers, rely on diarization from the audio alone. See AI note takers without a bot for that trade-off.

What affects diarization accuracy

Common causes of wrong speaker labels
FactorWhat goes wrongFix
Overlapping speechTwo people talk at once; the line goes to one of them or is split oddlyA facilitator who lets people finish; check overlaps by hand
Similar voicesTwo speakers merged into one labelUse names when addressing people; rename and check both
Short turns“Yes”, “right”, “mm-hm” are too short to fingerprintUsually harmless; ignore unless it is a vote or agreement
Distance and echoFar speakers sound alike and get mergedPut the device in the middle of the table, away from walls
Background noiseNoise treated as speech or masks voice differencesClose windows and doors, mute notifications, avoid cafés
One voice changingSomeone who moves, laughs or shouts is split into two speakersMerge by giving both labels the same name
Many speakersQuiet participants folded into louder onesAbove six to eight people, expect to review labels
Speakerphone or call audioRemote voices are compressed and sound similarUse computer speakers at moderate volume, not a tiny phone speaker

Researchers measure this with the diarization error rate (DER): the share of time that is missed speech, false-alarm speech, or speech given to the wrong speaker. In practice, what matters is simpler: are decisions and action items attributed to the right person? A wrong label on “mm-hm” is noise; a wrong label on “I’ll send the contract Friday” creates a wrong action item.

Tips for better speaker labels

  • Place one recording device centrally, about an arm’s length from the nearest speaker.
  • Start with a quick round of introductions: “Hi, I’m Lena from finance.” It helps you, and AI that reads names from context.
  • Address people by name when handing over (“Marcus, what do you think?”).
  • Ask people not to talk over each other on decisions and votes.
  • On calls, play remote audio through speakers, not headphones, so the device hears everyone.
  • Rename speakers right after the meeting while you still remember the voices.
  • Spot-check every decision and action item against the audio before sharing.

Speaker labels in AI Note Taker

While you record, AI Note Taker shows a live transcript without speaker labels. When you run the AI analysis, the recording is processed again and the transcript is replaced by a speaker-labeled version, split into turns with a timestamp on each. Where people’s names come up in the conversation, the app suggests them for the matching speakers; otherwise you see Speaker 1, Speaker 2 and so on, each in its own color.

  • Rename a speaker: on iPhone, open the full transcript and tap a speaker’s name; on the web, use the Speakers list above the transcript on the meeting page. The new name applies to every line by that speaker and syncs between devices.
  • Your names win: if you rename a speaker, a later re-analysis keeps your name instead of the suggestion.
  • Where names show up: in the transcript, in transcript search, in Markdown and PDF exports, and in the transcript Ask AI reads when you ask about a single meeting (a Pro feature).
  • Plan note: speaker labels come with the AI analysis. The free plan includes unlimited recordings and transcriptions plus one AI analysis; Pro includes unlimited analyses.
Full transcript with lines labeled Alex, Sara, Jordan and Priya, each with a timestamp, and a hint to tap a speaker to rename them

Tap a speaker to rename them

iPhone recording a meeting with a running timer, a live transcript without speaker labels, and Pause and Stop buttons

Live transcript while recording

Speaker labels are added after recording, when the AI analysis runs.

Related reading: meeting minutes vs notes, how to transcribe an in-person meeting and asking questions about a meeting transcript.

Frequently asked questions

Let AI take the notes for you

AI Note Taker records and transcribes your meetings, then writes the summary, decisions and action items. Free to start, no install needed.