From e86621e6b6190e16f2adfbfd06071421d9b558eb Mon Sep 17 00:00:00 2001 From: Sam Sykes Date: Wed, 8 Jul 2026 14:27:27 +0100 Subject: [PATCH 1/9] new docs on focus / speaker id --- .gitignore | 5 +- custom-words.txt | 1 + .../speaker-focus-for-voice-agents.mdx | 219 ++++++++++++++++++ .../speaker-memory-for-voice-agents.mdx | 211 +++++++++++++++++ 4 files changed, 435 insertions(+), 1 deletion(-) create mode 100644 docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx create mode 100644 docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx diff --git a/.gitignore b/.gitignore index 81ea2715..bcfc34f4 100644 --- a/.gitignore +++ b/.gitignore @@ -28,4 +28,7 @@ static/*.yaml .vercel .env*.local -.venv \ No newline at end of file +.venv + +# Temporary / local dev +/tmp diff --git a/custom-words.txt b/custom-words.txt index f00dfb5b..b0dfc137 100644 --- a/custom-words.txt +++ b/custom-words.txt @@ -342,3 +342,4 @@ sessiongroups melia مرحبا Theo +pipecat diff --git a/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx b/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx new file mode 100644 index 00000000..168fba86 --- /dev/null +++ b/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx @@ -0,0 +1,219 @@ +--- +description: "Direct a voice agent to respond to chosen speakers and treat others as background, and update the focus live mid-session." +keywords: + [ + speechmatics, + features, + speaker focus, + diarization, + voice agents, + realtime, + speech recognition, + automatic speech recognition, + asr, + ] +sidebar_position: 4 +--- + +# Speaker focus for voice agents + +Speaker focus lets a voice agent decide which speakers to respond to and how to treat everyone else, and update that choice live during a session. + +Put a voice agent in a room with more than one person and you hit a problem straight away: whose words should the agent act on? If two people are talking and a third asks a question, you don't want the agent replying to the crossfire. You want it locked onto the person talking to it, with everyone else treated as background it can hear but shouldn't answer. Speaker focus is how you do that. + +:::info +Speaker focus is a helper provided by the Speechmatics Python [voice SDK](/voice-agents/voice-sdk), which applies the focus rules on top of speaker [diarization](/speech-to-text/features/diarization) (labeling every word with a speaker: `S1`, `S2`, and so on). The Realtime API does not expose speaker focus directly today, and the voice SDK is Python only. Speechmatics provides the speech-to-text layer for voice agents, not the full pipeline. +::: + +## Focus and focus mode + +There are two levers. + +**Focus** is the set of speakers you care about. When a focus is set, only those speakers produce active transcript. Everyone else becomes passive. + +**Focus mode** decides what passive means: + +- `RETAIN` keeps the other speakers audible but marks them as background. Their words are still transcribed and still appear, carrying a passive marker so your agent knows not to respond. This is the default and the right choice most of the time: a background speaker can still be pulled into the conversation when a focused speaker invites them in. +- `IGNORE` drops the other speakers entirely. Their audio is not transcribed and does not trigger voice activity or end-of-utterance detection. Use it when you want the engine deaf to everyone else. + +A separate **ignore list** drops one specific speaker without touching the focus. It is useful for killing feedback when the agent's own audio output is picked up by the microphone, or for muting one person on request. + +Any speaker whose label starts and ends with double underscores (such as `__ASSISTANT__`) is excluded automatically, which keeps the agent's own voice out of the transcript. + +## Set and update focus + +You set an initial focus when you open the session, and you can update it at any point while the session runs. The two format strings decide how active and passive speakers are rendered into the transcript the LLM reads. + +The following pseudocode configures focus at session start and updates it live: + +```python +# At session start, with diarization on. The format strings control how +# active and passive speaker lines are rendered for the LLM. +configure_stt( + enable_diarization = true, + speaker_active_format = "@{speaker_id}: {text}", + speaker_passive_format = "@{speaker_id} [background]: {text}", +) + +# Live, at any point during the session: +set_speaker_focus(focus = ["S1"], mode = RETAIN) # others kept as background +set_speaker_focus(focus = ["S1"], mode = IGNORE) # others dropped entirely +ignore_speaker(add = "S2") # separate ignore list +set_speaker_focus(focus = [], mode = RETAIN) # reset, hear everyone +``` + +The format strings matter more than they look. The active speaker's line arrives as `@S1: can you help me`, and a background speaker as `@S2 [background]: yeah but what about lunch`. That `[background]` tag is the signal your system prompt teaches the LLM to ignore. + +A live focus update replaces the previous focus, it does not merge. To build up a multi-speaker focus one speaker at a time, track the set yourself and send the full list each time. + +## Drive focus from the agent + +Rather than a human flipping switches, you hand the LLM a small set of function tools and let it change the focus in response to plain speech. "Focus on me" becomes a tool call. + +The following pseudocode defines a sensible tool set: + +```python +focus_on_speaker(speaker_ids: string[]) + # Prioritize one or more speakers. Everyone else stays audible as + # background (RETAIN). Replaces any current focus. + "focus on me" -> focus_on_speaker(["S1"]) + "focus on me and her" -> focus_on_speaker(["S1", "S2"]) + +listen_only_to_speaker(speaker_ids: string[]) + # Listen ONLY to these speakers. Everyone else is dropped (IGNORE). + "only listen to me" -> listen_only_to_speaker(["S1"]) + +ignore_speaker(speaker_id: string) + # Add one speaker to the ignore list so their speech stops being + # transcribed. Good for killing echo or muting one person. + "ignore him" -> ignore_speaker("S2") + +listen_to_all_speakers() + # Reset. Clear focus and ignore lists, hear everyone equally again. + "listen to everyone again" -> listen_to_all_speakers() +``` + +The handlers behind these are thin. They map the tool call onto a focus state and push it to the engine: `focus_on_speaker` sets `RETAIN`, `listen_only_to_speaker` sets `IGNORE`, and both call the same live `set_speaker_focus`. `listen_to_all_speakers` sends empty lists and resets the mode to `RETAIN`. + +:::tip +Wire up the reset. If `listen_to_all_speakers` does not clear both the focus and the ignore list, the agent stays stuck on one person. +::: + +## System prompt for speaker focus + +The tools do nothing without a system prompt that teaches the LLM how to read the transcript and when to use them. Three things have to be spelled out. + +First, the LLM must understand the speaker tags. Every incoming line is prefixed, either `@S1:` for a raw label or `@Sam:` once a speaker is recognized by name. Lines carrying `[background]` are passive and should be left alone unless an active speaker brings that person in. + +Second, the LLM has to resolve "me." When someone says "focus on me," the word "me" means whoever is speaking, which is the ID prefixing their own message. A "focus on me" arriving on a `@S1:` line resolves to `S1`. Without this rule the model guesses, and it guesses badly. + +Third, the labels are internal. The agent must never read `S1` or `S2` aloud, and never echo a speaker tag into its reply. Real names are fine once known. + +The following prompt fragment covers the speaker rules: + +```markdown +# Speakers + +- Each message is prefixed with the speaker, like @S1: or, for a + recognized person, @Sam:. +- Messages prefixed @speaker_id [background]: are passive. Never respond + to or acknowledge them unless an active speaker invites you to. +- When a speaker says "me", "my" or "I", they mean the speaker ID + prefixing their own message. A request from @S1: refers to S1. +- Generic labels like S1 are internal. Never say them aloud or include a + speaker tag in your replies. Real names like "Sam" are fine. + +# Speaker focus + +- "Focus on me" -> focus_on_speaker with the requesting ID. +- "Only listen to me" -> listen_only_to_speaker with the ID(s). +- "Ignore him/her" -> ignore_speaker with that speaker's ID. +- "Listen to everyone" -> listen_to_all_speakers to reset. +``` + +## Example: switching to a single speaker + +This walks through "just listen to me" from a speaker labeled `S1`, with someone else talking as `S2`: + +1. The transcript line arrives as `@S1: just listen to me`, and `S2`'s chatter arrives as `@S2: ...`. +2. The LLM applies the prompt rule, resolves "me" to `S1`, and calls `listen_only_to_speaker(["S1"])`. +3. The handler sets focus to `["S1"]` with mode `IGNORE` and pushes it to the engine. +4. From that point the engine transcribes only `S1`. `S2` is dropped entirely: no transcript, no turn detection off their voice. +5. The agent confirms, something like "sure, just you now," and carries on. + +Saying "listen to everyone again" runs step 3 in reverse: empty focus, empty ignore list, mode back to `RETAIN`, everyone audible. + +## Speaker focus in the voice SDK, Pipecat and LiveKit + +The concepts above map onto real integrations with little code. The Speechmatics Python [voice SDK](/voice-agents/voice-sdk) exposes the two levers as `SpeakerFocusConfig` and `SpeakerFocusMode`; the Pipecat and LiveKit plugins wrap the same SDK under their own method names. + +For the voice SDK, set the focus on the session config and update it live with `update_diarization_config`: + +```python +from speechmatics.voice import SpeakerFocusConfig, SpeakerFocusMode + +# Session setup: set focus on the transcription config's speaker_config +speaker_config = SpeakerFocusConfig( + focus_speakers=["S1"], + focus_mode=SpeakerFocusMode.RETAIN, # others kept as background +) + +# Live update (inside a tool handler) +client.update_diarization_config( + SpeakerFocusConfig(focus_speakers=["S1"], focus_mode=SpeakerFocusMode.IGNORE) +) +``` + +For [Pipecat](https://github.com/pipecat-ai/pipecat), configure focus on the STT service and update it inside a tool handler: + +```python +# Session setup +stt = SpeechmaticsSTTService( + params=SpeechmaticsSTTService.InputParams( + enable_diarization=True, + speaker_active_format="@{speaker_id}: {text}", + speaker_passive_format="@{speaker_id} [background]: {text}", + ), +) + +# Live update (inside a tool handler) +stt.update_params( + SpeechmaticsSTTService.UpdateParams( + focus_speakers=["S1"], + focus_mode=SpeechmaticsSTTService.SpeakerFocusMode.RETAIN, + ) +) +``` + +For the [LiveKit Speechmatics plugin](https://github.com/livekit/agents/tree/main/livekit-plugins/livekit-plugins-speechmatics), use `update_speakers`: + +```python +# Session setup +stt = speechmatics.STT( + enable_diarization=True, + speaker_active_format="[Speaker {speaker_id}] {text}", + speaker_passive_format="[Speaker {speaker_id} *PASSIVE*] {text}", +) + +# Live update (inside a tool handler) +stt.update_speakers( + focus_speakers=["S1"], + focus_mode=SpeakerFocusMode.RETAIN, +) +``` + +Pick format strings to suit your model, and teach the LLM whichever format you chose. + +## Best practices for speaker focus + +`RETAIN` is the safer default for a chatty agent, because a background speaker can be brought in without a config change. `IGNORE` is blunter. Reach for it when you truly want the engine to stop listening, not just stop responding. + +Keep the agent's own voice out of the loop. Either use the `__ASSISTANT__` double-underscore convention or add the agent's speaker to the ignore list. Otherwise the agent's audio output is transcribed straight back in and it starts talking to itself. + +Speaker focus pairs with [speaker memory](/speech-to-text/features/speaker-memory-for-voice-agents). Once a voice is recognized by name, every focus and ignore operation can target that name instead of a throwaway label. + +## Next steps + +- [Speaker memory for voice agents](/speech-to-text/features/speaker-memory-for-voice-agents) — recognize returning speakers by name across sessions. +- [Voice SDK](/voice-agents/voice-sdk) — the Python SDK reference for `SpeakerFocusConfig` and speaker management. +- [Diarization](/speech-to-text/features/diarization) — separate a transcript into distinct speakers. diff --git a/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx b/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx new file mode 100644 index 00000000..03aae125 --- /dev/null +++ b/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx @@ -0,0 +1,211 @@ +--- +description: "Persist speaker identifiers so a voice agent recognizes returning speakers by name across sessions." +keywords: + [ + speechmatics, + features, + speaker identification, + speaker identifiers, + diarization, + voice agents, + realtime, + speech recognition, + automatic speech recognition, + asr, + ] +sidebar_position: 5 +--- + +# Speaker memory for voice agents + +Speaker memory lets a voice agent recognize a returning speaker by name across sessions, using durable speaker identifiers. + +Diarization gives you labels: `S1`, `S2`, `S3`. Those labels are only good for the life of one session. The person who was `S1` today might be `S2` tomorrow, or `S1` again by chance. The labels carry no memory. Speaker identifiers are how you give them memory. During a session the engine builds a voice fingerprint for each speaker, and you can save those fingerprints as durable identifiers. Feed them back into a later session and the engine attributes matching voices to a name you chose, `Sam` instead of `S1`. + +:::info +Speaker memory builds on [speaker identification](/speech-to-text/features/speaker-identification) with speaker [diarization](/speech-to-text/features/diarization) enabled. The Speechmatics Python [voice SDK](/voice-agents/voice-sdk) (Python only) exposes it through `known_speakers` and the speakers request. Speechmatics provides the speech-to-text layer for voice agents, not the full pipeline. +::: + +## How speaker identifiers work + +A speaker identifier is a voice fingerprint the engine produces once diarization is on. It is a string, tied to your Speechmatics account. That last point matters: identifiers are unique per account and do not work across accounts, so you cannot take a fingerprint from one account and use it under another. + +A speaker can accumulate more than one identifier over time, as the engine hears them in different conditions. A saved profile is therefore a name plus a list of identifiers, and you keep adding to that list as you see the person again. More identifiers mean better recognition. + +There are two halves to speaker memory: getting identifiers out of a live session, and feeding them back into a later one. + +## Get identifiers from a live session + +While a session is running, you request the current speakers. The engine replies with every speaker it has tracked, each with its label and its identifiers. + +The following pseudocode requests identifiers and persists the ones you want: + +```text +# Ask the engine for the fingerprints of everyone heard so far. +speakers = request_speakers() + +# speakers looks like: +# [ +# { label: "S1", speaker_identifiers: ["XX...XX"] }, +# { label: "S2", speaker_identifiers: ["YY...YY"] }, +# ] + +# Keep the ones you care about and persist them (your storage): +for s in speakers: + if s.label in wanted: + save_profile(name = wanted[s.label], identifiers = s.speaker_identifiers) +``` + +The engine returns everyone, so filter down to the speakers you want on your side. Recognition quality improves once each speaker has said a handful of words (five or so is a good rule of thumb), so do not fire the request the instant someone starts talking. + +Some integrations let you flag the request as final, meaning "give me the definitive fingerprints computed over the whole session." That is the best quality you will get and the natural thing to do at the end of a call. A mid-session request without the flag gives you whatever the engine has so far, which is fine for an interim save. + +## Reuse identifiers in a later session + +Saving the identifiers is up to you. A local file works and keeps things offline, but a database or a user record is just as valid. + +To use them, pass them as known speakers when you open the next session. Each entry pairs a label (now a real name) with the list of identifiers you saved. + +The following pseudocode configures known speakers at session start: + +```text +configure_stt( + enable_diarization = true, + known_speakers = [ + { label: "Sam", speaker_identifiers: ["XX...XX"] }, + { label: "Alice", speaker_identifiers: ["YY...YY"] }, + ], +) +``` + +From that point the engine attributes matching voices to `Sam` and `Alice` instead of `S1` and `S2`. Anyone it does not recognize still gets a fresh generic label, so known and unknown speakers coexist. + +## Drive voice memory from the agent + +As with [speaker focus](/speech-to-text/features/speaker-focus-for-voice-agents), you can give the LLM a couple of tools and let it manage voice memory through conversation. + +The following pseudocode defines the tools: + +```text +remember_voices(speakers: { speaker_id: string, name: string }[]) + # Save one or more speakers so they're recognized in future sessions. + # Only call when a speaker explicitly asks to be saved. The name must + # be their real name, never a generic label like "S1". Pass everyone + # to save in a single call. + "remember me, I'm Sam" -> remember_voices([{ speaker_id: "S1", name: "Sam" }]) + +forget_all_voices() + # Wipe every saved profile. + "clear voice memory" -> forget_all_voices() +``` + +The `remember_voices` handler needs care, because fetching fingerprints is asynchronous. Do not block the agent's reply waiting for them. The clean pattern is to stash the name mappings, fire the request, reply straight away, and save when the result arrives. + +The following pseudocode shows that non-blocking handler: + +```text +pending = {} # label -> real name, awaiting the speakers result + +on remember_voices(speakers): + for entry in speakers: + pending[entry.speaker_id] = entry.name + request_speakers() # result handled below + reply("Voice profile saved.") # don't wait + +on speakers_result(speakers): + for s in speakers: + name = pending.pop(s.label, none) + if name: + save_profile(name, s.speaker_identifiers) # merge, don't overwrite +``` + +Merge new identifiers into an existing profile of the same name rather than overwriting, so recognition improves each time. Snapshot then clear the pending map before working through the result, so overlapping requests do not double-process. + +## System prompt for voice memory + +The tools need guardrails in the prompt. The main one: do not save people just because they mention their name. Someone saying "I'm Sam, anyway, as I was saying" is not asking to be remembered. Only call `remember_voices` on an explicit request, such as "remember me" or "save my voice." + +The other rule is that the name must be a real name. `S1` is a label, not a name, and saving a profile called `S1` is worse than useless. If the agent does not know someone's name, it should ask before saving. + +The following prompt fragment covers the memory rules: + +```markdown +# Speaker memory + +- Only call remember_voices when a speaker explicitly asks to be saved, + such as "remember me", "save my voice" or "save our names". Do NOT + call just because someone says their name. +- The name must be the speaker's real name (e.g. "Sam"), never a generic + label like S1. If you don't know their name, ask for it first. +- Pass everyone to save in a single remember_voices call. +- "Forget all voices" or "clear voice memory" -> forget_all_voices. +``` + +## Example: remembering a speaker across sessions + +This walks through two sessions. + +**Session one.** Sam chats to the agent as `S1` and says "remember me, I'm Sam." The LLM calls `remember_voices([{ speaker_id: "S1", name: "Sam" }])`. The handler stashes `S1 -> Sam`, requests the speakers, and replies. The result comes back, the handler finds `S1`'s identifiers, and saves the profile `{ "label": "Sam", "speaker_identifiers": ["XX...XX"] }`. + +**Session two.** On startup you load that profile and pass it as a known speaker. Sam speaks, the engine matches his voice, and his transcript arrives as `@Sam: hello again`. The agent greets him by name, and because he is now a named speaker you can point [speaker focus](/speech-to-text/features/speaker-focus-for-voice-agents) at `Sam` directly. + +## Speaker memory in the voice SDK, Pipecat and LiveKit + +The concepts above map onto real integrations with little code: get identifiers out of one session, feed them back into the next as known speakers. The Speechmatics Python [voice SDK](/voice-agents/voice-sdk) exposes `known_speakers` (using `SpeakerIdentifier`) and a speakers request; the Pipecat and LiveKit plugins wrap the same SDK. + +For [Pipecat](https://github.com/pipecat-ai/pipecat), the result arrives on an event: + +```python +# Request fingerprints, then save them when they arrive +await stt.send_message("GetSpeakers") + +@stt.event_handler("on_speakers_result") +async def on_speakers_result(service, speakers): + for s in speakers["speakers"]: + save_profile(s["label"], s["speaker_identifiers"]) + +# Reuse in a later session +stt = SpeechmaticsSTTService( + params=SpeechmaticsSTTService.InputParams( + enable_diarization=True, + known_speakers=[ + SpeechmaticsSTTService.SpeakerIdentifier( + label="Sam", speaker_identifiers=["XX...XX"] + ) + ], + ), +) +``` + +For the [LiveKit Speechmatics plugin](https://github.com/livekit/agents/tree/main/livekit-plugins/livekit-plugins-speechmatics), the request is a single awaitable: + +```python +# Request fingerprints (waits for the result, with a short timeout) +speakers = await stt.get_speaker_ids() +for s in speakers: + save_profile(s.label, s.speaker_identifiers) + +# Reuse in a later session +stt = speechmatics.STT( + enable_diarization=True, + known_speakers=[ + SpeakerIdentifier(label="Sam", speaker_identifiers=["XX...XX"]), + ], +) +``` + +## Best practices for speaker memory + +Identifiers are per account. You cannot lift a fingerprint from one Speechmatics account and use it on another, so voice profiles belong to the account that created them. + +Quality improves with more identifiers. Rather than treating a profile as fixed, append new identifiers each time you save a returning speaker. + +Request fingerprints once there are a few words to work with, and request them again at the end of a call with the final flag if the integration offers it. That gives you identifiers computed over everything the engine heard, which recognizes better next time than an early snapshot. + +Speaker memory pairs with [speaker focus](/speech-to-text/features/speaker-focus-for-voice-agents). Once a voice is recognized as `Sam`, every focus and ignore operation can target `Sam` by name instead of a throwaway label. + +## Next steps + +- [Speaker focus for voice agents](/speech-to-text/features/speaker-focus-for-voice-agents) — direct the agent to specific speakers. +- [Voice SDK](/voice-agents/voice-sdk) — the Python SDK reference for `known_speakers` and speaker management. +- [Speaker identification](/speech-to-text/features/speaker-identification) — how identifiers are generated and scoped. From 56c0957a95db5fe6e2b6e0605b14f6216dfe7809 Mon Sep 17 00:00:00 2001 From: Sam Sykes Date: Wed, 8 Jul 2026 14:32:22 +0100 Subject: [PATCH 2/9] updated comments and cross-references --- .../features/speaker-focus-for-voice-agents.mdx | 3 ++- .../features/speaker-memory-for-voice-agents.mdx | 5 ++++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx b/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx index 168fba86..4489918a 100644 --- a/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx +++ b/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx @@ -215,5 +215,6 @@ Speaker focus pairs with [speaker memory](/speech-to-text/features/speaker-memor ## Next steps - [Speaker memory for voice agents](/speech-to-text/features/speaker-memory-for-voice-agents) — recognize returning speakers by name across sessions. -- [Voice SDK](/voice-agents/voice-sdk) — the Python SDK reference for `SpeakerFocusConfig` and speaker management. - [Diarization](/speech-to-text/features/diarization) — separate a transcript into distinct speakers. +- [Voice agents overview](/voice-agents/overview) — ways to build a voice agent with Speechmatics. +- [Voice SDK](/voice-agents/voice-sdk) — the Python SDK reference for `SpeakerFocusConfig` and speaker management. diff --git a/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx b/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx index 03aae125..ab3a44d8 100644 --- a/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx +++ b/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx @@ -30,6 +30,8 @@ Speaker memory builds on [speaker identification](/speech-to-text/features/speak A speaker identifier is a voice fingerprint the engine produces once diarization is on. It is a string, tied to your Speechmatics account. That last point matters: identifiers are unique per account and do not work across accounts, so you cannot take a fingerprint from one account and use it under another. +Speaker memory and [speaker identification](/speech-to-text/features/speaker-identification) use the same identifiers for different jobs. Speaker identification enrolls known speakers ahead of time from short audio clips and applies them in batch or realtime transcription. Speaker memory captures identifiers live during an agent session and reuses them in later sessions, so an agent can recognize a returning speaker without prior enrollment. + A speaker can accumulate more than one identifier over time, as the engine hears them in different conditions. A saved profile is therefore a name plus a list of identifiers, and you keep adding to that list as you see the person again. More identifiers mean better recognition. There are two halves to speaker memory: getting identifiers out of a live session, and feeding them back into a later one. @@ -207,5 +209,6 @@ Speaker memory pairs with [speaker focus](/speech-to-text/features/speaker-focus ## Next steps - [Speaker focus for voice agents](/speech-to-text/features/speaker-focus-for-voice-agents) — direct the agent to specific speakers. -- [Voice SDK](/voice-agents/voice-sdk) — the Python SDK reference for `known_speakers` and speaker management. - [Speaker identification](/speech-to-text/features/speaker-identification) — how identifiers are generated and scoped. +- [Voice agents overview](/voice-agents/overview) — ways to build a voice agent with Speechmatics. +- [Voice SDK](/voice-agents/voice-sdk) — the Python SDK reference for `known_speakers` and speaker management. From 4c168a85f5650709851046d19540c657b98076f4 Mon Sep 17 00:00:00 2001 From: Sam Sykes Date: Wed, 8 Jul 2026 14:36:31 +0100 Subject: [PATCH 3/9] updated cross-references --- .../features/speaker-memory-for-voice-agents.mdx | 4 ++-- docs/voice-agents/voice-sdk.mdx | 6 ++++++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx b/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx index ab3a44d8..667bffb0 100644 --- a/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx +++ b/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx @@ -23,7 +23,7 @@ Speaker memory lets a voice agent recognize a returning speaker by name across s Diarization gives you labels: `S1`, `S2`, `S3`. Those labels are only good for the life of one session. The person who was `S1` today might be `S2` tomorrow, or `S1` again by chance. The labels carry no memory. Speaker identifiers are how you give them memory. During a session the engine builds a voice fingerprint for each speaker, and you can save those fingerprints as durable identifiers. Feed them back into a later session and the engine attributes matching voices to a name you chose, `Sam` instead of `S1`. :::info -Speaker memory builds on [speaker identification](/speech-to-text/features/speaker-identification) with speaker [diarization](/speech-to-text/features/diarization) enabled. The Speechmatics Python [voice SDK](/voice-agents/voice-sdk) (Python only) exposes it through `known_speakers` and the speakers request. Speechmatics provides the speech-to-text layer for voice agents, not the full pipeline. +Speaker memory is an application of [speaker identification](/speech-to-text/features/speaker-identification), a standard capability of the Realtime and Batch APIs and their clients, not a voice-specific feature. Speaker identifiers and `known_speakers` are available in [realtime](/speech-to-text/realtime/speaker-identification) and [batch](/speech-to-text/batch/speaker-identification) speaker identification; the exact names vary by client. This page covers the voice-agent pattern: capturing identifiers live and reusing them across sessions, with speaker [diarization](/speech-to-text/features/diarization) enabled. ::: ## How speaker identifiers work @@ -153,7 +153,7 @@ This walks through two sessions. ## Speaker memory in the voice SDK, Pipecat and LiveKit -The concepts above map onto real integrations with little code: get identifiers out of one session, feed them back into the next as known speakers. The Speechmatics Python [voice SDK](/voice-agents/voice-sdk) exposes `known_speakers` (using `SpeakerIdentifier`) and a speakers request; the Pipecat and LiveKit plugins wrap the same SDK. +The concepts above map onto real integrations with little code: get identifiers out of one session, feed them back into the next as known speakers. `known_speakers` is available across the Realtime and Batch clients; the [voice SDK](/voice-agents/voice-sdk), Pipecat, and LiveKit expose it for the agent use case, with method names differing per client. For [Pipecat](https://github.com/pipecat-ai/pipecat), the result arrives on an event: diff --git a/docs/voice-agents/voice-sdk.mdx b/docs/voice-agents/voice-sdk.mdx index f352b780..54446ea7 100644 --- a/docs/voice-agents/voice-sdk.mdx +++ b/docs/voice-agents/voice-sdk.mdx @@ -58,6 +58,8 @@ You can also: - Ignore specific speakers - Provide known speakers for speaker identification +For voice-agent patterns built on these, see [Speaker focus for voice agents](/speech-to-text/features/speaker-focus-for-voice-agents) and [Speaker memory for voice agents](/speech-to-text/features/speaker-memory-for-voice-agents). + ### Voice SDK vs Realtime SDK - Use the Voice SDK when: @@ -400,6 +402,8 @@ In your event handler, you can use `is_active` to decide how to route segments: {pythonSpeakerFocusHandler} +For the agent-driven pattern, where the LLM changes focus in response to speech using function tools and a system prompt, see [Speaker focus for voice agents](/speech-to-text/features/speaker-focus-for-voice-agents). + #### Known speakers (speaker identification) `known_speakers` (list[SpeakerIdentifier], default: []) @@ -409,6 +413,8 @@ Pre-enrolled speaker identifiers for speaker identification. {pythonKnownSpeakers} +To capture identifiers live and reuse them across sessions so an agent recognizes returning speakers by name, see [Speaker memory for voice agents](/speech-to-text/features/speaker-memory-for-voice-agents). + ### Advanced configuration example From 3cab87bca0d4b7ada3851dbbd0021ecb4fead605 Mon Sep 17 00:00:00 2001 From: Sam Sykes Date: Wed, 8 Jul 2026 14:39:14 +0100 Subject: [PATCH 4/9] python linting for code --- .../features/speaker-memory-for-voice-agents.mdx | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx b/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx index 667bffb0..2cd85c22 100644 --- a/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx +++ b/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx @@ -42,7 +42,7 @@ While a session is running, you request the current speakers. The engine replies The following pseudocode requests identifiers and persists the ones you want: -```text +```python # Ask the engine for the fingerprints of everyone heard so far. speakers = request_speakers() @@ -70,7 +70,7 @@ To use them, pass them as known speakers when you open the next session. Each en The following pseudocode configures known speakers at session start: -```text +```python configure_stt( enable_diarization = true, known_speakers = [ @@ -88,7 +88,7 @@ As with [speaker focus](/speech-to-text/features/speaker-focus-for-voice-agents) The following pseudocode defines the tools: -```text +```python remember_voices(speakers: { speaker_id: string, name: string }[]) # Save one or more speakers so they're recognized in future sessions. # Only call when a speaker explicitly asks to be saved. The name must @@ -105,7 +105,7 @@ The `remember_voices` handler needs care, because fetching fingerprints is async The following pseudocode shows that non-blocking handler: -```text +```python pending = {} # label -> real name, awaiting the speakers result on remember_voices(speakers): From 2d1c0da9a6dfff1dbd2f171f69c844a5d3c4c76a Mon Sep 17 00:00:00 2001 From: Sam Sykes Date: Wed, 8 Jul 2026 15:41:35 +0100 Subject: [PATCH 5/9] set context as text for display --- docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx | 2 +- .../speech-to-text/features/speaker-memory-for-voice-agents.mdx | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx b/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx index 4489918a..ff9b1b0f 100644 --- a/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx +++ b/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx @@ -111,7 +111,7 @@ Third, the labels are internal. The agent must never read `S1` or `S2` aloud, an The following prompt fragment covers the speaker rules: -```markdown +```text # Speakers - Each message is prefixed with the speaker, like @S1: or, for a diff --git a/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx b/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx index 2cd85c22..bef34c00 100644 --- a/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx +++ b/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx @@ -131,7 +131,7 @@ The other rule is that the name must be a real name. `S1` is a label, not a name The following prompt fragment covers the memory rules: -```markdown +```text # Speaker memory - Only call remember_voices when a speaker explicitly asks to be saved, From 23998720c5a4b6de5472bc37807bdcd0cf653210 Mon Sep 17 00:00:00 2001 From: Sam Sykes Date: Wed, 8 Jul 2026 15:49:14 +0100 Subject: [PATCH 6/9] updated speaker identifier comments --- .../features/speaker-focus-for-voice-agents.mdx | 2 +- .../features/speaker-memory-for-voice-agents.mdx | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx b/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx index ff9b1b0f..f4090e27 100644 --- a/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx +++ b/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx @@ -208,7 +208,7 @@ Pick format strings to suit your model, and teach the LLM whichever format you c `RETAIN` is the safer default for a chatty agent, because a background speaker can be brought in without a config change. `IGNORE` is blunter. Reach for it when you truly want the engine to stop listening, not just stop responding. -Keep the agent's own voice out of the loop. Either use the `__ASSISTANT__` double-underscore convention or add the agent's speaker to the ignore list. Otherwise the agent's audio output is transcribed straight back in and it starts talking to itself. +Keep the agent's own voice out of the loop, otherwise its audio output is transcribed straight back in and it starts talking to itself. Any speaker enrolled with a label wrapped in double underscores (such as `__ASSISTANT__`) is excluded automatically, so you never have to ignore it explicitly. To use this, pass the agent's voice as a [known speaker](/speech-to-text/features/speaker-memory-for-voice-agents) at session start, labeled `__ASSISTANT__`, with a speaker identifier captured from an earlier session or from enrollment. The engine recognizes the agent's own audio and drops it before it reaches the transcript. This is the clean option for complex speaker or microphone setups, where the agent's voice can leak back in through the mic. You can pass several identifiers under the one label to cover that voice across different output paths or capture devices. Adding the agent's speaker to the ignore list at runtime also works. Speaker focus pairs with [speaker memory](/speech-to-text/features/speaker-memory-for-voice-agents). Once a voice is recognized by name, every focus and ignore operation can target that name instead of a throwaway label. diff --git a/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx b/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx index bef34c00..2e22780d 100644 --- a/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx +++ b/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx @@ -66,7 +66,7 @@ Some integrations let you flag the request as final, meaning "give me the defini Saving the identifiers is up to you. A local file works and keeps things offline, but a database or a user record is just as valid. -To use them, pass them as known speakers when you open the next session. Each entry pairs a label (now a real name) with the list of identifiers you saved. +To use them, pass them as known speakers when you open the next session. Each entry pairs a label (now a real name) with a list of identifiers. You can pass several identifiers under one label: this is how you cover the same voice heard through different devices or microphones (a laptop mic, a smartphone, an tablet, a handset), so recognition holds up wherever the person speaks from. The following pseudocode configures known speakers at session start: @@ -74,7 +74,8 @@ The following pseudocode configures known speakers at session start: configure_stt( enable_diarization = true, known_speakers = [ - { label: "Sam", speaker_identifiers: ["XX...XX"] }, + # One label, several identifiers for the same voice across devices. + { label: "Sam", speaker_identifiers: ["XX...XX", "ZZ...ZZ"] }, { label: "Alice", speaker_identifiers: ["YY...YY"] }, ], ) From 281aff1f23c894eb07ab0ac3f43eb13b257a3065 Mon Sep 17 00:00:00 2001 From: Sam Sykes Date: Wed, 8 Jul 2026 15:57:59 +0100 Subject: [PATCH 7/9] GDPR warning --- .../features/speaker-memory-for-voice-agents.mdx | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx b/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx index 2e22780d..75a0940c 100644 --- a/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx +++ b/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx @@ -30,6 +30,10 @@ Speaker memory is an application of [speaker identification](/speech-to-text/fea A speaker identifier is a voice fingerprint the engine produces once diarization is on. It is a string, tied to your Speechmatics account. That last point matters: identifiers are unique per account and do not work across accounts, so you cannot take a fingerprint from one account and use it under another. +:::warning +Speechmatics does not store speaker identifiers. They exist only for the session that produces them, so you must save them yourself and supply them again in every future session. Identifiers are derived from a person's voice and may qualify as biometric data under data-protection law (for example, GDPR). Treat them as sensitive personal data: obtain consent, store them securely, and support deletion. +::: + Speaker memory and [speaker identification](/speech-to-text/features/speaker-identification) use the same identifiers for different jobs. Speaker identification enrolls known speakers ahead of time from short audio clips and applies them in batch or realtime transcription. Speaker memory captures identifiers live during an agent session and reuses them in later sessions, so an agent can recognize a returning speaker without prior enrollment. A speaker can accumulate more than one identifier over time, as the engine hears them in different conditions. A saved profile is therefore a name plus a list of identifiers, and you keep adding to that list as you see the person again. More identifiers mean better recognition. @@ -66,7 +70,7 @@ Some integrations let you flag the request as final, meaning "give me the defini Saving the identifiers is up to you. A local file works and keeps things offline, but a database or a user record is just as valid. -To use them, pass them as known speakers when you open the next session. Each entry pairs a label (now a real name) with a list of identifiers. You can pass several identifiers under one label: this is how you cover the same voice heard through different devices or microphones (a laptop mic, a smartphone, an tablet, a handset), so recognition holds up wherever the person speaks from. +To use them, pass them as known speakers when you open the next session. Each entry pairs a label (now a real name) with a list of identifiers. You can pass several identifiers under one label: this is how you cover the same voice heard through different devices or microphones (a laptop mic, a smartphone, a tablet, a handset), so recognition holds up wherever the person speaks from. The following pseudocode configures known speakers at session start: From 08340147128b821307d826d39cab1ad79966c057 Mon Sep 17 00:00:00 2001 From: Edgars Adamovics Date: Tue, 21 Jul 2026 11:47:15 +0100 Subject: [PATCH 8/9] add foundational speaker identifiers doc and refactor related content Introduces a new page, "How speaker identifiers work," to centralize explanations of speaker identifiers, their scope, and persistence. This allows the "Speaker memory for voice agents" page to focus on the voice agent application pattern, reducing redundancy. Also includes minor improvements to the "Speaker focus for voice agents" page, like adding a title and code block line numbers. --- .../features/how-speaker-identifiers-work.mdx | 44 +++++++ .../speaker-focus-for-voice-agents.mdx | 17 +-- .../speaker-memory-for-voice-agents.mdx | 107 ++++++++---------- 3 files changed, 99 insertions(+), 69 deletions(-) create mode 100644 docs/speech-to-text/features/how-speaker-identifiers-work.mdx diff --git a/docs/speech-to-text/features/how-speaker-identifiers-work.mdx b/docs/speech-to-text/features/how-speaker-identifiers-work.mdx new file mode 100644 index 00000000..c41f72af --- /dev/null +++ b/docs/speech-to-text/features/how-speaker-identifiers-work.mdx @@ -0,0 +1,44 @@ +--- +title: How speaker identifiers work +description: Understand how the Speechmatics engine produces speaker identifiers and how they differ from diarization labels. +keywords: + [ + speechmatics, + speaker identifiers, + speaker identification, + diarization, + realtime, + batch, + ] +sidebar_position: 4 +--- + +# How speaker identifiers work + +Understand how the Speechmatics engine produces speaker identifiers and how they differ from diarization labels. + +Diarization assigns per-session labels: `S1`, `S2`, `S3`. These labels last only for one session. The person who is `S1` in one session may be `S2` in the next. Speaker identifiers give those speakers durable memory. During a session the engine builds a voice fingerprint for each speaker, and you can save those fingerprints as identifiers. Supplied to a later session, the engine attributes matching voices to a name you chose, such as `Sam` instead of `S1`. + +## What a speaker identifier is + +A speaker identifier is a voice fingerprint the engine produces once diarization is enabled. It is a string, unique per Speechmatics account. Identifiers do not work across accounts: a fingerprint created under one account cannot be used under another. + +:::info +Speaker identifiers and `known_speakers` are available in [realtime](/speech-to-text/realtime/speaker-identification) and [batch](/speech-to-text/batch/speaker-identification) speaker identification. The exact names vary by client. +::: + +A speaker can accumulate more than one identifier over time, as the engine hears them in different conditions. A saved profile is therefore a name plus a list of identifiers, and you extend that list each time you encounter the speaker. More identifiers improve recognition. + +## Speaker memory compared with speaker identification + +Speaker memory and [speaker identification](/speech-to-text/features/speaker-identification) use the same identifiers for different jobs. + +| Capability | When identifiers are captured | Typical use | +|---|---|---| +| Speaker identification | Enrolled ahead of time from short audio clips | Apply known speakers in batch or realtime transcription | +| Speaker memory | Captured live during an agent session | Recognize a returning speaker without prior enrollment | + +## Next steps + +- [Speaker memory for voice agents](/speech-to-text/features/speaker-memory-for-voice-agents) — capture and reuse identifiers across sessions. +- [Speaker identification](/speech-to-text/features/speaker-identification) — enroll known speakers ahead of time. diff --git a/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx b/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx index f4090e27..e4043865 100644 --- a/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx +++ b/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx @@ -1,4 +1,5 @@ --- +title: "Speaker focus for voice agents" description: "Direct a voice agent to respond to chosen speakers and treat others as background, and update the focus live mid-session." keywords: [ @@ -22,7 +23,7 @@ Speaker focus lets a voice agent decide which speakers to respond to and how to Put a voice agent in a room with more than one person and you hit a problem straight away: whose words should the agent act on? If two people are talking and a third asks a question, you don't want the agent replying to the crossfire. You want it locked onto the person talking to it, with everyone else treated as background it can hear but shouldn't answer. Speaker focus is how you do that. :::info -Speaker focus is a helper provided by the Speechmatics Python [voice SDK](/voice-agents/voice-sdk), which applies the focus rules on top of speaker [diarization](/speech-to-text/features/diarization) (labeling every word with a speaker: `S1`, `S2`, and so on). The Realtime API does not expose speaker focus directly today, and the voice SDK is Python only. Speechmatics provides the speech-to-text layer for voice agents, not the full pipeline. +Speaker focus is a helper provided by the Speechmatics Python [voice SDK](/voice-agents/voice-sdk), which applies the focus rules on top of speaker [diarization](/speech-to-text/features/diarization). The Realtime API does not expose speaker focus directly today, and the voice SDK is Python only. Speechmatics provides the speech-to-text layer for voice agents, not the full pipeline. ::: ## Focus and focus mode @@ -42,11 +43,11 @@ Any speaker whose label starts and ends with double underscores (such as `__ASSI ## Set and update focus -You set an initial focus when you open the session, and you can update it at any point while the session runs. The two format strings decide how active and passive speakers are rendered into the transcript the LLM reads. +You set an initial focus when you open the session, and you can update it at any point while the session runs. The two format strings decide how active and passive speakers are rendered into the transcript the LLM (large language model) reads. The following pseudocode configures focus at session start and updates it live: -```python +```python showLineNumbers # At session start, with diarization on. The format strings control how # active and passive speaker lines are rendered for the LLM. configure_stt( @@ -72,7 +73,7 @@ Rather than a human flipping switches, you hand the LLM a small set of function The following pseudocode defines a sensible tool set: -```python +```python showLineNumbers focus_on_speaker(speaker_ids: string[]) # Prioritize one or more speakers. Everyone else stays audible as # background (RETAIN). Replaces any current focus. @@ -111,7 +112,7 @@ Third, the labels are internal. The agent must never read `S1` or `S2` aloud, an The following prompt fragment covers the speaker rules: -```text +```text showLineNumbers # Speakers - Each message is prefixed with the speaker, like @S1: or, for a @@ -149,7 +150,7 @@ The concepts above map onto real integrations with little code. The Speechmatics For the voice SDK, set the focus on the session config and update it live with `update_diarization_config`: -```python +```python showLineNumbers from speechmatics.voice import SpeakerFocusConfig, SpeakerFocusMode # Session setup: set focus on the transcription config's speaker_config @@ -166,7 +167,7 @@ client.update_diarization_config( For [Pipecat](https://github.com/pipecat-ai/pipecat), configure focus on the STT service and update it inside a tool handler: -```python +```python showLineNumbers # Session setup stt = SpeechmaticsSTTService( params=SpeechmaticsSTTService.InputParams( @@ -187,7 +188,7 @@ stt.update_params( For the [LiveKit Speechmatics plugin](https://github.com/livekit/agents/tree/main/livekit-plugins/livekit-plugins-speechmatics), use `update_speakers`: -```python +```python showLineNumbers # Session setup stt = speechmatics.STT( enable_diarization=True, diff --git a/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx b/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx index 75a0940c..c1b984e3 100644 --- a/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx +++ b/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx @@ -1,53 +1,36 @@ --- -description: "Persist speaker identifiers so a voice agent recognizes returning speakers by name across sessions." +title: Speaker memory for voice agents +description: Persist speaker identifiers so a voice agent recognizes returning speakers by name across sessions. keywords: [ speechmatics, - features, speaker identification, speaker identifiers, diarization, voice agents, realtime, - speech recognition, - automatic speech recognition, - asr, ] sidebar_position: 5 --- # Speaker memory for voice agents -Speaker memory lets a voice agent recognize a returning speaker by name across sessions, using durable speaker identifiers. +Persist speaker identifiers so a voice agent recognizes a returning speaker by name across sessions. -Diarization gives you labels: `S1`, `S2`, `S3`. Those labels are only good for the life of one session. The person who was `S1` today might be `S2` tomorrow, or `S1` again by chance. The labels carry no memory. Speaker identifiers are how you give them memory. During a session the engine builds a voice fingerprint for each speaker, and you can save those fingerprints as durable identifiers. Feed them back into a later session and the engine attributes matching voices to a name you chose, `Sam` instead of `S1`. - -:::info -Speaker memory is an application of [speaker identification](/speech-to-text/features/speaker-identification), a standard capability of the Realtime and Batch APIs and their clients, not a voice-specific feature. Speaker identifiers and `known_speakers` are available in [realtime](/speech-to-text/realtime/speaker-identification) and [batch](/speech-to-text/batch/speaker-identification) speaker identification; the exact names vary by client. This page covers the voice-agent pattern: capturing identifiers live and reusing them across sessions, with speaker [diarization](/speech-to-text/features/diarization) enabled. -::: - -## How speaker identifiers work - -A speaker identifier is a voice fingerprint the engine produces once diarization is on. It is a string, tied to your Speechmatics account. That last point matters: identifiers are unique per account and do not work across accounts, so you cannot take a fingerprint from one account and use it under another. +Speaker memory is an application of [speaker identification](/speech-to-text/features/speaker-identification), a standard capability of the Realtime and Batch APIs, not a voice-specific feature. This page covers the voice-agent pattern: capturing identifiers live and reusing them across sessions with [diarization](/speech-to-text/features/diarization) enabled. For how identifiers are produced and scoped, see [How speaker identifiers work](/speech-to-text/features/how-speaker-identifiers-work). :::warning -Speechmatics does not store speaker identifiers. They exist only for the session that produces them, so you must save them yourself and supply them again in every future session. Identifiers are derived from a person's voice and may qualify as biometric data under data-protection law (for example, GDPR). Treat them as sensitive personal data: obtain consent, store them securely, and support deletion. +Speechmatics does not store speaker identifiers. Save them yourself and supply them again in every future session. Identifiers are derived from a person's voice and may qualify as biometric data under data-protection law such as GDPR. Obtain consent, store them securely, and support deletion. ::: -Speaker memory and [speaker identification](/speech-to-text/features/speaker-identification) use the same identifiers for different jobs. Speaker identification enrolls known speakers ahead of time from short audio clips and applies them in batch or realtime transcription. Speaker memory captures identifiers live during an agent session and reuses them in later sessions, so an agent can recognize a returning speaker without prior enrollment. - -A speaker can accumulate more than one identifier over time, as the engine hears them in different conditions. A saved profile is therefore a name plus a list of identifiers, and you keep adding to that list as you see the person again. More identifiers mean better recognition. - -There are two halves to speaker memory: getting identifiers out of a live session, and feeding them back into a later one. - ## Get identifiers from a live session -While a session is running, you request the current speakers. The engine replies with every speaker it has tracked, each with its label and its identifiers. +While a session is running, you request the current speakers. The engine returns every speaker it has tracked, each with its label and identifiers. The following pseudocode requests identifiers and persists the ones you want: ```python -# Ask the engine for the fingerprints of everyone heard so far. +# Request the fingerprints of every speaker heard so far speakers = request_speakers() # speakers looks like: @@ -56,21 +39,21 @@ speakers = request_speakers() # { label: "S2", speaker_identifiers: ["YY...YY"] }, # ] -# Keep the ones you care about and persist them (your storage): +# Filter to the speakers you want, then persist to your storage for s in speakers: if s.label in wanted: save_profile(name = wanted[s.label], identifiers = s.speaker_identifiers) ``` -The engine returns everyone, so filter down to the speakers you want on your side. Recognition quality improves once each speaker has said a handful of words (five or so is a good rule of thumb), so do not fire the request the instant someone starts talking. +The engine returns everyone, so filter to the speakers you want on your side. Recognition quality improves once each speaker has said around five words, so do not request identifiers the instant someone starts talking. -Some integrations let you flag the request as final, meaning "give me the definitive fingerprints computed over the whole session." That is the best quality you will get and the natural thing to do at the end of a call. A mid-session request without the flag gives you whatever the engine has so far, which is fine for an interim save. +Some integrations let you flag the request as final, returning the definitive fingerprints computed over the whole session. This gives the best quality and suits the end of a call. A mid-session request without the flag returns whatever the engine has so far, which is sufficient for an interim save. ## Reuse identifiers in a later session -Saving the identifiers is up to you. A local file works and keeps things offline, but a database or a user record is just as valid. +Storage is your responsibility. A local file, a database, or a user record all work. -To use them, pass them as known speakers when you open the next session. Each entry pairs a label (now a real name) with a list of identifiers. You can pass several identifiers under one label: this is how you cover the same voice heard through different devices or microphones (a laptop mic, a smartphone, a tablet, a handset), so recognition holds up wherever the person speaks from. +To reuse identifiers, pass them as known speakers when you open the next session. Each entry pairs a label, now a real name, with a list of identifiers. Passing several identifiers under one label covers the same voice heard through different devices, so recognition holds up wherever the person speaks from. The following pseudocode configures known speakers at session start: @@ -78,61 +61,60 @@ The following pseudocode configures known speakers at session start: configure_stt( enable_diarization = true, known_speakers = [ - # One label, several identifiers for the same voice across devices. + # One label, several identifiers for the same voice across devices { label: "Sam", speaker_identifiers: ["XX...XX", "ZZ...ZZ"] }, { label: "Alice", speaker_identifiers: ["YY...YY"] }, ], ) ``` -From that point the engine attributes matching voices to `Sam` and `Alice` instead of `S1` and `S2`. Anyone it does not recognize still gets a fresh generic label, so known and unknown speakers coexist. +The engine attributes matching voices to `Sam` and `Alice` instead of `S1` and `S2`. Anyone it does not recognize gets a fresh generic label, so known and unknown speakers coexist. ## Drive voice memory from the agent -As with [speaker focus](/speech-to-text/features/speaker-focus-for-voice-agents), you can give the LLM a couple of tools and let it manage voice memory through conversation. +As with [speaker focus](/speech-to-text/features/speaker-focus-for-voice-agents), you can give the LLM tools and let it manage voice memory through conversation. The following pseudocode defines the tools: ```python +# Save one or more speakers for recognition in future sessions remember_voices(speakers: { speaker_id: string, name: string }[]) - # Save one or more speakers so they're recognized in future sessions. - # Only call when a speaker explicitly asks to be saved. The name must - # be their real name, never a generic label like "S1". Pass everyone - # to save in a single call. "remember me, I'm Sam" -> remember_voices([{ speaker_id: "S1", name: "Sam" }]) +# Wipe every saved profile forget_all_voices() - # Wipe every saved profile. "clear voice memory" -> forget_all_voices() ``` -The `remember_voices` handler needs care, because fetching fingerprints is asynchronous. Do not block the agent's reply waiting for them. The clean pattern is to stash the name mappings, fire the request, reply straight away, and save when the result arrives. +The `remember_voices` handler needs care, because fetching fingerprints is asynchronous. Do not block the agent's reply waiting for them. Stash the name mappings, fire the request, reply immediately, and save when the result arrives. -The following pseudocode shows that non-blocking handler: +The following pseudocode shows the non-blocking handler: ```python pending = {} # label -> real name, awaiting the speakers result on remember_voices(speakers): + # Record the name mappings and request fingerprints without blocking for entry in speakers: pending[entry.speaker_id] = entry.name - request_speakers() # result handled below - reply("Voice profile saved.") # don't wait + request_speakers() + reply("Voice profile saved.") on speakers_result(speakers): + # Match returned identifiers to pending names and save for s in speakers: name = pending.pop(s.label, none) if name: save_profile(name, s.speaker_identifiers) # merge, don't overwrite ``` -Merge new identifiers into an existing profile of the same name rather than overwriting, so recognition improves each time. Snapshot then clear the pending map before working through the result, so overlapping requests do not double-process. +Merge new identifiers into an existing profile of the same name rather than overwriting, so recognition improves each time. Snapshot and clear the pending map before working through the result, so overlapping requests do not double-process. -## System prompt for voice memory +## Write the system prompt -The tools need guardrails in the prompt. The main one: do not save people just because they mention their name. Someone saying "I'm Sam, anyway, as I was saying" is not asking to be remembered. Only call `remember_voices` on an explicit request, such as "remember me" or "save my voice." +The tools need guardrails in the prompt. Do not save people because they mention their name: someone saying "I'm Sam, anyway" is not asking to be remembered. Call `remember_voices` only on an explicit request such as "remember me" or "save my voice." -The other rule is that the name must be a real name. `S1` is a label, not a name, and saving a profile called `S1` is worse than useless. If the agent does not know someone's name, it should ask before saving. +The name must be a real name. `S1` is a label, not a name. If the agent does not know someone's name, it should ask before saving. The following prompt fragment covers the memory rules: @@ -148,22 +130,25 @@ The following prompt fragment covers the memory rules: - "Forget all voices" or "clear voice memory" -> forget_all_voices. ``` -## Example: remembering a speaker across sessions +## Remember a speaker across sessions -This walks through two sessions. +This example walks through two sessions. -**Session one.** Sam chats to the agent as `S1` and says "remember me, I'm Sam." The LLM calls `remember_voices([{ speaker_id: "S1", name: "Sam" }])`. The handler stashes `S1 -> Sam`, requests the speakers, and replies. The result comes back, the handler finds `S1`'s identifiers, and saves the profile `{ "label": "Sam", "speaker_identifiers": ["XX...XX"] }`. +1. Sam chats to the agent as `S1` and says "remember me, I'm Sam." +2. The LLM calls `remember_voices([{ speaker_id: "S1", name: "Sam" }])`. +3. The handler stashes `S1 -> Sam`, requests the speakers, and replies. +4. The result returns, the handler finds `S1`'s identifiers, and saves the profile `{ "label": "Sam", "speaker_identifiers": ["XX...XX"] }`. -**Session two.** On startup you load that profile and pass it as a known speaker. Sam speaks, the engine matches his voice, and his transcript arrives as `@Sam: hello again`. The agent greets him by name, and because he is now a named speaker you can point [speaker focus](/speech-to-text/features/speaker-focus-for-voice-agents) at `Sam` directly. +On the next session startup, you load that profile and pass it as a known speaker. Sam speaks, the engine matches his voice, and his transcript arrives as `@Sam: hello again`. Because he is now a named speaker, you can point [speaker focus](/speech-to-text/features/speaker-focus-for-voice-agents) at `Sam` directly. -## Speaker memory in the voice SDK, Pipecat and LiveKit +## Use speaker memory in the voice SDK, Pipecat, and LiveKit -The concepts above map onto real integrations with little code: get identifiers out of one session, feed them back into the next as known speakers. `known_speakers` is available across the Realtime and Batch clients; the [voice SDK](/voice-agents/voice-sdk), Pipecat, and LiveKit expose it for the agent use case, with method names differing per client. +The pattern maps onto real integrations with little code: get identifiers out of one session, feed them back into the next as known speakers. `known_speakers` is available across the Realtime and Batch clients; the [voice SDK](/voice-agents/voice-sdk), Pipecat, and LiveKit expose it for the agent use case, with method names differing per client. For [Pipecat](https://github.com/pipecat-ai/pipecat), the result arrives on an event: ```python -# Request fingerprints, then save them when they arrive +# Request fingerprints, then save them when the event fires await stt.send_message("GetSpeakers") @stt.event_handler("on_speakers_result") @@ -187,7 +172,7 @@ stt = SpeechmaticsSTTService( For the [LiveKit Speechmatics plugin](https://github.com/livekit/agents/tree/main/livekit-plugins/livekit-plugins-speechmatics), the request is a single awaitable: ```python -# Request fingerprints (waits for the result, with a short timeout) +# Request fingerprints, waiting for the result with a short timeout speakers = await stt.get_speaker_ids() for s in speakers: save_profile(s.label, s.speaker_identifiers) @@ -201,19 +186,19 @@ stt = speechmatics.STT( ) ``` -## Best practices for speaker memory +## Best practices -Identifiers are per account. You cannot lift a fingerprint from one Speechmatics account and use it on another, so voice profiles belong to the account that created them. +Identifiers are per account. You cannot use a fingerprint from one Speechmatics account on another, so voice profiles belong to the account that created them. -Quality improves with more identifiers. Rather than treating a profile as fixed, append new identifiers each time you save a returning speaker. +Append new identifiers each time you save a returning speaker rather than treating a profile as fixed. More identifiers improve recognition. -Request fingerprints once there are a few words to work with, and request them again at the end of a call with the final flag if the integration offers it. That gives you identifiers computed over everything the engine heard, which recognizes better next time than an early snapshot. +Request fingerprints once there are a few words to work with, and request them again at the end of a call with the final flag if the integration offers it. This returns identifiers computed over everything the engine heard. -Speaker memory pairs with [speaker focus](/speech-to-text/features/speaker-focus-for-voice-agents). Once a voice is recognized as `Sam`, every focus and ignore operation can target `Sam` by name instead of a throwaway label. +Speaker memory pairs with [speaker focus](/speech-to-text/features/speaker-focus-for-voice-agents). Once a voice is recognized as `Sam`, every focus and ignore operation can target `Sam` by name. ## Next steps - [Speaker focus for voice agents](/speech-to-text/features/speaker-focus-for-voice-agents) — direct the agent to specific speakers. -- [Speaker identification](/speech-to-text/features/speaker-identification) — how identifiers are generated and scoped. -- [Voice agents overview](/voice-agents/overview) — ways to build a voice agent with Speechmatics. -- [Voice SDK](/voice-agents/voice-sdk) — the Python SDK reference for `known_speakers` and speaker management. +- [How speaker identifiers work](/speech-to-text/features/how-speaker-identifiers-work) — how identifiers are generated and scoped. +- [Voice agents overview](/voice-agents/overview) — ways to build a voice agent. +- [Voice SDK](/voice-agents/voice-sdk) — Python SDK reference for `known_speakers`. From 098303e9fdd8bc432d97a3869a940369bb813bde Mon Sep 17 00:00:00 2001 From: Edgars Adamovics Date: Tue, 21 Jul 2026 12:05:33 +0100 Subject: [PATCH 9/9] Refine speaker focus and memory documentation Enhances the presentation and clarity of speaker-related documentation pages. Updates `sidebar_label` and `sidebar_position` for improved navigation within the docs. Standardizes "Voice SDK" capitalization and adds line numbers to code examples for better readability. Includes minor wording improvements and adds "awaitable" to the custom words list. --- custom-words.txt | 1 + .../features/how-speaker-identifiers-work.mdx | 1 + .../speaker-focus-for-voice-agents.mdx | 23 +++++++++++-------- .../speaker-memory-for-voice-agents.mdx | 21 +++++++++-------- 4 files changed, 27 insertions(+), 19 deletions(-) diff --git a/custom-words.txt b/custom-words.txt index b0dfc137..45d28d3b 100644 --- a/custom-words.txt +++ b/custom-words.txt @@ -343,3 +343,4 @@ melia مرحبا Theo pipecat +awaitable diff --git a/docs/speech-to-text/features/how-speaker-identifiers-work.mdx b/docs/speech-to-text/features/how-speaker-identifiers-work.mdx index c41f72af..f9446225 100644 --- a/docs/speech-to-text/features/how-speaker-identifiers-work.mdx +++ b/docs/speech-to-text/features/how-speaker-identifiers-work.mdx @@ -1,5 +1,6 @@ --- title: How speaker identifiers work +sidebar_label: Speaker identifiers description: Understand how the Speechmatics engine produces speaker identifiers and how they differ from diarization labels. keywords: [ diff --git a/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx b/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx index e4043865..18275324 100644 --- a/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx +++ b/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx @@ -1,5 +1,6 @@ --- title: "Speaker focus for voice agents" +sidebar_label: "Speaker focus" description: "Direct a voice agent to respond to chosen speakers and treat others as background, and update the focus live mid-session." keywords: [ @@ -13,17 +14,17 @@ keywords: automatic speech recognition, asr, ] -sidebar_position: 4 +sidebar_position: 5 --- # Speaker focus for voice agents Speaker focus lets a voice agent decide which speakers to respond to and how to treat everyone else, and update that choice live during a session. -Put a voice agent in a room with more than one person and you hit a problem straight away: whose words should the agent act on? If two people are talking and a third asks a question, you don't want the agent replying to the crossfire. You want it locked onto the person talking to it, with everyone else treated as background it can hear but shouldn't answer. Speaker focus is how you do that. +A voice agent in a room with more than one person faces an immediate problem: whose words should it act on? If two people are talking and a third asks a question, the agent should not reply to the crossfire. It should stay locked onto the person addressing it, with everyone else treated as background it can hear but should not answer. Speaker focus provides that control. :::info -Speaker focus is a helper provided by the Speechmatics Python [voice SDK](/voice-agents/voice-sdk), which applies the focus rules on top of speaker [diarization](/speech-to-text/features/diarization). The Realtime API does not expose speaker focus directly today, and the voice SDK is Python only. Speechmatics provides the speech-to-text layer for voice agents, not the full pipeline. +Speaker focus is a helper provided by the Speechmatics Python [Voice SDK](/voice-agents/voice-sdk), which applies the focus rules on top of speaker [diarization](/speech-to-text/features/diarization). The Realtime API does not expose speaker focus directly today, and the Voice SDK is Python only. Speechmatics provides the speech-to-text layer for voice agents, not the full pipeline. ::: ## Focus and focus mode @@ -63,7 +64,7 @@ ignore_speaker(add = "S2") # separate ignore list set_speaker_focus(focus = [], mode = RETAIN) # reset, hear everyone ``` -The format strings matter more than they look. The active speaker's line arrives as `@S1: can you help me`, and a background speaker as `@S2 [background]: yeah but what about lunch`. That `[background]` tag is the signal your system prompt teaches the LLM to ignore. +The format strings determine what the LLM sees. The active speaker's line arrives as `@S1: can you help me`, and a background speaker as `@S2 [background]: yeah but what about lunch`. That `[background]` tag is the signal your system prompt teaches the LLM to ignore. A live focus update replaces the previous focus, it does not merge. To build up a multi-speaker focus one speaker at a time, track the set yourself and send the full list each time. @@ -106,7 +107,7 @@ The tools do nothing without a system prompt that teaches the LLM how to read th First, the LLM must understand the speaker tags. Every incoming line is prefixed, either `@S1:` for a raw label or `@Sam:` once a speaker is recognized by name. Lines carrying `[background]` are passive and should be left alone unless an active speaker brings that person in. -Second, the LLM has to resolve "me." When someone says "focus on me," the word "me" means whoever is speaking, which is the ID prefixing their own message. A "focus on me" arriving on a `@S1:` line resolves to `S1`. Without this rule the model guesses, and it guesses badly. +Second, the LLM has to resolve "me." When someone says "focus on me," the word "me" means whoever is speaking, which is the ID prefixing their own message. A "focus on me" arriving on a `@S1:` line resolves to `S1`. Without this rule the model has to guess, and often picks the wrong speaker. Third, the labels are internal. The agent must never read `S1` or `S2` aloud, and never echo a speaker tag into its reply. Real names are fine once known. @@ -144,11 +145,11 @@ This walks through "just listen to me" from a speaker labeled `S1`, with someone Saying "listen to everyone again" runs step 3 in reverse: empty focus, empty ignore list, mode back to `RETAIN`, everyone audible. -## Speaker focus in the voice SDK, Pipecat and LiveKit +## Speaker focus in the Voice SDK, Pipecat and LiveKit -The concepts above map onto real integrations with little code. The Speechmatics Python [voice SDK](/voice-agents/voice-sdk) exposes the two levers as `SpeakerFocusConfig` and `SpeakerFocusMode`; the Pipecat and LiveKit plugins wrap the same SDK under their own method names. +The concepts above map onto real integrations with little code. The Speechmatics Python [Voice SDK](/voice-agents/voice-sdk) exposes the two levers as `SpeakerFocusConfig` and `SpeakerFocusMode`; the Pipecat and LiveKit plugins wrap the same SDK under their own method names. -For the voice SDK, set the focus on the session config and update it live with `update_diarization_config`: +For the Voice SDK, set the focus on the session config and update it live with `update_diarization_config`: ```python showLineNumbers from speechmatics.voice import SpeakerFocusConfig, SpeakerFocusMode @@ -209,7 +210,11 @@ Pick format strings to suit your model, and teach the LLM whichever format you c `RETAIN` is the safer default for a chatty agent, because a background speaker can be brought in without a config change. `IGNORE` is blunter. Reach for it when you truly want the engine to stop listening, not just stop responding. -Keep the agent's own voice out of the loop, otherwise its audio output is transcribed straight back in and it starts talking to itself. Any speaker enrolled with a label wrapped in double underscores (such as `__ASSISTANT__`) is excluded automatically, so you never have to ignore it explicitly. To use this, pass the agent's voice as a [known speaker](/speech-to-text/features/speaker-memory-for-voice-agents) at session start, labeled `__ASSISTANT__`, with a speaker identifier captured from an earlier session or from enrollment. The engine recognizes the agent's own audio and drops it before it reaches the transcript. This is the clean option for complex speaker or microphone setups, where the agent's voice can leak back in through the mic. You can pass several identifiers under the one label to cover that voice across different output paths or capture devices. Adding the agent's speaker to the ignore list at runtime also works. +Keep the agent's own voice out of the loop, otherwise its audio output is transcribed straight back in and it starts talking to itself. Any speaker enrolled with a label wrapped in double underscores (such as `__ASSISTANT__`) is excluded automatically, so you never have to ignore it explicitly. + +To use this, pass the agent's voice as a [known speaker](/speech-to-text/features/speaker-memory-for-voice-agents) at session start, labeled `__ASSISTANT__`, with a speaker identifier captured from an earlier session or from enrollment. The engine recognizes the agent's own audio and drops it before it reaches the transcript. You can pass several identifiers under the one label to cover that voice across different output paths or capture devices. + +This is the clean option for complex speaker or microphone setups, where the agent's voice can leak back in through the mic. Adding the agent's speaker to the ignore list at runtime also works. Speaker focus pairs with [speaker memory](/speech-to-text/features/speaker-memory-for-voice-agents). Once a voice is recognized by name, every focus and ignore operation can target that name instead of a throwaway label. diff --git a/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx b/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx index c1b984e3..652376e6 100644 --- a/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx +++ b/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx @@ -1,5 +1,6 @@ --- title: Speaker memory for voice agents +sidebar_label: Speaker memory description: Persist speaker identifiers so a voice agent recognizes returning speakers by name across sessions. keywords: [ @@ -10,7 +11,7 @@ keywords: voice agents, realtime, ] -sidebar_position: 5 +sidebar_position: 6 --- # Speaker memory for voice agents @@ -29,7 +30,7 @@ While a session is running, you request the current speakers. The engine returns The following pseudocode requests identifiers and persists the ones you want: -```python +```python showLineNumbers # Request the fingerprints of every speaker heard so far speakers = request_speakers() @@ -57,7 +58,7 @@ To reuse identifiers, pass them as known speakers when you open the next session The following pseudocode configures known speakers at session start: -```python +```python showLineNumbers configure_stt( enable_diarization = true, known_speakers = [ @@ -76,7 +77,7 @@ As with [speaker focus](/speech-to-text/features/speaker-focus-for-voice-agents) The following pseudocode defines the tools: -```python +```python showLineNumbers # Save one or more speakers for recognition in future sessions remember_voices(speakers: { speaker_id: string, name: string }[]) "remember me, I'm Sam" -> remember_voices([{ speaker_id: "S1", name: "Sam" }]) @@ -90,7 +91,7 @@ The `remember_voices` handler needs care, because fetching fingerprints is async The following pseudocode shows the non-blocking handler: -```python +```python showLineNumbers pending = {} # label -> real name, awaiting the speakers result on remember_voices(speakers): @@ -118,7 +119,7 @@ The name must be a real name. `S1` is a label, not a name. If the agent does not The following prompt fragment covers the memory rules: -```text +```text showLineNumbers # Speaker memory - Only call remember_voices when a speaker explicitly asks to be saved, @@ -141,13 +142,13 @@ This example walks through two sessions. On the next session startup, you load that profile and pass it as a known speaker. Sam speaks, the engine matches his voice, and his transcript arrives as `@Sam: hello again`. Because he is now a named speaker, you can point [speaker focus](/speech-to-text/features/speaker-focus-for-voice-agents) at `Sam` directly. -## Use speaker memory in the voice SDK, Pipecat, and LiveKit +## Use speaker memory in the Voice SDK, Pipecat, and LiveKit -The pattern maps onto real integrations with little code: get identifiers out of one session, feed them back into the next as known speakers. `known_speakers` is available across the Realtime and Batch clients; the [voice SDK](/voice-agents/voice-sdk), Pipecat, and LiveKit expose it for the agent use case, with method names differing per client. +The pattern maps onto real integrations with little code: get identifiers out of one session, feed them back into the next as known speakers. `known_speakers` is available across the Realtime and Batch clients; the [Voice SDK](/voice-agents/voice-sdk), Pipecat, and LiveKit expose it for the agent use case, with method names differing per client. For [Pipecat](https://github.com/pipecat-ai/pipecat), the result arrives on an event: -```python +```python showLineNumbers # Request fingerprints, then save them when the event fires await stt.send_message("GetSpeakers") @@ -171,7 +172,7 @@ stt = SpeechmaticsSTTService( For the [LiveKit Speechmatics plugin](https://github.com/livekit/agents/tree/main/livekit-plugins/livekit-plugins-speechmatics), the request is a single awaitable: -```python +```python showLineNumbers # Request fingerprints, waiting for the result with a short timeout speakers = await stt.get_speaker_ids() for s in speakers: