diff --git a/.gitignore b/.gitignore index 81ea2715..bcfc34f4 100644 --- a/.gitignore +++ b/.gitignore @@ -28,4 +28,7 @@ static/*.yaml .vercel .env*.local -.venv \ No newline at end of file +.venv + +# Temporary / local dev +/tmp diff --git a/custom-words.txt b/custom-words.txt index f00dfb5b..45d28d3b 100644 --- a/custom-words.txt +++ b/custom-words.txt @@ -342,3 +342,5 @@ sessiongroups melia مرحبا Theo +pipecat +awaitable diff --git a/docs/speech-to-text/features/how-speaker-identifiers-work.mdx b/docs/speech-to-text/features/how-speaker-identifiers-work.mdx new file mode 100644 index 00000000..f9446225 --- /dev/null +++ b/docs/speech-to-text/features/how-speaker-identifiers-work.mdx @@ -0,0 +1,45 @@ +--- +title: How speaker identifiers work +sidebar_label: Speaker identifiers +description: Understand how the Speechmatics engine produces speaker identifiers and how they differ from diarization labels. +keywords: + [ + speechmatics, + speaker identifiers, + speaker identification, + diarization, + realtime, + batch, + ] +sidebar_position: 4 +--- + +# How speaker identifiers work + +Understand how the Speechmatics engine produces speaker identifiers and how they differ from diarization labels. + +Diarization assigns per-session labels: `S1`, `S2`, `S3`. These labels last only for one session. The person who is `S1` in one session may be `S2` in the next. Speaker identifiers give those speakers durable memory. During a session the engine builds a voice fingerprint for each speaker, and you can save those fingerprints as identifiers. Supplied to a later session, the engine attributes matching voices to a name you chose, such as `Sam` instead of `S1`. + +## What a speaker identifier is + +A speaker identifier is a voice fingerprint the engine produces once diarization is enabled. It is a string, unique per Speechmatics account. Identifiers do not work across accounts: a fingerprint created under one account cannot be used under another. + +:::info +Speaker identifiers and `known_speakers` are available in [realtime](/speech-to-text/realtime/speaker-identification) and [batch](/speech-to-text/batch/speaker-identification) speaker identification. The exact names vary by client. +::: + +A speaker can accumulate more than one identifier over time, as the engine hears them in different conditions. A saved profile is therefore a name plus a list of identifiers, and you extend that list each time you encounter the speaker. More identifiers improve recognition. + +## Speaker memory compared with speaker identification + +Speaker memory and [speaker identification](/speech-to-text/features/speaker-identification) use the same identifiers for different jobs. + +| Capability | When identifiers are captured | Typical use | +|---|---|---| +| Speaker identification | Enrolled ahead of time from short audio clips | Apply known speakers in batch or realtime transcription | +| Speaker memory | Captured live during an agent session | Recognize a returning speaker without prior enrollment | + +## Next steps + +- [Speaker memory for voice agents](/speech-to-text/features/speaker-memory-for-voice-agents) — capture and reuse identifiers across sessions. +- [Speaker identification](/speech-to-text/features/speaker-identification) — enroll known speakers ahead of time. diff --git a/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx b/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx new file mode 100644 index 00000000..18275324 --- /dev/null +++ b/docs/speech-to-text/features/speaker-focus-for-voice-agents.mdx @@ -0,0 +1,226 @@ +--- +title: "Speaker focus for voice agents" +sidebar_label: "Speaker focus" +description: "Direct a voice agent to respond to chosen speakers and treat others as background, and update the focus live mid-session." +keywords: + [ + speechmatics, + features, + speaker focus, + diarization, + voice agents, + realtime, + speech recognition, + automatic speech recognition, + asr, + ] +sidebar_position: 5 +--- + +# Speaker focus for voice agents + +Speaker focus lets a voice agent decide which speakers to respond to and how to treat everyone else, and update that choice live during a session. + +A voice agent in a room with more than one person faces an immediate problem: whose words should it act on? If two people are talking and a third asks a question, the agent should not reply to the crossfire. It should stay locked onto the person addressing it, with everyone else treated as background it can hear but should not answer. Speaker focus provides that control. + +:::info +Speaker focus is a helper provided by the Speechmatics Python [Voice SDK](/voice-agents/voice-sdk), which applies the focus rules on top of speaker [diarization](/speech-to-text/features/diarization). The Realtime API does not expose speaker focus directly today, and the Voice SDK is Python only. Speechmatics provides the speech-to-text layer for voice agents, not the full pipeline. +::: + +## Focus and focus mode + +There are two levers. + +**Focus** is the set of speakers you care about. When a focus is set, only those speakers produce active transcript. Everyone else becomes passive. + +**Focus mode** decides what passive means: + +- `RETAIN` keeps the other speakers audible but marks them as background. Their words are still transcribed and still appear, carrying a passive marker so your agent knows not to respond. This is the default and the right choice most of the time: a background speaker can still be pulled into the conversation when a focused speaker invites them in. +- `IGNORE` drops the other speakers entirely. Their audio is not transcribed and does not trigger voice activity or end-of-utterance detection. Use it when you want the engine deaf to everyone else. + +A separate **ignore list** drops one specific speaker without touching the focus. It is useful for killing feedback when the agent's own audio output is picked up by the microphone, or for muting one person on request. + +Any speaker whose label starts and ends with double underscores (such as `__ASSISTANT__`) is excluded automatically, which keeps the agent's own voice out of the transcript. + +## Set and update focus + +You set an initial focus when you open the session, and you can update it at any point while the session runs. The two format strings decide how active and passive speakers are rendered into the transcript the LLM (large language model) reads. + +The following pseudocode configures focus at session start and updates it live: + +```python showLineNumbers +# At session start, with diarization on. The format strings control how +# active and passive speaker lines are rendered for the LLM. +configure_stt( + enable_diarization = true, + speaker_active_format = "@{speaker_id}: {text}", + speaker_passive_format = "@{speaker_id} [background]: {text}", +) + +# Live, at any point during the session: +set_speaker_focus(focus = ["S1"], mode = RETAIN) # others kept as background +set_speaker_focus(focus = ["S1"], mode = IGNORE) # others dropped entirely +ignore_speaker(add = "S2") # separate ignore list +set_speaker_focus(focus = [], mode = RETAIN) # reset, hear everyone +``` + +The format strings determine what the LLM sees. The active speaker's line arrives as `@S1: can you help me`, and a background speaker as `@S2 [background]: yeah but what about lunch`. That `[background]` tag is the signal your system prompt teaches the LLM to ignore. + +A live focus update replaces the previous focus, it does not merge. To build up a multi-speaker focus one speaker at a time, track the set yourself and send the full list each time. + +## Drive focus from the agent + +Rather than a human flipping switches, you hand the LLM a small set of function tools and let it change the focus in response to plain speech. "Focus on me" becomes a tool call. + +The following pseudocode defines a sensible tool set: + +```python showLineNumbers +focus_on_speaker(speaker_ids: string[]) + # Prioritize one or more speakers. Everyone else stays audible as + # background (RETAIN). Replaces any current focus. + "focus on me" -> focus_on_speaker(["S1"]) + "focus on me and her" -> focus_on_speaker(["S1", "S2"]) + +listen_only_to_speaker(speaker_ids: string[]) + # Listen ONLY to these speakers. Everyone else is dropped (IGNORE). + "only listen to me" -> listen_only_to_speaker(["S1"]) + +ignore_speaker(speaker_id: string) + # Add one speaker to the ignore list so their speech stops being + # transcribed. Good for killing echo or muting one person. + "ignore him" -> ignore_speaker("S2") + +listen_to_all_speakers() + # Reset. Clear focus and ignore lists, hear everyone equally again. + "listen to everyone again" -> listen_to_all_speakers() +``` + +The handlers behind these are thin. They map the tool call onto a focus state and push it to the engine: `focus_on_speaker` sets `RETAIN`, `listen_only_to_speaker` sets `IGNORE`, and both call the same live `set_speaker_focus`. `listen_to_all_speakers` sends empty lists and resets the mode to `RETAIN`. + +:::tip +Wire up the reset. If `listen_to_all_speakers` does not clear both the focus and the ignore list, the agent stays stuck on one person. +::: + +## System prompt for speaker focus + +The tools do nothing without a system prompt that teaches the LLM how to read the transcript and when to use them. Three things have to be spelled out. + +First, the LLM must understand the speaker tags. Every incoming line is prefixed, either `@S1:` for a raw label or `@Sam:` once a speaker is recognized by name. Lines carrying `[background]` are passive and should be left alone unless an active speaker brings that person in. + +Second, the LLM has to resolve "me." When someone says "focus on me," the word "me" means whoever is speaking, which is the ID prefixing their own message. A "focus on me" arriving on a `@S1:` line resolves to `S1`. Without this rule the model has to guess, and often picks the wrong speaker. + +Third, the labels are internal. The agent must never read `S1` or `S2` aloud, and never echo a speaker tag into its reply. Real names are fine once known. + +The following prompt fragment covers the speaker rules: + +```text showLineNumbers +# Speakers + +- Each message is prefixed with the speaker, like @S1: or, for a + recognized person, @Sam:. +- Messages prefixed @speaker_id [background]: are passive. Never respond + to or acknowledge them unless an active speaker invites you to. +- When a speaker says "me", "my" or "I", they mean the speaker ID + prefixing their own message. A request from @S1: refers to S1. +- Generic labels like S1 are internal. Never say them aloud or include a + speaker tag in your replies. Real names like "Sam" are fine. + +# Speaker focus + +- "Focus on me" -> focus_on_speaker with the requesting ID. +- "Only listen to me" -> listen_only_to_speaker with the ID(s). +- "Ignore him/her" -> ignore_speaker with that speaker's ID. +- "Listen to everyone" -> listen_to_all_speakers to reset. +``` + +## Example: switching to a single speaker + +This walks through "just listen to me" from a speaker labeled `S1`, with someone else talking as `S2`: + +1. The transcript line arrives as `@S1: just listen to me`, and `S2`'s chatter arrives as `@S2: ...`. +2. The LLM applies the prompt rule, resolves "me" to `S1`, and calls `listen_only_to_speaker(["S1"])`. +3. The handler sets focus to `["S1"]` with mode `IGNORE` and pushes it to the engine. +4. From that point the engine transcribes only `S1`. `S2` is dropped entirely: no transcript, no turn detection off their voice. +5. The agent confirms, something like "sure, just you now," and carries on. + +Saying "listen to everyone again" runs step 3 in reverse: empty focus, empty ignore list, mode back to `RETAIN`, everyone audible. + +## Speaker focus in the Voice SDK, Pipecat and LiveKit + +The concepts above map onto real integrations with little code. The Speechmatics Python [Voice SDK](/voice-agents/voice-sdk) exposes the two levers as `SpeakerFocusConfig` and `SpeakerFocusMode`; the Pipecat and LiveKit plugins wrap the same SDK under their own method names. + +For the Voice SDK, set the focus on the session config and update it live with `update_diarization_config`: + +```python showLineNumbers +from speechmatics.voice import SpeakerFocusConfig, SpeakerFocusMode + +# Session setup: set focus on the transcription config's speaker_config +speaker_config = SpeakerFocusConfig( + focus_speakers=["S1"], + focus_mode=SpeakerFocusMode.RETAIN, # others kept as background +) + +# Live update (inside a tool handler) +client.update_diarization_config( + SpeakerFocusConfig(focus_speakers=["S1"], focus_mode=SpeakerFocusMode.IGNORE) +) +``` + +For [Pipecat](https://github.com/pipecat-ai/pipecat), configure focus on the STT service and update it inside a tool handler: + +```python showLineNumbers +# Session setup +stt = SpeechmaticsSTTService( + params=SpeechmaticsSTTService.InputParams( + enable_diarization=True, + speaker_active_format="@{speaker_id}: {text}", + speaker_passive_format="@{speaker_id} [background]: {text}", + ), +) + +# Live update (inside a tool handler) +stt.update_params( + SpeechmaticsSTTService.UpdateParams( + focus_speakers=["S1"], + focus_mode=SpeechmaticsSTTService.SpeakerFocusMode.RETAIN, + ) +) +``` + +For the [LiveKit Speechmatics plugin](https://github.com/livekit/agents/tree/main/livekit-plugins/livekit-plugins-speechmatics), use `update_speakers`: + +```python showLineNumbers +# Session setup +stt = speechmatics.STT( + enable_diarization=True, + speaker_active_format="[Speaker {speaker_id}] {text}", + speaker_passive_format="[Speaker {speaker_id} *PASSIVE*] {text}", +) + +# Live update (inside a tool handler) +stt.update_speakers( + focus_speakers=["S1"], + focus_mode=SpeakerFocusMode.RETAIN, +) +``` + +Pick format strings to suit your model, and teach the LLM whichever format you chose. + +## Best practices for speaker focus + +`RETAIN` is the safer default for a chatty agent, because a background speaker can be brought in without a config change. `IGNORE` is blunter. Reach for it when you truly want the engine to stop listening, not just stop responding. + +Keep the agent's own voice out of the loop, otherwise its audio output is transcribed straight back in and it starts talking to itself. Any speaker enrolled with a label wrapped in double underscores (such as `__ASSISTANT__`) is excluded automatically, so you never have to ignore it explicitly. + +To use this, pass the agent's voice as a [known speaker](/speech-to-text/features/speaker-memory-for-voice-agents) at session start, labeled `__ASSISTANT__`, with a speaker identifier captured from an earlier session or from enrollment. The engine recognizes the agent's own audio and drops it before it reaches the transcript. You can pass several identifiers under the one label to cover that voice across different output paths or capture devices. + +This is the clean option for complex speaker or microphone setups, where the agent's voice can leak back in through the mic. Adding the agent's speaker to the ignore list at runtime also works. + +Speaker focus pairs with [speaker memory](/speech-to-text/features/speaker-memory-for-voice-agents). Once a voice is recognized by name, every focus and ignore operation can target that name instead of a throwaway label. + +## Next steps + +- [Speaker memory for voice agents](/speech-to-text/features/speaker-memory-for-voice-agents) — recognize returning speakers by name across sessions. +- [Diarization](/speech-to-text/features/diarization) — separate a transcript into distinct speakers. +- [Voice agents overview](/voice-agents/overview) — ways to build a voice agent with Speechmatics. +- [Voice SDK](/voice-agents/voice-sdk) — the Python SDK reference for `SpeakerFocusConfig` and speaker management. diff --git a/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx b/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx new file mode 100644 index 00000000..652376e6 --- /dev/null +++ b/docs/speech-to-text/features/speaker-memory-for-voice-agents.mdx @@ -0,0 +1,205 @@ +--- +title: Speaker memory for voice agents +sidebar_label: Speaker memory +description: Persist speaker identifiers so a voice agent recognizes returning speakers by name across sessions. +keywords: + [ + speechmatics, + speaker identification, + speaker identifiers, + diarization, + voice agents, + realtime, + ] +sidebar_position: 6 +--- + +# Speaker memory for voice agents + +Persist speaker identifiers so a voice agent recognizes a returning speaker by name across sessions. + +Speaker memory is an application of [speaker identification](/speech-to-text/features/speaker-identification), a standard capability of the Realtime and Batch APIs, not a voice-specific feature. This page covers the voice-agent pattern: capturing identifiers live and reusing them across sessions with [diarization](/speech-to-text/features/diarization) enabled. For how identifiers are produced and scoped, see [How speaker identifiers work](/speech-to-text/features/how-speaker-identifiers-work). + +:::warning +Speechmatics does not store speaker identifiers. Save them yourself and supply them again in every future session. Identifiers are derived from a person's voice and may qualify as biometric data under data-protection law such as GDPR. Obtain consent, store them securely, and support deletion. +::: + +## Get identifiers from a live session + +While a session is running, you request the current speakers. The engine returns every speaker it has tracked, each with its label and identifiers. + +The following pseudocode requests identifiers and persists the ones you want: + +```python showLineNumbers +# Request the fingerprints of every speaker heard so far +speakers = request_speakers() + +# speakers looks like: +# [ +# { label: "S1", speaker_identifiers: ["XX...XX"] }, +# { label: "S2", speaker_identifiers: ["YY...YY"] }, +# ] + +# Filter to the speakers you want, then persist to your storage +for s in speakers: + if s.label in wanted: + save_profile(name = wanted[s.label], identifiers = s.speaker_identifiers) +``` + +The engine returns everyone, so filter to the speakers you want on your side. Recognition quality improves once each speaker has said around five words, so do not request identifiers the instant someone starts talking. + +Some integrations let you flag the request as final, returning the definitive fingerprints computed over the whole session. This gives the best quality and suits the end of a call. A mid-session request without the flag returns whatever the engine has so far, which is sufficient for an interim save. + +## Reuse identifiers in a later session + +Storage is your responsibility. A local file, a database, or a user record all work. + +To reuse identifiers, pass them as known speakers when you open the next session. Each entry pairs a label, now a real name, with a list of identifiers. Passing several identifiers under one label covers the same voice heard through different devices, so recognition holds up wherever the person speaks from. + +The following pseudocode configures known speakers at session start: + +```python showLineNumbers +configure_stt( + enable_diarization = true, + known_speakers = [ + # One label, several identifiers for the same voice across devices + { label: "Sam", speaker_identifiers: ["XX...XX", "ZZ...ZZ"] }, + { label: "Alice", speaker_identifiers: ["YY...YY"] }, + ], +) +``` + +The engine attributes matching voices to `Sam` and `Alice` instead of `S1` and `S2`. Anyone it does not recognize gets a fresh generic label, so known and unknown speakers coexist. + +## Drive voice memory from the agent + +As with [speaker focus](/speech-to-text/features/speaker-focus-for-voice-agents), you can give the LLM tools and let it manage voice memory through conversation. + +The following pseudocode defines the tools: + +```python showLineNumbers +# Save one or more speakers for recognition in future sessions +remember_voices(speakers: { speaker_id: string, name: string }[]) + "remember me, I'm Sam" -> remember_voices([{ speaker_id: "S1", name: "Sam" }]) + +# Wipe every saved profile +forget_all_voices() + "clear voice memory" -> forget_all_voices() +``` + +The `remember_voices` handler needs care, because fetching fingerprints is asynchronous. Do not block the agent's reply waiting for them. Stash the name mappings, fire the request, reply immediately, and save when the result arrives. + +The following pseudocode shows the non-blocking handler: + +```python showLineNumbers +pending = {} # label -> real name, awaiting the speakers result + +on remember_voices(speakers): + # Record the name mappings and request fingerprints without blocking + for entry in speakers: + pending[entry.speaker_id] = entry.name + request_speakers() + reply("Voice profile saved.") + +on speakers_result(speakers): + # Match returned identifiers to pending names and save + for s in speakers: + name = pending.pop(s.label, none) + if name: + save_profile(name, s.speaker_identifiers) # merge, don't overwrite +``` + +Merge new identifiers into an existing profile of the same name rather than overwriting, so recognition improves each time. Snapshot and clear the pending map before working through the result, so overlapping requests do not double-process. + +## Write the system prompt + +The tools need guardrails in the prompt. Do not save people because they mention their name: someone saying "I'm Sam, anyway" is not asking to be remembered. Call `remember_voices` only on an explicit request such as "remember me" or "save my voice." + +The name must be a real name. `S1` is a label, not a name. If the agent does not know someone's name, it should ask before saving. + +The following prompt fragment covers the memory rules: + +```text showLineNumbers +# Speaker memory + +- Only call remember_voices when a speaker explicitly asks to be saved, + such as "remember me", "save my voice" or "save our names". Do NOT + call just because someone says their name. +- The name must be the speaker's real name (e.g. "Sam"), never a generic + label like S1. If you don't know their name, ask for it first. +- Pass everyone to save in a single remember_voices call. +- "Forget all voices" or "clear voice memory" -> forget_all_voices. +``` + +## Remember a speaker across sessions + +This example walks through two sessions. + +1. Sam chats to the agent as `S1` and says "remember me, I'm Sam." +2. The LLM calls `remember_voices([{ speaker_id: "S1", name: "Sam" }])`. +3. The handler stashes `S1 -> Sam`, requests the speakers, and replies. +4. The result returns, the handler finds `S1`'s identifiers, and saves the profile `{ "label": "Sam", "speaker_identifiers": ["XX...XX"] }`. + +On the next session startup, you load that profile and pass it as a known speaker. Sam speaks, the engine matches his voice, and his transcript arrives as `@Sam: hello again`. Because he is now a named speaker, you can point [speaker focus](/speech-to-text/features/speaker-focus-for-voice-agents) at `Sam` directly. + +## Use speaker memory in the Voice SDK, Pipecat, and LiveKit + +The pattern maps onto real integrations with little code: get identifiers out of one session, feed them back into the next as known speakers. `known_speakers` is available across the Realtime and Batch clients; the [Voice SDK](/voice-agents/voice-sdk), Pipecat, and LiveKit expose it for the agent use case, with method names differing per client. + +For [Pipecat](https://github.com/pipecat-ai/pipecat), the result arrives on an event: + +```python showLineNumbers +# Request fingerprints, then save them when the event fires +await stt.send_message("GetSpeakers") + +@stt.event_handler("on_speakers_result") +async def on_speakers_result(service, speakers): + for s in speakers["speakers"]: + save_profile(s["label"], s["speaker_identifiers"]) + +# Reuse in a later session +stt = SpeechmaticsSTTService( + params=SpeechmaticsSTTService.InputParams( + enable_diarization=True, + known_speakers=[ + SpeechmaticsSTTService.SpeakerIdentifier( + label="Sam", speaker_identifiers=["XX...XX"] + ) + ], + ), +) +``` + +For the [LiveKit Speechmatics plugin](https://github.com/livekit/agents/tree/main/livekit-plugins/livekit-plugins-speechmatics), the request is a single awaitable: + +```python showLineNumbers +# Request fingerprints, waiting for the result with a short timeout +speakers = await stt.get_speaker_ids() +for s in speakers: + save_profile(s.label, s.speaker_identifiers) + +# Reuse in a later session +stt = speechmatics.STT( + enable_diarization=True, + known_speakers=[ + SpeakerIdentifier(label="Sam", speaker_identifiers=["XX...XX"]), + ], +) +``` + +## Best practices + +Identifiers are per account. You cannot use a fingerprint from one Speechmatics account on another, so voice profiles belong to the account that created them. + +Append new identifiers each time you save a returning speaker rather than treating a profile as fixed. More identifiers improve recognition. + +Request fingerprints once there are a few words to work with, and request them again at the end of a call with the final flag if the integration offers it. This returns identifiers computed over everything the engine heard. + +Speaker memory pairs with [speaker focus](/speech-to-text/features/speaker-focus-for-voice-agents). Once a voice is recognized as `Sam`, every focus and ignore operation can target `Sam` by name. + +## Next steps + +- [Speaker focus for voice agents](/speech-to-text/features/speaker-focus-for-voice-agents) — direct the agent to specific speakers. +- [How speaker identifiers work](/speech-to-text/features/how-speaker-identifiers-work) — how identifiers are generated and scoped. +- [Voice agents overview](/voice-agents/overview) — ways to build a voice agent. +- [Voice SDK](/voice-agents/voice-sdk) — Python SDK reference for `known_speakers`. diff --git a/docs/voice-agents/voice-sdk.mdx b/docs/voice-agents/voice-sdk.mdx index f352b780..54446ea7 100644 --- a/docs/voice-agents/voice-sdk.mdx +++ b/docs/voice-agents/voice-sdk.mdx @@ -58,6 +58,8 @@ You can also: - Ignore specific speakers - Provide known speakers for speaker identification +For voice-agent patterns built on these, see [Speaker focus for voice agents](/speech-to-text/features/speaker-focus-for-voice-agents) and [Speaker memory for voice agents](/speech-to-text/features/speaker-memory-for-voice-agents). + ### Voice SDK vs Realtime SDK - Use the Voice SDK when: @@ -400,6 +402,8 @@ In your event handler, you can use `is_active` to decide how to route segments: {pythonSpeakerFocusHandler} +For the agent-driven pattern, where the LLM changes focus in response to speech using function tools and a system prompt, see [Speaker focus for voice agents](/speech-to-text/features/speaker-focus-for-voice-agents). + #### Known speakers (speaker identification) `known_speakers` (list[SpeakerIdentifier], default: []) @@ -409,6 +413,8 @@ Pre-enrolled speaker identifiers for speaker identification. {pythonKnownSpeakers} +To capture identifiers live and reuse them across sessions so an agent recognizes returning speakers by name, see [Speaker memory for voice agents](/speech-to-text/features/speaker-memory-for-voice-agents). + ### Advanced configuration example