fix(avatar): preserve audio wrappers across avatar hot-swaps#5863
Open
longcw wants to merge 13 commits into
Open
fix(avatar): preserve audio wrappers across avatar hot-swaps#5863longcw wants to merge 13 commits into
longcw wants to merge 13 commits into
Conversation
AvatarSession.start() rebinds session.output.audio to a fresh DataStreamAudioOutput. On the first call the wrapper chain (Recorder, TranscriptSynchronizer) wraps it correctly, but a re-bind during a mid-session avatar switch overwrites the synchronizer-wrapped output with a raw sink, breaking audio/transcription sync and recording. Introduce _AudioSinkProxy, a transparent proxy auto-inserted at the bottom of any wrapper chain. Wrappers cache the proxy (not the leaf), so the leaf can be hot-swapped via the proxy without invalidating upstream references. When the proxy has no inner sink, flush() synthesizes a playback_finished so upstream wrappers don't hang. Add AgentOutput.set_audio_sink(sink, *, preserve_wrappers=False). With preserve_wrappers=True, walks the chain to find the proxy and swaps its downstream; otherwise behaves as the existing audio setter. Avatar plugins migrate to this API; AvatarSession.aclose() detaches the sink so the chain stays intact across aclose -> restart. Drops the "may be replaced by the avatar" warning in AvatarSession.start since the proxy makes mid-session rebinding correct by construction.
…ers=True) Route every avatar plugin's audio sink binding through the new AgentOutput.set_audio_sink API so mid-session hot-swaps (e.g. avatar switches) preserve the TranscriptSynchronizer / RecorderAudioOutput wrapper chain. Plugins migrated: anam, avatario, avatartalk, bey, bithuman, did, keyframe, liveavatar, runway, simli, tavus, trugen.
Covers: - auto-wrap inserts the proxy between a wrapper and a bare leaf - auto-wrap skipped when the downstream is already a proxy or a non-leaf - set_audio_sink default replaces the chain - set_audio_sink with preserve_wrappers swaps the proxy's inner in place - preserve_wrappers fallback when no proxy exists in the chain - proxy rejects a wrapper chain as inner (set_next_in_chain assert) - detached proxy synthesizes playback_finished on flush - swap routes new-leaf playback events to upstream listeners - swap disconnects the old leaf from the chain - on_attached/on_detached propagate to current inner and across swaps
Drop the leaf-only assertion in _AudioSinkProxy.set_next_in_chain — the base AudioOutput machinery cascades capture/flush and bubbles playback events through any chain, so the proxy can hold either a leaf or a wrapper chain without breaking the contract upstream.
The base class doesn't track which sink the avatar set, so nulling session.output.audio unconditionally could clobber a sink owned by someone else. The wrapper chain stays intact across hot-swaps anyway because the proxy preserves the wrappers regardless of what's in its downstream slot, so leaving the sink in place until it's replaced or the session tears down is fine.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
theomonnom
reviewed
May 27, 2026
Drop the preserve_wrappers flag: the wrapper-preserving leaf swap is now its own method, and full replacement stays as output.audio = sink.
…wrappers-on-avatar-swap
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
The detached no-op mode of _AudioSinkProxy synthesized a synchronous playback_finished during flush(), which re-entered _SyncedAudioOutput.on_playback_finished and caused a double rotate_segment while skipping end_audio_input. Nothing passes None in practice, so require a real sink instead of fixing the re-entrancy.
Comment on lines
+553
to
+557
| @property | ||
| def sample_rate(self) -> int | None: | ||
| if self._sample_rate is not None: | ||
| return self._sample_rate | ||
| return self.next_in_chain.sample_rate if self.next_in_chain else None |
Contributor
There was a problem hiding this comment.
🚩 sample_rate is now dynamic instead of fixed at construction time
Both _SyncedAudioOutput (synchronizer.py:553-557) and RecorderAudioOutput (recorder_io.py:366-370) changed from setting sample_rate at construction time (passing it to super().__init__) to computing it dynamically via a property that delegates to self.next_in_chain.sample_rate. This is intentional for the hot-swap use case — after swapping the leaf sink, the sample rate should reflect the new sink's requirements. However, in generation.py:418-428, the resampler is created lazily on the first frame and never recreated. If the sample rate changes after the first frame (e.g., from a hot-swap), the resampler won't be updated. This is a pre-existing limitation, not introduced by this PR, but it becomes more relevant now that hot-swapping is supported.
Was this helpful? React with 👍 or 👎 to provide feedback.
longcw
force-pushed
the
longc/preserve-audio-wrappers-on-avatar-swap
branch
from
2900ae1 to
b62cd5f
Compare
A swap while a flushed segment was still playing out removed the old sink's listeners before its playback_finished arrived, leaving the playback accounting unbalanced and wait_for_playout() hanging. Now the proxy clears the old sink's buffer and reports the orphaned segment as interrupted; a segment still being captured continues on the new sink, which reports it on its own.
theomonnom
reviewed
Jun 12, 2026
| else: | ||
| self._audio_sink.on_detached() | ||
|
|
||
| def swap_audio_endpoint(self, sink: AudioOutput) -> None: |
Member
There was a problem hiding this comment.
Suggested change
| def swap_audio_endpoint(self, sink: AudioOutput) -> None: | |
| def replace_audio_leadl(self, sink: AudioOutput) -> None: |
not a fan of the name
Member
There was a problem hiding this comment.
should it be something like replace_audio_sink?
Contributor
Author
There was a problem hiding this comment.
"lead" reads as the head of the chain, but the method replaces the tail, how about
- replace_audio_endpoint
- replace_audio_destination
- redirect_audio
- replace_audio_tail
theomonnom
approved these changes
Jun 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Avatar plugins set
session.output.audio = DataStreamAudioOutput(...)on every start. On the first start this works becauseAgentSession.start()wraps the sink with theTranscriptSynchronizer/RecorderAudioOutputchain afterwards; on a mid-session rebind (avatar switch) the raw assignment blows the chain away, silently breaking transcription sync and recording.Fix it by auto-inserting an
_AudioSinkProxyat the bottom of any wrapper chain. Wrappers cache the proxy, the proxy holds the swappable leaf — so hot-swaps preserve the wrappers above. NewAgentOutput.swap_audio_endpoint(sink)walks the chain to the proxy and swaps its downstream in place, leaving the wrappers attached; full replacement stays asoutput.audio = sink.Plugin migration
All 13 avatar plugins migrated to
swap_audio_endpoint(...): anam, avatario, avatartalk, bey, bithuman, did, keyframe, lemonslice, liveavatar, runway, simli, tavus, trugen.Example
examples/avatar_agents/audio_wavenow demonstrates hot-swapping through aswap_avatarRPC method: it tears down the current avatar (removing it from the room) and launches a fresh one under the same identity, while the audio wrappers and the listeners attached tosession.output.audiosurvive the swap.Related
Clean avatar-worker shutdown on swap depends on livekit/python-sdks#699.
close #4198