Skip to content

fix(voice): reset finalization gate on new turn + attach remote audio to DOM#19

Closed
sergiopesch wants to merge 1 commit into
fix/package-json-merge-conflictfrom
fix/voice-races-and-audio
Closed

fix(voice): reset finalization gate on new turn + attach remote audio to DOM#19
sergiopesch wants to merge 1 commit into
fix/package-json-merge-conflictfrom
fix/voice-races-and-audio

Conversation

@sergiopesch

Copy link
Copy Markdown
Owner

Two related but independent fixes in hooks/useVoiceInteraction.ts.

1. Dropped turn on fast re-speak

finalizeTurn sets isFinalizingTurnRef.current = true, then awaits onFinalTranscript (which is a fetch to /api/chat). If the user starts speaking again before that fetch resolves:

turn A: speech_stopped -> transcription.completed -> finalizeTurn()  [flag=true, awaiting fetch...]
turn B: speech_started    <- clears buffers, but does NOT clear the flag
turn B: speech_stopped
turn B: transcription.completed -> finalizeTurn()
                                   ^^^^^^^^^^^^^^
                                   if (isFinalizingTurnRef.current) return;   // 🔥 dropped

The user's second utterance is silently dropped.

Fix: reset isFinalizingTurnRef on input_audio_buffer.speech_started. The previous turn's in-flight fetch still completes normally; it just no longer blocks the next turn from finalizing.

2. Remote <audio> element not attached to DOM

On pc.ontrack we lazily constructed:

remoteAudioRef.current = new Audio();
remoteAudioRef.current.autoplay = true;

…and never appended it to the document. Most desktop browsers accept detached <audio> elements, but Safari on iOS is known to be inconsistent — autoplay can silently fail, and audio-route selection (speaker / earpiece / Bluetooth) can misbehave on detached nodes.

Fix: append to document.body with display: none + aria-hidden="true" on first creation. Remove it from its parent during teardown so repeated sessions don't leave orphaned nodes behind.

Why both in one PR

They're two lines of code each, both in the same hook, both about the lifetime of connection-scoped resources. Splitting them would create more review overhead than either deserves.

Verification

  • npm run validate (type-check + lint + prettier)
  • npm test — 12/12
  • npm run build — production build succeeds
  • ✅ No changes to the public hook API

Can't easily unit-test the race without a full Realtime-API harness; adding one was explicitly out of scope (would also have required jsdom or playwright). The fix is a two-line reset and the comment explains the invariant.

Depends on #12. Independent of #13/#14/#15/#16/#17/#18.

…to DOM

Two small but real issues in useVoiceInteraction.ts:

1. Dropped turn on fast re-speak.
   finalizeTurn() sets isFinalizingTurnRef.current = true, then awaits
   onFinalTranscript (a fetch to /api/chat). If the user starts a new
   turn before that fetch resolves, the server-emitted
   'input_audio_buffer.speech_started' event clears the buffers but
   does NOT clear the gate. The next 'input_audio_transcription.
   completed' event then hits:
       if (isFinalizingTurnRef.current) return;
   and gets silently dropped. The user's second utterance is lost.

   Fix: reset isFinalizingTurnRef to false on speech_started. The
   previous turn's in-flight fetch still completes and updates state;
   it just no longer blocks the next turn from finalizing.

2. Remote <audio> element not in the DOM.
   On pc.ontrack we lazily created an HTMLAudioElement with new Audio()
   and never appended it to the document. Most desktop browsers will
   play audio from a detached element, but Safari on iOS is known to
   be inconsistent -- autoplay can silently fail and audio-route
   selection (speaker vs earpiece vs Bluetooth) can misbehave on
   detached nodes.

   Fix: append to document.body with display:none + aria-hidden on
   first creation, and remove the element from its parent during
   teardown so repeated sessions don't leave orphaned <audio> nodes.

No changes to the public hook API. Verified with type-check, lint,
prettier, tests (12/12), production build.
@vercel

vercel Bot commented Apr 21, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
clitronic Ready Ready Preview, Comment Apr 21, 2026 10:32am

@sergiopesch
sergiopesch deleted the branch fix/package-json-merge-conflict April 21, 2026 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant