Local transcription frozen after 45-min recording; service won't recover

[hyper-charlie]

Issue

Local transcription froze midway through a 45-minute meeting recording. After the crash, all transcription is broken — even previously transcribed recordings can no longer be transcribed locally.

Steps to Reproduce

1. Record a 45-minute meeting
2. Attempt local transcription
3. Transcription freezes with WebSocket error: remote closed 1009
4. Attempt to transcribe any recording (new or old) → transcription fails

Expected Behavior

• Transcription should complete without freezing
• Service should recover after restart
• Previously transcribed recordings should still work

Actual Behavior

• WebSocket closes with code 1009 (message too large)
• Local transcription service appears to be in a broken state
• No recordings can be transcribed, even old ones that worked before

Environment

• Platform: macOS
• OS Version: macOS 26.3.1
• App Version: 1.0.13
• Build: 3b53e1b
• Architecture: aarch64
• Locale: en-US

Notes

• WebSocket error 1009 suggests message size exceeded server limits
• Service does not appear to recover automatically after crash

[devin-ai-integration]

Codebase Exploration Findings

Relevant Files & Components

WebSocket client layer:

• crates/ws-client/src/client.rs — WebSocket client using tokio_tungstenite::connect_async with no explicit WebSocketConfig (default tungstenite limits: 64 MiB message / 16 MiB frame)
• crates/ws-client/src/retry.rs — Connection retry logic (WebSocketConnectPolicy): 3 attempts, 750ms delay, 5s timeout. Only retries initial connection, not mid-stream failures.
• crates/ws-client/src/error.rs — Error::RemoteClosed { code, reason } — this is the error variant surfaced for close code 1009.

Server-side (local Whisper STT):

• crates/transcribe-whisper-local/src/service/streaming.rs — Axum WebSocket handler at /v1/listen. Uses WebSocketUpgrade with default message size limits (no .max_message_size() override). Processes audio through VAD → Whisper inference per speech chunk.
• plugins/local-stt/src/server/internal.rs — Starts the internal Axum server for local transcription.

Audio pipeline:

• crates/ws-utils/src/lib.rs — process_ws_message() and split_dual_audio_sources() — converts WS messages to audio samples. Uses try_send() on bounded channels (AUDIO_CHANNEL_CAPACITY = 1024) — silently drops audio when full (only logs a warning).
• crates/vad-ext/src/continuous.rs — speech_chunks() collects audio samples between SpeechStart and SpeechEnd transitions. The SpeechEnd variant carries the full samples: Vec<f32> for the entire speech segment. No max duration cap — if VAD doesn't detect a pause in a 45-minute meeting, a single chunk could grow very large.

Supervisor / recovery:

• plugins/listener/src/actors/session.rs — Session supervisor: RestForOne strategy, max_restarts: 3 in 15 seconds, reset after 30s. Once exhausted, the supervisor stops and all child actors (source, listener, recorder) die.
• plugins/listener/src/actors/root.rs — handle_supervisor_evt: when session supervisor terminates/fails, it sets state.supervisor = None and emits session_ended. No automatic re-creation of the supervisor — the session is permanently dead.
• plugins/local-stt/src/server/supervisor.rs — STT server supervisor: max_restarts: 100 in 180s (much more generous), but this only governs the STT server process, not individual transcription sessions.
• crates/ws-utils/src/manager.rs — ConnectionManager cancels old connections on new ones, but provides no recovery for failed connections.

Recent Related Changes

• 6ac06c35e (Mar 7, 2026) — "Improve ws-client retry/error handling and wire connect policy into listener clients (Improve ws-client retry/error handling and wire connect policy into listener clients #4453)" — Major rework of the WebSocket client. Added structured error types (including RemoteClosed), configurable WebSocketConnectPolicy, and retry callbacks. However, retry only applies to initial connection handshake, not to mid-stream disconnects like the 1009 error.
• 65d4e42a4 — ws-retry-desktop branch (WIP) — suggests active work on retry improvements.

Root Cause Analysis

1. WebSocket close code 1009 (Message Too Large):
The error originates from the server (internal Whisper) closing the connection. Both client and server use default tungstenite WebSocketConfig with no explicit size overrides. While the defaults are generous (64 MiB), the issue likely stems from the VAD speech chunk accumulation — SpeechEnd in vad-ext/src/continuous.rs carries the full audio samples for a continuous speech segment. In a 45-minute meeting with minimal silence:

• At 16kHz mono, 45 minutes of audio ≈ 43.2M samples × 4 bytes = ~165 MB as f32
• Even a few minutes of continuous speech without a VAD pause could produce chunks exceeding the default message limits

2. "All transcription broken" — No session recovery:
The critical path is:

1. WS error 1009 → listener actor crashes
2. Session supervisor (max_restarts: 3 / 15s) restarts it
3. If the large-message condition persists (e.g., accumulated audio buffer), it crashes again
4. After 3 rapid failures, supervisor stops → root.rs receives ActorTerminated/ActorFailed → sets supervisor = None → emits session_ended
5. No mechanism exists to re-create the supervisor — future transcription attempts may fail because the session management layer is in a terminal state
6. The STT server itself (governed by a separate, more generous supervisor with 100 restarts) likely remains running, but the session/listener layer above it is stuck

Potential Approaches

1. Cap VAD speech chunk size — Add a max_speech_duration to speech_chunks() that forces a SpeechEnd emission after N seconds of continuous speech, preventing unbounded chunk growth.

2. Configure explicit WebSocket message limits — Set max_message_size / max_frame_size on both the Axum WebSocketUpgrade (server) and tokio_tungstenite config (client) to known, matching values. Consider chunking large audio payloads.

3. Add mid-stream reconnection — The recent ws-client improvements (PR Improve ws-client retry/error handling and wire connect policy into listener clients #4453) only retry on initial connect. Extending this to handle mid-stream disconnects (especially for recoverable errors like 1009) would prevent permanent session death.

4. Increase session supervisor resilience — The current max_restarts: 3 in 15 seconds is very restrictive compared to the STT supervisor's 100 in 180s. Consider increasing it or adding exponential backoff to give transient errors more room to recover.

5. Add supervisor re-creation in root actor — When handle_supervisor_evt receives ActorTerminated or ActorFailed, instead of just setting supervisor = None, attempt to re-create the supervisor (possibly with a backoff) so that new sessions can still be started.

6. Improve try_send() backpressure — The current pattern in ws-utils/src/lib.rs silently drops audio. Adding backpressure or a bounded retry would provide better signal about capacity issues.

[m13v]

Hit something very similar building Fazm's voice mode. Long transcription sessions would freeze the Whisper pipeline and never recover, even after restarting the transcription service.

What fixed it for us: chunked processing with a hard timeout per chunk. We split audio into 30-second segments, process each independently, and if any single chunk takes longer than 10 seconds we kill it and move to the next. The key insight was that the decoder can get stuck in a loop on certain audio patterns (silence, background noise), and no amount of waiting will unstick it.

We also added a watchdog timer that monitors the transcription service. If no new tokens arrive for 5 seconds, it force-restarts the pipeline. The WebSocket 1009 error you're seeing (message too big) suggests the audio buffer is growing unbounded - you might want to cap the buffer size and process incrementally rather than accumulating the full 45 minutes.

[m13v]

Here's the transcription service we built for Fazm that handles long sessions with the chunked approach I mentioned: https://github.com/m13v/fazm/blob/main/Desktop/Sources/TranscriptionService.swift

The watchdog timer and chunk-based processing are in there. We also built a macOS AI agent (https://fazm.ai/gh) that does voice-first interaction, so we've spent a lot of time debugging these exact transcription reliability issues.