feat(asr): VAD-aware chunk boundaries and seam token dedup - #22
Merged
Conversation
Commit the exact ~30 minute MP3 from issue #18 plus its .srt/.vtt/.txt reference transcripts under testdata/reference/, renamed to shell-friendly names, with a README documenting provenance and usage. This is the input that reproduces the chunk-seam duplicated/dropped words, kept in-tree so the seam inspector needs no network access.
Download silero_vad.onnx from snakers4/silero-vad (MIT) pinned to release v6.2.1 with its sha256 verified via 'sha256sum -c' in the models-int8 and models-fp32 Makefile targets (new models-silero-vad helper) and in both Docker images. At v6.2.1 the ONNX file lives at src/silero_vad/data/, not the older files/ path. Exclude testdata/ from the Docker build context so the reference MP3 does not bloat it.
sileroVAD wraps one long-lived DynamicAdvancedSession (same pattern as the encoder). Silero is stateful but its recurrent state travels through the state/stateN tensors per call, so the session is safe to share across concurrent requests as long as each keeps its own vadState. Windows are fed in 512-sample (32ms) chunks with a 64-sample left context, carrying state between calls. A missing model file returns os.ErrNotExist so the caller can degrade gracefully rather than fail.
Introduce a boundaryOracle interface with three interchangeable implementations tried as a cascade: vadBoundaryOracle (centre of the longest Silero-detected silence), melEnergyBoundaryOracle (quietest smoothed mel-energy frame) and midpointBoundaryOracle (arithmetic midpoint, always decides). planChunksWithBoundaries takes the oracle and clamps its result into the legal overlap range so the emit-range tiling invariant holds for any oracle output; planChunks is now the nil-oracle midpoint case. Pure canary tests cover selection logic, cascade fallthrough order, clamping and the tiling invariant under adversarial oracles.
dedupSeam drops a leading token of window i+1 when its absolute encoder-frame timestep is within a small tolerance (~240ms) of window i's trailing tokens. One rule covers both issue #18 failure modes: a duplicate (same text) and a collision (different text), where the earlier, fully warmed-up window wins. decodedToken carries the absolute timestep used to line tokens up. Named constants, no flags. Table-driven canary tests cover exact duplicate within tolerance, duplicate outside tolerance, text-mismatch collision, no collision and the k boundary cases.
Build a per-request boundary oracle cascade (VAD -> mel energy -> midpoint) over the request's mel features and waveform, plan the chunk windows with it, and tag every decoded token with an absolute encoder-frame timestep. tdtDecode now buffers at most the first few owned tokens of each window after the first, resolves them through the seam deduper, streams the survivors in order and then streams the rest, so streaming order is preserved while seam duplicates are removed. Load the Silero VAD session in NewTranscriber when long audio is on (missing model warns once and falls back to mel energy) and release it in Close. Add MelFilterbank.HopLength for mel-frame-to-sample mapping.
Add --disable-vad-based-chunking, --disable-mel-based-chunking and --vad-model-path, wired through server.Config and asr.Options following the existing ChunkSeconds/LongAudio pattern. Each gets its PARAKEET_* env var for free via applyEnvDefaults.
A -tags=seaminspect Go test (no Python, no network) transcribes the reference MP3 through the full long-audio pipeline, logs the chosen boundary positions and which oracle decided each, and prints the transcribed text around every seam next to the .srt reference for the same time range so a human can eyeball each seam. Skips cleanly when ONNX Runtime, models or the audio are absent.
Add DD-014 (boundary cascade, interface rationale, VAD-only-on-overlaps, the loud-music limitation, rejected LCS/full-VAD-segmentation alternatives, the window-i-wins dedup rationale, the concurrency caveat, and the pinned Silero v6.2.1 tag+sha256+MIT license). Update AGENTS.md (new flags, files, model, env vars, testdata), README.md (flags table, models table, how chunk boundaries are chosen) and TODO.md.
The .agents/ directory is agentic documentation, not code documentation. Code comments and the README must stand on their own, so the DD-014 and DD-011 pointers are replaced with self-contained explanations.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the chunk-seam hallucinations reported in #18 (duplicated words like
to toand dropped words likefor., and howat the seams between overlapping windows in long-audio mode).The chunking introduced in #21 splits ownership of each overlap at its arithmetic midpoint. That midpoint is blind to content, so it often lands mid-word: each window timestamps tokens slightly differently, and a word straddling the boundary ends up emitted twice or by neither window. This PR moves the boundary onto silence and adds a seam-level token dedup as a second safety net. See DD-014 for the full rationale.
Boundary selection cascade
A
boundaryOracleinterface picks the split inside each overlap; the three implementations are alternative strategies for the same contract, tried in order:vadBoundaryOracle): runs the VAD over the overlap's waveform only (never the whole file) and picks the centre of the longest silence. Falls through when no window drops below the speech-probability threshold.melEnergyBoundaryOracle): quietest smoothed-energy frame from the already-extracted features. Robust fallback when the VAD is disabled or its model is missing.midpointBoundaryOracle): the previous behaviour, terminal fallback, so the result is never worse than before.Whatever an oracle returns is clamped so the emit ranges still tile the timeline with no gaps or duplicates (the invariant from #21 holds for ANY oracle output).
Seam token dedup (always on)
Tokens are tagged with absolute encoder-frame timesteps. At each seam, the first k tokens of window i+1 are compared against the tail of window i: collisions within ~240 ms are dropped from window i+1, because window i reaches the seam with a fully warmed-up LSTM while window i+1 is still warming up. Streaming buffers at most those k tokens per seam; everything else streams as before.
Silero VAD model
make modelsand shipped in the Docker images.New flags (env vars via the generic
PARAKEET_*mapping)-disable-vad-based-chunkingfalse-disable-mel-based-chunkingfalse-vad-model-path<models>/silero_vad.onnxValidation
gofmt -s,go vet,go build,go test -raceall clean.testdata/reference/now carries the exact ~30 min MP3 from Encoder crashes on audio (onnxruntime) #18 plus its reference transcripts, and a build-tag-gated seam inspector (go test -tags=seaminspect) transcribes it end to end and prints the text around every seam next to the reference SRT.Known limitation: with loud, dynamic music the energy heuristic degrades to roughly midpoint quality; the VAD layer is the robust answer there.
Fixes #18