feat(asr): opt-in long-audio chunking to stay under the encoder frame limit - #21
Merged
Conversation
The exported encoder crashes on audio longer than ~400s: its positional encoding table is [1,9999,1024] centred at 5000, so past 5000 encoder frames the relative-position slice goes out of range and self_attn Add fails on a broadcast mismatch (issues #14, #18). Split the mel feature sequence into overlapping windows sized in seconds (-chunk-seconds, default 300; -chunk-overlap-seconds, default 15), run each through the encoder and TDT decoder, and concatenate the tokens. Each window decodes in full so the LSTM state and previous-token feedback stay coherent, but only emits tokens whose timestep falls in its owned region; ownership of the shared overlap is split at its midpoint so adjacent emit ranges tile the timeline with no gaps or duplicated speech. Streaming filters by the same range, so no buffering is needed. Audio under the chunk size takes the single window path unchanged. planChunks and the frame math are pure and table-tested (tiling invariant, deterministic layout, overlap, validation, model-limit rejection). NewTranscriber rejects chunk sizes that would overrun the model limit. Fixes #14, #18.
Chunking is now opt-in. With -long-audio off (the default) audio over the model's single-pass limit is rejected with a clear error and a log warning pointing at the flag, instead of chunking or crashing. With it on, the overlapping-window path runs as before. planForAudio picks the coverage (single window, chunked, or ErrAudioTooLong) and is pure and table-tested. Chunk-size validation only runs when the mode is enabled.
achetronic
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The encoder crashes on audio over ~400s: its positional-encoding table is
[1,9999,1024]centred at 5000, so past 5000 encoder frames the relative-position slice goes out of range andself_attn/Addfails on a broadcast mismatch. Same root cause as #14 and #18.Features
-long-audioflag (default off). Off: audio over the limit is rejected with a clear error and a log warning. On: the audio is split into overlapping windows and stitched, dropping the overlap so seams are not duplicated.-chunk-seconds(default 300),-chunk-overlap-seconds(default 15).Verification
gofmt,go vet,go build,go test -racegreen.planForAudio/planChunksare pure and table-tested.mastercrashes on the same file.Fixes #14
Fixes #18