perf(record): keep one speech model loaded — 2948 ms → 77 ms to start recording - #13
Merged
Conversation
… in 77 ms instead of 3 seconds
Measured on the released build, `gipformer-65m`, from `session_start` to the daemon answering
`session started`:
cold 2948 ms
warmed 77 ms
Nearly three seconds, paid at the start of *every* meeting, with nothing on screen to say why the
transcript had not begun. It is not disk — a second construction in the same process costs the same
as the first — it is ONNX Runtime building the session, and the fix is not to build it again.
So the daemon keeps one decoder ready. Deliberately one: a second warm model would double resident
memory for the case where somebody switches languages *between* meetings, which is rarer than
recording twice in the same one, and `gipformer-65m` idles at about 150 MB. A two-lane session still
builds its second decoder, because a decoder holds mutable inference state and two lanes cannot
share one — all of the common case is instant, half of a rare one.
The slot is filled on demand rather than at startup: `POST /models/warm` when the record card opens,
and by the daemon itself after a session ends, so the *second* meeting is instant too — verified:
`ready again after the session: gipformer-65m`. It is given away rather than lent, which keeps every
question about a killed recording holding a borrowed decoder from existing. Removing a model clears
it, because a warm decoder pointing at deleted blobs is a crash waiting for the next recording.
A decoder built for another language is a miss, never a substitute. That is the one thing this must
not do: silently transcribing a Japanese meeting with a Vietnamese decoder because it happened to be
the one already loaded.
Along the way, `model_load` — the socket command whose whole purpose is "so the first recording is
not delayed by it" — was answering `cannot load: this binary was built without recognition support`
in builds that have it, exactly as `model_swap` was until yesterday. Both arms now work.
`/status` gains `ready`, beside the existing fields rather than wrapped around them, so the record
card can say "sẵn sàng ghi ngay" instead of leaving a pause unexplained. Warming with nothing
installed answers 200 and a reason: the browser suites caught it logging a 400 on a first run where
nothing was wrong.
…t is not an error The browser suites run against `--features bundled`, which compiles recognition out — so `POST /models/warm` has no route and the fallback answers 405, which the record card reported as a failed request on a screen where nothing was wrong. The same is true of the `-nomodels` release build a user can download: it browses a vault and cannot record. 404 and 405 now mean "this build cannot warm anything". Every other status is still a real failure.
Handling the 405 was not enough: the browser logs a failed request whatever the code does with the response, and the shell suite treats console errors as failures — correctly, because they usually are. So the interface asks first. `/status` carries `ready` only in a build with recognition, so an absent field means "no warming here" and the request is never sent. Present-but-null still means "nothing loaded yet", which is the case that does send it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measured on the released build,
gipformer-65m, fromsession_starttosession started:Nearly three seconds, paid at the start of every meeting, with nothing on screen explaining the
pause. Not disk — a second construction in the same process costs the same as the first — it is
ONNX Runtime building the session.
The shape
rarer case of switching language between meetings.
inference state. All of the common case is instant.
about a killed recording holding a borrowed decoder.
transcribing Japanese with a Vietnamese decoder is the failure this whole area exists to prevent.
the next recording.
Also fixed
model_load— the command whose comment says "so the first recording is not delayed by it" — wasanswering
cannot load: this binary was built without recognition supportin builds that haveit, the same defect
model_swaphad until yesterday. Both arms now work./statusgainsreadybeside the existing fields (not wrapped around them, so no client breaks),and the record card says "Sẵn sàng ghi ngay". Warming with nothing installed answers 200 with a
reason — the browser suites caught it logging a 400 on a first run where nothing was wrong.
Verified
1289 Rust tests (4 new in
warm.rs), 297 web tests,shell/language/full-flowsuites,clippy
-D warningson both feature sets, fmt, tsc, eslint.🤖 Generated with Claude Code