Skip to content

perf(record): keep one speech model loaded — 2948 ms → 77 ms to start recording - #13

Merged
meocong merged 3 commits into
masterfrom
feat/warm-models-and-compose
Aug 14, 2026
Merged

perf(record): keep one speech model loaded — 2948 ms → 77 ms to start recording#13
meocong merged 3 commits into
masterfrom
feat/warm-models-and-compose

Conversation

@meocong

@meocong meocong commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Measured on the released build, gipformer-65m, from session_start to session started:

ready before warming: null
cold  : 2948 ms to start recording
warming took 3024 ms → {"model":"gipformer-65m","language":null}
warmed:   77 ms to start recording
ready again after the session: {"model":"gipformer-65m"}      ← refilled by itself

Nearly three seconds, paid at the start of every meeting, with nothing on screen explaining the
pause. Not disk — a second construction in the same process costs the same as the first — it is
ONNX Runtime building the session.

The shape

  • One decoder, not a cache. A second warm model doubles resident memory (~150 MB idle) for the
    rarer case of switching language between meetings.
  • One lane. A two-lane session still builds its second decoder; two lanes cannot share mutable
    inference state. All of the common case is instant.
  • Given away, not lent. A session owns what it takes; the slot refills afterwards. No question
    about a killed recording holding a borrowed decoder.
  • A miss, never a substitute. A decoder built for another language is not reused — silently
    transcribing Japanese with a Vietnamese decoder is the failure this whole area exists to prevent.
  • Removing a model clears the slot: a warm decoder pointing at deleted blobs is a crash waiting for
    the next recording.

Also fixed

model_load — the command whose comment says "so the first recording is not delayed by it" — was
answering cannot load: this binary was built without recognition support in builds that have
it
, the same defect model_swap had until yesterday. Both arms now work.

/status gains ready beside the existing fields (not wrapped around them, so no client breaks),
and the record card says "Sẵn sàng ghi ngay". Warming with nothing installed answers 200 with a
reason — the browser suites caught it logging a 400 on a first run where nothing was wrong.

Verified

1289 Rust tests (4 new in warm.rs), 297 web tests, shell / language / full-flow suites,
clippy -D warnings on both feature sets, fmt, tsc, eslint.

🤖 Generated with Claude Code

… in 77 ms instead of 3 seconds

Measured on the released build, `gipformer-65m`, from `session_start` to the daemon answering
`session started`:

    cold      2948 ms
    warmed      77 ms

Nearly three seconds, paid at the start of *every* meeting, with nothing on screen to say why the
transcript had not begun. It is not disk — a second construction in the same process costs the same
as the first — it is ONNX Runtime building the session, and the fix is not to build it again.

So the daemon keeps one decoder ready. Deliberately one: a second warm model would double resident
memory for the case where somebody switches languages *between* meetings, which is rarer than
recording twice in the same one, and `gipformer-65m` idles at about 150 MB. A two-lane session still
builds its second decoder, because a decoder holds mutable inference state and two lanes cannot
share one — all of the common case is instant, half of a rare one.

The slot is filled on demand rather than at startup: `POST /models/warm` when the record card opens,
and by the daemon itself after a session ends, so the *second* meeting is instant too — verified:
`ready again after the session: gipformer-65m`. It is given away rather than lent, which keeps every
question about a killed recording holding a borrowed decoder from existing. Removing a model clears
it, because a warm decoder pointing at deleted blobs is a crash waiting for the next recording.

A decoder built for another language is a miss, never a substitute. That is the one thing this must
not do: silently transcribing a Japanese meeting with a Vietnamese decoder because it happened to be
the one already loaded.

Along the way, `model_load` — the socket command whose whole purpose is "so the first recording is
not delayed by it" — was answering `cannot load: this binary was built without recognition support`
in builds that have it, exactly as `model_swap` was until yesterday. Both arms now work.

`/status` gains `ready`, beside the existing fields rather than wrapped around them, so the record
card can say "sẵn sàng ghi ngay" instead of leaving a pause unexplained. Warming with nothing
installed answers 200 and a reason: the browser suites caught it logging a 400 on a first run where
nothing was wrong.
…t is not an error

The browser suites run against `--features bundled`, which compiles recognition out — so
`POST /models/warm` has no route and the fallback answers 405, which the record card reported as a
failed request on a screen where nothing was wrong. The same is true of the `-nomodels` release
build a user can download: it browses a vault and cannot record.

404 and 405 now mean "this build cannot warm anything". Every other status is still a real failure.
Handling the 405 was not enough: the browser logs a failed request whatever the code does with the
response, and the shell suite treats console errors as failures — correctly, because they usually
are.

So the interface asks first. `/status` carries `ready` only in a build with recognition, so an
absent field means "no warming here" and the request is never sent. Present-but-null still means
"nothing loaded yet", which is the case that does send it.
@meocong
meocong merged commit ff82219 into master Aug 14, 2026
10 checks passed
@meocong
meocong deleted the feat/warm-models-and-compose branch August 14, 2026 05:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant