fix(gpu): harden multi-user GPU contention — gate /health, align Ollama, jittered desk retry (ADR-0040)

[MikkoNumminen]

What

Answers "what happens when many users hit the shared GPU at once — do we have a queue?" and fixes the three sharp edges an audit surfaced. The design answer stays shed, not queue (right for an interactive desk on one 8B GPU); this PR makes the shed sound under a crowd.

Why not just use Ollama's queue?

We don't want a deep queue at all. Ollama's default OLLAMA_MAX_QUEUE=512 on one 8B GPU means an interactive "Tulkitse" click could wait minutes — a fast "busy, try again" beats a spinner that resolves in three minutes. The app-side gate also carries policy Ollama can't: cap this app at 2 concurrent so it stays a good neighbour to the sibling RAG, and degrade per-endpoint (ingest sheds and stores nothing so the client retries; the report swallows the shed and renders its deterministic layer). So the fix isn't to adopt Ollama's queue — it's to make Ollama agree with the gate.

Fixes (ADR-0040)

1. /health was the one ungated LLM path — it fired a real completion on every probe with no slot, adding concurrent GPU load and able to steal a slot from a live request. Now LlmGate.TryRunAsync (non-blocking WaitAsync(0)): probe only if a slot is free; a saturated gate is healthy (busy ⇒ the model is demonstrably loaded and generating) with no completion; a real failure still 503 llm_unavailable. /health consumers read status == "ok" and need no change.
2. Ollama aligned with the gate (docker-compose.yml): OLLAMA_NUM_PARALLEL=2 (both admitted slots run in parallel, not serialized) + shallow OLLAMA_MAX_QUEUE=8 (no hidden 512-deep backlog under the fast-shed gate). Env-overridable; safe on VRAM since this container runs only while the RAG is down.
3. Desk resilience (desk.html): a 503 retries a bounded few times with jittered backoff so transient sheds self-heal and a roomful of clerks desynchronize; a 429 (shared rate bucket, not the GPU) shows a distinct wait message and is never retried.

Verified

• Unit (LlmGateTests, 4 new; 170/170 API tests green): TryRunAsync runs when a slot is free; returns false without blocking or running when both slots are held (proving /health adds no GPU load); releases the slot even when work throws; RunAsync sheds LlmBusyException when no slot frees within the acquire timeout.
• Runtime, failure path (scratch API, Ollama unreachable): /health → 503 {"status":"llm_unavailable"} across three repeated probes — a leaked slot would have flipped the third to 200 ok, so this proves the slot is released each time; /interpret → 503 (the desk retry loop's trigger); /live/feedback → 200 (feedback stays visible with no LLM).
• Not driven live: the real-GPU idle probe (200 ok through the new wrapping) — the sibling RAG stack is up and owns the GPU, so the isolation rule blocks starting this project's Ollama. The completion call itself is byte-identical to before; only the slot wrapping is new, and that is unit-tested + exercised on the failure path.

Accepted, not fixed (documented in ADR-0040)

• POST /live/restructure holds a slot per item with no priority over live visitors — but it's operator-only (loopback-exempt, not in the public proxy allowlist).
• The browser-path rate bucket is shared across visitors (one Azure egress IP); at demo scale (240/60 s ≈ 4 req/s) it's ample, and the 429 message + jittered retry are the mitigation.

Docs

AI-first docs updated: yes — ADR-0040 (+ index row), operations.md containment table.

Deploy note

OLLAMA_NUM_PARALLEL / OLLAMA_MAX_QUEUE take effect on the next docker compose up -d ollama (container recreate) — a running retail-rag-ollama keeps its old env until then.

[MikkoNumminen]

Live real-GPU verification (deferred item in the PR body, now completed)

Ran master (post-merge) against the recreated Ollama container on the freed GPU (sibling RAG down).

Fix #2 — Ollama aligned: recreated container reports OLLAMA_NUM_PARALLEL=2, OLLAMA_MAX_QUEUE=8; Poro persisted through the recreate. Two concurrent /interpret completed in 1265 ms total ≈ 1× a single call (1511 ms), not the ~3 s of serialized execution — so both admitted slots genuinely run in parallel.

Fix #1 — /health gated:

• Idle: 200 {"status":"ok"} via a real 1-token probe (265 ms warm).
• Gate saturated (both slots held by 2 in-flight interprets): /health → 200 {"status":"ok"} in ~60 ms — no completion run, so it adds zero GPU load and steals no slot. The latency drop (60 ms vs 265 ms) is the short-circuit doing its job.

Shed-not-queue (the original question): with the 2 slots held, a 3rd concurrent /interpret returned 503 — the fast shed the desk now retries with jittered backoff.

Cold-model note: right after the container recreate, /health returned 503 llm_unavailable until the 4.9 GB model finished its ~45 s first load — pre-existing "warming?" behavior (the 10 s HealthTimeoutSeconds is shorter than a cold load), not introduced here.