Implement ADR-018 (persistent model + stateful sessions) + P0 performance ADRs

[Tovli]

Summary

Authors the first-milestone (EPIC 1–4) ADRs for the edge-inference performance work derived from docs/research/improvements-plan.md, and implements ADR-018 (persistent model instances + stateful inference sessions).

Headline change: the local Qwen chat backend now loads model weights once and reuses a resident session across turns, instead of rebuilding the engine and re-reading the GGUF on every chat — the dominant per-turn cost for a small on-device model.

ADRs (docs)

ADR	Decision	Status
018	Persistent model instances + stateful sessions	implemented here
019	In-loop incremental decoding + token streaming	proposed
020	Batched (single-pass) prefill	proposed
021	Memory-mapped verified GGUF loading	proposed
022	Two-tier quantized KV cache + attention-aware eviction	proposed
023	Baseline performance instrumentation	proposed

019–023 are design records only; just 018 is implemented in this PR.

ADR-018 implementation

Verified enabler: candle quantized_qwen2 attention discards/replaces its KV cache on a forward at index_pos == 0, so the engine is reusable without reloading weights.

• el-runtime — new InferenceEngine::reset_cache (release a conversation's KV, keep weights), distinct from rollback. reset()/close() return Result and propagate failures (state untouched on error); close(&mut self) releases the conversation while keeping the model resident and the session reusable; reset() preserves undrained events (turn isolation is the provider's job).
• el-engine-candle — QwenChatProvider holds the engine in a resident Mutex<ChatSession> (lazy Loaded→Active so builder config is final), reused per turn — the per-turn QwenEngine::from_path disk reload is gone. QwenEngine::reset_cache evicts candle's KV in place via a benign position-0 forward, gated by a cache_dirty flag so a partially-failed eviction is retried, not skipped; prompt/logits buffers are released (Vec::new), not just cleared. QwenChatProvider::end_session() exposes the explicit conversation release.
• el-chat — /reset, /system, and generation-error recovery release the conversation before reporting success, and stop (dropping the provider → KV freed via ownership) if it can't be cleared, rather than failing open.

Scope

In scope: AC-1 (weights load once), AC-2 (no per-turn reload), AC-4 (conversation memory measurable + explicitly released), and the engine seam.

Deferred (documented in the ADR-018 implementation status): AC-3 cross-turn prefix reuse / incremental prefill; safety-expert persistence; concurrent multi-session weight sharing (candle couples weights + KV cache — same root constraint noted in ADR-022).

Verification

• cargo test --workspace — green (el-runtime 24, el-engine-candle 20, 0 failures).
• cargo fmt --all -- --check — clean.
• cargo clippy --all-targets -- -D warnings — clean on the changed crates + el-chat.

Notes

• The safety follow-up ADRs (014–017, from docs/followups.md) are in a separate open PR (add safety ADRs #9) on codex/implement-followups; they are intentionally not part of this branch, so the index here lists 013 → 018–023.
• Hardened over several review rounds: KV-cleared-on-session-end privacy, fallible cache-clear retry, provider-owned event isolation, buffer release, and CLI fail-closed cleanup (see the ADR-018 "Review fixes" section).

🤖 Generated with Claude Code