Skip to content

Evict terminal chat runs to bound ChatRunRegistry memory#61

Draft
TYRMars wants to merge 1 commit into
mainfrom
claude/vibrant-dijkstra-Bz60a
Draft

Evict terminal chat runs to bound ChatRunRegistry memory#61
TYRMars wants to merge 1 commit into
mainfrom
claude/vibrant-dijkstra-Bz60a

Conversation

@TYRMars
Copy link
Copy Markdown
Owner

@TYRMars TYRMars commented May 30, 2026

Summary

Fixes #50. ChatRunRegistry never evicted terminal (completed/failed/cancelled) runs from its in-memory inner map, so memory grew unbounded over the lifetime of a long-lived server — one retained entry per distinct conversation_id ever run, each holding up to MAX_EVENTS_PER_RUN buffered records plus a live broadcast channel.

Fix

Lazy eviction on the existing mutation paths — no background task:

  • TTL: terminal runs are dropped TERMINAL_RETENTION_MS (5 min) after they finish, leaving a window for late WS reconnects to replay the final frames and observe channel closure before the state disappears.
  • LRU backstop: retained terminal runs are capped at MAX_RETAINED_TERMINAL (256), dropping the oldest by terminal timestamp — guards against a burst of conversations all completing inside the TTL window.

Active runs (terminated_at == None) are never counted against the cap or evicted. The terminal timestamp is stamped exactly once on the first terminal transition via mark_terminal, which also drives the one-shot abort-handle cleanup. The sweep (prune_terminal_locked) runs from try_start and from the terminal branch of push_frame / update, all already under the inner write lock, so it adds no new locking and only fires on run start / completion (not per token-delta).

Tests

Added coverage in chat_runs::tests (via a force_prune test seam that injects the clock so TTL is testable without sleeping):

  • terminal_run_is_evicted_after_ttl — kept within the window, swept past it
  • active_run_is_never_evicted — even with an absurd clock
  • cancelled_run_is_evicted_after_ttl
  • terminal_runs_are_capped_by_lru_backstop
  • restarting_an_evicted_conversation_succeeds

cargo test -p harness-server chat_runs:: (11 passed) and cargo clippy -p harness-server --all-targets -- -D warnings both green.


Generated by Claude Code

ChatRunRegistry retained every terminal (completed/failed/cancelled)
run in its in-memory map for the whole process lifetime, leaking one
entry per distinct conversation_id ever served.

Add lazy eviction on the existing mutation paths (no background task):
- TTL: terminal runs are dropped 5 minutes after completion, leaving a
  window for late WS reconnects to replay final frames.
- LRU backstop: cap retained terminal runs at 256, dropping the oldest
  by terminal timestamp, to guard against a burst completing inside the
  TTL window.

Active runs are never evicted. Stamp terminated_at on the first
terminal transition (mark_terminal) and sweep from try_start / terminal
push_frame / update. Closes #50.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ChatRunRegistry never evicts terminal runs — unbounded in-memory growth over server lifetime

2 participants