feat(backend): chat context compaction

[MesoX]

Chat context compaction: keep chats alive past the model's context window

Design doc: docs/adr/0012-context-compaction.md (ADR-0012)

Problem

Chats hard-fail the moment their message history exceeds a model's context window. The AI SDK (ai@6) reports real token usage after each call, but it exposes no context-window metadata on the model interface and no pre-call tokenizer. Providers also disagree on whether the window is discoverable at all: Google, OpenRouter, and vLLM expose it via their APIs, while OpenAI, Anthropic, and Bedrock do not. On top of that, the existing error handling only covered auth, rate-limit, and 5xx failures, so a context-overflow rejection killed the turn outright.

What this PR does

This change adds a two-tier, view-not-delete compaction model. It is fed by a single token estimator, all of its durable state is mutated through a single versioned compare-and-swap (CAS) writer, an always-on recovery net catches the overflow errors that the proactive path misses, and a deterministic context-editing pass prunes stale, bulky tool results without ever calling a model. Because top-level chats and sub-agents both run through the shared agent-runner/ToolLoopAgent, one implementation covers both.

The full rationale — including every alternative we rejected and why — lives in ADR-0012. The key decisions, summarized from it, are below.

Load-bearing principles

• View, not delete. The watermark and summary change what is sent to the model, never what is stored. Raw messages persist in the database untouched, so compaction is non-destructive in the data sense: a user can still read the full history, and a future "expand summary" UI comes for free. This reduces the "irreversible data loss" objection to a matter of UX courtesy rather than correctness. A summarized message is never hard-deleted.
• One estimator. Token counting lives in exactly one function operating over one neutral structure (CountUnit[]). Both Tier 1 (UIMessages) and Tier 2 (ModelMessages) normalize into it, and both count only model-bound parts (text, tool-call, tool-result, file, image). Divergence between the two tiers is therefore impossible by construction, because a tier cannot fire on a number the other tier never sees.
• One durable writer. Every mutation of compaction state (summaryWatermark, contextSummary, compactionDirty) goes through a single compare-and-swap keyed on a version column. Concurrency — for example a trigger run racing a user run — and the interaction between compaction and history-edit invalidation are both resolved by version rather than by comparing watermark values. That matters because a watermark can move backward on a history edit, and a value-based comparison would misread that as "not yet advanced" and write a stale summary over mutated history. On a CAS conflict the loser re-reads the row: if the winner already covered its prefix it skips, otherwise it retries once and then skips with a contended warning. There is no recompute loop and no livelock.
• Recovery is the net. When a 400/413 overflow error is caught, the messages are aggressively trimmed in memory (through the same Tier 2 adapter, so there is no bespoke trimmer) and the call is retried exactly once. Recovery flags compactionDirty and leaves the durable compaction to the next turn. Crucially, recovery stays on even when proactive compaction is globally disabled — it is the last line of defense, not a risk surface.

Mechanisms

• Window resolution. resolveContextWindow(provider, modelId) resolves a window per model, in order: a manual override (provider.modelMeta), then provider API auto-detection (Google / OpenRouter / vLLM), then the community-maintained litellm registry JSON (which covers OpenAI / Anthropic / Bedrock), and finally a conservative 8192 default. We deliberately do not maintain our own context-window table. Key normalization is boundary-safe, results are cached per provider+model with a TTL, the cache is evicted immediately on a modelMeta edit, default results use a short TTL, and API fetches use a 5 s timeout behind a single-flight guard.
• Token estimation. The estimator uses char/4 over text parts only (it never runs char/4 over a base64 image) and a modality table to size non-text parts. It is used only on the first turn, before any provider usage exists; every later turn uses the real, provider-reported usage.inputTokens. We accept the first-turn imprecision — guarded by a 1.15 cold-start margin and the recovery net — rather than ship a per-provider tokenizer.
• Tier 1 — cross-turn (durable). This runs in prepareChatTurn over the durable history. Its budget math works off the input budget (window − maxOutputReserve − safetyReserve), with a trigger ratio of 0.8 and a target of 0.5. The two ratios are deliberately distinct to give hysteresis, so compaction does not re-fire on the very next turn. The work is staged cheap-first: Stage 0 context-edits (pruning bulky old tool results with no model call), Stage 1 prunes the older prefix, and Stage 2 summarizes the prefix into a single synthetic message only if the conversation is still over target. Tool-call/result pairs stay atomic across the keep boundary, summarization is incremental (only the messages after the watermark), and a visible context-compacted event keeps the behavior fail-loud.
• Tier 2 — intra-turn (in-memory). This handles a single heavy response whose own tool loop bloats the window mid-loop. It runs in the SDK prepareStep hook on both streamText and generateText, summarizing old completed tool results while keeping recent steps verbatim and preserving call/result pairing. It fires only when genuinely near the limit, and it is not persisted — the next turn's Tier 1 folds the result into the durable summary.
• Sub-agents. Sub-agents start fresh on each invocation and carry no cross-turn history, so they use Tier 2 only. Each one resolves its own model's window and output limits, and recovery covers them as well through the shared runner.
• Config and kill switch. Compaction behavior is global (DEFAULT_COMPACTION_CONFIG); only the window/output size is per model. Setting COMPACTION_ENABLED=false disables all proactive compaction in production without a deploy, and recovery ignores that flag. Per-agent tuning was shipped and then removed, because no surveyed tool exposes it and the ratios already self-normalize to the model window.

Frontend

• Context-usage ring. A small ring next to the model selector fills as usedTokens / contextWindow, ramping from green to amber (≥ 0.7) to red (≥ 0.9), and rendering a neutral grey when the window is unknown or defaulted rather than guessing a ramp. The window comes from the currently selected model, and the numerator is the last response's peak contextTokens.
• Per-message stats. An (i) popover under each assistant response shows input/output tokens, time-to-first-token, and total generation time, reusing the existing tool-call timing mechanism.
• Force-compact on demand. The ring is clickable: POST /chats/:id/compact runs Tier 1 once regardless of the threshold and returns the post-compact usage so the ring refreshes immediately. A click is deferred while a response is streaming, and a confirmation dialog appears only when the drop would be significant.
• Compaction trace. Compaction is surfaced as a synthetic compact_context tool-call/result pair that reuses the tool-call UI. The trace part is stripped before convertToModelMessages, so it never replays to the provider as a phantom tool call.

Notable rejected alternatives (full list in ADR §Considered Options)

• Single-tier compaction (cross-turn only). Rejected because it cannot rescue a single response whose own tool loop overflows the window.
• Hard-deleting or truncating old messages. Rejected because it is irreversible and reproduces the "silently drops the middle" failure mode seen in gateway truncation. View-not-delete keeps the data and makes the action auditable.
• A homegrown context-window lookup table. Rejected because it is unmaintainable across providers; the litellm registry is the industry's "don't maintain your own table" answer.
• A real pre-call tokenizer. Rejected for v1 because it is a heavy per-provider dependency for a number the provider returns accurately after the very first call.
• Optimistic concurrency by comparing watermark values. Rejected because it breaks when invalidation moves the watermark backward; versioned CAS removes that monotonicity assumption.
• Compacting only to the trigger threshold. Rejected because it re-fires every turn (the thrash failure mode); distinct trigger and target ratios are what provide hysteresis.
• A token floor on the trigger (max(window × pct, 64000)). Rejected because it overflows sub-64k models, where the trigger would never fire. A floor belongs only on the window fallback, never on the trigger.

Schema changes (additive, nullable/defaulted)

• provider.modelMeta (JSONB) — per-model window/output overrides.
• chat/run gain contextSummary, summaryWatermark, compactionDirty, and version.
• Migration: apps/backend/drizzle/0046_context_compaction.sql.

Rollout is lazy and there is no backfill: existing chats compact only on their next turn. An eager backfill would create a thundering herd of summarize calls.

Cross-tenant safety

The submit route verifies that the body id belongs to the caller's workspace before a run starts. The compaction store is keyed by chat id alone, so without that check one workspace could mutate another's summary and watermark.

Observability

The feature emits structured metric:-tagged log lines: compaction.fired, summarize.latency_ms, recovery.*, context_window.fell_to_default, litellm.key_miss, cas.conflict, and context_edited.

Config

New apps/backend/.env.example keys, all optional with built-in defaults: COMPACTION_ENABLED, COMPACTION_TRIGGER_RATIO, COMPACTION_TARGET_RATIO, COMPACTION_RESERVE_RATIO, COMPACTION_KEEP_RECENT, COMPACTION_MIN_PRUNABLE_CHARS, and COMPACTION_MIN_RECENT_PRUNABLE_CHARS.

Tests

New suites cover compaction.test.ts, context-window.test.ts, token-estimate.test.ts, and recovery.test.ts; existing suites for agent-runner.test.ts, chat.test.ts, chat-execution.test.ts, sub-agent.test.ts, and packages/schemas/index.test.ts were extended. Recovery's per-provider overflow regex is fixture-tested across OpenAI/vLLM, Anthropic, Google, and Bedrock.

Known limitations (ADR §Open/deferred)

• A single oversized newest result. A tool result too large to fit even as the newest message still hard-errors; the fix is an ingestion cap at storage time, which is out of scope here.
• A few items are consciously deferred and are not needed for the "no more hard fails" goal: a live Pending → Running compaction trace (it currently renders post-hoc only), the estimate-vs-real divergence metric (log-only for now), and content-type tool-result media accounting.

🤖 Generated with Claude Code