perf(extract_llm): prompt-cache the preamble, global result cache, economic gate, and value-based triggering (measured ~8x underwater)

[OsherElhadad]

Problem statement

extract_llm is the only component that spends money to save money, and on Terminal-Bench it lost: 271 calls, $3.26, ~1,592 s cumulative latency (~450 ms/req) against ~197,548 unique tokens saved — measured ~8× underwater against the cache-aware value of those tokens. Worse, roughly 93% of its realized value came from the replay cache, not from the LLM: most calls re-derived a result the system already had, or should have had.

It also has the worst ergonomics of any component: a hand-tuned min_tokens threshold that must be re-picked per workload.

Current behavior

• No prompt caching. internal/cheapmodel/anthropic.go:32-48 sends the whole prompt as a single user message — no system block, no cache_control. The ~852-token invariant preamble is billed as fresh input on every call.
• Result cache is session-scoped. Keys carry a session prefix, so identical content in a different session is re-derived. Measured: 82 of 103 unique contents recurred across sessions.
• No economic gate. The component decides on size and cadence (extract_llm.go:66-71, :127-128 — llm_every_n_requests, llm_max_per_request), never on expected value. Note it does already carry a partial insight (:82-90): on a caching backend, skeletonizing small outputs "saves almost nothing yet costs the compaction LLM + one-time cache-write transitions → +30% billed cost." That reasoning is right and needs to become an actual gate.
• Trigger is a raw token threshold. min_tokens + trigger.min_request_tokens, both manual.

Motivation

The request is ~99.95% cached, so a token removed from a cached region is worth $0.20/M, not $2/M. An LLM call that costs ~$0.012 must therefore remove a lot of cached tokens to break even — and on a prompt-caching backend it frequently cannot. Either the component becomes cost-aware or it should be off by default on caching backends. See docs/results/improvement-plan.md §B3.

Desired behavior

A. Prompt-cache the fixed preamble

Move the invariant instructions into a cacheable stable system block with cache_control, so repeated calls pay cache-read ($0.20/M) instead of fresh input on ~852 tokens. Requires extending cheapmodel beyond its current single-user-message shape (anthropic.go:48) — keep the interface honest for the OpenAI backend too (internal/cheapmodel/openai.go), which has no explicit breakpoints. Measure the actual effect; do not assume it.

B. Investigate reusing the agent's own cached prefix

If the extraction model is the same as the agent model, and it is semantically safe, and the provider's caching rules allow it, and it does not poison the agent's own cache — investigate appending the extraction instruction as a final user message after the existing stable prefix, so extraction reuses the already-cached context.

Do not implement this blindly. Prototype, measure, and decide on evidence. Reasons it may be wrong: it would send a ~1.7M-token context to answer a question about one tool output (cache-read is cheap but not free); it risks a cache-write on the agent's prefix, which at 11.5× a read is exactly the mistake this whole workstream is about; and it couples the compaction model to the agent model. Record the decision either way.

C. Global content-hash result cache

Re-key extraction results primarily on content hash, globally, where semantically safe. The same exact content with the same extractor semantics should reuse the previous result.

The key must include everything that materially affects the result: content hash, prompt/extractor version, model, and the relevant component configuration. Getting this wrong means silently serving a stale extraction after a prompt change — add the version field first.

Contrast with #27, which is deliberately session-scoped: xdedup makes a conversational reference ("same as step N") that only makes sense in-session, while this caches a derived result that is context-free.

D. Economic gate

Only call the LLM when expected_saving_$ > expected_call_$. Compute the call cost from actual model pricing and token usage rather than hard-coding (~$0.012/call is today's estimate, not a constant). The gate should account for: expected number of future reuses, provider caching (which changes a saved token's value 10×), current context size, expected compression ratio, extraction model cost, remaining session horizon where estimable, and whether the content is likely to recur.

Where caching already makes extraction pointless, suppress it. On non-caching backends or high-value repeated content, allow it.

E. Better triggering

No fragile per-workload threshold. Investigate: context pressure, context growth rate, repetition, expected reuse, content type, previous component effectiveness, expected savings, model context-window pressure, amortized economics, and frequency of similar outputs. It must not run every step unnecessarily, and must fire when value is actually likely. Ship sensible defaults.

F. Metrics

Expose: extraction calls; calls avoided by result cache; calls suppressed by the economic gate; prompt-cache hit/read behavior; extraction LLM cost; gross savings; net savings after extraction cost; average latency; global cache hit rate; trigger reason.

Net-after-cost is the honest headline — the current /stats shape reports the tool's LLM cost (metrics/metrics.go:213-215) separately from savings, so nothing shows the component underwater.

Relevant code locations

• components/offload/extract_llm.go — :55 trigger, :62 llmSeen, :66-71 config, :82-90 the existing caching insight, :90-108 defaults/construction, :127-128 cadence, :277 concurrency.
• internal/cheapmodel/anthropic.go:16-48 (single-user-message shape — the blocker for A), openai.go, usage.go (token accounting for the cost model).
• internal/extract/ — extract.go:52-54 (ContentKey), cache.go (result cache), prompt.go (the preamble to split), starlark.go.
• components/component.go:101-135 — Ctx, CacheAware (the signal the economic gate needs).
• metrics/metrics.go:90-100, :213-215 — per-component stats and LLM cost.

Proposed architecture

1. cheapmodel gains a system-block + cache-control capable call shape; Anthropic sets a breakpoint, OpenAI ignores it.
2. prompt.go splits into invariant preamble (cached) and per-call variable part.
3. Result cache key becomes sha256(content) + extractorVersion + model + configFingerprint, global namespace.
4. An expectedValue() helper reading Ctx.CacheAware, request size, and observed historical compression ratio, gating the call.
5. Trigger derived from context pressure + recurrence rather than a bare threshold; old keys kept working.

Alternatives considered

• Delete extract_llm. Defensible on today's numbers, and the honest fallback if A–E do not get it above water. But it is the only component that can compress novel prose/log shapes no deterministic rule anticipates, so fix it first.
• Keep it but off by default on caching backends. Effectively what the economic gate does, expressed as a default rather than a computation. Weaker: it cannot exploit the cases where extraction is worth it.
• Batch multiple outputs per call. Amortizes the preamble, but couples unrelated outputs and complicates attribution. Consider only if A does not deliver.

Observability implications

Everything in F, plus: expose the trigger reason per activation (an operator's first question is "why did this run?"), and log a suppression reason when the gate declines. Feed the dashboard's per-component economics panel (#30) so it is obvious whether the component earns its place.

Storage / data-model implications

Result cache namespace changes from session-scoped to global. Bound it. On a key-schema change, old entries must be inert (miss), never mis-served — version the key.

Backward compatibility

Existing configs must keep working; min_tokens stays honored if set explicitly, with the smarter trigger as the default when it is not. Changing codesmart's behavior is benchmark-visible and must be measured. Cache-key change invalidates existing entries once — acceptable, note it.

Configuration design

Prefer defaults over knobs. Retain model, strategy; keep min_tokens/trigger as explicit overrides; add nothing that is only meaningful to someone who has read the source.

Testing plan

• Unit: the preamble is sent as a cached system block; the request shape is correct for Anthropic and degrades cleanly for OpenAI.
• Unit: result cache hit across two different sessions with identical content.
• Unit: a prompt/extractor version bump misses rather than serving stale.
• Unit: the economic gate suppresses when cache-aware and the output is small; permits on a non-caching backend; permits for high-reuse content.
• Unit: the cost model matches a known token count × known price.
• Unit: the trigger does not fire every step on a growing context.
• go test -race (the component runs concurrent per-output calls, :277).

Real-world benchmark plan

2–3 Terminal-Bench tasks where extract_llm was most active (from the codesmart dump) and 2–3 SWE-bench tasks. Record commit SHA, config, task ids, extraction calls, calls avoided, calls suppressed, prompt-cache reads, extraction cost, gross and unique tokens saved, net after cost, latency added, plus all four token tiers, total billed cost, steps, and reward. The bar is explicit: is extract_llm economically positive after these changes? If it is not, say so and recommend disabling it by default.

Acceptance criteria

• Preamble prompt-cached; measured effect reported (tokens + dollars).
• B prototyped and a documented decision recorded, with numbers.
• Global content-hash result cache with a version-inclusive key; cross-session reuse demonstrated.
• Economic gate implemented from real pricing; suppresses when uneconomic; reason logged.
• Trigger works with defaults, no per-workload tuning.
• All F metrics exposed, including net-after-cost.
• Benchmarked on real tasks; a clear verdict on whether the component is now positive.
• go test -race green.

Documentation updates

docs/components/extract_llm.md (rewrite — economics, triggering, caching, the honest verdict), docs/reference/config.md, docs/reference/presets.md if codesmart/agent/general change, and update improvement plan §B3 from hypothesis to result.

Dependencies

Independent of the others in implementation, but its benchmark numbers are cleaner after #25 removes cache-write noise. Feeds #30 (per-component economics). Its global-cache design should be contrasted with #27's deliberately session-scoped index.