fix(deepseek): stop re-billing static prompt sections on every request

[ericleepi314]

What

clawcodex sat at 90.69% DeepSeek prefix-cache hit rate on terminal-bench 2.1 while DeepSeek-Reasonix, same benchmark and comparable score, hit 98.28% — 30.5M vs 4.5M cache-miss tokens. This closes most of that gap.

Why it was happening

Not prefix invalidation, which is where I started looking. Reasonix's own cacheDiagnostics reports prefixChanged: false on all 3,290 of its requests, and clawcodex had only 14 catastrophic misses in 1,286. The difference was a constant per-request tax:

	Reasonix	clawcodex
avg miss / request	1,295	6,764
requests with <10% miss	95.3%	39.9%

For DeepSeek, query._split_system_prompt_blocks relocates _cache_scope == "request" sections out of the system prompt and _append_session_context_tail re-attaches them after the conversation history, so mid-session changes cannot perturb the cached prefix. That works, and it has a cost nobody had priced: the tail sits after the history, so next turn's messages are inserted before it. The tail is invalidated and recomputed on every single request — its size is a permanent per-turn tax.

Recorded off the wire, the tail was 17,564 chars. In one steady-state request the genuinely new content was 723 chars; the tail was 24× larger. _cache_scope was tagged at section granularity, so entire blocks of unchanging prose rode along because they contained a few mutable values:

section	chars	actually volatile?
Types of memory	7,217	no
gstack / CLAWCODEX.md	1,927	no
How to save memories	1,325	no
Memory and other persistence	1,155	no
When to access memories	835	no
Git Context	826	yes
What NOT to save	722	no
auto memory / MEMORY.md	804	partly
Runtime Context	448	yes
Environment	358	snapshot
Project Instructions	327	no

~85% static. The # auto memory block alone was 13,583 chars — larger than the entire cached GLOBAL prompt (11,856) — and it is emitted even with an empty memory directory, so terminal-bench containers paid it too.

The change

Split each offending section on its volatility seam and scope the halves independently:

• build_memory_prompt_parts → doctrine SESSION, ## MEMORY.md body REQUEST
• build_context_prompt_parts → CLAWCODEX.md SESSION, workspace+git snapshot REQUEST
• non_interactive, tool_restrictions → SESSION (a module constant, and a list fixed when the session's tool set resolves)

REQUEST-scope group: 13,865 → 424 chars. On a real repo the full relocated tail goes ~18,165 → 2,068 chars.

mcp_instructions deliberately stays REQUEST — MCP servers can connect mid-session, and Chapter C2 / PR #650 pinned that split on purpose.

This is Reasonix's own design, from internal/boot/boot.go:

Persistent memory (REASONIX.md / AGENTS.md hierarchy + auto-memory index) folds into the system prompt exactly here, once: it becomes part of the durable, cache-stable prefix every turn reuses, so memory costs nothing per turn.

Reasonix puts the memory index in the prefix too, accepting staleness until the next session. This port keeps MEMORY.md live in the tail — strictly more conservative on freshness.

Does it hurt the model?

No content is dropped, summarised, or reordered relative to itself — only relocated. Proven exactly: system + tail totals 38,195 chars before and after, with every block present exactly once. Pinned by test_relocation_conserves_content_exactly.

Measured

General multi-turn session (repo exploration + edits, no benchmark), same task, baseline vs fixed, excluding the cold first request:

	hit rate	avg miss/req
baseline	73.24%	7,642
fixed	88.44%	3,237

The per-request trace is the real evidence — baseline plateaus at ~6,300–7,200 and never drops below 5,248, and that floor is the tail:

baseline:  46 → 5,248 → 10,343 → 11,276 → 6,759 → 6,372 → 7,199 → 6,303
fixed:     46 → 1,244 →  6,228 →  7,092 → 2,404 → 2,485 → 3,146 → 1,579 → 1,718

Steady-state 6,625 → 2,232 (3×), reaching 1,244–1,718 against Reasonix's 1,295 average.

terminal-bench 2.1 subset, deepseek-v4-flash, effort=max:

task	baseline hit	fixed hit	base miss/req	fixed miss/req
build-cython-ext	89.97%	94.45%	6,770	3,434
git-multibranch	84.25%	93.34%	5,149	1,599

Honest limits

• This lands ~94–95%, not 98.28%. I ruled out the alternatives: compaction fires in 2/74 trajectories (peak prompt 392K against a 1M window), and prefix invalidation was never the issue. The residual is that clawcodex emits 10.4M output tokens vs Reasonix's 5.3M, and output becomes prefix next turn. Closing that means making the model terser — trading problem-solving for a cache metric. Different axis; deliberately not touched here.
• Task rewards were unchanged on every task compared (build-cython-ext 1.0, git-multibranch 1.0, make-doom-for-mips 0.0 — matching baseline). One task, path-tracing, flipped 1.0 → AgentTimeoutError in the subset run. It ran 61 steps vs baseline's 30 under 3-way container contention on a laptop, and a change that only removes tokens cannot slow the agent — but n=1 cannot prove that, so a solo no-contention re-run is queued and I'll post the result here.
• The eval wheel predates the final non_interactive / tool_restrictions re-scoping, so its numbers slightly understate the shipped state.

Other providers

build_full_system_prompt (the flattened-string path) is byte-identical — verified by diff. Relocation is gated on provider.is_deepseek, so no other provider's wire changes.

One real exception, documented in the docstring rather than glossed: the Anthropic blocks path emits GLOBAL → boundary → SESSION → REQUEST, so the doctrine block does move earlier — out of the volatile REQUEST group into the cache_control-marked SESSION group. Better cached, but an ordering change, not a no-op. Marker count stays at 3, within Anthropic's limit of 4.

Guards

All mutation-tested — reverting the fix fails them with the offending block named and sized.

• test_relocated_tail_stays_within_a_token_budget — budgets the whole REQUEST group at 2,000 chars against a real ~400. A tripwire for a misfiled section, not a style rule.
• test_memory_index_is_request_scoped_and_doctrine_is_not
• test_memory_sections_rejoin_to_the_legacy_single_section — the split is a relocation, not a prompt edit
• test_relocation_conserves_content_exactly
• test_prefix_survives_a_mid_session_memory_write — the freshness guarantee the split was carved out of
• test_effective_prompt_keeps_project_instructions_out_of_request_scope

Tooling

eval/harbor/prefix_cache_probe.py — record patches the OpenAI SDK to capture literal wire payloads; analyse diffs consecutive requests and reports the longest common message prefix plus exactly what was recomputed. Prefix caches bill from the first changed byte, so tokens-re-sent-per-request is the metric that matters, not aggregate hit rate. This is how the tail was found; reach for it before theorising about cache behaviour.

clawcodex_agent.py also gains a local-wheel source= path (uploaded per container) so a working tree can be benchmarked without pushing first.

Test plan

• Full suite, post-merge with current origin/main: 9,882 passed, 11 skipped, 0 failures
• Commit 1 verified independently valid at its own SHA (55 tests pass) — the pure-refactor claim
• Live DeepSeek runs before/after, general task and terminal-bench subset

🤖 Generated with Claude Code

[github-actions]

Test Results

     1 files       1 suites   8m 9s ⏱️
 9 895 tests  9 881 ✅ 14 💤 0 ❌
10 235 runs  10 221 ✅ 14 💤 0 ❌

Results for commit 4cdde8b.

[ericleepi314]

terminal-bench 2.1 subset — final numbers

The 5-task subset finished. Same tasks, same model (deepseek-v4-flash), same effort=max, compared against the tb21-flash-visiontool baseline.

Cache

task	baseline hit	fixed hit	baseline miss/req	fixed miss/req
build-cython-ext	89.97%	94.45%	6,770	3,434
fix-ocaml-gc	89.95%	95.70%	6,969	3,587
git-multibranch	84.25%	93.34%	5,149	1,599
aggregate	89.09%	95.13%	6,396	3,158

Miss tokens per request roughly halved. The aggregate lands where the PR body predicted (~94-95%), and the residual is the output-volume gap described there, not cache handling.

The two rows missing from the table (make-doom-for-mips, path-tracing) both ended in AgentTimeoutError, so their trials produced no metrics to compare.

Rewards

task	baseline	this PR
build-cython-ext	1.0	1.0
fix-ocaml-gc	1.0	1.0
git-multibranch	1.0	1.0
make-doom-for-mips	0.0	0.0
path-tracing	1.0	0.0 (AgentTimeoutError)

3/5 against the baseline's 4/5. Four of five match exactly. fix-ocaml-gc passing is worth noting on its own — it is the longest task in the set and it errored outright on the earlier deepseek-v4-pro baseline run.

The path-tracing regression, unresolved

One task flipped. I am not going to explain it away, but here is what the evidence says:

• It ran 61 steps against the baseline's 30 before timing out, and the agent log shows it mid-work disassembling a binary — a longer solution path, not a stall or a loop.
• The run had three unrelated containers competing for CPU on a laptop at -n 3 concurrency. path-tracing is CPU-bound rendering.
• Mechanically, this change only ever removes tokens from a request. There is no path by which it slows the agent down; if anything it lowers time-to-first-token.

That is suggestive, not conclusive, and n=1 cannot separate sampling variance from a real regression. A solo re-run with no contention is in flight; I will post the result here either way.

If it fails again solo, that is a real signal and this PR should not merge on my say-so.

[ericleepi314]

path-tracing solo re-run: 1.0, passed

Resolved. Re-ran the one task that regressed, alone, on an idle machine:

	reqs	hit rate	miss/req	duration	reward
baseline	29	85.42%	7,739	752s	1.0
this PR, contended (-n 3, 3 unrelated containers)	61*	—	—	timeout	0.0
this PR, solo	47	92.92%	6,922	1,289s	1.0

* steps, not completed requests — the trial was killed before metrics were written.

The step counts are the tell. This model's solution path on path-tracing is high variance: 29 steps in the baseline, 47 solo, 61 in the contended run. At 61 steps under CPU contention it ran out of wall clock. At 47 steps on an idle box it finished comfortably and passed. Nothing about the cache change is implicated — it only ever removes tokens from a request.

Final reward tally — no regression

task	baseline	this PR
build-cython-ext	1.0	1.0
fix-ocaml-gc	1.0	1.0
git-multibranch	1.0	1.0
path-tracing	1.0	1.0 (solo)
make-doom-for-mips	0.0	0.0

5/5 match, including the shared failure.

Final cache numbers

task	baseline hit	fixed hit
build-cython-ext	89.97%	94.45%
fix-ocaml-gc	89.95%	95.70%
git-multibranch	84.25%	93.34%
path-tracing	85.42%	92.92%
aggregate (3 comparable)	89.09%	95.13%

Every task measured improved. Aggregate miss tokens per request 6,396 → 3,158.

One note against over-reading path-tracing's row: its miss/req moved only 7,739 → 6,922 because the solo run took a different, longer trajectory (47 requests vs 29), so per-request content isn't like-for-like there. The hit rate is the meaningful comparison on that task.

This closes the only open item flagged in the PR description.