fix(cache): write cacheinject's breakpoints to the wire as metadata - #36
Conversation
cacheinject's breakpoints never reached the provider on Claude Code traffic. Measured over 40 captured requests: 46 breakpoints applied at the component level, 0 in the output body. The cause was structural. The component can only mark messages carrying content blocks; on this traffic those are exclusively assistant turns carrying tool_use. bifrost drops tool_use.id/name/input on unmarshal, so apply's losslessness guard discarded every change to such a message rather than splice a corrupted re-marshal — correct in itself, but it meant a component whose only possible targets were precisely the messages that could never be written back. Do not relax the guard; it prevents real corruption. Instead take the narrow exception the data model allows: cache_control is metadata, not content, so it needs no message model to express. When a component's only change to a message is an added cache_control key, write that key at its exact path on the ORIGINAL raw bytes via sjson. A write that reads no other field cannot drop one. metadataOnlyWrites enforces "only that" by diffing pre against post with the added keys removed; anything wider is still discarded. applyMetaWrites refuses if the raw block layout disagrees with the normalized view, and never overwrites a breakpoint the caller set. Fix the second, independent defect in the same change, because shipping the first alone produces a live 400. The provider caps cache_control at 4 across system + tools + messages together, and a component sees none of the first two — nor cache_control on blocks bifrost drops. On real traffic that hides all three of the agent's own breakpoints (2 in system, 1 on a tool_result block), so the component computed 3 free slots when 1 was free and emitted 6 on the wire. apply now counts them structurally from the raw body and passes the total as Ctx.ExistingBreakpoints; it also counts its own output and logs an error on a breach rather than waiting for the provider to reject the request. Make the failure class loud. A mutated-then-discarded component was indistinguishable from a working Reformat, which is why this survived two full benchmark studies. Pipeline.RecordDiscards attributes each thrown-away change back to the component that made it, surfacing as per-component discarded_changes and top_discarded in /stats. Both fields are additive; no existing /stats key is renamed or removed. Tests, all failing before this change: a cacheinject mark on an assistant tool_use message reaches the output body with id/name/input intact and nothing else altered; the wire total stays within 4 on the real (system=2, tools=0, messages=1) shape where the synthetic 60-message case previously produced 6; a discarded change increments the counter without inflating Runs. Replaying real captures, breakpoints now reach the wire on 86 of 92 requests (previously 1) with every tool_use provider field intact. Closes #32 Assisted-By: Claude Opus 5 Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
… measurement The components page credits cacheinject with the 97.8% cache-hit rate. It did not earn it: on that run its breakpoints never reached the wire (#32), so the rate is claude-code's own breakpoints, forwarded untouched. Note it in place rather than delete the row, so a reader comparing against an older build sees why the number moved. Assisted-By: Claude Opus 5 Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
OsherElhadad
left a comment
There was a problem hiding this comment.
Independent review — changes needed (do not merge as-is)
I did not write this code. Reviewed at db58dbf in a clean scratch worktree.
What I verified myself, and it holds up. Both claimed failures reproduce on origin/main with the exact quoted messages (expected cacheinject's breakpoint to change the body; 6 breakpoints on the wire). Full suite, -race on ./apply/ ./components/... ./metrics/ ./proxy/, and make lint are all green on the branch. The replay test passes on real captures with the reported numbers (capture-swe 16/19, capture-tb 70/73). The core mechanism is genuinely byte-clean: I stripped every added cache_control from the output of 115 real captured requests across three captures and the messages array came back byte-identical to the input in 115/115, including 8 requests where the mark landed on a tool_result block. tool_use.id/name/input survive. The correctness thesis is real and the fix is the right shape.
But there are five things wrong, two of them blocking.
1. BLOCKING — wireBreakpoints misses Bedrock cachePoint in system/tools, so the cap is still breachable
apply/metawrite.go:106-112 lists cachePoint only under messages.#.content.#.cachePoint. Bedrock puts cachePoint as its own array entry in system and tools — exactly the two places defect 2 is about. The count comes back 0 and the component budgets 4 free slots.
Constructed the shape and ran it: inbound system has one cachePoint, tools has one, 60 messages, provider Bedrock.
bedrock: pre-existing cachePoints in system+tools = 2; component added 4 message marks => wire total 6
Six on the wire, on this branch, and the cap-breach slog.Error at apply/apply.go:261 does not fire either — it uses the same blind counter. This is defect 2, un-fixed, on a provider the PR's own explicitBreakpointProvider and cacheAware() both declare in scope. docs/components/cacheinject.md:24 claims "the wire total is asserted never to exceed 4"; that assertion does not cover Bedrock.
Fix: add system.#.cachePoint and tools.#.cachePoint to breakpointPaths, and add the Bedrock shape to TestBreakpointCounting. Two lines and a test case. Note metawrite_test.go:47 uses only the Anthropic spelling, which is why this got through.
2. BLOCKING — discarded_changes misattributes, and can charge components that changed nothing on the wire
Two independent bugs in RecordDiscards, both reproduced:
(a) a REVERTED component is charged a discard. components/pipeline.go:98 sets rep.ChangedIdx = changedIdx(before, req.Input) before the switch below it that does req.Input = before on error / never-worse / offload-contract violation. A component that mutated and was then rolled back keeps a populated ChangedIdx and gets billed for someone else's discard:
testrevert: reverted=1 discarded=1 <- it contributed nothing; the pipeline undid it
top_discarded=[testrevert testrewrite]
Fix: move the ChangedIdx assignment into the default: (success) branch, or clear it wherever rep.Reverted = true is set.
(b) one discard is charged to every component that touched the message. With pipeline: [testrewrite, cacheinject] on one unmodellable message, exactly one change is discarded, but:
discarded per component: map[cacheinject:1 testrewrite:1]
top_discarded=[cacheinject testrewrite]
cacheinject is named as suppressed on a request where the discard was caused by the other component. Given the whole point of this counter is to make #32-class bugs loud, a counter that fingers innocent components will get ignored the first time someone chases a false positive. At minimum document it as "components implicated" rather than "changes discarded", or attribute to the last writer of each index.
3. slog.Error on cap breach fires on traffic context-guru never touched — including under bypass
apply/apply.go:261 counts the output unconditionally. Under bypass=true the pipeline is a no-op, yet a client that itself sends 5 breakpoints produces:
2026/08/10 07:20:45 ERROR context-guru: cache breakpoint count exceeds the provider cap breakpoints=5 cap=4 session=...
A hard ERROR blaming context-guru for the client's request. Same on the non-bypass path when inbound was already over 4. The comment says "should be unreachable: the component budgets against OuterBreakpoints" — also note that field is named ExistingBreakpoints, not OuterBreakpoints; fix the comment. Gate the log on n > max && n > wireBreakpoints(body) (i.e. we made it worse), or skip it entirely when bypass.
4. The stated root cause of "the third breakpoint is invisible" is wrong — bifrost is not the culprit
Every comment and doc paragraph in this PR says bifrost drops cache_control on tool_result blocks: components/component.go:132, components/reformat/cacheinject.go:149, apply/metawrite.go:117, docs/design.md:150, docs/components/cacheinject.md:21 and :157. I tested it directly and it is false — bifrost round-trips cache_control fine on both tool_result and tool_use:
cc on tool_result(string): rt={"role":"user","content":[{"type":"tool_result","cache_control":{"type":"ephemeral"}}]}
cc on tool_use: rt={"role":"assistant","content":[{"type":"tool_use","cache_control":{"type":"ephemeral"}}]}
The actual cause is this repo's own normalize(): a string-content tool_result is expanded into a synthetic role=tool message via toolMessage() (apply/apply.go:~455), which builds a fresh ChatMessage and never carries the block's cache_control forward. Confirmed:
norm[0] role=tool slotKind=1 blockCacheControlVisible=false <- string-form tool_result: mark lost
array-form norm[0] role=user slotKind=0 blockCacheControlVisible=true <- array-form: mark visible
The chosen fix still works, because it routes around visibility entirely. But five comments and two docs pages now assert a bifrost defect that does not exist, and the next person to touch this — or to consider the "upgrade bifrost" alternative the issue lists — will be reasoning from a false premise. Please correct the attribution to normalize/toolMessage. It is also a smaller latent bug worth its own line: toolMessage losing cache_control is the reason hasBreakpoint can double-mark.
5. ~+65% hot-path latency, and the expensive half is the observability counter
Benchmarked apply.Body with preset: general on a real 100 KB+ captured request, 100x, 3 counts, same box, this branch vs origin/main:
branch: 5164679 / 5161131 / 5133808 ns/op
main: 3063430 / 3407773 / 3184364 ns/op => +2.0 ms/request, ~+65%
Isolating: stubbing out changedIdx alone recovers ~1.2 ms; stubbing the two wireBreakpoints calls recovers ~0.7 ms. So most of the regression is changedIdx — a full json.Marshal of every message, twice, per component (components/pipeline.go:141), i.e. O(components × transcript) marshals added to the hot path purely for a diagnostic counter. It is as expensive as CloneMessages (3.9 ms vs 4.2 ms at 120 messages) and 113× tokensOf.
The repo tracks AddedLatencyMsAvg in /stats precisely because this matters. Cheapest fixes, in order of laziness: swap the double marshal for reflect.DeepEqual(before[i], after[i]) — I measured this at ~1.2 ms cheaper, recovering the whole changedIdx cost; and hoist wireBreakpoints(body) so it is computed once and reused for the breach check instead of twice.
The benchmark verdict — I disagree with one link in the chain, and it matters
The self-criticism is admirably honest and the per-step normalisation is legitimate in principle: with n=1 and a 3-step difference on traffic where cost correlates with steps at 0.95, the headline −15.2% is not attributable to placement, and inverting to +7.9%/step is the right instinct.
But acted=0 / 0 of 78 requests changed content tokens is not the proof it is presented as. acted counts content-token savings, and cacheonly guarantees no content change by construction — so acted=0 is a tautology of the arm, not evidence about placement. It rules out "content compaction confounded the result". It says nothing about the other live confounders: the arm still runs splitVolatileTail, which adds a system block and moves a breakpoint, and agent nondeterminism produced the 3-step difference itself. n=1 with a degenerate control cannot separate placement from either.
And the stated mechanism does not reproduce. The PR says the component "spends its one free slot at len−1, above claude-code's own breakpoint, shortening the readable prefix". I measured where the added mark actually lands on all three captures:
capture-swe: addedMarkAboveAgentOwn=0 addedBelowOrEqual=16 noMarkAdded=3
capture-tb: addedMarkAboveAgentOwn=0 addedBelowOrEqual=70 noMarkAdded=3
capture-swebench: addedMarkAboveAgentOwn=0 addedBelowOrEqual=20 noMarkAdded=3
Zero of 106. The agent's own mark is on the final message; ours consistently lands at highestAdded = highestAgentOwn − 1 (the mark() walk-down from len−1 hits the message below). By the component's own Rule 2 — "writes are billed as a SPAN, so an extra breakpoint BELOW the top costs exactly zero" — the +61.9% cache-write should not have this cause. So the paragraph naming v1's regression mechanism is unsupported, and the one number in the run that would cost money has no established mechanism at all. That is a reason to distrust the number as much as the mechanism.
My read: the cache-write direction should not block the merge, but the merge should not ship it enabled either. The correctness case stands on wire evidence alone, the honest cost signal is unresolved in both directions, and holding a correctness fix hostage to a properly-powered study is the wrong trade — the study needs this code to exist. Concretely: land the fix, and in the same PR drop cacheinject from general/codesmart/codesafe/balanced (config/config.go:127,152,165) pending the placement study. That is exactly the config-change remedy the issue anticipated, and it is one line each. Shipping it in the default presets on an n=1 signal pointing the wrong way, with no reproducible mechanism, is the one outcome I'd push back on.
Docs
docs/components/cacheinject.md:133 ships a literal Placeholder — filled by the cacheonly vs off measurement in #32 under a heading (## What placement is actually worth) that three other passages on the page link to as the authoritative answer (:31, :107). A reader following those links lands on nothing. Either put the n=1 result there with its caveats, or drop the section and the links until there is a measurement. Repo rule is docs must not present plans as shipped; a cross-referenced empty section is that rule's failure mode.
Otherwise the docs corrections are good and appropriately loud — the !!! danger blocks on cacheinject.md and results/components.md correctly retract the placement claims rather than quietly restating them, and docs/design.md:124-155 describes what the code does. Two fixes needed: the bifrost attribution above, and :24 overstates the cap guarantee (not true on Bedrock).
Metrics — backward compatible, correctly locked
Verified /stats is additive only: discarded_changes and top_discarded are new keys, nothing renamed or removed. deploy/harbor/*.py reads runs/acted/saved_tokens/saved_tokens_unique by .get()/subscript on unchanged names — swebench.py:298, measure.py:142, deep_analysis.py:96, analyze.py:90 all keep parsing. New counters are inside a.mu (metrics.go:115-116). The Discarded > 0 ⇒ early return guard correctly keeps Runs from double-counting, and the test pins it. One gap: an early return means a discard-attribution report never reaches the Mutated/Acted bookkeeping, which is right — but it also means Slog.Component (metrics.go:52) emits discarded_changes on every component line as a constant 0, which is dead field width on a hot log. Minor.
Security
Clean. No credential, token, key, or gateway URL anywhere in the diff or the PR body. Test fixtures use claude-x / toolu_abc placeholders.
Summary, ranked
| # | Severity | Issue |
|---|---|---|
| 1 | blocking | wireBreakpoints blind to Bedrock system/tools cachePoint → 6 on the wire, breach log silent too. Reproduced. |
| 2 | blocking | discarded_changes charges reverted components, and charges one discard to every component that touched the message. Both reproduced. |
| 3 | high | cap-breach ERROR fires on client-caused breaches and under bypass. Reproduced. |
| 4 | high | root cause misattributed to bifrost across 5 comments + 2 docs; it is this repo's toolMessage(). Disproved bifrost claim directly. |
| 5 | medium | +2.0 ms / +65% on apply.Body, mostly changedIdx's double marshal for a diagnostic. Measured on both branches. |
| 6 | medium | docs/components/cacheinject.md:133 is a Placeholder that three passages link to as the answer. |
| 7 | medium | benchmark's stated cost mechanism does not reproduce (0/106 marks land above the agent's own); acted=0 is a tautology of the arm, not proof about placement. |
| 8 | low | slog emits a constant-0 discarded_changes on every component line; OuterBreakpoints in the apply.go:262 comment should be ExistingBreakpoints. |
Verdict: changes needed. 1 and 2 are code bugs in the two things this PR exists to fix, both reproducible in under a minute. 3–5 are one-to-few-line fixes. On the economics I would not hold the merge — but I would drop cacheinject from the default presets in this same PR rather than after the study.
What I took on trust: the historical measurement counts (46 applied / 0 forwarded over 40 requests; 1,771 of 1,794 requests at (system=2, tools=0, messages=1); the 44-request live inbound/outbound distribution) and the astropy-12907 billing figures — I did not re-run the benchmark or re-derive those captures. Everything else in this review I reproduced.
Retracting my own framing of the cache-write findingI previously treated the +61.9% cache-write per step as the most consequential result here and said it "argues the honest follow-up is not ship this but measure placement properly." The independent review refuted the mechanism behind that reading, and it was right to. Two problems with the analysis — both cutting against the alarming interpretation:
So the number that supposedly costs money has no established mechanism behind it. n=1, degenerate control, contended box, and a mechanism that measurement contradicts. The cache-write direction should not block this merge. I got this wrong in a specific and repeatable way: I accepted an unfavourable number without checking whether the mechanism it was attributed to had actually operated. That is the same error as treating a favourable number the same way — and it is now the fourth instance in this workstream (see the What still blocks: the Bedrock What I'm adding as a merge requirement: drop |
Review follow-ups on the #32 metadata-write fix. Two of these are bugs the original change introduced. The cap was still breachable on Bedrock. `cachePoint` was only counted under `messages.#.content.#`, but Bedrock places it as its own entry in `system` and `tools` — precisely the two locations defect 2 is about. Constructed, that put 6 breakpoints on the wire with the new breach check also silent, since it reads the same blind counter. Both paths are counted now. `discarded_changes` misattributed twice, which matters because this is the counter whose whole job is catching #32-class bugs; a false-positive generator would not be trusted. `ChangedIdx` was recorded before the revert branches, so a rolled-back component was charged for a discard caused by a later one. And a single discarded message was charged to every component that had touched it. Now `ChangedIdx` is recorded only on the surviving path, and one discard goes to exactly one component — the last to change that message, whose state is what writeback actually threw away. Both have tests that fail without the fix. The breach ERROR blamed us for client-caused breaches. A request arriving already over the cap is forwarded untouched (fail open), so shouting about it named the wrong culprit; it now fires only when our own output exceeds both the cap and the inbound count. Correct the root cause, which was wrong in five comments and three doc pages. bifrost does NOT drop `cache_control` on `tool_result` — tested directly, it round-trips fine. The mark is dropped by this repo's own `toolMessage()` in `normalize`, which rebuilds the block into a synthetic role=tool message from text and tool_use_id alone. The fix works either way, but reasoning from a false premise is how the original bug survived two benchmark studies. Swap the diagnostic's double `json.Marshal` for `reflect.DeepEqual`: 20.52 ms/op -> 16.56 ms (-19.3%) and 3,206 fewer allocs on a realistic 80-message request. The writeback loop's own marshal is what decides whether to splice; this only needs to know which indices moved. Drop `cacheinject` from every preset. Its breakpoints only started reaching the provider with #32 and placement has never been shown to help, so enabling it by default ships an unmeasured policy on every request. The issue anticipated exactly this: "if the fix proves cacheinject harmful once live, the answer is to remove it from the default preset — a config change, not a new knob." That required separating two mechanisms that shared one config entry. The volatile-tail split was gated on `cacheinject` merely because it needed somewhere to hang, but the split is measured (-34.1% cost, 0% -> 96.7% hit in an isolated A/B) while placement is not. Dropping cacheinject alone would have silently disabled the split too, turning "disable an unproven component" into a real cost regression. So `cachesplit` is a marker component carrying the split, and the presets use it. It stays gated rather than unconditional so `off` remains a true passthrough control for A/B runs. Also retract the benchmark mechanism claim. The cache-write direction stands as recorded but has no established mechanism, and the one I proposed is disproven: 0 of 106 marks land above claude-code's own breakpoint across three captures — ours sits one message below, where Rule 2 says an extra breakpoint costs zero. `acted=0` also does not isolate placement, since `splitVolatileTail` is live in that arm. Assisted-By: Claude Opus 5 Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
…rve merges (#49) Five PRs landed in quick succession (#33, #36, #40, #42, #43), each updating its own docs. Nobody checked they were coherent together, and several pages described behaviour that no longer exists. The largest error: every one of the nine preset compositions in docs/reference/presets.md still ended in `cacheinject`, which #36 removed from all of them in favour of the new `cachesplit` marker component. `agent`, `aggressive` and `general` were also missing `extract_llm`, `general` was documented only in prose, and `balanced` was called "the default" when the proxy has defaulted to `codesmart` for some time. Each of the eleven pipelines is now verified component-for-component against the `presets` map. `cachesplit` was a registered component with no page and no nav entry; it has both now. That was the only registered/documented gap — every other components.Register call already had one. Reference pages were the other systematic gap. docs/reference/routes.md documented 5 of the ~45 fields the `Snapshot` struct serves; it now covers all of them, grouped, including `discarded_changes`/`top_discarded` (#36), the SSE quintet (#33), the cmdfilter ledgers (#42) and the observe namespace (#43). config.md gained the `store` block, five missing env vars, and lost a `cacheinject` example. Corrections carrying evidence discipline rather than just names: - cacheinject's placement section was headed "measured, not asserted" over a simulation, and its one live post-fix reading (n=1, +7.9% cost per step, +61.9% cache-write, mechanism unexplained, 0 of 106 marks landing where the suspected mechanism requires) was buried below the favourable numbers. The simulation is now labelled as one, and the negative live reading leads. - The root cause of the discarded `tool_result` breakpoint was attributed to bifrost. It is this repo's own `toolMessage()` in `normalize`. - routes.md now warns that `saved_tokens` is cumulative: the unique totals behind the two studies are 234,119 and 15,457 tokens, 21x and 8x smaller. - cmdfilter.md said 23 filters and first-line selectors; it is 24 filters over six leading lines. The four filters predicted to matter fired zero times and apt+gcc carried ~73% of live savings — recorded as a failed prediction, not quietly dropped. - The `repairLostResult` removal and extract_llm's exclusion from freeze-repair (its replacement is a sampled model output) are now in design.md. - docs described `extract` as the LLM component. `extract_llm` is; `extract` never calls a model. Historical results pages keep their original pipeline names, annotated with what changed since, rather than being rewritten to numbers the runs did not produce. Untouched: docs/results/terminal-bench-*.md and improvement-plan.md (held on #23), and extract_llm's economics (#34's subject). mkdocs build --strict passes; no orphan pages, no dangling nav entries. Assisted-By: Claude Opus 5 Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com> Co-authored-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
Re-measures context-guru on TB after the 15 cache/filter/observe PRs landed on main, as a fifth arm alongside the original four. The original study is left unchanged below it. Config is cgfinal = [format, dedup, cmdfilter, extract, cachesplit], chosen on per-component evidence rather than maximal token reduction: extract_llm is 82x underwater once its saved tokens are priced at the cache-read rate they actually bill at, failed_run acted 0 times while burning 28.8 s, and cacheinject was removed from every preset by #36. Result on 81 clean tasks: 61 solved vs baseline 53, total $79.32 vs $94.85, own LLM cost $0 vs the previous arm's $2.97, added latency 38.5 ms vs 449.8 ms. Two framing decisions the numbers force: The -16.4% aggregate is single-task sensitive -- path-tracing alone accounts for most of it, and an independent re-derivation with a stricter degenerate rule gave -13.7% dropping to -2.8% on the same exclusion. The median per-task ratio, -7.8% with 49/81 cheaper, is the figure to quote for a normal task. Both are published because they differ by 9 points. The one result needing no caveat is cache-write/cache-read returning to 1.86%, identical to baseline, where the previous arm ran 2.86%. That is the cache-write tax this study named as the deciding term on TB, and being a ratio rather than a sum it holds under every exclusion rule tried. Records what could NOT be verified: #40's freeze-TTL work has all five frozen_* counters at zero because its only callers are the three components this config excludes, so the arm is not evidence for or against it and none of the cost improvement may be credited to it. cachesplit likewise has zero legal opportunity on TB, because the Agent SDK never appends the git snapshot the CLI does. Regressions published rather than omitted: system-administration is +17.2% cost AND -2 solved, security +25.6%, fresh_input 3.8x baseline, and small tasks still inflate up to +311% at n=1 -- size-gating remains an unclaimed win. Also states plainly that cgfinal's raw model cost nearly ties the old arm and its cache-read is higher, so it wins mainly by not spending $2.97 on haiku. Limitations: headroom and rtk cannot be re-derived because their trial artifacts are pruned from disk, so those columns are cited rather than recomputed; single trial per task; one task still running at report time. Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
Re-measures context-guru on TB after the 15 cache/filter/observe PRs landed on main, as a fifth arm alongside the original four. The original study is left unchanged below it. Config is cgfinal = [format, dedup, cmdfilter, extract, cachesplit], chosen on per-component evidence rather than maximal token reduction: extract_llm is 82x underwater once its saved tokens are priced at the cache-read rate they actually bill at, failed_run acted 0 times while burning 28.8 s, and cacheinject was removed from every preset by #36. Result on 81 clean tasks: 61 solved vs baseline 53, total $79.32 vs $94.85, own LLM cost $0 vs the previous arm's $2.97, added latency 38.5 ms vs 449.8 ms. Two framing decisions the numbers force: The -16.4% aggregate is single-task sensitive -- path-tracing alone accounts for most of it, and an independent re-derivation with a stricter degenerate rule gave -13.7% dropping to -2.8% on the same exclusion. The median per-task ratio, -7.8% with 49/81 cheaper, is the figure to quote for a normal task. Both are published because they differ by 9 points. The one result needing no caveat is cache-write/cache-read returning to 1.86%, identical to baseline, where the previous arm ran 2.86%. That is the cache-write tax this study named as the deciding term on TB, and being a ratio rather than a sum it holds under every exclusion rule tried. Records what could NOT be verified: #40's freeze-TTL work has all five frozen_* counters at zero because its only callers are the three components this config excludes, so the arm is not evidence for or against it and none of the cost improvement may be credited to it. cachesplit likewise has zero legal opportunity on TB, because the Agent SDK never appends the git snapshot the CLI does. Regressions published rather than omitted: system-administration is +17.2% cost AND -2 solved, security +25.6%, fresh_input 3.8x baseline, and small tasks still inflate up to +311% at n=1 -- size-gating remains an unclaimed win. Also states plainly that cgfinal's raw model cost nearly ties the old arm and its cache-read is higher, so it wins mainly by not spending $2.97 on haiku. Limitations: headroom and rtk cannot be re-derived because their trial artifacts are pruned from disk, so those columns are cited rather than recomputed; single trial per task; one task still running at report time. Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
…rm (#59) * docs(benchmark): add the Terminal-Bench 2.0 four-way study + improvement plan Second benchmark of the study, after SWE-bench Verified: 89 open-ended terminal tasks, claude-code on aws/claude-sonnet-5, run live through the harness. Four arms, same as SWE: baseline (off passthrough), context-guru (codesmart), headroom (hd-cache), rtk. The claude-code trajectory parser, the cache-aware cost model and the summarizer are agent-specific, not benchmark-specific, so every number is computed identically to the SWE arms. Harnesses: terminalbench.py / _headroom.py / _rtk.py (thin adaptations of the SWE ones, dataset + jobs-root differ) and gen_tb_docs.py for the per-arm pages. What the run shows: the agent is ~98% cached here too, so cache-read is again the largest cost term — but cache-write, a rounding error on SWE-bench, becomes the deciding term on TB's ~1.7M-token contexts. Six baseline trials are degenerate (baseline aborted in 2-6 steps where the arms ran 50-160), which inflates the apparent regression; over the 83 clean tasks context-guru is -9.7% and headroom -16.0%, with only rtk regressing. That correction is stated up front on the comparison page and the six tasks are queued for re-run. improvement-plan.md carries the synthesis of both benchmarks: cost tracks agent steps (r=0.95), one cache-write costs 11.5 cache-reads, unique token removal is 0.02-0.13% of the billed total, and cache_control placement is metadata rather than hashed content — so moving a breakpoint is free. Also fixes swebench.py: captures and dumps now live under the run's jobs-root instead of a fixed /tmp path that start_proxy unlinks, which is how an earlier 472-request capture was truncated mid-analysis. Adds the cacheonly arm that isolates the prompt-cache lever from token reduction. Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com> * docs(benchmark): correct the Terminal-Bench cost conclusion and retract the xdedup premise Two corrections to the TB study, both from re-deriving the numbers from the row files rather than trusting the per-arm totals. 1. Six baseline trials are degenerate: the baseline aborted in 2-6 steps while the compaction arms ran 50-160. mteb-leaderboard, polyglot-rust-c and extract-moves-from-video alone account for $11.5 of apparent regression. On the 83 clean tasks context-guru costs $90.34 vs baseline $100.17 (-9.8% including its own haiku cost, -12.7% on model cost alone), solves +2, and takes 8.3% fewer steps. So TB does not invert the SWE result; the +1.7% headline was an artifact. headroom recomputes to about -16%; rtk remains a genuine regression. What IS different on TB survives the correction: cache-write, a rounding error on SWE-bench, is the deciding term on 1.7M-token contexts. 2. The cross-turn dedup premise is refuted. Measured on the raw captures (1,325 requests / 51 sessions), 232 of 232 re-sent large outputs live at exactly one stable message index, and 100% of consecutive turn pairs have the previous turn as a byte-identical prefix. The agent appends; it does not re-send. Those 5.46M tokens sit in the cached prefix and already bill at the cache-read rate, so an xdedup component would have no legal opportunity to act, and rewriting them would convert reads into writes at 11.5x. Independently re-checked: 0 of 77 large outputs ever appeared at a second index. Also documents the baseline's two-stage merge in REPRODUCE, because the intermediate rows file sums to $71.44 and does not reproduce the published $100.81 -- that gap is what surfaced both errors. Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com> * docs(benchmark): retract improvement-plan B2 — the expand tool works, the bug was a latency tautology The plan asserted that context_guru_expand is 'referenced 1,496x and callable 0x' because the tool is never registered on the streaming path, and made registering it the single biggest reward lever. Both halves are wrong. expand.Inject does fire on real streaming requests, and proxy.serve does buffer and aggregate SSE when markers are present -- there is no streaming short-circuit. More decisively, a live SWE run recorded bounces=1 with 3,372 tokens restored: RecordExpand has exactly one reachable call site, inside the continuation loop, only after a model-issued expand call resolves against the store, and all traffic was SSE. So restoration completed through the streaming path. The 4.8M figure was cumulative, re-counting each compaction every turn history is re-sent. Unique is 234,119 tokens behind 103 markers on TB and 15,457 behind 29 on SWE -- 21x and 8x smaller. Demand is genuinely low, not blocked. The real defect was a tautology: hasMarkers tested the raw body for the escaped marker sequence, and the injected tool description itself contains it, so every SSE response was buffered and the documented zero-added-latency fast path never engaged. Fixed by scoping the check to messages + system; marker-free TTFB went 1007ms -> 43ms, and live buffering fell from an implied 100% to 27.3%. Both this and the retracted C1 were premise errors from trusting the change-log dumps -- which only record messages a component already acted on -- over the raw request captures. That lesson is now recorded in the section. Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com> * docs(benchmark): record the mechanism-verification rule that four wrong premises taught An aggregate moving in the predicted direction is not evidence the predicted mechanism operated. Four premises in this plan were wrong, and each failed the same way: a derived artifact was trusted over the raw request stream. - C1 xdedup: a 39.8x re-send factor was read as tokens re-sent as new bytes. 232 of 232 large outputs sit at one stable message index; the agent appends, so those tokens are cached-prefix reads and the component could never have acted. - B2 expand: 'never registered on the streaming path' was false; a live agent restored 3,372 tokens through it. The 4.8M was a cumulative re-count against a 234k unique figure. - prefixpin: early-index churn measured 0 in ~6,500 comparisons on claude-code. An earlier 52% reading was concurrent sessions sharing a byte-identical first message and being diffed against each other. - async cache-write: -45%/-39% was read as the tail-protection working, but the protection only stripped context-guru's own breakpoints and never the agent's, so lower cache-write came from writing fewer breakpoints instead. Three of the four produced a number pointing the right way for the wrong reason, which is why they survived review. Records the five countermeasures, the most useful being: group lineages by append-only prefix match rather than a first-message hash, and instrument 'did the component act' separately from 'did the metric improve'. Also revises F2: cacheinject is not a dead component. It read as inert partly because its breakpoints were discarded by the writeback layer before reaching the wire (46 applied, 0 forwarded). Once forwarded, placement measures mildly harmful, so the open question is whether it belongs in the default preset. Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com> * docs(benchmark): extend the mechanism-verification rule with four more instances, two of them mine Four more premises fell the same way since F-1 was written, taking the count to eight: - cacheinject read as 'provably inert' when it was in fact applying 46 breakpoints and forwarding 0 -- the writeback layer discarded every one. Two benchmark studies concluded things about breakpoint placement while measuring a component whose output never left the process. - the follow-on claim that placement is HARMFUL (+61.9% cache-write/step) does not survive either: 0 of 106 marks land above the agent's own breakpoint, so the proposed mechanism is ruled out, and the arm's acted=0 is a tautology of its design rather than proof the delta was placement. - cachesplit cannot fire on Terminal-Bench at all. TB runs the Agent SDK, which never appends the git/env snapshot the CLI does: all 73 captured requests carry 3 system blocks and zero volatile-tail markers. Zero legal opportunity, the same shape as the refuted xdedup premise. - the same split is a silent no-op on Bedrock Converse, where cachePoint is its own array entry after the block, so the volatile half is inserted before it and the breakpoint still covers the churn -- while reporting Changed: true. Two of these were mine as orchestrator, and one was an UNFAVOURABLE number I accepted without checking its mechanism. That is the more useful half of the lesson: the bias is not optimism, it is incuriosity, and skepticism applied only to good news is not skepticism. Adds four countermeasures: a component reporting that it acted is not evidence it acted usefully; check the favourable metric had the opportunity to be caused by your change; verify the verifier (two 'defects' here were bugs in the checking script); and a sum over heterogeneous tasks can be one task -- an interim TB delta read -40.2% with a single trial carrying half of it, so report the median per-task ratio and a leave-one-out beside any aggregate. Rewrites F2's cacheinject entry as the full three-stage arc, since it is the clearest worked example of the rule in the document. Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com> * docs(benchmark): publish the merged-system Terminal-Bench arm Re-measures context-guru on TB after the 15 cache/filter/observe PRs landed on main, as a fifth arm alongside the original four. The original study is left unchanged below it. Config is cgfinal = [format, dedup, cmdfilter, extract, cachesplit], chosen on per-component evidence rather than maximal token reduction: extract_llm is 82x underwater once its saved tokens are priced at the cache-read rate they actually bill at, failed_run acted 0 times while burning 28.8 s, and cacheinject was removed from every preset by #36. Result on 81 clean tasks: 61 solved vs baseline 53, total $79.32 vs $94.85, own LLM cost $0 vs the previous arm's $2.97, added latency 38.5 ms vs 449.8 ms. Two framing decisions the numbers force: The -16.4% aggregate is single-task sensitive -- path-tracing alone accounts for most of it, and an independent re-derivation with a stricter degenerate rule gave -13.7% dropping to -2.8% on the same exclusion. The median per-task ratio, -7.8% with 49/81 cheaper, is the figure to quote for a normal task. Both are published because they differ by 9 points. The one result needing no caveat is cache-write/cache-read returning to 1.86%, identical to baseline, where the previous arm ran 2.86%. That is the cache-write tax this study named as the deciding term on TB, and being a ratio rather than a sum it holds under every exclusion rule tried. Records what could NOT be verified: #40's freeze-TTL work has all five frozen_* counters at zero because its only callers are the three components this config excludes, so the arm is not evidence for or against it and none of the cost improvement may be credited to it. cachesplit likewise has zero legal opportunity on TB, because the Agent SDK never appends the git snapshot the CLI does. Regressions published rather than omitted: system-administration is +17.2% cost AND -2 solved, security +25.6%, fresh_input 3.8x baseline, and small tasks still inflate up to +311% at n=1 -- size-gating remains an unclaimed win. Also states plainly that cgfinal's raw model cost nearly ties the old arm and its cache-read is higher, so it wins mainly by not spending $2.97 on haiku. Limitations: headroom and rtk cannot be re-derived because their trial artifacts are pruned from disk, so those columns are cited rather than recomputed; single trial per task; one task still running at report time. Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com> --------- Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com> Co-authored-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
Closes #32
cacheinject's breakpoints never reached the provider on Claude Code traffic. Measured over 40 captured requests: 46 breakpoints applied at the component level, 0 in the output body. The repo's flagship cache component was a no-op on its primary workload, and every conclusion drawn about breakpoint placement from the benchmark studies measured a component whose output was discarded before it was sent.Failing-then-passing evidence
Both new tests fail on
mainand pass here. Run against the unmodifiedapply.go+cacheinject.go:The second failure is defect 2 firing on its own: 6 breakpoints, which the provider rejects with a 400. That is why both defects had to land together — fixing the discard alone would have shipped a live 400.
Replaying real captured traffic through the two code paths:
capture-swe.jsonlcapture-tb.jsonloffarm (this PR's run)The 1-of-19 on
mainis the single message bifrost happens to round-trip; the replay test's gate therefore requires a majority, not merely nonzero.The decisive wire metric
Distribution of the total breakpoint count the provider sees, over the 44 requests captured live during this PR's benchmark run:
Exactly one breakpoint added — the single slot that was actually free — and never a cap breach. This also confirms the issue's traffic measurement on fresh traffic: 45 of 45 requests carry exactly
(system=2, tools=0, messages=1).Design choice: body-level metadata write
Took the issue's preferred option over the metadata-slot-kind alternative.
cache_controlis metadata, not content — it changes nothing the model reads, so it needs no bifrost message model to express. When a component's only change to a message is an addedcache_controlkey,applywrites that key at its exact path (messages.<i>.content.<b>.cache_control) on the original raw bytes viasjson. A write that reads no other field cannot drop one, so the losslessness concern does not arise rather than being traded away.The
!s.losslessguard is not relaxed. It prevents real corruption and stays exactly as strict:metadataOnlyWritesproves "only that" by diffing pre against post with the added keys removed and requiring equality. A text edit, a removed key, a changed block count, a role change — all still discarded.applyMetaWritesrefuses if the raw body's block layout disagrees with the normalized view, so a key can never land on the wrong block, and never overwrites a breakpoint the caller set.TestMetadataOnlyWritesRejectsNonMetadataChangespins all seven cases.Rejected the metadata-only slot kind: it adds a concept to the writeback layer for the same effect, and a body-level operation is also what lets defect 2 be fixed at all, since it can see
systemandtools.Defect 2: the budget was blind in two ways, not one
The issue names
system/tools. Replaying real captures surfaced a second blind spot the issue did not have: the agent's message breakpoint sits on atool_resultblock, and bifrost dropscache_controlthere too. So on real traffic all three of claude-code's breakpoints were invisible to the component, not two — it computedbudget = 4 − 0from what it could see.Counting in the component could never be correct, so
applycounts structurally from the raw body (wireBreakpoints, the same gjson-path approach ashasCacheBreakpoint) and passes the total asCtx.ExistingBreakpoints.applyalso counts its own output and logs an error on a breach, so a cap violation appears in telemetry instead of as a provider 400.Observability: discards are loud now
A mutated-then-discarded component was byte-indistinguishable from a working Reformat —
Mutated > 0, correctly excluded fromtop_passthrough, contributing nothing. That is why this survived two full benchmark studies.Pipeline.RecordDiscardsattributes each thrown-away change back to the component that made it (viaReport.ChangedIdx), surfacing as per-componentdiscarded_changesandtop_discardedin/stats. Both fields are additive — no existing/statskey is renamed or removed, sodeploy/harbor/*.pykeeps parsing unchanged. ADiscardedreport is attribution, not a run, and is asserted not to inflateRuns.Verification
Tests added: breakpoint reaches the wire with
tool_use.id/name/inputintact and nothing else altered; wire total within 4 on the real traffic shape; the 60-message synthetic case that produced 6; budget sees invisible breakpoints; four existing breakpoints exhaust the budget; the metadata-write allow/deny matrix; a discarded change increments the counter; and a capture-replay regression (gated onCONTEXT_GURU_CAPTURE, skipped in CI).Docs
docs/components/cacheinject.md— a prominent correction at the top. Every placement figure on that page measured a suppressed component. Theacted=0/ "placement contributes $0" findings are true as recorded (nothing reached the provider, so nothing could contribute) but are not evidence that placement lacks headroom — that question had never been asked. The volatile-tail-split figures are unaffected; that path never went through the discard.docs/design.md— the writeback losslessness constraint, the metadata exception and its guards, host-side breakpoint budgeting, and discard attribution.docs/results/improvement-plan.mdlives on thedocs/terminal-benchbranch, notmain, so its A5 correction is not in this PR's diff. A5 callscacheinject"provably inert" because the 4th breakpoint lands inside the same 20-block window as BP3 — that reasoning describes a component that placed the breakpoint. It never did. A5's conclusion may survive, but its stated reason is wrong and it should be restated on that branch before the sticky-anchor work builds on it.Benchmark — and what it does NOT show
cacheonlyvsoff, SWE-bench Verified,aws/claude-sonnet-5, n=1 per arm.off's second trial died on a Docker-compose error, so onlyastropy-12907completed in both arms:The −15.2% is not a saving — the agent took 3 fewer steps and cost tracks steps at corr 0.95 here. Per step: cost +7.9%, cache-write +61.9%.
No mechanism is established for the cache-write difference, and my original explanation is disproven. I claimed the mark lands at
len−1, above claude-code's own breakpoint. Measured across three captures: 0 of 106 marks land above — ours consistently sits one message below, where the policy's own Rule 2 says an extra breakpoint costs exactly zero.acted=0does not isolate placement. It rules out content compaction only. Thecacheonlyarm still runssplitVolatileTail, so the arm is "placement + split". A single trial per arm also cannot separate either from the step-count nondeterminism that produced the 3-step gap.So: one task, once, contended box, degenerate control, no mechanism. The cache-write direction should not block this PR, and placement's value has still never been measured. That is the reason for the preset change below, not the table.
Presets:
cacheinjectis no longer enabled by defaultIts breakpoints only began reaching the provider with this PR, and placement has never been shown to help — so shipping it on by default would enable an unmeasured policy on every request. The issue anticipated this exactly: "if the fix proves cacheinject harmful once live, the answer is to remove it from the default preset — a config change, not a new knob."
This required separating two mechanisms that shared one config entry.
splitVolatileTailwas gated oncacheinjectonly because it needed somewhere to hang, but the split is measured (−34.1% cost, 0% → 96.7% hit in an isolated A/B) while placement is not. Droppingcacheinjectalone would have silently disabled the split too — turning "disable an unproven component" into a real cost regression. So acachesplitmarker component now carries the split and the presets use it; it stays gated rather than unconditional sooffremains a true passthrough A/B control.TestNoPresetEnablesCacheinjectByDefaultlocks both halves in.Benchmark verdict — placement moves cache-write the WRONG way (n=1, not a claim)
The
offcontrol arm came back degenerate: 1 of 2 trials died on a Docker-composeRuntimeError(infrastructure, not the proxy), so the aggregate table is not comparable. One task,astropy-12907, completed in both arms:The headline −15.2% is not a win — it is the step count. The agent took 3 fewer steps, and cost tracks steps at corr 0.95 on this traffic. Normalising that out inverts the sign:
Verified this is purely placement: 0 of 78 requests changed content tokens (
acted=0across 39 runs), which is exactly what thecacheonlyarm exists to guarantee.Mechanism, and it is the one the issue told us to watch: the component spends its one free slot at
len−1, above claude-code's own breakpoint, shortening the readable prefix and converting reads into writes at 11.5× cost. That is v1's regression mechanism — now reachable for the first time, because before this fix the marks never left the process.This is n=1 with a degenerate control on a contended box. It is not a number, it is a sign — and it is the sign that costs money.
Recommendation
Merge this as a correctness fix, on the wire evidence, which is independent of cost. Then settle placement economics in a separate, properly-powered study before
cacheinjectstays in any default preset. The issue anticipated this outcome: "If the fix proves cacheinject harmful once live, the answer is to remove it from the default preset — a config change, not a new knob."Known gaps
/tmp/i32-runs/terminalbench_i32.py.cg-proxy-i32with the SWE run, so launching TB killed the SWE proxy mid-arm. Fixed with a distinct binary name; the stale logs are quarantined asINVALID-killed-run-*.len−1; if it lands above the agent's own it is the read→write conversion measured above. Placement policy is the follow-up, not this PR.docs/results/improvement-plan.mdis not onmain(it lives on thedocs/terminal-benchbranch, docs(benchmark): Terminal-Bench 2.0 four-way study + improvement plan #23), so its §A5 correction is not in this diff. A5 says cacheinject is "provably inert… while consuming the slot A1 needs" — pre-fix it consumed nothing, so that clause was false; post-fix it becomes true. It should be restated there before any sticky-anchor work builds on it.