docs(benchmark): Terminal-Bench 2.0 four-way study + improvement plan - #23
docs(benchmark): Terminal-Bench 2.0 four-way study + improvement plan#23OsherElhadad wants to merge 6 commits into
Conversation
Two corrections pushed — both found by trying to reproduce the published totalsWhile preparing the follow-up re-run I could not reconstruct the baseline's $100.81 from 1. The cost conclusion was wrong, and the direction flipsThe baseline is a two-stage merge: With the baseline correctly reconstructed, six trials are degenerate — the baseline aborted in 2-6 steps while the compaction arms did the real work:
Those three are $11.5 of apparent regression on their own. On the 83 clean tasks:
So Terminal-Bench does not invert the SWE-bench result. context-guru is cheaper, solves more, and takes fewer steps — the same shape as its −13% SWE win. The +1.7% headline was an artifact of six trials where the baseline didn't run. To be clear about what is not an error: the $102.55 figure is correct as printed — it is model cost ($99.29) plus context-guru's own LLM cost ($3.26). The mistake was comparing it against a baseline total contaminated by degenerate trials. The raw 89-task tables are kept as measured; the correction box, Headline, Verdict, and Bottom line now state the clean-set conclusion and why the two differ. The mechanism finding survives intact and is the durable result: cache-write, a rounding error on SWE-bench, is the deciding term on TB's ~1.7M-token contexts. 2. The
|
…rve merges (#49) Five PRs landed in quick succession (#33, #36, #40, #42, #43), each updating its own docs. Nobody checked they were coherent together, and several pages described behaviour that no longer exists. The largest error: every one of the nine preset compositions in docs/reference/presets.md still ended in `cacheinject`, which #36 removed from all of them in favour of the new `cachesplit` marker component. `agent`, `aggressive` and `general` were also missing `extract_llm`, `general` was documented only in prose, and `balanced` was called "the default" when the proxy has defaulted to `codesmart` for some time. Each of the eleven pipelines is now verified component-for-component against the `presets` map. `cachesplit` was a registered component with no page and no nav entry; it has both now. That was the only registered/documented gap — every other components.Register call already had one. Reference pages were the other systematic gap. docs/reference/routes.md documented 5 of the ~45 fields the `Snapshot` struct serves; it now covers all of them, grouped, including `discarded_changes`/`top_discarded` (#36), the SSE quintet (#33), the cmdfilter ledgers (#42) and the observe namespace (#43). config.md gained the `store` block, five missing env vars, and lost a `cacheinject` example. Corrections carrying evidence discipline rather than just names: - cacheinject's placement section was headed "measured, not asserted" over a simulation, and its one live post-fix reading (n=1, +7.9% cost per step, +61.9% cache-write, mechanism unexplained, 0 of 106 marks landing where the suspected mechanism requires) was buried below the favourable numbers. The simulation is now labelled as one, and the negative live reading leads. - The root cause of the discarded `tool_result` breakpoint was attributed to bifrost. It is this repo's own `toolMessage()` in `normalize`. - routes.md now warns that `saved_tokens` is cumulative: the unique totals behind the two studies are 234,119 and 15,457 tokens, 21x and 8x smaller. - cmdfilter.md said 23 filters and first-line selectors; it is 24 filters over six leading lines. The four filters predicted to matter fired zero times and apt+gcc carried ~73% of live savings — recorded as a failed prediction, not quietly dropped. - The `repairLostResult` removal and extract_llm's exclusion from freeze-repair (its replacement is a sampled model output) are now in design.md. - docs described `extract` as the LLM component. `extract_llm` is; `extract` never calls a model. Historical results pages keep their original pipeline names, annotated with what changed since, rather than being rewritten to numbers the runs did not produce. Untouched: docs/results/terminal-bench-*.md and improvement-plan.md (held on #23), and extract_llm's economics (#34's subject). mkdocs build --strict passes; no orphan pages, no dangling nav entries. Assisted-By: Claude Opus 5 Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com> Co-authored-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
Orchestrator cross-check of the final Terminal-Bench armI re-derived the numbers from the raw trial directories ( It survives, but the single-task sensitivity is worse than reported
Same direction, same order of magnitude, under two independent exclusion rules — that is the robustness that matters. But the leave-one-out is more severe than the agent's figure suggested. On my clean set:
That is not a criticism of the run — it is the §F-1.9 rule (a sum over heterogeneous tasks can be one task) doing its job, and the agent added the leave-one-out and median specifically because it caught this pattern early. It just needs to be the headline framing rather than a footnote when this lands in the docs. What I consider solidly establishedCache-write is the clean mechanistic result. cache-write/cache-read is 1.77–1.86%, indistinguishable from baseline's 1.86%, where the previous arm ran at 2.86% (+54%). That is the plan's "cache-write tax" eliminated, and cache-write is the term the published comparison names as deciding on Terminal-Bench. It does not depend on any exclusion rule. Reward improved on both rules (+7 to +8 solved), though at n=1 per task the flip churn (+11/−3) matters more than the net, and all three losses were verified as capability failures rather than information loss — HTTP 404, a wrong-cased flag, and a rejected non-fast-forward push. Notably two of the three used fewer steps than baseline, which argues against the "compaction hid something, agent redid work" mechanism. Own-LLM cost is $0 against the previous arm's $2.97, with Mechanisms verified fired, and one that could not be
Regressions that belong in the published table
Also carrying forward as a stated limitation: headroom's and rtk's TB artifacts have been pruned from disk, so those two columns must be cited from the published comparison and labelled as such, not re-derived alongside the others. |
…ent plan Second benchmark of the study, after SWE-bench Verified: 89 open-ended terminal tasks, claude-code on aws/claude-sonnet-5, run live through the harness. Four arms, same as SWE: baseline (off passthrough), context-guru (codesmart), headroom (hd-cache), rtk. The claude-code trajectory parser, the cache-aware cost model and the summarizer are agent-specific, not benchmark-specific, so every number is computed identically to the SWE arms. Harnesses: terminalbench.py / _headroom.py / _rtk.py (thin adaptations of the SWE ones, dataset + jobs-root differ) and gen_tb_docs.py for the per-arm pages. What the run shows: the agent is ~98% cached here too, so cache-read is again the largest cost term — but cache-write, a rounding error on SWE-bench, becomes the deciding term on TB's ~1.7M-token contexts. Six baseline trials are degenerate (baseline aborted in 2-6 steps where the arms ran 50-160), which inflates the apparent regression; over the 83 clean tasks context-guru is -9.7% and headroom -16.0%, with only rtk regressing. That correction is stated up front on the comparison page and the six tasks are queued for re-run. improvement-plan.md carries the synthesis of both benchmarks: cost tracks agent steps (r=0.95), one cache-write costs 11.5 cache-reads, unique token removal is 0.02-0.13% of the billed total, and cache_control placement is metadata rather than hashed content — so moving a breakpoint is free. Also fixes swebench.py: captures and dumps now live under the run's jobs-root instead of a fixed /tmp path that start_proxy unlinks, which is how an earlier 472-request capture was truncated mid-analysis. Adds the cacheonly arm that isolates the prompt-cache lever from token reduction. Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
…ct the xdedup premise Two corrections to the TB study, both from re-deriving the numbers from the row files rather than trusting the per-arm totals. 1. Six baseline trials are degenerate: the baseline aborted in 2-6 steps while the compaction arms ran 50-160. mteb-leaderboard, polyglot-rust-c and extract-moves-from-video alone account for $11.5 of apparent regression. On the 83 clean tasks context-guru costs $90.34 vs baseline $100.17 (-9.8% including its own haiku cost, -12.7% on model cost alone), solves +2, and takes 8.3% fewer steps. So TB does not invert the SWE result; the +1.7% headline was an artifact. headroom recomputes to about -16%; rtk remains a genuine regression. What IS different on TB survives the correction: cache-write, a rounding error on SWE-bench, is the deciding term on 1.7M-token contexts. 2. The cross-turn dedup premise is refuted. Measured on the raw captures (1,325 requests / 51 sessions), 232 of 232 re-sent large outputs live at exactly one stable message index, and 100% of consecutive turn pairs have the previous turn as a byte-identical prefix. The agent appends; it does not re-send. Those 5.46M tokens sit in the cached prefix and already bill at the cache-read rate, so an xdedup component would have no legal opportunity to act, and rewriting them would convert reads into writes at 11.5x. Independently re-checked: 0 of 77 large outputs ever appeared at a second index. Also documents the baseline's two-stage merge in REPRODUCE, because the intermediate rows file sums to $71.44 and does not reproduce the published $100.81 -- that gap is what surfaced both errors. Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
… the bug was a latency tautology The plan asserted that context_guru_expand is 'referenced 1,496x and callable 0x' because the tool is never registered on the streaming path, and made registering it the single biggest reward lever. Both halves are wrong. expand.Inject does fire on real streaming requests, and proxy.serve does buffer and aggregate SSE when markers are present -- there is no streaming short-circuit. More decisively, a live SWE run recorded bounces=1 with 3,372 tokens restored: RecordExpand has exactly one reachable call site, inside the continuation loop, only after a model-issued expand call resolves against the store, and all traffic was SSE. So restoration completed through the streaming path. The 4.8M figure was cumulative, re-counting each compaction every turn history is re-sent. Unique is 234,119 tokens behind 103 markers on TB and 15,457 behind 29 on SWE -- 21x and 8x smaller. Demand is genuinely low, not blocked. The real defect was a tautology: hasMarkers tested the raw body for the escaped marker sequence, and the injected tool description itself contains it, so every SSE response was buffered and the documented zero-added-latency fast path never engaged. Fixed by scoping the check to messages + system; marker-free TTFB went 1007ms -> 43ms, and live buffering fell from an implied 100% to 27.3%. Both this and the retracted C1 were premise errors from trusting the change-log dumps -- which only record messages a component already acted on -- over the raw request captures. That lesson is now recorded in the section. Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
…ng premises taught An aggregate moving in the predicted direction is not evidence the predicted mechanism operated. Four premises in this plan were wrong, and each failed the same way: a derived artifact was trusted over the raw request stream. - C1 xdedup: a 39.8x re-send factor was read as tokens re-sent as new bytes. 232 of 232 large outputs sit at one stable message index; the agent appends, so those tokens are cached-prefix reads and the component could never have acted. - B2 expand: 'never registered on the streaming path' was false; a live agent restored 3,372 tokens through it. The 4.8M was a cumulative re-count against a 234k unique figure. - prefixpin: early-index churn measured 0 in ~6,500 comparisons on claude-code. An earlier 52% reading was concurrent sessions sharing a byte-identical first message and being diffed against each other. - async cache-write: -45%/-39% was read as the tail-protection working, but the protection only stripped context-guru's own breakpoints and never the agent's, so lower cache-write came from writing fewer breakpoints instead. Three of the four produced a number pointing the right way for the wrong reason, which is why they survived review. Records the five countermeasures, the most useful being: group lineages by append-only prefix match rather than a first-message hash, and instrument 'did the component act' separately from 'did the metric improve'. Also revises F2: cacheinject is not a dead component. It read as inert partly because its breakpoints were discarded by the writeback layer before reaching the wire (46 applied, 0 forwarded). Once forwarded, placement measures mildly harmful, so the open question is whether it belongs in the default preset. Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
…e instances, two of them mine Four more premises fell the same way since F-1 was written, taking the count to eight: - cacheinject read as 'provably inert' when it was in fact applying 46 breakpoints and forwarding 0 -- the writeback layer discarded every one. Two benchmark studies concluded things about breakpoint placement while measuring a component whose output never left the process. - the follow-on claim that placement is HARMFUL (+61.9% cache-write/step) does not survive either: 0 of 106 marks land above the agent's own breakpoint, so the proposed mechanism is ruled out, and the arm's acted=0 is a tautology of its design rather than proof the delta was placement. - cachesplit cannot fire on Terminal-Bench at all. TB runs the Agent SDK, which never appends the git/env snapshot the CLI does: all 73 captured requests carry 3 system blocks and zero volatile-tail markers. Zero legal opportunity, the same shape as the refuted xdedup premise. - the same split is a silent no-op on Bedrock Converse, where cachePoint is its own array entry after the block, so the volatile half is inserted before it and the breakpoint still covers the churn -- while reporting Changed: true. Two of these were mine as orchestrator, and one was an UNFAVOURABLE number I accepted without checking its mechanism. That is the more useful half of the lesson: the bias is not optimism, it is incuriosity, and skepticism applied only to good news is not skepticism. Adds four countermeasures: a component reporting that it acted is not evidence it acted usefully; check the favourable metric had the opportunity to be caused by your change; verify the verifier (two 'defects' here were bugs in the checking script); and a sum over heterogeneous tasks can be one task -- an interim TB delta read -40.2% with a single trial carrying half of it, so report the median per-task ratio and a leave-one-out beside any aggregate. Rewrites F2's cacheinject entry as the full three-stage arc, since it is the clearest worked example of the rule in the document. Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
Re-measures context-guru on TB after the 15 cache/filter/observe PRs landed on main, as a fifth arm alongside the original four. The original study is left unchanged below it. Config is cgfinal = [format, dedup, cmdfilter, extract, cachesplit], chosen on per-component evidence rather than maximal token reduction: extract_llm is 82x underwater once its saved tokens are priced at the cache-read rate they actually bill at, failed_run acted 0 times while burning 28.8 s, and cacheinject was removed from every preset by #36. Result on 81 clean tasks: 61 solved vs baseline 53, total $79.32 vs $94.85, own LLM cost $0 vs the previous arm's $2.97, added latency 38.5 ms vs 449.8 ms. Two framing decisions the numbers force: The -16.4% aggregate is single-task sensitive -- path-tracing alone accounts for most of it, and an independent re-derivation with a stricter degenerate rule gave -13.7% dropping to -2.8% on the same exclusion. The median per-task ratio, -7.8% with 49/81 cheaper, is the figure to quote for a normal task. Both are published because they differ by 9 points. The one result needing no caveat is cache-write/cache-read returning to 1.86%, identical to baseline, where the previous arm ran 2.86%. That is the cache-write tax this study named as the deciding term on TB, and being a ratio rather than a sum it holds under every exclusion rule tried. Records what could NOT be verified: #40's freeze-TTL work has all five frozen_* counters at zero because its only callers are the three components this config excludes, so the arm is not evidence for or against it and none of the cost improvement may be credited to it. cachesplit likewise has zero legal opportunity on TB, because the Agent SDK never appends the git snapshot the CLI does. Regressions published rather than omitted: system-administration is +17.2% cost AND -2 solved, security +25.6%, fresh_input 3.8x baseline, and small tasks still inflate up to +311% at n=1 -- size-gating remains an unclaimed win. Also states plainly that cgfinal's raw model cost nearly ties the old arm and its cache-read is higher, so it wins mainly by not spending $2.97 on haiku. Limitations: headroom and rtk cannot be re-derived because their trial artifacts are pruned from disk, so those columns are cited rather than recomputed; single trial per task; one task still running at report time. Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
ef97bf4 to
4d00957
Compare
|
Superseded by #59 — this branch rebased onto Per your instruction this was held as a draft until the improved results existed. They now do: 61 solved vs baseline 53, cache-write back to baseline parity (1.86% against the previous arm's 2.86%), $0 own-LLM cost, and added latency down from 449.8 ms to 38.5 ms. One framing decision worth flagging before you read it: the −16.4% aggregate is dominated by a single task. Removing |
github-project-automation
Bot
moved this from New/ToDo
to Done
in Rossoctl Issue Prioritization
2 participants
What
The second benchmark of the study, after SWE-bench Verified: Terminal-Bench 2.0, 89 open-ended terminal tasks,
claude-codeonaws/claude-sonnet-5, run live through the harness. Same four arms as SWE — baseline (offpassthrough), context-guru (codesmart), headroom (hd-cache), rtk.The claude-code trajectory parser, the cache-aware cost model and the summarizer are agent-specific rather than benchmark-specific, so every number is computed identically to the SWE arms and the two benchmarks are directly comparable in methodology. Only the Harbor dataset and jobs-root differ.
Headline (89-task raw)
The correction, stated up front
Six baseline trials are degenerate — the baseline aborted in 2–6 steps (16–800 s) where the compaction arms ran 50–160 steps, so the cost delta on those six measures the baseline not doing the work.
extract-moves-from-videoalone is $24.10 of headroom's $24.65 "regression".Recomputed over the 83 clean tasks: context-guru −9.7%, headroom −16.0%, rtk +6.4%. Both proxies save on Terminal-Bench; only rtk regresses. The comparison page opens with this caveat and the six tasks are queued for re-run at low concurrency; the tables will be regenerated then.
What the study establishes
r = 0.95across every arm and both benchmarks.cache_controlis metadata, not hashed content — moving a breakpoint is free and lossless.docs/results/improvement-plan.mdcarries that synthesis and the prioritized work it implies.Also in this PR
swebench.py: captures and dumps move from a fixed/tmppath to the run's own jobs-root.start_proxyunlinks the capture path, so launching any new run truncated the capture an earlier analysis was computed from — which is how a 472-request capture was lost mid-analysis. Adds thecacheonlyarm, which removes no content tokens at all, so any cost delta vsoffis purely breakpoint placement.Reproduction
docs/results/REPRODUCE.md§7 (baseline: task-list generation,n=24concurrency feasibility on this box, the timeout/budget methodology, task-metadata extraction) and §7b (the three framework-arm commands). Every credential is read from the environment at run time; nothing is embedded.Verification
CGO_ENABLED=1 go build -tags cg_skeleton ./...andgo test -tags cg_skeleton ./...green (docs/harness only — no library change).