From ba12e6aad24126abfd265b35e4f63a7e1962ff19 Mon Sep 17 00:00:00 2001 From: Osher-Elhadad Date: Mon, 10 Aug 2026 18:34:19 +0000 Subject: [PATCH] docs(benchmark): final 89-task figures for the merged-system arm The run completed after the section was first published. The complete result is materially stronger and one framing claim needed correcting. cgfinal is the FIRST arm to beat baseline on the raw 89-task total -- $94.95 against $100.81 -- and solves the most, 65 of 89. No arm in the original four-way study managed either. That matters because the raw total is the harder bar: six degenerate baseline trials flatter the baseline column, which is why the correction box exists. Both reference arms reproduce their published totals exactly (baseline $100.81, cg-old $102.55), which is the check that the harness and cost model are unchanged across the two studies. Clean-set figures updated to the final 82-task set: 62 solved vs 54, steps 2,865 vs 3,117, $83.52 vs $98.70. The aggregate is -15.4% rather than the interim -16.4%, and path-tracing alone still carries about half of it (-8.3% without it), so the median per-task figure of -7.7% remains the one to quote. Adds two limitations that only the completed run revealed: caffe-cifar-10 is unscored (reward=None at 2,361 s of a 4,800 s budget), so it is in the clean set for cost and out for the solve count -- baseline also failed it; and five arm exceptions are genuine 4x wall-clock timeouts, counted as reward-0 as in the original study. mkdocs build --strict passes. Signed-off-by: Osher-Elhadad --- docs/results/terminal-bench-comparison.md | 59 ++++++++++++++++------- 1 file changed, 41 insertions(+), 18 deletions(-) diff --git a/docs/results/terminal-bench-comparison.md b/docs/results/terminal-bench-comparison.md index 1fd3fa8..c76de11 100644 --- a/docs/results/terminal-bench-comparison.md +++ b/docs/results/terminal-bench-comparison.md @@ -6,9 +6,10 @@ four-way](comparison.md) is the first. !!! tip "A fifth arm was added on 2026-08-10" The [merged-system section](#the-merged-system-a-fifth-arm-2026-08-10) below re-measures - context-guru after 15 PRs of cache/filter/observe work. Headline: **61 solved vs baseline 53**, - **cache-write back to baseline parity** (1.86% vs the previous arm's 2.86%), **$0** own-LLM - cost, and **−7.8% median per task** — read the median rather than the −16.4% aggregate, which + context-guru after 15 PRs of cache/filter/observe work. Headline: **65 of 89 solved — the most + of any arm** — and **$94.95, the first arm to beat baseline on the raw total**. Cache-write is + back to baseline parity (1.86% vs the previous arm's 2.86%), own-LLM cost is **$0**, and the + typical task is **−7.7%** — read the median rather than the −15.4% clean-set aggregate, which one task dominates. The original four-arm study is unchanged below it. The four original arms: @@ -86,28 +87,47 @@ per-component evidence, not on maximal token reduction. Three components exclude that figure is a single-task replay, it drops whole messages, and it is the one offloader that reaches inside the cached prefix. -### Results (81 clean tasks, paired per task) +### Results — all 89 tasks + +The run completed after the first publication of this section; these are the final figures. +Both reference arms reproduce their published totals **exactly** (baseline $100.81, +context-guru-old $102.55), which is the check that the harness and cost model are unchanged. + +| | baseline | cg (old) | headroom\* | rtk\* | **cg (merged)** | +|---|--:|--:|--:|--:|--:| +| solved / 89 | 56 | 58 | 64 | 55 | **65 (73.0%)** | +| **total billed** | $100.81 | $102.55 | $114.75 | $118.83 | **$94.95 (−5.8%)** | +| cache-write | 4.01M | 6.53M | 12.37M | 7.05M | **3.90M** | +| cache-hit | 98.15% | 96.86% | 94.1% | 97.3% | **98.16%** | +| own LLM cost | $0 | $3.26 | $0 | $0 | **$0** | + +\* headroom and rtk are **cited from the original study, not re-derived** — their trial +artifacts have been pruned from disk. + +**This is the first arm to beat baseline on the raw 89-task total.** No arm in the original +study managed that, and it also solves the most. The earlier correction box below explains why +the raw total is the harder bar: six degenerate baseline trials flatter the baseline column. + +### Results (82 clean tasks, paired per task) | | baseline | context-guru (old) | **context-guru (merged)** | |---|--:|--:|--:| -| solved | 53 | 55 | **61** | -| steps | 3,067 | 2,811 | 2,815 | -| cache-write | 3.90M | **5.04M** | **3.36M** | +| solved | 54 | — | **62** | +| steps | 3,117 | — | **2,865** | +| cache-write | 3.97M | — | **3.52M** | | **cache-write / cache-read** | **1.86%** | **2.86%** | **1.86%** | -| cache-hit | 98.16% | 97.18% | 98.12% | -| own LLM cost | $0 | $2.97 | **$0** | +| own LLM cost | $0 | $3.00 | **$0** | | context-guru added latency | — | 449.8 ms | **38.5 ms** | -| **total billed** | **$94.85** | $85.72 (−9.6%) | **$79.32 (−16.4%)** | +| **total billed** | **$98.70** | $86.34 | **$83.52 (−15.4%)** | !!! warning "Read the median, not the aggregate" - **−16.4% is single-task sensitive.** `path-tracing` alone contributes most of it: dropping - that one task gives **−9.1%**, and an independent re-derivation with a stricter degenerate - rule (76 clean tasks) gave **−13.7% → −2.8%** on the same exclusion. + **−15.4% is single-task sensitive.** `path-tracing` alone contributes about half of it: + dropping that one task gives **−8.3%**, and an independent re-derivation with a stricter + degenerate rule (76 clean tasks) gave **−13.7% → −2.8%** on the same exclusion. - The **median per-task ratio is 0.922, i.e. −7.8%**, with **49/81 tasks cheaper**. That is the - figure to quote for "what this does to a normal task"; the aggregate answers the different - question "what would the whole benchmark have cost". Both are reported because they differ - by 9 points. + The **median per-task ratio is −7.7%**, with **49/82 tasks cheaper**. That is the figure to + quote for "what this does to a normal task"; the aggregate answers the different question + "what would the whole benchmark have cost". Both are reported because they differ by half. Reward is **+11 / −3 = net +8** at a single trial per task, so the churn matters more than the net. All three losses were read from their verifier output and are **capability @@ -158,7 +178,10 @@ the predicted mechanism operated. Each claim below names the counter that proves - **Single trial per task**, as in the original study. - Every token figure here is **unique**, never cumulative — the measured overcount ratios were **44.5× (cmdfilter)** and **18.4× (extract)**. -- One task (`circuit-fibsqrt`) was still running at report time; 88 of 89 are scored. +- **`caffe-cifar-10` is unscored** (`reward=None` after 2,361 s of its 4,800 s budget): it is in + the clean set for cost and out of it for the solve count. Baseline also failed it. +- Five arm exceptions are genuine 4× wall-clock timeouts, counted as reward-0 as in the original + study. ---