docs: reconcile component and preset docs after the cache/filter/observe merges - #49
Conversation
…rve merges Five PRs landed in quick succession (#33, #36, #40, #42, #43), each updating its own docs. Nobody checked they were coherent together, and several pages described behaviour that no longer exists. The largest error: every one of the nine preset compositions in docs/reference/presets.md still ended in `cacheinject`, which #36 removed from all of them in favour of the new `cachesplit` marker component. `agent`, `aggressive` and `general` were also missing `extract_llm`, `general` was documented only in prose, and `balanced` was called "the default" when the proxy has defaulted to `codesmart` for some time. Each of the eleven pipelines is now verified component-for-component against the `presets` map. `cachesplit` was a registered component with no page and no nav entry; it has both now. That was the only registered/documented gap — every other components.Register call already had one. Reference pages were the other systematic gap. docs/reference/routes.md documented 5 of the ~45 fields the `Snapshot` struct serves; it now covers all of them, grouped, including `discarded_changes`/`top_discarded` (#36), the SSE quintet (#33), the cmdfilter ledgers (#42) and the observe namespace (#43). config.md gained the `store` block, five missing env vars, and lost a `cacheinject` example. Corrections carrying evidence discipline rather than just names: - cacheinject's placement section was headed "measured, not asserted" over a simulation, and its one live post-fix reading (n=1, +7.9% cost per step, +61.9% cache-write, mechanism unexplained, 0 of 106 marks landing where the suspected mechanism requires) was buried below the favourable numbers. The simulation is now labelled as one, and the negative live reading leads. - The root cause of the discarded `tool_result` breakpoint was attributed to bifrost. It is this repo's own `toolMessage()` in `normalize`. - routes.md now warns that `saved_tokens` is cumulative: the unique totals behind the two studies are 234,119 and 15,457 tokens, 21x and 8x smaller. - cmdfilter.md said 23 filters and first-line selectors; it is 24 filters over six leading lines. The four filters predicted to matter fired zero times and apt+gcc carried ~73% of live savings — recorded as a failed prediction, not quietly dropped. - The `repairLostResult` removal and extract_llm's exclusion from freeze-repair (its replacement is a sampled model output) are now in design.md. - docs described `extract` as the LLM component. `extract_llm` is; `extract` never calls a model. Historical results pages keep their original pipeline names, annotated with what changed since, rather than being rewritten to numbers the runs did not produce. Untouched: docs/results/terminal-bench-*.md and improvement-plan.md (held on #23), and extract_llm's economics (#34's subject). mkdocs build --strict passes; no orphan pages, no dangling nav entries. Assisted-By: Claude Opus 5 Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
Verified the three central claims mechanicallyPreset pipelines: 11/11 match the code exactly. I extracted the Worth recording a false alarm of my own: my first extractor reported 8 of 11 mismatched, which looked alarming. It was my regex — it scraped backticks out of the description column and off adjacent rows, so Component coverage: complete. Enumerated every
The code/doc disagreement you flagged is real — fixed in #50You were right not to retract the I confirmed the pool is genuinely live on Your instinct on which counter matters was also right and I used your framing in the commit: a drop is an observation silently given up, so a rising On the judgement callsLeaving the historical results pages' numbers intact and annotating what changed since is the right call — a run's recorded pipeline is what it actually ran, and restating it as the current one would fabricate a measurement. That is the same discipline as the F-1 rule. The evidence-discipline corrections are the most valuable part of this PR and easy to undervalue next to the preset fixes. Heading a simulation "measured, not asserted" while burying the one live post-fix reading below the favourable numbers is precisely how the cache-write finding got mis-framed in the first place — including by me. Leading with the negative reading (n=1, +7.9% cost, +61.9% cache-write per step, mechanism unexplained, suspected mechanism ruled out by 0-of-106 marks) is what the page should have said from the start. Same for reframing cmdfilter's zero-fire result as "the four filters predicted as Tier 1" rather than "the IaC and mobile-build filters" — the point is that the prediction failed, which is the transferable lesson, not which filters happened to be idle. Merging. |
Five PRs merged in quick succession (#33, #36, #40, #42, #43), each updating its own docs. Nobody checked they were coherent together. Several pages described behaviour that no longer exists.
Everything here is a doc change.
mkdocs build --strictpasses; no orphan pages, no nav entry pointing at a missing file.What was wrong, per file
docs/reference/presets.mdcacheinject, which #36 removed from every one of them.agent,aggressiveandgeneralwere also missingextract_llm;generalhad no row at all;balancedwas called "the default" (it'scodesmart); the "order matters" tip still explainedcacheinject's position. Every one of the eleven pipelines is now verified component-for-component against thepresetsmap.docs/components/cachesplit.mdcachesplitwas registered, in every caching preset, and had no page and no nav entry. Added, plus the nav entry.docs/components/cacheinject.mdcache_controlontool_result— it is this repo's owntoolMessage()innormalize. The split was still described as gated on this component.docs/reference/routes.mdSnapshotstruct serves. Now complete and grouped: savings, per-component (incl.discarded_changes/top_discarded, #36), CG's own cost, the SSE quintet (#33), freeze-replay (#40), cmdfilter ledgers (#42), mode + the observe namespace (#43).POST /compactand its query params were missing from the route table. Added a warning thatsaved_tokensis cumulative — the unique totals behind the two studies are 234,119 and 15,457 tokens, 21× and 8× smaller than the cumulative figure.docs/reference/config.mdstorewas one parenthesis; it now has a table (sliding TTL, the 1800→10000 change and why, pinning + the max/2 cap, expired-first eviction). Example pipeline usedcacheinject.--presetdefault wasbalanced. Five env vars undocumented:CHEAP_MODEL*,INJECT_EXPAND,CACHE_MODE,MODEL_INFO*,CONTEXT_GURU_CAPTURE. Added a note that async is not available.docs/components.mdcodesmart/codesafe/aggressiveentirely. Nocachesplitsection. cmdfilter: "23 filters", "first non-empty line". DSL: "matched first-by-sorted-name" (it's descendingpriority). Loss-typing said only whether a hint is appended, not which. Claimedextractis the LLM component.docs/components/cmdfilter.mddocs/components/dsl.mdwarningscap described as "warnings and test failures"; the code says warnings.docs/how-to/choose-a-preset.mdbalancedcalled "the default" with no warning that it omitsmask;aggressiveandagentcompositions missingextract_llm;generalhad no section despite being the recommended pipeline; "23 filters"; the LLM warning namedextractcode/rlm and listed the wrong presets.docs/how-to/recover-context.mdbounces=1, 3,372 tokens) — a fast path that never buffers could equally mean restoration is unreachable. The store section listed only rewind + sticky, missing frozen decisions, pinning, and extract_llm's exclusion from repair.docs/design.mdtoon,cachesplit,extract_llm,summarize. Calledextractthe LLM component (twice, incl. the mermaid degrade node). Metrics section listed 3 of the new field families. Config example usedcacheinject.Ctx.Modezero value and async's unavailability unstated.repairLostResult's removal and thecg:res:+cg:sum1:unification unrecorded.docs/how-to/measure-savings.mdtop_passthroughtip citedcacheinject, now in no preset. No mention oftop_discarded— the field that exists precisely because that signature hid a bug for two studies. No cumulative-vs-unique caveat.docs/examples/live-captures.mdcacheinjectcapture described the v1 policy (mark the message before the newest turn) — the +5.5% regression that was removed. Listedcacheinjectamong ageneral-preset run's passthroughs; that preset no longer contains it.docs/results/components.mdcacheinjectrow kept their original names (they are what the run used) but are now annotated with what changed since, incl. that a freshcodesmartrun has acachesplitrow instead.docs/get-started/overview.md,README.mdcachesplit(andtoon/extract_llm/summarizein overview). README preset list and component table had thecacheinjectcompositions and nocachesplitrow.Deliberately not changed
docs/results/terminal-bench-*.md,docs/results/improvement-plan.md— held on the unmergeddocs/terminal-benchbranch (docs(benchmark): Terminal-Bench 2.0 four-way study + improvement plan #23).extract_llm's economics — perf(extract_llm): economic gate, global result cache, and the honest verdict (#28) #34's subject; left for that PR. Noted, not touched.sync/observe.docs/results/observe-mode.md— already correct and scale-caveated by feat(proxy): add observe mode — measure what compaction would save, without enforcing it #43.Verification
mkdocs build --strict— passes clean (mkdocs installed into a throwaway venv fromrequirements-docs.txt).components.Register) enumerated and checked against the nav:cachesplitwas the only gap.config/config.goanddocs/reference/presets.mdand diffed — identical.cmdfilter_filters.goand from the doc table and diffed — identical.SnapshotJSON tag checked againstroutes.md.One thing I could not resolve without a code change
modes.Pool.Stats()(queued/pending/processed/dropped/errors) is not wired into/stats—proxy.Handler.statsnever calls it, andSnapshothas no field for it. Butdocs/how-to/operating-modes.md(added by #43) tells operators to readdroppedanderrors, and makes a point of contrastingdroppedwith headroom's dashboard hiding it. The doc describes a counter no consumer can currently reach.I left the doc alone rather than delete a section describing intended behaviour: the counters exist and are correct, only the serving is missing, and a one-line wiring fix in
proxy.gois the better resolution than a doc retraction. Flagging it for whoever owns #43.