Skip to content

docs: reconcile component and preset docs after the cache/filter/observe merges - #49

Merged
OsherElhadad merged 1 commit into
mainfrom
docs/post-merge-consistency
Aug 10, 2026
Merged

docs: reconcile component and preset docs after the cache/filter/observe merges#49
OsherElhadad merged 1 commit into
mainfrom
docs/post-merge-consistency

Conversation

@OsherElhadad

Copy link
Copy Markdown
Collaborator

Five PRs merged in quick succession (#33, #36, #40, #42, #43), each updating its own docs. Nobody checked they were coherent together. Several pages described behaviour that no longer exists.

Everything here is a doc change. mkdocs build --strict passes; no orphan pages, no nav entry pointing at a missing file.

What was wrong, per file

File What was wrong
docs/reference/presets.md All nine preset pipelines ended in cacheinject, which #36 removed from every one of them. agent, aggressive and general were also missing extract_llm; general had no row at all; balanced was called "the default" (it's codesmart); the "order matters" tip still explained cacheinject's position. Every one of the eleven pipelines is now verified component-for-component against the presets map.
docs/components/cachesplit.md Did not exist. cachesplit was registered, in every caching preset, and had no page and no nav entry. Added, plus the nav entry.
docs/components/cacheinject.md Placement section headed "measured, not asserted" over a simulation. The one live post-fix reading (n=1; +7.9% cost and +61.9% cache-write per step; mechanism unexplained, and the suspected mechanism ruled out by 0 of 106 marks landing above the agent's own breakpoint) sat below the favourable numbers. The simulation is now labelled as one and the negative reading leads. The intro box didn't say the component is in no preset. Root cause was attributed to bifrost dropping cache_control on tool_resultit is this repo's own toolMessage() in normalize. The split was still described as gated on this component.
docs/reference/routes.md Documented 5 of the ~45 fields the Snapshot struct serves. Now complete and grouped: savings, per-component (incl. discarded_changes / top_discarded, #36), CG's own cost, the SSE quintet (#33), freeze-replay (#40), cmdfilter ledgers (#42), mode + the observe namespace (#43). POST /compact and its query params were missing from the route table. Added a warning that saved_tokens is cumulative — the unique totals behind the two studies are 234,119 and 15,457 tokens, 21× and 8× smaller than the cumulative figure.
docs/reference/config.md Said "four top-level fields" for six. store was one parenthesis; it now has a table (sliding TTL, the 1800→10000 change and why, pinning + the max/2 cap, expired-first eviction). Example pipeline used cacheinject. --preset default was balanced. Five env vars undocumented: CHEAP_MODEL*, INJECT_EXPAND, CACHE_MODE, MODEL_INFO*, CONTEXT_GURU_CAPTURE. Added a note that async is not available.
docs/components.md Preset one-liner had the same wrong compositions, and omitted codesmart/codesafe/aggressive entirely. No cachesplit section. cmdfilter: "23 filters", "first non-empty line". DSL: "matched first-by-sorted-name" (it's descending priority). Loss-typing said only whether a hint is appended, not which. Claimed extract is the LLM component.
docs/components/cmdfilter.md "first non-empty line" in the how-it-works line, contradicting its own 6-line section below. Missing the ~73%-of-savings figure for apt+gcc, and the zero-fire result was phrased as "IaC and mobile-build filters" rather than as the four filters that had been predicted as Tier 1 — the F-1 point being that the prediction was wrong.
docs/components/dsl.md warnings cap described as "warnings and test failures"; the code says warnings.
docs/how-to/choose-a-preset.md balanced called "the default" with no warning that it omits mask; aggressive and agent compositions missing extract_llm; general had no section despite being the recommended pipeline; "23 filters"; the LLM warning named extract code/rlm and listed the wrong presets.
docs/how-to/recover-context.md The streaming section didn't say the fast path never engaged before #33, or why scoping (not marker-shape matching) was the fix. Added the #33 TTFB measurement, and the evidence that restoration does fire through the streaming path (bounces=1, 3,372 tokens) — a fast path that never buffers could equally mean restoration is unreachable. The store section listed only rewind + sticky, missing frozen decisions, pinning, and extract_llm's exclusion from repair.
docs/design.md Package map missing toon, cachesplit, extract_llm, summarize. Called extract the LLM component (twice, incl. the mermaid degrade node). Metrics section listed 3 of the new field families. Config example used cacheinject. Ctx.Mode zero value and async's unavailability unstated. repairLostResult's removal and the cg:res:+cg:sum1: unification unrecorded.
docs/how-to/measure-savings.md top_passthrough tip cited cacheinject, now in no preset. No mention of top_discarded — the field that exists precisely because that signature hid a bug for two studies. No cumulative-vs-unique caveat.
docs/examples/live-captures.md The cacheinject capture described the v1 policy (mark the message before the newest turn) — the +5.5% regression that was removed. Listed cacheinject among a general-preset run's passthroughs; that preset no longer contains it.
docs/results/components.md cmdfilter row said "23 shipped … first line" for a run that had 3. Historical pipeline header and the cacheinject row kept their original names (they are what the run used) but are now annotated with what changed since, incl. that a fresh codesmart run has a cachesplit row instead.
docs/get-started/overview.md, README.md Component diagrams missing cachesplit (and toon/extract_llm/summarize in overview). README preset list and component table had the cacheinject compositions and no cachesplit row.

Deliberately not changed

Verification

  • mkdocs build --strict — passes clean (mkdocs installed into a throwaway venv from requirements-docs.txt).
  • Every registered component (components.Register) enumerated and checked against the nav: cachesplit was the only gap.
  • All 11 preset pipelines extracted from both config/config.go and docs/reference/presets.md and diffed — identical.
  • All 24 builtin filter names extracted from cmdfilter_filters.go and from the doc table and diffed — identical.
  • Every Snapshot JSON tag checked against routes.md.

One thing I could not resolve without a code change

modes.Pool.Stats() (queued / pending / processed / dropped / errors) is not wired into /statsproxy.Handler.stats never calls it, and Snapshot has no field for it. But docs/how-to/operating-modes.md (added by #43) tells operators to read dropped and errors, and makes a point of contrasting dropped with headroom's dashboard hiding it. The doc describes a counter no consumer can currently reach.

I left the doc alone rather than delete a section describing intended behaviour: the counters exist and are correct, only the serving is missing, and a one-line wiring fix in proxy.go is the better resolution than a doc retraction. Flagging it for whoever owns #43.

…rve merges

Five PRs landed in quick succession (#33, #36, #40, #42, #43), each updating its
own docs. Nobody checked they were coherent together, and several pages described
behaviour that no longer exists.

The largest error: every one of the nine preset compositions in
docs/reference/presets.md still ended in `cacheinject`, which #36 removed from all
of them in favour of the new `cachesplit` marker component. `agent`, `aggressive`
and `general` were also missing `extract_llm`, `general` was documented only in
prose, and `balanced` was called "the default" when the proxy has defaulted to
`codesmart` for some time. Each of the eleven pipelines is now verified
component-for-component against the `presets` map.

`cachesplit` was a registered component with no page and no nav entry; it has both
now. That was the only registered/documented gap — every other
components.Register call already had one.

Reference pages were the other systematic gap. docs/reference/routes.md documented
5 of the ~45 fields the `Snapshot` struct serves; it now covers all of them,
grouped, including `discarded_changes`/`top_discarded` (#36), the SSE quintet
(#33), the cmdfilter ledgers (#42) and the observe namespace (#43). config.md
gained the `store` block, five missing env vars, and lost a `cacheinject` example.

Corrections carrying evidence discipline rather than just names:

- cacheinject's placement section was headed "measured, not asserted" over a
  simulation, and its one live post-fix reading (n=1, +7.9% cost per step,
  +61.9% cache-write, mechanism unexplained, 0 of 106 marks landing where the
  suspected mechanism requires) was buried below the favourable numbers. The
  simulation is now labelled as one, and the negative live reading leads.
- The root cause of the discarded `tool_result` breakpoint was attributed to
  bifrost. It is this repo's own `toolMessage()` in `normalize`.
- routes.md now warns that `saved_tokens` is cumulative: the unique totals behind
  the two studies are 234,119 and 15,457 tokens, 21x and 8x smaller.
- cmdfilter.md said 23 filters and first-line selectors; it is 24 filters over six
  leading lines. The four filters predicted to matter fired zero times and
  apt+gcc carried ~73% of live savings — recorded as a failed prediction, not
  quietly dropped.
- The `repairLostResult` removal and extract_llm's exclusion from freeze-repair
  (its replacement is a sampled model output) are now in design.md.
- docs described `extract` as the LLM component. `extract_llm` is; `extract` never
  calls a model.

Historical results pages keep their original pipeline names, annotated with what
changed since, rather than being rewritten to numbers the runs did not produce.
Untouched: docs/results/terminal-bench-*.md and improvement-plan.md (held on
#23), and extract_llm's economics (#34's subject).

mkdocs build --strict passes; no orphan pages, no dangling nav entries.

Assisted-By: Claude Opus 5
Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
@OsherElhadad

Copy link
Copy Markdown
Collaborator Author

Verified the three central claims mechanically

Preset pipelines: 11/11 match the code exactly. I extracted the presets map from config/config.go and the pipeline column from every table row in presets.md and diffed them — zero mismatches, including general, which had no row at all before.

Worth recording a false alarm of my own: my first extractor reported 8 of 11 mismatched, which looked alarming. It was my regex — it scraped backticks out of the description column and off adjacent rows, so agent appeared to end in a spurious mask and codesmart in a duplicated extract_llm. Isolating the pipeline cell gave 0. The doc was right and my check was wrong; worth stating rather than quietly re-running, since a verification tool that over-matches manufactures exactly the kind of "defect" that wastes a review cycle.

Component coverage: complete. Enumerated every components.Register call across the tree (14 registered, excluding _test.go) — every one has both a docs/components/*.md page and an mkdocs.yml nav entry. cachesplit was the only gap and it is now closed. That component being in every caching preset with no page was the worst of the findings: discoverable only inside the page named for the component that now ships disabled.

/stats coverage: 41 of 41 Snapshot JSON fields documented in routes.md, up from 5. That gap is what surfaced the unwired-pool bug below.

The code/doc disagreement you flagged is real — fixed in #50

You were right not to retract the operating-modes.md section. modes.Pool tracked all five counters correctly; metrics.Snapshot had no field and proxy.Handler.stats never called Stats(). So the doc instructed operators to watch dropped while describing something no consumer could reach.

I confirmed the pool is genuinely live on main (observe mode constructs it at proxy.go:146), so this was not a dormant async leftover. Wired it as observe_queue in #50, with a test that asserts the counters arrive over HTTP and fails without the wiring.

Your instinct on which counter matters was also right and I used your framing in the commit: a drop is an observation silently given up, so a rising dropped means the potential_* figures understate what compaction would have saved.

On the judgement calls

Leaving the historical results pages' numbers intact and annotating what changed since is the right call — a run's recorded pipeline is what it actually ran, and restating it as the current one would fabricate a measurement. That is the same discipline as the F-1 rule.

The evidence-discipline corrections are the most valuable part of this PR and easy to undervalue next to the preset fixes. Heading a simulation "measured, not asserted" while burying the one live post-fix reading below the favourable numbers is precisely how the cache-write finding got mis-framed in the first place — including by me. Leading with the negative reading (n=1, +7.9% cost, +61.9% cache-write per step, mechanism unexplained, suspected mechanism ruled out by 0-of-106 marks) is what the page should have said from the start.

Same for reframing cmdfilter's zero-fire result as "the four filters predicted as Tier 1" rather than "the IaC and mobile-build filters" — the point is that the prediction failed, which is the transferable lesson, not which filters happened to be idle.

Merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants