diff --git a/README.md b/README.md
index 6f2d4e3..d1fd9a8 100644
--- a/README.md
+++ b/README.md
@@ -79,7 +79,7 @@ Components implement one of two lossiness-typed interfaces and are stacked in co
```mermaid
flowchart TD
C["Component — Name() · Enabled(ctx)"]
- C --> R["Reformat: lossless repack
format · toon · cacheinject"]
+ C --> R["Reformat: lossless repack
format · toon · cachesplit · cacheinject"]
C --> O["Offload: drop + stash, returns cache_keys
skeleton · dedup · collapse · failed_run
cmdfilter · extract · extract_llm · smartcrush · mask · summarize"]
```
@@ -126,9 +126,10 @@ curl -s localhost:4000/anthropic/v1/messages \
```
Presets: **`codesmart`** (the default — the SWE-bench-winning cache-aware config
-`[format, dedup, failed_run, cmdfilter, extract_llm, extract, cacheinject]`), **`codesafe`** (the same
-minus the LLM pass — deterministic-only `[format, dedup, failed_run, cmdfilter, extract, collapse, cacheinject]`,
-zero model calls by policy), plus `general`, `agent`, `coding`, `mcp`, `balanced`, `safe`, `summarize`, `off`.
+`[format, dedup, failed_run, cmdfilter, extract_llm, extract, cachesplit]`), **`codesafe`** (the same
+minus the LLM pass — deterministic-only `[format, dedup, failed_run, cmdfilter, extract, collapse, cachesplit]`,
+zero model calls by policy), plus `general`, `agent`, `aggressive`, `coding`, `mcp`, `balanced`, `safe`,
+`summarize`, `off`.
`codesmart`'s LLM relevance-trimmer (`extract_llm`) engages only when a cheap model is configured
(`CHEAP_MODEL*`); without one it safely no-ops and behaves like `codesafe`.
See [docs/components.md](docs/components.md) and [docs/reference/presets.md](docs/reference/presets.md).
@@ -158,7 +159,8 @@ are in **[docs/components.md](docs/components.md)** and **[docs/results/componen
|---|---|---|
| `format` | Reformat | re-encodes pretty JSON tool output as compact JSON |
| `toon` | Reformat | re-encodes a uniform JSON array as TOON (header once, one row per item) |
-| `cacheinject` | Reformat | adds an Anthropic `cache_control` breakpoint on a stable prefix boundary |
+| `cachesplit` | Reformat | splits the volatile tail off the `system` prompt so the shared prefix stays cacheable (in the default presets) |
+| `cacheinject` | Reformat | places Anthropic `cache_control` breakpoints — **opt-in, in no preset**; placement is unmeasured |
| `dedup` | Offload | replaces a byte-identical earlier tool output with a pointer |
| `failed_run` | Offload | collapses superseded test/build runs, keeps the latest in full |
| `cmdfilter` | Offload | shrinks structured command output via declarative DSL filters |
diff --git a/docs/components.md b/docs/components.md
index 609b8b6..474743b 100644
--- a/docs/components.md
+++ b/docs/components.md
@@ -11,8 +11,8 @@ messages (`role:"tool"`; for Anthropic, `tool_result` blocks normalized to that
|---|---|---|---|---|---|
| `format` | Reformat | nothing (compacts JSON) | n/a (lossless) | pretty-printed JSON tool output | `min_tokens` (50) |
| `toon` | Reformat | nothing (re-encodes JSON arrays as TOON) | n/a (lossless) | uniform flat JSON object-arrays | `min_tokens` (50) |
-| `cacheinject` | Reformat | nothing (adds `cache_control`) | n/a (lossless) | Anthropic-family requests; **opt-in, in no preset** — placement is unmeasured (#32) | `ttl` (5m) |
-| `cachesplit` | Reformat | nothing (splits a `system` block) | n/a (lossless) | Anthropic-family requests; **in the default presets** — enables the measured volatile-tail split | — |
+| `cacheinject` | Reformat | nothing (adds `cache_control`) | n/a (lossless) | Anthropic-family requests; **opt-in, in no preset** — placement is unmeasured ([#36](https://github.com/rossoctl/context-guru/pull/36)) | `ttl` (5m) |
+| `cachesplit` | Reformat | nothing (splits a `system` block) | n/a (lossless) | Anthropic-family requests; **in every caching preset** — enables the measured volatile-tail split | — (no config) |
| `skeleton` | Offload | function/method bodies | via expand | fenced ` ```lang ` code blocks | `min_tokens` (80) |
| `dedup` | Offload | later byte-identical tool outputs | via expand | repeated identical outputs | `min_tokens` (100) |
| `collapse` | Offload | middle of an oversized output | via expand | any large tool output (fallback) | `max_tokens` (2000), `head_lines` (20), `tail_lines` (20) |
@@ -24,17 +24,26 @@ messages (`role:"tool"`; for Anthropic, `tool_result` blocks normalized to that
| `mask` | Offload | older tool outputs (age-based) | via expand | more than `keep_recent` outputs | `keep_recent` (3), `min_tokens` (100), `keep_head_chars` (96) |
| `summarize` | Offload (LLM) | the middle of the transcript → one summary | via expand | long trajectories | `summary_level` (regular), `keep_last` (3), `min_tokens` (500), `resummarize_tokens` (6000), `model.source`, `trigger` |
-Presets (`config`): `off` `[]` · `safe` `[format, cachesplit]` · `balanced`
-`[format, dedup, failed_run, cmdfilter, cachesplit]` · `aggressive` adds `smartcrush, extract` ·
+Presets (`config/config.go`), verbatim: **`codesmart`** (the proxy default)
+`[format, dedup, failed_run, cmdfilter, extract_llm, extract, cachesplit]` · **`codesafe`**
+`[format, dedup, failed_run, cmdfilter, extract, collapse, cachesplit]` (deterministic-only) ·
+`off` `[]` · `safe` `[format, cachesplit]` · `balanced`
+`[format, dedup, failed_run, cmdfilter, cachesplit]` · `aggressive`
+`[format, dedup, failed_run, cmdfilter, smartcrush, extract, extract_llm, cachesplit]` ·
`coding` `[format, skeleton, cmdfilter, cachesplit]` · `mcp` `[format, smartcrush, cachesplit]` ·
-**`agent`** `[format, dedup, failed_run, mask, extract, cachesplit]` — for long agentic sessions;
-`mask` is the biggest lever there (~27–30% content-token savings, no reward loss — see [RESULTS.md](RESULTS.md)) ·
-**`general`** `[format, toon, dedup, failed_run, cmdfilter, mask, extract, collapse, cachesplit]` — the
-recommended all-round pipeline: the reward-neutral levers of `agent` plus the situational shrinkers
-(`toon`/`cmdfilter`/`collapse`) that cost nothing when they don't fire. `balanced` is **not** recommended
-for agentic traffic — it omits `mask`, so it barely helps (6% vs 31% in the Terminal-Bench replay) ·
+**`agent`** `[format, dedup, failed_run, mask, extract, extract_llm, cachesplit]` — for long agentic
+sessions; `mask` is the biggest lever there (~27–30% content-token savings, no reward loss — see
+[RESULTS.md](RESULTS.md)) ·
+**`general`** `[format, toon, dedup, failed_run, cmdfilter, mask, extract, extract_llm, collapse, cachesplit]`
+— the recommended all-round pipeline: the reward-neutral levers of `agent` plus the situational
+shrinkers (`toon`/`cmdfilter`/`collapse`) that cost nothing when they don't fire. `balanced` is
+**not** recommended for agentic traffic — it omits `mask`, so it barely helps (6% vs 31% in the
+Terminal-Bench replay) ·
`summarize` `[summarize]` (run alone — it restructures the whole transcript).
+Every preset that touches caching carries `cachesplit`, never `cacheinject` — see
+[Presets](reference/presets.md).
+
**Dynamic, model-aware triggers.** Trigger thresholds can be expressed as **fractions of the model's
context window** (resolved dynamically via LiteLLM's public model map, no hand-maintained list):
`min_request_frac`, `min_output_frac`, and a hard `huge_output_frac` ("huge tool call" — act regardless of
@@ -48,10 +57,12 @@ store persists), so Offload markers are genuinely recoverable — not just descr
offloader also applies a **marker-inclusive** never-worse check per message, so a rewrite never grows a
message by the marker's tokens.
-**LLM-based components** (`extract` with `strategy: code`/`rlm`, and `summarize`) call a model, chosen by
+**LLM-based components** (`extract_llm`, and `summarize`) call a model, chosen by
`model.source`: `incoming` (default — reuse the proxied request's own model + key) or `config` (a dedicated
cheap model set via `CHEAP_MODEL*` env / the gateway's `CheapModel`). When no model is available they
-degrade — `extract` to its deterministic projection, `summarize` to a no-op. See [design.md](design.md#llm-components).
+degrade — `extract_llm` to a no-op (the deterministic `extract` beside it in every preset does the
+cheap pass), `summarize` to a no-op. `extract` itself never calls a model. See
+[design.md](design.md#llm-components).
Common gates every Offload respects: skip non-text (`Rewritable`) messages, skip content already
carrying a marker (no double-offload), and skip if the rewrite (marker + hint included) isn't
@@ -101,10 +112,12 @@ Places Anthropic `cache_control: {type: ephemeral}` breakpoints at the positions
billed input cost, so the provider KV cache is read rather than re-processed. Adds control
directives, changes no model-visible content.
-**In no preset — opt in explicitly.** Until [#32](https://github.com/rossoctl/context-guru/issues/32)
-its breakpoints never reached the provider on Claude Code traffic, so the placement policy has
-never been measured. The presets carry `cachesplit` instead, which enables the volatile-tail split
-(measured) without the placement (not).
+**In no preset — opt in explicitly.** Until [#36](https://github.com/rossoctl/context-guru/pull/36)
+its breakpoints never reached the provider on Claude Code traffic (46 applied, 0 forwarded), so the
+placement policy has never been measured. The presets carry [`cachesplit`](components/cachesplit.md)
+instead, which enables the volatile-tail split (measured) without the placement (not). The one live
+placement measurement since the fix is n=1 and mildly *harmful* per step, with no mechanism
+established — see [cacheinject](components/cacheinject.md#what-placement-is-actually-worth).
- **Lossiness:** none. **Shines:** Anthropic/Bedrock/Vertex agents that don't self-cache (the
savings lever is provider-side cache hits, invisible to `/stats` token counts). **Inert:**
@@ -112,6 +125,23 @@ never been measured. The presets carry `cachesplit` instead, which enables the v
breakpoint already present. `/stats` will list it under `top_passthrough` since it saves no
*content* tokens — that's expected, not dead weight.
+### `cachesplit`
+Splits the volatile tail of the top-level `system` array off its stable head — `[stable][volatile]`
+as two text blocks with the same concatenated text, breakpoint on the first — so the provider's
+cache boundary excludes the churn. Adjacent text blocks concatenate, so the model sees a
+byte-identical prompt.
+
+**In every caching preset.** It is a *marker* component: the `Reformat` method always skips, and the
+rewrite is body-level (`apply/prefixsplit.go`), gated on this name being in the pipeline. That
+separation exists so disabling breakpoint placement does not silently disable the split.
+
+- **Config:** none. **Lossiness:** none. **Shines:** Anthropic-family agents whose system prompt
+ carries a churning tail (env snapshot, git status, timestamp) in front of a breakpoint — measured
+ −34.1% mean cost on one Terminal-Bench task over three trials. **Inert:** implicit-prefix-cache
+ providers (OpenAI, Gemini), or a system block with no separable tail. Always in
+ `top_passthrough` (its saving is a provider-side cache effect). Full page:
+ [cachesplit](components/cachesplit.md).
+
---
## Offload (lossy, reversible)
@@ -184,19 +214,22 @@ after: [superseded by a later run] <> [full output: …] [run 2]
### `cmdfilter`
Shrinks tool output with **declarative DSL filters** (see below). Matches a filter on the output's
-first non-empty line, applies its 8-stage pipeline, stashes the original, and appends a recovery
-hint only when the filter was actually lossy. Ships 23 filters (test runners, build tools, package
-managers, IaC plans, verbose network clients) — see [cmdfilter](components/cmdfilter.md).
+first **six** non-empty lines (the selector), applies its 8-stage pipeline, stashes the original, and
+appends a recovery hint only when the filter was actually lossy — typed by *what* was lost. Ships
+**24** filters across 5 families (`builds` 10, `pkg` 7, `iac` 3, `net` 3, `tests` 1) — see
+[cmdfilter](components/cmdfilter.md).
```
before: pytest … 100 lines of PASSED + warnings + 1 failure
after: <> [full output: …]
```
-- **Config:** `filters` (inline filter YAML docs, added with no recompile), `disable_builtins`.
- `Enabled` only when ≥1 filter is loaded. **Shines:** noisy but structured command/log output
- (test runners, package managers, build tools). **Inert:** output whose first line matches no
- filter, or where filtering doesn't shrink it.
+- **Config:** `filters` (inline filter YAML docs, added with no recompile), `disable_builtins`,
+ `marker_mode`, `min_size` (500-byte floor — below it the marker routinely costs more than the
+ filter saves). `Enabled` only when ≥1 filter is loaded. **Shines:** noisy but structured
+ command/log output (test runners, package managers, build tools). **Inert:** output whose selector
+ matches no filter (logged in `cmdfilter_selector_misses`), output under `min_size`, or where
+ filtering doesn't shrink it.
### `extract`
**Deterministic, no-LLM.** Collapses only *obvious, provably redundant* noise: consecutively repeated
@@ -310,10 +343,11 @@ The summarizer is grounded in the **current task** (first user turn + recent tur
## The DSL filter engine
-`components/dsl` is a declarative, user-extensible text-filter engine (adapted from rtk), wrapped
-by `cmdfilter`. Filters are authored in YAML (no recompile), matched first-by-sorted-name, and each
-runs a fixed **8-stage** pipeline. Because filters drop lines they are lossy, which is why the
-wrapping `cmdfilter` component is an Offload (it stashes the original first).
+`components/dsl` is a declarative, user-extensible text-filter engine (adapted from rtk — Apache-2.0,
+see `THIRD-PARTY-NOTICES`), wrapped by `cmdfilter`. Filters are authored in YAML (no recompile),
+matched by **descending `priority` then by name**, and each runs a fixed **8-stage** pipeline. Because
+filters drop lines they are lossy, which is why the wrapping `cmdfilter` component is an Offload (it
+stashes the original first).
```mermaid
flowchart LR
@@ -322,26 +356,32 @@ flowchart LR
S6 --> S7[7 max_lines] --> S8[8 on_empty] --> O[output + Lossiness]
```
-Filter fields (all optional except `match`): `match` (regex vs the selector = first non-empty
-line), `strip_ansi`, `replace` (chained `pattern`→`replacement`, `$1` backrefs), `match_output`
-(whole-blob short-circuit: `pattern`/`message`/`unless`), `strip_lines_matching` **xor**
-`keep_lines_matching`, `truncate_lines_at` (per-line char cap), `head_lines`/`tail_lines`,
-`max_lines` (absolute cap with omission marker), `on_empty` (replacement when output is blank).
+Filter fields (all optional except `match`): `match` (regex vs the selector = the first six
+non-empty lines, compiled with `(?m)`), `family` (per-family `/stats` attribution), `priority`
+(match order, higher first), `strip_ansi`, `replace` (chained `pattern`→`replacement`, `$1`
+backrefs), `match_output` (whole-blob short-circuit: `pattern`/`message`/`unless`),
+`strip_lines_matching` **xor** `keep_lines_matching`, `truncate_lines_at` (per-line char cap),
+`head_lines`/`tail_lines`, `cap`/`cap_reduce` (a shared line-budget class), `max_lines` (absolute
+cap with omission marker, wins over `cap`), `on_empty` (replacement when output is blank).
-`Lossiness` reported back to `cmdfilter` (drives whether a recovery hint is appended): `None`
-(nothing dropped / reversible reformat), `Tail` (a clean contiguous tail dropped), `Whole`
-(non-contiguous or whole-blob loss).
+`Lossiness` reported back to `cmdfilter` (drives *which* recovery hint is appended): `None`
+(nothing dropped / reversible reformat → no hint), `Tail` (a clean contiguous tail dropped → the
+hint names the cut point, since re-reading from there is cheaper than a full expand), `Whole`
+(non-contiguous or whole-blob loss → the hint points at the expand tool). `Tail` and `Whole` used
+to share one hint text; they are now distinct.
```yaml
schema_version: 1
filters:
pytest:
description: keep failures + summary, drop passing noise
+ family: tests
+ priority: 10
match: "(pytest|=+ test session starts)"
strip_lines_matching: ["^\\s*$", " PASSED", "^\\.+$"]
- max_lines: 80
+ cap: buildlog # shared budget class; or a literal max_lines
on_empty: "pytest: all passed"
-tests: # inline; run via dsl.RunTests (a `verify` command)
+tests: # inline; run AT LOAD, and via dsl.RunTests
pytest:
- name: all-green
input: "pytest\n....\n"
@@ -349,4 +389,6 @@ tests: # inline; run via dsl.RunTests (a `verify` command)
```
Documents load with `schema_version: 1` and strict unknown-field rejection. Inline `tests`
-(input → expected) run via `dsl.RunTests`, so a filter ships with its own regression check.
+(input → expected) run **at load time** as well as via `dsl.RunTests`, so a filter whose tests fail
+never loads at all. Load also rejects duplicate filter names, an uncompilable regex, `strip` and
+`keep` both set, an unknown `cap` class, and `cap_reduce` without `cap`.
diff --git a/docs/components/cacheinject.md b/docs/components/cacheinject.md
index a0cc616..4371429 100644
--- a/docs/components/cacheinject.md
+++ b/docs/components/cacheinject.md
@@ -1,8 +1,11 @@
# cacheinject
-!!! info "Reformat — lossless"
+!!! info "Reformat — lossless. **In no preset: opt in explicitly.**"
Places Anthropic `cache_control` breakpoints at the positions that minimise billed
- input cost, so the provider KV cache is read rather than re-processed.
+ input cost, so the provider KV cache is read rather than re-processed. Placement has
+ never been shown to help, so no shipped preset enables it — the presets carry
+ [`cachesplit`](cachesplit.md), which enables the *measured* volatile-tail split. See
+ [Configuration](#configuration).
!!! danger "Every placement number below predates the fix in #32 — read this first"
Until #32, **this component's breakpoints never reached the wire on Claude Code
@@ -27,9 +30,15 @@
with a 400. That never fired in production *only* because the first defect
suppressed the marks; fixing one without the other would have produced a live 400.
- Both are fixed (see [design.md](../design.md) — the metadata-write exception).
- Breakpoints now reach the wire on **86 of 92** replayed captured requests, and the
- wire total is asserted never to exceed 4.
+ Both are fixed in [#36](https://github.com/rossoctl/context-guru/pull/36) (see
+ [design.md](../design.md) — the metadata-write exception). Breakpoints now reach the wire
+ on **86 of 92** replayed captured requests, and the wire total is asserted never to
+ exceed 4.
+
+ One earlier attribution is also corrected: bifrost does **not** drop `cache_control` on a
+ `tool_result` block — it round-trips it fine. The mark on that block is dropped by *this
+ repo's own* `toolMessage()` in `normalize`, which rebuilds the block into a synthetic
+ `role=tool` message from text + `tool_use_id` alone. The bug was ours.
**Consequence for every figure on this page:** the placement rows measured a
component whose output was discarded. The `acted=0` / "placement contributes $0"
@@ -86,10 +95,15 @@ after: [ tools ][ system ]…[ msg d−1 {cache_control} ]…[ newest turn {ca
^ rescues the stable head ^ writes this turn's growth
```
-## What it is worth — measured, not asserted
+## What it is worth — simulated
+
+Everything in this section is a **simulation** over a captured request stream, not a live
+billed comparison. It was calibrated to within 2.10 pp of the cache-hit rate a live 50-task
+run billed, which makes it a credible model of the billing rule — it does not make it a
+measurement of the policy. For the one live reading, see
+[What placement is actually worth](#what-placement-is-actually-worth).
-Simulated on a captured 91-request SWE-bench stream, calibrated to within 2.10 pp of
-the cache-hit rate a live 50-task run actually billed:
+Simulated on a captured 91-request SWE-bench stream:
| policy | cost | hit | vs the agent's own placement |
|---|--:|--:|--:|
@@ -120,9 +134,10 @@ saving against claude-code:
below a divergence when a prefix mutates mid-conversation — neither of which
claude-code needs.
-The measurable cost saving on claude-code traffic comes from the **cross-session
-prefix repairs** in `apply/prefixsplit.go`, which are gated on this component being
-configured but are a different mechanism. See below.
+The measurable cost saving on claude-code traffic comes from the **volatile-tail split** in
+`apply/prefixsplit.go` — a different mechanism, now carried by its own
+[`cachesplit`](cachesplit.md) component so it no longer depends on this one being configured.
+See below.
!!! warning "v1 was a regression"
v1 placed a single breakpoint on the message *before* the newest turn, which by
@@ -133,10 +148,13 @@ configured but are a different mechanism. See below.
## What placement is actually worth
-**Still unmeasured.** #32 made the policy reach the provider for the first time; it did
-not establish what the policy is worth. One `cacheonly` vs `off` pair was attempted and
-is reported here in full because it is all the evidence there is — not because it settles
-anything.
+**Still effectively unmeasured, and the one live reading is mildly negative.**
+[#36](https://github.com/rossoctl/context-guru/pull/36) made the policy reach the provider
+for the first time; it did not establish what the policy is worth. One `cacheonly` vs `off`
+pair was attempted and is reported here in full because it is all the evidence there is —
+not because it settles anything. **Per step it is +7.9% cost and +61.9% cache-write, at n=1,
+with no mechanism established and the obvious mechanism ruled out.** That is why placement
+ships off.
SWE-bench Verified, `aws/claude-sonnet-5`, n=1 per arm. `off`'s second trial died on a
Docker-compose error, so only `astropy-12907` completed in both arms:
diff --git a/docs/components/cachesplit.md b/docs/components/cachesplit.md
new file mode 100644
index 0000000..5644fc5
--- /dev/null
+++ b/docs/components/cachesplit.md
@@ -0,0 +1,84 @@
+# cachesplit
+
+!!! info "Reformat — lossless, and a marker component"
+ Enables the **volatile-tail split**: the top-level `system` array's churning tail is
+ separated from its stable head so the provider's cache boundary excludes the churn.
+ The model sees byte-identical text.
+
+`cachesplit` is in the default presets. It carries no logic of its own: the component's
+`Reformat` always reports `Skipped`, and the actual rewrite is body-level — it edits the
+top-level `system` array, which components never see — and lives in `apply/prefixsplit.go`,
+gated on this name being present in the pipeline.
+
+That is deliberate. `cachesplit` and [`cacheinject`](cacheinject.md) were one config entry
+until [#36](https://github.com/rossoctl/context-guru/pull/36), which separated them because
+their evidence is not comparable:
+
+- the **split** is measured — −34.1% cost and 0% → 96.7% cache hit in an isolated A/B;
+- **placement** has never been measured, because until #36 its breakpoints never reached the
+ provider at all.
+
+Splitting the config entry is what keeps disabling placement from silently disabling the
+split too. `apply` still honours `cacheinject` as a gate for the split, so an existing
+pipeline naming only `cacheinject` keeps working.
+
+## What the split does
+
+A cache entry hashes **everything before** its breakpoint, and no breakpoint position can
+exclude part of a single content block. Claude Code appends a live environment snapshot to
+the **end** of its main system block:
+
+```
+Current branch: main
+...
+Recent commits:
+0898367954 SWE-bench
+```
+
+Measured across 50 SWE-bench tasks that block is ~7,017 tokens, of which the first
+**6,921 (98.4%)** are byte-identical across sessions — but it is one cacheable unit with its
+breakpoint at the end, so the hash covers the churning tail and the shared 98.4% is
+re-written every session.
+
+The tail is real content: it cannot be moved or dropped without lying to the model about the
+repo state. It can be **split** — `[stable][volatile]` as two text blocks with the same
+concatenated text, breakpoint on the first. Adjacent text blocks concatenate, so the model
+sees a **byte-identical** prompt while the provider gains a hash boundary that excludes the
+churn. Asserted by `TestSplitIsConcatenationIdentical`.
+
+## What it is worth
+
+The full four-way measurement (structural target, mechanical verification, isolated live
+A/B, end-to-end agent run) is on the `cacheinject` page under
+[The volatile-tail split](cacheinject.md#the-volatile-tail-split), because that is where the
+work was done. The headline: **−34.1% mean cost on one Terminal-Bench task measured three
+times**, and `$0.0205` saved per warm session on Sonnet 5 in the isolated A/B. Read the
+caveats there — a second Terminal-Bench task was within noise and its numbers are not
+quoted.
+
+## Configuration
+
+None. It is a name in the pipeline:
+
+```yaml
+pipeline: [format, dedup, failed_run, cmdfilter, cachesplit]
+```
+
+## When it shines
+
+Anthropic-family agents whose system prompt carries a volatile tail (an environment
+snapshot, a timestamp, a git status) in front of a cache breakpoint.
+
+## When it's inert
+
+**Explicit-breakpoint providers only** (Anthropic, Bedrock, Vertex). Under an implicit
+longest-prefix cache (OpenAI, Gemini) the match already ends at the divergence, so a block
+boundary buys nothing. It is also inert when the system block has no separable volatile
+tail.
+
+Because the component itself always skips, `/stats` lists it under `top_passthrough` — that
+is expected, not dead weight. The split's saving is a provider-side cache effect, invisible
+to content-token counts.
+
+See also: [Components overview](../components.md) · [cacheinject](cacheinject.md) ·
+[Choose a preset](../how-to/choose-a-preset.md)
diff --git a/docs/components/cmdfilter.md b/docs/components/cmdfilter.md
index 61afd84..055247f 100644
--- a/docs/components/cmdfilter.md
+++ b/docs/components/cmdfilter.md
@@ -6,9 +6,10 @@
## How it works
`cmdfilter` shrinks tool output with **declarative DSL filters** (see
-[The DSL filter engine](dsl.md)). It matches a filter on the output's first non-empty line, applies
-its 8-stage pipeline, stashes the original, and appends a recovery hint only when the filter was
-actually lossy. It is `Enabled` only when ≥1 filter is loaded.
+[The DSL filter engine](dsl.md)). It matches a filter on the output's **first six non-empty lines**
+(the selector), applies its 8-stage pipeline, stashes the original, and appends a recovery hint only
+when the filter was actually lossy — typed by *what* was lost. Filters are tried in descending
+`priority`, then by name. It is `Enabled` only when ≥1 filter is loaded.
Deterministic filtering costs nothing — no LLM call, ~0 latency — and it is cache-safe: it acts on
the newest tool output, in the mutable tail.
@@ -35,8 +36,9 @@ and `$` anchor per line.
That is also the structural advantage: rtk's hook only sees Bash calls, so an agent's built-in
`Read`/`Grep`/`Glob` tools are invisible to it. A proxy sees every tool result regardless of origin.
-A `TestEveryBuiltinFilterHasTestsAndRoutes` guardrail asserts every filter's own test input actually
-routes to that filter, so a selector rewrite is verified rather than hoped for.
+A `TestEveryBuiltinFilterHasTestsAndRoutes` guardrail asserts every filter ships ≥1 inline test and
+that each test's input actually routes to *that* filter, so a selector rewrite is verified rather
+than hoped for.
### Size floor
@@ -91,11 +93,18 @@ shapes dominated the misses:
matches the command, so its patterns never had to name this shape; ported as-is, the filter would
miss the most common way gcc output starts.
-Meanwhile the IaC and mobile-build filters (`pulumi`, `terraform-plan`, `xcodebuild`, `gradle`) fired
-**zero** times on that dump. They are kept — they are correct, tested and cost nothing when inert —
-but the honest reading is that a filter set's value is decided by the workload, not by its size. The
+Between them those two shapes carry roughly **73% of the live savings** on that dump.
+
+Meanwhile the filters that had been *planned* as Tier 1 — `pulumi`, `terraform-plan`, `xcodebuild`,
+`gradle` — fired **zero** times on it. They are kept: they are correct, tested and cost nothing when
+inert. But the honest reading is that a filter set's value is decided by the workload, not by its
+size, and the prediction about which filters would matter was wrong. The
`cmdfilter_selector_misses` ledger exists so that stays measurable rather than assumed.
+!!! note "One benchmark, one dump"
+ The 73% / zero-fire split is Terminal-Bench tool output. A repo full of Terraform would
+ invert it. Read the ledger on *your* traffic before pruning a filter.
+
### A cautionary note on strip rules
The `apt` filter originally stripped `^debconf: `, which also swallowed
diff --git a/docs/components/dsl.md b/docs/components/dsl.md
index 8fb5aa9..38e6f05 100644
--- a/docs/components/dsl.md
+++ b/docs/components/dsl.md
@@ -56,7 +56,7 @@ The class names and the first four values are rtk's (`src/core/truncate.rs`); `b
| `cap` | lines | for |
|---|--:|---|
| `errors` | 20 | error lists — most actionable, shown the most |
-| `warnings` | 10 | warnings and test failures — lower signal density |
+| `warnings` | 10 | warnings — lower signal density than errors |
| `list` | 20 | flat lists (packages, services): one line per item |
| `inventory` | 50 | exhaustive lookups (installed packages, file listings) |
| `buildlog` | 80 | full build/plan transcripts — verbose, and the signal can sit anywhere |
diff --git a/docs/design.md b/docs/design.md
index ee4ca0a..41c139e 100644
--- a/docs/design.md
+++ b/docs/design.md
@@ -11,8 +11,8 @@ infrastructure the components sit on.
| Package | Role |
|---|---|
| `components/` | `Component`/`Reformat`/`Offload` interfaces, `Report`, `Ctx`, the `Pipeline`, the registry |
-| `components/reformat/` | lossless components: `format`, `cacheinject` |
-| `components/offload/` | lossy-reversible components: `skeleton`, `dedup`, `collapse`, `failed_run`, `cmdfilter`, `extract`, `smartcrush`, `mask` |
+| `components/reformat/` | lossless components: `format`, `toon`, `cacheinject`, `cachesplit` |
+| `components/offload/` | lossy-reversible components: `skeleton`, `dedup`, `collapse`, `failed_run`, `cmdfilter`, `extract`, `extract_llm`, `smartcrush`, `mask`, `summarize` |
| `components/dsl/` | declarative text-filter engine (wrapped by `cmdfilter`) |
| `components/all/` | blank-imports every component so `init()` registrations run |
| `schema/` | helpers over bifrost's schema: token counting, deep-clone, `MessageText`/`SetMessageText`, `Rewritable` |
@@ -57,7 +57,8 @@ YAML block as bytes), `NeedsModel` (declares it calls a cheap LLM — the model
yet wired).
- **Reformat** = lossless repack (`format` re-encodes JSON compact; `cacheinject` adds
- `cache_control`). No information leaves the wire, so nothing is stashed.
+ `cache_control`; `cachesplit` is a marker enabling a body-level split). No information leaves the
+ wire, so nothing is stashed.
- **Offload** = lossy-but-reversible. It drops bytes and returns the `cache_keys` under which
it stashed the originals. If it shrinks the request but returns no keys, the pipeline treats
it as a **failed offload and reverts** — you cannot silently lose data. Returning no keys and
@@ -220,8 +221,8 @@ bytes* (`expand.rawMarkerRe`, used by the host's streaming decision) must accept
## State: the Store
-One `Store` interface, in-memory TTL+LRU default (both hosts share it). Defaults: **10000s TTL,
-1000 entries, 100 sticky sessions**. It carries, keyed per session:
+One `Store` interface, in-memory TTL+LRU default (both hosts share it). Defaults: **10000s sliding
+TTL, 1000 entries, 100 sticky sessions**. It carries, keyed per session:
- **Rewind** — `cache_key → original bytes` (what the expand loop resolves).
- **Sticky** — the set of content ids already reduced on prior turns (for byte-stable output
@@ -272,12 +273,16 @@ set — the payload need not survive, only the knowledge that it existed):
the repair branch would buy a model call for nothing. There is no upside, so a lost `extract_llm`
decision simply declines and the message is forwarded verbatim. (Its entry is still pinned, so
the common case is that it is never lost at all.) Re-enabling it would need deterministic
- decoding *plus* a check that the re-derived bytes match the stored hash before splicing.
+ decoding *plus* a check that the re-derived bytes match the stored hash before splicing. The
+ dedicated `repairLostResult` path that used to attempt it was **removed** rather than left
+ disabled — a repair that can splice different bytes is not a repair worth keeping behind a flag.
+ `cg:res:` also unifies what were two keys (`cg:res:` + `cg:sum1:`) into one JSON value, so the
+ projection and its summary line cannot half-survive a drop.
`/stats` reports `frozen_hits`, `frozen_misses`, `frozen_dropped`, `frozen_repaired`, and
`frozen_flips` (= dropped − repaired; should be 0). `frozen_misses` is a *lookup* counter dominated
by the ordinary "not compacted yet" case — `frozen_dropped` is the one that measures harm. See
-[Routes](reference/routes.md#stats-freeze-replay-fields).
+[Routes](reference/routes.md#freeze-replay-health).
The related fail-*open* on `MaxCachedIdx`: `prevLen` returning 0 on a store miss yields
`MaxCachedIdx = -1`, and `Ctx.TailOnly` then permits mutating any index (measured on 11.2% of
@@ -301,19 +306,33 @@ vocabulary), `Aggregator` (in-process rollups behind `/stats`), `Tee` (fan-out),
of per-request percentages. It also reports:
- `wasted_tokens` / `bounces` — content offloaded then re-served via expand (a premature offload);
- `adjusted_saved` = saved − wasted (bounce-adjusted, may be negative);
-- `top_passthrough` — components that ran but never changed a request: dead weight to drop;
+- `top_passthrough` — components that ran but never changed a request: dead weight to drop. A
+ component that mutated without saving *content* tokens (`cachesplit`) is not listed;
+- `discarded_changes` (per component) / `top_discarded` — changes the writeback layer threw away.
+ Before this existed, a mutated-then-discarded component was indistinguishable from a working
+ Reformat, which is how the `cacheinject` bug survived two benchmark studies;
- `sse_streamed` / `sse_buffered` / `sse_buffered_pct` and `sse_ttfb_ms_avg` /
`sse_ttfb_ms_avg_buffered` — streaming health: how many SSE responses had to be buffered whole to
- be inspected for an expand call, and what that cost in time-to-first-byte.
+ be inspected for an expand call, and what that cost. The `_buffered` average is
+ time-to-*last*-byte by construction, so it is not comparable to `sse_ttfb_ms_avg`;
+- `frozen_hits` / `frozen_misses` / `frozen_dropped` / `frozen_repaired` / `frozen_flips` — the
+ cache-write cost line (see [Freeze lifetime](#freeze-lifetime-and-which-way-to-fail));
+- `cmdfilter_families` / `cmdfilter_filters` / `cmdfilter_selector_misses` — which command families
+ and individual filters pay off, and which output shapes matched nothing;
+- `saved_tokens` vs `saved_tokens_unique` and `overcount_ratio` — cumulative vs distinct. The agent
+ re-sends history verbatim every turn, so the cumulative figure double-counts. Quote the unique one;
+- `mode` / `sync_enforced`, and the `potential_*` / `projected_*` observe namespace.
Fields are only ever **added** to `/stats`; the harbor harnesses parse it, so no field is renamed
-or removed.
+or removed. The full field list is in [Routes](reference/routes.md#get-stats).
## Operating modes
Two modes, set explicitly by `mode:` (or `--mode` / `MODE`) and threaded onto
-`components.Ctx` as `Ctx.Mode`. Never inferred. `sync` is the default and reproduces
-pre-mode behavior byte for byte; a golden test compares the two entry points' output.
+`components.Ctx` as `Ctx.Mode`, whose zero value resolves to `sync`. Never inferred. `sync`
+reproduces pre-mode behavior byte for byte; a golden test compares the two entry points' output.
+An `async` mode is implemented on a held branch and is **not** available — the loader accepts
+only `sync` and `observe`.
See [Operating modes](how-to/operating-modes.md) for the operator's view. What follows is
the mechanism.
@@ -383,7 +402,7 @@ Unknown keys are rejected.
```yaml
preset: balanced
-pipeline: [format, dedup, failed_run, cmdfilter, cacheinject] # order + enable
+pipeline: [format, dedup, failed_run, cmdfilter, cachesplit] # order + enable
components:
collapse: { max_tokens: 2000, head_lines: 20, tail_lines: 20 }
smartcrush: { min_items: 5, keep_first: 3, keep_last: 2 }
@@ -396,8 +415,9 @@ component's config.
## LLM components
-Most components are deterministic. Two call an LLM: `extract` (`strategy: code`/`rlm`, a Starlark
-filter run in a sandbox) and `summarize` (whole-transcript summary). They implement `NeedsModel` and
+Most components are deterministic. Two call an LLM: `extract_llm` (`strategy: code`, a Starlark
+filter run in a sandbox) and `summarize` (whole-transcript summary). The deterministic `extract` is a
+separate component and never calls a model. They implement `NeedsModel` and
call `Ctx.Model` — a `ModelSpec` the host resolves per request:
```mermaid
@@ -405,7 +425,7 @@ flowchart LR
cfg["component config
model.source"] --> res{"ModelSpec.For(source)"}
res -->|incoming| inc["Incoming: request's own
model + upstream + key
(built in proxy.chat)"]
res -->|config| stat["Static: cheap model
(CHEAP_MODEL* env)"]
- res -->|nil| deg["degrade: extract→deterministic,
summarize→no-op"]
+ res -->|nil| deg["degrade: extract_llm→no-op,
summarize→no-op"]
inc --> call["Model.Complete(ctx, prompt)"]
stat --> call
```
@@ -413,6 +433,7 @@ flowchart LR
- **`incoming`** (default) reuses the proxied request's model + the gateway's key — zero extra config,
works through the eval-containers gateway. **`config`** uses a dedicated cheap model (`internal/cheapmodel`
Anthropic/OpenAI). The AuthBridge host offers only `config` (its incoming key is a placeholder).
-- The call is synchronous in the request path, so it's bounded (short timeout, retry) and **fail-open**:
- any error reverts the component (pipeline guarantee), and a missing model degrades gracefully.
+- The call is synchronous in the request path (in `sync` mode; in `observe` it happens off-path), so
+ it's bounded (short timeout, retry) and **fail-open**: any error reverts the component (pipeline
+ guarantee), and a missing model degrades gracefully.
- Reversibility is unchanged — the LLM output is still stashed under a `<>` marker for `expand`.
diff --git a/docs/examples/live-captures.md b/docs/examples/live-captures.md
index ef800da..02c0e87 100644
--- a/docs/examples/live-captures.md
+++ b/docs/examples/live-captures.md
@@ -39,9 +39,13 @@ after: [6]{id,name,status}:
… 4 more rows …
```
-### `cacheinject` — adds a cache breakpoint on the stable prefix
-The last content block of the message *before* the newest turn gains a `cache_control` directive
-(no model-visible content changes):
+### `cacheinject` — adds a cache breakpoint (opt-in, in no preset)
+A `cache_control` directive is attached to the last content block of a chosen message (no
+model-visible content changes). The v1 policy of always marking the message *before* the newest
+turn was a **+5.5% regression** and is gone; the current policy marks the newest message plus the
+last one still matching the previous turn. This capture predates
+[#36](https://github.com/rossoctl/context-guru/pull/36), so on Claude Code traffic a mark like this
+would have been discarded by the writeback layer rather than reaching the provider:
```json
{"role":"assistant","content":[{"type":"text",
"text":"Here is a fairly long answer … worth caching across turns.",
@@ -151,8 +155,10 @@ the `general` preset against a real gateway, `GET /stats` reported:
```
`dedup` + `mask` are deterministic; `extract_llm` made one real cheap-model call. Components that
-found nothing to do (`format`, `toon`, `cmdfilter`, `collapse`, `extract`, `failed_run`,
-`cacheinject`) sit in `top_passthrough` at zero cost — the pipeline is never worse.
+found nothing to do (`format`, `toon`, `cmdfilter`, `collapse`, `extract`, `failed_run`) sit in
+`top_passthrough` at zero cost — the pipeline is never worse. `cachesplit` is there too, but
+permanently and by design: its `Reformat` always skips because the split it enables is a body-level
+rewrite in `apply`.
## Reversibility, verified
diff --git a/docs/get-started/overview.md b/docs/get-started/overview.md
index 73d98db..517602a 100644
--- a/docs/get-started/overview.md
+++ b/docs/get-started/overview.md
@@ -49,8 +49,8 @@ config order:
```mermaid
flowchart TD
C["Component — Name() · Enabled(ctx)"]
- C --> R["Reformat: lossless repack
format · cacheinject"]
- C --> O["Offload: drop + stash, returns cache_keys
skeleton · dedup · collapse · failed_run
cmdfilter · extract · smartcrush · mask"]
+ C --> R["Reformat: lossless repack
format · toon · cachesplit · cacheinject"]
+ C --> O["Offload: drop + stash, returns cache_keys
skeleton · dedup · collapse · failed_run · cmdfilter
extract · extract_llm · smartcrush · mask · summarize"]
```
## Core concepts
diff --git a/docs/how-to/choose-a-preset.md b/docs/how-to/choose-a-preset.md
index 29526f0..83b68d8 100644
--- a/docs/how-to/choose-a-preset.md
+++ b/docs/how-to/choose-a-preset.md
@@ -40,26 +40,31 @@ is nothing to expand.
- **Fits:** any traffic where you want a zero-risk win and no reversibility surface.
- **Caveat:** `cachesplit`'s savings are provider-side cache hits, invisible to `/stats` token
counts — it will show up under `top_passthrough`. That's expected, not dead weight.
-- **Note:** breakpoint *placement* (`cacheinject`) is deliberately **not** here — it is unmeasured
- and opt-in since [#32](https://github.com/rossoctl/context-guru/issues/32). `cachesplit` carries
- the part with measured savings.
+- **Note:** breakpoint *placement* (`cacheinject`) is deliberately **not** here — it is opt-in
+ since [#36](https://github.com/rossoctl/context-guru/pull/36), and the one live reading since
+ its marks began reaching the provider is mildly *negative* per step at n=1 with no mechanism
+ established. `cachesplit` carries the part with measured savings.
### `balanced` — `[format, dedup, failed_run, cmdfilter, cachesplit]`
-The default. Adds three cheap, high-precision offloaders: exact-dup removal (`dedup`), superseded
-test/build runs (`failed_run`), and DSL command-log filtering (`cmdfilter`).
-
-- **Fits:** general agent traffic; the safe everyday choice.
-- **Caveat:** `cmdfilter` only fires when ≥1 filter is loaded and the output's first line matches
- one. It ships 23 filters covering test runners, build tools, package managers, IaC plans and
+Adds three cheap, high-precision offloaders: exact-dup removal (`dedup`), superseded test/build
+runs (`failed_run`), and DSL command-log filtering (`cmdfilter`).
+
+- **Fits:** general non-agentic traffic where you want a conservative win.
+- **Caveat:** **not** the choice for long agentic sessions — it omits `mask`, the biggest lever
+ there, and delivered 6% against `general`'s 31% in the Terminal-Bench replay. Use `codesmart`
+ (the actual default), `agent` or `general` for that.
+- **Caveat:** `cmdfilter` only fires when ≥1 filter is loaded and the output's selector matches
+ one. It ships **24** filters covering test runners, build tools, package managers, IaC plans and
verbose network clients; author more with a [custom DSL filter](custom-dsl-filter.md).
-### `aggressive` — `[format, dedup, failed_run, cmdfilter, smartcrush, extract, cachesplit]`
-`balanced` plus JSON-array crushing (`smartcrush`) and query-relevance projection (`extract`).
+### `aggressive` — `[format, dedup, failed_run, cmdfilter, smartcrush, extract, extract_llm, cachesplit]`
+`balanced` plus JSON-array crushing (`smartcrush`), deterministic noise collapse (`extract`) and
+the cheap-model relevance trimmer (`extract_llm`).
- **Fits:** you want more savings and accept structural/LLM offload with expand recovery.
-- **Caveat:** `extract` with `strategy: code`/`rlm` spends a model call (gated by its `trigger`);
- the default `deterministic` strategy is free. Keep the [store](recover-context.md) on so the
- extra offloads stay recoverable.
+- **Caveat:** `extract_llm` spends a model call (gated by its `trigger` and throttled per session);
+ `extract` beside it is free and deterministic. With no cheap model configured `extract_llm`
+ no-ops. Keep the [store](recover-context.md) on so the extra offloads stay recoverable.
### `coding` — `[format, skeleton, cmdfilter, cachesplit]`
Swaps in `skeleton`, which tree-sitter-parses fenced code blocks and replaces function bodies with
@@ -76,7 +81,7 @@ items plus any item carrying an error signal, drop the middle.
- **Fits:** MCP tools and REST list endpoints returning long uniform arrays.
- **Caveat:** inert on non-array output or arrays below `min_items`.
-### `agent` — `[format, dedup, failed_run, mask, extract, cachesplit]`
+### `agent` — `[format, dedup, failed_run, mask, extract, extract_llm, cachesplit]`
Tuned for long agentic sessions (e.g. Claude Code on SWE-bench) where the dominant cost is the
transcript of old tool outputs re-sent every turn.
@@ -86,6 +91,15 @@ transcript of old tool outputs re-sent every turn.
on a long session) with no reward loss ([Benchmarks](../RESULTS.md)). Order matters: lossless
first, then offload old-then-large, cache last.
+### `general` — `[format, toon, dedup, failed_run, cmdfilter, mask, extract, extract_llm, collapse, cachesplit]`
+The recommended all-round pipeline: the reward-neutral levers of `agent` plus the situational
+shrinkers (`toon`, `cmdfilter`, `collapse`) that cost nothing when they don't fire.
+
+- **Fits:** any agent or benchmark, when you don't want to pick per workload.
+- **Caveat:** it stacks one LLM component (`extract_llm`) and one blind fallback (`collapse`). It
+ deliberately does **not** stack `summarize` beside `mask` — they are overlapping old-context
+ reducers, and `mask` is the one kept.
+
### `summarize` — `[summarize]`
One LLM component that collapses the middle of the transcript into a single
`=== History Summary ===` message, keeping the head + last few turns.
@@ -96,7 +110,13 @@ One LLM component that collapses the middle of the transcript into a single
It needs a model; with none it no-ops.
!!! warning "LLM presets cost model calls"
- `aggressive` (via `extract` code/rlm) and `summarize` call a model. Both are gated by a
- `trigger` and reuse prior compactions per session, so they don't fire every turn. Pick the
- model with `model.source` (`incoming` reuses the request's own model+key; `config` uses a
- dedicated cheap model). See [LLM components](../design.md#llm-components).
+ `codesmart`, `aggressive`, `agent` and `general` (all via `extract_llm`) and `summarize` call
+ a model. Every call is gated by a `trigger` and throttled per session, and a prior compaction
+ is reused byte-for-byte on later turns, so they don't fire every turn. Pick the model with
+ `model.source` (`incoming` reuses the request's own model+key; `config` uses a dedicated cheap
+ model set via `CHEAP_MODEL*`). With no model available they no-op. See
+ [LLM components](../design.md#llm-components).
+
+!!! info "Every caching preset carries `cachesplit`, not `cacheinject`"
+ Breakpoint *placement* is in no preset — see the note under `safe` above and the
+ [`cacheinject` page](../components/cacheinject.md#what-placement-is-actually-worth).
diff --git a/docs/how-to/custom-dsl-filter.md b/docs/how-to/custom-dsl-filter.md
index 41e4d6d..827d208 100644
--- a/docs/how-to/custom-dsl-filter.md
+++ b/docs/how-to/custom-dsl-filter.md
@@ -135,15 +135,15 @@ components:
strip_lines_matching: ["^\\s*$", " PASSED", "^\\.+$"]
max_lines: 80
on_empty: "pytest: all passed"
- disable_builtins: false # keep the 23 shipped filters too
+ disable_builtins: false # keep the 24 shipped filters too
min_size: 500 # byte floor: below it the marker costs more than the saving
```
- `cmdfilter` is `Enabled` only when ≥1 filter is loaded.
- It ships [24 filters](../components/cmdfilter.md#the-shipped-filter-set); set
`disable_builtins: true` to run only your own.
-- The output's first non-empty line is the selector each filter's `match` is tested against, in
- descending `priority` then name order.
+- The output's first **six** non-empty lines are the selector each filter's `match` is tested
+ against, in descending `priority` then name order.
- Outputs smaller than `min_size` (default 500 bytes) are skipped entirely.
!!! tip
diff --git a/docs/how-to/measure-savings.md b/docs/how-to/measure-savings.md
index c3cb70a..5e26660 100644
--- a/docs/how-to/measure-savings.md
+++ b/docs/how-to/measure-savings.md
@@ -2,7 +2,8 @@
context-guru reports what it actually saved through the proxy's `GET /stats` endpoint, backed by
an in-process metrics aggregator. Savings are measured on **message content text** — what the model
-reads — not the JSON envelope, so a control directive like `cacheinject` never looks "worse".
+reads — not the JSON envelope, so a control directive like a `cache_control` breakpoint never looks
+"worse".
## `GET /stats`
@@ -16,14 +17,24 @@ The proxy exposes `GET /stats` with in-process savings rollups. Savings are **to
| `bounces` | how many offloads were re-served (the count behind `wasted_tokens`) |
| `adjusted_saved` | `saved − wasted` — bounce-adjusted, may be negative |
| `top_passthrough` | components that ran but never changed a request: dead weight to drop |
+| `top_discarded` | components whose changes the **writeback layer threw away** — they mutated but never reached the wire. Always worth investigating. |
+| `saved_tokens_unique` / `overcount_ratio` | distinct compactions, and how many times each was re-counted. Prefer the unique figure: the agent re-sends history verbatim every turn, so the cumulative `saved_tokens` is inflated. |
| `mode` | the operating mode these numbers came from: `sync` \| `observe` |
| `sync_enforced` | requests whose forwarded body context-guru actually shaped. **0 in observe mode by construction.** |
!!! tip "Reading top_passthrough"
- A component in `top_passthrough` isn't necessarily broken. `cacheinject` always lands there —
- its savings are provider-side KV-cache hits, invisible to content-token counts. But a
+ A component in `top_passthrough` isn't necessarily broken. `cachesplit` always lands there —
+ its saving is a provider-side KV-cache hit, invisible to content-token counts, and the
+ component itself deliberately always skips (the rewrite is body-level). But a
content-offloader that never fires is a candidate to drop from your pipeline.
+!!! warning "`top_discarded` is never expected"
+ An entry in `top_discarded` means a component ran, mutated the request, and the writeback
+ layer threw the change away before it reached the wire. Unlike `top_passthrough` this is
+ always worth investigating: it is exactly the signature that hid the `cacheinject` bug for
+ two whole benchmark studies, because a mutated-then-discarded component looks byte-identical
+ to a working Reformat. Check the per-component `discarded_changes` count.
+
!!! warning "Enforced vs hypothetical"
Everything above is what context-guru **actually did**. In
[observe mode](operating-modes.md#observe-measure-without-enforcing) nothing is applied,
diff --git a/docs/how-to/recover-context.md b/docs/how-to/recover-context.md
index 6819820..85c3b46 100644
--- a/docs/how-to/recover-context.md
+++ b/docs/how-to/recover-context.md
@@ -54,16 +54,26 @@ The loop reasons over a complete assistant message, but a streaming response arr
on SSE the proxy makes a per-request choice from the request bytes:
- **No marker in `messages`/`system`** → the model has nothing to expand, so the response is
- streamed straight through, byte for byte, with no added latency.
+ streamed straight through, byte for byte, with no added latency. This fast path only started
+ working in [#33](https://github.com/rossoctl/context-guru/pull/33): the marker check tested the
+ *whole* request body, which also matched the `context_guru_expand` tool description context-guru
+ injects itself, so it was unconditionally true and **every** SSE response was buffered. Measured
+ on a fake 1 s SSE upstream, medians of 12 trials: marker-free TTFB **1007 ms → 43 ms**;
+ marker-bearing stayed at 1008 ms, which is correct — it is being inspected.
- **A marker is present** → the response is read in full, reconstructed with `expand.AggregateSSE`,
and inspected. If it is a lone expand call the loop runs; otherwise the buffered bytes are
replayed to the client verbatim.
Buffering costs the client its streaming for that request (time-to-first-byte becomes
time-to-last-byte), which is why the marker test is narrow. It scans **only** `messages` and
-`system` — the model-visible content. It used to scan the whole request body, which also matched the
-`context_guru_expand` tool description we inject ourselves ("…replaced by a `<>` marker"),
-so it was always true and **every** stream was silently buffered (issue #26).
+`system` — the model-visible content, via `expand.HasMarkersInMessages`. Requiring the full marker
+shape would not have been enough on its own: the injected tool description contains the full shape
+too, so **scoping** is the actual fix (issue #26).
+
+!!! success "Restoration does fire through the streaming path"
+ Worth stating, because a fast path that never buffers could equally mean restoration is
+ unreachable. It is not: a real agent invoked restoration through the streaming path —
+ `bounces=1`, 3,372 tokens re-served. The loop works end to end on SSE, not only in tests.
`/stats` reports this directly, counted **once per client request** (not per upstream round, so a
request that drove several expand rounds is one sample): `sse_streamed`, `sse_buffered`,
@@ -104,9 +114,24 @@ The store is the whole reversibility mechanism. It defaults to an in-memory TTL+
refreshed on every read, so a stash an active session keeps touching does not expire under it.
It holds, per session:
-- **Rewind** — `cache_key → original bytes`, what the expand loop resolves.
+- **Rewind** — `cache_key → original bytes`, what the expand loop resolves. Fully evictable: these
+ are the large payloads.
- **Sticky** — the set of content ids already reduced on prior turns (byte-stable output across
turns).
+- **Frozen decisions** — the exact replacement bytes an offloader must replay so an already-cached
+ message stays byte-identical (`cg:frz:` for `mask`/`failed_run`, `cg:res:` for `extract_llm`'s
+ result, `cg:len:` for `apply`'s cache-boundary counter). These are **pinned** against LRU
+ eviction, because losing one is not a cache miss — it flips a message inside the provider's
+ cached prefix and the whole suffix is re-written at 11.5× the read price. The pin is capped at
+ half `max_entries` so one pathological session cannot starve the rewind stashes.
+
+A dropped frozen decision is re-derived where re-derivation is *reproducible* — `mask` and
+`failed_run` qualify, since their replacement is a pure function of `(content, config)`.
+`extract_llm` is deliberately excluded: its replacement is a **sampled** model output (no
+temperature, no seed sent), so re-deriving could splice *different* bytes into a cached prefix,
+which is the exact corruption the repair exists to prevent. Health counters:
+`frozen_hits` / `frozen_misses` / `frozen_dropped` / `frozen_repaired` / `frozen_flips` — see
+[Routes](../reference/routes.md#freeze-replay-health).
!!! warning "No store, no recovery"
Set `store.enabled: false` and offloads become **one-way** — a `store.Nop` is wired in and
diff --git a/docs/reference/config.md b/docs/reference/config.md
index e99c9cb..b869b5a 100644
--- a/docs/reference/config.md
+++ b/docs/reference/config.md
@@ -6,7 +6,7 @@ default pipeline; explicit fields override it.
## Config shape
-The document has four top-level fields (from the `Config` struct in
+The document has six top-level fields (from the `Config` struct in
`config/config.go`):
| Field | Type | Role |
@@ -14,19 +14,36 @@ The document has four top-level fields (from the `Config` struct in
| `preset` | string | Named default pipeline (see [Presets](presets.md)). |
| `pipeline` | `[]string` | Ordered component names — controls **order + enablement**. Overrides the preset's pipeline when present. |
| `components:` | map | Each component's typed config block, handed to its constructor verbatim. |
-| `store` | object | State store options (`enabled`, `ttl_seconds` (default **10000**, sliding), `max_entries`, …). |
+| `store` | object | State store options — see [`store`](#store) below. |
| `mode` | string | Operating mode: `sync` (default) \| `observe`. See [Operating modes](../how-to/operating-modes.md). |
| `observe` | object | Observe-mode tuning; ignored in sync mode. |
+### `store`
+
+| Field | Default | Purpose |
+|---|---|---|
+| `enabled` | `true` | Toggles the state store. `false` wires a `store.Nop`: nothing is stashed, so offloads become **one-way** and must run `marker_mode: off`. |
+| `ttl_seconds` | `10000` | Entry lifetime, and it **slides** — a `Get` refreshes the deadline, so an entry replayed every turn never ages out. Raised from 1800 because Terminal-Bench tasks average ~1975 s of wall clock and run to 4 h, so the old default expired live frozen decisions mid-task. |
+| `max_entries` | `1000` | LRU cap. Frozen-decision keys (`cg:frz:`, `cg:res:`, `cg:len:`) are **pinned** — exempt from LRU eviction, because losing one is cache-destructive rather than merely a miss. The pin is capped at half `max_entries`, and eviction reclaims **expired** entries first (pinned included). |
+| `max_sessions` | `100` | Cap on per-session sticky-id sets. |
+
+The pinned prefixes are a code-level property of the key layout, supplied by their owners via
+`store.Options.PinPrefixes` — not a YAML knob.
+
### `mode`
| Value | Behavior |
|---|---|
| `sync` (default) | Compact inline; the caller waits. Byte-identical to the behavior before modes existed. |
-| `observe` | Forward the request untouched and report what compaction *would* have saved, under `potential_*` / `projected_*` keys. |
+| `observe` | Forward the request untouched and report what compaction *would* have saved, under `potential_*` / `projected_*` keys. The request path never runs the pipeline and skips `expand.Inject` too (a tool declaration is a modification), so byte-identity is **structural**. |
Always explicit — nothing infers it from the rest of the configuration.
+!!! note "An `async` mode is designed but not shipped"
+ A third mode deferring compaction off the request path is implemented on a separate
+ branch and deliberately held pending a benchmark arm establishing a benefit. `sync` and
+ `observe` are the only values the loader accepts.
+
### `observe`
| Field | Default | Purpose |
@@ -42,10 +59,11 @@ Always explicit — nothing infers it from the rest of the configuration.
```yaml
preset: balanced
-pipeline: [format, dedup, failed_run, cmdfilter, cacheinject] # order + enable
+pipeline: [format, dedup, failed_run, cmdfilter, cachesplit] # order + enable
components:
collapse: { max_tokens: 2000, head_lines: 20, tail_lines: 20 }
smartcrush: { min_items: 5, keep_first: 3, keep_last: 2 }
+ cmdfilter: { min_size: 500 }
store: { ttl_seconds: 10000, max_entries: 1000 }
mode: sync # sync | observe
```
@@ -58,13 +76,17 @@ for every component's config block.
| Flag / env | Default | Purpose |
|---|---|---|
-| `--preset` / `PRESET` | `balanced` | Pipeline preset when no `--config`. |
+| `--preset` / `PRESET` | `codesmart` | Pipeline preset when no `--config`. |
| `--config` / `CONFIG` | — | YAML config file (overrides preset). |
| `LISTEN_ADDR` | `:4000` | Listen address. |
| `--openai-upstream` / `OPENAI_UPSTREAM` | `https://api.openai.com` | OpenAI upstream base. |
| `--anthropic-upstream` / `ANTHROPIC_UPSTREAM` | `https://api.anthropic.com` | Anthropic upstream base. |
| `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` | — | Real key injected on forward (gateway mode); empty = pass client auth through. |
+| `CHEAP_MODEL` (+ `CHEAP_MODEL_BASE` / `_KEY` / `_AUTH` / `_PROVIDER`) | — | Dedicated cheap model for the LLM components (`extract_llm`, `summarize`) — the `model.source: config` client. Without it they no-op. |
| `FORCE_MODEL` | — | Overwrite the request `model` (eval-containers uses `EVAL_MODEL`). |
+| `INJECT_EXPAND` | `auto` | Whether the `context_guru_expand` tool is advertised: `auto` (only when the request already declares tools and the store persists) \| `always` \| `never`. |
+| `CACHE_MODE` | `auto` | Cache-aware compaction: `auto` (on when the agent sets its own breakpoints) \| `on` \| `off`. |
+| `MODEL_INFO_URL` / `MODEL_INFO` | LiteLLM map | Source for context-window sizes (used by the fractional triggers). `MODEL_INFO=off` disables the lookup; fractions are then ignored and absolutes apply. |
| `--store` / `STORE` | on | Enable/disable the state store; `--store=false` disables offload reversibility. Wins over the file's `store:` block. |
| `--mode` / `MODE` | `sync` | Operating mode: `sync` \| `observe`. Wins over the file's `mode:`. |
@@ -74,3 +96,4 @@ for every component's config block.
|---|---|
| `CONTEXT_GURU_DEBUG=1` | Logs each tool output's token count + first line. |
| `CONTEXT_GURU_DUMP=` | Appends a before → after JSON record per rewritten message. |
+| `CONTEXT_GURU_CAPTURE=` | Appends the pristine inbound request body to a JSONL file — the input for offline replay. |
diff --git a/docs/reference/presets.md b/docs/reference/presets.md
index d55841a..2edece7 100644
--- a/docs/reference/presets.md
+++ b/docs/reference/presets.md
@@ -7,21 +7,30 @@ taken exactly from the `presets` map in `config/config.go`.
| Preset | Ordered pipeline | When to use |
|---|---|---|
-| `codesmart` | `format` → `dedup` → `failed_run` → `cmdfilter` → `extract_llm` → `extract` → `cacheinject` | **The default.** The SWE-bench-winning cache-aware config: structural offloaders + a cheap-model relevance-trimmer (`extract_llm`, routed to `CHEAP_MODEL`, gated so most turns make no model call) + deterministic `extract`. `extract_llm` no-ops (→ deterministic) when no cheap model is configured. |
-| `codesafe` | `format` → `dedup` → `failed_run` → `cmdfilter` → `extract` → `collapse` → `cacheinject` | `codesmart` minus the LLM pass — **deterministic-only, zero model calls by policy**. The safe control / the choice when you don't want an LLM on the hot path. |
+| `codesmart` | `format` → `dedup` → `failed_run` → `cmdfilter` → `extract_llm` → `extract` → `cachesplit` | **The default.** The SWE-bench-winning cache-aware config: structural offloaders + a cheap-model relevance-trimmer (`extract_llm`, routed to `CHEAP_MODEL`, gated so most turns make no model call) + deterministic `extract`. `extract_llm` no-ops (→ deterministic) when no cheap model is configured. |
+| `codesafe` | `format` → `dedup` → `failed_run` → `cmdfilter` → `extract` → `collapse` → `cachesplit` | `codesmart` minus the LLM pass — **deterministic-only, zero model calls by policy**. The safe control / the choice when you don't want an LLM on the hot path. |
| `off` | *(empty)* | Passthrough — no components. The baseline / A-B control. |
-| `safe` | `format` → `cacheinject` | Lossless only: repack JSON compactly and add `cache_control`. Zero risk of dropping content. |
-| `balanced` | `format` → `dedup` → `failed_run` → `cmdfilter` → `cacheinject` | Lossless repack + conservative offloads (dedupe, drop superseded/failed runs, filter command noise) + cache. |
-| `aggressive` | `format` → `dedup` → `failed_run` → `cmdfilter` → `smartcrush` → `extract` → `extract_llm` → `cacheinject` | `balanced` plus `smartcrush` (crush long homogeneous arrays), deterministic `extract` (noise collapse), and `extract_llm` (cheap-model relevance trim) for deeper savings. |
-| `coding` | `format` → `skeleton` → `cmdfilter` → `cacheinject` | Coding agents: `skeleton` reduces big source-file reads to their structure via tree-sitter. |
-| `mcp` | `format` → `smartcrush` → `cacheinject` | Tool/MCP servers returning long homogeneous JSON arrays (list endpoints, search hits). |
-| `agent` | `format` → `dedup` → `failed_run` → `mask` → `extract` → `cacheinject` | Long agentic sessions (e.g. Claude Code on SWE-bench) where re-sent tool outputs dominate cost. `mask` is the biggest lever — ~27% content-token savings with no task-reward loss (see [Benchmarks](../RESULTS.md)). |
+| `safe` | `format` → `cachesplit` | Lossless only: repack JSON compactly and split the volatile system tail so the shared prefix stays cacheable. Zero risk of dropping content. |
+| `balanced` | `format` → `dedup` → `failed_run` → `cmdfilter` → `cachesplit` | Lossless repack + conservative offloads (dedupe, drop superseded/failed runs, filter command noise) + the cache split. **Not recommended for agentic traffic** — it omits `mask`, the biggest lever there. |
+| `aggressive` | `format` → `dedup` → `failed_run` → `cmdfilter` → `smartcrush` → `extract` → `extract_llm` → `cachesplit` | `balanced` plus `smartcrush` (crush long homogeneous arrays), deterministic `extract` (noise collapse), and `extract_llm` (cheap-model relevance trim) for deeper savings. |
+| `coding` | `format` → `skeleton` → `cmdfilter` → `cachesplit` | Coding agents: `skeleton` reduces big source-file reads to their structure via tree-sitter. |
+| `mcp` | `format` → `smartcrush` → `cachesplit` | Tool/MCP servers returning long homogeneous JSON arrays (list endpoints, search hits). |
+| `agent` | `format` → `dedup` → `failed_run` → `mask` → `extract` → `extract_llm` → `cachesplit` | Long agentic sessions (e.g. Claude Code on SWE-bench) where re-sent tool outputs dominate cost. `mask` is the biggest lever — ~27% content-token savings with no task-reward loss (see [Benchmarks](../RESULTS.md)). |
+| `general` | `format` → `toon` → `dedup` → `failed_run` → `cmdfilter` → `mask` → `extract` → `extract_llm` → `collapse` → `cachesplit` | The recommended all-round pipeline: the reward-neutral levers of `agent` plus the situational shrinkers (`toon` / `cmdfilter` / `collapse`) that cost nothing when they don't fire. |
| `summarize` | `summarize` | Long trajectories where the transcript itself is the cost. **Runs alone** — it restructures the whole transcript (changes the message count), so no other component's in-place edits race the rebuild. |
!!! tip "Order matters"
Components run in pipeline order: lossless repack first, then offloads
- (old-then-large), with `cacheinject` last so it keeps the reduced prefix
- cacheable.
+ (old-then-large), with `cachesplit` last because it edits the top-level
+ `system` array rather than `messages`.
+
+!!! info "`cachesplit`, not `cacheinject`"
+ Every preset that touches caching carries [`cachesplit`](../components/cachesplit.md),
+ which enables the measured volatile-tail split. Breakpoint *placement*
+ ([`cacheinject`](../components/cacheinject.md)) is in **no** preset: its marks only
+ began reaching the provider in
+ [#36](https://github.com/rossoctl/context-guru/pull/36), and placement has never been
+ shown to help. Add it by hand if you want to run the placement study.
Not sure which to pick? See [Choose a preset](../how-to/choose-a-preset.md).
Every component's config lives in [Components](../components.md).
diff --git a/docs/reference/routes.md b/docs/reference/routes.md
index 4f0b532..39786b0 100644
--- a/docs/reference/routes.md
+++ b/docs/reference/routes.md
@@ -8,32 +8,135 @@ The proxy serves both provider dialects on one port (default `:4000`).
|---|---|
| `POST /openai/v1/chat/completions` | OpenAI chat dialect — runs the pipeline, forwards to the OpenAI upstream. |
| `POST /anthropic/v1/messages` | Anthropic Messages dialect — runs the pipeline, forwards to the Anthropic upstream. |
+| `POST /compact` | Stateless compaction: run the pipeline and return the rewritten body, no upstream call. `?provider=anthropic` switches dialect; `?preset=` / `x-context-guru-pipeline` override the pipeline; `?cache=on\|off\|auto` overrides cache-awareness. |
| `GET /healthz` | Liveness check. |
-| `GET /stats` | Savings rollups (token-weighted `Σ saved / Σ before`, plus `wasted_tokens`/`bounces` and per-component breakdown). |
-| `GET /expand?id=` | Recover an offloaded original by its `<>` id. |
+| `GET /stats` | Savings rollups and health counters — see below. |
+| `GET /expand?id=` | Recover an offloaded original by its `<>` id. Scoped to the caller's session. |
-### `/stats` freeze-replay fields
+!!! note "`POST /compact` (compaction-service mode)"
+ The [llm-d compaction service example](../examples/llm-d-service.md) uses this route with
+ the store disabled and `marker_mode: off`, so the returned body is clean, marker-free and
+ directly usable. See [Quickstart: Compaction service](../get-started/quickstart-compaction.md).
+
+## `GET /stats`
+
+Fields are only ever **added** to this payload (harnesses in `deploy/harbor` parse it), so a
+consumer that reads by key keeps working. The table below is the `Snapshot` struct in
+`metrics/metrics.go`, complete.
+
+### Savings
+
+| Field | Meaning |
+|---|---|
+| `requests` | Enforced requests aggregated. |
+| `tokens_before` / `tokens_after` | Content-token totals before and after the pipeline. |
+| `saved_tokens` / `savings_pct` | `before − after`, and the **token-weighted** ratio (`Σ saved / Σ before`), not a mean of per-request percentages. |
+| `wasted_tokens` | Content offloaded then re-served via `expand` — a premature offload. |
+| `bounces` | How many expand events produced that waste. |
+| `adjusted_saved` | `saved − wasted`. May be negative. |
+| `components` | Per-component rollup (see below). |
+| `top_passthrough` | Components that ran but never *changed* a request — dead weight. A component that mutated without saving content tokens (`cachesplit`, `cacheinject`) is **not** listed here. |
+| `top_discarded` | Components whose changes the **writeback layer threw away** at least once. Any entry needs investigating: the component ran, mutated, and had no effect on the wire. |
+
+Per-component (`components.` and `potential_components.`):
+
+| Field | Meaning |
+|---|---|
+| `runs` | Times the component ran. |
+| `acted` | Runs that actually saved tokens. |
+| `mutated` | Runs that changed the request at all — may save 0 content tokens. |
+| `reverted` | Runs the pipeline rolled back (error, panic, or grew the request). |
+| `saved_tokens` | **Cumulative** — re-counted every turn the compaction re-appears. |
+| `saved_tokens_unique` | **Unique** — each distinct compaction counted once, deduped by content key. |
+| `overcount_ratio` | `saved_tokens / saved_tokens_unique`. ~1.0 is honest; a large value means the cumulative figure is inflated by the agent re-sending history verbatim. |
+| `duration_ms` | Cumulative wall time this component spent on the hot path. |
+| `discarded_changes` | Changes the writeback layer threw away, attributed back to this component. |
+
+!!! warning "Cumulative is not unique"
+ `saved_tokens` counts the same compaction again on every later turn that carries it. A
+ figure like "4.8M tokens saved" is a *cumulative* total; the unique figures behind the
+ Terminal-Bench and SWE-bench runs are **234,119 tokens behind 103 markers** and
+ **15,457 behind 29** — 21× and 8× smaller respectively. Quote
+ `saved_tokens_unique`, and check `overcount_ratio` before citing either.
+
+### Context-guru's own cost
+
+| Field | Meaning |
+|---|---|
+| `llm_calls`, `llm_input_tokens`, `llm_output_tokens` | The cheap-model spend context-guru's *own* components incurred (`extract_llm`, `summarize`). Separate from the agent's spend; priced externally. |
+| `cg_added_ms_avg` | Mean ms context-guru added per request (normalize + pipeline + writeback). |
+| `upstream_ms_avg` | Mean provider round-trip on the active path. |
+| `upstream_ms_avg_bypassed` | Same on `x-context-guru-bypass` requests — the baseline for a with/without latency comparison. |
+
+### SSE streaming health
+
+Buffering a stream is the one thing that stops it being a stream, so it is counted. All four
+fields count **once per client request**, not per upstream round: a request that drove several
+expand rounds waited for all of them.
-Fields are only ever **added** to this payload (harnesses in `deploy/harbor` parse it), so
-a consumer that reads by key keeps working.
+| Field | Meaning |
+|---|---|
+| `sse_streamed` | Streaming responses passed straight through — the fast path. |
+| `sse_buffered` | Streaming responses read in full before the client saw a byte, because the request carried a marker that might produce an expand call. |
+| `sse_buffered_pct` | `buffered / (streamed + buffered) × 100`. |
+| `sse_ttfb_ms_avg` | Real time-to-first-byte, streamed-through requests only. |
+| `sse_ttfb_ms_avg_buffered` | Time-to-**last**-byte by construction — a buffered response is read in full before the client is written to, so its first byte cannot precede the buffer completing. Read it as "what buffering cost these requests", **not** as a latency comparable to `sse_ttfb_ms_avg`. |
+
+A high `sse_buffered_pct` on traffic that never expands is the regression to watch: the marker
+check used to match the expand tool's own description and so buffered **every** stream.
+
+### Freeze-replay health
+
+The cache-**write** cost line. A frozen decision replayed (`frozen_hits`) keeps an
+already-cached message byte-identical. One the store **drops** would flip that message's
+representation inside the provider's cached prefix and force the whole suffix to be re-written
+at 11.5× the read price — unless it is re-derived.
| Field | Meaning |
|---|---|
| `frozen_hits` | Replay lookups that found a stored decision and re-sent the same bytes. |
-| `frozen_misses` | Replay lookups that found nothing. **Dominated by the ordinary "not compacted yet" case** — it is a lookup counter, not an error counter. Read `frozen_dropped` for harm. |
-| `frozen_dropped` | Stored decisions the store actually **lost** (TTL expiry or eviction). Each is a chance for an already-cached message to flip representation. Counted per drop *event*. |
-| `frozen_repaired` | Dropped decisions later restored, so a replay can land again. |
+| `frozen_misses` | Replay lookups that found nothing. **Dominated by the ordinary "never frozen yet" case** — it is a lookup counter, not an error counter. Read it beside `frozen_dropped`, not instead of it. |
+| `frozen_dropped` | Stored decisions the store actually **lost** (TTL expiry or the pin cap). Counted per drop *event*. |
+| `frozen_repaired` | Dropped decisions re-derived, so a replay lands again. Only `mask` and `failed_run` qualify — `extract_llm` is deliberately excluded (its replacement is a *sampled* model output, so re-deriving could splice different bytes into the cached prefix). |
| `frozen_flips` | `frozen_dropped − frozen_repaired` — outstanding losses, i.e. drops that plausibly cost a suffix cache-write. **Should be 0.** |
-A healthy long-horizon run shows `frozen_hits` climbing with turn count and
-`frozen_dropped` at 0; a rising `frozen_dropped` means decisions are dying mid-session
-(TTL too short for the task, or the entry cap too small for the session's working set).
+A healthy long-horizon run shows `frozen_hits` climbing with turn count and `frozen_dropped` at
+0; a rising `frozen_dropped` means decisions are dying mid-session (TTL too short for the task,
+or the entry cap too small for the session's working set).
-!!! note "`POST /compact` (compaction-service mode)"
- The [llm-d compaction service example](../examples/llm-d-service.md) adds a
- stateless `POST /compact` route: it runs the pipeline and returns the
- rewritten body directly (`200` + JSON) with no upstream call, no store, and
- no markers. See [Quickstart: Compaction service](../get-started/quickstart-compaction.md).
+### cmdfilter attribution
+
+| Field | Meaning |
+|---|---|
+| `cmdfilter_families` | Per command family (`builds` / `tests` / `iac` / `pkg` / `net` / `other`): `acts`, `saved_tokens`, `saved_tokens_unique`. |
+| `cmdfilter_filters` | The same, per individual filter — which filters actually earn their place. |
+| `cmdfilter_selector_misses` | Output shapes that matched **no** filter, frequency-ranked. The backlog of filters worth writing. Bounded at 200 distinct selectors, first-seen wins. |
+
+### Operating mode
+
+| Field | Meaning |
+|---|---|
+| `mode` | The configured operating mode: `sync` \| `observe`. |
+| `sync_enforced` | Requests whose forwarded body context-guru actually shaped. **0 in observe mode by construction** — the machine-readable form of "context-guru did not modify requests". |
+
+Observe mode's results are **hypotheticals** and live under keys that never collide with an
+enforced metric, so a consumer cannot sum one into a real saving even by accident. All zero
+outside observe mode.
+
+| Field | Meaning |
+|---|---|
+| `observe_notice` | The banner. Present whenever hypotheticals are reported. |
+| `observe_hypothetical_requests` | Requests observed off-path. |
+| `actual_baseline_tokens` | What the agent really sent. Actual, not hypothetical. |
+| `projected_optimized_tokens` | What it would have sent under this pipeline. |
+| `potential_saved_tokens` / `potential_savings_pct` | The difference, and its ratio. |
+| `potential_components` | Per-component hypothetical contributions (same shape as `components`). |
+| `potential_overhead_ms_avg` | What compaction *would* have added per request — measured off-path, so it is what `sync` would cost, not what `observe` costs. |
+| `observe_llm_notice` | Warns that `llm_calls` / `llm_*_tokens` in observe mode are the cost of **measuring** off-path, not of enforcing. The spend is real (not hypothetical), so it stays where cost tooling reads it, labelled. |
+
+`cg_added_ms_avg` and the `llm_*` fields deliberately **do** accumulate in observe mode: they
+are real measurements and real spend. Zeroing them would hide a true number rather than protect
+anyone. See [Operating modes](../how-to/operating-modes.md).
## Per-request headers
diff --git a/docs/results/components.md b/docs/results/components.md
index 666c76f..0ace06d 100644
--- a/docs/results/components.md
+++ b/docs/results/components.md
@@ -21,6 +21,12 @@ All numbers are from the matched 50-task run (see [comparison.md](comparison.md)
## context-guru — pipeline `[format, dedup, failed_run, cmdfilter, extract_llm, extract, cacheinject]`
+!!! note "This is the pipeline as this run used it"
+ `codesmart` now ends in **`cachesplit`**, not `cacheinject` — breakpoint placement was removed
+ from every preset in [#36](https://github.com/rossoctl/context-guru/pull/36). The name is left
+ as-is here because it is what the run actually ran. See
+ [Presets](../reference/presets.md) for the current compositions.
+
Two type-enforced kinds: **Reformat** (lossless repack, no stash) and **Offload**
(lossy-but-reversible: leaves a `<>` marker + stashes the original, recoverable
via the `context_guru_expand` tool). Every component is fail-open isolated — an error or
@@ -55,13 +61,22 @@ Run: **0 acts** (cache-aware), but still scans every run-like output (~6.7 s tot
costliest *deterministic* detection).
### 4. `cmdfilter` (Offload)
-Declarative DSL filters keyed on a command output's first line (23 shipped, e.g. `pytest`,
+Declarative DSL filters keyed on the shape of a command output's leading lines (e.g. `pytest`,
`make`, `gradle`, `terraform-plan`, `pulumi`): strip blank/`PASSED`/progress lines, cap length,
keep failures.
> **Real example** (pytest session): `1140 → 1068 tok` — passing/blank noise stripped,
> failures + warnings kept verbatim.
Run: 3 acts.
+!!! note "The filter set has since tripled"
+ This run had **3** filters matched on the output's *first* line only.
+ [#42](https://github.com/rossoctl/context-guru/pull/42) took it to **24** filters, rewrote every
+ selector to match output shape over six leading lines, and added the per-family `/stats` ledger.
+ The 3-act figure is not a ceiling for the current set — but nor is 24 filters a promise of 8×
+ the savings: on the Terminal-Bench dump the four filters *predicted* to matter (`pulumi`,
+ `terraform-plan`, `xcodebuild`, `gradle`) fired zero times, and `apt` + `gcc` carried ~73% of
+ the savings instead. Re-measure rather than extrapolate.
+
### 5. `extract_llm` (Offload — the relevance-aware LLM pass)
A cheap **haiku**-class model writes a sandboxed **Starlark program** that trims *one*
tool output to what the agent needs next. It may delete or regex-rewrite, must preserve
@@ -114,8 +129,10 @@ cache hits across turns. No content change.
So the **97.8% cache-hit rate is not attributable to this component.** That rate is
claude-code's own breakpoints, which it sets on every request and which the proxy
- forwarded untouched. Fixed in #32; a re-run must re-measure this row rather than carry
- the number forward.
+ forwarded untouched. Fixed in [#36](https://github.com/rossoctl/context-guru/pull/36); a
+ re-run must re-measure this row rather than carry the number forward. In the meantime
+ placement has been removed from every preset, so a fresh `codesmart` run has no
+ `cacheinject` row at all — it has a `cachesplit` one.
---
diff --git a/mkdocs.yml b/mkdocs.yml
index ec85a19..87914c4 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -117,6 +117,7 @@ nav:
- Reformat:
- format: components/format.md
- toon: components/toon.md
+ - cachesplit: components/cachesplit.md
- cacheinject: components/cacheinject.md
- "Offload (reversible)":
- skeleton: components/skeleton.md