feat(cmdfilter): port rtk's filter coverage (3 -> 24) and fix three DSL loss-typing bugs

[OsherElhadad]

Closes #29

cmdfilter shipped a 3-filter starter set against rtk's 63. The DSL engine was already a faithful
port, so the gap was coverage — but the port is not a copy, and the interesting work turned out to be
in the two things measurement contradicted.

The three engine bugs, fixed with tests

1. truncate_lines_at never recorded loss. It ran before loss was initialized (dsl.go:192-200
   vs :202), so a real intra-line cut reported LossNone and cmdfilter emitted no recovery
   hint for a real loss. Now types as LossWhole — an intra-line cut is non-contiguous by nature,
   since every long line loses its own tail.
2. truncate_lines_at cut silently. A mid-line cut with no marker reads as corrupted output to a
   model. Now appends ..., sized to fit inside the cap so the line never grows.
3. recoveryHint collapsed LossTail and LossWhole. Both got the same "call
   context_guru_expand" text, making a cheap partial recovery look like an expensive whole-blob
   one and pushing the agent to the costly path. Now distinct: a tail cut names the cut point.

The port: 24 filters, every selector rewritten

rtk matches a shell command (^terraform\s+plan); a proxy only ever sees the output. Every
selector is rewritten to an output-shape signature. Copied verbatim, rtk's regexes compile fine and
never fire — so TestEveryBuiltinFilterHasTestsAndRoutes asserts each filter's own test input
actually routes to that filter. That test is what makes the rewrite verifiable instead of hopeful,
and it caught every shadowing regression below.

Ported (22): pytest npm-install make gradle xcodebuild gcc swift-build dotnet-build
turbo nx terraform-plan terraform-init pulumi liquibase ssh ping rsync
bundle-install poetry-install composer-install uv-sync brew-install quarto-render.
Filters whose output is indistinguishable are merged (terraform/tofu; the five pulumi subcommands).

Tier 3 not ported, with reasons recorded in docs/components/cmdfilter.md: the ~24
width-cap-only filters, the 11 blank-line-only linter filters, spring-boot, filter_stderr, rtk's
86 command-detection rules, and its ~50 native Rust filters.

Every success-collapse rule carries an unless guard (rtk ships 9 of 11 unguarded), each with an
explicit negative test: a build with a warning and a success marker must not collapse. In a proxy
the agent cannot re-run the command to find what a bare collapse swallowed.
TestEveryMatchOutputRuleIsGuarded fails the build if one is added without a guard. dotnet-build
needed care: dotnet prints 0 Error(s) on success, so a guard on the word "error" would never let it
collapse — it guards on the diagnostic form (error CS1002).

Also: shared cap budget classes by signal density instead of 24 hand-picked max_lines; a
500-byte size floor (rtk's MIN_TEE_SIZE); load-time guardrails (duplicate names rejected, inline
tests executed at load); a bounded selector-miss ledger; per-family metrics in /stats.

Two things measurement contradicted

I replayed the shipped selectors over recorded proxy traffic before trusting the plan. Both findings
changed the code.

1. The issue's Tier-1 priorities did not survive contact with the data. On a Terminal-Bench dump,
pulumi, terraform-plan, xcodebuild and gradle fired zero times. What dominated were two
shapes rtk has no filter for: apt/dpkg boilerplate (584 outputs, ~1.0 MB — the largest reachable
family) and gcc's <file>: In function 'main': header. I added both; apt gets 76% compression and
went on to carry ~73% of all savings in the live run. The IaC filters stay — correct, tested, free
when inert — but the docs now say plainly that a filter set's value comes from the workload, not its
size. That is what the miss ledger is for.

2. The one-line selector was a real bug in my own port. 112 pytest runs (311 KB) matched nothing
because the agent harness prepends its own preamble (Exit code 1, Internet access disabled) or
the report opens with a bare ERROR path::test — so pytest's session banner is never line 1. A
one-line selector ties a filter's reach to the agent's output framing rather than to the tool
that produced the output. selectorKey now spans the first 6 non-empty lines, with (?m) so ^/$
anchor per line:

workload	claimed before	claimed after
SWE-bench captured traffic (520 eligible outputs)	13 (2.5%)	124 (23.8%)
Terminal-Bench dump (223 eligible outputs)	140 (62.8%)	172 (77.1%)

Widening it made generic filters shadow specific ones — exactly the ordering hazard #29 flagged — and
the routing guardrail caught every case. Resolved with explicit priority: tool banners at 20,
gcc at -10 (its selector is the generic "some compiler said something" shape that also occurs
inside make/swift/dotnet output, so it is the last resort, not the first match).

A false positive caught in a live run: swift-build's ^Compiling claimed Cython output and
stripped its Compiling x.pyx because it changed lines — it fired 80× on a C build task with almost
no unique saving, visible directly in the new per-filter ledger. A generic verb is not a tool
signature. Fixed to require Swift identity, with a regression test asserting Cython and cargo output
are not claimed.

Benchmarks

Terminal-Bench, build-heavy tasks, off baseline vs codesmart, matched per task. Reward and
steps are the numbers that matter here — this is the failure mode that sank rtk on Terminal-Bench.

task	reward off → cs	steps off → cs	norm cost off → cs
build-pmars	1.0 → 1.0	32 → 34	$0.461 → $0.503
build-cython-ext	0.0 → 1.0	70 → 59	$1.177 → $1.213
compile-compcert	1.0 → 0.0	55 → 58	$1.602 → $1.128
aggregate	2/3 = 2/3	52.3 → 50.3 (−3.8%)	$3.240 → $2.843 (−12.3%)

Cache-hit 95.97% → 97.43%; cache-write 393k → 239k (−39%) — the lever that actually mattered
here. Total including context-guru's own LLM cost: $2.862 (−11.7%). cmdfilter: 128 acts / 153 runs,
0 reverts, 0 expand bounces, ~2 ms/request.

Reward is net neutral (2 = 2), not improved — and I am claiming neither flip. Both went in
opposite directions, and in each case the losing trial was an exception: true with agent_wall_s
pinned exactly at its budget (1800.0 s and 4800.0 s). Both flips are wall-clock timeouts, not
information loss. The cython "win" and the compcert "loss" are the same artifact.

The one real concern, and it is not the rtk failure mode. compcert's codesmart trial burned +17%
agent wall (4089 s → 4800 s) and hit its budget. Steps went down overall and aggregate reward held,
so this is not the "agent lost information → more steps" spiral that sank rtk. It is added latency:
cg_added_ms_avg 5 ms → 145 ms, upstream 12.9 s → 15.6 s/req — dominated by extract_llm
(17.8 s across 2 calls, 0 tokens saved), not cmdfilter. On a task already near its wall clock that
was enough to push it over. That belongs to #28, which independently concluded extract_llm is
net-negative on caching backends; it is not a reason to back off these filters.

Live per-family ledger (the new /stats fields):

cmdfilter: 128 acts, 0 reverted
  pkg    (apt)          — the largest reachable family, ~73% of live savings
  tests  (pytest)
  builds (swift-build ← the Cython false positive, now fixed)

Honest caveats, stated plainly:

• The SWE-bench control arm is unusable and I am not reporting it as a result. Another agent's
  proxy already held the port I had been assigned, so my first attempt was measuring through
  someone else's config; after moving to a verified-free port, 2 of 3 codesmart trials aborted with
  API Error: ConnectionRefused from proxy restarts. One clean trial is not a control. What it did
  give me is the capture file that exposed the selector bug — the more valuable outcome.
• n=1 per task. All three rewards carry single-trial noise; with two of the three decided by a
  timeout, the reward column should be read as "no regression detected", not as a measurement.
• This ran the PRE-FIX binary. The selector widening (2.5% → 23.8% match rate on SWE traffic) and
  the Cython false-positive fix landed after this run, so the merged version should produce more
  output than these numbers show.
• Unique savings are small in absolute terms (~3.6k tokens). On a ~96% cached agent this is a
  rounding error against the bill, consistent with the existing finding that cost tracks steps, not
  unique tokens. The case for cmdfilter is that it is free — no LLM, ~2 ms, cache-safe — not that
  it moves the bill on its own.
• The TB arm ran the pre-fix binary, so its cmdfilter numbers understate the merged version
  (the selector fix and the Cython fix both landed after it started).

Licensing

THIRD-PARTY-NOTICES (new) names rtk, github.com/rtk-ai/rtk, Apache-2.0, the
Copyright 2024 rtk-ai and rtk-ai Labs notice, and states that filter definitions and the DSL
pipeline are "adapted from, with modifications" — enumerating them. Apache-2.0 §4 applies to the
filter TOMLs as copyrightable expression; the existing prose credit in dsl.go is good practice but
not sufficient for a bulk port.

Verification

CGO_ENABLED=1 go build -tags cg_skeleton ./... · go test -tags cg_skeleton ./... ·
go test -race · make lint · gofmt -l — all clean. /stats additions are additive only, with
a test asserting the keys deploy/harbor/*.py parses still exist.

Docs: docs/components/cmdfilter.md (full filter table, what each preserves, the Tier-3 rationale,
the two measurement findings), docs/components/dsl.md (loss typing, cap classes, priority),
docs/how-to/custom-dsl-filter.md (both selector traps), THIRD-PARTY-NOTICES.

[OsherElhadad]

Orchestrator verification — the benchmark reading is correct, and the caveats are real

I re-derived everything from /tmp/i29-runs/ row files rather than taking the table on trust.

Both reward flips confirmed as wall-clock timeouts, not information loss

This is the load-bearing claim, and it checks out exactly as stated:

task	reward	steps	exception	agent_wall_s
build-cython-ext	0.0 → 1.0	70 → 59	True → False	1800.0 → 1386.1
compile-compcert	1.0 → 0.0	55 → 58	False → True	4089.4 → 4800.0
build-pmars	1.0 → 1.0	32 → 34	False → False	776.7 → 747.4

Each losing trial is an exception: true with agent_wall_s pinned exactly at its budget (1800.0 and 4800.0). Declining to claim either flip is the right call — they are the same artifact pointing opposite directions, and treating the cython "win" as a gain while explaining away the compcert "loss" would have been the tempting and wrong move.

Aggregate figures reproduce

	off	codesmart
billed cost	$3.240	$2.843 (−12.3%)
cache-read	9,379,052	9,183,987
cache-write	393,402	239,210 (−39.2%)
cache-hit	95.97%	97.43%

Cache-write −39% is the result that matters on this benchmark, and it is the lever the two-benchmark study identified as decisive on long-horizon contexts. Worth noting it lands in the same direction and similar magnitude as #35's async arms (−45% SWE / −39% TB), from a completely different mechanism.

The SWE control is worse than "unusable" — confirm it is excluded

astropy-12907  reward 0.0 steps 2  exception True   →  None (exception True)
astropy-14365  reward 0.0 steps 2  exception True   →  1.0 (34 steps)
astropy-8707   reward 0.0 steps 2  exception True   →  None (exception True)

The off arm is $0.000 total, zero cache-read, all three trials 2-step aborts. There is nothing there. The one codesmart trial that ran has no counterpart. Correctly reported as not-a-result; I confirmed no delta is quoted from it anywhere in the PR body.

On the extract_llm attribution

The claim that compcert's +17% wall came from extract_llm (17.8 s across 2 calls, 0 tokens saved) rather than cmdfilter is consistent with two independent findings: #34 concluded the component is net-negative on caching backends and recommends disabling it there, and #35 measured it making zero calls on Terminal-Bench. Three separate workstreams converging on the same component is worth more than any one of them.

That said, this does mean the merged codesmart preset would still carry the latency that pushed compcert over its budget. That is a preset-composition decision for the merge, not a defect in this PR — but it should not be lost: shipping #37 and #34 together, and acting on #34's recommendation, is what actually removes the observed harm.

What to fix before merge

The caveats are all stated in the PR body, which I've verified. The one thing I'd add: make it explicit that this run used the pre-fix binary, so the selector widening (2.5% → 23.8% SWE match rate) and the Cython false-positive fix are not reflected — the merged version should produce more output than these numbers, and a future reader comparing against a re-run needs to know that.

Verdict from my side: reward is net neutral with no regression detected, the cost and cache-write improvements are real, and the methodology disclosure is honest. The apt filter carrying ~73% of savings makes it the highest-stakes one — I've asked the independent reviewer to hunt for more cases of the ^debconf: class of over-broad pattern that your own safety test already caught once.

[OsherElhadad]

Superseded by #42 — same 7 commits rebased onto main after #33/#36/#40 merged. Could not rebase in place because branch protection blocks force-push to this branch.

The components/component.go conflict needed a manual merge, not the keep-both resolution that works for metrics/metrics.go: #36 adds an ExistingBreakpoints field to Ctx, while this branch adds a FilterStats field and a FilterStatsSink interface immediately after the struct. Concatenating the two sides drops the struct field into the interface body and yields invalid Go. Resolved with both fields inside Ctx and the interface separate; full gates re-run green.