Skip to content

fix(observe): gate the filter ledger behind Ctx.Stats() so observe records nothing enforced - #54

Merged
OsherElhadad merged 1 commit into
mainfrom
fix/observe-filter-leak
Aug 10, 2026
Merged

fix(observe): gate the filter ledger behind Ctx.Stats() so observe records nothing enforced#54
OsherElhadad merged 1 commit into
mainfrom
fix/observe-filter-leak

Conversation

@OsherElhadad

Copy link
Copy Markdown
Collaborator

Closes #46.

The leak

cmdfilter called c.FilterStats with no mode check, so an observe-only run — zero enforced requests, sync_enforced: 0 — still reported real-looking entries:

"cmdfilter_families": {"tests": {"acts": 1, "saved_tokens": 1787}},
"cmdfilter_filters":  {"pytest": {"acts": 1, "saved_tokens": 1787}},
"cmdfilter_selector_misses": [ ]

Those tokens were never saved — nothing was forwarded.

Why these three fields, and not the two observe deliberately shares

#43 correctly keeps two enforced-namespace fields populated in observe mode:

  • cg_added_ms_avg — a true measurement of the enforced path, and its reading of ~0 is observe's headline result. Zeroing it would hide a real number.
  • llm_calls/llm_*_tokens — real money actually spent measuring off-path. Relabelling real spend as potential_* would be a worse lie. Labelled via observe_llm_notice.

These three are different: no mode label, no potential_* counterpart. A consumer cannot tell them from real savings. That is exactly the failure #31 named as its primary correctness risk — a mislabelled hypothetical is worse than no number, because it silently inflates the product's own headline claim.

The fix is an accessor, not a call-site check

func (c *Ctx) Stats() FilterStatsSink {
	if c == nil || c.Mode == ModeObserve {
		return nil
	}
	return c.FilterStats
}

Gating at Ctx rather than at cmdfilter's two call sites is deliberate: a component author reaching for c.FilterStats has no reason to think about operating modes, so the next sink added to Ctx would reproduce this bug exactly. An accessor makes the safe path the only convenient one. The field's doc comment now says to read it through Stats().

Provenance

This is new leakage from #42 × #43 composing — neither PR's own review could see it, because the stats sink and observe mode landed independently and each was correct in isolation. Found by the integration review of the five merges, which is the class of defect per-PR review structurally cannot catch.

Test

TestObserveModeDoesNotRecordFilterStats drives the component in both modes and asserts sync does record — otherwise a broken sink would pass it for the wrong reason.

Verified non-vacuous. With the gate removed it reproduces the reported symptom:

--- FAIL: TestObserveModeDoesNotRecordFilterStats
    mode=observe: 1 ledger events (acts=1 misses=0), wantRecord=false

Gates

go build -tags cg_skeleton ./... · go test -tags cg_skeleton ./... · go test -race ./components/... ./proxy/... · gofmt -l · go vet — all clean.

Note on the remaining integration findings

#45 (the volatile-tail split is a silent no-op on Bedrock Converse), #47 (pin-budget exhaustion reopening the TailOnly fail-open on /compact only) and #48 (the selector-miss ledger bounding key count but not key size) are still open and unaddressed by this PR.

…cords nothing enforced

cmdfilter called c.FilterStats with no mode check, so an observe-only run --
zero enforced requests, sync_enforced 0 -- still reported real-looking
cmdfilter_families, cmdfilter_filters and cmdfilter_selector_misses entries.
Those tokens were never saved: nothing was forwarded.

Unlike the two fields observe deliberately shares (cg_added_ms_avg, a true
measurement of the enforced path whose reading of ~0 IS the headline result; and
context-guru's own model spend, which is real money and labelled by
observe_llm_notice), these three sit in the enforced namespace with no mode label
and no potential_* counterpart. A consumer cannot tell them from real savings.
That is the failure #31 named as its primary correctness risk -- a mislabelled
hypothetical is worse than no number, because it silently inflates the product's
own headline claim.

The gate is a Ctx.Stats() accessor rather than a check at cmdfilter's two call
sites. A component author reaching for c.FilterStats has no reason to think about
operating modes, so the next sink added to Ctx would reproduce this exactly; an
accessor makes the safe path the only convenient one. New leakage from #42 and
#43 composing -- neither PR's own review could see it, because the stats sink and
observe mode landed independently.

The test asserts sync DOES record, so it proves the gate rather than a dead sink,
and fails without the gate with the reported symptom (acts=1 in observe mode).

Signed-off-by: Osher-Elhadad <Osher.Elhadad@ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

fix(observe): cmdfilter's ledger leaks hypotheticals into unlabelled /stats fields

2 participants