Skip to content

fix(cmdfilter): the selector-miss ledger bounds key count but not key size — multimodal traffic fills it with base64 blobs #48

Description

@OsherElhadad

Found by integration review, replaying real captured traffic through the merged pipeline.

Problem

maxMissKeys = 200 bounds the number of selector-miss entries, not their size. selectorKey runs on the serialized content block, so on multimodal traffic it captures image payloads. Replaying capture-tb.jsonl filled the top 25 miss slots with base64 PNG data:

[{"type":"image","source":{"type":"base64","data":"iVBOR…

Two consequences:

  1. The ledger stops doing its job. feat(cmdfilter): port rtk's filter coverage (3 -> 24) and fix three DSL loss-typing bugs #42 added it to answer "which filter is worth writing next" (after rtk's parse_failures table). On multimodal traffic that answer is 200 near-identical image blobs — the actionable misses (Exit code 1, source listings) are crowded out.
  2. 200 multi-KB keys sit in the aggregator under its mutex, and they are reported in /stats on every scrape.

Not a crash and not a correctness bug, but the feature is effectively unusable on the traffic where it would matter, and it inflates the payload.

Desired behavior

  • Cap key length (a selector is a shape, so a short prefix is sufficient — well under 200 chars).
  • Skip non-text blocks entirely: an image block has no output shape a filter could match, so it should never enter the ledger. Ideally selectorKey returns empty for them and the miss is not recorded at all.
  • Consider normalising the key (collapse digits/hashes) so near-identical misses aggregate into one counted entry instead of 200 slots.

Relevant code

  • components/offload/cmdfilter.goselectorKey, the miss-recording call.
  • metrics/metrics.go — the miss map, maxMissKeys, SelectorMiss, topMisses.

Testing plan

  • Feed an image content block and assert no miss is recorded.
  • Feed a very long text selector and assert the stored key is truncated to the cap.
  • Feed 300 distinct misses and assert both the count cap and the size cap hold.
  • Assert the actionable misses from real traffic (Exit code 1, a source listing) still appear — the point is to keep the signal, not just shrink the payload.

Acceptance criteria

  • Non-text blocks never enter the ledger.
  • Key length is bounded.
  • Replaying capture-tb.jsonl yields a ledger whose top entries are real output shapes, not payloads.
  • /stats stays backward compatible.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    Status
    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions