test: parser conformance ratchet — cover the code that actually carries risk

[pitimon]

Test coverage in this repo is inverted relative to risk.

Verified

• 118 test files, of which 12 test scripts/graph/ — a SCIP code-graph indexer for the maintainer's own agent tooling, not part of the product. It runs in every ci:local.
• Parsers in the 8,816-line src/lib/rollout.js with no test file at all: hermes, codebuddy, craft, kimi. (droid, goose, kilo, roocode, zed, cursor, codex, claude are covered.)

CLAUDE.md:169-172 records the historical failure modes: dedup failures, cached-input semantics, snapshot-vs-cumulative confusion — 1.6× to 7× magnitude errors in the numbers the whole product exists to report. That is the class with no gate.

README:116 tells contributors "a new provider is usually one parser file away". Today that is true and unguarded: a new parser can land with no test and no CI objection.

Why a ratchet rather than a split

Splitting rollout.js is deferred by explicit decision, and splitting it would not have caught any of the recorded bugs — line count is not the risk, silent per-parser regression is. A conformance harness addresses the actual failure mode with zero churn to that decision.

Proposal

One table-driven test that enumerates every parse*Incremental export and, for each, requires a fixture directory whose parse output satisfies:

• Column sum: total = input + output + cache_creation + cache_read + reasoning
• No negative counts; model present (or the unknown placeholder, which fix(pricing): stop reporting the "unknown" placeholder as an unpriced model #94 defined)
• Buckets aligned to 30-minute UTC boundaries
• Idempotence: parsing the same fixture twice produces identical rows — the "run sync twice" lesson from CLAUDE.md:172, and the one that catches dedup-key instability

Parsers without a fixture go on an explicit allowlist, and CI fails if the allowlist grows. New parsers cannot opt out; existing gaps get closed as anyone touches them.

The ratchet assertion is the entire point. An allowlist that can grow is a TODO list, and this repo already has evidence about what happens to those.

Definition of done

• Harness enumerates parsers from source, not from a hand-written list — a parser added without a fixture must fail, not be silently skipped
• Allowlist committed with today's uncovered parsers and a count assertion that may only decrease
• At least the four uncovered parsers get fixtures, or a recorded reason they cannot (e.g. no obtainable sample log)
• Documented in CONTRIBUTING.md step 4 for adding a tool

Separately

scripts/graph/ (11 source + 12 test files) is personal tooling that pays rent in every ci:local run and inflates the coverage picture. Worth extracting to its own repo — filed as a separate chore.

[pitimon]

Harness shipped in #116 (main @ 6d9f329). The ratchet enumerates 24 parsers from src/lib/rollout.js source, so a new one without a fixture fails. Fixtures for craft, kimi, codebuddy; 21 on the allowlist with reasons, max_size pinned so it can only shrink.

Two things worth carrying forward:

The column invariant does not catch the failure class this issue is about. A parser that folds cache reads into input_tokens and inflates total_tokens to match is internally consistent — the sum balances while the number is ~4x too big. There is a test in the harness pinning that blind spot. Closing it needs real sample logs with known-correct expected values, not a structural check.

hermes still has no fixture — it reads a SQLite database across profile directories, so it needs a real schema rather than a JSONL line. That is the remaining item from this issue's DoD.

Separately, scripts/graph/ is out of the product gate (#115), so the coverage picture this issue describes is now honest: 902 product tests, none of them testing the indexer.

[pitimon]

Definition of done complete.

• Harness enumerates parsers from src/lib/rollout.js SOURCE (24 found), so a parser added without a fixture fails rather than being silently skipped — test(parsers): a new parser could land with no test and no CI objection #116.
• Allowlist committed with a count assertion that may only decrease, and fails if it is shorter without max_size being lowered in the same commit.
• All four named parsers have fixtures: craft, kimi, codebuddy in test(parsers): a new parser could land with no test and no CI objection #116, hermes in test(parsers): the hermes fixture the allowlist was standing in for #120. Allowlist 21 -> 20.
• Documented in CONTRIBUTING step 4, including what the fixture does NOT prove.

Two things carried forward rather than buried:

The column invariant does not catch the failure class this issue is about. A parser that folds cache reads into input_tokens and inflates total_tokens to match is internally consistent — the sum balances while the number is several times too big. There is a test in the harness pinning that blind spot so nobody reads it as covering more than it does. Closing it needs real sample logs with known-correct expected values, which is what the per-tool tests are for.

The invariant as documented was also wrong, found by running the same rule against a real 34,922-row queue: 8,236 false violations, all source=codex, all differing by exactly reasoning_output_tokens. Codex and every-code fold reasoning into output, as computeRowCost already knew. Corrected in CLAUDE.md and shared as expectedTotal() — #117.

20 parsers remain on the allowlist with recorded reasons. The ratchet is what gets those written as anyone touches them.