feat(queue): reclaim the 84% of the queue nothing ever reclaimed

[pitimon]

Summary

Addresses #103: sync --compact, plus the column invariant that lived only in prose now running in doctor.

Measured on a real install: 34,492 lines, 5,595 unique keys, 83.8% superseded, 11 MB — and readQueueData re-reads and re-dedups all of it on every endpoint call, 6-8 per dashboard refresh, auto-refresh every 30s.

Why the design risk is near zero

The readers already define the output. readQueueData keeps the last row per source|model|hour_start, so compaction keeps the last raw line per that key — not a re-serialised row. Byte-identical API responses follow by construction rather than by luck: the surviving bytes are the exact bytes the reader would have kept. A test reproduces the reader's dedup and asserts its output is identical before and after.

Unparseable lines are kept. They are invisible to every reader already, so dropping them changes no response — but a partial write worth investigating should survive a routine maintenance command.

Three things found while building this

1. The invariant as documented is wrong, and my check inherited the error

Running it against a real 34,922-row queue reported 8,236 violations. Every one was source=codex, and in every case the difference was exactly reasoning_output_tokens.

The rows were right. Codex and every-code fold reasoning into output_tokens, and computeRowCost (src/lib/pricing/index.js:309) already made that distinction by charging their reasoning at zero. CLAUDE.md said "sum of all columns" with no exception — the prose was the defect.

Corrected there, and expectedTotal() is now the single rule shared by doctor and the #104 parser harness, which had copied the same too-strict version.

After the fix: 13 findings, all inside 2 real codex buckets whose total exceeds its columns by 5,337. Left as a warn — genuine data, not a check artefact.

2. Automatic compaction is NOT shipped, though #103 proposed it

Wired up first, it immediately rewrote an 11 MB production queue — 34,924 lines down to 5,636 — during npm test.

Sync now recommends --compact past the threshold and does not act. Rewriting the user's data file unasked turns every path that reaches sync into a data-mutating path, and that is a decision for the person whose data it is.

3. The reason it could reach production data

test/init-uninstall.test.js spawned the notify handler in a child process with env: { ...process.env }. Overriding process.env.HOME in the parent never reaches a child, so it resolved ~/.tokentracker and ran a real sync.

Pre-existing — it had been appending rows to the maintainer's real queue all along. Compaction only made it visible. The helper now requires an isolated home and throws without one, with a test for that.

Verified empirically: ci:local no longer changes the real queue's mtime.

Test plan

• ci:local exit 0 — 914 root tests (+12), 256 dashboard
• Reader-view equality before and after compaction, on a fixture mixing claude and codex rows across two keys — queue: compact the append-only store and enforce the row invariant #103's actual definition of done, not the line count
• The surviving line is byte-identical to the original, including an unknown future field with non-alphabetical key order
• Interrupt safety: the rename is forced to fail, and the original is asserted intact with no temp file left behind
• No-op when nothing is superseded — the file is not rewritten at all
• doctor flags a column-sum violation, a negative count, a non-numeric column and a misaligned bucket; a codex row that folds reasoning is not flagged, and the same numbers from claude are
• sync --compact end-to-end on an isolated HOME: 6 lines → 1, keeping the latest
• Every touched function under the 50-line rule; queue-compact.js is 220 lines

Not in this PR

Automatic compaction (see above) and the concurrent-append test. The operation runs inside the existing sync lock, which is where a concurrent writer is already excluded.