queue: compact the append-only store and enforce the row invariant

[pitimon]

Measured on a real install

lines:        34,492
unique keys:   5,595
superseded:   28,897  (83.8%)
size:             11 MB

Five sixths of the file is dead weight. Every read path parses all of it.

src/lib/local-api.js:122-160 reads the entire queue.jsonl with readFileSync, re-parses every line and re-dedups, on every endpoint call. A dashboard refresh hits 6-8 endpoints; auto-refresh defaults to 30s. Nothing ever compacts — the only rewrite in the codebase is a one-off migration (sync.js:1836).

Appends are already deduped per totals-change (rollout.js:1541,1591), which is the right design for an append-only log. The gap is that superseded rows are never reclaimed.

This is not urgent today. It is monotonic, it is on the hot path of every render, and the fix carries close to zero design risk because the readers already define what the output must be: keep the last row per key. Compaction only has to produce what every reader already computes.

Proposal

sync --compact, and an automatic trigger when the superseded ratio crosses a threshold.

• Write to a temp file and rename — the atomic-replace pattern already used for the project queue at sync.js:2040-2043.
• Hold the sync lock for the whole operation. The mkdir mutex and heartbeat from fix: correct expired DeepSeek prices, enforce curated expiries, harden Host + sync lock #91 exist precisely so a long-running writer cannot be robbed mid-write; this is the second real user of that.
• Report what it reclaimed. Silent compaction of the file the README tells users to cat is the wrong default.

Second half — enforce the schema invariant

CLAUDE.md:56-63 states the column invariant in prose:

total = input + output + cache_creation + cache_read + reasoning

Nothing enforces it at runtime. A miswritten or corrupt row is aggregated and rendered, not flagged — and a parser bug of this shape is exactly the class CLAUDE.md:169-172 records at 1.6-7× magnitude.

Add a doctor check validating every row against it, plus bucket alignment (30-minute UTC) and non-negative counts. Same conversion as the curated expiry in #91 and the version lockstep in #96: a rule that lived in a document starts running.

Definition of done

• sync --compact reduces this install's 34,492 lines to ~5,595 with byte-identical API responses before and after — that equality is the real test, not the line count
• Compaction is safe against a concurrent append (test with the sync lock held)
• doctor fails on a fixture with a column-sum violation, a misaligned bucket, and a negative count
• Interrupted compaction leaves the original intact (kill between write and rename)

[pitimon]

Shipped in #117 (main @ 089b737). Reading this issue's definition of done against what actually landed:

DoD item	Status
sync --compact collapses the file with byte-identical API responses before and after	done — and by construction, not by luck. Compaction keeps the last RAW LINE per source|model|hour_start, which is exactly what readQueueData keeps, so the surviving bytes are the bytes the reader would have kept. A test reproduces the reader's dedup and asserts equality.
Interrupted compaction leaves the original intact	done — write-then-rename. A test forces the rename to fail and asserts the original is untouched and no temp file is left behind.
doctor fails on a column-sum violation, a misaligned bucket, a negative count	done as a warn, not a fail. The rows are already on disk and already being rendered; failing the health check reports a crisis the user cannot act on in the moment. What they can act on is which rows and how many, so the check names up to 5 examples.
Compaction is safe against a concurrent append (test with the sync lock held)	not tested. It runs inside the existing sync lock, which is where a concurrent writer is already excluded, but there is no test proving that. This is the honest gap.

Two deviations from the proposal, both deliberate

1. Automatic compaction is not shipped. The issue proposed a trigger past a threshold, and I built it. It immediately rewrote an 11 MB production queue — 34,924 lines down to 5,636 — during npm test.

The cause was a pre-existing test-isolation leak (test/init-uninstall.test.js spawned the notify handler in a child process with env: {...process.env}, so overriding process.env.HOME in the parent never reached it). That leak is fixed and guarded. But the lesson stands: making sync rewrite the user's data file turns every path that reaches sync into a data-mutating path. Sync now recommends --compact past the threshold and does not act.

2. The column invariant as documented was wrong. Running the check against a real 34,922-row queue reported 8,236 violations — every one source=codex, and in every case the difference was exactly reasoning_output_tokens.

The rows were right. Codex and every-code fold reasoning INTO output_tokens, and computeRowCost (src/lib/pricing/index.js:309) already made that distinction by charging their reasoning at zero. CLAUDE.md said "sum of all columns" with no exception — the prose was the defect. Corrected there, and expectedTotal() is now the one rule shared by doctor and the #104 parser harness, which had copied the same too-strict version.

After the fix: 13 findings, all inside 2 real codex buckets whose total exceeds its columns by 5,337. Genuine data, left visible rather than tuned away.

What is left

The concurrency test, and — if wanted — automatic compaction behind an explicit opt-in rather than a threshold. Ready to close otherwise.

[pitimon]

Definition of done complete. The concurrency item is closed in #123; the rest shipped in #117.

The concurrency test demonstrates BOTH halves rather than only the happy path, because compaction is safe due to the lock and not on its own: an append landing in the window between the temp write and the rename really is lost, the lock really does exclude a second holder, the positive case runs in sync's own lock-compact-release arrangement, and a source fact pins the ORDER in sync.js — reordering those lines would leave the other three green while reopening the window.

Two deviations from the proposal, both deliberate and both recorded above: automatic compaction is not shipped (wired up first, it rewrote an 11 MB production queue during npm test), and the doctor check is a warn rather than a fail because the rows are already on disk and being rendered — failing the health check would report a crisis the user cannot act on in the moment.