feat(usage): preserve usage history past the management read window with a daily rollup sidecar

devlog/_plan/260804_usage_rollup_preservation/000_research.md

@@ -0,0 +1,121 @@
+# 000 — Research: preserving usage history beyond the 64 MiB management read window
+
+## Problem
+
+`~/.opencodex/usage.jsonl` is append-only and unbounded (observed: 157 MB / 380k rows
+after ~5 weeks). `readUsageSnapshotForManagement` reads only the newest
+`managementUsageMaxReadBytes` (default 64 MiB) and caps parsed rows at
+`MANAGEMENT_USAGE_MAX_ENTRIES` (200k). Once the file outgrows the window, every
+`/api/usage` consumer silently loses the oldest days — the summary reports
+`historyTruncated: true` but the data is simply absent from `days`/`models`/
+`providers` and from `summary` totals. Observed in production on 2026-08-04:
+the 30d range was missing 11 of 30 days.
+
+Raising the byte window (256 MiB) or the entry cap (500k) only defers the loss and
+makes every read slower. The fix: fold rows that leave the read window into a
+compact daily aggregate sidecar, and make the reader merge "rollup (old) + raw
+tail (recent)".
+
+## Prior-art survey (lunasearch, 5 lanes, Tier-2 source-proven 2026-08-04)
+
+### Lane 1 — TSDB downsampling (James)
+
+- Prometheus: block-merge compaction, aggregates live in *separate* recording-rule
+  series; safety boundary is the immutable block. [Storage docs](https://prometheus.io/docs/prometheus/latest/storage/) (primary).
+- InfluxDB: raw bucket and aggregate bucket are separate retention domains; a
+  scheduled task windows recent data with an explicit lateness `offset`.
+  [Downsample and retain](https://docs.influxdata.com/enterprise_influxdb/v1/guides/downsample_and_retain/) (primary).
+- Thanos Compactor: rewrites blocks into aggregate chunks *after* a watermark
+  (raw→5m only after 40h); single-compactor rule; halt on overlap.
+  [Compactor](https://thanos.io/tip/components/compact.md/) (primary).
+- Takeaway: every mature system separates the raw store from the aggregate store
+  and only folds data past a stability boundary.
+
+### Lane 2 — append-only log compaction (Beauvoir)
+
+- Kafka never compacts the active segment; compaction reads older immutable
+  segments and atomically swaps results in. [Design](https://kafka.apache.org/41/design/design/) (primary).
+- Readers merge "stable compacted snapshot + live tail by validated offset
+  boundary". Ordering and offsets never change.
+- logrotate `copytruncate` documents a copy→truncate race that loses appends —
+  writer-uncoordinated truncation is inherently lossy.
+  [logrotate.conf(5)](https://man7.org/linux/man-pages/man5/logrotate.conf.5.html) (primary).
+- JSONL: compact only complete newline-terminated records; quarantine a partial
+  final line. [jsonlines.org](https://jsonlines.org/) (lead).
+
+### Lane 3 — crash-safe ordering (Sartre)
+
+- SQLite WAL: persist WAL first, then the DB, then reset the WAL — the aggregate
+  never advances ahead of its source. [WAL](https://www.sqlite.org/wal.html) (primary).
+- Atomic replace: temp file → fsync(temp) → rename → fsync(parent dir).
+  [fsync(2)] (primary).
+- Two files cannot be flushed atomically; make the raw log authoritative, keep a
+  durable monotonic watermark in the sidecar, replay the raw suffix after the
+  watermark on recovery. Advancing the watermark *after* the aggregate is durable
+  converts crashes into idempotent re-work, never loss.
+  [USENIX ATC15 crash-consistency](https://www.usenix.org/system/files/conference/atc15/atc15-paper-min.pdf) (primary research).
+
+### Lane 4 — LLM CLI usage stores (Pascal)
+
+- Claude Code: append-only JSONL transcripts + a *separate* `stats-cache.json`
+  aggregate for `/usage` — the same raw/aggregate split at much coarser grain.
+  [Claude directory docs](https://code.claude.com/docs/en/claude-directory) (primary).
+- Codex: documented unbounded-JSONL pathologies (issue #34061, ~755 GiB of
+  session JSONL) with no shipped rotation fix — nobody upstream has solved this
+  for us; also a warning that agents must not self-ingest their own logs
+  (issue #27131).
+- ccusage consumes schema-complete JSONL/SQLite token records; our
+  `PersistedUsageEntry` is already schema-complete, so folding to aggregates
+  loses only per-request identity, not meterability.
+
+### Lane 5 — exactly-once boundaries (Averroes)
+
+- High-water-mark pattern: persist `(aggregate, mark)` together; read strictly
+  after the mark; advance the mark only with the aggregate.
+  [GOV.UK HWM guidance](https://pg-bulk-ingest.docs.trade.gov.uk/high-watermark/) (primary).
+- A *source offset* answers "which records are consumed" (exactly-once); an
+  *event-time watermark* answers "when is a bucket final". For a local file the
+  offset is the correctness boundary — wall-clock time is unsafe.
+  [Flink streaming analytics](https://nightlies.apache.org/flink/flink-docs-stable/docs/learn-flink/streaming_analytics/) (primary).
+- Emit deltas and add, or emit snapshots and replace — never add repeated
+  cumulative snapshots (Beam accumulating-pane trap).
+
+## Design conclusions adopted
+
+1. **Offset cutline, not date cutline.** The rollup covers exactly the byte range
+   `[0, cutlineOffset)` of a specific file lineage; the reader takes raw rows
+   from `[cutlineOffset, EOF]`. Dedupe is structural (disjoint byte ranges), so
+   out-of-order timestamps cannot double-count and boundary days merge additively
+   in the day grid.
+2. **Rollup rows are deltas.** Each fold run appends aggregate rows covering only
+   the newly folded byte range. The reader sums all rollup rows plus the tail.
+   Repeated groups across fold runs are additive by construction.
+3. **Ordering: fold → fsync rollup → advance meta (temp+fsync+rename).** A crash
+   between rollup append and meta advance re-folds the same range next run; to
+   make that idempotent each fold writes a `segment` record keyed by
+   `(lineageId, fromOffset)` and the folder skips ranges already recorded.
+4. **Never truncate raw by default.** The window mechanism already ignores the
+   prefix; truncation is a policy decision deferred (documented for a follow-up).
+   No writer-coordination or copytruncate-style race exists as a result.
+5. **Token aggregates only; cost stays display-time.** Cost is linear in tokens
+   for a fixed `(provider, model, tier, longContext)` price row, so grouping by
+   those keys preserves exact display-time recomputation and keeps price-table
+   fixes retroactive. Long-context is per-request non-linear, hence it is part of
+   the group key, evaluated at fold time with the same predicate used today.
+6. **Two aggregation grains.** `summary`/`days` count *requests* (entry-level);
+   `models`/`providers` count *attributions* (attempt-level, combo-aware). The
+   rollup stores both kinds of rows explicitly rather than deriving one from the
+   other lossily.
+
+## Claim ledger
+
+| # | Claim | Source | Tier |
+|---|-------|--------|------|
+| 1 | Kafka excludes the active segment from compaction | kafka.apache.org/41/design | verified |
+| 2 | copytruncate documents a lossy race window | man7 logrotate.conf(5) | verified |
+| 3 | SQLite orders WAL-persist before DB-persist before WAL-reset | sqlite.org/wal.html | verified |
+| 4 | Atomic replace requires temp+fsync+rename+dir-fsync | fsync(2) man page | verified |
+| 5 | HWM advances only with its aggregate, source-derived not wall-clock | GOV.UK pg-bulk-ingest docs | verified |
+| 6 | Prometheus/Thanos fold only past a stability watermark | prometheus.io, thanos.io | verified |
+| 7 | Claude Code keeps raw JSONL + separate aggregate cache | code.claude.com | verified |
+| 8 | Upstream Codex has no shipped rotation fix for JSONL growth | github.com/openai/codex#34061 | verified |

devlog/_plan/260804_usage_rollup_preservation/001_roadmap.md

@@ -0,0 +1,120 @@
+# 001 — Roadmap: rollup sidecar as a rebuildable derived cache — rev2 (post-audit 002)
+
+## The one architectural decision everything else follows from
+
+**The raw log is never truncated, so the rollup is a derived cache, not a second
+source of truth.** Any invalidation — lineage change (inode/birthtime), schema
+version bump, price-table fingerprint mismatch — is answered by rebuilding the
+rollup from the full raw file in a background cooperative parse. This dissolves
+the two hardest problems found in research:
+
+- *Cost freezing*: rollup rows may store fold-time cost numbers (including exact
+  per-entry combo fail-closed decisions) because a price-table change flips the
+  fingerprint and triggers a rebuild. Display-time retroactivity is preserved by
+  rebuild, not by schema contortions.
+- *Exactness*: the fold processes whole entries with the same functions the
+  display path uses (`bumpStatus`, `addTokens`, `addEstimatedCost`,
+  `usageAttributions`, `foldAttributionStatuses`), so
+  `summarize(fold(prefix) ⊕ tail)` ≡ `summarize(prefix ⊕ tail)` within the
+  exactness domain defined in 010 (unique requestIds across entries — an
+  assumption backed by the current generator — and no multi-model combo
+  overflow into "other"),
+  enforced with a property test; out-of-domain behavior is additive and
+  carries dedicated documenting tests (002 B2).
+
+## Correctness boundary
+
+- Cutline is a **byte offset at a local-day boundary** of the raw file:
+  rollup covers `[0, cutlineOffset)`, the reader's tail starts at
+  `cutlineOffset`. Disjoint byte ranges make additive merge structurally
+  dedupe-free; a late-timestamped row after the cutline simply counts in the
+  tail (still additive, never double-counted).
+- **Single commit authority (002 B1):** the cutline is derived from validated
+  commit rows inside `usage-rollup.jsonl` itself — group rows first, a
+  `commit` row last, one fsync. Meta only carries lineage/fingerprint/throttle
+  via the existing `renameAtomicFile()`. Reader visibility and tail start
+  always agree because both derive from the same committed segments; a partial
+  append (no commit row / rowCount mismatch) is invisible garbage and the
+  retry re-folds the same range.
+- Only *complete days* older than `ROLLUP_MIN_AGE_DAYS = 9` are folded
+  (stability watermark; keeps the entire 7d window tail-exact, 002 B3).
+  Fold-eligibility check and the fold run are throttled and cooperative (same
+  yield discipline as `parseUsageTextCooperatively`).
+- **Same-lineage mutation policy (002 B5):** rebuild on lineage-tuple
+  mismatch, live size < committed cutline, or committed boundary-digest
+  mismatch (last 4 KiB of each committed segment re-verified at the next fold).
+  Deep hand-edits inside the folded prefix are answered by the documented
+  delete-the-rollup-files escape hatch (full rebuild), preserving the
+  hand-editable raw contract.
+
+## Range semantics
+
+- `all`: rollup days + tail — exact within the exactness domain (010; unique
+  requestIds assumed, no combo overflow into "other").
+- `7d`: tail-exact when `truncatedPrefixBytes === 0` — min-age 9 keeps the
+  fold away from it; an extreme-growth residual gap degrades to today's
+  reported truncation, never silently.
+- `30d`: tail rows keep ms-precision filtering; rollup rows are day-grain, so
+  the one boundary day partially inside the window is included whole (≤ ~24h
+  overcount). This is steady-state behavior for days 9–30, documented in
+  docs-site (002 wording fix: not an exceptional state).
+
+## Row grains (one JSONL file, `kind`-discriminated)
+
+1. `meta` (separate file): `version`, `lineageKey`, `priceFingerprint`,
+   throttle stamp — correctness-light.
+2. `commit`: `{ lineageKey, seg, attemptId, toOffset, rowCount, payloadDigest,
+   boundaryDigest, attributionSinceMs, oldestTimestampMs, foldedAt }` —
+   appended last; the segment's visibility gate and the cutline authority.
+   `attemptId` binds the commit to one append attempt so abandoned crash
+   leftovers can never collide with a retry (R2-1).
+3. `day` rows: key `(date, surfaceKey)` → entry-grain status counts,
+   attemptCount, token sums, cost sums, priced/unpriced/unmetered counts.
+4. `model` rows: key `(date, surfaceKey, providerKey, model)` →
+   distinct-request count, attemptCount, folded per-request status counts,
+   token sums, cost sums (attribution grain, combo-aware — exactly what
+   `buildModels`/day-model breakdown consume); `resolvedModel` rides along as
+   display-only first-seen. Separate `provider` rows store the within-entry
+   combo-deduped provider grain (not derivable from model rows).
+5. `key` rows: key `(date, admissionKind, apiKeyId?)` → request count,
+   lastUsedAtMs max (for `api-key-usage.ts`).
+
+Cardinality: O(days × surfaces × models) — a few KB per day.
+
+## Cost (002 B4)
+
+Long-context tier selection depends on per-request `usage.inputTokens`
+(`isLongContext`), so day-aggregated tokens cannot re-derive cost. The rollup
+stores fold-time cost sums; retroactivity is preserved by rebuild: a wide
+price fingerprint (full jawcode cost table + `EXPECTED_PRICE_OVERLAYS` +
+`PRIORITY_MULTIPLIERS` + `CONTEXT_TIERS` + hand-bumped
+`ROLLUP_COST_SEMANTICS_VERSION`) forces a full refold on any pricing or
+estimator-semantics change.
+
+## Reader merge
+
+`readUsageSnapshotForManagement` gains a `fromOffset` mode: tail read starts at
+`max(cutlineOffset, size - maxReadBytes)`. If `cutlineOffset < size - maxReadBytes`
+the un-folded gap is reported as residual truncation (`truncatedPrefixBytes > 0`)
+until the background fold catches up. `summarizeUsage` accepts optional rollup
+contributions and merges them into totals/days/models/providers before the
+display-time sort/cap steps. `historyTruncated` becomes false when rollup+tail
+jointly cover the file.
+
+## Implementation phases (1 decade doc = 1 PABCD cycle)
+
+- **010** `src/usage/rollup.ts` core: types, fold, file IO, meta, fingerprint,
+  rebuild, idempotent segments + focused tests.
+- **020** reader merge: `log.ts` offset tail, `summary.ts` merge,
+  `logs-usage-routes.ts` wiring + fold trigger, `api-key-usage.ts` merge,
+  config flag `usageRollupEnabled` (default true) + tests incl. the
+  fold⊕tail ≡ full property test.
+- **030** real-data validation against the 157 MB production copy, docs-site,
+  push + dev PR.
+
+## Out of scope (documented for follow-ups)
+
+- Raw-file truncation/archival policy (requires writer coordination; research
+  lane 2 shows uncoordinated truncation is lossy).
+- GUI changes beyond what the existing truncation metadata already drives.
+- Go runtime (retired).