Skip to content

feat(memory): consolidation run-audit + incident ledger — make "zero data-loss over 14 nights" verifiable (#309) - #314

Merged
Jaybhade merged 1 commit into
mainfrom
fix/309-consolidation-run-audit
Jul 19, 2026
Merged

feat(memory): consolidation run-audit + incident ledger — make "zero data-loss over 14 nights" verifiable (#309)#314
Jaybhade merged 1 commit into
mainfrom
fix/309-consolidation-run-audit

Conversation

@Jaybhade

Copy link
Copy Markdown
Owner

Closes #309.

Summary

§13.2 exit criterion 3 has two clauses — "runs unattended 14 nights" (already observable via the streak) and "zero data-loss incidents", which was ticked on faith with no telemetry. This adds a durable per-night run-audit ledger so that clause becomes a checked number.

A new consolidation_run table records, per (user_id, night): the outcome (complete/partial/quiet/error), per-job mutation counts (aggregated from each J*Result), attempts, a sticky incident flag, the caught error, and resumed_from. admin.service.getStats windows incidents to the trailing 14 nights and the exit-gate surfaces consolidationIncidents14d beside the streak. Additive audit only — the resume cursor (consolidation_progress) and the reversibility invariant (§2.3) are untouched.

Design choice (open question): a new table, not extending consolidation_progress — keeps the resume cursor single-purpose so the ledger can't perturb §08.3 idempotency/resume.

Incident definition: scoped to (a) any caught job error this PR. The orchestrator wraps the job loop in try/catch: on a throw it records outcome:'error', incident:true (with partial counts) and re-throws, preserving pg-boss's fail→retry→resume semantics. The recorder's UPSERT makes incident sticky (existing OR new) and bumps attempts, so a night that errored once then resumed-clean ends complete/attempts=2/incident=true — never indistinguishable from a clean night. Integrity-assertion incidents (condition (c)) are a deliberate follow-up (they'd need a DB read from the hermetic core); the sticky recorder already prevents a broken-then-resumed night from reading clean. Audit writes are best-effort (recordRunSafely) so a flaky ledger write never destabilises the (already-durable) consolidation.

Files

  • apps/api/src/db/schema.ts + drizzle/0032_gorgeous_jimmy_woo.sql (journal idx 32) — the consolidation_run table.
  • consolidation-queue.tsrunJob returns per-job counts; recordRun dep + sticky-incident UPSERT; one audit row per terminal path (quiet/complete/error) with re-throw on error.
  • admin.repository.tsconsolidationIncidents14d(sinceNight) aggregate (same admin-plane pattern as completedConsolidationNights).
  • exit-gate.ts / admin.service.tsconsolidationIncidents14d plumbed through the inputs, windowed by injected now; streak unchanged.
  • apps/admin/src/lib/api.ts + ops/page.tsx — contract + "(N incidents, 14d)" beside the streak row (the rich per-night panel is ux(admin): per-night consolidation health panel on the ops dashboard (§13.2) #310).

Acceptance criteria

  • After a night runs, a durable run record exists with per-job counts + null/non-null error — integration test asserts the persisted row.
  • A quiet day records a distinct quiet outcome; a night that throws records error+resume and is a non-clean night — integration test on both paths (errored attempts=1,incident=true → clean resume ends complete,attempts=2,incident=true,resumedFrom='J3').
  • GET /admin/stats exposes exitGate.consolidationIncidents14d beside the streak — integration + unit test; 15-nights-ago excluded proven via getStats(FROZEN_NOW=2026-07-19) (cutoff 2026-07-05).
  • Unit: injected job results with a thrown error → incident-flagged record; a clean set → zero incidents.
  • Migration applies cleanly to a freshly-rebuilt thebrain_test; the aggregate matches the admin-plane exit-gate read pattern; no new Date() in the math (injected now).

Verification (all green)

  • pnpm type-check — 7/7 · pnpm lint — 7/7 · pnpm build — 5/5
  • pnpm test — api 824 passed (126 files); changed-file re-run against the live DB: 3 files / 45 tests passed (incl. the sticky-incident resume + 15-nights-excluded cases)

No new off-box call (audit is DB-local — no privacy-fence surface); nothing hard-deleted; one migration, numbered last (idx 32), applied on top of a freshly-rebuilt test DB.

Co-authored-by: Claude Opus 4.8 noreply@anthropic.com

…data-loss over 14 nights" verifiable (#309)

§13.2 exit criterion 3 has two clauses: "runs unattended 14 nights" (observable
via the streak) and "zero data-loss incidents" (measured nowhere — ticked on
faith). This persists a per-night consolidation run-outcome record and derives a
`consolidationIncidents14d` the exit-gate exposes beside the streak.

Additive audit only: the reversibility invariant (§2.3) and the existing
`consolidation_progress` resume cursor are untouched.

- New `consolidation_run` table (migration 0032, journal idx 32), separate from
  the single-purpose resume cursor. PK (user_id, night), UPSERTED per invocation
  with a STICKY incident flag and an attempts counter.
- Orchestrator: `runJob` now RETURNS each J*Result's numeric mutation counts; the
  core aggregates them and writes ONE run-audit row per terminal path (quiet /
  complete / caught-error), re-throwing on error so pg-boss retry/resume is
  unchanged. Injected `now` (§2.6). Audit write is best-effort — never destabilises
  the durable consolidation.
- Exit-gate: `consolidationIncidents14d` plumbed through ExitGateInputs/Stats and
  buildExitGateStats; admin repo aggregate windowed by the injected `now`; ops
  dashboard shows the incident count beside the streak and folds "zero incidents"
  into the gate pass.

Incident condition scoped to (a) a caught job error this PR; the post-night
integrity assertion (c) is a deliberate follow-up (#310) — the sticky recorder
still ensures a broken-then-resumed night never reads as clean.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
@Jaybhade
Jaybhade merged commit 49a3d08 into main Jul 19, 2026
5 checks passed
@Jaybhade
Jaybhade deleted the fix/309-consolidation-run-audit branch July 19, 2026 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(memory): consolidation run-audit + incident ledger — make "zero data-loss over 14 nights" verifiable (§13.2)

2 participants