feat(memory): consolidation run-audit + incident ledger — make "zero data-loss over 14 nights" verifiable (#309) - #314
Merged
Conversation
…data-loss over 14 nights" verifiable (#309) §13.2 exit criterion 3 has two clauses: "runs unattended 14 nights" (observable via the streak) and "zero data-loss incidents" (measured nowhere — ticked on faith). This persists a per-night consolidation run-outcome record and derives a `consolidationIncidents14d` the exit-gate exposes beside the streak. Additive audit only: the reversibility invariant (§2.3) and the existing `consolidation_progress` resume cursor are untouched. - New `consolidation_run` table (migration 0032, journal idx 32), separate from the single-purpose resume cursor. PK (user_id, night), UPSERTED per invocation with a STICKY incident flag and an attempts counter. - Orchestrator: `runJob` now RETURNS each J*Result's numeric mutation counts; the core aggregates them and writes ONE run-audit row per terminal path (quiet / complete / caught-error), re-throwing on error so pg-boss retry/resume is unchanged. Injected `now` (§2.6). Audit write is best-effort — never destabilises the durable consolidation. - Exit-gate: `consolidationIncidents14d` plumbed through ExitGateInputs/Stats and buildExitGateStats; admin repo aggregate windowed by the injected `now`; ops dashboard shows the incident count beside the streak and folds "zero incidents" into the gate pass. Incident condition scoped to (a) a caught job error this PR; the post-night integrity assertion (c) is a deliberate follow-up (#310) — the sticky recorder still ensures a broken-then-resumed night never reads as clean. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #309.
Summary
§13.2 exit criterion 3 has two clauses — "runs unattended 14 nights" (already observable via the streak) and "zero data-loss incidents", which was ticked on faith with no telemetry. This adds a durable per-night run-audit ledger so that clause becomes a checked number.
A new
consolidation_runtable records, per(user_id, night): theoutcome(complete/partial/quiet/error), per-job mutationcounts(aggregated from eachJ*Result),attempts, a stickyincidentflag, the caughterror, andresumed_from.admin.service.getStatswindows incidents to the trailing 14 nights and the exit-gate surfacesconsolidationIncidents14dbeside the streak. Additive audit only — the resume cursor (consolidation_progress) and the reversibility invariant (§2.3) are untouched.Design choice (open question): a new table, not extending
consolidation_progress— keeps the resume cursor single-purpose so the ledger can't perturb §08.3 idempotency/resume.Incident definition: scoped to (a) any caught job error this PR. The orchestrator wraps the job loop in try/catch: on a throw it records
outcome:'error', incident:true(with partial counts) and re-throws, preserving pg-boss's fail→retry→resume semantics. The recorder's UPSERT makesincidentsticky (existing OR new) and bumpsattempts, so a night that errored once then resumed-clean endscomplete/attempts=2/incident=true— never indistinguishable from a clean night. Integrity-assertion incidents (condition (c)) are a deliberate follow-up (they'd need a DB read from the hermetic core); the sticky recorder already prevents a broken-then-resumed night from reading clean. Audit writes are best-effort (recordRunSafely) so a flaky ledger write never destabilises the (already-durable) consolidation.Files
apps/api/src/db/schema.ts+drizzle/0032_gorgeous_jimmy_woo.sql(journal idx 32) — theconsolidation_runtable.consolidation-queue.ts—runJobreturns per-job counts;recordRundep + sticky-incident UPSERT; one audit row per terminal path (quiet/complete/error) with re-throw on error.admin.repository.ts—consolidationIncidents14d(sinceNight)aggregate (same admin-plane pattern ascompletedConsolidationNights).exit-gate.ts/admin.service.ts—consolidationIncidents14dplumbed through the inputs, windowed by injectednow; streak unchanged.apps/admin/src/lib/api.ts+ops/page.tsx— contract + "(N incidents, 14d)" beside the streak row (the rich per-night panel is ux(admin): per-night consolidation health panel on the ops dashboard (§13.2) #310).Acceptance criteria
quietoutcome; a night that throws recordserror+resume and is a non-clean night — integration test on both paths (erroredattempts=1,incident=true→ clean resume endscomplete,attempts=2,incident=true,resumedFrom='J3').GET /admin/statsexposesexitGate.consolidationIncidents14dbeside the streak — integration + unit test; 15-nights-ago excluded proven viagetStats(FROZEN_NOW=2026-07-19)(cutoff2026-07-05).thebrain_test; the aggregate matches the admin-plane exit-gate read pattern; nonew Date()in the math (injectednow).Verification (all green)
pnpm type-check— 7/7 ·pnpm lint— 7/7 ·pnpm build— 5/5pnpm test— api 824 passed (126 files); changed-file re-run against the live DB: 3 files / 45 tests passed (incl. the sticky-incident resume + 15-nights-excluded cases)No new off-box call (audit is DB-local — no privacy-fence surface); nothing hard-deleted; one migration, numbered last (idx 32), applied on top of a freshly-rebuilt test DB.
Co-authored-by: Claude Opus 4.8 noreply@anthropic.com