feat: offline space backup and restore#244
Open
davidfarah2003 wants to merge 14 commits into
Open
Conversation
No-consumer native snapshots with ordered chunk sinks, restore initiation/upload/finalize with exact-ID sessions, canonical stream config and state validation, conservative pull-consumer checkpoints, exact per-phase backup/restore/infrastructure permission profiles, and space-auth trust validation. The deployer profile gains point-reads on the manager and delivery lease buckets for the preservation quiescence proof.
One fsynced, generation-bound journal drives the preserve/backup/restore state machine: cut-intent -> cut-committed -> ready, backup claims with coordinator/owner liveness and attempt-owned path slots (pending before creation, inode-proven after), restore-ready records carrying a liveness claim for the whole pre-commit window, listener proofs with dead-listener replacement for both restore and ordinary resume, and fail-closed recovery that removes only provable attempt residue and never surrenders ownership over unproven cleanup.
Two-phase preserve (prepare fences lifecycle and builds the resume inventory, commit stops children with every leave/deprovision suppressed), inventory validated against the exact resume control schema before any stop, durable commit tokens with idempotent commit/finalize recovery, resume that adopts retained principals and never provisions replacements, and abort that reconciles exits suppressed under the fence. Runtime providers (pty/tmux/cmux/orca) report authoritative child status so stops are provable.
Non-secret authority fingerprints bind full artifacts to the exact account identity, active signing set, and (in user mode) owner-secret, IdP, issuer, and actor-ledger inputs; drift fails closed before any store mutation. Restore never creates fresh auth.
cotal down --preserve-state fences Plane 3 before the inventory, proves control-plane quiescence on the wire (leases, not pidfiles), and journals the manager commitment before any stop so retries never need a live manager. cotal backup claims ownership before creating clone, broker run files, or destination, snapshots through a watchdog-owned isolated broker, and publishes the manifest atomically as the completion marker. cotal up --restore quarantines, re-snapshots sanitized bytes as the sole target input, asserts the exact post-restore inventory, and binds a provable normal listener; ordinary resume adopts or replaces its bound listener and opens only the exact preserved store inode. Explicit recovery lands as cotal clean restore-attempt. Replaces the interim channels export surface.
Operator sequence, artifact and checkpoint semantics, trust continuity, restore liveness claims and recovery recourse, and the maintenance cleanup verbs; regenerated docs bundle.
Env-gated failure points at snapshot initiation, first chunk, close, and manifest publication, a synthetic exact-ID handoff timeout per stream, and owned placeholder live smokes for conservation, user-mode restore, and fault/timeout replay coverage.
A fresh by_start_sequence durable is born with a native stream floor of opt_start_seq - 1 (verified on nats-server 2.14), so the zero-floor assertion made any space that had ever ACKed a chat message unrestorable. Validate the exact born floor and a zero delivered count instead - the conservative at-least-once replay semantics are unchanged. The restore fault hook also gains a pass discriminator (quarantine:/target:) so the partial-target-install path is reachable in tests.
Dev-only dependencies so the plan-required live smokes in bin/smoke can read broker state (pre-cut consumer floors, post-restore counts) directly; nothing ships in the published package.
All four artifact stages (initiation, first chunk, close, manifest) fail closed to ready with an absent destination and a clean retry on the same path; the exact-ID handoff timeout on the final stream - injected on the quarantine rehearsal AND the real-target pass - discards the whole populated attempt, returns the identical source inode, and the clean replay restores with no duplicate messages (counts proven through a verification re-cut and re-snapshot).
…Queues The unified born-floor rule is effectiveStart - 1 with effectiveStart = max(opt_start_seq ?? 1, stream first_seq): a TASK role durable that completed any work leaves a truncated WorkQueue whose fresh DeliverAll consumer is born at first_seq - 1, which the previous assertion refused as a non-born floor. At-least-once replay semantics are unchanged; the acked item is gone from the stream and everything remaining replays.
Live pre-cut consumer floors sampled through the backup inspect cred after daemon drain must equal every artifact checkpoint exactly; the artifact carries only the five canonical durables; all eight streams conserve messages and sequence bounds; unacked pre-cut work replays on ordinary resume while acked work stays gone. After a full restore, the member's pending entry is delivered under its preserved authorization and a first NEW post reaches it durably through the rebuilt fanout/inbox/ACL/DLV path. Anti-vacuous guards refuse an all-zero-floor mesh so the blind spot that hid two restore bugs cannot return.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| try { | ||
| return await task; | ||
| } finally { | ||
| if (this.resumeCommitTask === task) this.resumeCommitTask = undefined; |
| let brokerOwner: ProcessOwner | undefined; | ||
| try { | ||
| const handshakeDeadline = Date.now() + BOOT_TIMEOUT_MS; | ||
| while (!brokerOwner) { |
Comment on lines
+2
to
+20
| import { | ||
| chmodSync, | ||
| closeSync, | ||
| constants, | ||
| fstatSync, | ||
| fsyncSync, | ||
| lstatSync, | ||
| mkdirSync, | ||
| openSync, | ||
| readFileSync, | ||
| realpathSync, | ||
| readSync, | ||
| readdirSync, | ||
| renameSync, | ||
| rmSync, | ||
| statSync, | ||
| writeFileSync, | ||
| writeSync, | ||
| } from "node:fs"; |
| writeFileSync, | ||
| } from "node:fs"; | ||
| import { hostname } from "node:os"; | ||
| import { dirname, join, resolve } from "node:path"; |
| let sanitized: { path: string; identity: { dev: bigint; ino: bigint } } | undefined; | ||
| let broker: IsolatedBroker | undefined; | ||
| let quarantineBroker: IsolatedBroker | undefined; | ||
| const recordResources = (input: { owners?: readonly ProcessOwner[]; ownedPaths?: readonly AttemptOwnedPath[] }) => { |
Folds the manifest-extension manager resolution, detached-summary rendering, and five new smoke:ci suites in with the backup/restore maintenance surface (union of manager options, detach return shape, and the check/smoke:ci gate chains); docs bundle regenerated.
| else { fail++; console.log(` ✗ FAIL: ${name}`, extra ?? ""); } | ||
| }; | ||
| const wait = (ms: number) => new Promise((r) => setTimeout(r, ms)); | ||
| const until = async (cond: () => boolean | Promise<boolean>, ms = 10_000): Promise<boolean> => { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the interim channel-registry export with a complete offline backup/restore lifecycle for a space's broker-resident durable state, per the approved
.internalspace-backup plan.Surface
cotal down --preserve-state— maintenance cut: fences the manager and Plane-3 delivery before the inventory snapshot, proves control-plane death on the wire (leases, not pidfiles), retains credentials/durables/principals, and publishes an inode-boundreadyrecord only after every proof. Crash-recoverable at every boundary (durable prepare intent,cut-intent,cut-committed).cotal backup create <dir> [--only full|registry]— offline artifact from a ready cut: attempt-owned reflink clone, watchdog-owned isolated loopback broker with exact per-phase credentials in every auth mode, canonical stream/consumer validation, no-consumer snapshots, conservative ACK-floor checkpoints, atomicmanifest.jsoncompletion marker. Ownership precedes existence: the maintenance claim journals pending path slots before anything is created, and recovery deletes exactly the journaled inodes.cotal up(ordinary resume) — resumes the exact preserved source inode with a bound, provable listener (adopt-alive / replace-dead / refuse-ambiguous), and relaunches agents under their retained principals, never minting replacements.cotal up --restore <dir> [--restore-only registry] [--accept-missing-source]— pre-listener restore: staged pinned bytes, quarantine validation, sanitized re-snapshots as the sole target input, trust continuity (same space + signing authority, never fresh auth), exact post-restore stream inventory, and a journaled liveness claim covering the whole pre-commit window so concurrent commands refuse rather than roll back a live attempt.cotal clean restore-attempt|restore-fallback --attempt <id> --force— the only (explicit) recovery/cleanup verbs.Review
Adversarial mesh panel (engineering + security lanes) over three rounds; increments 1–2 double-APPROVED, including fixes for a critical concurrent-rollback race, a preserve-state wedge, quarantine sanitization, claim-first backup ownership, atomic completion markers, fail-closed cleanup, listener re-entry, and wire-truth quiescence. Final full-feature verdict pending the last test suite (see below).
Tests
Unit: workspace maintenance state machine (198 checks), manager preservation/resume, auth continuity, core backup, CLI. Live gates:
smoke:backup-perms:live(permission deny matrix),smoke:backup-restore:live(15 scenarios incl. six crash-boundary re-entries, claim races, backup→restore→backup→restore, source-immutability),smoke:backup-faults:live(per-stage fault injection + exact-ID timeout whole-attempt replay),smoke:backup-conservation:live(cursor stillness + post-restore delivery). These tests surfaced and fixed two real restore-breaking bugs (born ACK floors on by_start_sequence and truncated-WorkQueue DeliverAll durables). Outstanding:smoke:backup-usermode:live(user-auth lifecycle) is in flight and lands on this branch before merge.