Fix domain cutover snapshot consistency

docs/operations/mongodb-migrations.md

@@ -74,13 +74,15 @@ Run the mutating command only in an approved maintenance window:
 ```
 
 The confirmation is exact; `deploy` and `auto-deploy` cannot supply it. One
-fixed `locks\deploy.lock` is held from the fresh hash-bound backup and second
-dry restore through isolated candidate proof, writer stop with SCM recovery
-suspended, live stage/publication, target startup-barrier publication, stopped
-target-snapshot re-verification, one-at-a-time legacy deletion, target-active
-marker finalization, one target start and verification, recovery restoration,
-and auto-deploy refresh. The candidate uses only a generated non-production
-database and configured candidate port.
+fixed `locks\deploy.lock` is held while the legacy writer is stopped with SCM
+recovery suspended before the fresh hash-bound backup and protected evidence
+snapshot. That one quiesced snapshot is then used for the second dry restore,
+isolated candidate proof, live stage/publication, target startup-barrier
+publication, stopped target-snapshot re-verification, one-at-a-time legacy
+deletion, target-active marker finalization, one target start and verification,
+recovery restoration, and auto-deploy refresh. The maintenance window therefore
+includes candidate verification. The candidate uses only a generated
+non-production database and configured candidate port.
 
 Deletion is last. Before the first deletion intent, recovery reverses the
 publication when needed, removes only manifest-owned staging namespaces, proves
@@ -118,11 +120,14 @@ After a successful rollback, `mongo-consolidation-preview` and a newly confirmed
 legacy-compatible marker, active legacy release, and legacy schema still agree.
 Any outstanding one-shot rollback reconciliation must be completed first.
 The new cutover preserves the exact prior terminal JSON under protected
-`state\history`, retains its archive/evidence files, and creates a fresh backup,
-evidence record, owner token, and candidate database before publishing the new
-`PREVIEWED` state. A missing or mismatched marker, nonterminal state, active
-release mismatch, or target-schema legacy release blocks before backup or state
-publication.
+`state\history`, retains its archive/evidence files, stops the exact legacy
+writer, and creates a fresh backup, evidence record, owner token, and candidate
+database before publishing the new `PREVIEWED` state. A missing or mismatched
+marker, nonterminal state, active release mismatch, or target-schema legacy
+release blocks before writer stop, backup, or state publication. A writer-stop
+postcondition failure aborts before backup. After quiescence is proven, any
+snapshot initialization failure triggers guarded recovery of the exact prior
+marker and legacy release before normal SCM recovery is re-enabled.
 
 Every fresh consolidation now publishes a protected prepublication boundary
 before candidate work. An immutable evidence-and-owner-addressed binding records

docs/operations/windows-production.md

@@ -567,13 +567,19 @@ exact confirmation switch:
 .\prod.cmd mongo-consolidate -ConfirmDomainCollectionCutover
 ```
 
-The command proves a fresh hash-bound, dry-restored backup and an isolated
-candidate database/port before stopping the live writer. SCM recovery remains
-suspended for live mutation. It stages and verifies the 14 targets, publishes
-the target startup barrier, re-proves the exact target snapshot while the writer
-remains stopped, and only then drops legacy collections one at a time. After the
-target-active marker is durable, it starts and verifies the target release once.
-Automatic deployment cannot initiate or confirm it.
+The command first stops the exact live legacy writer with SCM recovery
+suspended, then creates the fresh hash-bound backup and protected evidence from
+that quiesced snapshot. It dry-restores and proves an isolated candidate
+database/port against the same snapshot, stages and verifies the 14 live targets,
+publishes the target startup barrier, re-proves the exact target snapshot while
+the writer remains stopped, and only then drops legacy collections one at a
+time. The maintenance window includes candidate verification. A failed writer
+stop postcondition aborts before backup. After quiescence is proven, a backup,
+evidence, or candidate initialization failure runs guarded prepublication
+recovery and restarts the exact legacy release without deleting or restoring
+production data.
+After the target-active marker is durable, the command starts and verifies the
+target release once. Automatic deployment cannot initiate or confirm it.
 
 For guarded recovery use:
 

docs/superpowers/plans/2026-08-12-domain-cutover-quiesced-snapshot.md

@@ -0,0 +1,114 @@
+# Domain Cutover Quiesced Snapshot Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Make the guarded domain-collection cutover take its backup, protected evidence, and candidate proof from one writer-quiesced snapshot.
+
+**Architecture:** Move writer quiescence into cutover-context initialization, immediately before the verified backup. Add an initialization-failure recovery seam that either reconciles an exact committed PREVIEWED pair through the existing state machine or restores and restarts the exact prior legacy release when no pair committed. Remove the later duplicate stop while retaining the fixed-root recheck before production migration.
+
+**Tech Stack:** PowerShell 7/Windows PowerShell 5.1, Pester 5.9, MongoDB migration scripts, native Windows service orchestration.
+
+## Global Constraints
+
+- Never weaken `C:\ProgramData\christopherbell.dev` ACLs.
+- Hold the existing fixed-root `deploy.lock` for the entire cutover and recovery sequence.
+- Do not stage, publish, rename, drop, or restore production data before the exact stopped snapshot and candidate proof succeed.
+- Preserve the current target-writer rule: target startup occurs only after legacy deletion completes.
+- Any pre-`DROP_STARTED` failure must restore the legacy writer without restoring the backup or deleting legacy collections.
+- Use the existing protected state, marker, fixed-root, and release-switch boundaries; add no ad hoc service/process commands.
+
+---
+
+### Task 1: Quiesce the exact snapshot and recover initialization failures
+
+**Files:**
+- Modify: `ops/production/windows/tests/Production.DomainCollections.Orchestration.Tests.ps1:378-515`
+- Modify: `ops/production/windows/modules/Production.DomainCollections.psm1:1169-1237`
+- Modify: `ops/production/windows/modules/Production.DomainCollections.psm1:1792-1846`
+- Modify: `docs/operations/windows-production.md`
+- Modify: `docs/operations/mongodb-migrations.md`
+
+**Interfaces:**
+- Consumes: `Stop-ProductionDomainCollectionWriter`, `Resolve-ProductionDomainCollectionPrepublicationPublication`, `Read-ProductionDomainSchemaDirection`, `Read-ProductionDomainCollectionProtectedState`, `Invoke-ProductionDomainCollectionFailureRecovery`, `Restore-ProductionDomainCollectionLegacyRelease`, `Start-ProductionDomainCollectionLegacy`, and `Set-ProductionWebsiteRecoveryPolicy`.
+- Produces: `Restore-ProductionDomainCollectionSnapshotInitializationFailure -Context <pre-snapshot context>` and a `New-ProductionDomainCollectionCutoverContext` result whose `writerStopped` field is always `$true`.
+
+- [ ] **Step 1: Write the failing orchestration tests**
+
+Add behavior tests that independently require:
+
+```powershell
+$script:events.IndexOf('stop-suspended') | Should -BeLessThan `
+    $script:events.IndexOf('backup-and-evidence')
+```
+
+and that make `New-ProductionDomainCollectionVerifiedBackup` throw after the
+real context initializer stops the writer, then assert one exact legacy
+restart, `Normal` recovery policy, zero preview/publication, zero staging, and
+zero deletion. Update the candidate-failure regression to require the writer
+was already stopped and that prepublication recovery still runs once.
+
+- [ ] **Step 2: Run the focused tests and verify RED**
+
+Run:
+
+```powershell
+Import-Module Pester -MinimumVersion 5.0 -ErrorAction Stop
+Invoke-Pester -Path ops/production/windows/tests/Production.DomainCollections.Orchestration.Tests.ps1 -Output Detailed
+```
+
+Expected: the new order assertion fails because `backup-and-evidence` precedes
+`stop-suspended`; the initialization-failure test fails because the legacy
+restart boundary is not invoked.
+
+- [ ] **Step 3: Implement the minimal quiescence and recovery change**
+
+In `New-ProductionDomainCollectionCutoverContext`, after exact target/legacy
+release validation and prior-marker capture, construct the minimal recovery
+context, call `Stop-ProductionDomainCollectionWriter`, and only then call
+`New-ProductionDomainCollectionVerifiedBackup` and preview. Set
+`writerStopped = $true` in the full context.
+
+Add `Restore-ProductionDomainCollectionSnapshotInitializationFailure` with
+this closed behavior:
+
+```powershell
+Resolve-ProductionDomainCollectionPrepublicationPublication -Config $Context.config
+$marker = Read-ProductionDomainSchemaDirection -Config $Context.config
+if ($marker -and [string]$marker.state -ceq 'ROLLBACK_IN_PROGRESS') {
+    $state = Read-ProductionDomainCollectionProtectedState -Config $Context.config
+    Invoke-ProductionDomainCollectionFailureRecovery -Context $state -PostDrop:$false
+    return
+}
+Restore-ProductionDomainCollectionLegacyRelease -State $Context
+Start-ProductionDomainCollectionLegacy -State $Context
+Set-ProductionWebsiteRecoveryPolicy -Policy Normal
+```
+
+Wrap the post-stop context initialization in `try/catch`; preserve the original
+failure unless recovery also fails, in which case throw an `AggregateException`
+with both causes. Remove the later stop at the orchestration call site.
+
+- [ ] **Step 4: Run focused GREEN and compatibility checks**
+
+Run the orchestration suite under PowerShell 7 and Windows PowerShell 5.1 with
+the explicit Pester 5.9 manifest. Parse the changed PowerShell files under both
+hosts. Run the domain command, deployment, writer-start, and operations suites
+that share the marker/service boundary.
+
+Expected: zero failures; ordering proves writer stop before backup and no
+duplicate stop.
+
+- [ ] **Step 5: Run real migration and full Windows verification**
+
+Run the marker-owned disposable Mongo harness and the full PowerShell 7
+production suite. Confirm the harness reports the full 52-kind, 126-index,
+14-target, 52-drop, and 468-boundary matrix with zero owned process/root residue.
+
+- [ ] **Step 6: Review, document, commit, and publish**
+
+Update both operations documents to state that maintenance downtime starts
+before the protected backup and evidence snapshot. Run `git diff --check`,
+request independent review, commit the cohesive change, push the branch, open a
+PR, wait for required CI, merge, refresh the isolated worktree to merged main,
+then rerun the guarded production cutover and exact HTTP/service/database
+verification.

docs/superpowers/specs/2026-08-12-domain-cutover-quiesced-snapshot-design.md

@@ -0,0 +1,55 @@
+# Domain Cutover Quiesced Snapshot Design
+
+## Problem
+
+The guarded domain-collection cutover creates a production backup and then
+captures protected evidence while the legacy writer is still running. Lease
+renewals can change documents between those two effects. The isolated candidate
+then restores the backup and correctly fails `restore-verify` because its exact
+checksums differ from the later evidence. Production recovery is safe, but the
+cutover cannot reliably progress.
+
+## Decision
+
+Quiesce the production writer, with automatic recovery suspended, before the
+verified backup and protected evidence are created. Keep the writer stopped
+through candidate restore, migration, application verification, production
+stage/publication, stopped-target re-verification, and legacy deletion. Start
+the target writer only after deletion completes, as the existing cutover
+contract already requires.
+
+This deliberately accepts several minutes of one-time maintenance downtime in
+exchange for one exact immutable snapshot across backup, evidence, candidate
+proof, publication, deletion proof, and rollback.
+
+## Recovery
+
+The existing stop boundary must prove the writer quiesced before backup begins;
+if that proof fails, cutover aborts without a snapshot effect. After quiescence
+is proven, a backup or preview failure must restart the exact active legacy
+release, restore its prior schema marker, and return website recovery policy to
+`Normal`. Once the protected PREVIEWED pair exists, failures continue through
+the existing evidence-bound prepublication recovery state machine. No failure
+before `DROP_STARTED` may restore the backup or delete legacy data.
+
+## Rejected Alternatives
+
+- A live preflight snapshot followed by a second final snapshot reduces
+  downtime but creates two evidence identities and a more complex crash/retry
+  protocol.
+- Ignoring or normalizing leases would weaken exact backup equivalence and is
+  incompatible with the destructive migration contract.
+
+## Verification
+
+- A focused orchestration regression must fail against the current order and
+  prove `stop-suspended` precedes `backup-and-evidence` and candidate work.
+- A pre-context backup/preview failure regression must prove exact legacy
+  restart, prior-marker preservation, recovery policy normalization, zero
+  staging, and zero deletion.
+- Candidate failure must prove the writer was already stopped and existing
+  guarded prepublication recovery ran.
+- Focused Pester must pass under PowerShell 7 and Windows PowerShell 5.1 with
+  Pester 5.9; changed files must parse under both hosts.
+- The disposable Mongo cutover matrix and the full relevant Windows suite must
+  remain green before publication.