Add disposable PostgreSQL and S3 backup/restore validation

[TheSilkky]

Add disposable PostgreSQL and S3 backup/restore validation

Priority

P1

Type

testing

Labels

Suggested GitHub labels:

• backlog
• testing
• deployment
• documentation

Branch scope

• Current branch: develop
• Current HEAD: 3555d1fd531ecd4b51b85779618bf4717514aa25
• Target branch: develop
• Original reviewed branch/ref: upgrade-codex-reports-and-repo-full-validation-security-review
• Original reviewed commit SHA: e609ff86028c81bd149839e03d1ffc0eb2ee9e4a
• Merged source pull request: #378
• Target release/version: unreleased
• Scope classification: follow-up-after-merge
• Revalidation status: Revalidated against develop at 3555d1fd531ecd4b51b85779618bf4717514aa25 on 2026-07-12; repeat if develop moves before issue creation.

Summary

Add an executable disposable restore drill that validates PostgreSQL metadata and S3-compatible encrypted blob state together.

Context

The cluster backup/restore and deletion-state documentation describes the required operator checks, but the technical review did not find or execute a repeatable automated drill that restores both durable backends and validates their relationship.

Issue #97 and PR #112 added the cluster runbook. Issue #201 and PR #286 added deletion-state restore guidance. This draft is limited to executable validation and the minimum documentation needed to run it safely.

Proposed change

Add an opt-in local test or script that:

• creates disposable PostgreSQL and S3-compatible services
• writes synthetic encrypted test state through supported application paths
• takes a coordinated test backup
• restores into fresh disposable services
• verifies expected metadata, encrypted objects, bundle behavior, and deletion-state handling
• reports missing or inconsistent restored state through sanitized test output

The drill must use generated values only and must clean up its services and artifacts.

Acceptance criteria

• The drill is explicitly opt-in and targets fresh disposable PostgreSQL and S3-compatible services.
• A coordinated backup and restore preserves expected synthetic metadata and encrypted-object state.
• Validation detects missing or inconsistent restored state without printing object keys, stored paths, credentials, payloads, or private endpoints.
• Deletion and tombstone states covered by the current runbook are checked after restore where applicable.
• Public viewer and bundle behavior remain fail closed when required restored state is unavailable.
• Coordination state is treated as disposable rather than durable evidence.
• The drill cleans up containers, networks, volumes, temporary files, and generated test artifacts.
• Documentation explains the opt-in command and its limitations without implying production readiness.

Tests / validation

• Run the new backup/restore drill twice against fresh disposable services.
• Run the drill's expected-failure cases with synthetic data.
• Run go test ./... if Go code changes.
• Run go vet ./... if Go or CI code changes.
• Run scripts/check-markdown-links.py if docs change.
• Run git diff --check.
• Verify cleanup leaves no drill containers, networks, volumes, or temporary artifacts.
• Revalidate against the target branch before public issue creation.

Out of scope

• Production backup automation, schedules, or retention guarantees.
• Cloud-provider infrastructure, Kubernetes, Terraform, or managed-service integration.
• Accessing real deployments, credentials, endpoints, accounts, or evidence.
• Backend decryption, raw-key handling, key custody, or playable exports.
• Rewriting the existing runbooks beyond narrow instructions for the test drill.

Notes

Report reference: docs/reports/2026-07-10-proofline-unreleased-technical-review.md, public validation follow-up.

Related completed work: issue #97 and PR #112; issue #201 and PR #286.