Epic: Configurable backup & restore for agent data volumes

[vybe]

Summary

Trinity has no backup for agent data volumes. All agent-generated binary/large-file data lives on per-agent named Docker volumes — agent-{name}-workspace (mounted at /home/developer), plus agent-{name}-public (FILES-001) and agent-{name}-shared (shared folders) — and the gitignored content/ directory inside the workspace volume. Git sync covers source code only; binaries are deliberately excluded (src/backend/services/git_service.py) and there is no Git LFS. The only backup tool, scripts/deploy/backup-database.sh, copies trinity.db and nothing else.

Net effect: host disk loss = total, unrecoverable loss of every agent's accumulated data. This epic develops the strategy for — and delivers — a configurable backup & restore capability for agent volumes, shipped as an entitled enterprise feature.

Context

Surfaced while reviewing distributed/binary storage for agents. The data itself is durable against container restart and recreation (the volume is re-attached to the new container) and even survives reset-to-main-preserve-state (the recovery path runs git reset --hard origin/main with no git clean, so gitignored files are left on disk). The gap is not ephemerality — it is the complete absence of any backup / snapshot / restore / portability story. The sharpest risk is single-host disk failure.

Positioned under the Enterprise epic because operators running fleets need configurable, governed backup: OSS ships the minimal primitives; enterprise adds policy, scheduling, retention, off-host destinations, and restore UX.

Design must respect sovereign infrastructure (Principle #6, docs/planning/TARGET_ARCHITECTURE.md): work on a single commodity server by default (local-disk target), with off-host / object-store destinations as opt-in configuration — never a hard cloud dependency.

Strategy questions to resolve (Phase 0)

• Snapshot mechanism: offline tar of a quiesced volume vs. live quiesce hook (SQLite/WAL-safe) vs. filesystem snapshot (overlay/zfs/btrfs).
• Relationship to Agent runtime data volumes: declared data paths with snapshot/restore and portable export #1169 (declared data_paths + snapshot/restore + portable export): is this epic the enterprise productization of Agent runtime data volumes: declared data paths with snapshot/restore and portable export #1169's primitive, or independent machinery? Decide first to avoid duplication.
• Scope of what to back up: workspace volume, public volume, shared volumes, content/ only, or declared data_paths.
• Destinations: local disk (default) → S3-compatible / object store → remote host. Encryption at rest.
• Scheduling & retention model: reuse APScheduler + the existing retention-sweep patterns.
• Consistency with running agents: quiesce vs. crash-consistent; documented guarantee.
• Restore UX: per-agent vs. full-fleet, in-place vs. new instance (ties into Agent runtime data volumes: declared data paths with snapshot/restore and portable export #1169 portability/export).
• Replica-safety interaction (Replica groups: horizontal scaling for single agents #927): a mutable per-agent data volume is what blocks horizontal scaling.

Acceptance Criteria (epic — tracked via child issues)

• Phase 0 strategy doc landed in docs/planning/ covering mechanism, destinations, consistency, retention, restore, and the Agent runtime data volumes: declared data paths with snapshot/restore and portable export #1169 relationship.
• Configurable backup policy (schedule, retention, included volumes/paths, destination) — persisted and admin-editable.
• At least one off-host destination (object-store / S3-compatible), with local disk as default and no mandatory cloud dependency.
• Backups encrypted at rest (AES-256-GCM via src/backend/services/credential_encryption.py).
• Consistent backup of a running agent (documented quiesce or crash-consistency guarantee).
• Restore path for per-agent and full-fleet, validated by a round-trip test (backup → wipe → restore → data intact).
• Shipped behind the enterprise entitlement seam (Spike: enterprise edition architecture — private module strategy for compliance features (SSO, SCIM, SIEM) #847): entitlement_service.register_module() + requires_entitlement() gating; OSS-only builds degrade gracefully.
• Observability: backup success/failure surfaced (operator queue / monitoring); last-backup timestamp per agent.

Technical Notes

• Volume creation & naming: src/backend/services/agent_service/crud.py (agent-{name}-workspace, -public, -shared); recreation re-attach in lifecycle.py.
• Existing DB-only backup/restore: scripts/deploy/backup-database.sh, scripts/deploy/restore-database.sh.
• Scheduling / retention patterns to reuse: src/backend/services/cleanup_service.py, db_vacuum_service.py, scheduler_service.py.
• Enterprise open-core seam (Spike: enterprise edition architecture — private module strategy for compliance features (SSO, SCIM, SIEM) #847): src/backend/enterprise/, two-track migrations (enterprise/backend/_migrations.py), entitlement gating in dependencies.py.
• Encryption: src/backend/services/credential_encryption.py (AES-256-GCM).
• Related: Agent runtime data volumes: declared data paths with snapshot/restore and portable export #1169 (declared data volumes + snapshot/restore + export — strong overlap, coordinate), Replica groups: horizontal scaling for single agents #927 (replica safety), FILES-001 follow-ups — needs refinement before pickup (12 items) #490 (FILES-001 follow-ups), Spike: evaluate Mesa filesystem for agent artifact storage #583 (Mesa filesystem spike), Configurable database backend: SQLAlchemy Core abstraction (SQLite + PostgreSQL) #300 (configurable DB backend, merged).