feat(doctor): detect static-with-DB drift + dark (undeclared) databases

[samo-agent]

Summary

• Check 1: static-db-drift (offline, record-based) — src/doctor/static-db-drift.ts
  Detects AppRecords where kind=static but DB fields (migrateCmd, dbBackend, previewDbBackend, databaseUrlEnv, envDbVars) are set. These migrations NEVER run in the static serve path. This is the DETECTION mirror of P1's validateStaticNoDb registration gate — surfaces already-registered apps (pre-P1 drift). One fail result per offending app; pass when the fleet is clean.

• Check 2: dark-db (on-VM SSH, folds into existing single-SSH batch) — checks.ts + parseDarkDbOutput in doctor.ts
  Extends appDbCheckTemplates with a sudo -u postgres psql probe listing non-system databases and roles on the VM. Fires when dbBackend is absent/"none" yet real app databases or roles exist (hand-installed Postgres outside the managed dblab flow). Fail-safe on probe error → unknown. System DBs (postgres/template0/template1) and system roles (pg_* prefix + postgres) are excluded at SQL query time and defensively in the parser.

• Single-SSH invariant preserved: dark-db is folded into the existing buildAuditScript / parseAuditOutput batch in auditVm. No separate SSH round-trip added.

• Static apps NOT skipped for dark-db: detecting a hand-installed Postgres on a kind=static VM is precisely the gamechangers scenario. STATIC_APP_SKIP_IDS deliberately excludes dark-db.

• static-db-drift in runFleetDoctor: injected in the LOCAL-ONLY checks block per-VM (same pattern as dblab-not-oversized). Always emits a result (pass or fail) so it appears in JSON output and participates in failingVms counter.

Root-cause analysis

P1 (#181) blocks NEW registrations with the kind=static + DB-fields combo. But existing apps (e.g. gamechangers) that were registered before P1 are already drifted: they appear as kind=static with migrateCmd/dbBackend set, and their migrations have never run in either env-create or prod-deploy (both static paths have no migrate phase). P2 (#182) ensures new onboarding defaults to node+DB. These two checks close the detection gap for the pre-existing drift.

Test plan

• test/doctor-drift-checks.test.ts (33 tests, all GREEN):

  • Check 1 fires for each DB field individually, for multiple fields, and does NOT fire for clean static / node+DB / absent-kind apps
  • Check 2 fires when probe shows undeclared app DB/role; does NOT fire when dbBackend is declared or only system DB/roles present; unknown on probe error
  • Both integrate into auditVm and runFleetDoctor: correct dark-db result in per-VM checks, failingVms counter correct, single-SSH invariant (1 call)
  • Fleet-doctor JSON includes static-db-drift per-VM with correct pass/fail

• No regressions: 1209 pass vs 1176 pass on base; 46 fail vs 47 fail on base (pre-existing failures identical, our changes actually reduce count by 1).

🤖 Generated with Claude Code

[samo-agent]

samorev verdict: PASS

Gate status

• M1 (MR-head pipeline): GREEN — CI test job on head 13b6747 is SUCCESS. Full suite verified: bunx tsc --noEmit (no error TS), root bun test 2226 pass / 0 fail (real Postgres service + SAMOHOST_REQUIRE_CADDY=1), @samo/auth SQLite 48/0, @samo/auth real-Postgres integration 27/0.
• M2 (samorev verdict): PASS (this verdict)
• M3 (walk evidence): N/A — internal doctor/fleet-ops CLI, no client UI/UX surface.

Reconciliation (build-agent local 46-fail vs CI 0-fail)

Confirmed LOCAL-ENVIRONMENT artifact, NOT a regression. CI runs the full suite with a postgres:16 service container and PG_TEST_URL set; local run had no PG_TEST_URL (pg/integration tests throw) plus local-only trigger tests. CI head is genuinely full-green (0 fail). NOTE: PR body's "46 fail vs 47 fail on base" describes the local env and is misleading in a PR description — non-blocking, worth a follow-up edit.

Findings (all confirmations, no blockers)

1. Reconciliation — CI head 13b6747 — INFO — full suite 0-fail, real PG + Caddy, tsc clean.
2. static-db-drift predicate is an EXACT field-by-field mirror of P1's validateStaticNoDb (feat(manifest): reject kind=static + DB fields (close silent static-with-DB drift gap) #181, merged): migrateCmd / dbBackend!=none / previewDbBackend!=none / databaseUrlEnv / envDbVars.length>0; kind!=static short-circuits. Fires for each field + multi-field; does NOT fire for clean static / node+DB / absent-kind. Wired LOCAL-ONLY per-VM in runFleetDoctor (mirrors dblab-not-oversized), always emits pass/fail into failingVms + JSON.
3. dark-db — single-SSH invariant CONFIRMED: added to appDbCheckTemplates, flows through the one remote(record, buildAuditScript(checks)) call in auditVm; diff adds NO new remote()/ssh/exec. Test 3e asserts callCount === 1. Fail-safe CONFIRMED: empty/error output → unknown, sweep continues (2h/2i). System exclusions in exact parity at SQL (postgres/template0/template1 DBs; pg_% + postgres roles) AND defensively in parser. Static apps deliberately NOT skipped (gamechangers case) — verified STATIC_APP_SKIP_IDS omits dark-db.
4. No false positives: declared dbBackend=dblab/template → pass even with DBs present (2d/2e); system-only DBs/roles → pass (2f/2g); clean node/static → failingVms=0 (3g/4b/4c).
5. Scope tight: only doctor.ts / fleet-doctor.ts / checks.ts / new static-db-drift.ts / test file. No env-create/app-script/onboarding/manifest change. No auto-remediation — no DROP/ALTER/DELETE/register mutation in the diff (detection only).
6. TDD genuine: RED 1e23e4a → GREEN dfcba85 → tsc fix 13b6747. 33 tests cover fires / does-not-fire / fail-safe / single-SSH / fleet-rollup. Re-review-after-postreview satisfied: the tsc-fix delta (databaseUrlEnv folded into fail description) reviewed — cosmetic, CI green.
7. Timeout/caps audit: NO setTimeout/setInterval/sleep/Promise.race/--timeout/max_* in diff (only async await auditVm). No LLM-bound wall-clock, no caps-as-termination. Secrets scan: clean (matches are check-IDs + test fixtures, no real credentials).

Suggested next steps (non-blocking)

• Follow-up: correct the PR-body "46 fail vs 47 fail" line — it reflects a local env without PG_TEST_URL, not CI (CI = 0 fail). Purely descriptive; does not block merge.