fix(heal-reliability): five surgical fixes for reliable fleet-wide vhost heal

[samo-agent]

Root-cause analysis

During the 2026-07-20 favicon rollout every VM had to receive hand-written vhosts over SSH because samohost app heal --apply failed silently on each one. Five distinct failure modes — all in the code, not in operator config (except Fix 5 which adds a knob):

Fix 1 — sudo tee to /tmp comparison file blocked

Root cause: heal-script.ts line ~366/423 wrote the mktemp comparison copy with sudo /usr/bin/tee "$_new_main_vhost". The VMs' NOPASSWD sudoers grants only cover /etc/caddy/sites.d/*.caddy — not arbitrary /tmp/* paths. The privileged write failed; bash -euo pipefail aborted the script before any vhost was touched.

Fix: Replace with plain cat > "$_new_main_vhost" <<'HEREDOC' (user-writable temp; no sudo). Only the final staged sites.d path retains sudo /usr/bin/tee. Heredoc delimiters unchanged (quoted for node, unquoted for static — preserves variable expansion semantics).

Fix 2 — Node-app sudoers missing four Caddy grants

Root cause: bootstrap.ts §6 node-app sudoers block omitted the four grants present in the static block (reload caddy, tee /etc/caddy/sites.d/*.caddy, mv --, rm -f). Without these, the node-app SSH user cannot write to sites.d even after Fix 1 corrects the temp-file path.

Fix: Add all four exact-path NOPASSWD grants to the node-app sudoers block. Re-running the bootstrap §6 section on existing node-app VMs (idempotent — overwrites /etc/sudoers.d/<app>-agent, visudo-validated) backfills the grants.

Fix 3 — Inline-Caddyfile pre-flight guard

Root cause: VMs with inline site blocks (field-record: http://field-record-1.samo.team { ... } directly in /etc/caddy/Caddyfile rather than import sites.d/*.caddy) would get a duplicate-site conflict if heal wrote to sites.d. The old code didn't detect this and would write, caddy validate would fail, rollback would fire.

Fix: Generated script now runs a pre-flight grep before any vhost work. If the import line is absent OR the app's mainHost appears as an inline block, the script emits inline-caddyfile and exits 0 (writes nothing). A separate operator migration (documented in docs/migrate-inline-caddyfile.md) is required for field-record before heal will attempt it.

Fix 4 — adopt-provenance flag for header-less vhosts

Root cause: Hand-written vhosts (favicon rollout hand-writes, game-changers, gregg-sites) lack the # generated by samohost/... provenance header on line 1. The heal script classifies them as drift-foreign and skips indefinitely, requiring manual re-writes on every generator change.

Fix: New adoptProvenance option on buildConfigHealScript. When set, if the live file content is byte-identical to the regenerated content modulo the missing header, the script prepends the header via the existing staged+atomic-mv+validate+rollback machinery and reports adopted. On the second run, the header is present → normal cmp → no-drift (idempotent). Default: false; drift-foreign skip preserved for unreviewed cases. CLI: samohost app heal <vm> <app> --apply --adopt-provenance.

Fix 5 — HEAL_VM_CAP env-overridable

Root cause: trigger.ts hardcoded const HEAL_VM_CAP = 2. During a fleet-wide change, 2 VMs/cycle means ceil(N/2) cycles to converge all N apps, causing multi-cycle lag. The default (blast-radius limiter) is correct for initial enable, but operators need a knob to raise it without a code change.

Fix: const HEAL_VM_CAP = Number(process.env["SAMOHOST_HEAL_VM_CAP"] ?? 2). Default behaviour unchanged.

---

Test evidence

• RED commit: f078f9b — 17/26 new tests fail (pinning each fix's behaviour)
• GREEN commit: c5b211b — all 26 pass; full suite 2303 pass / 1 pre-existing fail (auth/pg.test.ts requires PG_TEST_URL env, fails on origin/main too)

bun test test/heal-reliability.test.ts
→ 26 pass, 0 fail

bun test
→ 2303 pass, 1 fail (pre-existing: packages/auth/test/pg.test.ts, PG_TEST_URL not set)

What this PR does NOT do

• Does NOT change the SAMOHOST_APP_HEAL gate or its default (off). Merging activates ZERO new runtime behaviour on prod.
• Does NOT automate /etc/caddy/Caddyfile changes (heal-script.ts scope comment on line 22 preserved).
• Does NOT raise the default cap (stays 2).
• Does NOT run any per-VM state migrations (those are post-merge, supervised, per the runbooks).
• Does NOT cover game-changers deployment (owner decision pending).

Post-merge per-VM migration steps

See vmMigrationPlan in the structured output and docs/migrate-inline-caddyfile.md.

🤖 Generated with Claude Code