runs: every failure and refusal becomes a run row or a standing state

[mbertschler]

Summary

The audit trail is the trust contract, but several failure classes never reached it. This makes each one a visible run row or a standing state — the audit-trail foundation for #157 (F6, F14, F15, F26, F30).

New schema version: v25.

Changes

Schema (store/migrations.go, migrateV24ToV25)

• runs.status CHECK widens to admit two terminal states: refused (a preflight safety gate declined before the transfer began) and aborted (a running row reaped after the process that owned it died). Reuses the FK-off rebuild recipe (runs is referenced by files, hook_runs, packs, remote_objects, remote_packs, runs_audit).
• New destination_alarms table (STRICT, per the new-table convention): the per-destination standing-alarm latch.
• Snapshot regenerated via go test ./store -update-schema; TestSchemaSnapshot passes. (Three minimal destination_run_ids migration fixtures gained a runs table, which the v25 rebuild now reads — a real DB at those versions always has it.)

F6/F15 — rclone failures were undiagnosable

• Capture rclone's non-JSON stderr (bounded to 4 KiB) into RunResult.Stderr, and broaden JSON error capture to fatal/critical levels (a fatal backend/auth failure is logged at fatal, which the old error-only filter dropped — leaving the run row's ERROR column blank).
• Fold the stderr tail into the returned error, so the scheduler log and the run row carry the actual reason (auth, host key, path) instead of a bare exit status 1; synthesise a FailedFile so runs.error is never blank on a fatal failure. (kopia already recorded its full message — this matches it.)
• RunResult.DisplayErrors() fixes the errors=0-on-status=failed summary-line contradiction.

F26 — preflight refusals mint no run row

• Marker, kopia-connect, and layout-guard refusals now write a terminal refused run (via a new sync.ErrRefused sentinel) instead of vanishing into a run_id=0 error. A month-dead backup disk now produces visible red.
• A refused run consumes the cadence window (LatestFinishedRun counts it), so the scheduler backs off a dead destination rather than re-minting an identical refusal every tick.

F14 — killed agents leave phantom running runs

• Agent startup reaps its own orphaned running runs to aborted (store.AbortRunningRuns), loudly logged. aborted is excluded from LatestFinishedRun so a crashed run is re-attempted, not treated as a finished cadence window.

F30 — verify mismatches ring once then carry on

• A verify mismatch latches a per-destination alarm (destination_alarms) surfaced in squirrel runs, the TUI dashboard, and squirrel verify output until cleared.
• Both clear paths implemented: an auto-clear on a subsequent clean squirrel verify, and an explicit squirrel verify ack <dest>. Every raise and clear is recorded in runs_audit (tagged with the operator on ack), so deleting the derived latch row loses no forensic history — the raise/clear trail lives in runs/runs_audit, mirroring how peer_sync_state keeps a live overwrite-in-place row with history elsewhere.

Alarm-clear semantics (please confirm)

• The alarm latch is derived standing state, cleared in place (DELETE on clear) with the raise/clear recorded in the append-only runs_audit. This keeps it faithful to the no-loss invariant without an extra history table.
• Raising is idempotent — a repeated mismatch keeps the original "in alarm since" timestamp and run.
• Deliberately out of scope: blocking pushes to an alarmed destination. F30 asks for the latch to be surfaced until cleared; fail-closing subsequent syncs on a latched alarm is a larger behavioural change (and risks freezing a household's backups on a false positive) — the loud, standing surfacing is the first-class fix here, and enforcement can stack on this foundation.

Testing

• go vet ./... — clean
• go test ./... — all pass (new tests: alarm lifecycle, reaper + cadence-scope, rclone stderr/fatal capture + DisplayErrors, marker refusal mints a refused run, verify mismatch latch → auto-clear)
• golangci-lint run (v2.12.2) — 0 issues

Closes #157

🤖 Generated with Claude Code

https://claude.ai/code/session_014Skm2Qm6wWUB7dD4CpPTz7

---

Generated by Claude Code

[Copilot]

Pull request overview

This PR strengthens Squirrel’s audit-trail guarantees for runs by ensuring previously “silent” failure/refusal modes are persisted as terminal run rows and by introducing a per-destination standing alarm latch for verify mismatches. It expands run status semantics (refused, aborted), improves diagnosability for rclone failures (capturing non-JSON stderr and fatal/critical JSON levels), and surfaces standing alarms across CLI and TUI.

Changes:

• Extend run lifecycle tracking: add refused and aborted terminal run statuses; reap orphaned running runs to aborted at agent startup; count refused (but not aborted) in cadence “finished run” logic.
• Add standing verify alarm latch: new destination_alarms table with raise/clear recorded via runs_audit, with both auto-clear and explicit squirrel verify ack <dest> paths.
• Improve rclone failure evidence: capture bounded non-JSON stderr + broaden error-level capture; ensure failed runs don’t report errors=0 via RunResult.DisplayErrors().

Reviewed changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tui/styles.go	Color mapping updated to represent new run terminal statuses in the dashboard.
tui/format.go	Glyph mapping updated for refused and aborted statuses.
tui/dashboard.go	Dashboard now loads and renders standing destination alarms near the top.
sync/verify_remote.go	Verify run finalization now raises/clears destination alarms based on outcome.
sync/verify_remote_test.go	Adds coverage for alarm latch lifecycle via verify runs.
sync/sync.go	Introduces refusal sentinel + refusal run recording; improves rclone fatal error evidence handling.
sync/sync_test.go	Asserts marker refusals wrap ErrRefused and mint a refused run row.
sync/rclone.go	Captures non-JSON stderr, broadens JSON error levels, adds DisplayErrors(), refactors stats handling.
sync/rclone_test.go	Adds tests for fatal/raw stderr capture, stderr bounding, and DisplayErrors().
sync/packed.go	Layout guard refusal now wraps ErrRefused.
sync/packed_test.go	Updates expectations for layout-guard refusals to be refused.
sync/kopia.go	Connect-without-init refusal now wraps ErrRefused.
sync/kopia_test.go	Updates expectations for kopia connect refusal to be refused.
sync/handler.go	Ensures handler-run finish logic derives terminal status via ErrRefused sentinel.
sync/content_addressed.go	Layout guard refusal now wraps ErrRefused.
sync/content_addressed_test.go	Updates expectations for layout-guard refusals to be refused.
store/schema.sql	Regenerated schema snapshot for v25 including destination_alarms and widened runs.status CHECK.
store/runs.go	Adds new statuses, updates terminal-status logic, cadence selection logic, and adds AbortRunningRuns().
store/runs_audit.go	Adds new audit transition constants for abort and alarm raise/clear.
store/migrations.go	Adds v25 migration: rebuild runs table to widen status CHECK; create destination_alarms.
store/destination_run_ids_test.go	Updates minimal migration fixtures to include runs table needed by v25 rebuild.
store/destination_alarms.go	Implements destination alarm latch CRUD and audit recording.
store/destination_alarms_test.go	Adds tests for alarm lifecycle and audit trail behavior.
store/abort_runs_test.go	Adds tests for aborting orphaned runs and cadence finished-run scope.
cmd/squirrel/verify.go	Surfaces alarm transitions in verify output; wires in verify ack subcommand.
cmd/squirrel/verify_ack.go	New command to explicitly clear a standing verify alarm with operator attribution.
cmd/squirrel/sync.go	Uses DisplayErrors() in sync summary output to avoid errors=0-on-failure contradiction.
cmd/squirrel/runs.go	Prints active standing alarms below the runs listing.
cmd/squirrel/agent.go	Reaps orphaned running runs to aborted at startup and logs loudly.

Comments suppressed due to low confidence (1)

sync/rclone.go:359

• rcloneExitError() and its doc refer to a captured stderr tail, but stderr capture is currently a bounded prefix (see appendStderr and tests). Renaming the parameter/local variable and updating the comment would keep terminology consistent and reduce confusion when debugging failures.

// rcloneExitError wraps a non-zero rclone exit, folding in the captured
// stderr tail when present so the scheduler log and the run row carry the
// actual reason (auth, host key, path) rather than a bare "exit status 1"
// (#157, F6/F15).
func rcloneExitError(waitErr error, stderrTail string) error {

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.