Skip to content

runs: every failure and refusal becomes a run row or a standing state #157

Description

@mbertschler

Problem (friction log F6, F14, F15, F26, F30 — S1/S2, all confirmed live)

The audit trail is the trust contract, but several failure classes never reach it:

  • F6/F15 — rclone failures are undiagnosable and evidence-free. CLI and scheduler surface only rclone: rclone exit: exit status 1 (no stderr anywhere), the summary line claims errors=0 on a status=failed run, and the permanent run row's ERROR column is empty — while kopia failures record their full message. Diagnosing the walk's cloudbox failure required hand-driving rclone against squirrel's generated conf.
  • F26 — preflight refusals mint no run row. An "unplugged" USB disk's marker refusal (run_id=0) exists only in agent stderr; every squirrel surface just shows the last success quietly aging. A month-dead backup disk produces zero red anywhere.
  • F14 — killed agents leave phantom running runs forever, rendered as a live elapsed-ticking banner in the TUI dashboard hours later.
  • F30 — verify mismatches ring once, then everything carries on. The tamper drill was caught perfectly, but the next scheduled sync pushed to the flagged destination as if nothing happened; the alarm lives only in a scrolled-away runs row.

Plan

  • Capture rclone stderr (bounded) into the run row's error and the scheduler error log; fix the errors=0-on-failure contradiction.
  • Preflight refusals (marker, kopia-connect, layout guard) write a run row with a new refused status — schema migration + snapshot regen.
  • Agent startup reaps its own orphaned running runs to aborted.
  • Verify mismatch latches a per-destination alarm state (new table or destination-state column), surfaced in runs/TUI until an operator clears it (squirrel verify passing clean clears it automatically or via an explicit ack — decide in the PR).

One theme: every failure becomes a run row, every abnormal condition a standing state.

See design/friction-log.md for the live evidence per finding.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions