agent: per-destination scheduler workers + transfer timeouts

[mbertschler]

Problem (friction log F25, S1 — confirmed live, twice)

The agent's scheduler is one serial worker with no per-run time bound. When the walk's S3 endpoint stopped accepting writes, the in-flight rclone hung and everything froze — no cloudbox pushes, no NAS→HTPC peer-sync, no kopia snapshots, across all volumes — for 12.6 minutes until the run failed; during a later full endpoint outage, indefinitely, until an operator killed the rclone process by hand. No rclone timeout fired, no run failed, and the only trace was a kicked line with no finished. In the reference household this means a dark cloud destination silently stops local NAS→HTPC replication too.

Decision (Martin, 2026-07-24 — option 4b: one PR)

Go straight to per-destination worker queues plus timeouts in a single change:

• One in-flight run per destination; different destinations run concurrently (bounded overall parallelism). A slow offsite can never delay LAN pairs.
• Per-run stall handling: --contimeout/--timeout on every rclone invocation, plus a squirrel-side wall-clock/no-progress bound that kills the child, fails the run with a real error (composes with runs: every failure and refusal becomes a run row or a standing state #157's stderr capture), and lets the pair retry next cadence.
• In-flight visibility: expose "running since HH:MM" per pair (consumed by status: per volume×destination coverage + durability panel (CLI, then TUI in this issue) #159's status grid; the TUI already shows in-flight runs — make them per-pair and truthful after runs: every failure and refusal becomes a run row or a standing state #157's orphan reaping).

Notes

• Keep per-volume index runs serialized against that volume's syncs (pre-sync index ordering must hold); the concurrency unit is the destination, not the volume.
• Scheduler-log discipline (kicked/skipped/finished/error) stays; skipped reason="in-flight sync run" becomes per-pair.
• Acceptance: re-run walk scenario "endpoint goes dark" on the testbed — other destinations keep syncing, the stuck pair fails within the bound with a diagnosable error.

See design/friction-log.md F25.

[mbertschler]

Landed by #181, which merged into the claude/issue-157-run-rows-standing-states branch and reached main with #174 — the stacked base is why the Closes #160 keyword never auto-closed this.

Verified on main (09c2760): per-destination worker queues in agent/dispatcher.go + agent/scheduler.go, with per-volume index runs still serialized against that volume's syncs, rclone timeouts and the squirrel-side stall bound, and per-pair in-flight visibility feeding #159's status grid.

Closing as completed.

---

Generated by Claude Code