feat: persistent watcher daemon — supervision survives walking away (PRD #10)

CLAUDE.md

@@ -34,6 +34,13 @@ Bach is a CLI-first personal agentic development session launcher that turns dai
 - `/board` (REPL) - Kanban board now shows liveness dots (●/◐/✓) and observer summary lines on executing/qa cards; `⚑ needs you` marker when observer flags needs_human
 - `/show <tN>` (REPL) - Task detail now includes an Observer section (summary, confidence, needs_human, session state) when the observer has data
 
+### PRD#10 — Persistent Watcher Daemon
+- `uv run bach daemon run` - Start the foreground supervisor loop; reconciles live task-linked sessions vs supervised watcher processes every `daemon-reconcile-seconds` (default 30); single-instance guarded; exits cleanly on SIGTERM/SIGINT
+- `uv run bach daemon status` - Report whether the daemon is running (pid) and list the supervised artifact set (derived from the process table)
+- `uv run bach daemon stop` - Send SIGTERM to the recorded daemon pid and clear the pidfile
+- `uv run bach daemon install` - Install `com.bach.daemon` as a macOS LaunchAgent (RunAtLoad + KeepAlive); prints `bach daemon run` fallback if launchctl fails
+- `uv run bach daemon uninstall` - Unload and remove the LaunchAgent plist; idempotent
+
 ## File References
 - `docs/internal/DISCOVERY.md` - real goal and scope
 - `docs/internal/PRD.md` - requirements

docs/adr/017-persistent-watcher-daemon.md

@@ -0,0 +1,239 @@
+# ADR-017 — Persistent Watcher Daemon (PRD #10)
+
+**Status:** Accepted
+**Date:** 2026-06-13
+**Extends:** ADR-015 (session observability, watcher-per-session run-and-exit),
+             ADR-016 (loop control, AFK runner).
+
+## Context
+
+ADR-015 introduced a per-session watcher (`session_watcher.py`) that runs as a
+foreground, run-and-exit process. It is spawned in one of two ways:
+
+1. **watch_on_launch** — `task_service.launch_task` spawns a detached watcher
+   when the user has set `watch_on_launch: on`. This is per-launch, per-user.
+2. **`bach sessions watch`** — the user manually runs the watcher for a named
+   session.
+
+Both mechanisms share the same limitation: if the user restarts their machine,
+closes their terminal, or starts a new session from a different terminal window,
+no watcher is running and observations go dark. There is no process that
+reconciles the desired supervision state against what is actually running.
+
+PRD #10 adds a **persistent daemon** that closes this gap. The daemon is a
+long-running supervisor that continuously ensures one watcher is alive for every
+live, task-linked session — without requiring the user to manually spawn or
+re-spawn watchers.
+
+Two design questions arose that required explicit decisions:
+
+1. **What is the supervision state model?** Naively, a bookkeeping file could
+   record which artifacts have a running watcher. But a state file introduces
+   failure modes: it can go stale after a crash, diverge from reality, and
+   cannot be queried atomically with process existence checks.
+
+2. **Who terminates supervised watchers?** The daemon could actively kill
+   watchers when their session ends. But watchers already self-terminate on
+   session end (via spool events and transcript EOF). An active kill path would
+   duplicate responsibility and risk killing a watcher in mid-observation.
+
+## Decision
+
+### 1. Supervisor-of-subprocesses: thin reconcile loop
+
+The daemon runs a **single-threaded synchronous reconcile loop** with no
+threads and no asyncio. Each cycle:
+
+1. Compute the **desired set** — artifact paths whose sessions are live and
+   task-linked (not ended, not in a terminal status).
+2. Compute the **supervised set** — artifact paths that currently have a live
+   `session-watch` child process (see §2 below).
+3. Compute the **gap** = desired − supervised.
+4. Call `spawn_watcher(artifact)` for each artifact in the gap.
+5. Sleep `daemon_reconcile_seconds` (default 30 s).
+
+Each session watcher is a separate detached process (`bach internal
+session-watch <artifact>`) — the existing `session_watcher.py` run-and-exit
+process, completely unchanged. The daemon is a thin coordination layer above
+the already-working watcher processes; it adds no new observation logic.
+
+Rationale for single-threaded synchronous design:
+- One cycle is O(n) in the number of active sessions, typically < 5 processes
+  for a single developer. No concurrency benefit exists at this scale.
+- Threads or asyncio would add cancellation complexity, exception propagation
+  issues, and test infrastructure overhead that far exceeds the value.
+- The supervisor loop is fully injectable (all side-effecting deps are
+  keyword-only args to `run_supervisor`) so it is trivially unit-testable with
+  fakes and no real subprocess/time involvement.
+
+### 2. Supervision state is derived from the OS process table
+
+**The single source of truth for "which artifacts are supervised" is the OS
+process table, not a bookkeeping file.**
+
+`list_supervised_artifacts()` (in `runtimes/watcher_processes.py`) calls
+`ps -axo pid,command`, scans for lines containing the unique marker string
+`"internal session-watch"`, and extracts the artifact path argument. The result
+is a `set[Path]` of currently-watched artifacts.
+
+This is the load-bearing invariant. The reasons it was chosen over a state
+file:
+
+- **Auto-dedup vs `watch_on_launch`.** If the user has `watch_on_launch: on`
+  and the daemon is also running, the per-launch watcher and the daemon's
+  watcher would both try to supervise the same artifact. The ps-derived
+  supervised set naturally deduplicates: if `watch_on_launch` already spawned
+  a watcher, it appears in `ps` and the daemon's reconcile skips it. No
+  coordination between the two spawn paths is needed.
+- **Restart-safe.** If the daemon crashes and restarts, the supervised set is
+  reconstructed from the actual running processes. Orphaned watchers from the
+  previous daemon run are discovered and counted; there is no orphan list to
+  reconcile with.
+- **No state file to corrupt.** A bookkeeping file can go out of sync with
+  reality. `ps` cannot.
+
+The consequence of this design is the **no-kill-path invariant**: because the
+supervised set is derived from `ps`, the daemon does not track watcher PIDs
+individually. It sees "artifact X has a watcher" but not "watcher PID is
+12345". There is therefore no mechanism to selectively terminate a specific
+watcher. Watchers self-terminate when their session ends (via spool events or
+transcript EOF from ADR-015 §3). Ended or terminal-status artifacts drop from
+the desired set; they are simply ignored on the next cycle. The watcher dies
+on its own, and on the cycle after that, it disappears from `ps` and is not
+missed.
+
+### 3. Crash backoff
+
+If a watcher crashes immediately after spawning (e.g. bad artifact state, disk
+error), the reconcile loop would respawn it every cycle, hammering the process
+table. The daemon uses per-artifact exponential backoff to prevent this.
+
+After each spawn attempt (success or failure), a backoff entry is recorded in
+an **in-memory dict** (not a file). The next spawn for that artifact is
+suppressed until the cooldown window expires. The window starts at 10 s and
+doubles on each successive crash, capped at 5 minutes.
+
+The backoff state is reset on daemon restart because it is in-memory. This is
+deliberate: a fresh daemon restart (via KeepAlive or manual `bach daemon run`)
+gives every artifact a clean slate. The KeepAlive restart boundary naturally
+bounds how long stale backoff state persists.
+
+### 4. Single-instance guard
+
+`acquire()` in `services/daemon_lifecycle.py` reads a pidfile at
+`~/.bach/daemon.pid`, checks liveness of the recorded pid via `os.kill(pid, 0)`,
+and either:
+- Writes our PID and returns success.
+- Returns a refusal with the live PID.
+- Reclaims a stale pidfile (dead PID or corrupt content) and returns success.
+
+Stale pidfile reclaim is automatic so a crashed daemon never permanently
+blocks a restart.
+
+### 5. macOS launchd integration
+
+`bach daemon install` writes a LaunchAgent plist to `~/Library/LaunchAgents/`
+and loads it via `launchctl load`. The plist uses:
+
+- `RunAtLoad: true` — starts the daemon on user login without manual action.
+- `KeepAlive: true` — launchd restarts the daemon if it crashes.
+- `StandardOutPath` / `StandardErrorPath` — daemon output goes to
+  `~/.bach/logs/daemon.stdout.log` / `daemon.stderr.log`.
+
+On `launchctl` failure (non-macOS, SIP restrictions, sandboxed environment),
+the plist is still written and a fallback command (`bach daemon run`) is
+printed so the user always has a manual path.
+
+`bach daemon uninstall` unloads and removes the plist. Both are idempotent.
+
+## Known limitations / accepted trade-offs
+
+*These are deliberate design choices, not bugs.*
+
+**(a) Pidfile acquire has a TOCTOU window.**
+Between reading the pidfile and writing it, another process could write its own
+PID (a race). This is accepted for a single-user macOS development tool:
+the only realistic concurrent caller is the user running `bach daemon run`
+twice in rapid succession, which is a user error that results in one of them
+refusing cleanly. Using `flock()` would add platform-specific code and a
+dependency on `fcntl` with no real safety gain for a single-user tool.
+
+**(b) `ps` may truncate long artifact paths.**
+On some platforms, `ps` truncates command lines beyond ~1000 characters. A
+task artifact with a path longer than this would appear in `ps` with a
+truncated command string, causing `list_supervised_artifacts` to miss it and
+the daemon to attempt a duplicate spawn. At most one extra spawn would occur:
+on the following cycle, the newly-spawned watcher appears in `ps` with the
+full path (it was spawned from this machine with the full argv), so it is
+detected and the artifact is no longer in the gap. The consequence is one
+redundant spawn, not an infinite loop. This is accepted over using
+platform-specific proc APIs (Linux `/proc/<pid>/cmdline`, macOS
+`libproc.h`) which would fragment the codebase for a purely theoretical
+edge case.
+
+**(c) Backoff state is in-memory; it does not clear when a watcher recovers.**
+If a watcher crashes once, accumulates a backoff cooldown, then succeeds on the
+next attempt, the backoff entry persists in memory until the daemon restarts.
+On the next restart the backoff dict is empty, so the artifact gets a clean
+slate immediately. The `KeepAlive` restart boundary means staleness is bounded
+by the daemon restart frequency. This is conservative-by-design: a slightly
+extended cooldown after a recovery is preferable to hammering a flaky artifact
+with no delay.
+
+## Consequences
+
+- **`bach daemon run` is a foreground command.** It runs until SIGTERM or
+  SIGINT. The launchd LaunchAgent is the intended production deployment;
+  foreground is for development and testing.
+
+- **No daemon required for basic use.** The existing `watch_on_launch` path
+  and `bach sessions watch` remain fully functional. The daemon is an opt-in
+  enhancement for persistent supervision.
+
+- **`daemon_reconcile_seconds` is a new config knob** (default 30). Lower
+  values mean faster gap detection but more `ps` calls. Tunable via
+  `bach config set daemon-reconcile-seconds <N>`.
+
+- **Watcher logs accumulate.** Each watcher's stdout/stderr is appended to
+  `~/.bach/logs/watcher-<stem>.log`. The daemon's own output goes to
+  `daemon.stdout.log` / `daemon.stderr.log` under the same directory. GC is
+  a future concern.
+
+- **SSH check-in workflow is pull-only.** The daemon writes observer verdicts
+  to task artifacts and the board. There are no push notifications. A user
+  returning over SSH runs `bach sessions list`, `/board`, or `bach daemon status`
+  to pull current state. See `docs/how-to/run-the-daemon.md`.
+
+## Alternatives rejected
+
+- **Daemon with in-process threads / asyncio.** Threading or asyncio would
+  allow concurrent watcher goroutines but introduce cancellation complexity,
+  shared-state races, and test overhead that far exceeds the benefit for a
+  single-developer tool running O(5) concurrent sessions.
+
+- **Bookkeeping file as supervision state.** A JSON or YAML file recording
+  watched artifacts would be simpler to query but would go stale after crashes
+  and require its own GC and consistency maintenance. The process table is
+  always fresh and never lies.
+
+- **Daemon kills watchers on session end.** Adding a kill path would require
+  tracking individual watcher PIDs and handling races between kill and natural
+  self-termination. Watchers already self-terminate reliably (ADR-015 §3);
+  adding a redundant kill path would increase complexity with no correctness
+  benefit.
+
+- **flock() for single-instance guard.** Adds `fcntl` import (POSIX-only),
+  held-lock semantics, and platform-specific cleanup. The pidfile approach with
+  stale reclaim gives identical single-user safety at half the complexity.
+
+## References
+
+- ADR-015: `adr/015-session-observability-codex-parity.md`
+- ADR-016: `adr/016-loop-control.md`
+- Implementation: `src/bach/services/daemon_supervisor.py` (reconcile + run_supervisor + backoff)
+- Implementation: `src/bach/runtimes/watcher_processes.py` (spawn_watcher, list_supervised_artifacts)
+- Implementation: `src/bach/services/daemon_lifecycle.py` (pidfile single-instance guard)
+- Implementation: `src/bach/services/launchd_service.py` (plist render + install/uninstall)
+- Implementation: `src/bach/cli/daemon_cmd.py` (run/status/stop/install/uninstall)
+- Config: `src/bach/config/settings.py` (`daemon_reconcile_seconds`)
+- How-to: `docs/how-to/run-the-daemon.md`