feat: add autodeploy alerting, NAS deploy sync, and dispatch guardrails#625
Open
inivosdigital wants to merge 46 commits into
Open
feat: add autodeploy alerting, NAS deploy sync, and dispatch guardrails#625inivosdigital wants to merge 46 commits into
inivosdigital wants to merge 46 commits into
Conversation
inivosdigital
force-pushed
the
fm/ship-firstmate-autodeploy-alert-relay
branch
2 times, most recently
from
July 16, 2026 02:56
3b4bbeb to
dcb5dd5
Compare
added 28 commits
July 15, 2026 23:30
Two treehouse worktrees of the same project share a leaf directory basename, so Compose's default project-naming collides across them on container/volume/network names. fm-spawn.sh now derives a stable name from the worktree's own pool-slot path and drops it at .treehouse-compose-project (git-excluded, like the other turn-end pointer files), recorded in meta as compose_project=. fm-brief.sh tells ship/scout crewmates to consume it on every docker compose invocation and to avoid fixed host ports as a second collision vector.
…against backgrounded server call
…r-compose note to fm-brief.sh header comment
…compose isolation test
Claude crewmates and secondmates are separate claude processes launched by fm-spawn.sh; they never inherit the primary's .claude/settings.local.json, so they only compact at the hard context wall instead of proactively. Add CLAUDE_CODE_AUTO_COMPACT_WINDOW=200000 to the claude launch template. The var is capped at the model's real context window, so a fixed 200000 lands auto-compaction around 200k regardless of whether the spawned model's window is 200k or (as verified live for --model sonnet on this account, which carries the 1M-context beta) ~1M. The primary's own two-var form (WINDOW=1000000 + PCT_OVERRIDE=20) assumes a 1M window and is not safe to copy blindly for a smaller-window model/account combination. grok, codex, opencode, and pi launches are left untouched: grok's binary is not verified to honor Claude Code env vars, and the other three ignore them harmlessly but should not carry a var meant for a harness they aren't.
Bump CLAUDE_CODE_AUTO_COMPACT_WINDOW from 200000 to 300000 for claude crewmates and secondmates, matching the primary session's own updated target. The override is capped at the model's real context window, so it is a no-op below 300000 and only takes effect above it; today's standard --model sonnet crewmate spawn resolves to the ~1M-context beta, well above the new threshold. Updated the fm-spawn.sh header comment, launch_template() comment, and the harness-adapters SKILL.md fact to describe this conditional behavior instead of the "regardless of window" framing that only held at the old 200000 value.
content_in_default() only ever diffed a branch against the fetched refs/remotes/origin/$name, so a no-mistakes-mode task whose branch was merged into the LOCAL default branch by hand (e.g. after its upstream PR was closed unmerged) was never recognized as landed and teardown refused it. It now also checks the local refs/heads/$name ref.
…tests content_in_default()'s diff-based comparison (from local main's own prior rewrite) no longer depends on git merge-tree --write-tree, so the two tests that used to skip themselves on older git now run and pass here.
Extends config/crew-dispatch.json's schema with model/effort/ultracode axes per the resource-tiering design: docs/examples/crew-dispatch.json gains rules for the firstmate-repo supervision-backbone carve-out, safety-critical/security/migration work (opus/xhigh, ultracode-flagged), and architectural/product tradeoffs (opus/xhigh). Adds the four hardened guardrails as mechanical checks rather than prose aspirations: - bin/fm-tier-guard.sh: firstmate-observable escalation triggers (diff size and elapsed time vs. the trivial tier's envelope, plus a general heavy-scale ceiling for any tier). - bin/fm-risk-tripwire.sh: a keyword/path trip-wire against a task's brief and diff, independent of the natural-language dispatch match. - bin/fm-ultracode-guard.sh: flag/reviewed/check subcommands that mechanically confirm a genuinely independent second pass ran before an ultracode-flagged task reaches PR-ready. - A documented model-verification gate in AGENTS.md section 4, extending the existing unverified-harness discipline to model names. Verification tiering (light/standard/heavy no-mistakes runs) is deliberately out of scope here and left for a follow-up change.
… distinct exit codes
…d distinct exit codes
… guardrail coverage
…herdr server-race
The read-tree portability rewrite restructured content_in_default() into an if/elif that checks exactly one candidate ref: the fetched origin ref when a remote exists, falling back to the local ref only when there is no remote at all. That silently dropped the case a no-mistakes-mode project with a closed, unmerged PR but work fast-forwarded into local main by hand - the fetched origin ref exists but does not contain the work, so the local ref is never even checked. Restore checking both candidates in sequence via a shared content_matches_ref helper, matching the pre-rewrite behavior tests/fm-teardown.test.sh's local-land regression test expects.
A merged PR does not, by itself, reach the live site: deployed apps run from a separate checkout under /mnt/nas/experiments/<name>/, managed by pm2, independent from the projects/<name> clone used for development. This gap required a manual pull+restart to notice and fix. bin/fm-nas-deploy-sync.sh fast-forwards a project's live NAS checkout and restarts its pm2 process(es), looked up from a captain-private data/nas-deployments.md mapping (best-effort, non-fatal, mirrors fm-fleet-sync.sh's dirty/diverged safety exactly). bin/fm-teardown.sh now calls it automatically after a landed ship task, including local-only merges.
…-backend.md with the NAS auto-deploy-sync feature and cmux liveness fix.
Add config/critical-services (local, gitignored): one systemd unit name per line, blank lines and full-line "#" comments ignored. bin/fm-bootstrap.sh reads it in the detect-only diagnostics and prints one read-only SERVICE_FAILED line per failed unit via systemctl is-failed (no root), with a "failed since" timestamp when available. An absent or empty file, or a host without systemctl, is a no-op; nothing is ever auto-restarted. Wire it into AGENTS.md's config list and section 3's diagnostic-handling list, docs/configuration.md, a copyable docs/examples/critical-services, and tests.
Swap remotes on this working copy so origin is the captain's fork and upstream is the read-only parent template, and keep the fork alive: - fm-merge-local.sh: after a local fast-forward, best-effort push the default branch to origin when it exists, so a push-backed local-only default branch never silently drifts behind local main. Skipped silently for no-remote local-only projects. - fm-bootstrap.sh: always-on UPSTREAM_DRIFT diagnostic reporting how far local main diverges from upstream/main in both directions plus days-since-merge-base, escalating wording past 30 commits behind or a 10-day-old merge-base. The upstream fetch lives only in the locked mutating fleet-sync sweep; the count-and-report runs unconditionally from the already-present upstream ref, never fetching in the detect-only path. - Document the local-only landing pattern and the opportunistic, non-blocking upstream PR in AGENTS.md, the UPSTREAM_DRIFT trigger in the bootstrap-diagnostics skill, and the inverted remote layout in CONTRIBUTING.md.
…late Post-swap origin is the captain's fork, so /updatefirstmate only converges the fleet onto that fork (a no-op on the primary source of truth) and never pulls the read-only upstream template. Folding upstream-template improvements into local main is the separate reviewed reconciliation task surfaced by the UPSTREAM_DRIFT bootstrap line, not a self-update. Note it inline in AGENTS.md section 12 and in the updatefirstmate skill.
The turn-end guard blocked essentially every turn during a legitimate watcher re-arm, not just a genuine supervision gap. fm_watcher_healthy requires a live, identity-matched watcher lock with zero grace, but a backgrounded re-arm needs a brief moment, roughly 0.25-0.3s measured, more under load, to fork the new watcher and register its lock after the old one released it. The Stop hook fired inside that gap on essentially every re-arm, so it blocked with no real supervision lapse. The pull-based warning banner tolerates this same gap because it checks only beacon freshness with a 300s grace. The turn-end guard's sharper live-lock check had no equivalent tolerance. Replace the single fm_watcher_healthy check with a bounded poll: retry every 0.2s, mirroring the watcher-arm script's own confirm loop, and block only once a deadline passes still unhealthy. The deadline defaults to FM_ARM_CONFIRM_TIMEOUT (10s), the same budget the arm gives itself to confirm, and is independently overridable via FM_TURNEND_ARM_WAIT. A watcher that is genuinely missing, dead, or identity-mismatched for the whole window still blocks; it now takes up to that deadline instead of blocking on the first check. This intentionally slows the truly-blind path (no arm in flight at all), which is the documented tradeoff. Update docs/turnend-guard.md's Shared Predicate section, the contract's one authoritative statement, to describe the bounded poll instead of an immediate block. Testing: tests/fm-turnend-guard.test.sh's test_hook_runs_fast now pins the bounded-poll latency instead of asserting a near-instant return, and a new test_hook_allows_watcher_that_registers_mid_poll confirms a lock that registers partway through the window is let through rather than blocked. Other block-path tests default to a short override so the suite stays fast; ran the lint gate and the full behavior suite (41 tests, all passing).
…ndoff tasks-axi gate
…unch signal-race fix
…ard.md with turn-end-guard poll fix
Add a config/autodeploy-logs sweep to fm-watch.sh's heartbeat fleet-scan. Each configured status log's last line is read on every heartbeat, and a failure (an ALERT rollup or a STUCK:/FAILED: line in fleet-sync's convention) surfaces once as a check wake carrying that line until it clears; a later healthy run re-arms the alarm. Mirrors bootstrap's critical_services_check but fires periodically, so a deploy failure between sessions is caught instead of waiting for the next session start. Absent config, a healthy "ok" last line, or a missing/unreadable log (a NAS hiccup) are all silent and non-fatal, so the check stays cheap enough to run every heartbeat without spurious alarms. Config is gitignored and absent by default, so it is a no-op for fleets that do not opt in.
Factor the failure predicate into the shared bin/fm-autodeploy-lib.sh (fm_autodeploy_line_failed), sourced by both the fm-watch.sh sweep and a new fm-bootstrap.sh autodeploy_logs_check, so an idle fleet catches a deploy failure at the next session start exactly as critical-services already does. One owner for the match set; docs and colocated tests updated to match.
On a hung NAS mount the tail read of an autodeploy status log could block the watcher heartbeat (and the bootstrap check). Wrap the read in the shared fm-autodeploy-lib.sh behind a bounded timeout and treat a timeout as unreadable, preserving the existing fail-quiet behavior for missing or unreadable logs. Both read sites go through the one shared helper.
…t; document timeout knob
…todeploy-lib.sh, and watcher's autodeploy sweep
The rebase onto the compressed AGENTS.md structure (upstream kunchenguid#626) kept this branch's own feature documentation (autodeploy alerting, NAS deploy sync, dispatch guardrails, and the read-only-upstream/local-only landing contract) by folding it into the new compressed sections instead of reverting the compression or duplicating the old layout.
inivosdigital
force-pushed
the
fm/ship-firstmate-autodeploy-alert-relay
branch
from
July 16, 2026 03:40
dcb5dd5 to
9d6ba4a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
The developer (the "captain" in this firstmate orchestration session) asked for a full status and roadmap report across their 12-project portfolio, then directed the assistant to take over as the prime supervising session after terminating a stale/orphaned session. They asked to resume three previously parked validation jobs (an auto-compaction change, a falsework runner/scheduler feature, and a tiering system change) plus dispatch a crewmate to redeploy the live untethered companion app to current main and restart its dead data-harvester process. They later asked which model/effort tier each crewmate was running on, requested an expert UX/UI crewmate be assigned to overhaul carscanner's navigation and visual design so it looks professional and demo-ready, and explicitly approved merging a previously reported fix for a crewmate-steer message-delivery glitch (though the assistant discovered the branch needed rebuilding before merging). Constraints emphasized throughout include: never merge without explicit approval, avoid running too many heavy validations concurrently due to tight machine memory (prior OOM crashes), and keep supervision continuously active ("no turn ends blind") while awaiting the captain's decisions on several blocked/paused tasks.
What Changed
fm-autodeploy-lib.sh) and a one-time warning fires when no timeout mechanism is available so alerting doesn't silently go inert.fm-nas-deploy-sync.sh), with fetch/timeout guarding and pm2 cluster-status aggregation.fm-risk-tripwire.sh,fm-tier-guard.sh,fm-ultracode-guard.sh) and quota-balanced crew-dispatch selection, alongside supporting fixes to teardown landed-work detection, turnend-guard watcher re-arm polling, per-worktree docker-compose isolation, and claude crewmate auto-compact tuning.Risk Assessment
✅ Low: This round is a small, well-scoped follow-up fix (68 lines across 6 files) that precisely implements the two auto-fix instructions from round 1: a one-time AUTODEPLOY_INERT bootstrap diagnostic and documentation of FM_AUTODEPLOY_LOG_READ_TIMEOUT. The new command-v-shadowing detection logic is correct, consistent with the existing critical_services_check/no-systemctl test pattern, properly wired into AGENTS.md and the bootstrap-diagnostics skill, and covered by two new tests (content present vs. comment-only) that both pass sound reasoning checks.
Testing
test
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
⏭️ **Rebase** - skipped
.agents/skills/afk/SKILL.md- branch carries 42 commit(s) that exist on your local main branch but were never pushed to origin/main; rebasing would bundle this unrelated work (66 file(s)) into the PR:Push main to origin, or rebase your branch onto origin/main, before gating.
🔧 **Review** - 4 issues found → auto-fixed ✅
bin/fm-autodeploy-lib.sh:25- When none of timeout, gtimeout, or perl are on PATH, fm_autodeploy_read_last_line always returns 1, so both autodeploy_scan and autodeploy_logs_check silently no-op forever on that host. Unlike other bootstrap gaps in this repo (MISSING: tool diagnostics for treehouse/no-mistakes/tasks-axi, NEEDS_GH_AUTH, etc.), there is no diagnostic telling the captain the whole autodeploy-alert feature is inert, so a deploy could fail indefinitely with zero alerting and no clue why. Worth at least a one-time bootstrap diagnostic line when config/autodeploy-logs is present but no timeout mechanism is available.bin/fm-autodeploy-lib.sh:24- The perl fork/exec/process-group/SIGALRM fallback (used only when timeout and gtimeout are both absent) is never exercised by any added test - every test runs on a host where coreutilstimeoutis present, so this branch's non-trivial signal/process-group handling is unverified.bin/fm-watch.sh:471- autodeploy_marker_path collapses every non-alphanumeric character in the log path to '' viatr -c 'A-Za-z0-9' '_'. Two distinct configured log paths that differ only in punctuation (e.g. /var/log/app-a/status.log vs /var/log/app_a/status.log) hash to the same surfaced-marker file, so their dedupe/re-arm state would cross-contaminate. Low likelihood given user-authored config with few entries, but worth a hash-based marker name if this ever needs hardening.bin/fm-autodeploy-lib.sh:18- FM_AUTODEPLOY_LOG_READ_TIMEOUT is a new override knob but isn't added to docs/configuration.md's environment-variable timeout list, where every other timeout knob in this repo (FM_NAS_SYNC_TIMEOUT, FM_CHECK_TIMEOUT, FM_UPSTREAM_FETCH_TIMEOUT, etc.) is documented.🔧 Fix: bootstrap: warn when autodeploy alerting is inert; document timeout knob
✅ Re-checked - no issues remain.
🔧 **Test** - 1 issue found → auto-fixed ✅
command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"🔧 Fix: placeholder - awaiting background test run completion
✅ Re-checked - no issues remain.
command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"test✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.