feat: add mechanical dispatch guardrails and NAS auto-deploy-sync#624
Open
inivosdigital wants to merge 47 commits into
Open
feat: add mechanical dispatch guardrails and NAS auto-deploy-sync#624inivosdigital wants to merge 47 commits into
inivosdigital wants to merge 47 commits into
Conversation
inivosdigital
force-pushed
the
fm/fix-fm-secondmate-safety-tmp-leak
branch
2 times, most recently
from
July 16, 2026 02:50
7096746 to
3bb7741
Compare
added 28 commits
July 15, 2026 23:29
Two treehouse worktrees of the same project share a leaf directory basename, so Compose's default project-naming collides across them on container/volume/network names. fm-spawn.sh now derives a stable name from the worktree's own pool-slot path and drops it at .treehouse-compose-project (git-excluded, like the other turn-end pointer files), recorded in meta as compose_project=. fm-brief.sh tells ship/scout crewmates to consume it on every docker compose invocation and to avoid fixed host ports as a second collision vector.
…against backgrounded server call
…r-compose note to fm-brief.sh header comment
…compose isolation test
Claude crewmates and secondmates are separate claude processes launched by fm-spawn.sh; they never inherit the primary's .claude/settings.local.json, so they only compact at the hard context wall instead of proactively. Add CLAUDE_CODE_AUTO_COMPACT_WINDOW=200000 to the claude launch template. The var is capped at the model's real context window, so a fixed 200000 lands auto-compaction around 200k regardless of whether the spawned model's window is 200k or (as verified live for --model sonnet on this account, which carries the 1M-context beta) ~1M. The primary's own two-var form (WINDOW=1000000 + PCT_OVERRIDE=20) assumes a 1M window and is not safe to copy blindly for a smaller-window model/account combination. grok, codex, opencode, and pi launches are left untouched: grok's binary is not verified to honor Claude Code env vars, and the other three ignore them harmlessly but should not carry a var meant for a harness they aren't.
Bump CLAUDE_CODE_AUTO_COMPACT_WINDOW from 200000 to 300000 for claude crewmates and secondmates, matching the primary session's own updated target. The override is capped at the model's real context window, so it is a no-op below 300000 and only takes effect above it; today's standard --model sonnet crewmate spawn resolves to the ~1M-context beta, well above the new threshold. Updated the fm-spawn.sh header comment, launch_template() comment, and the harness-adapters SKILL.md fact to describe this conditional behavior instead of the "regardless of window" framing that only held at the old 200000 value.
content_in_default() only ever diffed a branch against the fetched refs/remotes/origin/$name, so a no-mistakes-mode task whose branch was merged into the LOCAL default branch by hand (e.g. after its upstream PR was closed unmerged) was never recognized as landed and teardown refused it. It now also checks the local refs/heads/$name ref.
…tests content_in_default()'s diff-based comparison (from local main's own prior rewrite) no longer depends on git merge-tree --write-tree, so the two tests that used to skip themselves on older git now run and pass here.
Extends config/crew-dispatch.json's schema with model/effort/ultracode axes per the resource-tiering design: docs/examples/crew-dispatch.json gains rules for the firstmate-repo supervision-backbone carve-out, safety-critical/security/migration work (opus/xhigh, ultracode-flagged), and architectural/product tradeoffs (opus/xhigh). Adds the four hardened guardrails as mechanical checks rather than prose aspirations: - bin/fm-tier-guard.sh: firstmate-observable escalation triggers (diff size and elapsed time vs. the trivial tier's envelope, plus a general heavy-scale ceiling for any tier). - bin/fm-risk-tripwire.sh: a keyword/path trip-wire against a task's brief and diff, independent of the natural-language dispatch match. - bin/fm-ultracode-guard.sh: flag/reviewed/check subcommands that mechanically confirm a genuinely independent second pass ran before an ultracode-flagged task reaches PR-ready. - A documented model-verification gate in AGENTS.md section 4, extending the existing unverified-harness discipline to model names. Verification tiering (light/standard/heavy no-mistakes runs) is deliberately out of scope here and left for a follow-up change.
… distinct exit codes
…d distinct exit codes
… guardrail coverage
…herdr server-race
The read-tree portability rewrite restructured content_in_default() into an if/elif that checks exactly one candidate ref: the fetched origin ref when a remote exists, falling back to the local ref only when there is no remote at all. That silently dropped the case a no-mistakes-mode project with a closed, unmerged PR but work fast-forwarded into local main by hand - the fetched origin ref exists but does not contain the work, so the local ref is never even checked. Restore checking both candidates in sequence via a shared content_matches_ref helper, matching the pre-rewrite behavior tests/fm-teardown.test.sh's local-land regression test expects.
A merged PR does not, by itself, reach the live site: deployed apps run from a separate checkout under /mnt/nas/experiments/<name>/, managed by pm2, independent from the projects/<name> clone used for development. This gap required a manual pull+restart to notice and fix. bin/fm-nas-deploy-sync.sh fast-forwards a project's live NAS checkout and restarts its pm2 process(es), looked up from a captain-private data/nas-deployments.md mapping (best-effort, non-fatal, mirrors fm-fleet-sync.sh's dirty/diverged safety exactly). bin/fm-teardown.sh now calls it automatically after a landed ship task, including local-only merges.
…-backend.md with the NAS auto-deploy-sync feature and cmux liveness fix.
Add config/critical-services (local, gitignored): one systemd unit name per line, blank lines and full-line "#" comments ignored. bin/fm-bootstrap.sh reads it in the detect-only diagnostics and prints one read-only SERVICE_FAILED line per failed unit via systemctl is-failed (no root), with a "failed since" timestamp when available. An absent or empty file, or a host without systemctl, is a no-op; nothing is ever auto-restarted. Wire it into AGENTS.md's config list and section 3's diagnostic-handling list, docs/configuration.md, a copyable docs/examples/critical-services, and tests.
Swap remotes on this working copy so origin is the captain's fork and upstream is the read-only parent template, and keep the fork alive: - fm-merge-local.sh: after a local fast-forward, best-effort push the default branch to origin when it exists, so a push-backed local-only default branch never silently drifts behind local main. Skipped silently for no-remote local-only projects. - fm-bootstrap.sh: always-on UPSTREAM_DRIFT diagnostic reporting how far local main diverges from upstream/main in both directions plus days-since-merge-base, escalating wording past 30 commits behind or a 10-day-old merge-base. The upstream fetch lives only in the locked mutating fleet-sync sweep; the count-and-report runs unconditionally from the already-present upstream ref, never fetching in the detect-only path. - Document the local-only landing pattern and the opportunistic, non-blocking upstream PR in AGENTS.md, the UPSTREAM_DRIFT trigger in the bootstrap-diagnostics skill, and the inverted remote layout in CONTRIBUTING.md.
…late Post-swap origin is the captain's fork, so /updatefirstmate only converges the fleet onto that fork (a no-op on the primary source of truth) and never pulls the read-only upstream template. Folding upstream-template improvements into local main is the separate reviewed reconciliation task surfaced by the UPSTREAM_DRIFT bootstrap line, not a self-update. Note it inline in AGENTS.md section 12 and in the updatefirstmate skill.
fm_test_tmproot installed its EXIT trap and appended to FM_TEST_CLEANUP_DIRS from inside a command-substitution subshell, so neither ever reached the calling script: the trap fired (and deleted the just-created dir) the instant the subshell exited, and the array mutation never propagated back. Every run of the secondmate suites therefore leaked its temp root, growing worse under SIGKILL (e.g. the OOM killer) since no trap can run for that signal regardless. Move registration to an on-disk manifest, which survives the subshell boundary, and install the EXIT trap once at source time in the real top-level shell. Add a startup sweep for leftover <prefix>.<pid>.XXXXXX roots whose owning PID is no longer alive, as the actual safety net for the SIGKILL case, and call it from both secondmate suites before they claim their own root.
…ge-local push, and stale comments
inivosdigital
force-pushed
the
fm/fix-fm-secondmate-safety-tmp-leak
branch
from
July 16, 2026 03:30
3bb7741 to
af4e536
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
The captain (user) asked their firstmate orchestration agent for a status report on all 12 projects and their roadmaps, then directed it to take over as the prime supervising session (terminating a stale/orphaned session) after finding the fleet lock held by an unresponsive session. They asked to resume three previously-parked validation jobs (firstmate auto-compaction, falsework runner/scheduler, firstmate tiering system) and to dispatch a crewmate to handle untethered app ops: redeploy the live companion site to the latest main branch and restart a dead data-harvester process. They also asked which model/effort tier the crewmates were running on, requested an expert UX/UI crewmate for carscanner to fix its navigation and visual polish so it would impress viewers being shown the app, and explicitly approved merging a specific bug fix (a false "failed" status message glitch during crewmate steering). Constraints emphasized throughout included memory-safety (avoiding OOM by serializing heavy validation jobs), never merging without explicit approval, and maintaining continuous "no turn ends blind" supervision over in-flight work while awaiting the captain's decisions on blocked items (a UX sharing question and an AI-reviewer quota exhaustion issue).
What Changed
bin/fm-risk-tripwire.sh(floors model/effort and flags ultracode review on risky work),bin/fm-tier-guard.sh(escalates a task's tier once its diff or elapsed time outgrows its envelope), andbin/fm-ultracode-guard.sh(tracks and gates the independent-review requirement) - wired intofm-spawn.sh,fm-brief.sh, andfm-dispatch-select.sh.bin/fm-nas-deploy-sync.shto best-effort sync and restart a project's live NAS deployment after a landed ship task, driven by a newdata/nas-deployments.mdlookup and bounded by a timeout; extendfm-merge-local.shto also push a fast-forwarded localmaintooriginwhen a remote is present.SERVICE_FAILED) at session start, isolate docker-compose project names per worktree, recognize work fast-forwarded into localmainas landed, and fix leaked/randomized temp-root cleanup in secondmate test suites, plus corresponding docs and test coverage across all of the above.Risk Assessment
✅ Low: The branch's stated goal (randomize the per-task /tmp root and guarantee its cleanup on any spawn failure) is implemented correctly and covered by dedicated tests including legacy-path backward compatibility; the large surrounding body of accumulated commits (NAS deploy sync, risk-tripwire, tier-guard, ultracode-guard, docker-compose isolation, upstream-drift reporting) consists of well-bounded, best-effort, heavily-tested scripts that consistently fail safe (never force, never silently discard, bounded timeouts on every NAS/network touch), and the one safety-invariant change I scrutinized closely (teardown's landed-work check now also accepting the local default-branch ref) is backed by a tree-equality proof and a dedicated regression test, so it does not weaken the no-data-loss guarantee.
Testing
The configured baseline (tests/*.test.shunder tmux) had already passed prior to this session; on top of that I ran the three suites most directly touched by this fix (fm-secondmate-safety, fm-secondmate-lifecycle-e2e, fm-gotmp) standalone and all 69 assertions passed. To get beyond pass/fail, I produced direct filesystem evidence of the actual leak-and-reclaim behavior: found 24 real leaked temp directories in /tmp from pre-fix runs (old<prefix>.<random>naming with no PID, exactly the bug described in commit 481fd7f), then manually exercised the fixedtests/lib.shhelpers to show a normal exit now fully removes the new<prefix>.<pid>.XXXXXXroot and its manifest, and — the critical SIGKILL case a trap can never handle — that a killed process's leaked root and manifest are reclaimed byfm_test_sweep_stale_tmprootson the next invocation. Running the real affected suites confirmed zero new leaks were added. Overall result: the fix works as intended, no issues found.Evidence: Demo A: normal-exit cleanup via manifest-backed EXIT trap
Evidence: Demo B: SIGKILL leaks a root, next invocation's sweep reclaims it
Evidence: Demo C: real pre-existing pre-fix leaked directories found on this machine (old naming scheme)
Evidence: Demo D: real fm-secondmate-safety.test.sh + fm-secondmate-lifecycle-e2e.test.sh run, all passing, zero new leaks
Evidence: Demo E: fm-gotmp.test.sh run, all passing
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
🔧 **Rebase** - 2 issues found → auto-fixed ✅
AGENTS.md- merge conflict rebasing onto origin/maindocs/scripts.md- merge conflict rebasing onto origin/main🔧 Fix applied.
✅ Re-checked - no issues remain.
bin/fm-teardown.sh:373- bin/fm-teardown.sh: content_in_default() (a live git fetch + 3-way tree-merge check) can run twice in one teardown - once inside work_is_landed() (line 364, called when the worktree has unpushed commits and the PR wasn't merged) and again unconditionally inside the new mark_nas_deploy_sync_eligible() (line 373). This duplicates a network fetch and, on a transient fetch failure, could make the two calls disagree (the safety gate passes but NAS-sync eligibility silently comes back false, or vice versa). Low impact since the NAS sync is explicitly best-effort, but worth caching the result of the first call and reusing it for the second.✅ **Test** - passed
✅ No issues found.
command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"bash tests/fm-secondmate-safety.test.sh (39 assertions, all ok)bash tests/fm-secondmate-lifecycle-e2e.test.sh (24 assertions, all ok)bash tests/fm-gotmp.test.sh (6 assertions, all ok)Manual demo A: sourced tests/lib.sh in an isolated TMPDIR, called fm_test_tmproot, confirmed the manifest-backed EXIT trap removes the new <prefix>.<pid>.XXXXXX root and manifest on normal process exitManual demo B: backgrounded a process that created a temp root via fm_test_tmproot, SIGKILL'd it (no trap can run), confirmed the root+manifest were left behind, then ran fm_test_sweep_stale_tmproots from a fresh shell and confirmed both were reclaimedManual demo C: inspected real /tmp for pre-existing fm-secondmate-safety.*/fm-secondmate-lifecycle.* directories from before this fix (old <prefix>.<random> naming with no PID segment) as live evidence the original leak bug was real and reproduced on this machine across many prior runsBefore/after count of leaked-scheme temp directories in real /tmp across the fm-secondmate-safety.test.sh + fm-secondmate-lifecycle-e2e.test.sh run (stayed flat at 24 pre-existing pre-fix leaks; the fixed run itself created zero new leaks)✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.