Skip to content

feat: add mechanical dispatch guardrails and NAS auto-deploy-sync#624

Open
inivosdigital wants to merge 47 commits into
kunchenguid:mainfrom
inivosdigital:fm/fix-fm-secondmate-safety-tmp-leak
Open

feat: add mechanical dispatch guardrails and NAS auto-deploy-sync#624
inivosdigital wants to merge 47 commits into
kunchenguid:mainfrom
inivosdigital:fm/fix-fm-secondmate-safety-tmp-leak

Conversation

@inivosdigital

Copy link
Copy Markdown

Intent

The captain (user) asked their firstmate orchestration agent for a status report on all 12 projects and their roadmaps, then directed it to take over as the prime supervising session (terminating a stale/orphaned session) after finding the fleet lock held by an unresponsive session. They asked to resume three previously-parked validation jobs (firstmate auto-compaction, falsework runner/scheduler, firstmate tiering system) and to dispatch a crewmate to handle untethered app ops: redeploy the live companion site to the latest main branch and restart a dead data-harvester process. They also asked which model/effort tier the crewmates were running on, requested an expert UX/UI crewmate for carscanner to fix its navigation and visual polish so it would impress viewers being shown the app, and explicitly approved merging a specific bug fix (a false "failed" status message glitch during crewmate steering). Constraints emphasized throughout included memory-safety (avoiding OOM by serializing heavy validation jobs), never merging without explicit approval, and maintaining continuous "no turn ends blind" supervision over in-flight work while awaiting the captain's decisions on blocked items (a UX sharing question and an AI-reviewer quota exhaustion issue).

What Changed

  • Add three mechanical dispatch guardrails - bin/fm-risk-tripwire.sh (floors model/effort and flags ultracode review on risky work), bin/fm-tier-guard.sh (escalates a task's tier once its diff or elapsed time outgrows its envelope), and bin/fm-ultracode-guard.sh (tracks and gates the independent-review requirement) - wired into fm-spawn.sh, fm-brief.sh, and fm-dispatch-select.sh.
  • Add bin/fm-nas-deploy-sync.sh to best-effort sync and restart a project's live NAS deployment after a landed ship task, driven by a new data/nas-deployments.md lookup and bounded by a timeout; extend fm-merge-local.sh to also push a fast-forwarded local main to origin when a remote is present.
  • Harden bootstrap, teardown, and spawn: surface failed critical systemd services (SERVICE_FAILED) at session start, isolate docker-compose project names per worktree, recognize work fast-forwarded into local main as landed, and fix leaked/randomized temp-root cleanup in secondmate test suites, plus corresponding docs and test coverage across all of the above.

Risk Assessment

✅ Low: The branch's stated goal (randomize the per-task /tmp root and guarantee its cleanup on any spawn failure) is implemented correctly and covered by dedicated tests including legacy-path backward compatibility; the large surrounding body of accumulated commits (NAS deploy sync, risk-tripwire, tier-guard, ultracode-guard, docker-compose isolation, upstream-drift reporting) consists of well-bounded, best-effort, heavily-tested scripts that consistently fail safe (never force, never silently discard, bounded timeouts on every NAS/network touch), and the one safety-invariant change I scrutinized closely (teardown's landed-work check now also accepting the local default-branch ref) is backed by a tree-equality proof and a dedicated regression test, so it does not weaken the no-data-loss guarantee.

Testing

The configured baseline (tests/*.test.sh under tmux) had already passed prior to this session; on top of that I ran the three suites most directly touched by this fix (fm-secondmate-safety, fm-secondmate-lifecycle-e2e, fm-gotmp) standalone and all 69 assertions passed. To get beyond pass/fail, I produced direct filesystem evidence of the actual leak-and-reclaim behavior: found 24 real leaked temp directories in /tmp from pre-fix runs (old <prefix>.<random> naming with no PID, exactly the bug described in commit 481fd7f), then manually exercised the fixed tests/lib.sh helpers to show a normal exit now fully removes the new <prefix>.<pid>.XXXXXX root and its manifest, and — the critical SIGKILL case a trap can never handle — that a killed process's leaked root and manifest are reclaimed by fm_test_sweep_stale_tmproots on the next invocation. Running the real affected suites confirmed zero new leaks were added. Overall result: the fix works as intended, no issues found.

Evidence: Demo A: normal-exit cleanup via manifest-backed EXIT trap
=== Demo A: normal exit reclaims the temp root via EXIT trap + manifest ===
TMPDIR (isolated for this demo): /tmp/gotmp-demo.kDHMQx
Before run, TMPDIR contents:
total 0
drwx------    2 orangepi orangepi    40 Jul 15 19:43 .
drwxrwxrwt 1762 root     root     38840 Jul 15 19:43 ..

--- running demo script (sources tests/lib.sh, calls fm_test_tmproot, exits normally) ---
created: /tmp/gotmp-demo.kDHMQx/demo-suite.1884480.ZS3Ptv
manifest contents:
/tmp/gotmp-demo.kDHMQx/demo-suite.1884480.ZS3Ptv

--- after normal exit, TMPDIR contents (temp root + manifest should be GONE) ---
total 0
drwx------    2 orangepi orangepi    40 Jul 15 19:43 .
drwxrwxrwt 1762 root     root     38840 Jul 15 19:43 ..
Evidence: Demo B: SIGKILL leaks a root, next invocation's sweep reclaims it
=== Demo B: SIGKILL leaks a temp root (no trap can run); next invocation's sweep reclaims it ===
TMPDIR (isolated for this demo): /tmp/gotmp-demo.SINaxR

--- launching a background process that creates a temp root, then SIGKILL-ing it (simulating an OOM kill) ---
created: /tmp/gotmp-demo.SINaxR/demo-suite.1885689.ie8ONC
manifest contents:
/tmp/gotmp-demo.SINaxR/demo-suite.1885689.ie8ONC
victim pid: 1885689
victim process is dead: /bin/bash: line 47: kill: (1885689) - No such process
exit=1

--- TMPDIR contents immediately after SIGKILL (temp root + manifest LEAKED, exactly the bug) ---
total 4
drwx------    3 orangepi orangepi    80 Jul 15 19:43 .
drwxrwxrwt 1763 root     root     38860 Jul 15 19:43 ..
drwx------    3 orangepi orangepi    60 Jul 15 19:43 demo-suite.1885689.ie8ONC
-rw-------    1 orangepi orangepi    49 Jul 15 19:43 .fm-test-cleanup.1885689.Puuotd

--- running fm_test_sweep_stale_tmproots("demo-suite") from a fresh shell (simulates the next test invocation) ---

--- TMPDIR contents after the sweep (leaked root + manifest should be GONE) ---
total 0
drwx------    2 orangepi orangepi    40 Jul 15 19:43 .
drwxrwxrwt 1763 root     root     38860 Jul 15 19:43 ..
Evidence: Demo C: real pre-existing pre-fix leaked directories found on this machine (old naming scheme)
=== Pre-existing evidence: leaked temp roots from PRE-FIX runs (old naming scheme, no PID segment) ===
These accumulated across many days (Jul 14-15) of running the test suite BEFORE this fix,
because fm_test_tmproot's trap/array registration never reached the calling script (subshell bug).

drwxrwxr-x   5 orangepi orangepi  120 Jul 15 17:58 /tmp/fm-secondmate-lifecycle.3DDzGu
drwxrwxr-x   5 orangepi orangepi  120 Jul 15 12:55 /tmp/fm-secondmate-lifecycle.4kJC5Y
drwxrwxr-x   5 orangepi orangepi  120 Jul 15 19:06 /tmp/fm-secondmate-lifecycle.72NK7n
drwxrwxr-x   5 orangepi orangepi  120 Jul 15 12:15 /tmp/fm-secondmate-lifecycle.9gBE57
drwxrwxr-x   5 orangepi orangepi  120 Jul 15 02:44 /tmp/fm-secondmate-lifecycle.Ag1MOP
drwxrwxr-x   5 orangepi orangepi  120 Jul 15 17:11 /tmp/fm-secondmate-lifecycle.bpdkPC
drwxrwxr-x   5 orangepi orangepi  120 Jul 15 18:27 /tmp/fm-secondmate-lifecycle.CGYgtS
drwxrwxr-x   5 orangepi orangepi  120 Jul 15 15:49 /tmp/fm-secondmate-lifecycle.CvM4S1
drwxrwxr-x   5 orangepi orangepi  120 Jul 15 19:34 /tmp/fm-secondmate-lifecycle.d6FpXy
drwxrwxr-x   5 orangepi orangepi  120 Jul 15 13:17 /tmp/fm-secondmate-lifecycle.EAGgKR
drwxrwxr-x   5 orangepi orangepi  120 Jul 14 23:22 /tmp/fm-secondmate-lifecycle.EtYows
drwxrwxr-x   5 orangepi orangepi  120 Jul 15 14:17 /tmp/fm-secondmate-lifecycle.fea16V
drwxrwxr-x   5 orangepi orangepi  120 Jul 15 11:34 /tmp/fm-secondmate-lifecycle.gnagLT
drwxrwxr-x   5 orangepi orangepi  120 Jul 15 17:11 /tmp/fm-secondmate-lifecycle.lTj5YG
drwxrwxr-x   5 orangepi orangepi  120 Jul 15 19:17 /tmp/fm-secondmate-lifecycle.LzkdCR
drwxrwxr-x   5 orangepi orangepi  120 Jul 15 15:48 /tmp/fm-secondmate-lifecycle.nZizo4
drwxrwxr-x   5 orangepi orangepi  120 Jul 15 19:36 /tmp/fm-secondmate-lifecycle.sw75XW
drwxrwxr-x   5 orangepi orangepi  120 Jul 15 18:43 /tmp/fm-secondmate-lifecycle.u5D9eb
drwxrwxr-x   5 orangepi orangepi  120 Jul 15 18:24 /tmp/fm-secondmate-lifecycle.usu2qo
drwxrwxr-x   5 orangepi orangepi  120 Jul 15 19:38 /tmp/fm-secondmate-lifecycle.W4ZCZT
drwxrwxr-x 161 orangepi orangepi 4240 Jul 15 19:35 /tmp/fm-secondmate-safety.AMaMFy
drwxrwxr-x  58 orangepi orangepi 1620 Jul 15 19:18 /tmp/fm-secondmate-safety.mJgdk2
drwxrwxr-x 161 orangepi orangepi 4240 Jul 15 19:39 /tmp/fm-secondmate-safety.VCUeGV
drwxrwxr-x 161 orangepi orangepi 4240 Jul 15 19:37 /tmp/fm-secondmate-safety.vK9K56

Count of leaked pre-fix directories:
24

Sample directory name (old scheme = <prefix>.<random>, NEW scheme = <prefix>.<pid>.<random>):
fm-secondmate-safety.AMaMFy
Evidence: Demo D: real fm-secondmate-safety.test.sh + fm-secondmate-lifecycle-e2e.test.sh run, all passing, zero new leaks
=== Running the actual fixed suites: fm-secondmate-safety.test.sh and fm-secondmate-lifecycle-e2e.test.sh (real /tmp, no isolation) ===
ok - FM_HOME parameterizes data and state paths
ok - fm-lock status is scoped per home
Note: switching to '94c7df48b7b7fb6bae40c4e069120fcc0593d628'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

ok - seed allows overlapping project clone lists and drops the owns/owner routing
ok - home seed validation rejects duplicate home routes
ok - home seed validation rejects duplicate id routes
ok - home seed validation rejects nested home routes
Note: switching to '94c7df48b7b7fb6bae40c4e069120fcc0593d628'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

leased worktree for dash
ok - home seeding durably leases treehouse-acquired dash homes under the secondmate id
Note: switching to '94c7df48b7b7fb6bae40c4e069120fcc0593d628'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

ok - home seeding returns rejected acquired homes through treehouse
Note: switching to '94c7df48b7b7fb6bae40c4e069120fcc0593d628'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

ok - home seed rollback warns when treehouse-acquired return fails
ok - home seeding leaves unsafe acquired active homes untouched
ok - home seeding rolls back failed clone attempts without residue
ok - home seeding refuses direct seed without filled charter text
ok - home seeding refuses unfilled placeholder charters
ok - home seeding refuses empty normalized charter fields
Note: switching to '94c7df48b7b7fb6bae40c4e069120fcc0593d628'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

ok - home seeding scaffolds, registers, and spawns a project-less home end to end
ok - home seeding validates reused project-less charters before mutation
ok - home seeding refuses project-less conversion of a populated home
ok - home seeding refuses project-less homes whose projects directory cannot be inspected
ok - home seeding refuses project-less homes with symlinked projects directories
ok - home seeding refuses project-less homes with non-directory projects paths
ok - home seeding refuses project-less homes whose project registry cannot be inspected
ok - home seeding fails loudly on accidental project omission and rejects mixed --no-projects
ok - home seeding refuses local-only projects
ok - home seeding refuses registry delimiter home paths
Note: switching to '94c7df48b7b7fb6bae40c4e069120fcc0593d628'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

Note: switching to '94c7df48b7b7fb6bae40c4e069120fcc0593d628'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

Note: switching to '94c7df48b7b7fb6bae40c4e069120fcc0593d628'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

Note: switching to '94c7df48b7b7fb6bae40c4e069120fcc0593d628'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

ok - home seeding refuses active home and repo root
Note: switching to '94c7df48b7b7fb6bae40c4e069120fcc0593d628'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

ok - home seeding refuses homes marked for another id
Note: switching to '94c7df48b7b7fb6bae40c4e069120fcc0593d628'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -


... [3490 bytes truncated] ...

experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

ok - home seeding skips initialized existing no-mistakes clones
Note: switching to '94c7df48b7b7fb6bae40c4e069120fcc0593d628'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

ok - home seeding refuses uninitialized existing no-mistakes clones
Note: switching to '94c7df48b7b7fb6bae40c4e069120fcc0593d628'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

ok - home seeding refuses project destinations outside the subhome
Note: switching to '94c7df48b7b7fb6bae40c4e069120fcc0593d628'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

Note: switching to '94c7df48b7b7fb6bae40c4e069120fcc0593d628'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

Note: switching to '94c7df48b7b7fb6bae40c4e069120fcc0593d628'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

ok - home seeding refuses operational directories outside the subhome
Note: switching to '94c7df48b7b7fb6bae40c4e069120fcc0593d628'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

Note: switching to '94c7df48b7b7fb6bae40c4e069120fcc0593d628'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

Note: switching to '94c7df48b7b7fb6bae40c4e069120fcc0593d628'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

ok - home seeding refuses symlinked leaf files
ok - secondmate spawn validates homes before launch
ok - secondmate spawn refuses operational directories outside the subhome
ok - fm-send refuses a bare firstmate window with no metadata in this home
ok - secondmate teardown retires empty homes and releases routing
ok - secondmate teardown refuses to hide failed leased-home return
ok - secondmate teardown raw-removes plain-clone homes
ok - secondmate force teardown discards child work
ok - secondmate force teardown preserves child worktree after unproven lock refusal
ok - force teardown allows operational directory symlinks inside the subhome
ok - force teardown refuses operational directory symlinks outside the subhome
ok - secondmate teardown refuses homes containing registered nested homes
ok - secondmate teardown refuses nested homes from the child registry
ok - force teardown validates subhome before child cleanup
ok - force teardown refuses child worktrees inside the active home
ok - force teardown refuses child worktrees inside the firstmate repo
ok - force teardown refuses unregistered child worktree paths
ok - secondmate teardown path-boundary matrix refuses unmarked/ancestor/active-descendant/repo-descendant homes
ok - idle kind=secondmate pane is healthy and not stale
ok - secondmate charter brief is idle by default and does not self-initiate work
ok - fm-backlog-handoff aborts atomically on unmatched, in-flight, and unregistered targets
ok - fm-backlog-handoff refuses Done items under whitespace section headings and unsafe homes

real	0m36.867s
user	0m12.564s
sys	0m18.697s

Note: switching to '94c7df48b7b7fb6bae40c4e069120fcc0593d628'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

ok - seed: registry scope+projects, charter copied, clones+origins, no-mistakes init in subhome only
warning: secondmate design sync skipped before launch: primary default-branch commit cannot be resolved
ok - spawn: launches in the subhome with persistent charter, records routing meta
ok - send: a bare fm-<id> secondmate routes to the meta window with the from-firstmate marker
skip: tasks-axi not found or too old (backlog handoff delegates to it)
ok - recovery: respawns from the durable registry and persistent home
ok - teardown: removes the home, then clears meta and the registry route

real	0m5.693s
user	0m1.239s
sys	0m1.678s
Evidence: Demo E: fm-gotmp.test.sh run, all passing
ok - fm-spawn creates gotmp dir and records tasktmp in meta
ok - fm-teardown removes the dir pointed to by tasktmp= in meta
ok - fm-teardown skips gracefully when tasktmp= is absent (backward compat)
ok - fm-teardown removes legacy deterministic tasktmp roots
ok - fm-teardown skips gracefully when tasktmp= points to a nonexistent dir
ok - fm-teardown skips unsafe tasktmp paths from mutable meta

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

🔧 **Rebase** - 2 issues found → auto-fixed ✅
  • ⚠️ AGENTS.md - merge conflict rebasing onto origin/main
  • ⚠️ docs/scripts.md - merge conflict rebasing onto origin/main

🔧 Fix applied.
✅ Re-checked - no issues remain.

⚠️ **Review** - 1 info
  • ℹ️ bin/fm-teardown.sh:373 - bin/fm-teardown.sh: content_in_default() (a live git fetch + 3-way tree-merge check) can run twice in one teardown - once inside work_is_landed() (line 364, called when the worktree has unpushed commits and the PR wasn't merged) and again unconditionally inside the new mark_nas_deploy_sync_eligible() (line 373). This duplicates a network fetch and, on a transient fetch failure, could make the two calls disagree (the safety gate passes but NAS-sync eligibility silently comes back false, or vice versa). Low impact since the NAS sync is explicitly best-effort, but worth caching the result of the first call and reusing it for the second.
✅ **Test** - passed

✅ No issues found.

  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"
  • bash tests/fm-secondmate-safety.test.sh (39 assertions, all ok)
  • bash tests/fm-secondmate-lifecycle-e2e.test.sh (24 assertions, all ok)
  • bash tests/fm-gotmp.test.sh (6 assertions, all ok)
  • Manual demo A: sourced tests/lib.sh in an isolated TMPDIR, called fm_test_tmproot, confirmed the manifest-backed EXIT trap removes the new <prefix>.<pid>.XXXXXX root and manifest on normal process exit
  • Manual demo B: backgrounded a process that created a temp root via fm_test_tmproot, SIGKILL'd it (no trap can run), confirmed the root+manifest were left behind, then ran fm_test_sweep_stale_tmproots from a fresh shell and confirmed both were reclaimed
  • Manual demo C: inspected real /tmp for pre-existing fm-secondmate-safety.*/fm-secondmate-lifecycle.* directories from before this fix (old <prefix>.<random> naming with no PID segment) as live evidence the original leak bug was real and reproduced on this machine across many prior runs
  • Before/after count of leaked-scheme temp directories in real /tmp across the fm-secondmate-safety.test.sh + fm-secondmate-lifecycle-e2e.test.sh run (stayed flat at 24 pre-existing pre-fix leaks; the fixed run itself created zero new leaks)
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

@inivosdigital
inivosdigital force-pushed the fm/fix-fm-secondmate-safety-tmp-leak branch 2 times, most recently from 7096746 to 3bb7741 Compare July 16, 2026 02:50
Michael added 28 commits July 15, 2026 23:29
Two treehouse worktrees of the same project share a leaf directory
basename, so Compose's default project-naming collides across them on
container/volume/network names. fm-spawn.sh now derives a stable name
from the worktree's own pool-slot path and drops it at
.treehouse-compose-project (git-excluded, like the other turn-end
pointer files), recorded in meta as compose_project=. fm-brief.sh tells
ship/scout crewmates to consume it on every docker compose invocation
and to avoid fixed host ports as a second collision vector.
…r-compose note to fm-brief.sh header comment
Claude crewmates and secondmates are separate claude processes launched
by fm-spawn.sh; they never inherit the primary's .claude/settings.local.json,
so they only compact at the hard context wall instead of proactively.

Add CLAUDE_CODE_AUTO_COMPACT_WINDOW=200000 to the claude launch template.
The var is capped at the model's real context window, so a fixed 200000
lands auto-compaction around 200k regardless of whether the spawned model's
window is 200k or (as verified live for --model sonnet on this account,
which carries the 1M-context beta) ~1M. The primary's own two-var form
(WINDOW=1000000 + PCT_OVERRIDE=20) assumes a 1M window and is not safe to
copy blindly for a smaller-window model/account combination.

grok, codex, opencode, and pi launches are left untouched: grok's binary
is not verified to honor Claude Code env vars, and the other three ignore
them harmlessly but should not carry a var meant for a harness they aren't.
Bump CLAUDE_CODE_AUTO_COMPACT_WINDOW from 200000 to 300000 for claude
crewmates and secondmates, matching the primary session's own updated
target.

The override is capped at the model's real context window, so it is a
no-op below 300000 and only takes effect above it; today's standard
--model sonnet crewmate spawn resolves to the ~1M-context beta, well
above the new threshold. Updated the fm-spawn.sh header comment,
launch_template() comment, and the harness-adapters SKILL.md fact to
describe this conditional behavior instead of the "regardless of
window" framing that only held at the old 200000 value.
content_in_default() only ever diffed a branch against the fetched
refs/remotes/origin/$name, so a no-mistakes-mode task whose branch was
merged into the LOCAL default branch by hand (e.g. after its upstream
PR was closed unmerged) was never recognized as landed and teardown
refused it. It now also checks the local refs/heads/$name ref.
…tests

content_in_default()'s diff-based comparison (from local main's own prior
rewrite) no longer depends on git merge-tree --write-tree, so the two
tests that used to skip themselves on older git now run and pass here.
Extends config/crew-dispatch.json's schema with model/effort/ultracode
axes per the resource-tiering design: docs/examples/crew-dispatch.json
gains rules for the firstmate-repo supervision-backbone carve-out,
safety-critical/security/migration work (opus/xhigh, ultracode-flagged),
and architectural/product tradeoffs (opus/xhigh).

Adds the four hardened guardrails as mechanical checks rather than prose
aspirations:
- bin/fm-tier-guard.sh: firstmate-observable escalation triggers (diff
  size and elapsed time vs. the trivial tier's envelope, plus a general
  heavy-scale ceiling for any tier).
- bin/fm-risk-tripwire.sh: a keyword/path trip-wire against a task's
  brief and diff, independent of the natural-language dispatch match.
- bin/fm-ultracode-guard.sh: flag/reviewed/check subcommands that
  mechanically confirm a genuinely independent second pass ran before an
  ultracode-flagged task reaches PR-ready.
- A documented model-verification gate in AGENTS.md section 4, extending
  the existing unverified-harness discipline to model names.

Verification tiering (light/standard/heavy no-mistakes runs) is
deliberately out of scope here and left for a follow-up change.
The read-tree portability rewrite restructured content_in_default() into
an if/elif that checks exactly one candidate ref: the fetched origin ref
when a remote exists, falling back to the local ref only when there is no
remote at all. That silently dropped the case a no-mistakes-mode project
with a closed, unmerged PR but work fast-forwarded into local main by
hand - the fetched origin ref exists but does not contain the work, so
the local ref is never even checked. Restore checking both candidates in
sequence via a shared content_matches_ref helper, matching the pre-rewrite
behavior tests/fm-teardown.test.sh's local-land regression test expects.
A merged PR does not, by itself, reach the live site: deployed apps run from a
separate checkout under /mnt/nas/experiments/<name>/, managed by pm2,
independent from the projects/<name> clone used for development. This gap
required a manual pull+restart to notice and fix.

bin/fm-nas-deploy-sync.sh fast-forwards a project's live NAS checkout and
restarts its pm2 process(es), looked up from a captain-private
data/nas-deployments.md mapping (best-effort, non-fatal, mirrors
fm-fleet-sync.sh's dirty/diverged safety exactly). bin/fm-teardown.sh now
calls it automatically after a landed ship task, including local-only merges.
Michael and others added 19 commits July 15, 2026 23:30
…-backend.md with the NAS auto-deploy-sync feature and cmux liveness fix.
Add config/critical-services (local, gitignored): one systemd unit name
per line, blank lines and full-line "#" comments ignored. bin/fm-bootstrap.sh
reads it in the detect-only diagnostics and prints one read-only
SERVICE_FAILED line per failed unit via systemctl is-failed (no root), with
a "failed since" timestamp when available. An absent or empty file, or a host
without systemctl, is a no-op; nothing is ever auto-restarted.

Wire it into AGENTS.md's config list and section 3's diagnostic-handling
list, docs/configuration.md, a copyable docs/examples/critical-services,
and tests.
Swap remotes on this working copy so origin is the captain's fork and
upstream is the read-only parent template, and keep the fork alive:

- fm-merge-local.sh: after a local fast-forward, best-effort push the
  default branch to origin when it exists, so a push-backed local-only
  default branch never silently drifts behind local main. Skipped
  silently for no-remote local-only projects.
- fm-bootstrap.sh: always-on UPSTREAM_DRIFT diagnostic reporting how far
  local main diverges from upstream/main in both directions plus
  days-since-merge-base, escalating wording past 30 commits behind or a
  10-day-old merge-base. The upstream fetch lives only in the locked
  mutating fleet-sync sweep; the count-and-report runs unconditionally
  from the already-present upstream ref, never fetching in the
  detect-only path.
- Document the local-only landing pattern and the opportunistic,
  non-blocking upstream PR in AGENTS.md, the UPSTREAM_DRIFT trigger in
  the bootstrap-diagnostics skill, and the inverted remote layout in
  CONTRIBUTING.md.
…late

Post-swap origin is the captain's fork, so /updatefirstmate only
converges the fleet onto that fork (a no-op on the primary source of
truth) and never pulls the read-only upstream template. Folding
upstream-template improvements into local main is the separate reviewed
reconciliation task surfaced by the UPSTREAM_DRIFT bootstrap line, not a
self-update. Note it inline in AGENTS.md section 12 and in the
updatefirstmate skill.
fm_test_tmproot installed its EXIT trap and appended to
FM_TEST_CLEANUP_DIRS from inside a command-substitution subshell, so
neither ever reached the calling script: the trap fired (and deleted
the just-created dir) the instant the subshell exited, and the array
mutation never propagated back. Every run of the secondmate suites
therefore leaked its temp root, growing worse under SIGKILL (e.g. the
OOM killer) since no trap can run for that signal regardless.

Move registration to an on-disk manifest, which survives the subshell
boundary, and install the EXIT trap once at source time in the real
top-level shell. Add a startup sweep for leftover
<prefix>.<pid>.XXXXXX roots whose owning PID is no longer alive, as
the actual safety net for the SIGKILL case, and call it from both
secondmate suites before they claim their own root.
@inivosdigital
inivosdigital force-pushed the fm/fix-fm-secondmate-safety-tmp-leak branch from 3bb7741 to af4e536 Compare July 16, 2026 03:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant