feat(spawn): add memory headroom tripwire before dispatch by lukeforshort · Pull Request #659 · kunchenguid/firstmate

lukeforshort · 2026-07-17T01:52:51Z

Intent

Captain-ordered design (2026-07-15): build a memory headroom tripwire in bin/fm-spawn.sh so firstmate refuses to dispatch a new crewmate/scout/secondmate when this machine's memory is already under pressure. Context: this box runs WSL2 under a deliberately tight ~6GB memory cap (see /mnt/c/Users/snydi/.wslconfig, read live rather than hard-coded since the cap can change); two OOM incidents on 2026-07-10 killed the whole fleet (tmux server, no-mistakes daemon, dbus) because concurrent agent/validation dispatch had no headroom check. This is explicitly a tripwire, not a queue: no new queueing machinery or wait loop was added, because the existing backlog already serves as the queue and firstmate retries deferred dispatches later on its own. Design decisions made along the way: (1) measurement is used%=(MemTotal-MemAvailable)/MemTotal from /proc/meminfo, read with pure bash builtins (no forks, no jq/awk) to keep the check cheap; (2) default threshold is 80%, overridable via FM_SPAWN_MEM_MAX_PCT; (3) an explicit FM_SPAWN_MEM_FORCE=1 bypass exists for a captain-ordered emergency dispatch; (4) on trip, fm-spawn.sh must refuse BEFORE any side effect (no window, no worktree lease, no meta file, no hook install) and exit non-zero, printing a loud 'DEFERRED:' stderr message with the measured used%, the threshold, and a note that firstmate should retry once headroom recovers; (5) the check runs once per task launch so it also applies per-pair inside a batch id=repo dispatch (batch keeps its existing 'one failure does not stop the rest' contract), and applies identically to --secondmate launches since those are full agent processes too; (6) an unreadable/unparseable /proc/meminfo must fail OPEN (spawn proceeds) so the tripwire can never brick dispatch on a weird environment. Implementation: new bin/fm-mem-guard.sh defines fm_mem_guard_check (sourced by fm-spawn.sh), with a FM_MEMINFO_PATH_OVERRIDE test seam so the new hermetic unit-test suite (tests/fm-mem-guard.test.sh, 7 cases: under-threshold proceeds, at/over-threshold defers with the DEFERRED message and non-zero exit, force-bypass proceeds, unreadable meminfo proceeds, unparseable meminfo proceeds, custom threshold honored) never touches the real /proc/meminfo. fm-spawn.sh calls fm_mem_guard_check immediately after batch-dispatch detection and before ID is even assigned, i.e. before every existing side effect for both single-task and secondmate spawns. Verified locally: shellcheck clean at the pinned 0.11.0 version across all bin/ and tests/ scripts, the new test suite green, and the pre-existing fm-spawn-batch/fm-spawn-dispatch-profile/fm-backend suites still green (no regression). Also live-fired fm-spawn.sh against a scratch FM_HOME on this machine: FM_SPAWN_MEM_MAX_PCT=1 produced the DEFERRED message and left the scratch state dir completely empty (zero meta/window artifacts), while the default threshold passed straight through the tripwire to the pre-existing missing-brief error further down the script, confirming the tripwire does not interfere with normal dispatch. Deliberately out of scope per the captain's brief: no AGENTS.md change - section 7's existing 'a backend spawn refusal must be surfaced to the captain as a blocker' wording already covers surfacing this kind of dispatch refusal, so no new documentation line was added there; the fix is entirely mechanics, documented in bin/fm-mem-guard.sh's own header and bin/fm-spawn.sh's header comment per the firstmate-coding-guidelines skill's decision tree.

What Changed

Added bin/fm-mem-guard.sh defining fm_mem_guard_check, which reads used% = (MemTotal-MemAvailable)/MemTotal from /proc/meminfo with pure bash builtins and refuses dispatch above a default 80% threshold (FM_SPAWN_MEM_MAX_PCT), with an FM_SPAWN_MEM_FORCE bypass, fail-open on unreadable/unparseable meminfo, a FM_MEMINFO_PATH_OVERRIDE test seam, and a stderr warning on unparseable knob values.
Wired the check into bin/fm-spawn.sh immediately after batch-dispatch detection and before any side effect, so single-task, per-pair batch, and --secondmate launches all defer before creating a window/worktree/meta/hook, printing a loud DEFERRED: message and exiting non-zero; bin/fm-watch.sh now traps TERM so its lock is always released.
Added a hermetic tests/fm-mem-guard.test.sh suite and hardened existing tests for host-dependent node/jq assumptions and missing default sessions; documented the tripwire and watcher signal trap in docs/architecture.md, docs/configuration.md, and docs/scripts.md.

Risk Assessment

✅ Low: The change is a small, well-bounded, fail-open tripwire with hermetic unit coverage; the round-2 fixes correctly addressed the CI-breaking exec bit and the silent knob coercion, and the guard is placed ahead of every spawn side effect with no behavior change on the normal dispatch path.

Testing

Beyond the already-green baseline suite, I ran the new hermetic mem-guard unit suite plus the spawn/backend suites for regression, then produced product-level evidence by driving the real fm-spawn.sh CLI against a scratch FM_HOME with real git projects and this machine's actual /proc/meminfo (36% used of the 5.8 GB WSL cap): pinning the threshold to 1% produced the loud DEFERRED refusal with measured used%, threshold and retry note, exit 1, and a verified-empty state dir, brief dir, worktree lease list and tmux window list, confirming the refusal precedes every side effect; force-bypass, typo'd-bypass warning, per-pair batch defers, secondmate parity, untouched normal dispatch at the default threshold, and the missing/garbage-meminfo fail-open all behaved as designed. The captured CLI transcript is the artifact; the full tests/*.test.sh suite passed and the worktree and scratch home were left clean.

Evidence: Live fm-spawn.sh memory tripwire CLI transcript (10 scenarios, real /proc/meminfo)

=== Memory headroom tripwire: live fm-spawn.sh transcript ===
FM_HOME=/tmp/fm-memguard-live.gpZLGi (scratch fleet home, real git project under projects/demoapp)

### 1. Real memory on this WSL2 box right now - the number the tripwire actually reads
MemTotal:        6068968 kB
MemAvailable:    3865936 kB
=> used 36% of the 5.8 GB WSL cap

### 2. Dispatch a crewmate while the box counts as under pressure (threshold pinned to 1%)
DEFERRED: memory headroom tripwire tripped - used 36% >= threshold 1%; firstmate should retry this dispatch once headroom recovers (bypass with FM_SPAWN_MEM_FORCE=1 for a captain-ordered emergency dispatch)
[exit code: 1]

### 3. Did the refusal land BEFORE any side effect?
-- state/ (would hold fix-login-k3.meta + hook tokens):
   (empty - no meta, no hook token)
-- data/fix-login-k3/ (would hold the brief):
   (absent)
-- git worktree leases in projects/demoapp:
   /tmp/fm-memguard-live.gpZLGi/projects/demoapp  beb948d [master]
-- tmux windows for this task:
   0 window(s) named fm-fix-login-k3
### 4. Captain-ordered emergency dispatch: FM_SPAWN_MEM_FORCE=1 bypasses the tripwire
$ FM_SPAWN_MEM_MAX_PCT=1 FM_SPAWN_MEM_FORCE=1 /home/clarice/.no-mistakes/worktrees/ffc1b9ed10d3/01KXJHKAZ9AZTYKC9NPY6WF6W8/bin/fm-spawn.sh fix-login-k3 projects/demoapp --harness claude
   error: no brief at /tmp/fm-memguard-live.gpZLGi/data/fix-login-k3/brief.md
   [exit code: 1]

   ^ no DEFERRED: the launch passed the tripwire and reached the pre-existing missing-brief error.

### 5. A typo'd bypass value does NOT silently disable the safety net
$ FM_SPAWN_MEM_MAX_PCT=1 FM_SPAWN_MEM_FORCE=true /home/clarice/.no-mistakes/worktrees/ffc1b9ed10d3/01KXJHKAZ9AZTYKC9NPY6WF6W8/bin/fm-spawn.sh fix-login-k3 projects/demoapp --harness claude
   WARNING: FM_SPAWN_MEM_FORCE='true' ignored - only FM_SPAWN_MEM_FORCE=1 bypasses the memory headroom check
   DEFERRED: memory headroom tripwire tripped - used 36% >= threshold 1%; firstmate should retry this dispatch once headroom recovers (bypass with FM_SPAWN_MEM_FORCE=1 for a captain-ordered emergency dispatch)
   [exit code: 1]


### 6. Normal dispatch at the default 80% threshold is untouched by the tripwire
$ /home/clarice/.no-mistakes/worktrees/ffc1b9ed10d3/01KXJHKAZ9AZTYKC9NPY6WF6W8/bin/fm-spawn.sh fix-login-k3 projects/demoapp --harness claude
   error: no brief at /tmp/fm-memguard-live.gpZLGi/data/fix-login-k3/brief.md
   [exit code: 1]

   ^ box is at 36% used, well under 80: no DEFERRED, straight through to the same missing-brief error.

### 7. Batch dispatch: the check runs once PER PAIR (both defer, batch still exits non-zero)
$ FM_SPAWN_MEM_MAX_PCT=1 /home/clarice/.no-mistakes/worktrees/ffc1b9ed10d3/01KXJHKAZ9AZTYKC9NPY6WF6W8/bin/fm-spawn.sh a-task=projects/demoapp b-task=projects/otherapp --harness claude
   DEFERRED: memory headroom tripwire tripped - used 36% >= threshold 1%; firstmate should retry this dispatch once headroom recovers (bypass with FM_SPAWN_MEM_FORCE=1 for a captain-ordered emergency dispatch)
   batch: FAILED to spawn a-task (projects/demoapp)
   DEFERRED: memory headroom tripwire tripped - used 36% >= threshold 1%; firstmate should retry this dispatch once headroom recovers (bypass with FM_SPAWN_MEM_FORCE=1 for a captain-ordered emergency dispatch)
   batch: FAILED to spawn b-task (projects/otherapp)
   [exit code: 1]


### 8. Secondmate launches are full agent processes, so they defer identically
$ FM_SPAWN_MEM_MAX_PCT=1 /home/clarice/.no-mistakes/worktrees/ffc1b9ed10d3/01KXJHKAZ9AZTYKC9NPY6WF6W8/bin/fm-spawn.sh triage-sm /tmp/fm-memguard-live.gpZLGi/sm-home --secondmate
   DEFERRED: memory headroom tripwire tripped - used 36% >= threshold 1%; firstmate should retry this dispatch once headroom recovers (bypass with FM_SPAWN_MEM_FORCE=1 for a captain-ordered emergency dispatch)
   [exit code: 1]


### 9. Final side-effect sweep across every attempt above
-- state/:
   (empty)
-- worktree leases:
   /tmp/fm-memguard-live.gpZLGi/projects/demoapp  beb948d [master]
   /tmp/fm-memguard-live.gpZLGi/projects/otherapp  f68fa80 [master]
-- tmux fm-* windows for these task ids:
   0
### 10. Fail-open: a missing or garbage meminfo must never brick dispatch
$ FM_MEMINFO_PATH_OVERRIDE=/nonexistent/meminfo FM_SPAWN_MEM_MAX_PCT=1 fm-spawn.sh fix-login-k3 projects/demoapp --harness claude
   error: no brief at /tmp/fm-memguard-live.gpZLGi/data/fix-login-k3/brief.md
   [exit code: 1]
$ FM_MEMINFO_PATH_OVERRIDE=/tmp/fm-memguard-live.gpZLGi/garbage.meminfo FM_SPAWN_MEM_MAX_PCT=1 fm-spawn.sh fix-login-k3 projects/demoapp --harness claude
   error: no brief at /tmp/fm-memguard-live.gpZLGi/data/fix-login-k3/brief.md
   [exit code: 1]
   ^ no DEFERRED even with the threshold pinned to 1%: unmeasurable memory lets dispatch proceed.

Evidence: Key excerpt: tripwire refuses before any side effect

### 1. Real memory on this WSL2 box right now - the number the tripwire actually reads
MemTotal: 6068968 kB
MemAvailable: 3865936 kB
=> used 36% of the 5.8 GB WSL cap

### 2. Dispatch a crewmate while the box counts as under pressure (threshold pinned to 1%)
DEFERRED: memory headroom tripwire tripped - used 36% >= threshold 1%; firstmate should retry this dispatch once headroom recovers (bypass with FM_SPAWN_MEM_FORCE=1 for a captain-ordered emergency dispatch)
[exit code: 1]

### 3. Did the refusal land BEFORE any side effect?
-- state/ (would hold fix-login-k3.meta + hook tokens):
(empty - no meta, no hook token)
-- data/fix-login-k3/ (would hold the brief):
(absent)
-- git worktree leases in projects/demoapp:
/tmp/fm-memguard-live.gpZLGi/projects/demoapp beb948d [master]
-- tmux windows for this task:
0 window(s) named fm-fix-login-k3

### 6. Normal dispatch at the default 80% threshold is untouched by the tripwire
$ fm-spawn.sh fix-login-k3 projects/demoapp --harness claude
error: no brief at /tmp/fm-memguard-live.gpZLGi/data/fix-login-k3/brief.md
[exit code: 1]
^ box is at 36% used, well under 80: no DEFERRED, straight through to the same missing-brief error.

### 7. Batch dispatch: the check runs once PER PAIR (both defer, batch still exits non-zero)
$ fm-spawn.sh a-task=projects/demoapp b-task=projects/otherapp --harness claude
DEFERRED: memory headroom tripwire tripped - used 36% >= threshold 1%; ...
batch: FAILED to spawn a-task (projects/demoapp)
DEFERRED: memory headroom tripwire tripped - used 36% >= threshold 1%; ...
batch: FAILED to spawn b-task (projects/otherapp)
[exit code: 1]

- Outcome: 🔧 1 issue found → auto-fixed (3) ✅ across 4 runs (1h34m42s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 info

🚨 tests/fm-mem-guard.test.sh:1 - The new test file is committed as mode 100644 and is the only non-executable file matching tests/*.test.sh. CI (.github/workflows/ci.yml:57) runs for test_script in tests/*.test.sh; do "$test_script"; done under set -eu, which will fail with exit 126 (permission denied) and break the build. The suite passed locally only because it was invoked as bash tests/fm-mem-guard.test.sh. Fix with chmod +x tests/fm-mem-guard.test.sh.
⚠️ bin/fm-spawn.sh:287 - The comment claims the per-pair check lets a batch "land its first task and defer the rest as pressure rises", but the batch loop re-execs every pair within milliseconds, well before a just-spawned agent's RSS grows. All pairs therefore read essentially the same pre-spawn MemAvailable: a 5-pair batch at 70% used passes all five checks and can still OOM - the exact concurrent-dispatch shape the 2026-07-10 incidents came from. The tripwire does protect the sequential-dispatch case; the comment should be corrected to claim only that, or the batch path needs a real mechanism (e.g. a post-spawn settle before the next pair's check).
ℹ️ bin/fm-mem-guard.sh:51 - A non-numeric FM_SPAWN_MEM_MAX_PCT (e.g. a typo'd 95%) is silently coerced to 80, giving the operator a stricter gate than requested with no diagnostic. The same shape applies at line 27, where FM_SPAWN_MEM_FORCE=true/yes is silently not a bypass. Emit a one-line stderr warning on an unparseable knob value rather than falling back silently.

🔧 Fix: fix mem-guard test exec bit and warn on bad knob values
1 info still open:

ℹ️ bin/fm-mem-guard.sh:1 - bin/fm-mem-guard.sh is a sourced-only library (defines fm_mem_guard_check, no main body) but does not use the -lib.sh suffix that all sixteen other sourced-only libraries in bin/ use (fm-gate-refuse-lib.sh, fm-classify-lib.sh, fm-tangle-lib.sh, etc.), and it is committed mode 100755 so invoking it directly is a silent no-op. Existing lib exec bits are already mixed, so this is a naming consistency note only, not a defect.

🔧 **Test** - 1 issue found → auto-fixed (3) ✅

🚨 tests failed with exit code 1
command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"

🔧 Fix: fix host-dependent node and jq assumptions in tests
1 error still open:

🚨 tests failed with exit code 1
command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"

🔧 Fix: skip herdr lab tests without a running default session
1 error still open:

🚨 tests failed with exit code 1
command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"

🔧 Fix: trap TERM in watcher so its lock is always released
✅ Re-checked - no issues remain.

command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"
bash tests/fm-mem-guard.test.sh - 9 hermetic unit cases (under/at/over threshold, force bypass, unreadable + unparseable meminfo fail-open, custom threshold, bad knob warnings)
bash tests/fm-spawn-batch.test.sh, bash tests/fm-spawn-dispatch-profile.test.sh, bash tests/fm-backend.test.sh - no regression in spawn paths the tripwire sits in front of
for t in tests/*.test.sh; do bash "$t"; done - full suite, rc=0 (needed a >2min budget)
Manual end-to-end: FM_SPAWN_MEM_MAX_PCT=1 bin/fm-spawn.sh fix-login-k3 projects/demoapp --harness claude against a scratch FM_HOME reading the real /proc/meminfo -> DEFERRED message + exit 1
Manual side-effect sweep after each refusal: find $FM_HOME/state -mindepth 1, brief-dir existence check, git worktree list, tmux list-windows -a | grep -c fm-<id> - all empty/zero
Manual: FM_SPAWN_MEM_FORCE=1 bypass proceeds; FM_SPAWN_MEM_FORCE=true warns and still defers
Manual: default 80% threshold on a 36%-used box passes straight through the tripwire to the pre-existing missing-brief error
Manual: bin/fm-spawn.sh a-task=projects/demoapp b-task=projects/otherapp -> DEFERRED printed once per pair, batch exits non-zero
Manual: bin/fm-spawn.sh triage-sm <home> --secondmate -> defers identically
Manual fail-open: FM_MEMINFO_PATH_OVERRIDE=/nonexistent/meminfo and a garbage meminfo both proceed with the threshold pinned to 1%

⚠️ **Document** - 1 info

ℹ️ AGENTS.md:427 - AGENTS.md:427 says a spawn refusal "must be surfaced to the captain as a blocker; never silently retry", while the new tripwire's DEFERRED: message instructs firstmate to retry once headroom recovers. Narrowly read there is no conflict: line 427 enumerates backend spawn refusals (missing dependency, unauthenticated socket, version gate), and the memory tripwire is not a backend refusal. The author also explicitly scoped AGENTS.md out of this change on that exact reasoning. Left unedited deliberately - deciding whether AGENTS.md needs an explicit DEFERRED: carve-out (retry later rather than escalate) is a judgment call about the operating manual's intent, not a substantiated doc gap.

⏭️ **Lint** - skipped

⚠️ linter found issues (exit code 1)

🔧 Fix: no lint fix needed; code clean under pinned shellcheck 0.11.0
1 warning still open:

⚠️ linter found issues (exit code 1)

🔧 Fix: no lint fix needed; code clean under pinned shellcheck 0.11.0
1 warning still open:

⚠️ linter found issues (exit code 1)

🔧 Fix: no lint fix needed; changed scripts shellcheck-clean under pinned 0.11.0
1 warning still open:

⚠️ linter found issues (exit code 1)

🔧 Fix: no code fix; changed files clean under shellcheck 0.11.0
1 warning still open:

⚠️ linter found issues (exit code 1)

✅ **Push** - passed

✅ No issues found.

Refuse a crewmate/scout/secondmate launch before any side effect when used memory is at or above FM_SPAWN_MEM_MAX_PCT (default 80), printing a DEFERRED: message; FM_SPAWN_MEM_FORCE=1 bypasses for emergencies. Prevents OOM-driven fleet kills on this machine's tight WSL2 memory cap.

… values

…sion

…signal trap

lukeforshort added 6 commits July 17, 2026 09:53

no-mistakes(review): fix mem-guard test exec bit and warn on bad knob…

401307c

… values

no-mistakes(test): fix host-dependent node and jq assumptions in tests

1b0cb01

no-mistakes(test): skip herdr lab tests without a running default ses…

c411c45

…sion

no-mistakes(test): trap TERM in watcher so its lock is always released

397c45e

no-mistakes(document): document memory headroom tripwire and watcher …

107e526

…signal trap

lukeforshort force-pushed the fm/dispatch-headroom-tripwire-m6 branch from 48dc2ad to 107e526 Compare July 17, 2026 01:56

github-actions Bot mentioned this pull request Jul 17, 2026

[firstmate#659] feat(spawn): add memory headroom tripwire before dispatch kunchenguid/wheelhouse#1426

Closed

4 tasks

lukeforshort closed this Jul 17, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(spawn): add memory headroom tripwire before dispatch#659

feat(spawn): add memory headroom tripwire before dispatch#659
lukeforshort wants to merge 6 commits into
kunchenguid:mainfrom
lukeforshort:fm/dispatch-headroom-tripwire-m6

lukeforshort commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

lukeforshort commented Jul 17, 2026

Intent

What Changed

Risk Assessment

Testing

Pipeline

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant