Skip to content

fix(herdr): record a stopped default session in the lab fleet-state tripwire#660

Closed
lukeforshort wants to merge 6 commits into
kunchenguid:mainfrom
lukeforshort:fm/testpath-hermeticity-y4
Closed

fix(herdr): record a stopped default session in the lab fleet-state tripwire#660
lukeforshort wants to merge 6 commits into
kunchenguid:mainfrom
lukeforshort:fm/testpath-hermeticity-y4

Conversation

@lukeforshort

Copy link
Copy Markdown

Intent

Make firstmate's test suite hermetic against machine-specific jq and node tool locations

What Changed

  • Relax the Herdr lab fleet-state tripwire so it snapshots the default session whether or not it is running, instead of hard-refusing on any host without a live default server; a start or stop of that session during lab work still trips the before/after comparison. Synced the matching wording in bin/fm-herdr-lab.sh, bin/fm-brief.sh, and docs/herdr-backend.md.
  • Add fm_test_base_path and fm_test_command_mask helpers to tests/lib.sh that symlink only the named real tools into a scoped realbin dir and mask command -v for a tool, so tests resolve host-installed jq/node without dragging in unrelated binaries or silently satisfying absence checks.
  • Rewire fm-x-mode, fm-dispatch-select, and fm-session-start tests onto the new helpers and add a fm-herdr-lab test covering the stopped-default tripwire path.

Risk Assessment

✅ Low: Test-hermeticity refactor plus a narrow, intentional tripwire relaxation that preserves change detection (running flag still recorded and compared), backed by a new regression test and matching doc/comment updates.

Testing

Baseline suite already passed; I re-ran the four test files touched by this change and all pass. To show the intent end-to-end, I exercised the two new helpers directly on this host, whose tool layout (jq at ~/.local/bin off the bare BASE_PATH, plus a real /usr/bin/node) is exactly the machine-specific hazard the change targets: fm_test_base_path imports only the requested jq via a realbin symlink and never node, and fm_test_command_mask makes command -v node fail while leaving other tools resolvable, with a captured counterfactual proving a bare BASE_PATH cannot find jq here. Non-visual, test-infrastructure change, so evidence is a CLI transcript rather than a rendered surface.

Evidence: Hermeticity demo transcript (helpers vs. this host's jq/node layout)
==================================================================
 HERMETICITY DEMO -- host tool layout that would defeat naive tests
==================================================================

[host facts]
  jq   -> /home/clarice/.local/bin/jq      (NOT on /usr/bin:/bin:/usr/sbin:/sbin)
  node -> /home/clarice/.nvm/versions/node/v22.23.1/bin/node  (nvm)
  BUT a real system node also exists: /usr/bin/node

------------------------------------------------------------------
[1] fm_test_base_path: imports ONLY the requested tool (jq), via a
    realbin symlink -- never dragging node in beside it.
------------------------------------------------------------------
  returned BASE_PATH = /tmp/hermetic-demo.waUmyX/realbin:/usr/bin:/bin:/usr/sbin:/sbin
  realbin contents:
    total 0
    lrwxrwxrwx 1 clarice clarice 27 Jul 17 09:33 jq -> /home/clarice/.local/bin/jq

  -> jq resolvable under the returned PATH?
     /tmp/hermetic-demo.waUmyX/realbin/jq
  -> node NOT imported (only jq was requested):
     OK: node absent from realbin

------------------------------------------------------------------
[2] Counterfactual: a bare BASE_PATH (old code) canNOT find jq on
    this host, which is exactly why the helper is needed.
------------------------------------------------------------------
  bare BASE_PATH jq lookup: NOT FOUND (would break fm-x-mode / fm-dispatch-select)

------------------------------------------------------------------
[3] fm_test_command_mask: makes 'command -v node' fail even though
    /usr/bin/node is on BASE_PATH and would silently satisfy
    detection in the forced-missing-node session-start test.
------------------------------------------------------------------
  without mask, under bare BASE_PATH:
     /usr/bin/node <- real node found, forced-absence defeated
  with mask sourced:
     OK: command -v node FAILS -> forced absence holds
  mask leaves other tools alone:
     OK: command -v sh still works (/usr/bin/sh)

==================================================================
 Both machine-specific hazards on THIS host are neutralized.
==================================================================

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"
  • bash tests/fm-x-mode.test.sh (uses fm_test_base_path for jq)
  • bash tests/fm-dispatch-select.test.sh (uses fm_test_base_path for jq)
  • bash tests/fm-session-start.test.sh (uses fm_test_command_mask for node)
  • bash tests/fm-herdr-lab.test.sh (new stopped-default tripwire test)
  • Direct helper demo: sourced tests/lib.sh, called fm_test_base_path (verified only jq symlinked into realbin, node absent) and fm_test_command_mask (verified command -v node fails under BASE_PATH while command -v sh still works), plus the bare-BASE_PATH counterfactual showing jq is unfindable without the helper
✅ **Document** - passed

✅ No issues found.

⚠️ **Lint** - 1 warning
  • ⚠️ linter found issues (exit code 1)

🔧 Fix: no code lint issues; failure is shellcheck 0.11.0 pin unmet
1 warning still open:

  • ⚠️ linter found issues (exit code 1)

🔧 Fix: no code lint issues; unmet shellcheck 0.11.0 version pin
1 warning still open:

  • ⚠️ linter found issues (exit code 1)

🔧 Fix: no code lint issues; local shellcheck 0.11.0 pin unmet
1 warning still open:

  • ⚠️ linter found issues (exit code 1)
✅ **Push** - passed

✅ No issues found.

fm-dispatch-select.test.sh and fm-x-mode.test.sh hard-coded a BASE_PATH
that misses jq when it lives outside /usr/bin:/bin (e.g. ~/.local/bin).
Generalize fm-x-mode.test.sh's JQ_DIR trick into a shared
fm_test_base_path helper in tests/lib.sh and use it from both suites.

fm-session-start.test.sh's forced-missing-node cases only removed node
from the fakebin dir, so a real node elsewhere on BASE_PATH (e.g. apt's
/usr/bin/node) silently satisfied detection. Mask `command -v node` the
same way the file already masks tmux for its Herdr cases.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant