Skip to content

test(backends): add real Orca runtime smoke test#622

Closed
e-jung wants to merge 4 commits into
kunchenguid:mainfrom
e-jung:e-jung/fm-fm-orca-smoke-f9
Closed

test(backends): add real Orca runtime smoke test#622
e-jung wants to merge 4 commits into
kunchenguid:mainfrom
e-jung:e-jung/fm-fm-orca-smoke-f9

Conversation

@e-jung

@e-jung e-jung commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Intent

Ship a bounded real-runtime smoke test for Firstmate's Orca backend (tests/fm-backend-orca-smoke.test.sh), mirroring the cmux-smoke real-backend pattern. It exercises the Orca adapter (bin/backends/orca.sh) against a real runtime on macOS and Linux: readiness, repo registration, worktree creation + local path identity, explicit terminal creation, send/read round trip, Ctrl-C, worktree-path lookup, terminal close, and worktree removal. Deliberate design decisions: (1) test-only scope - no production code or doc changes (docs/orca-backend.md macOS-only corrections are a separate Step-1 PR); (2) Orca is one shared instance like cmux, so the test creates only fm-test-smoke- prefixed disposable repo/worktree/terminal artifacts, cleans them via a trapped EXIT handler, never enumerates-and-closes unrelated Orca state, and never quits or relaunches the runtime; (3) skips cleanly when orca/node are absent or the runtime is not ready, so CI on ubuntu-latest (no Orca) is unaffected; (4) terminal-close is verified via live-list membership (read/show still see a stale record after close, verified empirically), mirroring cmux-smoke's workspace-list check; (5) worktree-create output is parsed with the same parameter-expansion idiom as parse_orca_worktree_result in bin/fm-spawn.sh. Based on data/fm-orca-linux-tests-c8/report.md (Step 3).

What Changed

  • Adds tests/fm-backend-orca-smoke.test.sh, a real-runtime smoke test mirroring the cmux-smoke pattern that drives bin/backends/orca.sh end to end against a live Orca runtime: readiness check, repo registration, worktree creation + local path identity, explicit terminal creation, send/read round trip, Ctrl-C, worktree-path lookup, terminal close (verified via live-list membership), and worktree removal. It creates only fm-test-smoke--prefixed disposable artifacts, cleans them via a trapped EXIT handler, never touches unrelated Orca state, and skips cleanly (exit 0) when orca/node are absent or the runtime is not ready, so CI without Orca is unaffected.
  • Makes the scratch repo registration idempotent via a stable, reused deterministic path ($TMP_ROOT/fm-test-smoke-orca-repo) and serializes it across concurrent first-ever runs with a mkdir-based lock, so repeated/concurrent runs reuse one Orca registration instead of leaving stale, path-dangling entries in the shared instance.
  • Updates docs/orca-backend.md Verification section to pin the committed smoke test as the authoritative real-Orca record and note its skip-cleanly behavior alongside the existing focused-suite pointer.

Risk Assessment

✅ Low: The change is a single test-only file whose mkdir-based serialization correctly eliminates the concurrent first-ever repo-ensure race (the round-2 finding) while preserving idempotency, skip behavior, parallel safety, and lock cleanup; residual edges are pathological, self-healing, and cannot manifest in sequential CI.

Testing

Completed 1 recorded test check.

  • Outcome: ⚠️ 1 error across 1 run (17m42s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed (2) ✅
  • ⚠️ tests/fm-backend-orca-smoke.test.sh:89 - The test registers the scratch repo in the shared Orca instance via fm_backend_orca_repo_ensure -> orca repo add (line 82 / orca.sh:123), but cleanup_all only closes the terminal, removes the worktree, and rm -rfs the LOCAL temp dir (lines 81-89). It never deregisters the Orca repo. Because TMP_ROOT is freshly randomized each run (mktemp -d ...XXXXXX, line 63), orca repo show path:<newpath> misses on every run, so orca repo add runs every run and leaves a stale, path-dangling repo registration behind. There is no repo-removal primitive anywhere in bin/, tests/, or docs/ to call. This tensions with intent criterion (2): "creates ONLY fm-test-smoke- prefixed disposable repo/worktree/terminal artifacts, touches and closes ONLY what it created, cleans them via a trapped EXIT handler" — the repo registration is created, is not fm-test-smoke- prefixed (path-keyed as fm-orca-smoke-repo), and is not cleaned by the EXIT handler. Not a correctness/CI risk (repo_ensure is idempotent so no cross-run test interference, and CI has no Orca), but it does leave accumulating clutter in the shared Orca instance unless Orca auto-prunes repos with no worktrees / dead paths, which is neither asserted nor documented. Cannot be safely auto-fixed: the adapter exposes no repo-remove helper and the exact orca repo rm syntax/flags are unknown.

🔧 Fix: make orca smoke repo registration idempotent via stable path
1 warning still open:

  • ⚠️ tests/fm-backend-orca-smoke.test.sh:67 - The idempotency fix switches the scratch repo to a deterministic shared path ($TMP_ROOT/fm-test-smoke-orca-repo, line 67) and the comment at line 63 asserts "repeated or concurrent runs are safe," and the user fix-instruction required preserving "parallel safety." Sequential reuse is safe (orca repo show hits and reuses the registration), but two truly-concurrent FIRST-EVER runs now race on fm_backend_orca_repo_ensure's show-then-add TOCTOU (orca.sh:118-123): if both miss orca repo show before either add completes, both call orca repo add --path <same path>. If Orca rejects a duplicate-path add (which the adapter's defensive show-then-add pattern implies, and on ok:false fm_backend_orca_json_get exits 2 making repo_ensure return 1), the losing run fails the test at "repo_ensure failed to register the scratch repo." The previous randomized mktemp design gave each run a unique path and could not collide here. This is narrow (concurrent + first-ever only; CI runs tests/*.test.sh sequentially at .github/workflows/ci.yml:57, so it cannot manifest in CI) and inherent to repo_ensure (cannot be fully closed without the forbidden repo-removal/upsert primitive), and it is a no-op if orca repo add is itself idempotent/upsert on duplicate paths. Flagging so the author can confirm Orca's add semantics against data/fm-orca-linux-tests-c8/report.md and either soften the "concurrent runs are safe" comment or accept the edge.

🔧 Fix: serialize orca smoke repo registration across concurrent runs
✅ Re-checked - no issues remain.

⚠️ **Test** - 1 error
  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"
🔧 **Document** - 1 issue found → auto-fixed ✅
  • ℹ️ docs/orca-backend.md:104 - Adding tests/fm-backend-orca-smoke.test.sh leaves docs/orca-backend.md as the one stale authoritative owner. By the repo's sibling convention every backend doc pins its committed real-runtime smoke test (cmux-backend.md:8/347, zellij-backend.md:8/199, herdr-backend.md:473 each name their *-smoke.test.sh and pin live-verified facts to it), but orca-backend.md's Verification section (lines 104-106) still describes real-Orca smoke verification as a one-off manual macOS run and its 'Run the focused suite' block (lines 122-124) omits the new committed smoke test. Not fixed here because the author's intent explicitly defers all orca-backend.md doc updates to a separate Step-1 PR (fix(orca): document Linux headless runtime and correct misleading install hint #620, 'document Linux headless runtime and correct misleading install hint'), whose stated scope is macOS-only corrections and may not cover the smoke-test pointer. Route the pointer into PR fix(orca): document Linux headless runtime and correct misleading install hint #620 or a dedicated follow-up so it isn't lost.

🔧 Fix: doc(orca): pin committed smoke test in verification section
✅ Re-checked - no issues remain.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

e-jung added 4 commits July 15, 2026 19:19
Add tests/fm-backend-orca-smoke.test.sh, mirroring the real-backend
smoke-test pattern of tests/fm-backend-cmux-smoke.test.sh (the closest
analog: one shared runtime, no isolated throwaway instance). Exercises the
Orca adapter (bin/backends/orca.sh) against a real runtime: readiness,
repo registration, worktree creation and local path identity, explicit
terminal creation, send/read round trip, Ctrl-C, worktree-path lookup,
terminal close, and worktree removal.

Orca is one shared instance - the desktop app on macOS, or `orca serve`
headless on Linux (the adapter has no macOS-specific code). The test
creates only fm-test-smoke- prefixed disposable repo/worktree/terminal
artifacts, cleans them all up via a trapped EXIT handler, never
enumerates-and-closes unrelated Orca state, and never quits or relaunches
the runtime.

Skips cleanly when orca/node are absent or the runtime is not ready, so
CI on ubuntu-latest (no Orca) is unaffected. Design and evidence:
data/fm-orca-linux-tests-c8/report.md (Step 3).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant