test(backends): add real Orca runtime smoke test#622
Closed
e-jung wants to merge 4 commits into
Closed
Conversation
Add tests/fm-backend-orca-smoke.test.sh, mirroring the real-backend smoke-test pattern of tests/fm-backend-cmux-smoke.test.sh (the closest analog: one shared runtime, no isolated throwaway instance). Exercises the Orca adapter (bin/backends/orca.sh) against a real runtime: readiness, repo registration, worktree creation and local path identity, explicit terminal creation, send/read round trip, Ctrl-C, worktree-path lookup, terminal close, and worktree removal. Orca is one shared instance - the desktop app on macOS, or `orca serve` headless on Linux (the adapter has no macOS-specific code). The test creates only fm-test-smoke- prefixed disposable repo/worktree/terminal artifacts, cleans them all up via a trapped EXIT handler, never enumerates-and-closes unrelated Orca state, and never quits or relaunches the runtime. Skips cleanly when orca/node are absent or the runtime is not ready, so CI on ubuntu-latest (no Orca) is unaffected. Design and evidence: data/fm-orca-linux-tests-c8/report.md (Step 3).
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Ship a bounded real-runtime smoke test for Firstmate's Orca backend (tests/fm-backend-orca-smoke.test.sh), mirroring the cmux-smoke real-backend pattern. It exercises the Orca adapter (bin/backends/orca.sh) against a real runtime on macOS and Linux: readiness, repo registration, worktree creation + local path identity, explicit terminal creation, send/read round trip, Ctrl-C, worktree-path lookup, terminal close, and worktree removal. Deliberate design decisions: (1) test-only scope - no production code or doc changes (docs/orca-backend.md macOS-only corrections are a separate Step-1 PR); (2) Orca is one shared instance like cmux, so the test creates only fm-test-smoke- prefixed disposable repo/worktree/terminal artifacts, cleans them via a trapped EXIT handler, never enumerates-and-closes unrelated Orca state, and never quits or relaunches the runtime; (3) skips cleanly when orca/node are absent or the runtime is not ready, so CI on ubuntu-latest (no Orca) is unaffected; (4) terminal-close is verified via live-list membership (read/show still see a stale record after close, verified empirically), mirroring cmux-smoke's workspace-list check; (5) worktree-create output is parsed with the same parameter-expansion idiom as parse_orca_worktree_result in bin/fm-spawn.sh. Based on data/fm-orca-linux-tests-c8/report.md (Step 3).
What Changed
tests/fm-backend-orca-smoke.test.sh, a real-runtime smoke test mirroring the cmux-smoke pattern that drivesbin/backends/orca.shend to end against a live Orca runtime: readiness check, repo registration, worktree creation + local path identity, explicit terminal creation, send/read round trip, Ctrl-C, worktree-path lookup, terminal close (verified via live-list membership), and worktree removal. It creates onlyfm-test-smoke--prefixed disposable artifacts, cleans them via a trapped EXIT handler, never touches unrelated Orca state, and skips cleanly (exit 0) whenorca/nodeare absent or the runtime is not ready, so CI without Orca is unaffected.$TMP_ROOT/fm-test-smoke-orca-repo) and serializes it across concurrent first-ever runs with a mkdir-based lock, so repeated/concurrent runs reuse one Orca registration instead of leaving stale, path-dangling entries in the shared instance.docs/orca-backend.mdVerification section to pin the committed smoke test as the authoritative real-Orca record and note its skip-cleanly behavior alongside the existing focused-suite pointer.Risk Assessment
✅ Low: The change is a single test-only file whose mkdir-based serialization correctly eliminates the concurrent first-ever repo-ensure race (the round-2 finding) while preserving idempotency, skip behavior, parallel safety, and lock cleanup; residual edges are pathological, self-healing, and cannot manifest in sequential CI.
Testing
Completed 1 recorded test check.
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 1 issue found → auto-fixed (2) ✅
tests/fm-backend-orca-smoke.test.sh:89- The test registers the scratch repo in the shared Orca instance via fm_backend_orca_repo_ensure ->orca repo add(line 82 / orca.sh:123), but cleanup_all only closes the terminal, removes the worktree, andrm -rfs the LOCAL temp dir (lines 81-89). It never deregisters the Orca repo. Because TMP_ROOT is freshly randomized each run (mktemp -d ...XXXXXX, line 63),orca repo show path:<newpath>misses on every run, soorca repo addruns every run and leaves a stale, path-dangling repo registration behind. There is no repo-removal primitive anywhere in bin/, tests/, or docs/ to call. This tensions with intent criterion (2): "creates ONLY fm-test-smoke- prefixed disposable repo/worktree/terminal artifacts, touches and closes ONLY what it created, cleans them via a trapped EXIT handler" — the repo registration is created, is not fm-test-smoke- prefixed (path-keyed as fm-orca-smoke-repo), and is not cleaned by the EXIT handler. Not a correctness/CI risk (repo_ensure is idempotent so no cross-run test interference, and CI has no Orca), but it does leave accumulating clutter in the shared Orca instance unless Orca auto-prunes repos with no worktrees / dead paths, which is neither asserted nor documented. Cannot be safely auto-fixed: the adapter exposes no repo-remove helper and the exactorca repo rmsyntax/flags are unknown.🔧 Fix: make orca smoke repo registration idempotent via stable path
1 warning still open:
tests/fm-backend-orca-smoke.test.sh:67- The idempotency fix switches the scratch repo to a deterministic shared path ($TMP_ROOT/fm-test-smoke-orca-repo, line 67) and the comment at line 63 asserts "repeated or concurrent runs are safe," and the user fix-instruction required preserving "parallel safety." Sequential reuse is safe (orca repo show hits and reuses the registration), but two truly-concurrent FIRST-EVER runs now race on fm_backend_orca_repo_ensure's show-then-add TOCTOU (orca.sh:118-123): if both missorca repo showbefore eitheraddcompletes, both callorca repo add --path <same path>. If Orca rejects a duplicate-path add (which the adapter's defensive show-then-add pattern implies, and on ok:false fm_backend_orca_json_get exits 2 making repo_ensure return 1), the losing run fails the test at "repo_ensure failed to register the scratch repo." The previous randomized mktemp design gave each run a unique path and could not collide here. This is narrow (concurrent + first-ever only; CI runs tests/*.test.sh sequentially at .github/workflows/ci.yml:57, so it cannot manifest in CI) and inherent to repo_ensure (cannot be fully closed without the forbidden repo-removal/upsert primitive), and it is a no-op iforca repo addis itself idempotent/upsert on duplicate paths. Flagging so the author can confirm Orca's add semantics against data/fm-orca-linux-tests-c8/report.md and either soften the "concurrent runs are safe" comment or accept the edge.🔧 Fix: serialize orca smoke repo registration across concurrent runs
✅ Re-checked - no issues remain.
command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"🔧 **Document** - 1 issue found → auto-fixed ✅
docs/orca-backend.md:104- Adding tests/fm-backend-orca-smoke.test.sh leaves docs/orca-backend.md as the one stale authoritative owner. By the repo's sibling convention every backend doc pins its committed real-runtime smoke test (cmux-backend.md:8/347, zellij-backend.md:8/199, herdr-backend.md:473 each name their *-smoke.test.sh and pin live-verified facts to it), but orca-backend.md's Verification section (lines 104-106) still describes real-Orca smoke verification as a one-off manual macOS run and its 'Run the focused suite' block (lines 122-124) omits the new committed smoke test. Not fixed here because the author's intent explicitly defers all orca-backend.md doc updates to a separate Step-1 PR (fix(orca): document Linux headless runtime and correct misleading install hint #620, 'document Linux headless runtime and correct misleading install hint'), whose stated scope is macOS-only corrections and may not cover the smoke-test pointer. Route the pointer into PR fix(orca): document Linux headless runtime and correct misleading install hint #620 or a dedicated follow-up so it isn't lost.🔧 Fix: doc(orca): pin committed smoke test in verification section
✅ Re-checked - no issues remain.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.