diff --git a/AGENTS.md b/AGENTS.md index 4dd5acb..b2d4b94 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -86,6 +86,10 @@ Markdown-carrying surfaces reuse: the shell-discovery errors, the `run` prefligh it. `--help` is the one deliberate restatement (`AFTER_HELP` in `src/cli/help.rs`), hard-wrapped and backtick-free because clap renders into a terminal; keep the two in step by hand. +Which platforms that requirement is honored on — and why preparing on Windows but dispatching from +WSL is a correctness boundary rather than a preference — is stated once under "Platform support" in +`docs/developer_overview.md`. + **Where user-facing warnings come from.** Library modules (`pipeline`, `workspace`, `sandbox`, `adapters`) never print. They return warning strings on their result struct — `#[serde(skip)]` when that struct is also a serialized artifact — and the `cli` handler prints them with the `⚠ ` prefix. diff --git a/README.md b/README.md index 595de5e..65b8759 100644 --- a/README.md +++ b/README.md @@ -38,10 +38,14 @@ The installed CLI is the primary manual. Start with `eval-magic --help`, and use ## Install Git is required at runtime, plus a POSIX shell with `jq`: the dispatch and judge recipes eval-magic -generates are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. On Windows, run them in -Git Bash (Git for Windows) or WSL, and install `jq` separately — Git for Windows does not bundle it. +generates are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. The shell that runs them +has to resolve the same paths the workspace was prepared with. On Windows that is Git Bash (Git for +Windows), with `jq` installed separately — Git for Windows does not bundle it. WSL resolves a +different filesystem namespace, so run eval-magic inside WSL rather than dispatching into it. Set `EVAL_MAGIC_SH` to select a specific `sh`. +Windows support runs through Git Bash and is deprecated: a future release will require WSL. + Prebuilt binaries for macOS, Linux, and Windows are attached to each [GitHub release](https://github.com/slowdini/eval-magic/releases). diff --git a/docs/developer_overview.md b/docs/developer_overview.md index 0ecc36c..efe3adc 100644 --- a/docs/developer_overview.md +++ b/docs/developer_overview.md @@ -66,6 +66,27 @@ following authorities: infer one harness's flags or event shapes from another harness. - Tests and golden artifacts for behavior that crosses a module or CLI boundary. +## Platform support + +| Tier | Platform | Verified by | +| --- | --- | --- | +| Supported | Linux, macOS | the `ubuntu-latest` CI job | +| Deprecated | Windows, through Git Bash (Git for Windows) | the `windows-latest` CI job | +| Unsupported | preparing a workspace on Windows and dispatching it from WSL | — | + +Windows support is deprecated in favor of WSL, and its removal is gated on #256, which replaces +the generated POSIX recipes with a runner-driven `eval-magic dispatch`. Until that lands, the +Windows runner stays green and Windows-native behavior is held to the same bar as any other +platform: a Windows failure is a real failure, not an accepted gap. Do not add new Windows-native +accommodation in the meantime. + +The unsupported row is a correctness boundary rather than a preference. A generated recipe carries +the absolute paths of the host that prepared the workspace. Git Bash shares the Windows filesystem, +so those paths resolve; WSL resolves its own namespace, where a `C:\…` path names nothing. Nothing +in the tree translates between the two, so the split fails quietly instead of loudly. +`POSIX_TOOLING_REQUIREMENT` (`src/core/runtime.rs`) is the single wording every user-facing surface +reuses to state this; `src/cli/help.rs` restates it for clap by hand. + ## Make and verify a change Trace the user-visible behavior from the CLI handler into library-owned logic and artifacts before diff --git a/src/cli/help.rs b/src/cli/help.rs index 9dc74fb..69afc86 100644 --- a/src/cli/help.rs +++ b/src/cli/help.rs @@ -9,9 +9,12 @@ pub(super) const AFTER_HELP: &str = "\ REQUIREMENTS: Git, plus a POSIX shell with jq, xargs, tr, and wc. The dispatch and judge - recipes in the generated RUNBOOK.md are POSIX command lines — on Windows, - run them in Git Bash (Git for Windows) or WSL, and install jq separately. - Set EVAL_MAGIC_SH to select a specific sh. + recipes in the generated RUNBOOK.md are POSIX command lines, and the shell + that runs them has to resolve the same paths the workspace was prepared + with. On Windows that is Git Bash (Git for Windows), with jq installed + separately. WSL resolves a different filesystem namespace, so run + eval-magic inside WSL rather than dispatching into it. Set EVAL_MAGIC_SH + to select a specific sh. EXAMPLES: # Scaffold a first eval and prepare its isolated comparison environments diff --git a/src/cli/run/orchestrate/shell.rs b/src/cli/run/orchestrate/shell.rs index a8c8a6e..621c60c 100644 --- a/src/cli/run/orchestrate/shell.rs +++ b/src/cli/run/orchestrate/shell.rs @@ -3,8 +3,13 @@ //! //! Unlike [`super::git`], a missing shell is a warning rather than an error. //! `run` never dispatches — it prepares a workspace, and that workspace is -//! correct whatever shell prepared it. Preparing on Windows and dispatching from -//! WSL is a legitimate split, so this reports the gap and lets the run finish. +//! correct whatever shell prepared it, so this reports the gap and lets the run +//! finish. +//! +//! The shell that eventually dispatches still has to resolve the paths this host +//! wrote into the recipes, which keeps the gap host-local: Git Bash shares the +//! Windows filesystem, WSL resolves its own. See [`POSIX_TOOLING_REQUIREMENT`] +//! for the declared rule the warnings below defer to. use std::path::Path; @@ -39,8 +44,8 @@ pub(super) fn preflight_posix_tooling() -> Vec { fn tooling_warning(shell: Result<&Path, &str>, missing: Option<&str>) -> Option { if let Err(reason) = shell { return Some(format!( - "{reason} The workspace and recipes below are still correct — prepare here and \ - dispatch them from a POSIX shell." + "{reason} The workspace and recipes below are still correct — dispatch them from a \ + POSIX shell on this host." )); } let reason = missing?; @@ -73,6 +78,19 @@ mod tests { assert!(warning.contains("Git Bash"), "{warning}"); } + /// An unqualified "dispatch them from a POSIX shell" reads as an invitation + /// to prepare here and dispatch from WSL — the one split + /// [`POSIX_TOOLING_REQUIREMENT`] rules out, and the one that fails quietly. + #[test] + fn a_missing_shell_confines_dispatch_to_the_host_that_prepared_the_workspace() { + let warning = tooling_warning(Err("no POSIX shell found. Use Git Bash."), None) + .expect("a host with no POSIX shell must be told"); + assert!( + warning.contains("this host"), + "the warning must keep dispatch on the preparing host: {warning}" + ); + } + /// A shell that is missing a recipe tool names the tool and the shell whose /// PATH was searched, and still points at the declared requirement — Git for /// Windows resolves a shell but bundles no `jq`, which is exactly this case. diff --git a/src/core/runtime.rs b/src/core/runtime.rs index ee3bcce..630e444 100644 --- a/src/core/runtime.rs +++ b/src/core/runtime.rs @@ -105,9 +105,19 @@ pub fn run_git(args: &[&str], cwd: &Path) -> GitOutput { /// for Windows supplies `sh`, `xargs`, `tr`, and `wc` but not `jq`, so guidance /// naming only the shell would send an operator to a setup that still walls out /// at the judge step. +/// +/// It also separates the two Windows options rather than listing them side by +/// side. A generated recipe carries the preparing host's absolute paths, so the +/// dispatching shell has to resolve those same paths. Git Bash does — it shares +/// the Windows filesystem. WSL does not: `C:\…` names nothing in its namespace, +/// so WSL is correct only when eval-magic itself runs inside it. Nothing here +/// translates between the two, and a split across that boundary fails quietly +/// rather than loudly. pub(crate) const POSIX_TOOLING_REQUIREMENT: &str = "eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, \ - `tr`, and `wc`. Run them in a POSIX shell with `jq` installed — on Windows, use Git Bash \ - (Git for Windows) or WSL. Set EVAL_MAGIC_SH to select a specific `sh`."; + `tr`, and `wc`. Run them in a POSIX shell with `jq` installed that resolves the same paths \ + this workspace was prepared with — on Windows, Git Bash (Git for Windows). WSL resolves a \ + different filesystem namespace, so run eval-magic inside WSL rather than dispatching into \ + it. Set EVAL_MAGIC_SH to select a specific `sh`."; /// `sh` locations inside a Git for Windows install, given the `git --exec-path` /// directory (`/mingw64/libexec/git-core` — hence three levels up). @@ -358,6 +368,25 @@ mod tests { assert!(shell.is_file(), "{} is not a file", shell.display()); } + /// The declared requirement has to separate the two Windows options rather + /// than list them as equivalent. Git Bash shares the Windows filesystem, so a + /// workspace prepared by a native run dispatches from it correctly. WSL + /// resolves a different namespace, where the `C:\…` paths a native run wrote + /// name nothing — so WSL is only correct when eval-magic itself runs inside + /// it. Listing the two side by side invites a split that silently cannot work. + #[test] + fn the_declared_requirement_places_wsl_around_eval_magic_not_downstream_of_it() { + assert!( + POSIX_TOOLING_REQUIREMENT.contains("Git Bash"), + "{POSIX_TOOLING_REQUIREMENT}" + ); + assert!( + POSIX_TOOLING_REQUIREMENT.contains("inside WSL"), + "WSL must be named as where eval-magic runs, not somewhere to dispatch \ + into: {POSIX_TOOLING_REQUIREMENT}" + ); + } + #[test] fn require_posix_toolchain_names_the_tool_that_is_missing() { let error = require_posix_toolchain(&["eval-magic-not-a-real-tool"]) diff --git a/tests/golden/claude-code/manifest-nomodel.golden.md b/tests/golden/claude-code/manifest-nomodel.golden.md index 9d30538..9f0ea4a 100644 --- a/tests/golden/claude-code/manifest-nomodel.golden.md +++ b/tests/golden/claude-code/manifest-nomodel.golden.md @@ -8,7 +8,7 @@ Total dispatches: 2 In an agent session, read `dispatch.json` (sibling of this file) instead of this manifest. Each task has a `dispatch_prompt_path` field pointing at the file that holds the full prompt — dispatch the task with a short "read this file and follow it" instruction rather than inlining the prompt — plus exact paths for `run.json` and `timing.json`. -**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed — on Windows, use Git Bash (Git for Windows) or WSL. Set EVAL_MAGIC_SH to select a specific `sh`. +**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed that resolves the same paths this workspace was prepared with — on Windows, Git Bash (Git for Windows). WSL resolves a different filesystem namespace, so run eval-magic inside WSL rather than dispatching into it. Set EVAL_MAGIC_SH to select a specific `sh`. After all dispatches (Claude Code): diff --git a/tests/golden/claude-code/manifest.golden.md b/tests/golden/claude-code/manifest.golden.md index a72f47f..83499d7 100644 --- a/tests/golden/claude-code/manifest.golden.md +++ b/tests/golden/claude-code/manifest.golden.md @@ -8,7 +8,7 @@ Total dispatches: 2 In an agent session, read `dispatch.json` (sibling of this file) instead of this manifest. Each task has a `dispatch_prompt_path` field pointing at the file that holds the full prompt — dispatch the task with a short "read this file and follow it" instruction rather than inlining the prompt — plus exact paths for `run.json` and `timing.json`. -**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed — on Windows, use Git Bash (Git for Windows) or WSL. Set EVAL_MAGIC_SH to select a specific `sh`. +**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed that resolves the same paths this workspace was prepared with — on Windows, Git Bash (Git for Windows). WSL resolves a different filesystem namespace, so run eval-magic inside WSL rather than dispatching into it. Set EVAL_MAGIC_SH to select a specific `sh`. After all dispatches (Claude Code): diff --git a/tests/golden/claude-code/runbook.golden.md b/tests/golden/claude-code/runbook.golden.md index 265e33a..713eace 100644 --- a/tests/golden/claude-code/runbook.golden.md +++ b/tests/golden/claude-code/runbook.golden.md @@ -4,7 +4,7 @@ This runbook is for a human driving the run from a terminal. Work from this iter and copy-paste each step. The workspace is self-contained — you should not need the surrounding repo. -> **Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed — on Windows, use Git Bash (Git for Windows) or WSL. Set EVAL_MAGIC_SH to select a specific `sh`. +> **Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed that resolves the same paths this workspace was prepared with — on Windows, Git Bash (Git for Windows). WSL resolves a different filesystem namespace, so run eval-magic inside WSL rather than dispatching into it. Set EVAL_MAGIC_SH to select a specific `sh`. - **Skill under test:** widget-skill - **Mode:** revision — comparing `old_skill` vs `new_skill` diff --git a/tests/golden/cline/manifest.golden.md b/tests/golden/cline/manifest.golden.md index df07631..5103519 100644 --- a/tests/golden/cline/manifest.golden.md +++ b/tests/golden/cline/manifest.golden.md @@ -8,7 +8,7 @@ Total dispatches: 2 In an agent session, read `dispatch.json` (sibling of this file) instead of this manifest. Each task has a `dispatch_prompt_path` field pointing at the file that holds the full prompt — dispatch the task with a short "read this file and follow it" instruction rather than inlining the prompt — plus exact paths for `run.json` and `timing.json`. -**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed — on Windows, use Git Bash (Git for Windows) or WSL. Set EVAL_MAGIC_SH to select a specific `sh`. +**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed that resolves the same paths this workspace was prepared with — on Windows, Git Bash (Git for Windows). WSL resolves a different filesystem namespace, so run eval-magic inside WSL rather than dispatching into it. Set EVAL_MAGIC_SH to select a specific `sh`. After all dispatches (Cline): diff --git a/tests/golden/cline/runbook.golden.md b/tests/golden/cline/runbook.golden.md index f858583..f6ce60e 100644 --- a/tests/golden/cline/runbook.golden.md +++ b/tests/golden/cline/runbook.golden.md @@ -4,7 +4,7 @@ This runbook is for a human driving the run from a terminal. Work from this iter and copy-paste each step. The workspace is self-contained — you should not need the surrounding repo. -> **Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed — on Windows, use Git Bash (Git for Windows) or WSL. Set EVAL_MAGIC_SH to select a specific `sh`. +> **Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed that resolves the same paths this workspace was prepared with — on Windows, Git Bash (Git for Windows). WSL resolves a different filesystem namespace, so run eval-magic inside WSL rather than dispatching into it. Set EVAL_MAGIC_SH to select a specific `sh`. - **Skill under test:** widget-skill - **Mode:** revision — comparing `old_skill` vs `new_skill` diff --git a/tests/golden/codex/manifest-noguard.golden.md b/tests/golden/codex/manifest-noguard.golden.md index 117a33c..8312d24 100644 --- a/tests/golden/codex/manifest-noguard.golden.md +++ b/tests/golden/codex/manifest-noguard.golden.md @@ -8,7 +8,7 @@ Total dispatches: 2 In an agent session, read `dispatch.json` (sibling of this file) instead of this manifest. Each task has a `dispatch_prompt_path` field pointing at the file that holds the full prompt — dispatch the task with a short "read this file and follow it" instruction rather than inlining the prompt — plus exact paths for `run.json` and `timing.json`. -**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed — on Windows, use Git Bash (Git for Windows) or WSL. Set EVAL_MAGIC_SH to select a specific `sh`. +**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed that resolves the same paths this workspace was prepared with — on Windows, Git Bash (Git for Windows). WSL resolves a different filesystem namespace, so run eval-magic inside WSL rather than dispatching into it. Set EVAL_MAGIC_SH to select a specific `sh`. After all dispatches (Codex): diff --git a/tests/golden/codex/manifest.golden.md b/tests/golden/codex/manifest.golden.md index 7c802ff..75b3ee4 100644 --- a/tests/golden/codex/manifest.golden.md +++ b/tests/golden/codex/manifest.golden.md @@ -8,7 +8,7 @@ Total dispatches: 2 In an agent session, read `dispatch.json` (sibling of this file) instead of this manifest. Each task has a `dispatch_prompt_path` field pointing at the file that holds the full prompt — dispatch the task with a short "read this file and follow it" instruction rather than inlining the prompt — plus exact paths for `run.json` and `timing.json`. -**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed — on Windows, use Git Bash (Git for Windows) or WSL. Set EVAL_MAGIC_SH to select a specific `sh`. +**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed that resolves the same paths this workspace was prepared with — on Windows, Git Bash (Git for Windows). WSL resolves a different filesystem namespace, so run eval-magic inside WSL rather than dispatching into it. Set EVAL_MAGIC_SH to select a specific `sh`. After all dispatches (Codex): diff --git a/tests/golden/codex/runbook.golden.md b/tests/golden/codex/runbook.golden.md index 959daf3..6bbb1b9 100644 --- a/tests/golden/codex/runbook.golden.md +++ b/tests/golden/codex/runbook.golden.md @@ -4,7 +4,7 @@ This runbook is for a human driving the run from a terminal. Work from this iter and copy-paste each step. The workspace is self-contained — you should not need the surrounding repo. -> **Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed — on Windows, use Git Bash (Git for Windows) or WSL. Set EVAL_MAGIC_SH to select a specific `sh`. +> **Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed that resolves the same paths this workspace was prepared with — on Windows, Git Bash (Git for Windows). WSL resolves a different filesystem namespace, so run eval-magic inside WSL rather than dispatching into it. Set EVAL_MAGIC_SH to select a specific `sh`. - **Skill under test:** widget-skill - **Mode:** revision — comparing `old_skill` vs `new_skill` diff --git a/tests/golden/opencode/manifest.golden.md b/tests/golden/opencode/manifest.golden.md index a25dd60..9acfc20 100644 --- a/tests/golden/opencode/manifest.golden.md +++ b/tests/golden/opencode/manifest.golden.md @@ -8,7 +8,7 @@ Total dispatches: 2 In an agent session, read `dispatch.json` (sibling of this file) instead of this manifest. Each task has a `dispatch_prompt_path` field pointing at the file that holds the full prompt — dispatch the task with a short "read this file and follow it" instruction rather than inlining the prompt — plus exact paths for `run.json` and `timing.json`. -**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed — on Windows, use Git Bash (Git for Windows) or WSL. Set EVAL_MAGIC_SH to select a specific `sh`. +**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed that resolves the same paths this workspace was prepared with — on Windows, Git Bash (Git for Windows). WSL resolves a different filesystem namespace, so run eval-magic inside WSL rather than dispatching into it. Set EVAL_MAGIC_SH to select a specific `sh`. After all dispatches (OpenCode): diff --git a/tests/golden/opencode/runbook.golden.md b/tests/golden/opencode/runbook.golden.md index 23efe8a..bdd0107 100644 --- a/tests/golden/opencode/runbook.golden.md +++ b/tests/golden/opencode/runbook.golden.md @@ -4,7 +4,7 @@ This runbook is for a human driving the run from a terminal. Work from this iter and copy-paste each step. The workspace is self-contained — you should not need the surrounding repo. -> **Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed — on Windows, use Git Bash (Git for Windows) or WSL. Set EVAL_MAGIC_SH to select a specific `sh`. +> **Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed that resolves the same paths this workspace was prepared with — on Windows, Git Bash (Git for Windows). WSL resolves a different filesystem namespace, so run eval-magic inside WSL rather than dispatching into it. Set EVAL_MAGIC_SH to select a specific `sh`. - **Skill under test:** widget-skill - **Mode:** revision — comparing `old_skill` vs `new_skill`