Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,10 @@ Markdown-carrying surfaces reuse: the shell-discovery errors, the `run` prefligh
it. `--help` is the one deliberate restatement (`AFTER_HELP` in `src/cli/help.rs`), hard-wrapped and
backtick-free because clap renders into a terminal; keep the two in step by hand.

Which platforms that requirement is honored on — and why preparing on Windows but dispatching from
WSL is a correctness boundary rather than a preference — is stated once under "Platform support" in
`docs/developer_overview.md`.

**Where user-facing warnings come from.** Library modules (`pipeline`, `workspace`, `sandbox`,
`adapters`) never print. They return warning strings on their result struct — `#[serde(skip)]` when
that struct is also a serialized artifact — and the `cli` handler prints them with the `⚠ ` prefix.
Expand Down
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,14 @@ The installed CLI is the primary manual. Start with `eval-magic --help`, and use
## Install

Git is required at runtime, plus a POSIX shell with `jq`: the dispatch and judge recipes eval-magic
generates are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. On Windows, run them in
Git Bash (Git for Windows) or WSL, and install `jq` separately — Git for Windows does not bundle it.
generates are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. The shell that runs them
has to resolve the same paths the workspace was prepared with. On Windows that is Git Bash (Git for
Windows), with `jq` installed separately — Git for Windows does not bundle it. WSL resolves a
different filesystem namespace, so run eval-magic inside WSL rather than dispatching into it.
Set `EVAL_MAGIC_SH` to select a specific `sh`.

Windows support runs through Git Bash and is deprecated: a future release will require WSL.

Prebuilt binaries for macOS, Linux, and Windows are attached to each
[GitHub release](https://github.com/slowdini/eval-magic/releases).

Expand Down
21 changes: 21 additions & 0 deletions docs/developer_overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,27 @@ following authorities:
infer one harness's flags or event shapes from another harness.
- Tests and golden artifacts for behavior that crosses a module or CLI boundary.

## Platform support

| Tier | Platform | Verified by |
| --- | --- | --- |
| Supported | Linux, macOS | the `ubuntu-latest` CI job |
| Deprecated | Windows, through Git Bash (Git for Windows) | the `windows-latest` CI job |
| Unsupported | preparing a workspace on Windows and dispatching it from WSL | — |

Windows support is deprecated in favor of WSL, and its removal is gated on #256, which replaces
the generated POSIX recipes with a runner-driven `eval-magic dispatch`. Until that lands, the
Windows runner stays green and Windows-native behavior is held to the same bar as any other
platform: a Windows failure is a real failure, not an accepted gap. Do not add new Windows-native
accommodation in the meantime.

The unsupported row is a correctness boundary rather than a preference. A generated recipe carries
the absolute paths of the host that prepared the workspace. Git Bash shares the Windows filesystem,
so those paths resolve; WSL resolves its own namespace, where a `C:\…` path names nothing. Nothing
in the tree translates between the two, so the split fails quietly instead of loudly.
`POSIX_TOOLING_REQUIREMENT` (`src/core/runtime.rs`) is the single wording every user-facing surface
reuses to state this; `src/cli/help.rs` restates it for clap by hand.

## Make and verify a change

Trace the user-visible behavior from the CLI handler into library-owned logic and artifacts before
Expand Down
9 changes: 6 additions & 3 deletions src/cli/help.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,12 @@
pub(super) const AFTER_HELP: &str = "\
REQUIREMENTS:
Git, plus a POSIX shell with jq, xargs, tr, and wc. The dispatch and judge
recipes in the generated RUNBOOK.md are POSIX command lines — on Windows,
run them in Git Bash (Git for Windows) or WSL, and install jq separately.
Set EVAL_MAGIC_SH to select a specific sh.
recipes in the generated RUNBOOK.md are POSIX command lines, and the shell
that runs them has to resolve the same paths the workspace was prepared
with. On Windows that is Git Bash (Git for Windows), with jq installed
separately. WSL resolves a different filesystem namespace, so run
eval-magic inside WSL rather than dispatching into it. Set EVAL_MAGIC_SH
to select a specific sh.

EXAMPLES:
# Scaffold a first eval and prepare its isolated comparison environments
Expand Down
26 changes: 22 additions & 4 deletions src/cli/run/orchestrate/shell.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,13 @@
//!
//! Unlike [`super::git`], a missing shell is a warning rather than an error.
//! `run` never dispatches — it prepares a workspace, and that workspace is
//! correct whatever shell prepared it. Preparing on Windows and dispatching from
//! WSL is a legitimate split, so this reports the gap and lets the run finish.
//! correct whatever shell prepared it, so this reports the gap and lets the run
//! finish.
//!
//! The shell that eventually dispatches still has to resolve the paths this host
//! wrote into the recipes, which keeps the gap host-local: Git Bash shares the
//! Windows filesystem, WSL resolves its own. See [`POSIX_TOOLING_REQUIREMENT`]
//! for the declared rule the warnings below defer to.

use std::path::Path;

Expand Down Expand Up @@ -39,8 +44,8 @@ pub(super) fn preflight_posix_tooling() -> Vec<String> {
fn tooling_warning(shell: Result<&Path, &str>, missing: Option<&str>) -> Option<String> {
if let Err(reason) = shell {
return Some(format!(
"{reason} The workspace and recipes below are still correct — prepare here and \
dispatch them from a POSIX shell."
"{reason} The workspace and recipes below are still correct — dispatch them from a \
POSIX shell on this host."
));
}
let reason = missing?;
Expand Down Expand Up @@ -73,6 +78,19 @@ mod tests {
assert!(warning.contains("Git Bash"), "{warning}");
}

/// An unqualified "dispatch them from a POSIX shell" reads as an invitation
/// to prepare here and dispatch from WSL — the one split
/// [`POSIX_TOOLING_REQUIREMENT`] rules out, and the one that fails quietly.
#[test]
fn a_missing_shell_confines_dispatch_to_the_host_that_prepared_the_workspace() {
let warning = tooling_warning(Err("no POSIX shell found. Use Git Bash."), None)
.expect("a host with no POSIX shell must be told");
assert!(
warning.contains("this host"),
"the warning must keep dispatch on the preparing host: {warning}"
);
}

/// A shell that is missing a recipe tool names the tool and the shell whose
/// PATH was searched, and still points at the declared requirement — Git for
/// Windows resolves a shell but bundles no `jq`, which is exactly this case.
Expand Down
33 changes: 31 additions & 2 deletions src/core/runtime.rs
Original file line number Diff line number Diff line change
Expand Up @@ -105,9 +105,19 @@ pub fn run_git(args: &[&str], cwd: &Path) -> GitOutput {
/// for Windows supplies `sh`, `xargs`, `tr`, and `wc` but not `jq`, so guidance
/// naming only the shell would send an operator to a setup that still walls out
/// at the judge step.
///
/// It also separates the two Windows options rather than listing them side by
/// side. A generated recipe carries the preparing host's absolute paths, so the
/// dispatching shell has to resolve those same paths. Git Bash does — it shares
/// the Windows filesystem. WSL does not: `C:\…` names nothing in its namespace,
/// so WSL is correct only when eval-magic itself runs inside it. Nothing here
/// translates between the two, and a split across that boundary fails quietly
/// rather than loudly.
pub(crate) const POSIX_TOOLING_REQUIREMENT: &str = "eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, \
`tr`, and `wc`. Run them in a POSIX shell with `jq` installed — on Windows, use Git Bash \
(Git for Windows) or WSL. Set EVAL_MAGIC_SH to select a specific `sh`.";
`tr`, and `wc`. Run them in a POSIX shell with `jq` installed that resolves the same paths \
this workspace was prepared with — on Windows, Git Bash (Git for Windows). WSL resolves a \
different filesystem namespace, so run eval-magic inside WSL rather than dispatching into \
it. Set EVAL_MAGIC_SH to select a specific `sh`.";

/// `sh` locations inside a Git for Windows install, given the `git --exec-path`
/// directory (`<root>/mingw64/libexec/git-core` — hence three levels up).
Expand Down Expand Up @@ -358,6 +368,25 @@ mod tests {
assert!(shell.is_file(), "{} is not a file", shell.display());
}

/// The declared requirement has to separate the two Windows options rather
/// than list them as equivalent. Git Bash shares the Windows filesystem, so a
/// workspace prepared by a native run dispatches from it correctly. WSL
/// resolves a different namespace, where the `C:\…` paths a native run wrote
/// name nothing — so WSL is only correct when eval-magic itself runs inside
/// it. Listing the two side by side invites a split that silently cannot work.
#[test]
fn the_declared_requirement_places_wsl_around_eval_magic_not_downstream_of_it() {
assert!(
POSIX_TOOLING_REQUIREMENT.contains("Git Bash"),
"{POSIX_TOOLING_REQUIREMENT}"
);
assert!(
POSIX_TOOLING_REQUIREMENT.contains("inside WSL"),
"WSL must be named as where eval-magic runs, not somewhere to dispatch \
into: {POSIX_TOOLING_REQUIREMENT}"
);
}

#[test]
fn require_posix_toolchain_names_the_tool_that_is_missing() {
let error = require_posix_toolchain(&["eval-magic-not-a-real-tool"])
Expand Down
2 changes: 1 addition & 1 deletion tests/golden/claude-code/manifest-nomodel.golden.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Total dispatches: 2

In an agent session, read `dispatch.json` (sibling of this file) instead of this manifest. Each task has a `dispatch_prompt_path` field pointing at the file that holds the full prompt — dispatch the task with a short "read this file and follow it" instruction rather than inlining the prompt — plus exact paths for `run.json` and `timing.json`.

**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed — on Windows, use Git Bash (Git for Windows) or WSL. Set EVAL_MAGIC_SH to select a specific `sh`.
**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed that resolves the same paths this workspace was prepared with — on Windows, Git Bash (Git for Windows). WSL resolves a different filesystem namespace, so run eval-magic inside WSL rather than dispatching into it. Set EVAL_MAGIC_SH to select a specific `sh`.

After all dispatches (Claude Code):

Expand Down
2 changes: 1 addition & 1 deletion tests/golden/claude-code/manifest.golden.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Total dispatches: 2

In an agent session, read `dispatch.json` (sibling of this file) instead of this manifest. Each task has a `dispatch_prompt_path` field pointing at the file that holds the full prompt — dispatch the task with a short "read this file and follow it" instruction rather than inlining the prompt — plus exact paths for `run.json` and `timing.json`.

**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed — on Windows, use Git Bash (Git for Windows) or WSL. Set EVAL_MAGIC_SH to select a specific `sh`.
**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed that resolves the same paths this workspace was prepared with — on Windows, Git Bash (Git for Windows). WSL resolves a different filesystem namespace, so run eval-magic inside WSL rather than dispatching into it. Set EVAL_MAGIC_SH to select a specific `sh`.

After all dispatches (Claude Code):

Expand Down
2 changes: 1 addition & 1 deletion tests/golden/claude-code/runbook.golden.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ This runbook is for a human driving the run from a terminal. Work from this iter
and copy-paste each step. The workspace is self-contained — you should not need the surrounding
repo.

> **Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed — on Windows, use Git Bash (Git for Windows) or WSL. Set EVAL_MAGIC_SH to select a specific `sh`.
> **Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed that resolves the same paths this workspace was prepared with — on Windows, Git Bash (Git for Windows). WSL resolves a different filesystem namespace, so run eval-magic inside WSL rather than dispatching into it. Set EVAL_MAGIC_SH to select a specific `sh`.

- **Skill under test:** widget-skill
- **Mode:** revision — comparing `old_skill` vs `new_skill`
Expand Down
2 changes: 1 addition & 1 deletion tests/golden/cline/manifest.golden.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Total dispatches: 2

In an agent session, read `dispatch.json` (sibling of this file) instead of this manifest. Each task has a `dispatch_prompt_path` field pointing at the file that holds the full prompt — dispatch the task with a short "read this file and follow it" instruction rather than inlining the prompt — plus exact paths for `run.json` and `timing.json`.

**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed — on Windows, use Git Bash (Git for Windows) or WSL. Set EVAL_MAGIC_SH to select a specific `sh`.
**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed that resolves the same paths this workspace was prepared with — on Windows, Git Bash (Git for Windows). WSL resolves a different filesystem namespace, so run eval-magic inside WSL rather than dispatching into it. Set EVAL_MAGIC_SH to select a specific `sh`.

After all dispatches (Cline):

Expand Down
2 changes: 1 addition & 1 deletion tests/golden/cline/runbook.golden.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ This runbook is for a human driving the run from a terminal. Work from this iter
and copy-paste each step. The workspace is self-contained — you should not need the surrounding
repo.

> **Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed — on Windows, use Git Bash (Git for Windows) or WSL. Set EVAL_MAGIC_SH to select a specific `sh`.
> **Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed that resolves the same paths this workspace was prepared with — on Windows, Git Bash (Git for Windows). WSL resolves a different filesystem namespace, so run eval-magic inside WSL rather than dispatching into it. Set EVAL_MAGIC_SH to select a specific `sh`.

- **Skill under test:** widget-skill
- **Mode:** revision — comparing `old_skill` vs `new_skill`
Expand Down
2 changes: 1 addition & 1 deletion tests/golden/codex/manifest-noguard.golden.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Total dispatches: 2

In an agent session, read `dispatch.json` (sibling of this file) instead of this manifest. Each task has a `dispatch_prompt_path` field pointing at the file that holds the full prompt — dispatch the task with a short "read this file and follow it" instruction rather than inlining the prompt — plus exact paths for `run.json` and `timing.json`.

**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed — on Windows, use Git Bash (Git for Windows) or WSL. Set EVAL_MAGIC_SH to select a specific `sh`.
**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed that resolves the same paths this workspace was prepared with — on Windows, Git Bash (Git for Windows). WSL resolves a different filesystem namespace, so run eval-magic inside WSL rather than dispatching into it. Set EVAL_MAGIC_SH to select a specific `sh`.

After all dispatches (Codex):

Expand Down
2 changes: 1 addition & 1 deletion tests/golden/codex/manifest.golden.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Total dispatches: 2

In an agent session, read `dispatch.json` (sibling of this file) instead of this manifest. Each task has a `dispatch_prompt_path` field pointing at the file that holds the full prompt — dispatch the task with a short "read this file and follow it" instruction rather than inlining the prompt — plus exact paths for `run.json` and `timing.json`.

**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed — on Windows, use Git Bash (Git for Windows) or WSL. Set EVAL_MAGIC_SH to select a specific `sh`.
**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed that resolves the same paths this workspace was prepared with — on Windows, Git Bash (Git for Windows). WSL resolves a different filesystem namespace, so run eval-magic inside WSL rather than dispatching into it. Set EVAL_MAGIC_SH to select a specific `sh`.

After all dispatches (Codex):

Expand Down
2 changes: 1 addition & 1 deletion tests/golden/codex/runbook.golden.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ This runbook is for a human driving the run from a terminal. Work from this iter
and copy-paste each step. The workspace is self-contained — you should not need the surrounding
repo.

> **Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed — on Windows, use Git Bash (Git for Windows) or WSL. Set EVAL_MAGIC_SH to select a specific `sh`.
> **Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed that resolves the same paths this workspace was prepared with — on Windows, Git Bash (Git for Windows). WSL resolves a different filesystem namespace, so run eval-magic inside WSL rather than dispatching into it. Set EVAL_MAGIC_SH to select a specific `sh`.

- **Skill under test:** widget-skill
- **Mode:** revision — comparing `old_skill` vs `new_skill`
Expand Down
2 changes: 1 addition & 1 deletion tests/golden/opencode/manifest.golden.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Total dispatches: 2

In an agent session, read `dispatch.json` (sibling of this file) instead of this manifest. Each task has a `dispatch_prompt_path` field pointing at the file that holds the full prompt — dispatch the task with a short "read this file and follow it" instruction rather than inlining the prompt — plus exact paths for `run.json` and `timing.json`.

**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed — on Windows, use Git Bash (Git for Windows) or WSL. Set EVAL_MAGIC_SH to select a specific `sh`.
**Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed that resolves the same paths this workspace was prepared with — on Windows, Git Bash (Git for Windows). WSL resolves a different filesystem namespace, so run eval-magic inside WSL rather than dispatching into it. Set EVAL_MAGIC_SH to select a specific `sh`.

After all dispatches (OpenCode):

Expand Down
2 changes: 1 addition & 1 deletion tests/golden/opencode/runbook.golden.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ This runbook is for a human driving the run from a terminal. Work from this iter
and copy-paste each step. The workspace is self-contained — you should not need the surrounding
repo.

> **Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed — on Windows, use Git Bash (Git for Windows) or WSL. Set EVAL_MAGIC_SH to select a specific `sh`.
> **Requires:** eval-magic's dispatch and judge recipes are POSIX command lines built on `jq`, `xargs`, `tr`, and `wc`. Run them in a POSIX shell with `jq` installed that resolves the same paths this workspace was prepared with — on Windows, Git Bash (Git for Windows). WSL resolves a different filesystem namespace, so run eval-magic inside WSL rather than dispatching into it. Set EVAL_MAGIC_SH to select a specific `sh`.

- **Skill under test:** widget-skill
- **Mode:** revision — comparing `old_skill` vs `new_skill`
Expand Down
Loading