eval-magic

A CLI for measuring whether an agent skill changes behavior.

eval-magic runs the same task in two controlled conditions—such as a new skill versus no skill, or an edited skill versus its previous version—and grades both results against shared assertions. It builds isolated task workspaces, stages skills, generates harness-specific dispatch instructions, ingests transcripts and final state, and produces comparison artifacts. You dispatch the agent sessions with Claude Code, Cline, Codex, OpenCode, or a descriptor-backed harness of your own.

The installed CLI is the primary manual. Start with eval-magic --help, and use eval-magic <command> --help whenever you reach a new phase.

Install

Git is required at runtime, plus a POSIX shell with jq: the dispatch and judge recipes eval-magic generates are POSIX command lines built on jq, xargs, tr, and wc. The shell that runs them has to resolve the same paths the workspace was prepared with. On Windows that is Git Bash (Git for Windows), with jq installed separately — Git for Windows does not bundle it. WSL resolves a different filesystem namespace, so run eval-magic inside WSL rather than dispatching into it. Set EVAL_MAGIC_SH to select a specific sh.

Windows support runs through Git Bash and is deprecated: a future release will require WSL.

Prebuilt binaries for macOS, Linux, and Windows are attached to each GitHub release.

macOS or Linux:

curl --proto '=https' --tlsv1.2 -LsSf \
  https://github.com/slowdini/eval-magic/releases/latest/download/eval-magic-installer.sh | sh

Windows PowerShell:

powershell -ExecutionPolicy Bypass -c \
  "irm https://github.com/slowdini/eval-magic/releases/latest/download/eval-magic-installer.ps1 | iex"

Or build and install from crates.io:

cargo install eval-magic

Confirm the installation with eval-magic --version.

Quickstart

Start in a skill directory containing SKILL.md:

cd path/to/my-skill
eval-magic init

init creates evals/evals.json with one valid seed case. Edit the prompt and expected behavior to describe a realistic task, add concrete assertions as the eval matures, then check the file:

eval-magic validate

Prepare the first comparison with the default Claude Code harness:

eval-magic run

Or select another registered harness:

eval-magic run --harness cline
eval-magic run --harness codex
eval-magic run --harness opencode

run prepares the campaign; it does not dispatch agents. Review the printed task and model-usage summary before continuing. Then read the generated RUNBOOK.md from beginning to end. It contains the exact dispatch, ingest, judge, finalize, and eval-magic teardown commands for that campaign and harness.

After finalization, open the generated benchmark.json to compare pass rates, token and duration measurements, and validity warnings. Use eval-magic aggregate --help when you need to combine multiple campaigns.

To evaluate an edit already in your working tree, snapshot the committed version and compare it with the edited file:

eval-magic snapshot --label baseline --ref HEAD
eval-magic run --mode revision

The command help and generated runbook describe baseline selection and the rest of the workflow.

How it works

Each eval case runs once per condition and repetition in its own clean Git repository. The two arms receive the same task and fixtures; only the condition under test changes. Assertions can combine LLM judgment with runner-owned command checks, transcript checks, and final diff limits. Scripted turns resume one native harness session so follow-up answers remain part of the same conversation.

Most harness features are declared in TOML descriptors. See the current registry and resolved data instead of relying on a static compatibility table:

eval-magic harness list
eval-magic harness show codex

Documentation and contributing

• eval-magic --help and subcommand help cover the complete CLI workflow and every flag.
• eval-magic docs lists the offline, version-matched guides embedded in the binary.
• eval-magic docs byoh explains descriptor authoring, testing, layering, and contribution. Its repository source is docs/guides/byoh.md.
• eval-magic docs isolation explains how live or installed skill sources can contaminate a comparison and how to verify isolation. Its source is docs/guides/isolation.md.
• docs/developer_overview.md maps the codebase, sources of truth, verification workflow, and internal documentation.

Issues and planned work are tracked in the GitHub issue tracker.

Development

Development carries the same host requirement as use: a POSIX shell with jq. The scripted-turn tests spawn #!/bin/sh harness stubs through the resolved shell and do not skip, so the suite cannot pass without one. Tests that need jq or symlink creation report a skip instead.

cargo fmt --check
cargo build
cargo test
cargo clippy --all-targets -- -D warnings

See AGENTS.md for repository conventions.

License

MIT