OpenEval

OpenEval is a local-first, harness-agnostic evaluation dashboard for agent CLIs. Every harness is a JSON descriptor; the bundled Claude Code, Codex, and ncode adapters use the same format as user descriptors under harnesses/. OpenEval runs repeatable cases, grades results with deterministic and rubric-based checks, persists run history to SQLite, and turns the results into an operator-friendly Next.js dashboard.

The project is designed for people who want to compare agent behavior on practical software tasks, inspect the exact traces behind a score, and keep their private run history on their own machine.

Launch Film

openeval-launch-preview.mp4

The 29.5-second v0.1.1 launch film shows the real OpenEval dashboard: live sessions, collection history, repeatable runs, comparisons, coverage, telemetry, and accuracy audits. The current application release is v0.1.5.

Download the full-resolution MP4 · Open the v0.1.1 media release · Latest release: v0.1.5

Build Credits

• OpenEval: developed with Noumena Code, Claude Code (including Fable 5 and Sonnet 5), and OpenAI Codex. Fable 5 contributed substantially to the application itself.
• Launch film: GPT-5.6 Sol (High) led the edit, composition, visual QA, and final release pass; GPT-5.6 Luna (xhigh) contributed additional iteration passes.
• Production stack: Codex, ImageGen, TouchDesigner, HyperFrames, GSAP, CDP Recorder, and FFmpeg.
• Acknowledgment: huge thanks to @KingBootoshi from righttointelligence.org.

What's New in v0.1.5

OpenEval v0.1.5 is the release-hardening pass: clean installs are documented and gated, evidence delivery is bounded, judge jobs are durable, API recovery is more consistent, and the dense observation surfaces are easier to read on desktop and mobile.

Safer installation and release checks

• Supported Node 20 and npm 10 versions are explicit in the package engines, .nvmrc, doctor output, README, and contributor guide.
• .env.example documents data roots, harness defaults, judge settings, host allowlists, scan budgets, and transcript streaming controls.
• npm run verify:ci and npm run verify:release provide one repeatable validation path from clean install through production build.
• The full dependency audit is clean, and the candidate-scope public-upload audit checks tracked and untracked release material for local paths, identities, and secret-like leakage.

More durable evidence

• Judge selections and lease ownership are persisted and fenced, so a stale or interrupted worker cannot silently replace current evidence.
• Transcript records, artifact previews, byte ranges, and report bundles stay bounded or stream from disk with explicit truncation, range, hash, and provenance metadata.
• Collection, report, artifact, cancel, and event-stream failures share readable JSON/SSE recovery envelopes.
• Polling coalesces duplicate requests, and source revisions invalidate stale outcome judgments instead of displaying old scores as current.

Easier to operate

• Harness selection falls back to an available configured harness when no explicit harness is supplied and gives a concrete install/configuration error when none is available.
• Onboarding, judge setup, retry/readiness states, review receipts, mixed-method explanations, Timeline copy, and evidence denominators are clearer.
• Mobile navigation, New Run ordering, evidence grouping, and compact status/recovery surfaces are less crowded.

The release gate verifies deterministic and corpus-backed evidence while preserving Unknown for trace, visual, LLM-judge, provider, assistive-technology, and human-review surfaces that were not exercised by the corresponding runtime proof.

Read the complete v0.1.5 release notes · Compare v0.1.4...v0.1.5

What's New in v0.1.4

OpenEval v0.1.4 is the broad stability and evaluation-flow release. It ties the Evaluate pages together, makes benchmark evidence easier to watch and compare, preserves the strong local observation surfaces, and adds explicit boundaries wherever the local product does not yet have provider or human proof.

A more coherent evaluation workflow

• Runs, Leaderboard, Compare, Cases, New Run, and Accuracy now read as one ordered Evaluate workflow instead of isolated pages.
• Run diagnostics use explicit chart titles, numeric ticks, units, accessible point labels, ranked throughput bars, and honest measured/estimated/missing cost treatment.
• Run detail has a watchable evaluation pulse with current/next case, lifecycle phase, progress, timing, connection state, bounded replay, and direct evidence navigation.
• Compare surfaces A/B pass-rate, throughput, visual-contract, and error deltas, plus bounded side-by-side visual artifact review.
• New Run begins with a focused Core suite and provides Visual lab, Reasoning, and Everything presets with planned execution and budget visibility.
• Cases provides a direct Core-suite starter path and a Creative sampler route for operators who do not yet know which benchmark to choose.
• Selected cases that disappear under a filter remain visible and recoverable; “Clear visible,” “Clear all,” and “Show selected” have distinct meanings.

A wider, more useful benchmark library

• The runnable corpus now contains 35 cases across agentic SWE, reasoning, single-tool, and visual-code work.
• Visual/code coverage includes supplied-data SVG, pixel art, 3D depth, isometric voxel worlds, data-story cards, route planners, accessible forms, dashboards, posters, diagrams, sortable tables, responsive layouts, and Markdown runbooks.
• The catalog keeps deterministic structural checks separate from screenshot, pixel-quality, judge, and human review. Bytes, hashes, selectors, and element counts are evidence receipts, not visual-quality verdicts.
• The ledger now includes 200 user-flow stories across brand-new, beginner, intermediate, and expert journeys, plus a 250-row UX pass covering sizing, visibility, contrast, color, responsive density, and interaction states.

Stronger accuracy and evidence posture

• The strict corpus audit covers 35/35 oracle scripts and 35/35 known-bad scripts.
• Accuracy surfaces expose evidence tiers, weaknesses, uncertainties, direct next actions, judge posture, and the distinction between configuration-only and executed proof.
• Product-flow coverage, capability nuclei, evidence-lens rows, and runnable benchmark cases have separate manifests and denominators.
• Unknown and pending states remain visible for unexercised provider, trace, visual, LLM-judge, assistive, and human-review evidence.

Observation remains the deepest surface

• Live, Collection, and Timeline retain bounded semantic transcript and tool-call views for Codex, Claude, ncode, and other configured sources.
• Tool calls/results are paired by identity where available, with names, status, duration, bounded arguments/results, and tool-search records.
• Raw transcripts remain authoritative; derived windows, FTS fields, DOM projections, and API payloads stay bounded.
• Collection and Timeline report discovered, scanned, parsed, dropped, unscanned, signal, judged, heuristic, and no-signal populations explicitly.
• Measured, inferred, missing, malformed, stale, archived, and incomplete provenance remains visible for model, token, cost, duration, tools, and trace structure.
• Child traces and judge sessions retain lineage and stay out of unrelated outcome denominators.

Stability, accessibility, and responsive behavior

• Run cancellation now propagates through in-flight harness/grader work and closes terminal SSE streams honestly.
• Loading, error, retry, and empty states are present across the primary evaluation and observation routes.
• Onboarding is session-aware, route-aware, focus-contained, Escape-dismissible, and labeled for assistive technology.
• Harness discovery reports failed probes, missing binaries, wrapper failures, and revoked authentication with a concrete recovery action.
• API requests are bounded and field-tagged; mutation routes enforce local/same-origin safety; artifact access is server-owned and symlink-safe.
• The sidebar release tag now comes from package metadata and links directly to the matching GitHub release tag.
• Primary evaluation routes were checked at 390×844 with zero positive document overflow and no captured browser console warnings/errors.

Release proof

The release source passed TypeScript typecheck, the full test suite, lint, production build, strict accuracy audit, doctor, public-upload audit, and diff hygiene checks. The release deliberately does not claim every harness has fresh authenticated provider success, that artifact structure proves pixel quality, or that source/browser checks replace full assistive-technology and human visual review.

Read the complete v0.1.4 release notes · Compare v0.1.3...v0.1.4

What's New in v0.1.3

OpenEval v0.1.3 makes large local transcript collections faster to inspect and much harder to misread:

• 93.5% smaller Live list payload: the measured 100-session response fell from 2.65 MB to 171 KB, while full trace, tool, queue, file, and usage detail remains available on demand.
• Exact source inventory: Gemini discovery now counts verified ~/.gemini/tmp/**/logs.json artifacts instead of unrelated configuration JSON; same-machine known-source discovery improved from 60 ms to 26 ms median.
• Visible data fidelity: Collection separates parseable and detect-only files and reports measured, inferred, missing, malformed, stale, archived, and incomplete evidence explicitly.
• Honest populations: Live shows discovered, scanned, parsed, dropped, and unscanned files; Timeline distinguishes signal, judged, heuristic, and no-signal sessions and displays the actual outcome denominator.
• Stronger proof UX: artifact byte receipts, visual-contract boundaries, strict accuracy gates, accessible case rows and session drawers, and resilient loading/error states make evidence easier to audit.

Read the v0.1.3 release notes · Compare v0.1.2...v0.1.3

Highlights

• Descriptor-first harnesses: bundled ncode, Claude Code, and Codex harnesses plus user harnesses under harnesses/ are all validated JSON descriptors.
• Fast at scale: a fingerprint-validated scan cache and a single shared full-history parse keep warm Collection/home loads near-instant at 1,000+ sessions, and Live polls send bytes, not megabytes, when nothing changed.
• Weighted graders: combine shell checks, file assertions, transcript regexes, JSON path checks, trace-shape checks, git diff checks, and optional LLM rubric judges.
• Run dashboard: browse recent runs, pass rates, token and cost summaries, case outcomes, and per-case trace detail.
• Live trace view: inspect recent local CLI sessions with measured/inferred/missing/malformed provenance for usage, cost, model, duration, tool calls, and trace structure; discovered/scanned/parsed counts keep the latest-session slice explicit.
• Collection: every harness's sessions on the machine — full-history totals, parseable versus detect-only inventory, session-evidence fidelity, weekly usage rollups, top projects, OpenRouter-based cost estimates (always tilde-marked), and a permanent archive so sessions survive harness log pruning.
• Full-text search: an FTS5 index over every transcript's conversational text; one search box across Claude Code, Codex, ncode, and any other parseable source, with a read-only transcript viewer for every hit.
• Timeline & Impact: adoption markers (skills, MCP servers, subagents, models), before/after impact deltas with explicit outcome denominators and confound flags, automatic change-point detection, and opt-in LLM-judge outcome refinement (OpenRouter or a local CLI; verdicts persist and upgrade the heuristic scores).
• Accuracy audits: review whether cases have deterministic proof, known-bad rejection, oracle coverage, and rubric backstops.
• Local-first storage: run history, workdirs, transcripts, SQLite WAL files, and personal stats live under data/, which is ignored by Git.
• Fixture-based repeatability: each case can copy a tiny repository from fixtures/, initialize git, run the harness, and grade the changed workdir.

Dashboard Tour

Start the app and open the dashboard:

nvm use 20
npm ci
npm run doctor
npm run dev

Open http://localhost:3000 (or the port printed by Next.js). Node 20 and npm 10 are the supported release toolchain. The dashboard itself is local-first, but running an evaluation requires at least one supported agent CLI installed and authenticated on the same machine.

For a clean release-style check, stop the dev server before running the build:

npm run verify:release
npm start

next dev and next build both write .next; always restart the server after switching between them.

The dashboard currently exposes these primary routes:

Route	Purpose
/	Summary dashboard with total runs, case count, latest pass rate, average tokens, recent runs, case library breakdown, and last-run metrics.
/runs	Historical run table with status, pass/fail counts, runner, cost, token, and duration information.
/runs/new	New-run wizard for choosing harness, runner, parallelism, samples, category/tag filters, and explicit cases.
/runs/[id]	Run detail page with case list, grader results, live run events, final answers, tool calls, cost, duration, and usage.
/runs/[id]/case/[caseId]	Full per-case trace with transcript entries, tool calls, grader output, and artifact links.
/runs/[id]/bench	Bench view for per-case cost, token, duration, and throughput diagnostics.
/runs/leaderboard	Cross-harness comparison by pass rate, cost, tokens, and speed.
/runs/compare	Side-by-side comparison of selected runs.
/harnesses	Harness discovery, binary availability, capabilities, version probe, and sample command display.
/accuracy	Case quality audit: evidence tiers, oracle coverage, known-bad rejection, and weak grader warnings.
/live	Live local trace intelligence from the selected harness descriptor's liveTrace roots, or the descriptor-driven default harness.
/collection	All harnesses at once: source discovery, full-history totals (archive included), weekly usage rollups, top projects, and full-text search across every transcript.
/collection/session	Read-only transcript viewer for any discovered session file (search hits and session tables link here).
/collection/timeline	Adoption timeline, before/after impact analysis, detected metric shifts, and LLM-judge outcome refinement.
/cases	Case library browser with categories, tags, prompts, setup, and grader definitions.
/settings	Operator settings surface for local configuration.

Documentation

Document	Purpose
Harness Authoring	Write and validate .harness.json descriptors.
Case Authoring	Write .case.json cases, fixtures, runner settings, budgets, oracles, visual contracts, and grader arrays.
Graders	Understand every grader type, parameters, evidence tier, weights, forbidden checks, and pass-threshold math.
Architecture	Follow the dataflow from case selection through workdir prep, harness spawn, parsing, SQLite persistence, dashboard routes, SSE, and live traces.
Redaction	Layered redaction: local paths, credential shapes, and the optional on-device Rampart PII model.

npm scripts

Script	Command	Purpose
dev	next dev	Start the local Next.js dashboard.
build	next build	Build the production Next.js app.
start	next start	Serve the production build.
lint	next lint	Run the Next.js ESLint pass.
typecheck	tsc --noEmit	Run TypeScript without emitting files.
doctor	tsx scripts/doctor.ts	Diagnose dev-runtime health: Node vs .nvmrc, better-sqlite3 binding, stale .next cache (--fix clears), port 3000, read-only DB quick_check, disk headroom.
test	node --import tsx --test --test-reporter=dot tests/*.test.ts	Run the full test suite (what CI runs); judge by exit code — the dot reporter keeps output terse.
test:live	node --import tsx --test tests/live.test.ts	Run only the live-trace tests.
test:telemetry	node --import tsx --test tests/telemetry.test.ts	Run only the telemetry tests.
test:redaction	node --import tsx --test tests/redaction.test.ts	Run only the redaction-pipeline tests.
pricing:check	tsx scripts/check-pricing-catalog.ts	Compare the bundled pricing catalog against live OpenRouter rates.
bench:live	tsx scripts/bench-live.ts	Benchmark live-session parser throughput on deterministic corpora.
bench:scan	tsx scripts/perf/scan-bench.ts	Benchmark the end-to-end scan pipeline (cold/warm/hot) on an isolated corpus.
run:eval	tsx lib/cli/run.ts	Start evaluation runs from the CLI.
run:case	tsx lib/cli/run.ts --case	Run one case id through the evaluation CLI.
run:headless	tsx lib/cli/run.ts --runner headless	Run evaluations with the headless runner preselected.
selftest	tsx lib/cli/selftest.ts	Validate DB access, cases, harness descriptors/parsers, no-op baselines, known-bad rejection, and oracles.
judge:windows	tsx scripts/judge-windows.ts	Timeline helper: judge sessions inside adoption-marker windows.
audit:accuracy	tsx lib/cli/accuracy.ts	Audit case proof strength, oracle coverage, known-bad coverage, and evidence tiers.
audit:accuracy:strict	tsx lib/cli/accuracy.ts --strict --strict-known-bad	Release gate for malformed cases, dangling oracle scripts, deterministic/trace gaps, and missing known-bad fixtures.
report	tsx lib/cli/report.ts	Generate a Markdown run report or portable bundle (--bundle, --redact); every run also records a reproducibility manifest.

The test* scripts set OPENEVAL_DATA_ROOT=.test-data so test runs never touch your real data/ history.

What OpenEval Tests

Cases live in cases/ and are grouped by category.

Category	What it exercises	Examples
agentic-swe	Multi-step software edits against small repositories. These cases exercise reading files, understanding tests, editing source, and running commands.	FizzBuzz bug fix, CommonJS to ESM conversion, shape feature addition, multi-bug pipeline repair.
single-tool	Narrow tasks that isolate one tool or one tiny workflow.	Count errors, sum numbers, create README, preserve locked config.
reasoning	No-code reasoning tasks where the final answer can be checked by regex or rubric.	Code trace, recursion, word problem, proof sketch, counterfactual planning.
visual-code	Generated UI/SVG/web artifacts with deterministic contracts and preview-oriented checks.	SVG status card, web eval dashboard.

Each .case.json file declares:

• identity: id, category, name, description, tags, difficulty
• prompt: the exact instruction sent to the harness
• setup: optional fixture copy and git initialization
• runner budget: max turns, timeout, model, permissions, extra args
• graders: weighted checks that decide pass/fail
• pass threshold: the weighted ratio required to pass

Grader System

The grader registry is implemented in lib/grader/index.ts. A case can combine multiple graders and assign optional weights. A failed grader with forbidden: true is a hard failure even when the weighted score would otherwise pass.

Common grader types include:

Grader	What it proves
exit_code	A command exited with status 0.
tests_pass	A test command passed, including TAP-style pass/fail parsing where available.
file_contains	A file matches a required regex.
file_exists	A required file exists, or does not exist when negated.
file_eq	A file exactly matches expected text.
files_unchanged	Protected files match their fixture baseline.
git_diff_contains	The final diff contains an expected pattern.
checksum	A file or artifact matches an expected checksum.
regex_match	Final answer, stdout, or transcript matches a regex.
json_path	A JSON value at a dotted path equals the expected value.
step	Tool-call traces follow expected shape, order, or count constraints.
rubric_llm	A separate judge harness grades a response against a rubric.
manual	Marks a case for human review instead of fully automated proof.

The accuracy audit in lib/accuracy.ts classifies grader evidence into deterministic, trace, visual, LLM-judge, and manual tiers. Use it to identify cases that need stronger proof.

npm run audit:accuracy
npm run audit:accuracy:strict

Harnesses

OpenEval separates "how to run an agent CLI" from "how to grade a case" through the HarnessAdapter interface in lib/adapters/types.ts.

Built-in harnesses:

• ncode: descriptor for NCode stream JSON output.
• claude-code: stream JSON adapter for Claude Code-compatible output.
• codex: adapter for Codex CLI/App session event formats.

Descriptor harnesses:

• Add harnesses/<id>.harness.json for CLIs that emit JSON or JSONL with stable field paths.
• The descriptor is loaded by lib/adapters/loader.ts and converted into a generic adapter by lib/adapters/generic.ts.
• harnesses/hermes.harness.json is the example descriptor.

Discovery:

npm run run:eval -- --list-harnesses

Discovery probes PATH, well-known paths declared by each adapter, and <bin> --version. The Harnesses page shows availability, version, capabilities, and the sample command OpenEval would run.

Running Evaluations

Run one case:

npm run run:case -- --harness codex swe-fix-fizzbuzz

Replace codex with an available harness from npm run run:eval -- --list-harnesses. If you omit --harness, OpenEval uses OPENEVAL_DEFAULT_HARNESS when set and otherwise selects an available built-in harness after probing. If no supported CLI is installed, the command reports the missing binaries and the discovery command to run next.

Run the full suite with the default harness:

npm run run:eval -- --runner headless --parallel 4

Run a filtered slice:

npm run run:eval -- --category agentic-swe --tag bugfix

Compare harnesses with repeated samples:

npm run run:eval -- --harness claude-code --harness codex --samples 3

Useful flags:

Flag	Meaning
--case <id>	Run one case id.
--category <name>	Restrict to one category.
--tag <tag>	Restrict to cases with a tag.
--parallel <n>	Number of worker slots.
--samples <n>	Repeat each selected case n times.
--runner headless	Spawn the harness directly.
--runner tmux	Run each harness command inside tmux for attachable observation.
--harness <id>	Select one registered harness adapter.
--model <name>	Pass a model selection through to adapters that support it.
--list-harnesses	Print harness discovery results and exit.

CLI exit codes:

Code	Meaning
0	Every case reached a terminal status with no infrastructure error. Individual cases may still be failed.
1	The run itself failed, or a general CLI error.
2	Infrastructure error: any case ended in error (runner crash, workdir prep failure, judge unavailable) or never reached a terminal status (stranded).
3	A grader crashed. Takes precedence over 2.

Local Data and Privacy

OpenEval intentionally keeps operator stats local.

Ignored runtime data:

• data/eval.db
• data/eval.db-wal
• data/eval.db-shm
• data/live-cache.db
• data/transcripts/
• data/workdirs/
• .codex/
• .ncode/
• state.yaml
• tsconfig.tsbuildinfo

This means you can make a public GitHub repository without publishing local run history, traces, private machine paths, or personal dashboard totals. Your dashboard will still keep using the existing local data/ directory as long as you do not delete it.

The fixture files under fixtures/ may contain intentionally fake secret-like strings, such as API_KEY=prod-secret-12345, because some cases test whether an agent preserves protected files instead of deleting or leaking them. Treat those as adversarial test data, not credentials.

Run the public upload audit before publishing:

bash scripts/public-upload-audit.sh --all

The audit checks the tracked and untracked non-ignored candidate files for local-only data, the disallowed public-facing identity string, private machine paths, and an inventory of secret-like fixture files that should be reviewed when fixtures change. Run it against the exact candidate you intend to stage or archive; do not rely on a clean result from an older tag.

GitHub Project Hygiene

The repository includes public-facing GitHub scaffolding:

• CONTRIBUTING.md for local setup, contribution areas, case quality expectations, and pull request checks.
• SECURITY.md for private vulnerability reporting and local-data boundaries.
• SUPPORT.md for synthetic reproduction guidance.
• .github/pull_request_template.md for verification and privacy checks.
• .github/ISSUE_TEMPLATE/bug_report.yml for public-safe bug reports.
• .github/ISSUE_TEMPLATE/evaluation_case.yml for case and grader proposals.

OpenEval is open source under the MIT License.

Project Layout

app/                         Next.js App Router pages and API routes
app/api/runs/                run creation, run detail, case detail, telemetry, events
app/api/live/                live trace intelligence API
app/api/harnesses/           harness discovery and leaderboard APIs
components/                  dashboard, run detail, live view, tables, controls
lib/adapters/                harness registry, built-ins, generic descriptor loader
lib/runner/                  headless and tmux runner implementations
lib/grader/                  grader registry and LLM judge path
lib/cli/                     CLI entrypoints for runs, selftest, accuracy audit
lib/cases.ts                 case loading and zod validation
lib/db.ts                    SQLite schema, migrations, and query helpers
lib/live.ts                  local trace parsing and live dashboard summaries
lib/redaction.ts             display redaction for local paths/usernames
cases/                       public case definitions and oracle scripts
fixtures/                    tiny repositories and inputs copied into workdirs
harnesses/                   descriptor-driven harness definitions
scripts/                     maintenance and public-readiness helpers
data/                        ignored local SQLite DB, transcripts, and workdirs

Development

Install dependencies:

npm install

Run the dashboard:

npm run dev

Verify TypeScript:

npm run typecheck

Run the full test suite (what CI runs):

npm test

Or focused subsets:

npm run test:live
npm run test:telemetry
npm run test:redaction

Run internal selftests:

npm run selftest

Build for production:

npm run build
npm start

Troubleshooting

When local dev misbehaves, run the doctor first — the most common failures are environmental, not app bugs:

npm run doctor          # diagnose
npm run doctor -- --fix # also clear a flagged .next cache
npm run doctor -- --json

What it checks and what the findings mean:

Symptom	Doctor finding	Fix
Missing-chunk errors, blank pages, hung dev server	Stale or incomplete .next cache	rm -rf .next (or --fix), restart npm run dev. Always restart the dev server after next build.
ERR_DLOPEN_FAILED / NODE_MODULE_VERSION mismatch on startup	better-sqlite3 binding fails to load	npm rebuild better-sqlite3 — happens after switching Node majors.
Dev server silently starts on port 3001	Port 3000 occupied	The doctor reports the process report-only; stop the stale server yourself if it is a leftover.
Node behaves differently than CI	Running Node differs from .nvmrc	nvm use, then rebuild native modules.
Dashboard errors reading history	data/eval.db fails PRAGMA quick_check	Back up data/ before anything else; the check itself is strictly read-only.
Builds or runs fail with ENOSPC-shaped errors	Low disk headroom	Free space; workdirs and WAL files grow during runs.

The doctor never kills processes and never writes to data/ — its only mutating action is clearing .next, and only with --fix.

Adding a Case

1. Add a tiny fixture repository under fixtures/<fixture-name>/ if the case needs files.
2. Add cases/<category>/<case-id>.case.json.
3. Include deterministic graders wherever possible.
4. Add known-bad oracle scripts when a case needs accuracy-audit confidence.
5. Run the case locally.
6. Run npm run audit:accuracy and strengthen weak proof surfaces before treating the case as reliable.

Example:

{
  "id": "swe-fix-fizzbuzz",
  "category": "agentic-swe",
  "name": "Fix the FizzBuzz bug",
  "tags": ["bugfix", "node"],
  "prompt": "There is a bug in src/fizzbuzz.js: multiples of 15 should print FizzBuzz. Fix it and run npm test.",
  "setup": {
    "type": "fixture",
    "fixture": "fizzbuzz-repo",
    "init_git": true
  },
  "runner": {
    "max_turns": 18,
    "timeout_seconds": 240,
    "permission_mode": "bypassPermissions"
  },
  "graders": [
    { "type": "tests_pass", "command": "npm test" },
    { "type": "file_contains", "path": "src/fizzbuzz.js", "pattern": "FizzBuzz" }
  ],
  "pass_threshold": 1.0
}

Adding a Descriptor Harness

Use a descriptor when a CLI emits Claude stream JSON, Codex JSONL, generic JSONL with stable field paths, or line-oriented text.

1. Create harnesses/<id>.harness.json.
2. Declare binary names and optional well-known paths.
3. Map output fields such as final text, usage, cost, duration, error state, and tool calls.
4. Run npm run run:eval -- --list-harnesses.
5. Start with one simple case and inspect the transcript before running larger suites.

Operating Notes

• Node 20 with npm 10 is the supported release toolchain. Node 22+ may work, but it is outside the release-tested range and native modules should be rebuilt after a runtime change.
• The app uses better-sqlite3, so native dependency installation must succeed for the local platform.
• macOS and Linux are the release-tested platforms. Windows may work when a compatible better-sqlite3 prebuild is available; otherwise install the platform's native Node build prerequisites before running npm ci.
• The optional tmux runner also requires bash and tmux; use the default headless runner when either is unavailable.
• rubric_llm uses a separate judge harness; it should not silently reuse the harness under test.
• Missing usage data is shown as missing, not as a real zero.
• Local path redaction is for display; raw local paths may still exist server-side in local-only transcripts.
• Public repository cleanup should be validated with bash scripts/public-upload-audit.sh --all before pushing.