diff --git a/CHANGELOG.md b/CHANGELOG.md index 296532d1..3e23ed3f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,67 @@ All notable changes to cap-evolve are documented here. The format follows longer looks like a capability regression. ### Added +- **Benchmark zoo + `cap-evolve benchmark add|verify|list` + a declarative manifest** + (#141). Onboarding a benchmark no longer means hand-writing a `CapabilityAdapter` + subclass and pruning a ~100-line `capevolve.yaml`. Comparing the two generic bundled + templates showed what actually repeats — module preamble, the dataset→`Task` loop, the + infra-error branch of `score`, the match helper, the `Score(...)` construction, and + nearly the whole spec — so `benchmark.yaml` declares that, and `target.py` keeps the + one thing that is genuinely per-benchmark: `run(task, ctx, *, seed=0)`. There is + deliberately no config language for `run()`. Measured on the same benchmark + (`toy_calc`): **78 → 36 hand-authored lines (−54%)**; `adapters/adapter.py` and + `capevolve.yaml` are generated (0 hand-authored). `scoring: custom` keeps a bespoke + predicate as code (see the new `json_extract` entry's per-field partial credit). + `benchmarks/` is the curated, verifier-gated library; both bundled entries are + zero-API and run end to end to a sealed test number. + `cap-evolve benchmark verify` **executes** the benchmark rather than parsing its + manifest: the real `cap-evolve check` gate, a dataset load through the real adapter, + the seeded split plus the honest-gate floor (`val >= MIN_VAL_TASKS` + a non-empty + sealed test split, so a 3-task dataset fails at verify rather than mid-run inside + `gate.decide`), and a **real zero-API smoke eval** — every val task through `live()` + → `run_target()` → `score()`, twice, comparing rollout fingerprints and rewards, + which is what catches a non-deterministic `run_target()` (`check` never runs the + target). `verified.json` records the measured reward, split sizes and dataset + SHA-256; `benchmark list` reads that stamp from disk, not the manifest's `verified:` + flag. A zoo entry keeps the manifest, scorer and dataset **inside** the project dir + so #142's tamper guard covers them by construction (a grader at the benchmark root + is declared-but-unprotected). `cap-evolve --help` now generates the subcommand + listing from `COMMANDS` + handler docstrings instead of a literal usage string that + goes stale. Docs: `docs/BENCHMARK_ZOO.md`, `benchmarks/README.md`. + + **Review follow-up (#233).** `verify` ran everything it claimed but *concluded* + nothing from the results — a review made it pass on a broken benchmark in 9 of 12 + attacks. It now draws conclusions. A seed capability that already scores 1.0 has **no + headroom** and is a hard failure (opt out per-benchmark with + `allow_saturated_baseline: true`), closing the `score()`-wired-to-1.0, + `run()`-returns-`task.target` and `run()`-reads-the-answer-key hacks; a + **degenerate-scorer probe** scores a synthetically correct rollout against a + deliberately wrong one and requires the rewards to differ, so a scorer that ignores + its input fails at any baseline. Splits are checked for **genuine disjointness and a + non-empty train, on the realized split** (`train == val == test` passed before — #99 + found the repo's own headline τ² number came from exactly that), and are built as a + real `Splits` rather than a throwaway class. Every path key (`tasks_file`, + `target_module`, `capability_path`, `split_ids_file`) must be a **plain relative path + whose resolved parent is inside the project dir** — an allowlist checked once in + `load_manifest`, after `target_module: ../../pwned.py` executed code outside the + project dir during `verify`. `protected_paths` is now **additive** (unioned with the + layout defaults and #197's globs, never substituted), the protected-paths step asserts + on what the **runtime guard actually resolves from the generated `capevolve.yaml`** + rather than on what the manifest claims (#189's wrong-artifact class), and an + **under-declaration sweep** flags any `.py` or answer-key-ish file under `project/` + outside `capability_path/` that the guard would not hash — previously a third author's + `helpers.py` / `scorer2.py` were silently unprotected and tampering went undetected. + `verified.json` now stamps the grader and manifest hashes alongside the dataset, and + `benchmark list` **re-checks every hash**, so a hand-written stamp and a stale one + both read `verified: false` with the reason in `stale_reason`. A `score()` without + `scoring: custom` is a hard error (it silently overrode the declared mode and made + `benchmark list` lie); content-duplicate task rows are refused; `--description` is + emitted as a quoted YAML scalar (a newline used to redefine manifest keys); + `tasks(split)` honours its argument instead of handing out the sealed test split; an + empty or uncompilable target is a dataset error; and `--refresh` **keeps a hand-edited + `adapters/adapter.py`** instead of clobbering it, so overriding one generated hook is + a supported edit rather than work the next manifest change deletes. + - **SWE-bench oracle mode + calibrated smoke selection.** The SWE-bench adapter gains `SWEBENCH_ORACLE=1`, which attaches the "Oracle" retrieval context (the file[s] the gold patch touches, from `princeton-nlp/SWE-bench_Lite_oracle`'s `text` field) to the diff --git a/benchmarks/README.md b/benchmarks/README.md new file mode 100644 index 00000000..1631a5bf --- /dev/null +++ b/benchmarks/README.md @@ -0,0 +1,76 @@ +# The cap-evolve benchmark zoo + +A curated, **verifier-gated** library of ready-to-run benchmarks. Every entry is +declared by a `benchmark.yaml` manifest plus one file of real code, and carries a +`verified.json` stamp recording what `cap-evolve benchmark verify` actually measured +— not a hand-committed "verified" flag. + +```bash +cap-evolve benchmark list # the zoo + each entry's verified status +cap-evolve benchmark add my_bench # scaffold a new one (manifest + one code file) +cap-evolve benchmark add my_bench --from-zoo toy_calc # start from an existing entry +cap-evolve benchmark verify my_bench # check gate + a REAL smoke eval, then stamp +cap-evolve run --spec my_bench/project/capevolve.yaml --project my_bench/project +``` + +## Entries + +| Benchmark | Measures | Scoring | Tasks | API cost | +|---|---|---|---|---| +| [`toy_calc/`](toy_calc/) | Arithmetic accuracy of a deterministic stand-in agent | `exact` | 8 | none | +| [`json_extract/`](json_extract/) | Structured-JSON extraction with per-field partial credit | `custom` | 12 | none | + +Both are zero-API and fully deterministic, so they run in CI and are the reference +examples for the manifest format. + +## Layout + +``` +/ + README.md + mock_script.json # optional: the deterministic edit the `mock` optimizer applies + verified.json # written by `verify` — the EVIDENCE (measured reward + hashes) + project/ # ← this IS the cap-evolve project dir + benchmark.yaml # the declarative manifest (you write this) + target.py # run(task, ctx, *, seed=0) [+ score()] (you write this) + tasks.jsonl # the dataset + seed_capability/ # the artifact the optimizer edits + adapters/adapter.py # GENERATED — a bare ManifestAdapter subclass + capevolve.yaml # GENERATED from the manifest +``` + +Everything the grader depends on lives **inside** `project/`. That is deliberate: +[#142](../docs/HONEST_EVAL.md)'s tamper guard can only hash paths under the project +dir, so a manifest or scorer parked at the benchmark root would be declared-but- +unprotected. This layout makes the guard cover them by construction. + +## What is declarative vs what is code + +Declarative in `benchmark.yaml`: dataset file + field mapping, scoring mode, metric +direction, capability path, split seed/ratios/pinned ids, trial count, protected +paths. + +Code in `target.py`: `run(task, ctx, *, seed=0)` — how the target agent runs. There +is no config language for it because running an agent is real logic, and a DSL that +reimplemented Python would be worse than the Python it replaced. Optionally +`score(task, rollout)` for a bespoke predicate (`scoring: custom`); the five built-in +modes (`exact`/`contains`/`regex`/`numeric`/`custom`) cover the common cases. + +## What `verify` executes + +Not a manifest parse. In order: + +1. manifest parse + field validation (an unknown key is a hard error); +2. dataset load **through the real adapter** (missing / duplicate-id / empty → fail); +3. `cap-evolve check` on the generated project (stubs, task stability, scorer + determinism, pure `materialize`); +4. seeded split + the honest-gate floor — `val >= MIN_VAL_TASKS` and a non-empty + sealed test split, so a 3-task dataset fails **here**, not mid-run inside + `gate.decide`; +5. a **real zero-API smoke eval**: every val task through `live()` → `run_target()` → + `score()`, **twice**, comparing rollout fingerprints and rewards — this is what + catches a non-deterministic `run_target()`, which `check` never runs; +6. protected-paths resolution: the grader, dataset and manifest must all be covered. + +Then `verified.json` records the measured val reward, the split sizes, the dataset +SHA-256 and the list of steps that ran. diff --git a/benchmarks/json_extract/README.md b/benchmarks/json_extract/README.md new file mode 100644 index 00000000..7ac74fd7 --- /dev/null +++ b/benchmarks/json_extract/README.md @@ -0,0 +1,17 @@ +# Benchmark: json_extract + +Structured-JSON extraction accuracy with **per-field partial credit**, on a +deterministic zero-API extractor. Zero cost, so it runs in CI. + +Declared in [`project/benchmark.yaml`](project/benchmark.yaml). The only code is +[`project/target.py`](project/target.py): `run(task, ctx, *, seed=0)` plus a +`score(task, rollout)` — this benchmark uses `scoring: custom` because partial +credit over parsed JSON is real logic, not something a config key should express. + +```bash +cap-evolve benchmark verify json_extract +cap-evolve run --spec json_extract/project/capevolve.yaml --project json_extract/project +``` + +The seed prompt asks for prose, so it scores 0. Adding `[JSON]` earns the name +field (1/3) and `[FIELDS]` earns all three (3/3) — a graded, non-binary signal. diff --git a/benchmarks/json_extract/mock_script.json b/benchmarks/json_extract/mock_script.json new file mode 100644 index 00000000..a9ceb7c0 --- /dev/null +++ b/benchmarks/json_extract/mock_script.json @@ -0,0 +1,5 @@ +{ + "edits": [ + {"file": "prompt.txt", "op": "ensure_contains", "text": "\n[JSON] Reply with a single JSON object and nothing else.\n[FIELDS] Include every field: name, city, year."} + ] +} diff --git a/benchmarks/json_extract/project/adapters/adapter.py b/benchmarks/json_extract/project/adapters/adapter.py new file mode 100644 index 00000000..8199d5a3 --- /dev/null +++ b/benchmarks/json_extract/project/adapters/adapter.py @@ -0,0 +1,7 @@ +from cap_evolve.zoo import ManifestAdapter + + +class Adapter(ManifestAdapter): + """json_extract — everything is declared in ../benchmark.yaml.""" + + manifest_path = __file__ diff --git a/benchmarks/json_extract/project/benchmark.yaml b/benchmarks/json_extract/project/benchmark.yaml new file mode 100644 index 00000000..0055e697 --- /dev/null +++ b/benchmarks/json_extract/project/benchmark.yaml @@ -0,0 +1,31 @@ +# cap-evolve benchmark manifest — the DECLARATIVE half of a benchmark. +# The only code is target.py (run + a custom score). Flat keys only. +name: json_extract +description: Structured-JSON extraction accuracy (per-field partial credit) for a deterministic zero-API extractor whose prompt is optimized. + +# --- dataset --------------------------------------------------------------- +tasks_file: tasks.jsonl +id_field: id +input_field: input +target_field: target + +# --- scoring --------------------------------------------------------------- +scoring: custom # per-field partial credit lives in target.py:score +metric_direction: higher + +# --- what is optimized ----------------------------------------------------- +capability_path: seed_capability +target_module: target.py + +# --- splits (seeded once; test is sealed) ---------------------------------- +split_seed: 0 +split_train: 0.5 +split_val: 0.25 +split_test: 0.25 +split_ids_file: "" +num_trials: 1 + +# --- protected paths (#142 tamper guard hashes exactly these) -------------- +protected_paths: [adapters, benchmark.yaml, target.py, tasks.jsonl] + +verified: false diff --git a/benchmarks/json_extract/project/capevolve.yaml b/benchmarks/json_extract/project/capevolve.yaml new file mode 100644 index 00000000..a21d6c41 --- /dev/null +++ b/benchmarks/json_extract/project/capevolve.yaml @@ -0,0 +1,25 @@ +# GENERATED from benchmark.yaml by `cap-evolve benchmark add` — edit the +# manifest and re-run `cap-evolve benchmark add --refresh`, not this file. +capabilities: [system-prompt] +capability_path: seed_capability +actions: [edit] +optimizer_skill: mock +optimizer_model: "" +algorithm_skill: hill-climb +algorithm_focus: all +dataset_source: adapter +split_seed: 0 +split_train: 0.5 +split_val: 0.25 +split_test: 0.25 +split_ids_file: "" +num_trials: 1 +metric_directions: [higher] +gate_mode: paired +gate_k_se: 1.0 +max_iterations: 5 +stall: 2 +store: copy +# Declared by the manifest; #142's tamper guard hashes exactly these at baseline +# and re-hashes them after every optimizer step. +protected_paths: [adapters, benchmark.yaml, target.py, tasks.jsonl, capevolve.yaml, *gold*.json, *gold*.jsonl, *gold*.yaml, *gold*.yml, *gold*.csv, *gold*.txt, **/*gold*.json, **/*gold*.jsonl, **/*gold*.yaml, **/*gold*.yml, **/*gold*.csv, **/*gold*.txt] diff --git a/benchmarks/json_extract/project/seed_capability/prompt.txt b/benchmarks/json_extract/project/seed_capability/prompt.txt new file mode 100644 index 00000000..3727ac46 --- /dev/null +++ b/benchmarks/json_extract/project/seed_capability/prompt.txt @@ -0,0 +1 @@ +You are a helpful assistant. Describe what the user tells you. diff --git a/benchmarks/json_extract/project/target.py b/benchmarks/json_extract/project/target.py new file mode 100644 index 00000000..9683ce91 --- /dev/null +++ b/benchmarks/json_extract/project/target.py @@ -0,0 +1,68 @@ +"""json_extract's runner + a CUSTOM scorer — the two things a manifest can't declare. + +A deterministic zero-API stand-in extractor. The candidate prompt controls its +behavior through two markers, so prompt edits provably move the score with no model +calls: + + ``[JSON]`` emit a JSON object instead of prose + ``[FIELDS]`` include all three fields (name/city/year) rather than just the name + +Scoring is ``custom`` because partial credit over parsed JSON fields is real logic +(a per-field comparison after a parse that can fail) — exactly the kind of thing a +config language should NOT try to express. Everything else is in ``benchmark.yaml``. +""" + +from __future__ import annotations + +import json +import re +from pathlib import Path + +from cap_evolve import Score + +_FIELDS = ("name", "city", "year") + + +def _extract(text: str) -> dict: + """The stand-in's (deterministic) understanding of one sentence.""" + m = re.match(r"(?P.+?) was born in (?P.+?) in (?P\d{4})", text) + if not m: + return {} + return {"name": m["name"], "city": m["city"], "year": int(m["year"])} + + +def run(task, ctx, *, seed: int = 0): + prompt = (Path(ctx) / "prompt.txt").read_text(encoding="utf-8") + facts = _extract(str(task.input)) + if "[JSON]" not in prompt: + return {"output": f"Sure! That sentence is about {facts.get('name', 'someone')}.", + "trace": "prose (prompt did not ask for JSON)"} + keep = _FIELDS if "[FIELDS]" in prompt else ("name",) + return {"output": json.dumps({k: facts[k] for k in keep if k in facts}, + sort_keys=True), + "trace": f"json fields={list(keep)}"} + + +def score(task, rollout) -> Score: + """Per-field partial credit over parsed JSON — the bespoke half, as code.""" + if rollout.error: + return Score(task_id=task.id, reward=0.0, trial_rewards=[0.0], + feedback=f"Rollout failed ({rollout.error}); infrastructure " + "noise, not a prompt defect.") + want = json.loads(str(task.target)) + try: + got = json.loads(str(rollout.output or "")) + if not isinstance(got, dict): + raise ValueError("top level is not an object") + except Exception as e: # noqa: BLE001 + return Score(task_id=task.id, reward=0.0, trial_rewards=[0.0], + feedback=f"output was not a JSON object ({e}); the prompt must " + "instruct the agent to reply with a single JSON object " + f"holding the {list(_FIELDS)} fields.") + hits = [k for k in _FIELDS if str(got.get(k, "")) == str(want[k])] + reward = len(hits) / len(_FIELDS) + missing = [k for k in _FIELDS if k not in hits] + fb = ("all fields correct" if not missing else + f"got {len(hits)}/{len(_FIELDS)} fields; wrong or absent: {missing}. The " + "prompt should name every required field explicitly — never hard-code values.") + return Score(task_id=task.id, reward=reward, feedback=fb, trial_rewards=[reward]) diff --git a/benchmarks/json_extract/project/tasks.jsonl b/benchmarks/json_extract/project/tasks.jsonl new file mode 100644 index 00000000..c3ac7e04 --- /dev/null +++ b/benchmarks/json_extract/project/tasks.jsonl @@ -0,0 +1,12 @@ +{"id": "x01", "input": "Ada Lovelace was born in London in 1907 and later worked on compilers.", "target": "{\"city\": \"London\", \"name\": \"Ada Lovelace\", \"year\": 1907}"} +{"id": "x02", "input": "Grace Hopper was born in Baltimore in 1914 and later worked on compilers.", "target": "{\"city\": \"Baltimore\", \"name\": \"Grace Hopper\", \"year\": 1914}"} +{"id": "x03", "input": "Alan Turing was born in Maida Vale in 1921 and later worked on compilers.", "target": "{\"city\": \"Maida Vale\", \"name\": \"Alan Turing\", \"year\": 1921}"} +{"id": "x04", "input": "Katherine Johnson was born in Hampton in 1928 and later worked on compilers.", "target": "{\"city\": \"Hampton\", \"name\": \"Katherine Johnson\", \"year\": 1928}"} +{"id": "x05", "input": "Barbara Liskov was born in New York in 1935 and later worked on compilers.", "target": "{\"city\": \"New York\", \"name\": \"Barbara Liskov\", \"year\": 1935}"} +{"id": "x06", "input": "Edsger Dijkstra was born in Rotterdam in 1942 and later worked on compilers.", "target": "{\"city\": \"Rotterdam\", \"name\": \"Edsger Dijkstra\", \"year\": 1942}"} +{"id": "x07", "input": "Margaret Hamilton was born in Paris in 1949 and later worked on compilers.", "target": "{\"city\": \"Paris\", \"name\": \"Margaret Hamilton\", \"year\": 1949}"} +{"id": "x08", "input": "Donald Knuth was born in Lansing in 1956 and later worked on compilers.", "target": "{\"city\": \"Lansing\", \"name\": \"Donald Knuth\", \"year\": 1956}"} +{"id": "x09", "input": "Radia Perlman was born in Portsmouth in 1903 and later worked on compilers.", "target": "{\"city\": \"Portsmouth\", \"name\": \"Radia Perlman\", \"year\": 1903}"} +{"id": "x10", "input": "Leslie Lamport was born in Brooklyn in 1910 and later worked on compilers.", "target": "{\"city\": \"Brooklyn\", \"name\": \"Leslie Lamport\", \"year\": 1910}"} +{"id": "x11", "input": "Frances Allen was born in Peru in 1917 and later worked on compilers.", "target": "{\"city\": \"Peru\", \"name\": \"Frances Allen\", \"year\": 1917}"} +{"id": "x12", "input": "Ken Thompson was born in New Orleans in 1924 and later worked on compilers.", "target": "{\"city\": \"New Orleans\", \"name\": \"Ken Thompson\", \"year\": 1924}"} diff --git a/benchmarks/json_extract/verified.json b/benchmarks/json_extract/verified.json new file mode 100644 index 00000000..2a8bae86 --- /dev/null +++ b/benchmarks/json_extract/verified.json @@ -0,0 +1,26 @@ +{ + "ok": true, + "at": "2026-07-30T14:08:15+00:00", + "cap_evolve": "0.1.0", + "val_reward": 0.0, + "n_tasks": 12, + "splits": { + "train": 6, + "val": 3, + "test": 3 + }, + "steps": [ + "manifest parsed + validated", + "dataset loaded through the adapter: 12 task(s)", + "cap-evolve check executed on the generated project", + "splits computed: {'train': 6, 'val': 3, 'test': 3}", + "REAL smoke eval: 3 val task(s) x 2 passes through live() -> run_target() -> score()", + "degenerate-scorer probe: 3 task(s) scored with a correct vs a deliberately-wrong output", + "under-declaration sweep: every .py and answer-key-ish file under project/ (outside seed_capability/) is guard-hashed", + "protected paths resolved from the generated spec: ['adapters/adapter.py', 'benchmark.yaml', 'capevolve.yaml', 'target.py', 'tasks.jsonl']" + ], + "problems": [], + "dataset_sha256": "8e7d164f1d3f7440d86a604881006d3112cf055f6b7b3851a0408df5a29039a6", + "target_sha256": "33744fc166d055fa017654fb8b7c0400b0e033b1a41d7f5376848599219d9fd7", + "manifest_sha256": "11d012d109abff8f58d06d3b110ccfb842fd474597cede8c70f8b7b7617ab853" +} diff --git a/benchmarks/toy_calc/README.md b/benchmarks/toy_calc/README.md new file mode 100644 index 00000000..a775742d --- /dev/null +++ b/benchmarks/toy_calc/README.md @@ -0,0 +1,21 @@ +# Benchmark: toy_calc + +Arithmetic accuracy of a deterministic zero-API stand-in agent whose system prompt +is optimized. The smallest end-to-end proof in the repo: no model calls, runs in +seconds, and the optimization provably moves the number. + +Declared in [`project/benchmark.yaml`](project/benchmark.yaml). The only code is +[`project/target.py`](project/target.py) — one function, `run(task, ctx, *, seed=0)`. +The stand-in computes correctly only when the candidate prompt contains `[CALC]`, +so adding it (which the `mock` optimizer does) raises val 0.0 → 1.0. + +```bash +cap-evolve benchmark verify toy_calc +CAPEVOLVE_MOCK_SCRIPT=$PWD/mock_script.json \ + cap-evolve run --spec toy_calc/project/capevolve.yaml --project toy_calc/project +# -> baseline_val 0.0 -> test_reward 1.0 (gate-accepted, test sealed) +``` + +`examples/toy_calc/` keeps the hand-written adapter form of the same benchmark — it +is the "before" side of the boilerplate measurement, and the reference for what a +custom `CapabilityAdapter` looks like when the manifest does not fit. diff --git a/benchmarks/toy_calc/mock_script.json b/benchmarks/toy_calc/mock_script.json new file mode 100644 index 00000000..b79ebb76 --- /dev/null +++ b/benchmarks/toy_calc/mock_script.json @@ -0,0 +1,5 @@ +{ + "edits": [ + {"file": "prompt.txt", "op": "ensure_contains", "text": "\n[CALC] Compute the arithmetic expression exactly and output ONLY the resulting number."} + ] +} diff --git a/benchmarks/toy_calc/project/adapters/adapter.py b/benchmarks/toy_calc/project/adapters/adapter.py new file mode 100644 index 00000000..aaf52ab2 --- /dev/null +++ b/benchmarks/toy_calc/project/adapters/adapter.py @@ -0,0 +1,7 @@ +from cap_evolve.zoo import ManifestAdapter + + +class Adapter(ManifestAdapter): + """toy_calc — everything is declared in ../benchmark.yaml.""" + + manifest_path = __file__ diff --git a/benchmarks/toy_calc/project/benchmark.yaml b/benchmarks/toy_calc/project/benchmark.yaml new file mode 100644 index 00000000..f68961ef --- /dev/null +++ b/benchmarks/toy_calc/project/benchmark.yaml @@ -0,0 +1,32 @@ +# cap-evolve benchmark manifest — the DECLARATIVE half of a benchmark. +# The only code is target.py's run(task, ctx, *, seed=0). Flat keys only, so the +# zero-dependency spec reader can parse it. +name: toy_calc +description: Arithmetic accuracy of a deterministic zero-API stand-in agent whose system prompt is optimized. + +# --- dataset --------------------------------------------------------------- +tasks_file: tasks.jsonl +id_field: id +input_field: input +target_field: target + +# --- scoring --------------------------------------------------------------- +scoring: exact # exact | contains | regex | numeric | custom +metric_direction: higher + +# --- what is optimized ----------------------------------------------------- +capability_path: seed_capability +target_module: target.py + +# --- splits (seeded once; test is sealed) ---------------------------------- +split_seed: 0 +split_train: 0.5 +split_val: 0.25 +split_test: 0.25 +split_ids_file: "" +num_trials: 1 # the stand-in is deterministic + +# --- protected paths (#142 tamper guard hashes exactly these) -------------- +protected_paths: [adapters, benchmark.yaml, target.py, tasks.jsonl] + +verified: false # `cap-evolve benchmark verify` records the evidence diff --git a/benchmarks/toy_calc/project/capevolve.yaml b/benchmarks/toy_calc/project/capevolve.yaml new file mode 100644 index 00000000..a21d6c41 --- /dev/null +++ b/benchmarks/toy_calc/project/capevolve.yaml @@ -0,0 +1,25 @@ +# GENERATED from benchmark.yaml by `cap-evolve benchmark add` — edit the +# manifest and re-run `cap-evolve benchmark add --refresh`, not this file. +capabilities: [system-prompt] +capability_path: seed_capability +actions: [edit] +optimizer_skill: mock +optimizer_model: "" +algorithm_skill: hill-climb +algorithm_focus: all +dataset_source: adapter +split_seed: 0 +split_train: 0.5 +split_val: 0.25 +split_test: 0.25 +split_ids_file: "" +num_trials: 1 +metric_directions: [higher] +gate_mode: paired +gate_k_se: 1.0 +max_iterations: 5 +stall: 2 +store: copy +# Declared by the manifest; #142's tamper guard hashes exactly these at baseline +# and re-hashes them after every optimizer step. +protected_paths: [adapters, benchmark.yaml, target.py, tasks.jsonl, capevolve.yaml, *gold*.json, *gold*.jsonl, *gold*.yaml, *gold*.yml, *gold*.csv, *gold*.txt, **/*gold*.json, **/*gold*.jsonl, **/*gold*.yaml, **/*gold*.yml, **/*gold*.csv, **/*gold*.txt] diff --git a/benchmarks/toy_calc/project/seed_capability/prompt.txt b/benchmarks/toy_calc/project/seed_capability/prompt.txt new file mode 100644 index 00000000..0f4d6a5c --- /dev/null +++ b/benchmarks/toy_calc/project/seed_capability/prompt.txt @@ -0,0 +1 @@ +You are a helpful assistant. Answer the user as best you can. diff --git a/benchmarks/toy_calc/project/target.py b/benchmarks/toy_calc/project/target.py new file mode 100644 index 00000000..47f9b98a --- /dev/null +++ b/benchmarks/toy_calc/project/target.py @@ -0,0 +1,35 @@ +"""toy_calc's target runner — the ONE piece that is code, not config. + +A deterministic zero-API stand-in agent. It computes the arithmetic correctly only +when the candidate prompt contains the ``[CALC]`` marker, so optimizing the prompt +provably raises the score with no model calls. Everything else about this benchmark +(dataset, scoring mode, splits, metric direction, protected paths) is declared in +``benchmark.yaml``. +""" + +from __future__ import annotations + +from pathlib import Path + +_ALLOWED = set("0123456789 +-*") + + +def _safe_eval(expr: str) -> int: + if not set(expr) <= _ALLOWED: # arithmetic only + raise ValueError(f"unsafe expr: {expr!r}") + return int(eval(expr, {"__builtins__": {}}, {})) # noqa: S307 (sandboxed) + + +def run(task, ctx, *, seed: int = 0): + """Run the stand-in agent. ``seed`` is accepted per contract but unused: exact.""" + prompt = (Path(ctx) / "prompt.txt").read_text(encoding="utf-8") + has_calc = "[CALC]" in prompt + if not has_calc: + # without the instruction the stand-in rambles and gets it wrong + return {"output": f"I think {task.input} is roughly some number.", + "trace": "prompt_had_calc=False"} + try: + out = str(_safe_eval(str(task.input))) + except Exception as e: # noqa: BLE001 + out = f"error: {e}" + return {"output": out, "trace": "prompt_had_calc=True"} diff --git a/benchmarks/toy_calc/project/tasks.jsonl b/benchmarks/toy_calc/project/tasks.jsonl new file mode 100644 index 00000000..64b56a9d --- /dev/null +++ b/benchmarks/toy_calc/project/tasks.jsonl @@ -0,0 +1,8 @@ +{"id": "a1", "input": "3 + 4", "target": "7"} +{"id": "a2", "input": "10 - 6", "target": "4"} +{"id": "a3", "input": "2 * 5", "target": "10"} +{"id": "a4", "input": "9 + 1", "target": "10"} +{"id": "a5", "input": "8 - 3", "target": "5"} +{"id": "a6", "input": "6 * 2", "target": "12"} +{"id": "a7", "input": "7 + 5", "target": "12"} +{"id": "a8", "input": "4 * 4", "target": "16"} diff --git a/benchmarks/toy_calc/verified.json b/benchmarks/toy_calc/verified.json new file mode 100644 index 00000000..90ff4301 --- /dev/null +++ b/benchmarks/toy_calc/verified.json @@ -0,0 +1,26 @@ +{ + "ok": true, + "at": "2026-07-30T14:08:15+00:00", + "cap_evolve": "0.1.0", + "val_reward": 0.0, + "n_tasks": 8, + "splits": { + "train": 4, + "val": 2, + "test": 2 + }, + "steps": [ + "manifest parsed + validated", + "dataset loaded through the adapter: 8 task(s)", + "cap-evolve check executed on the generated project", + "splits computed: {'train': 4, 'val': 2, 'test': 2}", + "REAL smoke eval: 2 val task(s) x 2 passes through live() -> run_target() -> score()", + "degenerate-scorer probe: 2 task(s) scored with a correct vs a deliberately-wrong output", + "under-declaration sweep: every .py and answer-key-ish file under project/ (outside seed_capability/) is guard-hashed", + "protected paths resolved from the generated spec: ['adapters/adapter.py', 'benchmark.yaml', 'capevolve.yaml', 'target.py', 'tasks.jsonl']" + ], + "problems": [], + "dataset_sha256": "2ba70d56bce14825146be518b5766b14bdabd9c43a8129db633be4f21afdc323", + "target_sha256": "25cfd9f142539b85c33e392299d919cc426ffe5cdcd395bdc9bc716de55f98f8", + "manifest_sha256": "87e751daa3278e2fd6e8b6833b6f84e51edf3d39710bb47d80e16dd7b4352d19" +} diff --git a/core/cap_evolve/__init__.py b/core/cap_evolve/__init__.py index f31644c3..0e939564 100644 --- a/core/cap_evolve/__init__.py +++ b/core/cap_evolve/__init__.py @@ -23,6 +23,7 @@ from .stats import aggregate, bootstrap_ci, combined_stderr, mean, pass_at_k, pass_k, stderr from .trials import run_trials_pool from .types import Candidate, Rollout, Score, Task +from .zoo import BenchmarkError, ManifestAdapter __version__ = "0.1.0" @@ -59,5 +60,7 @@ "Rollout", "Score", "Task", + "BenchmarkError", + "ManifestAdapter", "__version__", ] diff --git a/core/cap_evolve/cli.py b/core/cap_evolve/cli.py index c6bae3c1..2ef4fb6c 100644 --- a/core/cap_evolve/cli.py +++ b/core/cap_evolve/cli.py @@ -5,12 +5,11 @@ declares, threading the run dir between them. The honesty guarantees live in ``cap_evolve`` (splits/gate/seal); ``cap-evolve`` just orchestrates. -Subcommands: - cap-evolve version - cap-evolve splits --ids ... [--seed N] [--ratios a,b,c] - cap-evolve check [project_dir] - cap-evolve run --spec .capevolve/project/capevolve.yaml (sequences phase skills) - [--resume [--run-ts TS]] resume an interrupted run in place +The subcommand list is ``COMMANDS`` and nothing else — ``cap-evolve --help`` renders it +from there plus each handler's first docstring line, and each handler owns its own +``--help``. There is deliberately no second copy of the list here or in ``main()``: +five parallel branches adding subcommands all conflicted on that literal usage string, +and a stale copy makes the documented-CLI test pass vacuously. ``run`` is intentionally minimal in Phase 0 and grows as phase skills land; it already resolves the manifest and validates the spec so the wiring is testable. @@ -46,22 +45,87 @@ def _find_skills_dir() -> Path | None: def _cmd_version(argv): + """Print the installed cap-evolve version as JSON.""" print(json.dumps({"cap-evolve": __version__})) return 0 def _cmd_splits(argv): + """Compute the seeded train/val/test split for a set of task ids.""" from .__main__ import _cmd_splits as f return f(argv) def _cmd_check(argv): + """Verify a project's adapter is fully implemented and deterministic.""" project = Path(argv[0]) if argv else Path(".capevolve/project") rep = run_check(project) print(json.dumps(rep.to_dict(), indent=2)) return 0 if rep.ok else 1 +def _cmd_benchmark(argv): + """Manage the benchmark zoo: list | add | verify a declarative benchmark.""" + import argparse + + from . import zoo + + p = argparse.ArgumentParser( + prog="cap-evolve benchmark", description=_cmd_benchmark.__doc__, + epilog=("examples:\n" + " cap-evolve benchmark list\n" + " cap-evolve benchmark add my_bench --description 'what it measures'\n" + " cap-evolve benchmark add my_bench --from-zoo toy_calc\n" + " cap-evolve benchmark add my_bench --refresh # regen project from manifest\n" + " cap-evolve benchmark verify my_bench\n"), + formatter_class=argparse.RawDescriptionHelpFormatter) + sub = p.add_subparsers(dest="action", required=True) + sub.add_parser("list", help="list the zoo with each entry's verified status") + a = sub.add_parser("add", help="scaffold a draft benchmark (manifest + one code file)") + a.add_argument("name", help="benchmark name, or a path to create it at") + a.add_argument("--description", default="", help="one line: what it measures") + a.add_argument("--from-zoo", default="", help="copy an existing zoo benchmark") + a.add_argument("--tasks", type=int, default=8, help="placeholder task count (default 8)") + a.add_argument("--refresh", action="store_true", + help="regenerate the derived project files from an existing manifest") + v = sub.add_parser("verify", help="check gate + REAL smoke eval, then stamp verified.json") + v.add_argument("name", help="zoo name or path to the benchmark dir") + v.add_argument("--smoke-tasks", type=int, default=0, + help="cap the smoke eval at N val tasks (0 = the whole val split)") + v.add_argument("--no-stamp", action="store_true", help="do not write verified.json") + args = p.parse_args(argv) + + try: + if args.action == "list": + print(json.dumps({"zoo": str(zoo.zoo_dir()), "benchmarks": zoo.index()}, indent=2)) + return 0 + if args.action == "add": + if ".." in Path(args.name).parts: + raise zoo.BenchmarkError( + f"benchmark add {args.name!r}: a `..` component is refused. A " + "benchmark is created inside cwd or at an explicit absolute path; " + "`../../../x` silently scaffolding four levels up is always a typo.") + dest = Path(args.name) + if "/" not in args.name and not dest.exists() and not args.refresh: + dest = Path.cwd() / args.name + info = zoo.add(dest, name=Path(args.name).name, description=args.description, + from_zoo=args.from_zoo, n_tasks=args.tasks, refresh=args.refresh) + info["next"] = f"cap-evolve benchmark verify {info['dir']}" + print(json.dumps(info, indent=2)) + return 0 + bench = zoo.resolve(args.name) + rep = zoo.verify(bench, smoke_tasks=args.smoke_tasks) + out = rep.to_dict() + if not args.no_stamp: + out["stamp"] = str(zoo.stamp(bench, rep)) + print(json.dumps(out, indent=2)) + return 0 if rep.ok else 1 + except zoo.BenchmarkError as e: + # stdout stays exactly one JSON object, even on the error path. + print(json.dumps({"ok": False, "error": str(e)}, indent=2)) + return 1 + + # Old hill-climb skill names → (skill, focus). The three byte-identical clones are # now one ``hill-climb`` skill parameterized by ``--focus``. _ALGO_FOCUS_ALIASES = { @@ -95,6 +159,7 @@ def _resolve_skills(skills_dir: Path) -> dict: def _cmd_run(argv): + """Sequence a whole optimization run: baseline -> algorithm -> sealed test -> report.""" import argparse import subprocess from .specfile import read_yaml @@ -619,13 +684,23 @@ def _cmd_estimate(argv): "run": _cmd_run, "estimate": _cmd_estimate, "dashboard": _cmd_dashboard, + "benchmark": _cmd_benchmark, } def main(argv=None) -> int: argv = list(sys.argv[1:] if argv is None else argv) if not argv or argv[0] in ("-h", "--help"): - print("usage: cap-evolve {version|splits|check|run|estimate|dashboard} [args]", file=sys.stderr) + # GENERATED from COMMANDS + each handler's docstring. There is deliberately no + # second literal copy of the subcommand list: five parallel branches adding a + # subcommand all conflicted on that string, and a stale listing breaks the + # "every documented `cap-evolve ` exists" test (#203/#214). + print("usage: cap-evolve {" + "|".join(COMMANDS) + "} [args]\n", file=sys.stderr) + for name, fn in COMMANDS.items(): + doc = ((fn.__doc__ or "").strip().splitlines() or [""])[0] + print(f" {name:<10} {doc}", file=sys.stderr) + print("\nrun `cap-evolve --help` for a command's own options.", + file=sys.stderr) return 0 if argv else 2 fn = COMMANDS.get(argv[0]) if fn is None: diff --git a/core/cap_evolve/zoo.py b/core/cap_evolve/zoo.py new file mode 100644 index 00000000..fe2cb9c6 --- /dev/null +++ b/core/cap_evolve/zoo.py @@ -0,0 +1,1407 @@ +"""The benchmark zoo — a declarative manifest, a scaffolder, and a real verifier. + +Onboarding a benchmark used to mean hand-writing a whole ``CapabilityAdapter`` +subclass plus a ~100-line ``capevolve.yaml``. Comparing the two *generic* bundled +templates (``templates/adapters/jsonl_litellm`` vs ``huggingface_litellm``) shows +what actually repeats: the module preamble, the JSONL→``Task`` loop, the +error-rollout branch of ``score``, the match-mode helper, the ``Score(...)`` +construction, and nearly the entire spec file. What does NOT repeat is exactly one +thing — **how you run the target agent** — plus, occasionally, a bespoke match +predicate. + +So this module makes the repeating half declarative and leaves the other half as +code, deliberately: + + * ``benchmark.yaml`` declares dataset wiring, the match mode, split policy, + metric direction and protected paths. + * ``target.py`` defines ONE function, ``run(task, ctx, *, seed=0)``, returning + the agent's output. Optionally ``score(task, rollout)`` when + ``scoring: custom`` — a real predicate is real logic and a config language + that reimplemented it would be worse than the Python it replaced. + +``ManifestAdapter`` then IS the adapter: ``tasks()`` and ``score()`` come from the +manifest, ``run_target()`` delegates to ``target.run``. The project's +``adapters/adapter.py`` is a 3-line subclass, so the whole adapter contract is +satisfied without the user seeing it. + +``verify`` is not a manifest parser, and it does not stop at "it ran". It runs the +real ``cap-evolve check`` gate AND a real zero-API smoke through the adapter (every +val task, twice, comparing rollouts and rewards), then **draws conclusions from the +results**: a benchmark whose seed capability already scores 1.0 has no headroom and +FAILS, and a ``score()`` that returns the same reward for a deliberately wrong +output FAILS the degenerate-scorer probe. It also requires genuinely disjoint +splits with a non-empty train, containment of every declared path inside the +project dir, and that the paths the *runtime guard actually resolves* — not the +ones the manifest claims — cover the grader, dataset and declaration. Then it +stamps ``verified.json`` with the reward it measured plus hashes of the dataset, +grader and manifest, which ``index()`` re-checks so a forged or stale stamp reads +as unverified. + +Scope note on the determinism check: the two smoke passes run back-to-back in one +process, so it catches a sampler without a seed, not drift on a coarser clock (a +``run()`` keyed on ``int(time.time()) % 2`` looks identical inside one second). It +is a necessary condition, not a proof of reproducibility. + +Pure stdlib (json + hashlib + importlib), like the rest of ``core``. +""" + +from __future__ import annotations + +import hashlib +import importlib.util +import inspect +import json +import math +import re +from dataclasses import dataclass, field +from pathlib import Path + +from . import splits as _splits +from .adapter import IMPLEMENT_MARKER, CapabilityAdapter +from .specfile import read_yaml +from .splits import Splits, make_splits +from .types import Rollout, Score, Task + +MANIFEST_NAME = "benchmark.yaml" +STAMP_NAME = "verified.json" + +#: A zoo entry is ``/project/`` — the manifest, the target module, the dataset +#: and the generated ``adapters/adapter.py`` all live in ONE dir, which is also the +#: cap-evolve project dir. That is not cosmetic: #142's tamper guard can only hash +#: paths *under* the project dir, so a grader parked at the benchmark root would be +#: declared-but-unprotected (its own ``protected_paths_unmatched`` event). Keeping the +#: manifest + scorer + task data inside the project dir makes the guard cover them by +#: construction. ``/`` itself stays the run base, so run dirs land in +#: ``/run_/`` rather than polluting the zoo root. +PROJECT_SUBDIR = "project" + +#: Minimum val tasks for the acceptance gate to mean anything. #113 puts the same +#: floor inside ``gate.decide`` itself; read it from ``splits`` when that has landed +#: so the two can never disagree, and fall back to the same literal when it has not. +MIN_VAL_TASKS = getattr(_splits, "MIN_VAL_TASKS", 2) + +#: Manifest keys whose value is a path INSIDE the benchmark's project dir. Each one +#: is fed to ``root / value`` — and ``target_module`` is then *imported*, so an +#: unchecked value is arbitrary code execution during ``verify`` (a reviewer's +#: ``target_module: ../../pwned.py`` ran code outside the project dir and still +#: verified clean). These are validated by ``_contained`` at manifest-load time, +#: before anything is read or imported. +_PATH_FIELDS = ("tasks_file", "target_module", "capability_path", "split_ids_file") + +SCORING_MODES = ("exact", "contains", "regex", "numeric", "custom") + +#: Every field a manifest may declare, with its default. Anything else is a typo +#: and a hard error — a silently-ignored key in an honesty-critical config is how +#: "I declared it" and "it applied" drift apart. +_FIELDS: dict = { + "name": "", + "description": "", + "tasks_file": "tasks.jsonl", + "id_field": "id", + "input_field": "input", + "target_field": "target", + "scoring": "exact", + "metric_direction": "higher", + "capability_path": "seed_capability", + "target_module": "target.py", + "split_seed": 0, + "split_train": 0.5, + "split_val": 0.25, + "split_test": 0.25, + "split_ids_file": "", + "protected_paths": [], + "num_trials": 1, + "verified": False, + #: Loud, explicit opt-out of the headroom requirement, for a reference benchmark + #: that is genuinely saturated at baseline (a regression fixture, not an + #: optimization target). Named for what it costs, not for what it disables: a + #: benchmark with no headroom cannot demonstrate improvement, which is the one + #: thing `baseline` exists to confirm, so it must be a deliberate declaration. + "allow_saturated_baseline": False, +} + + +class BenchmarkError(RuntimeError): + """A benchmark manifest is invalid, or its benchmark does not verify.""" + + +def _contained(root: Path, value: str, key: str) -> Path: + """``root / value``, proven to stay inside ``root``. An ALLOWLIST, not a denylist. + + Two conditions, both required: the declared value must be a *plain relative + path* (no absolute path, no drive, no ``..`` component, no leading ``~``), and + the resolved parent must still be inside the resolved ``root``. The second + condition is what a string check cannot give you — it catches a symlinked + subdirectory pointing out of the tree, which every ``..``-denylist in this repo + has missed (six times in this batch). + + Same shape as the guard PR #210 added at ``gepa.py``: resolve, then compare + parents. Denylisting substrings is not attempted, deliberately. + """ + raw = str(value) + p = Path(raw) + if p.is_absolute() or p.drive or raw.startswith("~") or ".." in p.parts: + raise BenchmarkError( + f"{key}={raw!r} must be a plain relative path inside the benchmark's " + f"project dir ({root}). Absolute paths, `~` and `..` are refused: " + f"{key} is read (and for target_module, IMPORTED) by verify, so a value " + "escaping the project dir would execute code and load data that #142's " + "tamper guard structurally cannot hash — the guard only covers paths " + "under the project dir.") + target = (root / p).resolve() + parent = target.parent if not target.is_dir() else target + try: + inside = parent.is_relative_to(root.resolve()) + except AttributeError: # pragma: no cover — py<3.9 + inside = str(parent).startswith(str(root.resolve())) + if not inside: + raise BenchmarkError( + f"{key}={raw!r} resolves to {target}, whose parent is OUTSIDE the " + f"benchmark's project dir ({root.resolve()}) — most likely through a " + "symlinked directory. Refused: see above.") + return root / p + + +# --------------------------------------------------------------------------- +# manifest +# --------------------------------------------------------------------------- + + +def find_manifest(start: Path) -> Path: + """The nearest ``benchmark.yaml`` at or above ``start``. + + Lets the scaffolded ``adapters/adapter.py`` be a bare subclass: the manifest + lives one dir up (the project dir), which is also where the dataset, the target + module and the seed capability live. + """ + p = Path(start).resolve() + for cand in (p, *p.parents) if p.is_dir() else (p.parent, *p.parents): + m = cand / MANIFEST_NAME + if m.is_file(): + return m + raise BenchmarkError( + f"no {MANIFEST_NAME} found at or above {start} — a manifest benchmark needs " + f"one. Run `cap-evolve benchmark add ` to scaffold it." + ) + + +def load_manifest(path: Path) -> dict: + """Read + validate a ``benchmark.yaml``. Raises ``BenchmarkError`` on anything off.""" + path = Path(path) + if path.is_dir(): + path = path / MANIFEST_NAME + if not path.is_file(): + raise BenchmarkError(f"missing manifest: {path}") + try: + raw = read_yaml(path.read_text(encoding="utf-8")) or {} + except Exception as e: # noqa: BLE001 + raise BenchmarkError(f"{path} did not parse as YAML: {e}") from e + if not isinstance(raw, dict): + raise BenchmarkError(f"{path} must be a YAML mapping, got {type(raw).__name__}") + + unknown = sorted(set(raw) - set(_FIELDS)) + if unknown: + raise BenchmarkError( + f"{path}: unknown manifest key(s) {unknown}. Known keys: " + f"{sorted(_FIELDS)}. A misspelled key would be silently ignored, so this " + "is a hard error." + ) + m = {**_FIELDS, **raw} + m["root"] = str(path.parent.resolve()) + if not str(m["name"]).strip(): + m["name"] = path.parent.name + # `name` is interpolated into a generated Python docstring and a module name, so a + # multi-line value would emit broken code rather than a broken config. + if "\n" in str(m["name"]) or "\r" in str(m["name"]): + raise BenchmarkError( + f"{path}: name must be a single line (it is interpolated into the " + "generated adapter shim's docstring and module name).") + if str(m["scoring"]).lower() not in SCORING_MODES: + raise BenchmarkError( + f"{path}: scoring={m['scoring']!r} is not one of {list(SCORING_MODES)}. " + "Use `custom` and define score(task, rollout) in the target module for a " + "bespoke predicate." + ) + m["scoring"] = str(m["scoring"]).lower() + if str(m["metric_direction"]).lower() not in ("higher", "lower"): + raise BenchmarkError( + f"{path}: metric_direction must be 'higher' or 'lower', got " + f"{m['metric_direction']!r}") + m["metric_direction"] = str(m["metric_direction"]).lower() + pp = m["protected_paths"] + if isinstance(pp, str): + pp = [pp] + if not isinstance(pp, (list, tuple)): + raise BenchmarkError( + f"{path}: protected_paths must be a YAML list (got " + f"{type(pp).__name__}). Write it as `protected_paths: [adapters, " + "tasks.jsonl]`.") + m["protected_paths"] = [str(x) for x in pp if str(x).strip()] + # Containment, at the ONE point every caller routes through. Doing it here rather + # than at each use site means `tasks()`, `_load_target_module`, `verify` and the + # generated spec all inherit it — there is no second path to `root / value`. + root = Path(m["root"]) + for key in _PATH_FIELDS: + if str(m[key]).strip(): + _contained(root, str(m[key]), key) + return m + + +def _load_target_module(manifest: dict): + """Import the manifest's target module (the one function that is real code).""" + mod_path = _contained(Path(manifest["root"]), str(manifest["target_module"]), + "target_module") + if not mod_path.is_file(): + raise BenchmarkError( + f"target_module {mod_path} does not exist — it must define " + "run(task, ctx, *, seed=0). Scaffold one with `cap-evolve benchmark add`.") + spec = importlib.util.spec_from_file_location( + f"capevolve_bench_{manifest['name']}_target", mod_path) + mod = importlib.util.module_from_spec(spec) + assert spec and spec.loader + spec.loader.exec_module(mod) # type: ignore[union-attr] + if not callable(getattr(mod, "run", None)): + raise BenchmarkError(f"{mod_path} must define a callable run(task, ctx, *, seed=0)") + # A `score()` that is not declared is a manifest that LIES about how the benchmark + # is graded — and `benchmark list` then reports the declared mode next to a + # verified badge for a benchmark scored by arbitrary code. Same reasoning as the + # unknown-key hard error, one level deeper: declaration must match behaviour. + if manifest["scoring"] != "custom" and callable(getattr(mod, "score", None)): + raise BenchmarkError( + f"{mod_path} defines score(task, rollout) but {MANIFEST_NAME} declares " + f"scoring: {manifest['scoring']!r}. A code scorer silently overriding the " + "declared mode makes the manifest — and `benchmark list` — report a " + "grading mode that is not the one in effect. Either set `scoring: custom` " + "to declare it, or delete score() and let the declared mode grade.") + return mod + + +# --------------------------------------------------------------------------- +# scoring (the declarative half) +# --------------------------------------------------------------------------- + + +def _num(s: str): + """The first number in ``s``, scientific notation included, or None.""" + m = re.search(r"-?\d+(?:\.\d+)?(?:[eE][-+]?\d+)?", str(s).replace(",", "")) + return float(m.group()) if m else None + + +def match(output: str, target: str, mode: str) -> bool: + """Does ``output`` satisfy ``target`` under ``mode``? (the built-in predicates) + + Exact semantics, because a scorer that surprises its author produces silent 0.0 + rows: + + * ``exact`` — case-insensitive equality of the stripped strings. + * ``contains`` — the stripped target appears anywhere in the output, + case-insensitively. An empty target is rejected in ``tasks()`` (it would match + everything). + * ``regex`` — ``re.search``, so the target is an UNANCHORED pattern: target ``7`` + credits ``17`` and ``0.7``. Anchor it yourself (``^7$``) when you mean exactly. + Every target is ``re.compile``-validated at dataset load. + * ``numeric`` — the FIRST number found in each side, compared with + ``math.isclose(rel_tol=1e-9, abs_tol=1e-12)``: relative tolerance so large + magnitudes are not spuriously unequal (a fixed ``abs(a-b) < 1e-9`` called + ``1e9`` vs ``1e9 + 2e-7`` different, which is one float step), and a small + absolute floor so values near zero still compare. ``_num`` accepts scientific + notation and strips thousands separators; ``"in 2024 the answer is 7"`` yields + ``2024``, not ``7``, so put the answer first or use ``regex``. + """ + out, tgt = (output or "").strip(), (target or "").strip() + if mode == "contains": + return tgt.lower() in out.lower() + if mode == "regex": + return re.search(tgt, out) is not None + if mode == "numeric": + a, b = _num(out), _num(tgt) + return (a is not None and b is not None + and math.isclose(a, b, rel_tol=1e-9, abs_tol=1e-12)) + return out.lower() == tgt.lower() # exact + + +# --------------------------------------------------------------------------- +# the adapter +# --------------------------------------------------------------------------- + + +class ManifestAdapter(CapabilityAdapter): + """A full ``CapabilityAdapter`` driven by ``benchmark.yaml`` + ``target.run``. + + Subclass it with an empty body in ``adapters/adapter.py``; the manifest is found + by walking up from that subclass's own file. + """ + + manifest_path: str | None = None # override to point elsewhere + + def __init__(self, manifest: Path | str | None = None): + start = manifest or self.manifest_path + if start is None: + try: + start = Path(inspect.getfile(type(self))) + except (TypeError, OSError): # defined in a REPL/exec — fall back to cwd + start = Path.cwd() + self.manifest = load_manifest(find_manifest(Path(start))) + self.root = Path(self.manifest["root"]) + self._target = _load_target_module(self.manifest) + self._tasks: list[Task] | None = None + + # --- declarative: where tasks come from -------------------------------- + + def tasks(self, split: str) -> list[Task]: + """The tasks for ``split`` — HONOURED, not ignored. + + It used to return the full list for every split, so ``tasks("test")`` handed + out the sealed test set to any caller who trusted the base contract. The + manifest already declares the seed, ratios and any pinned id file, so the same + partition ``verify`` and the run dir use is derivable here. ``harness`` still + asks for ``"all"`` and filters by the frozen ids, so nothing re-splits mid-run. + """ + allt = self._all_tasks() + if str(split).lower() in ("all", ""): + return allt + sp = self._splits([t.id for t in allt]) + ids = set(sp.ids(str(split).lower())) + return [t for t in allt if t.id in ids] + + def _splits(self, ids: list[str]) -> Splits: + """The manifest's declared partition: pinned ids when given, else seed+ratios.""" + m = self.manifest + if str(m["split_ids_file"]).strip(): + sf = _contained(self.root, str(m["split_ids_file"]), "split_ids_file") + sd = json.loads(sf.read_text(encoding="utf-8")) + return Splits(train=[str(x) for x in sd.get("train", [])], + val=[str(x) for x in sd.get("val", [])], + test=[str(x) for x in sd.get("test", [])], + seed=int(m["split_seed"])) + return make_splits(ids, seed=int(m["split_seed"]), + ratios=(float(m["split_train"]), float(m["split_val"]), + float(m["split_test"]))) + + def _all_tasks(self) -> list[Task]: + if self._tasks is not None: + return list(self._tasks) + m = self.manifest + path = _contained(self.root, str(m["tasks_file"]), "tasks_file") + if not path.is_file(): + raise BenchmarkError( + f"dataset file missing: {path} (declared as tasks_file=" + f"{m['tasks_file']!r} in {self.root / MANIFEST_NAME}). Create it — one " + 'JSON object per line, e.g. {"id": "t1", "input": "...", ' + '"target": "..."} — or point tasks_file at the real dataset.') + out: list[Task] = [] + seen: set[str] = set() + by_content: dict = {} + for lineno, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1): + line = line.strip() + if not line: + continue + try: + d = json.loads(line) + except Exception as e: # noqa: BLE001 + raise BenchmarkError(f"{path}:{lineno} is not valid JSON: {e}") from e + tid = str(d.get(m["id_field"], "") or f"t{lineno}") + if tid in seen: + raise BenchmarkError( + f"{path}:{lineno} duplicate task id {tid!r} — ids key the splits, " + "so duplicates would silently drop tasks and leak across splits.") + seen.add(tid) + # CONTENT duplicates, not just id duplicates. Fresh ids on identical + # (input, target) rows split cleanly and pass every honesty check, but a + # val task that is byte-identical to a train task is memorization dressed + # as generalization — the id-level guard below could not see it, so the + # reviewer's duplicate-every-row attack verified clean. + key = (json.dumps(d.get(m["input_field"]), sort_keys=True, default=str), + json.dumps(d.get(m["target_field"]), sort_keys=True, default=str)) + if key in by_content: + raise BenchmarkError( + f"{path}:{lineno} task {tid!r} is a CONTENT duplicate of " + f"{by_content[key]!r}: identical input and target under a different " + "id. Distinct ids make it split cleanly, so the same task can land " + "in train and val — the gate then rewards memorization, and the " + "sealed test number measures recall of a seen row. De-duplicate the " + "dataset.") + by_content[key] = tid + tgt = d.get(m["target_field"]) + # An absent/empty target is a FREE POINT under `contains` (`"" in + # anything` is True) and a match-everything pattern under `regex`. A + # dataset row missing its answer must be an error, not a gift. + if m["scoring"] != "custom" and not str(tgt if tgt is not None else "").strip(): + raise BenchmarkError( + f"{path}:{lineno} has no {m['target_field']!r} value (got {tgt!r}), " + f"but scoring is {m['scoring']!r} — an empty target scores 1.0 for " + "free under `contains`/`regex`, silently inflating the benchmark. " + "Give the row a target, or use `scoring: custom` if the answer is " + "not a single field.") + if m["scoring"] == "regex": + try: + re.compile(str(tgt)) + except re.error as e: + raise BenchmarkError( + f"{path}:{lineno} target {str(tgt)!r} is not a valid regex " + f"({e}) — under `scoring: regex` the target IS the pattern. " + "Validated at load so it fails as a dataset error rather than " + "mid-eval inside score().") from e + out.append(Task(id=tid, input=d.get(m["input_field"]), + target=tgt, + metadata={k: v for k, v in d.items() + if k not in (m["id_field"], m["input_field"], + m["target_field"])})) + if not out: + raise BenchmarkError(f"{path} contains no tasks") + self._tasks = out + return list(out) + + # --- code: how the target runs ----------------------------------------- + + def run_target(self, task: Task, ctx, *, seed: int = 0) -> Rollout: + res = self._target.run(task, ctx, seed=seed) + if isinstance(res, Rollout): + return res + if isinstance(res, dict): + return Rollout(task_id=task.id, **{k: v for k, v in res.items() + if k != "task_id"}) + return Rollout(task_id=task.id, output=res, trace=res) + + # --- declarative (or custom code): scoring ------------------------------ + + def score(self, task: Task, rollout: Rollout) -> Score: + custom = getattr(self._target, "score", None) + if self.manifest["scoring"] == "custom": + if not callable(custom): + raise BenchmarkError( + f"{self.manifest['target_module']} declares scoring: custom but " + "defines no score(task, rollout).") + return custom(task, rollout) + # No `if callable(custom)` fallthrough: an undeclared score() is refused at + # module load (_load_target_module), so reaching here means the declared mode + # IS the effective mode. `benchmark list`'s `scoring` column is now honest. + if rollout.error: + return Score(task_id=task.id, reward=0.0, trial_rewards=[0.0], + feedback=f"Rollout failed ({rollout.error}); infrastructure " + "noise, not a capability defect — do not optimize " + "against it.") + mode = self.manifest["scoring"] + ok = match(str(rollout.output or ""), str(task.target), mode) + got = str(rollout.output or "").strip().replace("\n", " ")[:200] + fb = ("correct" if ok else + f"output did not satisfy the expected answer under '{mode}' scoring; " + f"the agent produced {got!r}. Guide it toward the required " + "answer format/content — never hard-code answers.") + return Score(task_id=task.id, reward=1.0 if ok else 0.0, feedback=fb, + trial_rewards=[1.0 if ok else 0.0]) + + # The seed capability is read straight out of ``ctx`` (the candidate dir) by + # ``target.run``, so making a candidate live needs no global side effect. + def apply(self, candidate_dir, edits: dict | None = None) -> None: + self.materialize(candidate_dir, edits) + + +# --------------------------------------------------------------------------- +# spec generation — the other half of the boilerplate +# --------------------------------------------------------------------------- + + +def spec_from_manifest(manifest: dict) -> str: + """Render the ``capevolve.yaml`` a manifest implies (so nobody authors one).""" + m = manifest + # The UNION, not the manifest's list: #197's protected_paths replaces its defaults + # wholesale, so emitting only the declared four silently switched off the answer-key + # globs. verify now asserts against this same generated file, which is what the + # runtime guard actually reads. + prot = effective_protected(m) + return f"""# GENERATED from {MANIFEST_NAME} by `cap-evolve benchmark add` — edit the +# manifest and re-run `cap-evolve benchmark add --refresh`, not this file. +capabilities: [system-prompt] +capability_path: {m['capability_path']} +actions: [edit] +optimizer_skill: mock +optimizer_model: "" +algorithm_skill: hill-climb +algorithm_focus: all +dataset_source: adapter +split_seed: {m['split_seed']} +split_train: {m['split_train']} +split_val: {m['split_val']} +split_test: {m['split_test']} +split_ids_file: "{m['split_ids_file']}" +num_trials: {m['num_trials']} +metric_directions: [{m['metric_direction']}] +gate_mode: paired +gate_k_se: 1.0 +max_iterations: 5 +stall: 2 +store: copy +# Declared by the manifest; #142's tamper guard hashes exactly these at baseline +# and re-hashes them after every optimizer step. +protected_paths: [{', '.join(prot)}] +""" + + +def default_protected(manifest: dict) -> list[str]: + """The paths a manifest benchmark must protect: grader + dataset + declaration.""" + out = ["adapters", MANIFEST_NAME, str(manifest["target_module"]), + str(manifest["tasks_file"])] + if str(manifest["split_ids_file"]).strip(): + out.append(str(manifest["split_ids_file"])) + return out + + +def effective_protected(manifest: dict) -> list[str]: + """What the generated spec must declare: the manifest's list UNIONED with the + layout defaults — #197's globs plus this benchmark's grader/dataset/declaration. + + ADDITIVE, deliberately. #197's ``protected_paths`` *replaces* its defaults + wholesale, so a manifest that declared its four known paths silently switched off + the ``*gold*`` answer-key globs — a reviewer added ``helpers.py``, ``scorer2.py`` + and ``answers_gold.json``, tampered with all three, and nothing noticed. Union is + the only default that fails safe: declaring one more path can never *un*protect + something. A benchmark that genuinely needs a default off edits the generated + spec, which is now a supported override (see ``add``). + """ + return list(dict.fromkeys([*manifest["protected_paths"], + *default_protected(manifest), + *_protect_default_globs()])) + + +def _protect_default_globs() -> tuple: + """#197's default globs when ``protect`` is importable, else its literal set. + + Duplicated-with-fallback rather than imported-hard so this module still works on a + pre-#142 checkout (the branch merges in either order). ``protect`` wins when + present, so the two can never drift once merged. + """ + try: + from . import protect + return tuple(protect._DEFAULT_GLOBS) + except (ImportError, AttributeError): # pragma: no cover — pre-#142 checkout + return ("adapters", "capevolve.yaml", + "*gold*.json", "*gold*.jsonl", "*gold*.yaml", "*gold*.yml", + "*gold*.csv", "*gold*.txt", + "**/*gold*.json", "**/*gold*.jsonl", "**/*gold*.yaml", + "**/*gold*.yml", "**/*gold*.csv", "**/*gold*.txt") + + +#: Data suffixes that could hold an answer key. Used by the under-declaration sweep, +#: which flags any *code* or *gold-ish data* file in the project dir that the runtime +#: guard would not hash — the four names ``verify`` used to hardcode covered only the +#: two committed examples, so anything a third author added was neither protected nor +#: flagged. +_GOLDISH_SUFFIXES = (".json", ".jsonl", ".yaml", ".yml", ".csv", ".tsv", ".txt") +_GOLDISH_HINTS = ("gold", "answer", "label", "solution", "truth", "key", "expected") + + +def resolve_declared(project_dir: Path) -> set: + """``{project-relative path}`` the tamper guard will hash, read from the GENERATED + ``capevolve.yaml`` — the artifact the runtime guard reads. + + Delegates to ``protect.resolve_protected`` when #142 has landed. The fallback is a + deliberately small re-implementation (dirs expand to their subtree, globs via + ``Path.glob``) so ``verify``'s protected-paths and under-declaration steps are not + silently skipped on a pre-#142 checkout — the two branches merge in either order, + and a guard that quietly does nothing is the exact failure this review found. + """ + pdir = Path(project_dir).resolve() + try: + from . import protect + return set(protect.resolve_protected(pdir)) + except ImportError: # pragma: no cover — pre-#142 checkout + pass + cfg = pdir / "capevolve.yaml" + declared: list = [] + if cfg.is_file(): + d = (read_yaml(cfg.read_text(encoding="utf-8")) or {}).get("protected_paths") + if isinstance(d, str): + d = [d] + declared = [str(x) for x in d if str(x).strip()] if isinstance(d, (list, tuple)) else [] + out: set = set() + + def _add(p: Path) -> None: + if not p.is_file() or "__pycache__" in p.parts: + return + try: + rel = p.relative_to(pdir) + except ValueError: + return + out.add(str(rel).replace("\\", "/")) + + for pat in declared: + direct = pdir / str(pat).lstrip("/") + if direct.is_dir(): + for c in direct.rglob("*"): + _add(c) + elif direct.is_file(): + _add(direct) + else: + for hit in pdir.glob(str(pat).lstrip("/")): + if hit.is_dir(): + for c in hit.rglob("*"): + _add(c) + else: + _add(hit) + return out + + +def under_declared(project_dir: Path, manifest: dict, protected: set) -> list[str]: + """Files inside the project dir that the runtime guard will NOT hash but should. + + Everything the optimizer must not rewrite is code or ground truth. So: every + ``.py`` in the project dir (a helper the grader imports is as much the grader as + ``adapter.py``), plus every data file whose name suggests an answer key. The seed + capability dir is excluded — it is the target, the one thing that MUST be + writable — and so are run dirs and caches. + + Returns project-relative paths, sorted. Detection, not silent protection: an + author who genuinely wants a file writable declares it, rather than finding out + from a tampered run. + """ + pdir = Path(project_dir).resolve() + cap = (pdir / str(manifest["capability_path"])).resolve() + out = [] + for p in sorted(pdir.rglob("*")): + if not p.is_file(): + continue + rel = p.relative_to(pdir) + parts = rel.parts + if any(x in (".git", "__pycache__", ".pytest_cache", ".mypy_cache", + ".ruff_cache") or x.startswith("run_") for x in parts): + continue + try: + if p.resolve().is_relative_to(cap): + continue # the seed capability IS the target + except (AttributeError, OSError, ValueError): # pragma: no cover + pass + name = rel.name.lower() + goldish = (p.suffix.lower() in _GOLDISH_SUFFIXES + and any(h in name for h in _GOLDISH_HINTS)) + if not (p.suffix == ".py" or goldish): + continue + if str(rel).replace("\\", "/") not in protected: + out.append(str(rel).replace("\\", "/")) + return out + + +# --------------------------------------------------------------------------- +# add (scaffold) +# --------------------------------------------------------------------------- + +_TARGET_STUB = '''"""The ONE piece of a benchmark that is real code, not config. + +``run(task, ctx, *, seed=0)`` runs the agent under test and returns its output. +``ctx`` is the live candidate dir — read the artifact being optimized out of it +(here: ``prompt.txt``). Return a str, or a dict of ``Rollout`` fields +(``output``/``trace``/``cost_usd``/``tokens``/``error``/``metadata``). + +Everything else — dataset wiring, splits, scoring mode, metric direction, +protected paths — is declared in ``{manifest}``. + +If your runner is STOCHASTIC you MUST forward ``seed`` to it, or pass^k and the +significance gate degenerate. Optionally define ``score(task, rollout) -> Score`` +for a bespoke predicate (set ``scoring: custom``). +""" + +from __future__ import annotations + +from pathlib import Path + + +def run(task, ctx, *, seed: int = 0): + # TODO replace this placeholder runner with a real one (an LLM call, a + # benchmark harness invocation, a subprocess). The placeholder echoes the + # task input when the candidate prompt asks it to, so the scaffold is a + # complete, runnable, verifiable benchmark from the first minute. + prompt = (Path(ctx) / "prompt.txt").read_text(encoding="utf-8") + if "[ECHO]" in prompt: + return str(task.input) + return f"I am not sure about {{task.input}}." +''' + +_PROMPT_STUB = ("You are a helpful assistant. Answer the user's question.\n") + +_README = """# Benchmark: {name} + +{description} + +Declared in [`project/{manifest}`](project/{manifest}); the only code is +[`project/{target}`](project/{target}) — one function, `run(task, ctx, *, seed=0)`. + +Everything under `project/` is the cap-evolve project dir, so #142's tamper guard +hashes the manifest, the scorer and the dataset by construction. + +```bash +cap-evolve benchmark verify {name} # check gate + real zero-API smoke eval +cap-evolve run --spec {name}/project/capevolve.yaml --project {name}/project +``` +""" + + +def add(dest: Path, *, name: str = "", description: str = "", from_zoo: str = "", + n_tasks: int = 8, refresh: bool = False) -> dict: + """Scaffold a benchmark at ``dest`` (or copy ``from_zoo``), wired end to end. + + Layout: ``dest/project/`` holds the manifest, the one-function target module, the + dataset, the seed capability AND the generated ``adapters/adapter.py`` + + ``capevolve.yaml``; ``dest/`` is the run base. Everything the grader depends on + is therefore inside the project dir, which is the only place #142's tamper guard + can hash. ``refresh`` regenerates just the derived files from an existing manifest. + """ + dest = Path(dest) + name = name or dest.name + proj = dest / PROJECT_SUBDIR + if from_zoo: + import shutil + src = resolve(from_zoo) + if dest.exists() and not refresh: + raise BenchmarkError(f"{dest} already exists") + shutil.copytree(src, dest, dirs_exist_ok=refresh) + elif not (proj / MANIFEST_NAME).exists(): + if proj.exists() and any(proj.iterdir()): + raise BenchmarkError(f"{proj} exists and is not empty") + proj.mkdir(parents=True, exist_ok=True) + (proj / MANIFEST_NAME).write_text(_manifest_text(name, description), + encoding="utf-8") + (proj / "target.py").write_text( + _TARGET_STUB.format(manifest=MANIFEST_NAME), encoding="utf-8") + (proj / "tasks.jsonl").write_text("".join( + json.dumps({"id": f"t{i}", "input": f"question {i}", + "target": f"question {i}"}) + "\n" + for i in range(1, n_tasks + 1)), encoding="utf-8") + cap = proj / "seed_capability" + cap.mkdir(exist_ok=True) + (cap / "prompt.txt").write_text(_PROMPT_STUB, encoding="utf-8") + (dest / "README.md").write_text(_README.format( + name=name, description=description or "(describe the benchmark here)", + manifest=MANIFEST_NAME, target="target.py"), encoding="utf-8") + elif not refresh: + raise BenchmarkError( + f"{proj / MANIFEST_NAME} already exists — pass --refresh to regenerate " + "the derived project files from it.") + + m = load_manifest(proj) + (proj / "adapters").mkdir(parents=True, exist_ok=True) + # The whole adapter: the manifest is found by walking up from this file. + shim = proj / "adapters" / "adapter.py" + generated = _shim_text(m) + kept = [] + # THE OVERRIDE PATH. `--refresh` used to overwrite this file unconditionally, so an + # author who overrode one generated hook (`trajectories()`, a custom `live()`) lost + # it silently the next time they edited the manifest — "hand-edit a file the tool + # overwrites" was the only answer to "I need to override one method". Now a shim + # whose bytes differ from the generated text is treated as AUTHORED and left alone; + # `capevolve.yaml` is still re-derived, since it holds no logic to override. The + # kept files are reported, so a stale hand-edited shim is visible, not silent. + if shim.is_file() and shim.read_text(encoding="utf-8") != generated: + kept.append("adapters/adapter.py") + else: + shim.write_text(generated, encoding="utf-8") + (proj / "capevolve.yaml").write_text(spec_from_manifest(m), encoding="utf-8") + info = {"name": m["name"], "dir": str(dest), "project": str(proj), + "manifest": str(proj / MANIFEST_NAME), "files": sorted( + str(p.relative_to(dest)) for p in dest.rglob("*") if p.is_file())} + if kept: + info["kept_hand_edited"] = kept + info["note"] = ( + f"left {kept} untouched: its content differs from the generated shim, so it " + "is treated as an authored override. Delete it and re-run --refresh to go " + "back to the generated version.") + return info + + +def _shim_text(manifest: dict) -> str: + """The generated ``adapters/adapter.py``. Compared byte-wise by ``--refresh`` to + tell an untouched shim from an authored override.""" + return ("from cap_evolve.zoo import ManifestAdapter\n\n\n" + "class Adapter(ManifestAdapter):\n" + f' """{manifest["name"]} — everything is declared in ' + f'../{MANIFEST_NAME}."""\n\n' + " manifest_path = __file__\n") + + +def _yaml_scalar(value: str) -> str: + """One YAML scalar, quoted so it cannot become extra keys. + + ``json.dumps`` emits a double-quoted string with ``\\n``/``"``/``\\`` escaped, and + YAML's double-quoted style is a superset of JSON string syntax — so a JSON string + IS a valid single-line YAML scalar. That is the whole fix for a ``--description`` + containing a newline, which previously redefined manifest keys below it. Free via + stdlib; a dumper dependency would buy nothing. + """ + return json.dumps(str(value)) + + +def _manifest_text(name: str, description: str) -> str: + return f"""# cap-evolve benchmark manifest — the DECLARATIVE half of a benchmark. +# The only code is target.py's run(task, ctx, *, seed=0). Flat keys only, so the +# zero-dependency spec reader can parse it. +name: {_yaml_scalar(name)} +description: {_yaml_scalar(description or "TODO one line: what capability this benchmark measures")} + +# --- dataset --------------------------------------------------------------- +tasks_file: tasks.jsonl # one JSON object per line +id_field: id +input_field: input +target_field: target + +# --- scoring --------------------------------------------------------------- +scoring: exact # exact | contains | regex | numeric | custom +metric_direction: higher # higher | lower + +# --- what is optimized ----------------------------------------------------- +capability_path: seed_capability +target_module: target.py + +# --- splits (seeded once; test is sealed) ---------------------------------- +split_seed: 0 +split_train: 0.5 +split_val: 0.25 +split_test: 0.25 +split_ids_file: "" # pin an official split instead of ratios +num_trials: 1 # raise if the runner is stochastic + +# --- protected paths (#142 tamper guard hashes exactly these) -------------- +protected_paths: [adapters, {MANIFEST_NAME}, target.py, tasks.jsonl] + +verified: false # `cap-evolve benchmark verify` flips this +""" + + +# --------------------------------------------------------------------------- +# the zoo index +# --------------------------------------------------------------------------- + + +def zoo_dir() -> Path: + """The bundled ``benchmarks/`` library (env override for a private zoo).""" + import os + env = os.environ.get("CAPEVOLVE_BENCHMARKS_DIR") + if env and Path(env).is_dir(): + return Path(env) + here = Path(__file__).resolve() + for parent in here.parents: + d = parent / "benchmarks" + if d.is_dir(): + return d + return Path("benchmarks") + + +def index() -> list[dict]: + """Every benchmark in the zoo, with its verified status (read from DISK). + + The stamp is read from ``verified.json``, not from the manifest's ``verified:`` + flag — a committed flag is a claim, the stamp is evidence, and the stamp carries + the reward that was actually measured. + """ + out = [] + d = zoo_dir() + if not d.is_dir(): + return out + for mpath in sorted(d.glob(f"*/{PROJECT_SUBDIR}/{MANIFEST_NAME}")): + try: + m = load_manifest(mpath) + except BenchmarkError as e: + out.append({"name": mpath.parent.parent.name, "error": str(e)}) + continue + bench = mpath.parent.parent + # The stamp is only evidence if it is a real verify result AND still matches the + # files on disk. `stamp_state` re-hashes; a forged or stale stamp reads as + # unverified with the reason visible in `stale_reason`. + state = stamp_state(bench, m) + st = state.get("stamp") or {} + row = {"name": m["name"], "dir": str(bench), + "description": m["description"], + # The EFFECTIVE grading mode. `scoring: exact` with a code scorer is now + # refused at load, so the declared mode is the one in effect — but say so + # explicitly rather than making the reader infer it. + "scoring": m["scoring"], + "metric_direction": m["metric_direction"], + "verified": state["verified"], + "verified_at": st.get("at"), "smoke_val_reward": st.get("val_reward"), + "n_tasks": st.get("n_tasks")} + if not state["verified"] and state["why"]: + row["stale"] = bool(state["stale"]) + row["stale_reason"] = state["why"] + if bool(m["allow_saturated_baseline"]): + row["allow_saturated_baseline"] = True + out.append(row) + return out + + +def resolve(name_or_path: str) -> Path: + """A benchmark dir from a zoo name or a filesystem path.""" + for cand in (Path(name_or_path), zoo_dir() / name_or_path): + if (cand / PROJECT_SUBDIR / MANIFEST_NAME).is_file(): + return cand + if (cand / MANIFEST_NAME).is_file(): # given the project dir directly + return cand.parent + raise BenchmarkError( + f"no benchmark {name_or_path!r}: no {PROJECT_SUBDIR}/{MANIFEST_NAME} under " + f"{Path(name_or_path).resolve()} or {zoo_dir() / name_or_path}. Zoo entries: " + f"{[b['name'] for b in index()]}") + + +# --------------------------------------------------------------------------- +# verify — the part that must actually verify +# --------------------------------------------------------------------------- + + +@dataclass +class VerifyReport: + name: str = "" + ok: bool = False + steps: list = field(default_factory=list) # what was EXECUTED, in order + problems: list = field(default_factory=list) + notes: list = field(default_factory=list) + val_reward: float | None = None + n_tasks: int | None = None + splits: dict = field(default_factory=dict) + protected: list = field(default_factory=list) + + def to_dict(self) -> dict: + return {"name": self.name, "ok": self.ok, "steps": self.steps, + "problems": self.problems, "notes": self.notes, + "val_reward": self.val_reward, "n_tasks": self.n_tasks, + "splits": self.splits, "protected": self.protected} + + +#: The deliberately-wrong output handed to ``score()`` by the degenerate-scorer probe. +#: Chosen to satisfy no plausible target under any built-in mode, and to be obvious in +#: a failure message. +_WRONG_SENTINEL = "__CAPEVOLVE_DELIBERATELY_WRONG__" + + +def _rollout_fingerprint(r: Rollout) -> str: + """Hash of everything about a rollout that a deterministic runner must repeat. + + ``cost_usd``/``tokens`` are excluded: a real metered runner reports them with + float jitter, and they are not the thing being verified. + """ + d = r.to_dict() + d.pop("cost_usd", None) + d.pop("tokens", None) + return hashlib.sha256(json.dumps(d, sort_keys=True, default=str) + .encode("utf-8")).hexdigest()[:16] + + +def verify(bench_dir: Path, *, smoke_tasks: int = 0) -> VerifyReport: + """Prove a benchmark works: check gate + REAL smoke eval + honest splits. + + Executes, in order: + 1. manifest parse + field validation; + 2. dataset load through the real adapter (missing/duplicate/empty → fail); + 3. ``cap_evolve.check.run_check`` on the generated project (stubs, task + stability, scorer determinism, materialize); + 4. seeded (or pinned) split + the honesty floor: ``MIN_VAL_TASKS`` val, a + non-empty sealed test, a non-empty train, and genuine disjointness — all + asserted on the REALIZED split, so ``train == val == test`` fails; + 5. a REAL zero-API smoke: every val task through ``live`` → ``run_target`` → + ``score``, TWICE, comparing rollout fingerprints and rewards — this is what + catches a non-deterministic ``run_target``, which ``check`` never runs; + 5a. HEADROOM: a seed capability that already scores 1.0 everywhere fails, + unless the manifest declares ``allow_saturated_baseline: true``; + 5b. the DEGENERATE-SCORER probe: a deliberately wrong output must score + differently from the real rollout on at least one task; + 6. protected paths, checked against what ``protect.resolve_protected`` actually + resolves from the generated spec (the artifact the runtime guard reads), + plus a sweep for any ``.py`` / answer-key-ish file the guard would miss. + + ``smoke_tasks`` caps step 5 (0 = the whole val split). + """ + rep = VerifyReport() + bench_dir = Path(bench_dir) + proj = bench_dir / PROJECT_SUBDIR + + # 1. manifest + try: + m = load_manifest(proj) + except BenchmarkError as e: + rep.problems.append(str(e)) + return rep + rep.name = m["name"] + rep.steps.append("manifest parsed + validated") + + if not (proj / "adapters" / "adapter.py").is_file(): + rep.problems.append( + f"no generated project at {proj} — run `cap-evolve benchmark add " + f"{bench_dir} --refresh` to regenerate the adapter shim + capevolve.yaml " + "from the manifest.") + return rep + + # 2. dataset, through the real adapter + from .check import load_adapter + try: + adapter = load_adapter(proj) + tasks = adapter.tasks("all") + except Exception as e: # noqa: BLE001 + rep.problems.append(f"dataset/adapter load failed: {e}") + return rep + rep.n_tasks = len(tasks) + rep.steps.append(f"dataset loaded through the adapter: {len(tasks)} task(s)") + + # 3. the real check gate + from .check import run_check + creport = run_check(proj) + rep.steps.append("cap-evolve check executed on the generated project") + if not creport.ok: + rep.problems.extend(f"cap-evolve check: {p}" for p in creport.problems) + rep.notes.extend(f"check: {n}" for n in creport.notes) + + # 4. honest splits — a benchmark must not be able to ship without them + if str(m["split_ids_file"]).strip(): + sf = _contained(proj, str(m["split_ids_file"]), "split_ids_file") + if not sf.is_file(): + rep.problems.append(f"split_ids_file {sf} does not exist") + return rep + sd = json.loads(sf.read_text(encoding="utf-8")) + # A real ``Splits``, not a throwaway ``type("S", (), ...)``: the dataclass is + # what every other honesty check in the repo consumes, so the seal semantics + # and the id types are the same ones ``harness.ensure_splits`` sees. + sp = Splits(train=[str(x) for x in sd.get("train", [])], + val=[str(x) for x in sd.get("val", [])], + test=[str(x) for x in sd.get("test", [])], + seed=int(m["split_seed"])) + else: + sp = make_splits([t.id for t in tasks], seed=int(m["split_seed"]), + ratios=(float(m["split_train"]), float(m["split_val"]), + float(m["split_test"]))) + rep.splits = {"train": len(sp.train), "val": len(sp.val), "test": len(sp.test)} + rep.steps.append(f"splits computed: {rep.splits}") + + # Disjointness + a non-empty train, asserted on the REALIZED split (not on the + # declared ratios): whatever produced `sp` — pinned ids or ratios — is what the + # run will use. #99 found the repo's own headline tau^2 number came from a + # train==val==test==50 run, so "the sealed number is a fit metric" is not a + # hypothetical failure mode; it already happened once here. + tr, va, te = set(sp.train), set(sp.val), set(sp.test) + leak = sorted(te & (tr | va)) + if leak: + rep.problems.append( + f"test split OVERLAPS train/val on {len(leak)} task id(s) (e.g. " + f"{leak[:5]}) — the 'sealed' number would be measured on data the " + "optimizer trained against, making it a fit metric, not a held-out " + "result. train/val/test must be disjoint.") + tv = sorted(tr & va) + if tv: + rep.problems.append( + f"train and val OVERLAP on {len(tv)} task id(s) (e.g. {tv[:5]}) — the " + "acceptance gate would score candidates on the very tasks reflection " + "read, so every accept is measuring memorization.") + if not tr: + rep.problems.append( + "train split is EMPTY — there is nothing for the optimizer to reflect " + f"over, so no candidate can be proposed from evidence. Raise split_train " + f"above {m['split_train']} in {proj / MANIFEST_NAME}" + + (" or add train ids to the split_ids_file." + if str(m["split_ids_file"]).strip() else ".")) + if len(sp.val) < MIN_VAL_TASKS: + rep.problems.append( + f"val split has {len(sp.val)} task(s), below the honest-gate minimum of " + f"{MIN_VAL_TASKS}: SE(Δ) would have {max(len(sp.val) - 1, 0)} degrees of " + f"freedom, so every accept/reject is meaningless (the gate itself refuses " + f"this mid-run). This benchmark has {len(tasks)} task(s) total — add more " + f"tasks, or raise split_val above {m['split_val']} in " + f"{proj / MANIFEST_NAME}.") + if not sp.test: + rep.problems.append( + "test split is EMPTY — there is no sealed held-out set, so this benchmark " + f"cannot produce an honest headline number. Add tasks or raise split_test " + f"above {m['split_test']}.") + if rep.problems: + return rep # a dishonest split makes the smoke number meaningless + + # 5. a REAL smoke eval through the adapter, twice. + val_ids = set(sp.val) + smoke = [t for t in tasks if t.id in val_ids] + if smoke_tasks: + smoke = smoke[:smoke_tasks] + cap = proj / str(m["capability_path"]) + if not cap.is_dir(): + rep.problems.append( + f"capability_path {cap} is not a directory — the optimizer needs a seed " + "artifact dir to edit.") + return rep + passes: list[dict] = [] + for attempt in (1, 2): + got: dict = {} + try: + with adapter.live(cap) as ctx: + for t in smoke: + r = adapter.run_target(t, ctx, seed=0) + s = adapter.score(t, r) + got[t.id] = (_rollout_fingerprint(r), round(float(s.reward), 9)) + except NotImplementedError as e: + rep.problems.append( + f"smoke eval pass {attempt} hit an unimplemented method ({e}) — " + f"implement it in {m['target_module']} (run/score) before verifying.") + return rep + except Exception as e: # noqa: BLE001 + rep.problems.append( + f"smoke eval pass {attempt} raised {type(e).__name__}: {e} — the " + "benchmark cannot be run, so it cannot be verified.") + return rep + passes.append(got) + rep.steps.append( + f"REAL smoke eval: {len(smoke)} val task(s) x 2 passes through live() -> " + "run_target() -> score()") + drift = sorted(k for k in passes[0] if passes[0][k] != passes[1][k]) + if drift: + rep.problems.append( + f"NON-DETERMINISTIC: {len(drift)} task(s) produced a different rollout or " + f"reward on an identical re-run with seed=0 — e.g. {drift[:3]}: " + f"{[passes[0][k] for k in drift[:3]]} vs {[passes[1][k] for k in drift[:3]]}. " + "A benchmark whose rollouts drift at a fixed seed cannot produce a " + f"reproducible number. Make {m['target_module']}'s run() a function of " + "(task, candidate, seed) only, and forward `seed` to any sampler.") + rewards = [v[1] for v in passes[0].values()] + rep.val_reward = round(sum(rewards) / len(rewards), 6) if rewards else None + rep.notes.append(f"smoke val reward (seed capability) = {rep.val_reward}") + + # 5a. HEADROOM. verify already had this signal and only *noted* it, so a score() + # hard-wired to 1.0 and a run() returning task.target both verified clean. A + # benchmark whose seed capability is already perfect cannot demonstrate an + # improvement — the one thing `baseline` exists to confirm — so it is a problem. + if rewards and min(rewards) >= 1.0: + if bool(m["allow_saturated_baseline"]): + rep.notes.append( + "SATURATED BASELINE ALLOWED: the seed capability scores 1.0 on every " + "smoke task and the manifest declares allow_saturated_baseline: true. " + "This benchmark has NO headroom and cannot show an improvement; it is " + "only usable as a regression fixture.") + else: + rep.problems.append( + f"NO HEADROOM: the seed capability already scores {rep.val_reward} on " + f"all {len(rewards)} smoke val task(s). A benchmark that is perfect at " + "baseline cannot demonstrate an improvement, so optimizing it is " + "meaningless — and this is the signature of the two commonest reward " + "hacks: a score() hard-wired to a constant, and a run() that returns " + "task.target (or reads the answer key off disk). Make the tasks harder, " + "weaken the seed capability, or — for a genuinely saturated reference " + "fixture — declare `allow_saturated_baseline: true` in " + f"{proj / MANIFEST_NAME}.") + + # 5b. DEGENERATE-SCORER PROBE. Hand `score()` a synthetically CORRECT rollout (the + # task's own target as the output — that is what "correct" means for every declared + # mode) and a deliberately WRONG one, and require the two rewards to differ on at + # least one task. + # + # Note the comparison is correct-vs-wrong, NOT wrong-vs-the-seed-rollout: the seed + # capability is usually already wrong, so comparing against it would score 0.0 both + # times on a perfectly good benchmark. Correct-vs-wrong is a property of the SCORER + # alone, so it holds at any baseline — which is what catches a + # `return Score(reward=1.0)` on a benchmark whose baseline is imperfect, a case the + # headroom rule (5a) cannot see. + if smoke: + probe: dict = {} + try: + for t in smoke: + good = Rollout(task_id=t.id, output=str(t.target), + trace="__CAPEVOLVE_PROBE_CORRECT__") + bad = Rollout(task_id=t.id, output=_WRONG_SENTINEL, + trace=_WRONG_SENTINEL) + probe[t.id] = (round(float(adapter.score(t, good).reward), 9), + round(float(adapter.score(t, bad).reward), 9)) + except Exception as e: # noqa: BLE001 + rep.problems.append( + f"degenerate-scorer probe raised {type(e).__name__}: {e} — score() must " + "handle any rollout it is handed (a real run produces wrong answers by " + "definition), so a scorer that crashes on one cannot grade a run.") + return rep + rep.steps.append( + f"degenerate-scorer probe: {len(smoke)} task(s) scored with a correct vs a " + "deliberately-wrong output") + discriminating = sorted(k for k, (g, b) in probe.items() if g != b) + if not discriminating: + rep.problems.append( + "SCORER DOES NOT DISCRIMINATE: a correct output and a deliberately " + f"wrong one ({_WRONG_SENTINEL!r}) received the SAME reward on every one " + f"of {len(smoke)} smoke task(s) — e.g. " + f"{ {k: {'correct': probe[k][0], 'wrong': probe[k][1]} for k in list(probe)[:3]} }" + ". score() is therefore not a function of the agent's output: it grades " + "a constant, so every number this benchmark produces is fiction and no " + f"optimizer can learn from it. Fix score() in {m['target_module']} (or " + f"the {m['scoring']!r} mode's target field).") + else: + rep.notes.append( + f"scorer discriminates on {len(discriminating)}/{len(smoke)} smoke " + "task(s) (a correct output outscores a deliberately wrong one)") + + # 6. protected paths — checked against the artifact the RUNTIME GUARD reads. + # + # This step used to read the manifest's `protected_paths` while #197's + # `resolve_protected` reads the GENERATED `capevolve.yaml`. Weakening only the + # generated spec left verify reporting OK with `rep.protected == + # ['adapters/adapter.py']`: the same wrong-artifact bug as #189 (a guard reading a + # committed manifest instead of disk). Now the manifest is only the *source*, and + # every assertion below is on what `protect` actually resolves. + try: + protected = resolve_declared(proj) + except Exception as e: # noqa: BLE001 — a guard that cannot resolve is a problem + rep.problems.append( + f"resolving the protected set for {proj} raised {type(e).__name__}: {e} — " + "the runtime tamper guard cannot determine what to hash, so it would refuse " + "to run. Fix the generated capevolve.yaml's protected_paths.") + return rep + rep.protected = sorted(protected) + + # Assert on the RESOLVED set: every file the grader depends on must be one the guard + # will hash. A string match against a declaration proves nothing. + must_files = [str(m["target_module"]), str(m["tasks_file"]), MANIFEST_NAME, + "adapters/adapter.py"] + if str(m["split_ids_file"]).strip(): + must_files.append(str(m["split_ids_file"])) + missing = sorted(x for x in must_files if x not in protected) + if missing: + rep.problems.append( + f"the runtime tamper guard will NOT hash {missing}. What it actually " + f"resolves from {proj / 'capevolve.yaml'} is {sorted(protected)} — so the " + "grader / dataset / manifest is optimizer-writable and a candidate could " + "'improve' by rewriting it. This is checked against the generated spec " + f"(what the guard reads), not {MANIFEST_NAME} (what it was derived from); " + f"if they have diverged, re-run `cap-evolve benchmark add {bench_dir} " + "--refresh`.") + + # 6a. UNDER-DECLARATION SWEEP. The four names above only ever covered the two + # committed examples: a third author's `helpers.py` / `scorer2.py` / + # `answers_gold.json` were neither protected nor flagged, and tampering with all + # three went undetected. Flag every Python module and answer-key-ish data file in + # the project dir that the guard would not hash. + stray = under_declared(proj, m, protected) + if stray: + rep.problems.append( + f"UNDER-DECLARED: {len(stray)} file(s) inside {proj} are code or answer-key " + f"data that the tamper guard will not hash: {stray}. A helper the grader " + "imports is as much the grader as adapter.py, and an answer key the " + "optimizer can rewrite is a free 1.0. Add them to protected_paths in " + f"{proj / MANIFEST_NAME} and re-run `cap-evolve benchmark add {bench_dir} " + f"--refresh` — or move them under {m['capability_path']}/ if they are " + "genuinely part of the artifact being optimized.") + else: + rep.steps.append( + "under-declaration sweep: every .py and answer-key-ish file under " + f"{PROJECT_SUBDIR}/ (outside {m['capability_path']}/) is guard-hashed") + if not rep.problems: + rep.steps.append( + f"protected paths resolved from the generated spec: {rep.protected}") + + rep.ok = not rep.problems + return rep + + +#: The files a stamp hashes. The dataset changing invalidates the evidence, but so does +#: the GRADER changing (``target.py``) and the DECLARATION changing (``benchmark.yaml``) +#: — a contributor who verifies then edits the scorer ships a badge certifying code +#: that no longer exists. Keys are stable so ``index()`` can re-check each one. +_STAMPED_KEYS = ("dataset_sha256", "target_sha256", "manifest_sha256") + + +def stamp_hashes(proj: Path, manifest: dict) -> dict: + """``{stamp key -> sha256}`` for the dataset, the grader and the declaration.""" + return { + "dataset_sha256": _file_sha( + _contained(Path(proj), str(manifest["tasks_file"]), "tasks_file")), + "target_sha256": _file_sha( + _contained(Path(proj), str(manifest["target_module"]), "target_module")), + "manifest_sha256": _file_sha(Path(proj) / MANIFEST_NAME), + } + + +def stamp(bench_dir: Path, rep: VerifyReport) -> Path: + """Persist the verification EVIDENCE (measured reward + hashes), not a claim.""" + bench_dir = Path(bench_dir) + import datetime + proj = bench_dir / PROJECT_SUBDIR + payload = { + "ok": rep.ok, + "at": datetime.datetime.now(datetime.timezone.utc).isoformat(timespec="seconds"), + "cap_evolve": __import__("cap_evolve").__version__, + "val_reward": rep.val_reward, "n_tasks": rep.n_tasks, "splits": rep.splits, + "steps": rep.steps, "problems": rep.problems, + **stamp_hashes(proj, load_manifest(proj)), + } + p = bench_dir / STAMP_NAME + p.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8") + return p + + +def stamp_state(bench_dir: Path, manifest: dict) -> dict: + """Is ``verified.json`` real, current evidence? ``{"verified", "stale", "why"}``. + + Three ways a badge is refused: + + * ``ok`` is false, or the fields verify actually writes are absent — a + hand-written ``{"ok": true}`` has no ``steps``, so it is not a verify result; + * a recorded hash does not match the file on disk — a stale stamp certifying a + dataset or grader that has since been edited; + * the stamp does not parse. + + Previously ``index()`` read only ``st["ok"]``, so "a committed flag is a claim, the + stamp is evidence" was false as implemented: the stamp was a differently-located + claim. The hashes were written and never compared. + """ + bench_dir = Path(bench_dir) + p = bench_dir / STAMP_NAME + if not p.is_file(): + return {"verified": False, "stale": False, "why": "no verified.json"} + try: + st = json.loads(p.read_text(encoding="utf-8")) + except Exception as e: # noqa: BLE001 + return {"verified": False, "stale": True, "why": f"unreadable verified.json: {e}", + "stamp": {}} + if not isinstance(st, dict): + return {"verified": False, "stale": True, + "why": "verified.json is not a JSON object", "stamp": {}} + if not st.get("ok"): + return {"verified": False, "stale": False, "why": "stamp records ok: false", + "stamp": st} + # Require the fields we rely on. A stamp without `steps` was not produced by + # `verify` — it was typed. This is the cheap half of un-forging the badge. + missing = [k for k in ("steps", "val_reward", "splits", "n_tasks", *_STAMPED_KEYS) + if k not in st] + if missing: + return {"verified": False, "stale": True, "stamp": st, + "why": f"verified.json is missing {missing} — `cap-evolve benchmark " + "verify` always writes these, so this stamp was hand-written, " + "not measured. Re-run verify."} + if not st.get("steps"): + return {"verified": False, "stale": True, "stamp": st, + "why": "verified.json records no executed steps, so nothing was " + "actually verified. Re-run verify."} + try: + actual = stamp_hashes(bench_dir / PROJECT_SUBDIR, manifest) + except BenchmarkError as e: + return {"verified": False, "stale": True, "stamp": st, + "why": f"cannot re-hash the stamped files: {e}"} + drifted = {k: {"stamped": st.get(k), "actual": actual[k]} + for k in _STAMPED_KEYS if st.get(k) != actual[k]} + if drifted: + return {"verified": False, "stale": True, "stamp": st, "drifted": drifted, + "why": f"STALE STAMP: {sorted(drifted)} changed since verification " + f"({ {k: (str(v['stamped'])[:12], str(v['actual'])[:12]) for k, v in drifted.items()} }). " + "The badge certifies a dataset/grader/manifest that no longer " + "exists on disk. Re-run `cap-evolve benchmark verify`."} + return {"verified": True, "stale": False, "why": "", "stamp": st} + + +def _file_sha(p: Path) -> str | None: + p = Path(p) + if not p.is_file(): + return None + return hashlib.sha256(p.read_bytes()).hexdigest() + + +def _selfcheck() -> None: + """ponytail self-check for the declarative predicates (``python -m cap_evolve.zoo``). + + A module, not a script: ``zoo.py`` uses relative imports, so ``python zoo.py`` + cannot work — this runs as ``python -m cap_evolve.zoo``. + """ + assert match("Paris", "paris", "exact") and not match("Paris, FR", "paris", "exact") + assert match("The capital is Paris.", "Paris", "contains") + assert match("answer: 42", r"\d+", "regex") and not match("none", r"\d+", "regex") + assert match("the answer is 1,024 units", "1024", "numeric") + assert not match("no number", "7", "numeric") + assert IMPLEMENT_MARKER # imported for the stub contract + print("zoo predicate self-check: OK") + + +if __name__ == "__main__": + _selfcheck() diff --git a/core/tests/test_benchmark_zoo.py b/core/tests/test_benchmark_zoo.py new file mode 100644 index 00000000..5566dff6 --- /dev/null +++ b/core/tests/test_benchmark_zoo.py @@ -0,0 +1,622 @@ +"""The benchmark zoo: declarative manifest, ``add`` scaffold, and a verifier that +actually verifies. + +The point of these tests is the last part. Every guard in this epic that measured +the wrong artifact passed its own test vacuously, so each breakage case below +*breaks a real benchmark* and asserts ``verify`` fails on it with an actionable +message — a stubbed ``score()``, a non-deterministic ``run_target()``, a dataset too +small for an honest gate, and a missing dataset file. +""" + +from __future__ import annotations + +import json +import shutil +import subprocess +import sys +from pathlib import Path + +import pytest + +REPO = Path(__file__).resolve().parents[2] +CORE = REPO / "core" +ZOO = REPO / "benchmarks" +sys.path.insert(0, str(CORE)) + +from cap_evolve import zoo # noqa: E402 + + +@pytest.fixture +def bench(tmp_path): + """A working copy of the bundled toy_calc zoo entry.""" + d = tmp_path / "toy_calc" + shutil.copytree(ZOO / "toy_calc", d) + shutil.rmtree(d / "__pycache__", ignore_errors=True) + shutil.rmtree(d / "project" / "__pycache__", ignore_errors=True) + (d / zoo.STAMP_NAME).unlink(missing_ok=True) + return d + + +# --- manifest --------------------------------------------------------------- + + +def test_manifest_rejects_unknown_key(tmp_path): + p = tmp_path / zoo.MANIFEST_NAME + p.write_text("name: x\ntasks_fil: tasks.jsonl\n", encoding="utf-8") + with pytest.raises(zoo.BenchmarkError) as e: + zoo.load_manifest(p) + assert "tasks_fil" in str(e.value), "a typo'd key must not be silently ignored" + + +def test_manifest_rejects_bad_scoring_and_direction(tmp_path): + p = tmp_path / zoo.MANIFEST_NAME + p.write_text("name: x\nscoring: fuzzy\n", encoding="utf-8") + with pytest.raises(zoo.BenchmarkError, match="fuzzy"): + zoo.load_manifest(p) + p.write_text("name: x\nmetric_direction: sideways\n", encoding="utf-8") + with pytest.raises(zoo.BenchmarkError, match="metric_direction"): + zoo.load_manifest(p) + + +def test_builtin_predicates(): + assert zoo.match("Paris", "paris", "exact") + assert not zoo.match("Paris, FR", "paris", "exact") + assert zoo.match("the capital is Paris.", "Paris", "contains") + assert zoo.match("answer: 42", r"\d+", "regex") + assert zoo.match("about 1,024 units", "1024", "numeric") + assert not zoo.match("no number here", "7", "numeric") + + +# --- add -------------------------------------------------------------------- + + +def test_add_scaffolds_a_benchmark_that_verifies(tmp_path): + info = zoo.add(tmp_path / "demo", description="does the agent echo") + files = set(info["files"]) + assert {"project/benchmark.yaml", "project/target.py", "project/tasks.jsonl", + "project/adapters/adapter.py", "project/capevolve.yaml", + "project/seed_capability/prompt.txt"} <= files + # The whole adapter is a bare subclass — that IS the boilerplate reduction. + shim = (tmp_path / "demo" / "project" / "adapters" / "adapter.py").read_text() + assert "ManifestAdapter" in shim + assert len([ln for ln in shim.splitlines() if ln.strip()]) <= 6 + rep = zoo.verify(tmp_path / "demo") + assert rep.ok, rep.problems + + +def test_add_refuses_to_clobber_and_refresh_regenerates(tmp_path): + zoo.add(tmp_path / "demo") + with pytest.raises(zoo.BenchmarkError, match="already exists"): + zoo.add(tmp_path / "demo") + spec = tmp_path / "demo" / "project" / "capevolve.yaml" + spec.write_text("clobbered\n", encoding="utf-8") + zoo.add(tmp_path / "demo", refresh=True) + assert "capability_path" in spec.read_text() + + +def test_generated_spec_declares_the_protected_paths(tmp_path): + zoo.add(tmp_path / "demo") + spec = (tmp_path / "demo" / "project" / "capevolve.yaml").read_text() + line = next(ln for ln in spec.splitlines() if ln.startswith("protected_paths:")) + for must in ("adapters", zoo.MANIFEST_NAME, "target.py", "tasks.jsonl"): + assert must in line, f"{must} must be protected — it is the grader/data" + + +# --- the zoo index ---------------------------------------------------------- + + +def test_zoo_index_reads_the_stamp_from_disk_not_the_manifest_flag(bench, monkeypatch): + monkeypatch.setenv("CAPEVOLVE_BENCHMARKS_DIR", str(bench.parent)) + mpath = bench / "project" / zoo.MANIFEST_NAME + # The manifest *claims* verified: false, but a stamp is the evidence. + assert "verified: false" in mpath.read_text() + entry = next(b for b in zoo.index() if b["name"] == "toy_calc") + assert entry["verified"] is False + rep = zoo.verify(bench) + assert rep.ok, rep.problems + zoo.stamp(bench, rep) + entry = next(b for b in zoo.index() if b["name"] == "toy_calc") + assert entry["verified"] is True and entry["n_tasks"] == 8 + # ... and a *claimed* verified: true with no stamp is still unverified. + (bench / zoo.STAMP_NAME).unlink() + mpath.write_text(mpath.read_text().replace("verified: false", "verified: true")) + assert next(b for b in zoo.index() if b["name"] == "toy_calc")["verified"] is False + + +def test_bundled_zoo_entry_verifies(): + rep = zoo.verify(ZOO / "toy_calc") + assert rep.ok, rep.problems + assert rep.n_tasks == 8 and rep.splits["test"] >= 1 + assert rep.val_reward == 0.0, "the seed prompt lacks [CALC]; headroom must exist" + + +# --- verify EXECUTES the benchmark ------------------------------------------ + + +def test_verify_runs_the_real_thing_not_just_the_manifest(bench): + rep = zoo.verify(bench) + joined = " | ".join(rep.steps) + assert "cap-evolve check executed" in joined + assert "REAL smoke eval" in joined and "run_target() -> score()" in joined + assert rep.val_reward is not None, "a parse-only verify could not produce a reward" + + +def test_verify_catches_a_stubbed_score(bench): + m = bench / "project" / zoo.MANIFEST_NAME + m.write_text(m.read_text().replace("scoring: exact", "scoring: custom")) + t = bench / "project" / "target.py" + t.write_text(t.read_text() + '\n\ndef score(task, rollout):\n' + ' raise NotImplementedError("IMPLEMENT ME: score(task, rollout)")\n') + rep = zoo.verify(bench) + assert not rep.ok + assert any("unimplemented adapter methods" in p and "score" in p + for p in rep.problems), rep.problems + + +def test_verify_catches_a_nondeterministic_run_target(bench): + t = bench / "project" / "target.py" + t.write_text(t.read_text().replace( + " prompt = (Path(ctx)", + " import random\n" + " return {'output': str(random.random()), 'trace': 'unseeded'}\n" + " prompt = (Path(ctx)")) + rep = zoo.verify(bench) + assert not rep.ok + p = next(p for p in rep.problems if "NON-DETERMINISTIC" in p) + assert "seed=0" in p and "forward `seed`" in p, p + + +def test_verify_refuses_a_tiny_dataset_before_the_gate_can_surprise_anyone(bench): + """A 3-task dataset must fail loudly at verify, not mid-run inside gate.decide.""" + ds = bench / "project" / "tasks.jsonl" + ds.write_text("\n".join(ds.read_text().splitlines()[:3]) + "\n") + rep = zoo.verify(bench) + assert not rep.ok + assert any(f"below the honest-gate minimum of {zoo.MIN_VAL_TASKS}" in p + for p in rep.problems), rep.problems + assert any("test split is EMPTY" in p for p in rep.problems), rep.problems + + +def test_verify_catches_a_missing_dataset_file(bench): + (bench / "project" / "tasks.jsonl").unlink() + rep = zoo.verify(bench) + assert not rep.ok + p = " ".join(rep.problems) + assert "dataset file missing" in p and "tasks.jsonl" in p, rep.problems + + +def test_verify_catches_duplicate_task_ids(bench): + """Duplicate ids silently drop tasks and leak across splits — a hard error.""" + ds = bench / "project" / "tasks.jsonl" + lines = ds.read_text().splitlines() + ds.write_text("\n".join(lines + [lines[0]]) + "\n") + rep = zoo.verify(bench) + assert not rep.ok and "duplicate task id" in " ".join(rep.problems) + + +def test_verify_checks_the_generated_spec_not_the_manifest(bench): + """B5: the guard reads capevolve.yaml, so verify must check capevolve.yaml. + + Weaken ONLY the generated spec and leave the manifest declaring all four paths. + The old check read the manifest and reported OK while ``rep.protected`` sitting + right next to it showed a single file — the same wrong-artifact bug as #189. + """ + cfg = bench / "project" / "capevolve.yaml" + cfg.write_text(cfg.read_text().replace( + cfg.read_text().split("protected_paths: ")[1].splitlines()[0], + "[adapters/adapter.py]")) + m = (bench / "project" / zoo.MANIFEST_NAME).read_text() + assert "target.py" in m, "the manifest must still declare it — that is the point" + rep = zoo.verify(bench) + assert not rep.ok, "verify read the manifest instead of the artifact the guard reads" + assert any("runtime tamper guard will NOT hash" in p or + "does not declare" in p for p in rep.problems), rep.problems + + +def test_protection_is_additive_not_replacing(bench): + """A manifest's protected_paths must UNION with #197's defaults, never replace them. + + #197's own ``protected_paths`` replaces its defaults wholesale, so a manifest that + declared its four known paths silently switched OFF the ``*gold*`` answer-key globs. + """ + m = zoo.load_manifest(bench / "project") + eff = zoo.effective_protected(m) + for want in ("adapters", zoo.MANIFEST_NAME, "target.py", "tasks.jsonl"): + assert want in eff, want + assert any("gold" in p for p in eff), \ + "declaring paths must not switch off the answer-key globs" + spec = (bench / "project" / "capevolve.yaml").read_text() + assert "*gold*.json" in spec, "the generated spec must carry the union" + + +def test_verify_flags_a_third_authors_new_files(bench): + """Under-declaration must be DETECTED, not just uncovered for four hardcoded names. + + A reviewer added helpers.py / scorer2.py / answers_gold.json to a zoo entry: all + three were silently unprotected and verify still said ok: true. + """ + proj = bench / "project" + (proj / "helpers.py").write_text("X = 1\n", encoding="utf-8") + (proj / "scorer2.py").write_text("def grade(t, r):\n return 1.0\n", encoding="utf-8") + (proj / "answers_gold.json").write_text('{"a1": "2"}\n', encoding="utf-8") + rep = zoo.verify(bench) + assert not rep.ok, "a third author's code + answer key must not verify silently" + stray = " ".join(rep.problems) + assert "UNDER-DECLARED" in stray, rep.problems + # the two modules are FLAGGED (they need declaring)... + for name in ("helpers.py", "scorer2.py"): + assert name in stray, f"{name} not flagged: {rep.problems}" + # ...and the answer key is already COVERED, because the union keeps #197's *gold* + # globs alive instead of letting the manifest's list replace them. + assert "answers_gold.json" in rep.protected, rep.protected + + # declaring them closes the finding: verify passes and all three are guard-hashed. + m = bench / "project" / zoo.MANIFEST_NAME + m.write_text(m.read_text().replace( + "protected_paths: [adapters, benchmark.yaml, target.py, tasks.jsonl]", + "protected_paths: [adapters, benchmark.yaml, target.py, tasks.jsonl, " + "helpers.py, scorer2.py]"), encoding="utf-8") + zoo.add(bench, refresh=True) + rep2 = zoo.verify(bench) + assert rep2.ok, rep2.problems + for name in ("helpers.py", "scorer2.py", "answers_gold.json"): + assert name in rep2.protected, (name, rep2.protected) + + +def test_verify_fails_a_saturated_baseline_and_allows_a_declared_opt_out(bench): + """B1: a benchmark that is already perfect has no headroom to optimize.""" + t = bench / "project" / "target.py" + t.write_text(t.read_text() + ''' + +def _cheat(task): + return str(task.target) +''', encoding="utf-8") + # make run() echo the gold answer + t.write_text(t.read_text().replace( + ' prompt = (Path(ctx) / "prompt.txt").read_text(encoding="utf-8")', + ' return {"output": str(task.target), "trace": "gold"}\n' + ' prompt = (Path(ctx) / "prompt.txt").read_text(encoding="utf-8")'), + encoding="utf-8") + rep = zoo.verify(bench) + assert not rep.ok and any("NO HEADROOM" in p for p in rep.problems), rep.problems + # ...and the loudly-declared opt-out for a genuinely saturated reference fixture + m = bench / "project" / zoo.MANIFEST_NAME + m.write_text(m.read_text() + "allow_saturated_baseline: true\n", encoding="utf-8") + rep2 = zoo.verify(bench) + assert not any("NO HEADROOM" in p for p in rep2.problems), rep2.problems + assert any("SATURATED BASELINE ALLOWED" in n for n in rep2.notes), rep2.notes + + +def test_verify_catches_a_constant_scorer(bench): + """B1: the degenerate-scorer probe — a score() that ignores its input.""" + t = bench / "project" / "target.py" + t.write_text(t.read_text() + ''' + +def score(task, rollout): + from cap_evolve.types import Score + return Score(task_id=task.id, reward=1.0, feedback="perfect", trial_rewards=[1.0]) +''', encoding="utf-8") + # declared, so B2's hard error is not what fires here + m = bench / "project" / zoo.MANIFEST_NAME + m.write_text(m.read_text().replace("scoring: exact", "scoring: custom"), + encoding="utf-8") + rep = zoo.verify(bench) + assert not rep.ok + assert any("DOES NOT DISCRIMINATE" in p or "NO HEADROOM" in p + for p in rep.problems), rep.problems + + +def test_undeclared_custom_scorer_is_a_hard_error(bench): + """B2: a score() that silently overrides the declared mode makes `list` lie.""" + t = bench / "project" / "target.py" + t.write_text(t.read_text() + ''' + +def score(task, rollout): + from cap_evolve.types import Score + return Score(task_id=task.id, reward=1.0, trial_rewards=[1.0]) +''', encoding="utf-8") + rep = zoo.verify(bench) + assert not rep.ok + assert any("scoring: custom" in p for p in rep.problems), rep.problems + + +@pytest.mark.parametrize("key,value", [ + ("target_module", "../../pwned.py"), + ("tasks_file", "../../evil.jsonl"), + ("capability_path", "../../elsewhere"), + ("split_ids_file", "/etc/passwd"), +]) +def test_path_fields_are_contained_in_the_project_dir(bench, key, value): + """B4: `target_module: ../../pwned.py` EXECUTED code outside the project dir.""" + m = bench / "project" / zoo.MANIFEST_NAME + txt = m.read_text() + import re as _re + txt = _re.sub(rf"(?m)^{key}:.*$", f"{key}: {value}", txt) + if f"{key}:" not in txt: + txt += f"\n{key}: {value}\n" + m.write_text(txt, encoding="utf-8") + with pytest.raises(zoo.BenchmarkError) as e: + zoo.load_manifest(bench / "project") + assert key in str(e.value) and "relative path" in str(e.value) + + +def test_target_module_escape_never_imports(bench, tmp_path): + """The containment guard must fire BEFORE the module is executed.""" + marker = tmp_path / "PWNED" + (tmp_path / "pwned.py").write_text( + f"from pathlib import Path\n" + f"Path({str(marker)!r}).write_text('x')\n" + "def run(task, ctx, *, seed=0):\n return 'x'\n", encoding="utf-8") + m = bench / "project" / zoo.MANIFEST_NAME + m.write_text(m.read_text().replace("target_module: target.py", + "target_module: ../../pwned.py"), + encoding="utf-8") + rep = zoo.verify(bench) + assert not rep.ok + assert not marker.exists(), "code outside the project dir was EXECUTED during verify" + + +def test_zero_holdout_and_empty_train_are_refused(bench): + """B6: train == val == test passed the honesty floor; #99's tau^2 number was one.""" + ids = [t.id for t in zoo.ManifestAdapter(bench / "project").tasks("all")] + sf = bench / "project" / "splits.json" + sf.write_text(json.dumps({"train": ids, "val": ids, "test": ids}), encoding="utf-8") + m = bench / "project" / zoo.MANIFEST_NAME + m.write_text(m.read_text().replace('split_ids_file: ""', + "split_ids_file: splits.json"), encoding="utf-8") + rep = zoo.verify(bench) + assert not rep.ok + assert any("OVERLAP" in p for p in rep.problems), rep.problems + + # and an empty train split, via ratios + m.write_text(m.read_text().replace("split_ids_file: splits.json", + 'split_ids_file: ""') + .replace("split_train: 0.5", "split_train: 0.0") + .replace("split_val: 0.25", "split_val: 0.9") + .replace("split_test: 0.25", "split_test: 0.1"), encoding="utf-8") + rep2 = zoo.verify(bench) + assert not rep2.ok + assert any("train split is EMPTY" in p for p in rep2.problems), rep2.problems + + +def test_content_duplicate_tasks_are_refused(bench): + """Fresh ids on identical rows split cleanly, so val became a copy of train.""" + d = bench / "project" / "tasks.jsonl" + rows = [json.loads(x) for x in d.read_text().splitlines() if x.strip()] + d.write_text("".join(json.dumps(r) + "\n" for r in + rows + [{**r, "id": r["id"] + "_dup"} for r in rows]), + encoding="utf-8") + rep = zoo.verify(bench) + assert not rep.ok + assert any("CONTENT duplicate" in p for p in rep.problems), rep.problems + + +def test_forged_and_stale_stamps_read_as_unverified(bench, monkeypatch, tmp_path): + """B3: dataset_sha256 was written and never compared.""" + zoo_root = tmp_path / "zoo" + zoo_root.mkdir() + shutil.copytree(bench, zoo_root / "toy_calc") + monkeypatch.setenv("CAPEVOLVE_BENCHMARKS_DIR", str(zoo_root)) + b = zoo_root / "toy_calc" + m = zoo.load_manifest(b / "project") + + # forged: a hand-written stamp with no evidence + (b / zoo.STAMP_NAME).write_text(json.dumps( + {"ok": True, "val_reward": 0.99, "n_tasks": 999}), encoding="utf-8") + assert zoo.stamp_state(b, m)["verified"] is False + assert zoo.index()[0]["verified"] is False + assert "hand-written" in zoo.index()[0]["stale_reason"] + + # real stamp, then edit the dataset + zoo.stamp(b, zoo.verify(b)) + assert zoo.stamp_state(b, m)["verified"] is True + d = b / "project" / "tasks.jsonl" + d.write_text(d.read_text() + json.dumps({"id": "zz", "input": "1+1", + "target": "2"}) + "\n", encoding="utf-8") + st = zoo.stamp_state(b, m) + assert st["verified"] is False and st["stale"] is True + assert "dataset_sha256" in st["why"] + + # a stamped grader change also invalidates it + zoo.stamp(b, zoo.verify(b)) + t = b / "project" / "target.py" + t.write_text(t.read_text() + "\n# touched\n", encoding="utf-8") + assert zoo.stamp_state(b, m)["verified"] is False + assert "target_sha256" in zoo.stamp_state(b, m)["why"] + + +def test_description_newline_cannot_redefine_manifest_keys(tmp_path): + """B7: --description was interpolated unquoted into YAML.""" + zoo.add(tmp_path / "b", name="b", + description="oops\nscoring: contains\ntasks_file: /etc/hosts") + m = zoo.load_manifest(tmp_path / "b" / "project") + assert m["scoring"] == "exact", m + assert m["tasks_file"] == "tasks.jsonl", m + assert "\n" in m["description"], "the description itself must survive intact" + + +def test_refresh_keeps_a_hand_edited_adapter_shim(bench): + """N2: --refresh clobbered an authored override with no warning.""" + shim = bench / "project" / "adapters" / "adapter.py" + override = shim.read_text() + "\n def trajectories(self, *a, **k):\n return []\n" + shim.write_text(override, encoding="utf-8") + info = zoo.add(bench, refresh=True) + assert shim.read_text() == override, "an authored override was clobbered" + assert info.get("kept_hand_edited") == ["adapters/adapter.py"], info + # an untouched shim is still re-derived + shim.write_text(zoo._shim_text(zoo.load_manifest(bench / "project")), encoding="utf-8") + assert "kept_hand_edited" not in zoo.add(bench, refresh=True) + + +def test_tasks_honours_the_split_argument(bench): + """N1: tasks("test") handed out the sealed test split to any caller.""" + a = zoo.ManifestAdapter(bench / "project") + allt = {t.id for t in a.tasks("all")} + tr = {t.id for t in a.tasks("train")} + va = {t.id for t in a.tasks("val")} + te = {t.id for t in a.tasks("test")} + assert tr | va | te == allt + assert not (te & (tr | va)), "tasks() must not leak test ids into train/val" + assert te != allt, 'tasks("test") returned the whole dataset' + + +def test_empty_target_is_not_a_free_point(tmp_path): + """N5: match("anything", "", "contains") is True — a missing answer scored 1.0.""" + zoo.add(tmp_path / "b", name="b") + proj = tmp_path / "b" / "project" + m = proj / zoo.MANIFEST_NAME + m.write_text(m.read_text().replace("scoring: exact", "scoring: contains"), + encoding="utf-8") + (proj / "tasks.jsonl").write_text( + json.dumps({"id": "t1", "input": "q", "target": ""}) + "\n", encoding="utf-8") + with pytest.raises(zoo.BenchmarkError, match="free"): + zoo.ManifestAdapter(proj).tasks("all") + + +def test_bad_regex_target_fails_at_load(tmp_path): + """N3: an invalid pattern surfaced as a smoke-eval crash, not a dataset error.""" + zoo.add(tmp_path / "b", name="b") + proj = tmp_path / "b" / "project" + m = proj / zoo.MANIFEST_NAME + m.write_text(m.read_text().replace("scoring: exact", "scoring: regex"), + encoding="utf-8") + (proj / "tasks.jsonl").write_text( + json.dumps({"id": "t1", "input": "q", "target": "([unclosed"}) + "\n", + encoding="utf-8") + with pytest.raises(zoo.BenchmarkError, match="valid regex"): + zoo.ManifestAdapter(proj).tasks("all") + + +def test_stamp_records_measured_evidence(bench): + rep = zoo.verify(bench) + p = zoo.stamp(bench, rep) + d = json.loads(p.read_text()) + assert d["ok"] is True and d["val_reward"] == 0.0 and d["n_tasks"] == 8 + assert d["dataset_sha256"] and d["steps"], "the stamp must carry the evidence" + + +# --- CLI -------------------------------------------------------------------- + + +def _cli(*args, cwd=None): + return subprocess.run([sys.executable, "-m", "cap_evolve.cli", *args], + capture_output=True, text=True, cwd=cwd, + env={"PATH": "/usr/bin:/bin", "PYTHONPATH": str(CORE)}) + + +def test_cli_help_lists_benchmark_from_the_generated_listing(): + """`--help` renders COMMANDS + docstrings — no second literal list to go stale.""" + from cap_evolve import cli + assert "benchmark" in cli.COMMANDS + r = _cli("--help") + assert "benchmark" in r.stderr + for name in cli.COMMANDS: + assert name in r.stderr, f"{name} missing from the generated listing" + src = (CORE / "cap_evolve" / "cli.py").read_text() + assert "version|splits|check|run|estimate|dashboard" not in src, \ + "the literal usage string is back — it will go stale" + + +def test_cli_benchmark_add_verify_roundtrip_stdout_is_one_json_object(tmp_path): + r = _cli("benchmark", "add", "demo", "--description", "d", cwd=tmp_path) + assert r.returncode == 0, r.stderr + json.loads(r.stdout) # exactly ONE object + r = _cli("benchmark", "verify", "demo", cwd=tmp_path) + assert r.returncode == 0, r.stdout + r.stderr + out = json.loads(r.stdout) + assert out["ok"] is True and out["val_reward"] is not None + assert (tmp_path / "demo" / zoo.STAMP_NAME).is_file() + + +def test_cli_benchmark_verify_exits_nonzero_on_a_broken_benchmark(tmp_path): + _cli("benchmark", "add", "demo", cwd=tmp_path) + (tmp_path / "demo" / "project" / "tasks.jsonl").unlink() + r = _cli("benchmark", "verify", "demo", "--no-stamp", cwd=tmp_path) + assert r.returncode == 1 + assert json.loads(r.stdout)["ok"] is False + + +def test_cli_benchmark_list_shows_the_bundled_zoo(tmp_path): + r = _cli("benchmark", "list", cwd=tmp_path) + assert r.returncode == 0, r.stderr + names = [b["name"] for b in json.loads(r.stdout)["benchmarks"]] + assert "toy_calc" in names + + +def test_cli_benchmark_error_path_still_prints_one_json_object(tmp_path): + r = _cli("benchmark", "verify", "nope", cwd=tmp_path) + assert r.returncode == 1 + assert json.loads(r.stdout)["ok"] is False # not a traceback + + +# --- end to end ------------------------------------------------------------- + + +def test_zoo_benchmark_runs_end_to_end_to_a_sealed_test_number(bench, tmp_path, monkeypatch): + """A zoo benchmark, driven by the manifest adapter, reaches a sealed test number.""" + from cap_evolve import Budget, RunDir, TestSealError, harness + from cap_evolve.check import load_adapter + + proj = bench / "project" + adapter = load_adapter(proj) + seed = tmp_path / "seed" + shutil.copytree(proj / "seed_capability", seed) + run_dir = RunDir.create(tmp_path / ".capevolve", ts="zoo", + budget=Budget(max_iterations=3, stall=2)) + harness.ensure_splits(adapter, run_dir, seed=0) + base = harness.baseline(adapter, seed, run_dir=run_dir) + assert base.reward == 0.0, "seed prompt lacks [CALC]" + + monkeypatch.setenv("CAPEVOLVE_MOCK_SCRIPT", str(bench / "mock_script.json")) + monkeypatch.setenv("CAPEVOLVE_CORE", str(CORE)) + optimizer = harness.optimizer_from_command([ + "python3", str(REPO / "skills" / "optimizers" / "run-optimizer" / "scripts" / "run.py"), + "--name", "mock", "--workdir", "{workdir}", "--prompt", "{prompt}"]) + step = harness.run_step(adapter, run_dir=run_dir, + parent_dir=run_dir.candidate_dir("seed"), + optimizer=optimizer, instructions="raise val pass rate", + current_val=base, + gate_kwargs={"mode": "significant", "k_se": 1.0}) + assert step["accepted"] is True + payload = harness.finalize(adapter, run_dir=run_dir, + best_dir=run_dir.candidate_dir(run_dir.best_id)) + assert payload["test"]["reward"] == 1.0 + with pytest.raises(TestSealError): + harness.finalize(adapter, run_dir=run_dir, + best_dir=run_dir.candidate_dir(run_dir.best_id)) + + +def test_custom_scoring_entry_gives_graded_partial_credit(tmp_path): + """json_extract proves `scoring: custom` works AND the signal is non-binary.""" + from cap_evolve.check import load_adapter + d = tmp_path / "json_extract" + shutil.copytree(ZOO / "json_extract", d) + shutil.rmtree(d / "project" / "__pycache__", ignore_errors=True) + rep = zoo.verify(d) + assert rep.ok, rep.problems + adapter = load_adapter(d / "project") + tasks = adapter.tasks("all") + cap = tmp_path / "cap" + cap.mkdir() + + def val(prompt: str) -> float: + (cap / "prompt.txt").write_text(prompt, encoding="utf-8") + rs = [adapter.score(t, adapter.run_target(t, cap, seed=0)).reward for t in tasks] + return sum(rs) / len(rs) + + assert val("prose please") == 0.0 + assert val("[JSON] reply as json") == pytest.approx(1 / 3) + assert val("[JSON] json\n[FIELDS] all fields") == 1.0 + + +def test_every_bundled_zoo_entry_verifies_and_is_stamped(): + """The zoo is verifier-GATED: every entry must verify from a clean checkout.""" + entries = zoo.index() + assert len(entries) >= 2, entries + for e in entries: + assert "error" not in e, e + rep = zoo.verify(Path(e["dir"])) + assert rep.ok, (e["name"], rep.problems) + assert (Path(e["dir"]) / zoo.STAMP_NAME).is_file(), \ + f"{e['name']} ships without a verified.json stamp" + assert e["verified"] is True, e diff --git a/docs/ADAPTER_CONTRACT.md b/docs/ADAPTER_CONTRACT.md index 70e15dc3..1518a4ca 100644 --- a/docs/ADAPTER_CONTRACT.md +++ b/docs/ADAPTER_CONTRACT.md @@ -5,6 +5,11 @@ agent-specific glue is confined to a small adapter you implement once, in `.capevolve/project/adapters/adapter.py`. It subclasses `CapabilityAdapter` (`core/cap_evolve/adapter.py`). +> Before writing an adapter by hand, check whether the declarative manifest covers +> you: `cap-evolve benchmark add ` needs one `run(task, ctx, *, seed=0)` +> function and a `benchmark.yaml` — see [`BENCHMARK_ZOO.md`](BENCHMARK_ZOO.md). +> `ManifestAdapter` implements this contract for you. + ## The three required methods These three are `@abstractmethod` — `cap-evolve check` refuses to run until all diff --git a/docs/BENCHMARK_ZOO.md b/docs/BENCHMARK_ZOO.md new file mode 100644 index 00000000..31bdfd38 --- /dev/null +++ b/docs/BENCHMARK_ZOO.md @@ -0,0 +1,226 @@ +# The benchmark zoo — declarative benchmarks + `cap-evolve benchmark` + +Onboarding a benchmark was a from-scratch, per-user effort: write a +`CapabilityAdapter` subclass, prune a ~100-line `capevolve.yaml`, and author a +bespoke `optimizer/INSTRUCTIONS.md`. This page documents the lighter path — a +declarative `benchmark.yaml`, one file of real code, and a `verify` command that +actually exercises the benchmark. + +`benchmarks/` is the curated library. `docs/ADAPTER_CONTRACT.md` still governs the +full-adapter path, which stays the escape hatch for anything the manifest cannot +express. + +## What actually repeats (the measurement) + +Diffing the two *generic* bundled templates (`templates/adapters/jsonl_litellm` vs +`huggingface_litellm`) — 78 changed lines out of 127, but the changes are almost +entirely the dataset-loading block: + +| Repeats verbatim across every benchmark | Genuinely per-benchmark | +|---|---| +| module preamble + `sys.path` juggling | **how the target agent runs** | +| the JSONL/dataset → `Task` loop | a bespoke match predicate (sometimes) | +| `if rollout.error:` infra-noise branch of `score` | | +| the `exact`/`contains`/`regex` match helper | | +| `Score(task_id=…, reward=…, feedback=…, trial_rewards=[…])` construction | | +| the whole `capevolve.yaml` except ~6 values | | + +So the manifest declares the left column and `target.py` keeps the right one. There +is deliberately **no config language for `run()`** — running an agent is real logic, +and a DSL that reimplemented Python would be worse than the Python it replaced. + +### Measured reduction (same benchmark, `toy_calc`, hand-authored non-blank lines) + +| | Before (hand-written adapter) | After (manifest) | +|---|---|---| +| Python | 43 (`adapter.py`: 3 methods + `apply`) | **18** (`target.py`: one `run()`) | +| YAML | 35 (`capevolve.yaml` from the template) | **18** (`benchmark.yaml`) | +| `adapters/adapter.py` | — | 0 — **generated** | +| `capevolve.yaml` | — | 0 — **generated** | +| **Total** | **78** | **36** (−54%) | + +For the documented generic LLM case (`jsonl_litellm`) the before number is 101 +(88 Python + 13 YAML) against the same 36. + +## Layout + +``` +benchmarks// + README.md + mock_script.json # optional: the deterministic edit the `mock` optimizer applies + verified.json # written by `verify` — the measured evidence + project/ # ← this IS the cap-evolve project dir + benchmark.yaml # you write this + target.py # you write this + tasks.jsonl + seed_capability/ + adapters/adapter.py # GENERATED (a bare ManifestAdapter subclass) + capevolve.yaml # GENERATED from the manifest +``` + +Everything the grader depends on lives **inside** `project/`, because #142's tamper +guard can only hash paths under the project dir. A manifest or scorer at the +benchmark root would be declared-but-unprotected (the guard logs +`protected_paths_unmatched` and moves on). This layout closes that by construction — +`resolve_protected` on a scaffolded benchmark returns +`['adapters/adapter.py', 'benchmark.yaml', 'target.py', 'tasks.jsonl']` — and the +containment allowlist on every path key makes the layout *enforced* rather than merely +conventional. Anything else you add under `project/` that is code or an answer key is +either covered by the unioned globs or flagged by `verify`'s under-declaration sweep. + +## The manifest + +```yaml +name: toy_calc +description: Arithmetic accuracy of a deterministic zero-API stand-in agent. + +tasks_file: tasks.jsonl # one JSON object per line +id_field: id +input_field: input +target_field: target + +scoring: exact # exact | contains | regex | numeric | custom +metric_direction: higher # higher | lower + +capability_path: seed_capability +target_module: target.py + +split_seed: 0 +split_train: 0.5 +split_val: 0.25 +split_test: 0.25 +split_ids_file: "" # pin an official split instead of ratios +num_trials: 1 # raise if the runner is stochastic + +protected_paths: [adapters, benchmark.yaml, target.py, tasks.jsonl] +verified: false +allow_saturated_baseline: false # opt out of the headroom requirement, loudly +``` + +An **unknown key is a hard error** — a silently-ignored key in an honesty-critical +config is how "I declared it" and "it applied" drift apart. For the same reason a +`score()` in the target module with anything but `scoring: custom` is *also* a hard +error: a code scorer silently overriding the declared mode made both the manifest and +`benchmark list` report a grading mode that was not the one in effect. + +Scoring semantics, spelled out because a surprising scorer produces silent `0.0` rows: +`exact` is case-insensitive equality of stripped strings; `contains` is +case-insensitive substring (an empty target is rejected at load — it would match +everything); `regex` is `re.search`, i.e. the target is an **unanchored pattern**, so +target `7` credits `17` — anchor it yourself (`^7$`) and note every target is +`re.compile`-validated at load; `numeric` compares the **first** number on each side +with `math.isclose(rel_tol=1e-9, abs_tol=1e-12)` (relative, so large magnitudes are +not spuriously unequal) and accepts scientific notation. + +## The code + +```python +def run(task, ctx, *, seed: int = 0): + """ctx is the live candidate dir. Return a str, or a dict of Rollout fields.""" + prompt = (Path(ctx) / "prompt.txt").read_text(encoding="utf-8") + ... + return {"output": answer, "trace": transcript, "cost_usd": spend} +``` + +If the runner is stochastic you **must** forward `seed`, or pass^k and the +significance gate degenerate. For a bespoke predicate set `scoring: custom` and add +`score(task, rollout) -> Score` in the same file — see `benchmarks/json_extract`, +which does per-field partial credit over parsed JSON. + +## Commands + +```bash +cap-evolve benchmark list # the zoo + each entry's verified status +cap-evolve benchmark add my_bench --description "what it measures" +cap-evolve benchmark add my_bench --from-zoo toy_calc +cap-evolve benchmark add my_bench --refresh # regenerate the derived project files +cap-evolve benchmark verify my_bench # check gate + REAL smoke eval, then stamp +cap-evolve run --spec my_bench/project/capevolve.yaml --project my_bench/project +``` + +`add` writes a benchmark that is **runnable and verifiable from the first minute**: +the placeholder `run()` echoes the task input when the candidate prompt contains +`[ECHO]`, so `verify` passes and `run` shows a real gate decision before you have +written a line. + +## What `verify` executes + +Every guard in this area that measured the wrong artifact passed its own test +vacuously, so `verify` deliberately runs the benchmark rather than parsing it: + +1. manifest parse + field validation; +2. dataset load **through the real adapter** — missing file, duplicate ids, + **content-duplicate rows** (identical input+target under fresh ids, which split + cleanly and made val a copy of train), an empty target under a matching mode, an + uncompilable `regex` target, or an empty dataset all fail here; +3. `cap_evolve.check.run_check` on the generated project (stubs, task stability, + scorer determinism, pure `materialize`); +4. seeded (or pinned) split + the **honesty floor**, asserted on the **realized** + split: `val >= MIN_VAL_TASKS`, a non-empty sealed test, a **non-empty train**, and + **genuine disjointness**. A 3-task dataset fails *here*, not mid-run inside + `gate.decide` (#113) — and so does `train == val == test`, which is how #99's + headline τ² number turned out to be a fit metric; +5. a **real zero-API smoke eval** — every val task through `live()` → + `run_target()` → `score()`, **twice**, comparing rollout fingerprints and + rewards. This is what catches a non-deterministic `run_target()`; `check` never + runs the target at all. *Scope:* both passes run back-to-back in one process, so + this catches an unseeded sampler, not drift on a coarser clock — a necessary + condition, not a proof of reproducibility; + - **5a. headroom** — a seed capability that already scores `1.0` on every smoke + task is a **hard failure**, not a note. A benchmark that is perfect at baseline + cannot demonstrate an improvement, which is the one thing `baseline` exists to + confirm, and it is the signature of the two commonest reward hacks (a `score()` + wired to a constant; a `run()` returning `task.target` or reading the answer key + off disk). A genuinely saturated *reference fixture* opts out loudly with + `allow_saturated_baseline: true`; + - **5b. the degenerate-scorer probe** — `score()` is handed a synthetically + **correct** rollout and a deliberately **wrong** one, and the two rewards must + differ on at least one task. Correct-vs-wrong is a property of the scorer alone, + so it holds at any baseline: a `score()` that ignores its input fails even on a + benchmark whose baseline is imperfect; +6. **protected paths, checked against the artifact the runtime guard reads.** The + assertion is on what `protect.resolve_protected()` resolves from the *generated* + `capevolve.yaml` — not on what `benchmark.yaml` claims. Weakening only the + generated spec used to leave `verify` reporting OK with + `rep.protected == ['adapters/adapter.py']`, the same wrong-artifact bug as #189; + - **6a. the under-declaration sweep** — every `.py` and answer-key-ish data file + under `project/` (outside `capability_path/`) must be guard-hashed. The four + hardcoded names only ever covered the two bundled examples, so anything a third + author added was neither protected nor flagged. + +Every path key (`tasks_file`, `target_module`, `capability_path`, `split_ids_file`) +must be a **plain relative path whose resolved parent is inside `project/`** — an +allowlist, checked once in `load_manifest`, so no use site can bypass it. Without it +`target_module: ../../pwned.py` executed code outside the project dir during +`verify`, from a location #142's guard structurally cannot hash. + +`protected_paths` is **additive**: the manifest's list is UNIONed with the layout +defaults *and* #197's globs, never substituted for them. #197's own `protected_paths` +replaces its defaults wholesale, so declaring four paths silently switched off the +`*gold*` answer-key globs. Union is the only default that fails safe — declaring one +more path can never *un*protect something. + +`verified.json` then records the measured val reward, split sizes, SHA-256 of the +**dataset, the grader and the manifest**, and the ordered list of steps that ran. +`cap-evolve benchmark list` reads that stamp from **disk** and **re-checks every +hash**: a hand-written stamp (no `steps`, no hashes) and a stale one (dataset or +`target.py` edited after verifying) both read `verified: false` with the reason in +`stale_reason`. Without that comparison the stamp was just a differently-located +claim. + +## Overriding a generated file + +`adapters/adapter.py` is generated, but `--refresh` no longer clobbers it. If its +bytes differ from the generated shim it is treated as an **authored override** and +left alone; `capevolve.yaml` is still re-derived, since it holds no logic to override. +`benchmark add --refresh` reports what it kept: + +```json +{"kept_hand_edited": ["adapters/adapter.py"], "note": "left [...] untouched: ..."} +``` + +So overriding one `CapabilityAdapter` hook (`trajectories()`, a custom `live()`) is a +supported edit rather than something the next manifest change silently deletes. + +Exit code is 1 on any problem, and stdout is exactly one JSON object on every path, +including errors. diff --git a/templates/adapters/README.md b/templates/adapters/README.md index cd046ccd..31f89880 100644 --- a/templates/adapters/README.md +++ b/templates/adapters/README.md @@ -7,6 +7,14 @@ write code when your task loading or scoring is genuinely custom. Copy a template into `.capevolve/project/adapters/`, drop `model_config.py` next to it, set credentials in a `.env`, and run `cap-evolve check && cap-evolve run`. +> **Simpler path first:** for the common case you may not need an adapter at all. +> `cap-evolve benchmark add ` scaffolds a **declarative** benchmark — a +> `benchmark.yaml` manifest plus one `run(task, ctx, *, seed=0)` function — and +> `cap-evolve benchmark verify ` proves it works. See +> [`docs/BENCHMARK_ZOO.md`](../../docs/BENCHMARK_ZOO.md) and the curated +> [`benchmarks/`](../../benchmarks/) library. Come back here when your task loading or +> scoring genuinely needs the full adapter. + ## Templates | Template | Best for | What it optimizes | Task source |