Benchmark zoo + cap-evolve benchmark add/verify + declarative manifest

CHANGELOG.md

@@ -21,6 +21,67 @@ All notable changes to cap-evolve are documented here. The format follows
   longer looks like a capability regression.
 
 ### Added
+- **Benchmark zoo + `cap-evolve benchmark add|verify|list` + a declarative manifest**
+  (#141). Onboarding a benchmark no longer means hand-writing a `CapabilityAdapter`
+  subclass and pruning a ~100-line `capevolve.yaml`. Comparing the two generic bundled
+  templates showed what actually repeats — module preamble, the dataset→`Task` loop, the
+  infra-error branch of `score`, the match helper, the `Score(...)` construction, and
+  nearly the whole spec — so `benchmark.yaml` declares that, and `target.py` keeps the
+  one thing that is genuinely per-benchmark: `run(task, ctx, *, seed=0)`. There is
+  deliberately no config language for `run()`. Measured on the same benchmark
+  (`toy_calc`): **78 → 36 hand-authored lines (−54%)**; `adapters/adapter.py` and
+  `capevolve.yaml` are generated (0 hand-authored). `scoring: custom` keeps a bespoke
+  predicate as code (see the new `json_extract` entry's per-field partial credit).
+  `benchmarks/` is the curated, verifier-gated library; both bundled entries are
+  zero-API and run end to end to a sealed test number.
+  `cap-evolve benchmark verify` **executes** the benchmark rather than parsing its
+  manifest: the real `cap-evolve check` gate, a dataset load through the real adapter,
+  the seeded split plus the honest-gate floor (`val >= MIN_VAL_TASKS` + a non-empty
+  sealed test split, so a 3-task dataset fails at verify rather than mid-run inside
+  `gate.decide`), and a **real zero-API smoke eval** — every val task through `live()`
+  → `run_target()` → `score()`, twice, comparing rollout fingerprints and rewards,
+  which is what catches a non-deterministic `run_target()` (`check` never runs the
+  target). `verified.json` records the measured reward, split sizes and dataset
+  SHA-256; `benchmark list` reads that stamp from disk, not the manifest's `verified:`
+  flag. A zoo entry keeps the manifest, scorer and dataset **inside** the project dir
+  so #142's tamper guard covers them by construction (a grader at the benchmark root
+  is declared-but-unprotected). `cap-evolve --help` now generates the subcommand
+  listing from `COMMANDS` + handler docstrings instead of a literal usage string that
+  goes stale. Docs: `docs/BENCHMARK_ZOO.md`, `benchmarks/README.md`.
+
+  **Review follow-up (#233).** `verify` ran everything it claimed but *concluded*
+  nothing from the results — a review made it pass on a broken benchmark in 9 of 12
+  attacks. It now draws conclusions. A seed capability that already scores 1.0 has **no
+  headroom** and is a hard failure (opt out per-benchmark with
+  `allow_saturated_baseline: true`), closing the `score()`-wired-to-1.0,
+  `run()`-returns-`task.target` and `run()`-reads-the-answer-key hacks; a
+  **degenerate-scorer probe** scores a synthetically correct rollout against a
+  deliberately wrong one and requires the rewards to differ, so a scorer that ignores
+  its input fails at any baseline. Splits are checked for **genuine disjointness and a
+  non-empty train, on the realized split** (`train == val == test` passed before — #99
+  found the repo's own headline τ² number came from exactly that), and are built as a
+  real `Splits` rather than a throwaway class. Every path key (`tasks_file`,
+  `target_module`, `capability_path`, `split_ids_file`) must be a **plain relative path
+  whose resolved parent is inside the project dir** — an allowlist checked once in
+  `load_manifest`, after `target_module: ../../pwned.py` executed code outside the
+  project dir during `verify`. `protected_paths` is now **additive** (unioned with the
+  layout defaults and #197's globs, never substituted), the protected-paths step asserts
+  on what the **runtime guard actually resolves from the generated `capevolve.yaml`**
+  rather than on what the manifest claims (#189's wrong-artifact class), and an
+  **under-declaration sweep** flags any `.py` or answer-key-ish file under `project/`
+  outside `capability_path/` that the guard would not hash — previously a third author's
+  `helpers.py` / `scorer2.py` were silently unprotected and tampering went undetected.
+  `verified.json` now stamps the grader and manifest hashes alongside the dataset, and
+  `benchmark list` **re-checks every hash**, so a hand-written stamp and a stale one
+  both read `verified: false` with the reason in `stale_reason`. A `score()` without
+  `scoring: custom` is a hard error (it silently overrode the declared mode and made
+  `benchmark list` lie); content-duplicate task rows are refused; `--description` is
+  emitted as a quoted YAML scalar (a newline used to redefine manifest keys);
+  `tasks(split)` honours its argument instead of handing out the sealed test split; an
+  empty or uncompilable target is a dataset error; and `--refresh` **keeps a hand-edited
+  `adapters/adapter.py`** instead of clobbering it, so overriding one generated hook is
+  a supported edit rather than work the next manifest change deletes.
+
 - **SWE-bench oracle mode + calibrated smoke selection.** The SWE-bench adapter gains
   `SWEBENCH_ORACLE=1`, which attaches the "Oracle" retrieval context (the file[s] the
   gold patch touches, from `princeton-nlp/SWE-bench_Lite_oracle`'s `text` field) to the

benchmarks/README.md

@@ -0,0 +1,76 @@
+# The cap-evolve benchmark zoo
+
+A curated, **verifier-gated** library of ready-to-run benchmarks. Every entry is
+declared by a `benchmark.yaml` manifest plus one file of real code, and carries a
+`verified.json` stamp recording what `cap-evolve benchmark verify` actually measured
+— not a hand-committed "verified" flag.
+
+```bash
+cap-evolve benchmark list                       # the zoo + each entry's verified status
+cap-evolve benchmark add my_bench               # scaffold a new one (manifest + one code file)
+cap-evolve benchmark add my_bench --from-zoo toy_calc    # start from an existing entry
+cap-evolve benchmark verify my_bench            # check gate + a REAL smoke eval, then stamp
+cap-evolve run --spec my_bench/project/capevolve.yaml --project my_bench/project
+```
+
+## Entries
+
+| Benchmark | Measures | Scoring | Tasks | API cost |
+|---|---|---|---|---|
+| [`toy_calc/`](toy_calc/) | Arithmetic accuracy of a deterministic stand-in agent | `exact` | 8 | none |
+| [`json_extract/`](json_extract/) | Structured-JSON extraction with per-field partial credit | `custom` | 12 | none |
+
+Both are zero-API and fully deterministic, so they run in CI and are the reference
+examples for the manifest format.
+
+## Layout
+
+```
+<benchmark>/
+  README.md
+  mock_script.json          # optional: the deterministic edit the `mock` optimizer applies
+  verified.json             # written by `verify` — the EVIDENCE (measured reward + hashes)
+  project/                  # ← this IS the cap-evolve project dir
+    benchmark.yaml          # the declarative manifest (you write this)
+    target.py               # run(task, ctx, *, seed=0) [+ score()] (you write this)
+    tasks.jsonl             # the dataset
+    seed_capability/        # the artifact the optimizer edits
+    adapters/adapter.py     # GENERATED — a bare ManifestAdapter subclass
+    capevolve.yaml          # GENERATED from the manifest
+```
+
+Everything the grader depends on lives **inside** `project/`. That is deliberate:
+[#142](../docs/HONEST_EVAL.md)'s tamper guard can only hash paths under the project
+dir, so a manifest or scorer parked at the benchmark root would be declared-but-
+unprotected. This layout makes the guard cover them by construction.
+
+## What is declarative vs what is code
+
+Declarative in `benchmark.yaml`: dataset file + field mapping, scoring mode, metric
+direction, capability path, split seed/ratios/pinned ids, trial count, protected
+paths.
+
+Code in `target.py`: `run(task, ctx, *, seed=0)` — how the target agent runs. There
+is no config language for it because running an agent is real logic, and a DSL that
+reimplemented Python would be worse than the Python it replaced. Optionally
+`score(task, rollout)` for a bespoke predicate (`scoring: custom`); the five built-in
+modes (`exact`/`contains`/`regex`/`numeric`/`custom`) cover the common cases.
+
+## What `verify` executes
+
+Not a manifest parse. In order:
+
+1. manifest parse + field validation (an unknown key is a hard error);
+2. dataset load **through the real adapter** (missing / duplicate-id / empty → fail);
+3. `cap-evolve check` on the generated project (stubs, task stability, scorer
+   determinism, pure `materialize`);
+4. seeded split + the honest-gate floor — `val >= MIN_VAL_TASKS` and a non-empty
+   sealed test split, so a 3-task dataset fails **here**, not mid-run inside
+   `gate.decide`;
+5. a **real zero-API smoke eval**: every val task through `live()` → `run_target()` →
+   `score()`, **twice**, comparing rollout fingerprints and rewards — this is what
+   catches a non-deterministic `run_target()`, which `check` never runs;
+6. protected-paths resolution: the grader, dataset and manifest must all be covered.
+
+Then `verified.json` records the measured val reward, the split sizes, the dataset
+SHA-256 and the list of steps that ran.

benchmarks/json_extract/README.md

@@ -0,0 +1,17 @@
+# Benchmark: json_extract
+
+Structured-JSON extraction accuracy with **per-field partial credit**, on a
+deterministic zero-API extractor. Zero cost, so it runs in CI.
+
+Declared in [`project/benchmark.yaml`](project/benchmark.yaml). The only code is
+[`project/target.py`](project/target.py): `run(task, ctx, *, seed=0)` plus a
+`score(task, rollout)` — this benchmark uses `scoring: custom` because partial
+credit over parsed JSON is real logic, not something a config key should express.
+
+```bash
+cap-evolve benchmark verify json_extract
+cap-evolve run --spec json_extract/project/capevolve.yaml --project json_extract/project
+```
+
+The seed prompt asks for prose, so it scores 0. Adding `[JSON]` earns the name
+field (1/3) and `[FIELDS]` earns all three (3/3) — a graded, non-binary signal.

benchmarks/json_extract/mock_script.json

@@ -0,0 +1,5 @@
+{
+  "edits": [
+    {"file": "prompt.txt", "op": "ensure_contains", "text": "\n[JSON] Reply with a single JSON object and nothing else.\n[FIELDS] Include every field: name, city, year."}
+  ]
+}

benchmarks/json_extract/project/adapters/adapter.py

@@ -0,0 +1,7 @@
+from cap_evolve.zoo import ManifestAdapter
+
+
+class Adapter(ManifestAdapter):
+    """json_extract — everything is declared in ../benchmark.yaml."""
+
+    manifest_path = __file__

benchmarks/json_extract/project/benchmark.yaml

@@ -0,0 +1,31 @@
+# cap-evolve benchmark manifest — the DECLARATIVE half of a benchmark.
+# The only code is target.py (run + a custom score). Flat keys only.
+name: json_extract
+description: Structured-JSON extraction accuracy (per-field partial credit) for a deterministic zero-API extractor whose prompt is optimized.
+
+# --- dataset ---------------------------------------------------------------
+tasks_file: tasks.jsonl
+id_field: id
+input_field: input
+target_field: target
+
+# --- scoring ---------------------------------------------------------------
+scoring: custom                # per-field partial credit lives in target.py:score
+metric_direction: higher
+
+# --- what is optimized -----------------------------------------------------
+capability_path: seed_capability
+target_module: target.py
+
+# --- splits (seeded once; test is sealed) ----------------------------------
+split_seed: 0
+split_train: 0.5
+split_val: 0.25
+split_test: 0.25
+split_ids_file: ""
+num_trials: 1
+
+# --- protected paths (#142 tamper guard hashes exactly these) --------------
+protected_paths: [adapters, benchmark.yaml, target.py, tasks.jsonl]
+
+verified: false

benchmarks/json_extract/project/capevolve.yaml

@@ -0,0 +1,25 @@
+# GENERATED from benchmark.yaml by `cap-evolve benchmark add` — edit the
+# manifest and re-run `cap-evolve benchmark add --refresh`, not this file.
+capabilities: [system-prompt]
+capability_path: seed_capability
+actions: [edit]
+optimizer_skill: mock
+optimizer_model: ""
+algorithm_skill: hill-climb
+algorithm_focus: all
+dataset_source: adapter
+split_seed: 0
+split_train: 0.5
+split_val: 0.25
+split_test: 0.25
+split_ids_file: ""
+num_trials: 1
+metric_directions: [higher]
+gate_mode: paired
+gate_k_se: 1.0
+max_iterations: 5
+stall: 2
+store: copy
+# Declared by the manifest; #142's tamper guard hashes exactly these at baseline
+# and re-hashes them after every optimizer step.
+protected_paths: [adapters, benchmark.yaml, target.py, tasks.jsonl, capevolve.yaml, *gold*.json, *gold*.jsonl, *gold*.yaml, *gold*.yml, *gold*.csv, *gold*.txt, **/*gold*.json, **/*gold*.jsonl, **/*gold*.yaml, **/*gold*.yml, **/*gold*.csv, **/*gold*.txt]

benchmarks/json_extract/project/seed_capability/prompt.txt

@@ -0,0 +1 @@
+You are a helpful assistant. Describe what the user tells you.

benchmarks/json_extract/project/target.py

@@ -0,0 +1,68 @@
+"""json_extract's runner + a CUSTOM scorer — the two things a manifest can't declare.
+
+A deterministic zero-API stand-in extractor. The candidate prompt controls its
+behavior through two markers, so prompt edits provably move the score with no model
+calls:
+
+  ``[JSON]``   emit a JSON object instead of prose
+  ``[FIELDS]`` include all three fields (name/city/year) rather than just the name
+
+Scoring is ``custom`` because partial credit over parsed JSON fields is real logic
+(a per-field comparison after a parse that can fail) — exactly the kind of thing a
+config language should NOT try to express. Everything else is in ``benchmark.yaml``.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+from pathlib import Path
+
+from cap_evolve import Score
+
+_FIELDS = ("name", "city", "year")
+
+
+def _extract(text: str) -> dict:
+    """The stand-in's (deterministic) understanding of one sentence."""
+    m = re.match(r"(?P<name>.+?) was born in (?P<city>.+?) in (?P<year>\d{4})", text)
+    if not m:
+        return {}
+    return {"name": m["name"], "city": m["city"], "year": int(m["year"])}
+
+
+def run(task, ctx, *, seed: int = 0):
+    prompt = (Path(ctx) / "prompt.txt").read_text(encoding="utf-8")
+    facts = _extract(str(task.input))
+    if "[JSON]" not in prompt:
+        return {"output": f"Sure! That sentence is about {facts.get('name', 'someone')}.",
+                "trace": "prose (prompt did not ask for JSON)"}
+    keep = _FIELDS if "[FIELDS]" in prompt else ("name",)
+    return {"output": json.dumps({k: facts[k] for k in keep if k in facts},
+                                 sort_keys=True),
+            "trace": f"json fields={list(keep)}"}
+
+
+def score(task, rollout) -> Score:
+    """Per-field partial credit over parsed JSON — the bespoke half, as code."""
+    if rollout.error:
+        return Score(task_id=task.id, reward=0.0, trial_rewards=[0.0],
+                     feedback=f"Rollout failed ({rollout.error}); infrastructure "
+                              "noise, not a prompt defect.")
+    want = json.loads(str(task.target))
+    try:
+        got = json.loads(str(rollout.output or ""))
+        if not isinstance(got, dict):
+            raise ValueError("top level is not an object")
+    except Exception as e:  # noqa: BLE001
+        return Score(task_id=task.id, reward=0.0, trial_rewards=[0.0],
+                     feedback=f"output was not a JSON object ({e}); the prompt must "
+                              "instruct the agent to reply with a single JSON object "
+                              f"holding the {list(_FIELDS)} fields.")
+    hits = [k for k in _FIELDS if str(got.get(k, "")) == str(want[k])]
+    reward = len(hits) / len(_FIELDS)
+    missing = [k for k in _FIELDS if k not in hits]
+    fb = ("all fields correct" if not missing else
+          f"got {len(hits)}/{len(_FIELDS)} fields; wrong or absent: {missing}. The "
+          "prompt should name every required field explicitly — never hard-code values.")
+    return Score(task_id=task.id, reward=reward, feedback=fb, trial_rewards=[reward])

benchmarks/json_extract/project/tasks.jsonl

@@ -0,0 +1,12 @@
+{"id": "x01", "input": "Ada Lovelace was born in London in 1907 and later worked on compilers.", "target": "{\"city\": \"London\", \"name\": \"Ada Lovelace\", \"year\": 1907}"}
+{"id": "x02", "input": "Grace Hopper was born in Baltimore in 1914 and later worked on compilers.", "target": "{\"city\": \"Baltimore\", \"name\": \"Grace Hopper\", \"year\": 1914}"}
+{"id": "x03", "input": "Alan Turing was born in Maida Vale in 1921 and later worked on compilers.", "target": "{\"city\": \"Maida Vale\", \"name\": \"Alan Turing\", \"year\": 1921}"}
+{"id": "x04", "input": "Katherine Johnson was born in Hampton in 1928 and later worked on compilers.", "target": "{\"city\": \"Hampton\", \"name\": \"Katherine Johnson\", \"year\": 1928}"}
+{"id": "x05", "input": "Barbara Liskov was born in New York in 1935 and later worked on compilers.", "target": "{\"city\": \"New York\", \"name\": \"Barbara Liskov\", \"year\": 1935}"}
+{"id": "x06", "input": "Edsger Dijkstra was born in Rotterdam in 1942 and later worked on compilers.", "target": "{\"city\": \"Rotterdam\", \"name\": \"Edsger Dijkstra\", \"year\": 1942}"}
+{"id": "x07", "input": "Margaret Hamilton was born in Paris in 1949 and later worked on compilers.", "target": "{\"city\": \"Paris\", \"name\": \"Margaret Hamilton\", \"year\": 1949}"}
+{"id": "x08", "input": "Donald Knuth was born in Lansing in 1956 and later worked on compilers.", "target": "{\"city\": \"Lansing\", \"name\": \"Donald Knuth\", \"year\": 1956}"}
+{"id": "x09", "input": "Radia Perlman was born in Portsmouth in 1903 and later worked on compilers.", "target": "{\"city\": \"Portsmouth\", \"name\": \"Radia Perlman\", \"year\": 1903}"}
+{"id": "x10", "input": "Leslie Lamport was born in Brooklyn in 1910 and later worked on compilers.", "target": "{\"city\": \"Brooklyn\", \"name\": \"Leslie Lamport\", \"year\": 1910}"}
+{"id": "x11", "input": "Frances Allen was born in Peru in 1917 and later worked on compilers.", "target": "{\"city\": \"Peru\", \"name\": \"Frances Allen\", \"year\": 1917}"}
+{"id": "x12", "input": "Ken Thompson was born in New Orleans in 1924 and later worked on compilers.", "target": "{\"city\": \"New Orleans\", \"name\": \"Ken Thompson\", \"year\": 1924}"}

benchmarks/json_extract/verified.json

@@ -0,0 +1,26 @@
+{
+  "ok": true,
+  "at": "2026-07-30T14:08:15+00:00",
+  "cap_evolve": "0.1.0",
+  "val_reward": 0.0,
+  "n_tasks": 12,
+  "splits": {
+    "train": 6,
+    "val": 3,
+    "test": 3
+  },
+  "steps": [
+    "manifest parsed + validated",
+    "dataset loaded through the adapter: 12 task(s)",
+    "cap-evolve check executed on the generated project",
+    "splits computed: {'train': 6, 'val': 3, 'test': 3}",
+    "REAL smoke eval: 3 val task(s) x 2 passes through live() -> run_target() -> score()",
+    "degenerate-scorer probe: 3 task(s) scored with a correct vs a deliberately-wrong output",
+    "under-declaration sweep: every .py and answer-key-ish file under project/ (outside seed_capability/) is guard-hashed",
+    "protected paths resolved from the generated spec: ['adapters/adapter.py', 'benchmark.yaml', 'capevolve.yaml', 'target.py', 'tasks.jsonl']"
+  ],
+  "problems": [],
+  "dataset_sha256": "8e7d164f1d3f7440d86a604881006d3112cf055f6b7b3851a0408df5a29039a6",
+  "target_sha256": "33744fc166d055fa017654fb8b7c0400b0e033b1a41d7f5376848599219d9fd7",
+  "manifest_sha256": "11d012d109abff8f58d06d3b110ccfb842fd474597cede8c70f8b7b7617ab853"
+}

benchmarks/toy_calc/README.md

@@ -0,0 +1,21 @@
+# Benchmark: toy_calc
+
+Arithmetic accuracy of a deterministic zero-API stand-in agent whose system prompt
+is optimized. The smallest end-to-end proof in the repo: no model calls, runs in
+seconds, and the optimization provably moves the number.
+
+Declared in [`project/benchmark.yaml`](project/benchmark.yaml). The only code is
+[`project/target.py`](project/target.py) — one function, `run(task, ctx, *, seed=0)`.
+The stand-in computes correctly only when the candidate prompt contains `[CALC]`,
+so adding it (which the `mock` optimizer does) raises val 0.0 → 1.0.
+
+```bash
+cap-evolve benchmark verify toy_calc
+CAPEVOLVE_MOCK_SCRIPT=$PWD/mock_script.json \
+  cap-evolve run --spec toy_calc/project/capevolve.yaml --project toy_calc/project
+# -> baseline_val 0.0  ->  test_reward 1.0  (gate-accepted, test sealed)
+```
+
+`examples/toy_calc/` keeps the hand-written adapter form of the same benchmark — it
+is the "before" side of the boilerplate measurement, and the reference for what a
+custom `CapabilityAdapter` looks like when the manifest does not fit.