Benchmark zoo + cap-evolve benchmark add/verify + declarative manifest - #233
Benchmark zoo + cap-evolve benchmark add/verify + declarative manifest#233OsherElhadad wants to merge 2 commits into
cap-evolve benchmark add/verify + declarative manifest#233Conversation
…ifest Onboarding a benchmark was a from-scratch effort: a hand-written CapabilityAdapter subclass plus a ~100-line capevolve.yaml. Diffing the two generic bundled templates (jsonl_litellm vs huggingface_litellm) shows what actually repeats — module preamble, the dataset->Task loop, the infra-error branch of score, the match helper, the Score(...) construction, and nearly the whole spec. What does not repeat is one thing: how the target agent runs. So benchmark.yaml declares the repeating half and target.py keeps run(task, ctx, *, seed=0) as code. There is deliberately no config language for run() — running an agent is real logic, and a DSL reimplementing Python would be worse than the Python it replaced. `scoring: custom` keeps a bespoke predicate as code too. Measured on the same benchmark (toy_calc): 78 -> 36 hand-authored non-blank lines (-54%); adapters/adapter.py and capevolve.yaml are generated (0 hand-authored). `cap-evolve benchmark verify` EXECUTES the benchmark rather than parsing its manifest: the real check gate, a dataset load through the real adapter, the seeded split plus the honest-gate floor (val >= MIN_VAL_TASKS and a non-empty sealed test split, so a 3-task dataset fails at verify rather than mid-run inside gate.decide), and a real zero-API smoke eval — every val task through live() -> run_target() -> score(), twice, comparing rollout fingerprints and rewards, which is what catches a non-deterministic run_target() (check never runs the target). verified.json records the measured reward, split sizes and dataset SHA-256; `benchmark list` reads that stamp from disk, not the manifest's `verified:` flag. A zoo entry keeps the manifest, scorer and dataset INSIDE the project dir so #142's tamper guard covers them by construction — a first pass parked them at the benchmark root and resolve_protected() hashed only adapters/adapter.py, i.e. a declared-but-unprotected grader. `cap-evolve --help` now generates the subcommand listing from COMMANDS plus each handler's docstring, so adding a subcommand no longer edits a literal usage string five branches all conflict on.
skillberry-bot
added
enhancement
|
🏷️ Automatic Labeling I've analyzed this pull request and added the following labels:
These labels were selected based on the PR title, description, and changed files. If you believe any labels are incorrect or missing, feel free to adjust them manually. |
🔬 EvidenceAll commands run in 1. Baseline suite (before any change)2. The boilerplate, measured3. The whole generated adapter$ cat benchmarks/toy_calc/project/adapters/adapter.py
from cap_evolve.zoo import ManifestAdapter
class Adapter(ManifestAdapter):
"""toy_calc — everything is declared in ../benchmark.yaml."""
manifest_path = __file__4. Scaffold from scratch → verify (no hand-editing)5.
|
🔍 Review — PR #233CHANGES REQUESTED. The scaffolder, the manifest, the CLI wiring and the two zoo entries are solid, and the four claimed breakage cases all reproduce. But the headline claim — " BlockingB1. The smoke eval measures if rewards and all(r == rewards[0] for r in rewards) and rewards[0] == 1.0:
rep.notes.append("the seed capability already scores 1.0 on every smoke task — ...")So the two canonical reward hacks both verify clean: # attack 1: appended to target.py; manifest still says `scoring: exact`
def score(task, rollout):
return Score(task_id=task.id, reward=1.0, feedback="perfect", trial_rewards=[1.0])# attack 2: run() returns the gold answer
def run(task, ctx, *, seed=0):
return {"output": str(task.target), "trace": "I just read the gold answer"}Consequence: the zoo's central promise ("verifier-gated library") does not hold. A Fix — cheap and sufficient: B2. Scoring if callable(custom): # a custom scorer always wins over the declared mode
return custom(task, rollout)A manifest that declares B3. A hand-written
The stamp file I wrote by hand does not even contain B4. Both are
Fix: after resolving, require B5.
(I weakened only B6. A zero-holdout / degenerate split passes the honesty floor — The floor checks only Consequence: the PR's headline is "a 3-task dataset fails at verify, not mid-run", but the worse failure — a test split that is literally the training set, so the sealed number is a fit metric — passes silently. A B7.
Here the injected keys land above the real ones so the later definitions win, but that is an accident of key ordering in the template — an injected Non-blocking(7 items.) N1. Today's callers are safe ( N2. The PR asks "what happens when a zoo benchmark needs to override one generated method?" I tried it: adding a N3. Unanchored
N4.
N5.
N6. Non-determinism at coarser-than-one-second granularity escapes the twice-run fingerprint — The two passes run back-to-back in one process, so a Real LLM-backed runners drift on much longer scales (cache state, rate-limit retries), so the guard is genuinely useful but should not be described as the determinism proof. The N7. Nits
Can I make
|
| # | Attack | Result |
|---|---|---|
| 1 | score() appended to target.py returning reward=1.0 unconditionally (manifest still says scoring: exact) |
PASSED VERIFY, val_reward: 1.0, badge verified: True |
| 2 | run() returns str(task.target) — echoes the gold answer |
PASSED VERIFY, val_reward: 1.0 |
| 3 | run() reads tasks.jsonl off disk and returns the matching target |
PASSED VERIFY, val_reward: 1.0 |
| 4 | Every val task is a content-duplicate of a train task (new ids, identical input/target) | PASSED VERIFY, 16 tasks, splits {train:8,val:4,test:4} |
| 5 | tasks_file: ../../evil.jsonl (dataset outside the benchmark dir), undeclared |
blocked — but only by the protected_paths string check, not by any containment rule |
| 6 | Same, with ../../evil.jsonl added to protected_paths |
PASSED VERIFY — dataset genuinely loaded from /tmp/atk/evil.jsonl |
| 7 | target_module: ../../pwned.py (writes a file at import), declared |
PASSED VERIFY — /tmp/atk/PWNED created; code executed outside the project dir |
| 8 | Hand-written verified.json with {"ok": true, "val_reward": 0.99, "n_tasks": 999}, no steps, bogus sha |
FORGED BADGE — benchmark list reports verified: True |
| 9 | Verify + stamp, then edit tasks.jsonl |
STALE STAMP STILL VERIFIED (2ba70d56… vs actual c7f45566…) |
| 10 | split_ids_file with train == val == test == all ids (zero holdout) |
PASSED VERIFY, {train:8,val:8,test:8}, no overlap note |
| 11 | Ratios train 0.0 / val 0.9 / test 0.1 |
PASSED VERIFY, {train:0,val:7,test:1} |
| 12 | Weaken protected_paths in the generated capevolve.yaml only (manifest intact), on the #197 merge |
PASSED VERIFY while rep.protected == ['adapters/adapter.py'] |
Blocked as claimed: stubbed score(), non-deterministic run() (planted random.random() — caught, message names seed=0 and forward \seed`), 3-task dataset, missing dataset file, duplicate task ids, under-declared protected_paths` in the manifest. Those six all reproduce exactly as the PR describes. What none of them cover is whether the benchmark measures anything.
Protected-paths generalization
No — a third author's new file is silently unprotected. The #197 fix is real for the two committed examples: on the merged tree resolve_protected(project) returns exactly ['adapters/adapter.py', 'benchmark.yaml', 'target.py', 'tasks.jsonl'], a clean protect.verify returns [], and touching target.py raises TamperError: … modified target.py. Verified literally (output below).
But protected_paths is a static list in the manifest, and _FIELDS defaults it to [] → default_protected() → the same four hardcoded names. A third author who adds anything else gets nothing:
third-author protected set: ['adapters/adapter.py', 'benchmark.yaml', 'target.py', 'tasks.jsonl']
verify third-author layout ok= True problems= []
=> NEW FILES SILENTLY UNPROTECTED (tamper undetected)
I added helpers.py (a module target.py could import), scorer2.py (a second grader) and answers_gold.json (an answer key with a non-gold*-prefixed name — #197's default glob is *gold* on data suffixes, and answers_gold.json does match *gold*, yet it is not protected here because the manifest's explicit protected_paths replaces the defaults wholesale). verify reported ok: true, then all three were modified and protect.verify stayed silent.
So the PR's claim that under-declaration is detected is true only for the four names verify hardcodes in must (zoo.py:767). Anything a third author adds is neither protected nor flagged — it is the same trap the author already fell into once, just moved one file over. The layout convention works for the two committed examples and no further. A generalizing fix: protect project/ minus capability_path minus the run dir by default (deny-list rather than allow-list) — the seed capability is the only thing that should be writable, and that is exactly what the manifest already names.
Boilerplate claim
Independently reproduced, and it holds — using an AST-based count (non-blank, non-comment, docstrings excluded, which is stricter than "non-blank" and therefore fairer to the "before" side):
18 benchmarks/toy_calc/project/target.py <- hand-authored
18 benchmarks/toy_calc/project/benchmark.yaml <- hand-authored
3 benchmarks/toy_calc/project/adapters/adapter.py (generated)
21 benchmarks/toy_calc/project/capevolve.yaml (generated)
43 benchmarks/json_extract/project/target.py <- hand-authored (custom scorer)
88 templates/adapters/jsonl_litellm/adapter.py
13 templates/adapters/jsonl_litellm/capevolve.yaml
98 templates/adapters/huggingface_litellm/adapter.py
562 core/cap_evolve/zoo.py
36 hand-authored (18 + 18) — exactly the PR's number. Against jsonl_litellm as the "before" I get 88 + 13 = 101, matching the PR's alternate figure, so 36 → −64% there and the −54% claim against a 78-line baseline is if anything conservative. Excluding generated files is the right call; I'd note that capevolve.yaml at 21 lines is larger than the 18-line manifest it's generated from, so the win is that it's derived, not that it's shorter.
The complexity is not free, though: it moved into zoo.py's 562 lines that every zoo author now depends on and none of them owns. That is a good trade for two benchmarks and a fine trade for twenty — but the concrete cost surfaced in N2: an author who needs to override one generated hook edits the generated shim and --refresh deletes it without a word. The abstraction has no supported override path, which is the usual way generated-code wins turn into 3am debugging.
Merge-order note
I trial-merged all three. Recommend #197 → #195 → #137, i.e. the author's order, with one correction to their expectation:
- Protected-paths tamper guard: verify the optimizer never edited scoring/eval/task files #197 (
feat/issue-142-protected-paths) — conflicts only inCHANGELOG.md;core/auto-merges. After merging,resolve_protectedreturns the claimed 4-path manifest andTamperErrorfires on a tamperedtarget.py. Must go first, because B5 is only visible onceprotectexists (pre-Protected-paths tamper guard: verify the optimizer never edited scoring/eval/task files #197verifytakes theImportErrorbranch andrep.protectedstays[], hiding the manifest/spec skew). - Guard tiny/empty val splits and add a Student-t small-sample correction to the paired gate #195 (
fix/issue-113-small-samples) — clean merge, no conflicts. After merging,zoo.MIN_VAL_TASKS == splits.MIN_VAL_TASKS == 2, so thegetattratzoo.py:71picks up the shared constant exactly as claimed. On this branch (pre-Guard tiny/empty val splits and add a Student-t small-sample correction to the paired gate #195)splitshas noMIN_VAL_TASKSand the fallback literal2is used — same value, so the Guard tiny/empty val splits and add a Student-t small-sample correction to the paired gate #195-absent path does not silently use a different floor. Good. - CLI ergonomics overhaul: structured flags, did-you-mean, per-subcommand help, UTF-8/exit codes #137 (
feat/issue-137-cli-ergonomics) — 5 conflicts incli.py, not the "small textual conflict inmain()" the PR predicts. The module docstring,_cmd_check,_cmd_splits,_cmd_runandmain()all collide, because CLI ergonomics overhaul: structured flags, did-you-mean, per-subcommand help, UTF-8/exit codes #137 rewrites each handler onto its_parser()helper and replaces themain()help branch with_usage()+-V. Resolution is mechanical (take CLI ergonomics overhaul: structured flags, did-you-mean, per-subcommand help, UTF-8/exit codes #137's side everywhere, keep this PR's"benchmark": _cmd_benchmarkrow) andgit checkout -malready produces aCOMMANDSdict containing all seven entries — but it needs_cmd_benchmarkconverted to_parser()for consistency, so budget more than a docstring fixup. No conflict with the intent: both branches generate the listing fromCOMMANDS, and CLI ergonomics overhaul: structured flags, did-you-mean, per-subcommand help, UTF-8/exit codes #137's_usage()is the better implementation (padded width, docstring-less fallback), so keeping it costs this PR nothing.
CLI integration verdict: benchmark appears in the generated listing with no manual edit, on both this branch and the merge — COMMANDS is the single source and --help renders it. No conflict with #214's test. #214 owns core/tests/test_documented_cli.py; this PR's assertion lives in its own test_cli_help_lists_benchmark_from_the_generated_listing (test_benchmark_zoo.py:233-235) and only greps cli.py for the dead literal version|splits|check|run|estimate|dashboard, which #137 also removes — the two agree rather than compete. stdout is one JSON object on every path I could reach, including verify nope (exit 1, {"ok": false, ...}) and a broken benchmark (exit 1). Note main()'s help output goes to stderr, so it doesn't pollute the JSON contract.
ManifestAdapter / contract: satisfies all three required methods plus the defaulted hooks; cap-evolve check on the generated project returns ok: true, stubs: []. Per the standing caveat I did not infer anything from that — a raising materialize() still yields ok: true, which is precisely why B1 matters: verify inherits check's blind spot for outcome and adds a mechanical execution check on top, not a semantic one.
Seal: the smoke eval does not consume the seal. verify runs on val only, creates no run dir, and the sealed test ids appear in zero artifacts:
splits: {'train': ['a5','a2','a6','a3'], 'val': ['a1','a4'], 'test': ['a8','a7']}
stamp ok: True | test ids a7/a8 present in verified.json (sha excluded): False False
run dirs created under bench: [] seal artifacts: []
splits deterministic across re-derivation: True
The TestSealError-on-second-finalize path also passes in the e2e test. Splits are seeded once and reproducible. (B6 is about a manifest that declares a degenerate split, not about verify leaking one.)
Verification I re-ran
Suite — 203 passed, as claimed:
$ cd /tmp/rv-233 && PYTHONPATH=/tmp/rv-233/core python -m pytest core/tests -q
........................................................................ [ 35%]
........................................................................ [ 70%]
........................................................... [100%]
203 passed in 63.38s (0:01:03)
compileall on core + benchmarks: exit 0, silent.
Real verify on the bundled toy_calc (exit 0) — all six steps present:
"steps": [
"manifest parsed + validated",
"dataset loaded through the adapter: 8 task(s)",
"cap-evolve check executed on the generated project",
"splits computed: {'train': 4, 'val': 2, 'test': 2}",
"REAL smoke eval: 2 val task(s) x 2 passes through live() -> run_target() -> score()",
"protected paths declared + resolved: ['adapters', 'benchmark.yaml', 'target.py', 'tasks.jsonl']"
],
"problems": [], "val_reward": 0.0, "n_tasks": 8, "splits": {"train": 4, "val": 2, "test": 2}
The smoke eval is real: I confirmed by planting random.random() in run() and watching it fail with the fingerprint diff quoted in the table.
cap-evolve check on the generated project:
{"ok": true, "stubs": [], "notes": ["tasks('val') -> 8 task(s)", "scorer deterministic (probe reward=0.0000)", "materialize() callable (dry-run into temp copy; host untouched)"]}
json_extract graded signal — 0.0 / ⅓ / 1.0 reproduced exactly:
prose: 0.0 | [JSON]: 0.3333 | [JSON]+[FIELDS]: 1.0
E2E to a sealed test number: test_zoo_benchmark_runs_end_to_end_to_a_sealed_test_number passes — baseline 0.0, gate-accepted, test.reward == 1.0, second finalize raises TestSealError.
Four claimed breakage cases (all reproduce, exit 1, actionable message): stubbed score() → "unimplemented adapter methods … score"; non-deterministic run() → "NON-DETERMINISTIC: 2 task(s) … seed=0 … forward \seed`"; 3-task dataset → "val split has 1 task(s), below the honest-gate minimum of 2"**and**"test split is EMPTY"; missing dataset → "dataset file missing … tasks.jsonl". Plus duplicate ids and manifest-level under-declared protected_paths`.
#197 merge:
$ git merge origin/feat/issue-142-protected-paths # CHANGELOG.md only
resolve_protected: ['adapters/adapter.py', 'benchmark.yaml', 'target.py', 'tasks.jsonl']
clean verify: []
TamperError: cap-evolve TAMPER DETECTED: 1 protected file(s) changed under …/project — modified target.py.
zoo.verify ok: True | rep.protected: ['adapters/adapter.py','benchmark.yaml','target.py','tasks.jsonl']
…and the generalization probe, same tree:
third-author protected set: ['adapters/adapter.py','benchmark.yaml','target.py','tasks.jsonl']
verify third-author layout ok= True problems= []
=> NEW FILES SILENTLY UNPROTECTED (tamper undetected)
#195 merge: clean; zoo.MIN_VAL_TASKS = 2 | splits.MIN_VAL_TASKS = 2.
#137 merge: CONFLICT (content): core/cap_evolve/cli.py, 5 conflict hunks (lines 8, 72, 88, 193, 818).
--help:
$ python -m cap_evolve.cli --help # stderr
usage: cap-evolve {version|splits|check|run|estimate|dashboard|benchmark} [args]
benchmark Manage the benchmark zoo: list | add | verify a declarative benchmark.
benchmark --help renders its own subparsers with the documented examples.
Summary: the mechanical half of verify is genuinely real and I could not fake my way past the six guards the PR names. The semantic half doesn't exist yet — B1 (no headroom/degenerate-scorer assertion), B3 (unauthenticated stamp), B4 (path containment) and B5 (check rep.protected, not declared) are the four that turn "it ran" into "it measures". B1 and B5 are each a handful of lines and reuse signal the code already computes.
The review made `verify` pass on a broken benchmark in 9 of 12 attacks. Its mechanical claims were all true — it really does run `live()` -> `run_target()` -> `score()` twice for every val task — but it drew no conclusion from the results. This fixes the seven blocking findings plus the third-author protection gap. verify now concludes: - HEADROOM is a hard failure, not a note. A seed capability already scoring 1.0 has nothing to optimize, which is the one thing `baseline` exists to confirm and the signature of the two commonest reward hacks (a score() wired to a constant, a run() returning task.target or reading the answer key off disk). A genuinely saturated reference fixture opts out loudly with allow_saturated_baseline: true. - A DEGENERATE-SCORER PROBE scores a synthetically correct rollout against a deliberately wrong one and requires the rewards to differ. Correct-vs-wrong is a property of the scorer alone, so it catches a score() that ignores its input at any baseline — which the headroom rule cannot see. - SPLITS must be genuinely disjoint with a non-empty train, asserted on the REALIZED split. train == val == test passed the old floor; #99 found the repo's own headline tau^2 number came from exactly that. Built as a real Splits rather than a throwaway type("S", (), ...). - CONTAINMENT is an allowlist, the shape PR #210 used at gepa.py: every path key must be a plain relative path whose resolved parent is inside the project dir, checked once in load_manifest so no use site can bypass it. `target_module: ../../pwned.py` previously EXECUTED code outside the project dir during verify, from a location #142's guard structurally cannot hash. Denylists have failed six times in this batch; this is not a seventh. - PROTECTED PATHS are asserted on what protect.resolve_protected() actually resolves from the GENERATED capevolve.yaml — the artifact the runtime guard reads — not on what benchmark.yaml claims. Weakening only the spec left verify reporting OK with rep.protected == ['adapters/adapter.py'], the same wrong-artifact bug as #189. - protected_paths is now ADDITIVE: unioned with the layout defaults and #197's globs, never substituted. #197's own list replaces its defaults wholesale, so declaring four paths silently switched off the *gold* answer-key globs. Union is the only default that fails safe. Plus an UNDER-DECLARATION SWEEP flagging any .py or answer-key-ish file under project/ (outside capability_path/) the guard would not hash: a third author's helpers.py / scorer2.py were silently unprotected and tampering all three went undetected. - The STAMP is evidence, not a differently-located claim. verified.json now records the grader and manifest hashes alongside the dataset, and `benchmark list` RE-CHECKS every one: a hand-written stamp (no steps, no hashes) and a stale one both read verified: false with the reason in stale_reason. The dataset_sha256 was written and never compared. - A score() without `scoring: custom` is a hard error. It silently overrode the declared mode, so both the manifest and `benchmark list` reported a grading mode that was not in effect — the unknown-key hard error, one level deeper. - --description is emitted as a quoted YAML scalar. A newline redefined manifest keys. Also: content-duplicate task rows are refused (fresh ids on identical rows split cleanly, so val became a copy of train); tasks(split) honours its argument instead of handing the sealed test split to any caller; an empty or uncompilable regex target is a dataset error rather than a free 1.0 / a mid-eval crash; numeric uses math.isclose(rel_tol=...) and accepts scientific notation; and --refresh KEEPS a hand-edited adapters/adapter.py instead of clobbering it, so overriding one generated hook is a supported edit rather than work the next manifest change deletes. All 12 attacks now fail. 220 passed (was 203, + 17 new tests), 0 failed apart from the known #200 dashboard port flake; compileall clean.
🔧 Review fixesThanks — the summary landed: "the mechanical claims are all true; the semantic ones are not." New commit
|
| # | Finding | State at a03e6a44 |
Evidence |
|---|---|---|---|
| B1 | headroom is only a note | not fixed | zoo.py:754-757 verbatim rep.notes.append("…no headroom to optimize…") |
| B2 | stray score() overrides declared mode |
not fixed | zoo.py:302 verbatim if callable(custom): # a custom scorer always wins |
| B3 | stamp never re-checked | not fixed | grep -n dataset_sha256 zoo.py → 1 hit, the write at :792 |
| B4 | no path containment | not fixed | grep -c 'resolve().parent|is_relative_to' zoo.py → 0 |
| B5 | protected step reads the manifest | not fixed | zoo.py:760 verbatim declared = m["protected_paths"] or … |
| B6 | zero-holdout passes | not fixed | zoo.py:684 verbatim type("S", (), {...})(), no overlap check |
| B7 | --description unquoted into YAML |
not fixed | zoo.py:487 verbatim description: {description or …} |
So everything below is new work, verified by me rather than assumed.
Attacks closed
All 12 re-run against a working copy of bundled toy_calc on the pushed commit. 12/12 now fail (was 3/12).
| # | Attack | Result | Refused because |
|---|---|---|---|
| 1 | score() hard-wired to 1.0 | FAILS | NO HEADROOM: seed capability scores 1.0 on all 2 smoke val tasks |
| 2 | run() returns task.target | FAILS | NO HEADROOM: seed capability scores 1.0 on all 2 smoke val tasks |
| 3 | run() reads the answer key off disk | FAILS | NO HEADROOM: seed capability scores 1.0 on all 2 smoke val tasks |
| 4 | val == train content duplicates (new ids) | FAILS | dataset/adapter load failed: tasks.jsonl:9 task 'a1_dup' is a CONTENT duplicate of 'a1' |
| 5 | tasks_file: ../../evil.jsonl (declared in protected_paths) | FAILS | tasks_file='../../evil.jsonl' must be a plain relative path inside the project dir |
| 6 | target_module: ../../pwned.py (code execution) |
FAILS | target_module='../../pwned.py' must be a plain relative path inside the project dir — side-effect file created: False |
| 7 | forged verified.json | FAILS | verified.json is missing ['steps', 'splits', 'dataset_sha256', 'target_sha256', 'manifest_sha256'] — hand-written, not measured |
| 8 | stale stamp (dataset edited after verify) | FAILS | STALE STAMP: dataset_sha256 changed since verification (2ba70d56bce1 → 1991103bc2fd) |
| 9 | train == val == test (zero holdout) | FAILS | test split OVERLAPS train/val on 8 task id(s) (and train ∩ val on 8) |
| 10 | ratios train 0.0 / val 0.9 / test 0.1 | FAILS | train split is EMPTY ({'train': 0, 'val': 7, 'test': 1}) |
| 11 | generated spec weakened, manifest intact (wrong-artifact skew) | FAILS | the runtime guard will NOT hash ['benchmark.yaml', 'target.py', 'tasks.jsonl'] (it resolves only ['adapters/adapter.py']) |
| 12 | stray score() overriding scoring: exact | FAILS | dataset/adapter load failed: target.py defines score() but the manifest declares scoring: 'exact' |
Full literal attack output (all 12)
### ATTACK 1: score() hard-wired to 1.0
verify ok = False
problem: NO HEADROOM: the seed capability already scores 1.0 on all 2 smoke val task(s). A benchmark that is perfect at baseline cannot demonstrate an improvement, so optimizing it is meaningless — and this is the signature of the two commonest reward hacks: a score() hard-wired to a constant, and a run() that returns task.target (or reads the answer key off disk). Make the tasks harder, weaken the seed capability, or — for a genuinely saturated reference fixture — declare `allow_saturated_baseline: true` in /var/folders/zh/srgnbq_97qvb6002zsr40tgc0000gn/T/tmpljivcltz/toy_calc/project/benchmark.yaml.
problem: SCORER DOES NOT DISCRIMINATE: a correct output and a deliberately wrong one ('__CAPEVOLVE_DELIBERATELY_WRONG__') received the SAME reward on every one of 2 smoke task(s) — e.g. {'a1': {'correct': 1.0, 'wrong': 1.0}, 'a4': {'correct': 1.0, 'wrong': 1.0}}. score() is therefore not a function of the agent's output: it grades a constant, so every number this benchmark produces is fiction and no optimizer can learn from it. Fix score() in target.py (or the 'custom' mode's target field).
### ATTACK 2: run() returns task.target
verify ok = False
problem: NO HEADROOM: the seed capability already scores 1.0 on all 2 smoke val task(s). A benchmark that is perfect at baseline cannot demonstrate an improvement, so optimizing it is meaningless — and this is the signature of the two commonest reward hacks: a score() hard-wired to a constant, and a run() that returns task.target (or reads the answer key off disk). Make the tasks harder, weaken the seed capability, or — for a genuinely saturated reference fixture — declare `allow_saturated_baseline: true` in /var/folders/zh/srgnbq_97qvb6002zsr40tgc0000gn/T/tmpmv694v94/toy_calc/project/benchmark.yaml.
### ATTACK 3: run() reads the answer key off disk
verify ok = False
problem: NO HEADROOM: the seed capability already scores 1.0 on all 2 smoke val task(s). A benchmark that is perfect at baseline cannot demonstrate an improvement, so optimizing it is meaningless — and this is the signature of the two commonest reward hacks: a score() hard-wired to a constant, and a run() that returns task.target (or reads the answer key off disk). Make the tasks harder, weaken the seed capability, or — for a genuinely saturated reference fixture — declare `allow_saturated_baseline: true` in /var/folders/zh/srgnbq_97qvb6002zsr40tgc0000gn/T/tmp4hmrryje/toy_calc/project/benchmark.yaml.
### ATTACK 4: val == train content duplicates (new ids)
verify ok = False
problem: dataset/adapter load failed: /private/var/folders/zh/srgnbq_97qvb6002zsr40tgc0000gn/T/tmpecqtuzw8/toy_calc/project/tasks.jsonl:9 task 'a1_dup' is a CONTENT duplicate of 'a1': identical input and target under a different id. Distinct ids make it split cleanly, so the same task can land in train and val — the gate then rewards memorization, and the sealed test number measures recall of a seen row. De-duplicate the dataset.
### ATTACK 5: tasks_file: ../../evil.jsonl (declared in protected_paths)
verify ok = False
problem: tasks_file='../../evil.jsonl' must be a plain relative path inside the benchmark's project dir (/private/var/folders/zh/srgnbq_97qvb6002zsr40tgc0000gn/T/tmps5x3kwl6/toy_calc/project). Absolute paths, `~` and `..` are refused: tasks_file is read (and for target_module, IMPORTED) by verify, so a value escaping the project dir would execute code and load data that #142's tamper guard structurally cannot hash — the guard only covers paths under the project dir.
### ATTACK 6: target_module: ../../pwned.py (CODE EXECUTION)
verify ok = False
problem: target_module='../../pwned.py' must be a plain relative path inside the benchmark's project dir (/private/var/folders/zh/srgnbq_97qvb6002zsr40tgc0000gn/T/tmp91_oqafw/toy_calc/project). Absolute paths, `~` and `..` are refused: target_module is read (and for target_module, IMPORTED) by verify, so a value escaping the project dir would execute code and load data that #142's tamper guard structurally cannot hash — the guard only covers paths under the project dir.
>>> code executed outside project dir? False
### ATTACK 7: hand-written verified.json
index() -> verified=False stale=True
reason: verified.json is missing ['steps', 'splits', 'dataset_sha256', 'target_sha256', 'manifest_sha256'] — `cap-evolve benchmark verify` always writes these, so this stamp was hand-written, not measured. Re-run verify.
(control) a REAL stamp reads verified=True
### ATTACK 8: dataset mutated after stamping
index() -> verified=False stale=True
reason: STALE STAMP: ['dataset_sha256'] changed since verification ({'dataset_sha256': ('2ba70d56bce1', '1991103bc2fd')}). The badge certifies a dataset/grader/manifest that no longer exists on disk. Re-run `cap-evolve benchmark verify`.
### ATTACK 9: train == val == test (zero holdout)
verify ok = False
problem: test split OVERLAPS train/val on 8 task id(s) (e.g. ['a1', 'a2', 'a3', 'a4', 'a5']) — the 'sealed' number would be measured on data the optimizer trained against, making it a fit metric, not a held-out result. train/val/test must be disjoint.
problem: train and val OVERLAP on 8 task id(s) (e.g. ['a1', 'a2', 'a3', 'a4', 'a5']) — the acceptance gate would score candidates on the very tasks reflection read, so every accept is measuring memorization.
### ATTACK 10: ratios train 0.0 / val 0.9 / test 0.1
verify ok = False
problem: train split is EMPTY — there is nothing for the optimizer to reflect over, so no candidate can be proposed from evidence. Raise split_train above 0.0 in /var/folders/zh/srgnbq_97qvb6002zsr40tgc0000gn/T/tmp20x8ihwt/toy_calc/project/benchmark.yaml.
(manifest still declares all four; only the generated spec was weakened)
### ATTACK 11: generated spec weakened, manifest intact (wrong-artifact skew)
verify ok = False
problem: the runtime tamper guard will NOT hash ['benchmark.yaml', 'target.py', 'tasks.jsonl']. What it actually resolves from /var/folders/zh/srgnbq_97qvb6002zsr40tgc0000gn/T/tmp6z1ta6k9/toy_calc/project/capevolve.yaml is ['adapters/adapter.py'] — so the grader / dataset / manifest is optimizer-writable and a candidate could 'improve' by rewriting it. This is checked against the generated spec (what the guard reads), not benchmark.yaml (what it was derived from); if they have diverged, re-run `cap-evolve benchmark add /var/folders/zh/srgnbq_97qvb6002zsr40tgc0000gn/T/tmp6z1ta6k9/toy_calc --refresh`.
problem: UNDER-DECLARED: 1 file(s) inside /var/folders/zh/srgnbq_97qvb6002zsr40tgc0000gn/T/tmp6z1ta6k9/toy_calc/project are code or answer-key data that the tamper guard will not hash: ['target.py']. A helper the grader imports is as much the grader as adapter.py, and an answer key the optimizer can rewrite is a free 1.0. Add them to protected_paths in /var/folders/zh/srgnbq_97qvb6002zsr40tgc0000gn/T/tmp6z1ta6k9/toy_calc/project/benchmark.yaml and re-run `cap-evolve benchmark add /var/folders/zh/srgnbq_97qvb6002zsr40tgc0000gn/T/tmp6z1ta6k9/toy_calc --refresh` — or move them under seed_capability/ if they are genuinely part of the artifact being optimized.
### ATTACK 12: stray score() overriding scoring: exact
verify ok = False
problem: dataset/adapter load failed: /private/var/folders/zh/srgnbq_97qvb6002zsr40tgc0000gn/T/tmpau3ozsxk/toy_calc/project/target.py defines score(task, rollout) but benchmark.yaml declares scoring: 'exact'. A code scorer silently overriding the declared mode makes the manifest — and `benchmark list` — report a grading mode that is not the one in effect. Either set `scoring: custom` to declare it, or delete score() and let the declared mode grade.
Notes on two of them:
- Feat/cap evolve dashboard #6 is the code-execution case.
pwned.pywrites a marker at import. The run reportsside-effect file created: False— the containment check fires inload_manifest, before_load_target_moduleimports anything. There is a dedicated test for exactly this ordering (test_target_module_escape_never_imports), because "rejected after execution" would be no fix at all. - fix: repair quickstart command — add toy_calc run.sh (closes #3) #4 was still passing after my first pass and I only caught it by re-running the matrix rather than trusting the fixes. Content-duplicate rows (identical
input+targetunder fresh ids) split cleanly and passed every honesty check. Fixed at the existing id-duplicate guard intasks(), which is where all callers already route.
Third-author files are protected automatically
Root cause was as you diagnosed: protected_paths replaced #197's derived defaults wholesale, and verify only hardcoded four names in must. Two changes:
- Protection is additive.
effective_protected()unions the manifest's list with the layout defaults and Protected-paths tamper guard: verify the optimizer never edited scoring/eval/task files #197's_DEFAULT_GLOBS, and the generatedcapevolve.yamlcarries the union. Declaring one more path can now never unprotect something — which is the only default that fails safe. (Note this also meansanswers_gold.jsonis protected with no declaration at all: the*gold*globs are alive again.) - Under-declaration is detected. A sweep over
project/flags every.pyand answer-key-ish data file (gold|answer|label|solution|truth|key|expected× data suffixes) that the guard would not hash, excludingcapability_path/— the one thing that must stay writable — plus run dirs and caches.
Run on the #197 merge (protect.py + harness.py from feat/issue-142-protected-paths), using the real protect.verify() with a real RunDir:
=== 1. a third author adds three files to the zoo entry
added project/helpers.py
added project/scorer2.py
added project/answers_gold.json
=== 2. verify now DETECTS the under-declaration (it used to say ok: true)
ok = False
problem: UNDER-DECLARED: 2 file(s) inside /var/folders/zh/srgnbq_97qvb6002zsr40tgc0000gn/T/tmpd4182zzu/toy_calc/project are code or answer-key data that the tamper guard will not hash: ['helpers.py', 'scorer2.py']. A helper the grader imports is as much the grader as adapter.py, and an answer key the optimizer can rewrite is a free 1.0. Add them to protected_paths in /var/folders/zh/srgnbq_97qvb6002zsr40tgc0000gn/T/tmpd4182zzu/toy_calc/project/benchmark.yaml and re-run `cap-evolve benchmark add /var/folders/zh/srgnbq_97qvb6002zsr40tgc0000gn/T/tmpd4182zzu/toy_calc --refresh` — or move them under seed_capability/ if they are genuinely part of the artifact being optimized.
answers_gold.json already covered by the UNIONED *gold* globs: True
=== 3. declare the two modules, regenerate the spec, verify
verify ok = True problems = []
protect.resolve_protected -> ['adapters/__pycache__/adapter.cpython-314.pyc', 'adapters/adapter.py', 'answers_gold.json', 'benchmark.yaml', 'capevolve.yaml', 'helpers.py', 'scorer2.py', 'target.py', 'tasks.jsonl']
project/helpers.py IS guard-hashed: True
project/scorer2.py IS guard-hashed: True
project/answers_gold.json IS guard-hashed: True
=== 4. tamper with each file; the REAL #197 guard must raise TamperError
--- project/helpers.py
TamperError raised: True
message: cap-evolve TAMPER DETECTED during tampering with helpers.py: 1 protected file(s) changed under /private/var/folders/zh/srgnbq_97qvb6002zsr40tgc0000gn/T/tmpd4182zzu/toy_calc/project — modified helpers.py. Protected paths are the sc
--- project/scorer2.py
TamperError raised: True
message: cap-evolve TAMPER DETECTED during tampering with scorer2.py: 1 protected file(s) changed under /private/var/folders/zh/srgnbq_97qvb6002zsr40tgc0000gn/T/tmpd4182zzu/toy_calc/project — modified scorer2.py. Protected paths are the sc
--- project/answers_gold.json
TamperError raised: True
message: cap-evolve TAMPER DETECTED during tampering with answers_gold.json: 1 protected file(s) changed under /private/var/folders/zh/srgnbq_97qvb6002zsr40tgc0000gn/T/tmpd4182zzu/toy_calc/project — modified answers_gold.json. Protected pa
=== 5. control: the seed capability stays WRITABLE (it is the target)
editing seed_capability/prompt.txt -> verify returns []
ALL THREE THIRD-AUTHOR FILES ARE PROTECTED, AND TAMPERING EACH RAISES TamperError.
So: the two modules are flagged (they need declaring — detection, not silent protection, because an author who genuinely wants a file writable should say so), the answer key is already covered, and once declared all three raise TamperError naming the file. The seed capability stays writable, as it must.
Numbered response to all 19 findings
Blocking
- B1 —
verifycannot tell a working benchmark from a self-answering one. Fixed, both halves you suggested.min(rewards) >= 1.0is nowproblems.append, not a note; and a degenerate-scorer probe runsscore()on a synthetically correct rollout (output=str(task.target)) versus a deliberately wrong one and requires the rewards to differ on ≥1 task. One deviation from your sketch worth flagging: I first compared wrong-vs-the-seed-rollout as written, and it false-positived on the good bundledtoy_calc— the seed capability is already wrong, so both scored 0.0 and a healthy benchmark failed. Correct-vs-wrong is a property of the scorer alone, so it holds at any baseline. Opt-out for a genuinely saturated reference fixture:allow_saturated_baseline: true, which downgrades to a loud note naming the cost ("NO headroom… only usable as a regression fixture"). Closes attacks 1, 2, 3. - B2 —
score()silently overrides the declared mode. Fixed as you proposed: an undeclaredscore()is a hard error at module load, naming both remedies. Theif callable(custom)fallthrough inManifestAdapter.scoreis gone, so reaching the declared-mode branch now proves the declared mode is the effective one — which is what makesbenchmark list'sscoringcolumn honest. Closes attack 12. - B3 — forged / stale stamp. Fixed.
stamp_state()requires the fieldsverifyactually writes (a hand-typed{"ok": true}has nosteps) and re-hashes;index()reportsverified: falsewithstale: trueand astale_reason. Per your suggestion I also stamptarget_sha256andmanifest_sha256— the grader changing invalidates the evidence as much as the data. Closes attacks 7, 8. - B4 — path escape / arbitrary code execution. Fixed with an allowlist, no denylist.
_contained()requires (a) a plain relative path — not absolute, no drive, no~, no..component — and (b) the resolved parent inside the resolved root, which is what catches a symlinked subdirectory that a string check cannot. Called once fromload_manifest, the single choke point every caller routes through, for all four path keys. Same shape as GEPA eval-cache hits now carry output/trace (no more hollow reflective dataset) #210'sgepa.py:232,237. Closes attacks 5, 6. - B5 — wrong artifact. Fixed exactly as you specified: the assertion is on the set
protect.resolve_protected()resolves from the generatedcapevolve.yaml, and the manifest is only the source. As you predicted, this also subsumes the old string check. There is a fallback resolver for a pre-Protected-paths tamper guard: verify the optimizer never edited scoring/eval/task files #142 checkout (the branches merge in either order) so the step is never silently skipped — a guard that quietly does nothing being the exact failure mode this review found. Closes attack 11. - B6 — zero holdout. Fixed on the realized split:
test & (train|val),train & val, and an empty train are allproblems.append, andsplit_ids_filebuilds a realSplits(...)instead oftype("S", (), ...). Closes attacks 9, 10. - B7 — YAML injection. Fixed via
json.dumps(YAML's double-quoted style is a superset of JSON string syntax, so this is a valid scalar and needs no dumper dependency).nameadditionally rejects newlines, since it is interpolated into generated Python, where a broken value emits broken code rather than broken config. Proof below.
Non-blocking
- N1 —
tasks(split)ignoressplit. Fixed by honouring it, not raising: I tried raising first and it brokecheck.py:115, which callstasks("val")— a good reminder that the sibling caller matters._splits()derives the manifest's declared partition;harnessstill asks for"all"and filters by frozen ids, so nothing re-splits mid-run. Test assertstasks("test")is neither the full dataset nor overlapping train/val. - N2 —
--refreshclobbers a hand-edited shim. Fixed, and this is my answer to the structural criticism (below). A shim whose bytes differ from the generated text is treated as an authored override and left alone;capevolve.yamlis still re-derived, since it holds no logic to override.addreportskept_hand_editedso a stale override is visible rather than silent. - N3 — unanchored
regex. Declining the anchor change, documenting instead —re.fullmatchwould silently change the meaning of every existingregextarget, andexactalready covers equality. Thematch()docstring anddocs/BENCHMARK_ZOO.mdnow state the target is an unanchored pattern with the7/17example. I did take the other half: every target isre.compile-validated intasks(), so a bad pattern is a dataset error naming the line, not a smoke-eval crash. - N4 — fixed absolute tolerance + first-number. Partially fixed:
math.isclose(rel_tol=1e-9, abs_tol=1e-12)(relative, with a small floor so near-zero still compares) and_numnow accepts scientific notation. Declining atolerance:manifest key — no benchmark needs it yet, and an unused config knob in an honesty-critical file is a liability. The first-number semantics and the"in 2024 the answer is 7"case are documented exactly. - N5 — empty target scores 1.0. Fixed in
tasks(): an absent/empty target under a non-custommode is aBenchmarkErrornaming the line and why ("scores 1.0 for free undercontains/regex"). - N6 — 1-second determinism granularity. Fixed as a claim, not a mechanism: the module docstring and the docs now say the two back-to-back passes catch an unseeded sampler and are "a necessary condition, not a proof of reproducibility". Making it a real multi-scale check would mean sleeping in
verify, which is a bad trade. Thanks for confirming thecost_usd/tokensexclusion. - N7 — missing trailing newline. Fixed.
Nits
zoo.py:69mid-modulefrom . import splits as _splits+noqa: E402— fixed, moved to the import block next tomake_splits,noqagone.zooin__init__on the hot path — declining.stamp()does__import__("cap_evolve").__version__and the generated shim importscap_evolve.zoo, so removing it risks a circular-import regression for a saving I have not measured. Worth its own change with a measurement, not a drive-by here.type("S", (), {...})()→ realSplits(...)— fixed (part of B6).cli.py:104strip("./").count("/")→ fixed, now"/" not in args.name.benchmark add ../../../../tmp/x— fixed, a..component is refused with a message; stdout stays one JSON object and exit is 1.
The structural criticism: 562 lines with no override path
Taken, and it is the fairest point in the review — a generated-code win with no supported override is how these turn into 3am debugging. Concretely fixed at N2: --refresh now byte-compares adapters/adapter.py against the generated shim and keeps an authored override, reporting kept_hand_edited in the JSON. So "I need to override one CapabilityAdapter hook" has an answer that survives the next manifest edit, documented under a new Overriding a generated file heading.
I did not add a plugin/hook system. The shim is a real Python subclass — overriding a method there is already the language's own extension mechanism, and the only thing broken was the tool deleting it. Removing the clobber is the whole fix; an override registry would be the abstraction the review is rightly warning about.
Correction to my prediction
You are right and I was wrong: #137 is 5 conflict hunks in cli.py, not the "small textual conflict in main()" I wrote. Re-measured on the pushed commit:
$ git merge --no-commit --no-ff origin/feat/issue-137-cli-ergonomics
Auto-merging core/cap_evolve/cli.py
CONFLICT (content): Merge conflict in core/cap_evolve/cli.py
Automatic merge failed; fix conflicts and then commit the result.
$ grep -c '^<<<<<<<' core/cap_evolve/cli.py
5
Adopting your resolution: take #137's side everywhere, keep the COMMANDS row, and convert _cmd_benchmark to _parser(). Merge order unchanged (#197 → #195 → #137).
YAML injection is inert
$ cap-evolve benchmark add b --description $'oops
scoring: contains
tasks_file: /etc/hosts
verified: true
split_test: 0.0'
# resulting benchmark.yaml (head):
# cap-evolve benchmark manifest — the DECLARATIVE half of a benchmark.
# The only code is target.py's run(task, ctx, *, seed=0). Flat keys only, so the
# zero-dependency spec reader can parse it.
name: "b"
description: "oops\nscoring: contains\ntasks_file: /etc/hosts\nverified: true\nsplit_test: 0.0"
# parsed manifest — every injected key is INERT:
{
"scoring": "exact",
"tasks_file": "tasks.jsonl",
"verified": false,
"split_test": 0.25,
"description": "oops\nscoring: contains\ntasks_file: /etc/hosts\nverified: true\nsplit_test: 0.0"
}
The description itself survives intact — quoting, not stripping.
Re-proved what already worked
Both real e2e runs, the four original breakage cases, non-determinism detection and the seal are all still green, as suite tests rather than pasted transcripts (they were already tests):
core/tests/test_benchmark_zoo.py::test_manifest_rejects_unknown_key PASSED [ 2%]
core/tests/test_benchmark_zoo.py::test_manifest_rejects_bad_scoring_and_direction PASSED [ 4%]
core/tests/test_benchmark_zoo.py::test_builtin_predicates PASSED [ 7%]
core/tests/test_benchmark_zoo.py::test_add_scaffolds_a_benchmark_that_verifies PASSED [ 9%]
core/tests/test_benchmark_zoo.py::test_add_refuses_to_clobber_and_refresh_regenerates PASSED [ 11%]
core/tests/test_benchmark_zoo.py::test_generated_spec_declares_the_protected_paths PASSED [ 14%]
core/tests/test_benchmark_zoo.py::test_zoo_index_reads_the_stamp_from_disk_not_the_manifest_flag PASSED [ 16%]
core/tests/test_benchmark_zoo.py::test_bundled_zoo_entry_verifies PASSED [ 19%]
core/tests/test_benchmark_zoo.py::test_verify_runs_the_real_thing_not_just_the_manifest PASSED [ 21%]
core/tests/test_benchmark_zoo.py::test_verify_catches_a_stubbed_score PASSED [ 23%]
core/tests/test_benchmark_zoo.py::test_verify_catches_a_nondeterministic_run_target PASSED [ 26%]
core/tests/test_benchmark_zoo.py::test_verify_refuses_a_tiny_dataset_before_the_gate_can_surprise_anyone PASSED [ 28%]
core/tests/test_benchmark_zoo.py::test_verify_catches_a_missing_dataset_file PASSED [ 30%]
core/tests/test_benchmark_zoo.py::test_verify_catches_duplicate_task_ids PASSED [ 33%]
core/tests/test_benchmark_zoo.py::test_verify_checks_the_generated_spec_not_the_manifest PASSED [ 35%]
core/tests/test_benchmark_zoo.py::test_protection_is_additive_not_replacing PASSED [ 38%]
core/tests/test_benchmark_zoo.py::test_verify_flags_a_third_authors_new_files PASSED [ 40%]
core/tests/test_benchmark_zoo.py::test_verify_fails_a_saturated_baseline_and_allows_a_declared_opt_out PASSED [ 42%]
core/tests/test_benchmark_zoo.py::test_verify_catches_a_constant_scorer PASSED [ 45%]
core/tests/test_benchmark_zoo.py::test_undeclared_custom_scorer_is_a_hard_error PASSED [ 47%]
core/tests/test_benchmark_zoo.py::test_path_fields_are_contained_in_the_project_dir[target_module-../../pwned.py] PASSED [ 50%]
core/tests/test_benchmark_zoo.py::test_path_fields_are_contained_in_the_project_dir[tasks_file-../../evil.jsonl] PASSED [ 52%]
core/tests/test_benchmark_zoo.py::test_path_fields_are_contained_in_the_project_dir[capability_path-../../elsewhere] PASSED [ 54%]
core/tests/test_benchmark_zoo.py::test_path_fields_are_contained_in_the_project_dir[split_ids_file-/etc/passwd] PASSED [ 57%]
core/tests/test_benchmark_zoo.py::test_target_module_escape_never_imports PASSED [ 59%]
core/tests/test_benchmark_zoo.py::test_zero_holdout_and_empty_train_are_refused PASSED [ 61%]
core/tests/test_benchmark_zoo.py::test_content_duplicate_tasks_are_refused PASSED [ 64%]
core/tests/test_benchmark_zoo.py::test_forged_and_stale_stamps_read_as_unverified PASSED [ 66%]
core/tests/test_benchmark_zoo.py::test_description_newline_cannot_redefine_manifest_keys PASSED [ 69%]
core/tests/test_benchmark_zoo.py::test_refresh_keeps_a_hand_edited_adapter_shim PASSED [ 71%]
core/tests/test_benchmark_zoo.py::test_tasks_honours_the_split_argument PASSED [ 73%]
core/tests/test_benchmark_zoo.py::test_empty_target_is_not_a_free_point PASSED [ 76%]
core/tests/test_benchmark_zoo.py::test_bad_regex_target_fails_at_load PASSED [ 78%]
core/tests/test_benchmark_zoo.py::test_stamp_records_measured_evidence PASSED [ 80%]
core/tests/test_benchmark_zoo.py::test_cli_help_lists_benchmark_from_the_generated_listing PASSED [ 83%]
core/tests/test_benchmark_zoo.py::test_cli_benchmark_add_verify_roundtrip_stdout_is_one_json_object PASSED [ 85%]
core/tests/test_benchmark_zoo.py::test_cli_benchmark_verify_exits_nonzero_on_a_broken_benchmark PASSED [ 88%]
core/tests/test_benchmark_zoo.py::test_cli_benchmark_list_shows_the_bundled_zoo PASSED [ 90%]
core/tests/test_benchmark_zoo.py::test_cli_benchmark_error_path_still_prints_one_json_object PASSED [ 92%]
core/tests/test_benchmark_zoo.py::test_zoo_benchmark_runs_end_to_end_to_a_sealed_test_number PASSED [ 95%]
core/tests/test_benchmark_zoo.py::test_custom_scoring_entry_gives_graded_partial_credit PASSED [ 97%]
core/tests/test_benchmark_zoo.py::test_every_bundled_zoo_entry_verifies_and_is_stamped PASSED [100%]
============================== 42 passed in 1.88s ==============================
test_zoo_benchmark_runs_end_to_end_to_a_sealed_test_number—toy_calc: baseline val0.0→ gate-accepted → sealedtest.reward == 1.0, and a secondfinalizeraisesTestSealError.test_custom_scoring_entry_gives_graded_partial_credit—json_extract: prose0.0,[JSON]⅓,[JSON]+[FIELDS]1.0.- The four original breakage cases:
test_verify_catches_a_stubbed_score,test_verify_catches_a_nondeterministic_run_target(plantedrandom, message still namesseed=0andforward \seed`),test_verify_refuses_a_tiny_dataset_before_the_gate_can_surprise_anyone,test_verify_catches_a_missing_dataset_file`. Plus duplicate ids. - Seal untouched:
verifyruns on val only, creates no run dir, writes no test ids.
Both bundled entries still verify clean and are re-stamped (their capevolve.yaml gained the unioned globs and their stamps gained the two new hashes):
cap-evolve benchmark verify toy_calc — the full executed step list
{
"name": "toy_calc",
"ok": true,
"steps": [
"manifest parsed + validated",
"dataset loaded through the adapter: 8 task(s)",
"cap-evolve check executed on the generated project",
"splits computed: {'train': 4, 'val': 2, 'test': 2}",
"REAL smoke eval: 2 val task(s) x 2 passes through live() -> run_target() -> score()",
"degenerate-scorer probe: 2 task(s) scored with a correct vs a deliberately-wrong output",
"under-declaration sweep: every .py and answer-key-ish file under project/ (outside seed_capability/) is guard-hashed",
"protected paths resolved from the generated spec: ['adapters/adapter.py', 'benchmark.yaml', 'capevolve.yaml', 'target.py', 'tasks.jsonl']"
],
"problems": [],
"notes": [
"check: tasks('val') -> 2 task(s)",
"check: scorer deterministic (probe reward=0.0000)",
"check: materialize() callable (dry-run into temp copy; host untouched)",
"smoke val reward (seed capability) = 0.0",
"scorer discriminates on 2/2 smoke task(s) (a correct output outscores a deliberately wrong one)"
],
"val_reward": 0.0,
"n_tasks": 8,
"splits": {
"train": 4,
"val": 2,
"test": 2
},
"protected": [
"adapters/adapter.py",
"benchmark.yaml",
"capevolve.yaml",
"target.py",
"tasks.jsonl"
]
}
cap-evolve benchmark list
{
"zoo": "/private/tmp/fx-233b/benchmarks",
"benchmarks": [
{
"name": "json_extract",
"dir": "/private/tmp/fx-233b/benchmarks/json_extract",
"description": "Structured-JSON extraction accuracy (per-field partial credit) for a deterministic zero-API extractor whose prompt is optimized.",
"scoring": "custom",
"metric_direction": "higher",
"verified": true,
"verified_at": "2026-07-30T14:08:15+00:00",
"smoke_val_reward": 0.0,
"n_tasks": 12
},
{
"name": "toy_calc",
"dir": "/private/tmp/fx-233b/benchmarks/toy_calc",
"description": "Arithmetic accuracy of a deterministic zero-API stand-in agent whose system prompt is optimized.",
"scoring": "exact",
"metric_direction": "higher",
"verified": true,
"verified_at": "2026-07-30T14:08:15+00:00",
"smoke_val_reward": 0.0,
"n_tasks": 8
}
]
}
Suite + compileall
E ? +
core/tests/test_dashboard_launch.py:56: AssertionError
=========================== short test summary info ============================
FAILED core/tests/test_dashboard_launch.py::test_maybe_launch_spawns_when_available
1 failed, 220 passed in 65.96s (0:01:05)
$ compileall
exit=0
$ python -m cap_evolve.zoo
zoo predicate self-check: OK
220 passed (baseline 203 + 17 new), 0 failed other than test_dashboard_launch.py::test_maybe_launch_spawns_when_available, which is the known #200 flake — port 7878 was occupied on this box (lsof -ti:7878 → 40333), so maybe_launch picked 7882. Confirmed pre-existing by stashing every change on this commit and re-running that file alone: same single failure. Excluding just that file: 214 passed.
compileall on core + benchmarks: exit 0, silent. skills/ untouched, so no manifest rebuild.
Files touched
CHANGELOG.md | 33 ++
benchmarks/json_extract/project/capevolve.yaml | 2 +-
benchmarks/json_extract/verified.json | 10 +-
benchmarks/toy_calc/project/capevolve.yaml | 2 +-
benchmarks/toy_calc/verified.json | 10 +-
core/cap_evolve/cli.py | 7 +-
core/cap_evolve/zoo.py | 719 ++++++++++++++++++++++---
core/tests/test_benchmark_zoo.py | 290 +++++++++-
docs/ADAPTER_CONTRACT.md | 2 +-
docs/BENCHMARK_ZOO.md | 100 +++-
10 files changed, 1078 insertions(+), 97 deletions(-)
…s-ref The anti-rot assertion was `"Honest limits" in project_md`, which matched an incidental cross-reference earlier in PROJECT.md — so deleting the whole section (all 5 caveats) or renaming the heading both left the test green. It now anchors on `## Honest limits`, slices the section body, asserts a distinctive phrase per caveat, and pins the count at 5. Also: - PROJECT.md: qualify "check does not run run_target" — check.py:178-182 does, behind CAPEVOLVE_CHECK_TRIAL_PROBE=1 and CAPEVOLVE_N_TRIALS>1 (warn-only). - PROJECT.md: name LOW_CONFIDENCE_VAL_TASKS = 5 instead of "the threshold". - PROJECT.md / capevolve.yaml: quote the stable prefix of the gate transcript, since #195 appends a suffix after `n=2`. - README.md: forward link to benchmarks/toy_calc/ (the declarative form), making the #233 cross-reference bidirectional.
B1 (blocking). `CHEAP_REAL_MODEL=claude-haiku-4-5` — the fallback offered in three
docs for users with no local model — produced `preflight: OK`, exit 0, empty stderr
and `test_reward: 0.0` on every task. Two causes, both fixed in run.sh:
* the bundled adapter forwards `seed=`, which Anthropic rejects
(`UnsupportedParamsError`); the adapter correctly turns that into reward 0.0, so
the whole rung scored zero behind a clean-looking run. `export LITELLM_DROP_PARAMS=1`.
* every preflight check sat inside `if MODEL.startswith("ollama/")`, so the one
documented path nobody had exercised was the one path the preflight did not cover —
and it printed OK, actively vouching for the dead run. The hosted branch now does one
real 1-token completion through `model_config.llm_kwargs()` and exits non-zero on any
failure, with the endpoint and every credential redacted from the message.
Secondary trap on the same path: `API_BASE=localhost:11434` was exported
unconditionally and `model_config.py:95` reads the generic `API_BASE` before any
provider special-casing, so a hosted MODEL inherited a localhost endpoint. Now exported
only for an `ollama/` MODEL.
MEASURED, hosted variant, run for real: 98 s, $0.011478, 30 metric_calls,
baseline_val 0.0 -> sealed test_reward 1.0. The six eval costs sum to the total
exactly. That retires the prior $0.21 ESTIMATE, which was 20x high because
pricing.py assumes a 3000-in/800-out rollout and a one-line date is far smaller.
Review findings also addressed:
* Dropped "real paired gate" (5x) for "real accept/reject decision", and added an
Honest-limits entry stating plainly that this run never rejects on merit: val
saturates at 1.0 on iteration 1, so the two later rejections are against a ceiling,
and at num_trials 1 the decision is Δ > 0, not a significance test.
* Three progress lines moved to stderr (#116), and the single-object stdout contract no
longer depends on `--dashboard off` defaulting: run.sh filters the CLI's stdout to the
last object, so `json.loads(stdout)` works under `CAPEVOLVE_DASHBOARD=auto` too (#217).
* Corrected the #197 wording. `tasks.jsonl` matches NO default glob; the answer key is
covered by the `dataset_source` fold-in at protect.py:169, which only happens when
`protected_paths` is omitted. Declaring the key drops it — verified.
* Corrected the call-count comment in the preset (15 -> 30, with the derivation) and
the stale CHEAP_REAL_MAX_USD default in run.sh's header (2.0 -> 3.0).
* Replaced the "36 s" figure, which appeared in neither measurement, with "~30 s" and
the observed spread (30.4 / 33.9 / 34.2 s).
* Softened the zoo line: the determinism guarantee question is deferred, not settled —
named the `determinism: exact|statistical|none` grade #233 will need.
* Noted the pass^k `{"2": 0.0}` artifact as pre-existing #112, and that the local
rung's $0 is a metering artifact (the runner's cost there is time).
* Two new tests: the run.sh YAML-rewrite heredoc over the committed preset (it is a
line-prefix rewriter, so a no-op means the paid rung runs `mock`), and the measured
`optimizer_max_turns >= 40` floor. Both verified to fail on a broken invariant.
Declined, filed instead: the relative `optimizer_instructions_file` bug is #252 (the
absolute-path workaround stays, with the `check`-passes-what-`run`-ignores disagreement
named in the comment); the optimizer stderr discarded on success is #251 — the reason a
warning alone would not have surfaced this bug.
Closes #141
What this does
Onboarding a benchmark was a from-scratch, per-user effort: hand-write a
CapabilityAdaptersubclass, prune a ~100-linecapevolve.yaml, author a bespokeoptimizer/INSTRUCTIONS.md. This adds the three pieces #141 asks for — a curated verifier-gated library (benchmarks/), acap-evolve benchmark add|verify|listsubcommand, and a declarative manifest — plus the docs.The boilerplate, measured first
I diffed the two generic bundled templates (
templates/adapters/jsonl_litellmvshuggingface_litellm): 78 changed lines out of 127, and the changes are almost entirely the dataset-loading block.sys.pathjugglingTaskloopif rollout.error:infra-noise branch ofscoreexact/contains/regexmatch helperScore(task_id=…, reward=…, feedback=…, trial_rewards=[…])capevolve.yamlexcept ~6 valuesMeasured reduction — same benchmark (
toy_calc), hand-authored non-blank linesadapter.py: 3 methods +apply)target.py: onerun())capevolve.yamlfrom the template)benchmark.yaml)adapters/adapter.pycapevolve.yamlFor the documented generic LLM case (
jsonl_litellm) the "before" is 101 (88 Python + 13 YAML) against the same 36.Declarative vs code (and why)
benchmark.yamldeclares: dataset file + field mapping, scoring mode, metric direction, capability path, split seed/ratios/pinned ids, trial count, protected paths. An unknown key is a hard error — a silently-ignored key in an honesty-critical config is exactly how "I declared it" and "it applied" drift apart.target.pystays code:run(task, ctx, *, seed=0)— how the target agent runs. There is deliberately no config language for it. Running an agent is real logic; a DSL reimplementing Python would be worse than the Python it replaced, and the repo's norm is lazy-correct with zero runtime deps. Same reasoning forscoring: custom+score(task, rollout): the newjson_extractentry does per-field partial credit over parsed JSON, which no config key should try to express. The five built-in modes (exact/contains/regex/numeric/custom) cover the rest.ManifestAdapterthen is the adapter, soadapters/adapter.pyis a generated 3-line subclass and the contract is satisfied without the user seeing it.Exactly what
verifyexecutesNot a manifest parse. Every guard in this epic that measured the wrong artifact passed its own test vacuously (#189 counts, #202 chart DOM, #207
npm test, #208install.sh, #213 frontmatter), soverifyruns the benchmark:cap_evolve.check.run_checkon the generated project (stubs, task stability, scorer determinism, purematerialize);val >= MIN_VAL_TASKS(read fromsplitswhen Guard tiny/empty val splits and add a small-sample correction to the paired gate #113 lands, same literal otherwise) and a non-empty sealed test split — a 3-task dataset fails here, not mid-run insidegate.decide;live()→run_target()→score(), twice, comparing rollout fingerprints and rewards. This is the step that catches a non-deterministicrun_target()—checknever runs the target at all;verified.jsonrecords the measured val reward, split sizes, dataset SHA-256 and the ordered step list.benchmark listreads that stamp from disk: a committedverified:flag is a claim, the stamp is evidence (there's a test that a flag flipped totruewith no stamp still reads unverified).Note on reusing
check: the review that found a raisingmaterialize()still yieldsok: trueis why steps 4–6 exist independently rather than trustingcheckalone — and whyverifyalso asserts on the smoke outcome, not justok.Protected paths / #142
A zoo entry keeps the manifest, scorer and dataset inside
project/(which is the cap-evolve project dir). That is not cosmetic: my first pass parked them at the benchmark root, and #197'sresolve_protected()hashed onlyadapters/adapter.py— a declared-but-unprotected grader, caught by actually running the guard rather than reading it. With the current layout the guard resolves all four declared paths and a tamperedtarget.pyraisesTamperError. Evidence in the comment below.CLI
cap-evolve --helpnow generates the subcommand listing fromCOMMANDS+ each handler's first docstring line (a test asserts the old literalversion|splits|check|run|estimate|dashboardstring is gone). This is the same shape as #137/#214, so addingbenchmarkneeded no manual edit to a string five branches conflict on. Each handler owns its own--help; stdout stays exactly one JSON object on every path, including errors ({"ok": false, "error": …}), and exit is 1 on any problem.Expected merge order
protected_paths, andverifystep 6 importsprotectwhen present (gracefulImportErrornote otherwise, so it merges in either order).fix/issue-113-small-samples—MIN_VAL_TASKSis read fromsplitswhen present, so after that merge the verify floor andgate.decideshare one constant automatically. No conflict either way.feat/issue-137-cli-ergonomics— both generate the listing fromCOMMANDS; expect a small textual conflict inmain()/the module docstring, resolvable by keeping CLI ergonomics overhaul: structured flags, did-you-mean, per-subcommand help, UTF-8/exit codes #137's_parser()helper and this PR'sCOMMANDSrow.No overlap with #199/#191/#193/#190/#205 beyond the
COMMANDSdict row.Verification
Full output in the 🔬 Evidence comment. Summary:
203 passed in 60.94s(baseline 179 + 24 new), 0 failed.compilealloncore+benchmarks: clean (exit 0). Noskills/changes, so no manifest rebuild.toy_calc:baseline_val 0.0 → test_reward 1.0,test_delta 1.0, gate-accepted, test sealed.json_extract(custom scoring):baseline_val 0.0 → test_reward 1.0, and a graded signal — prose 0.0,[JSON]1/3,[JSON]+[FIELDS]1.0.benchmark add demo→verify demo→ok: truewith no hand-editing.score(), non-deterministicrun_target(), 3-task dataset (tiny val + empty test), missing dataset file. Plus duplicate task ids and under-declaredprotected_paths.['adapters/adapter.py', 'benchmark.yaml', 'target.py', 'tasks.jsonl']; clean verify[]; tamperingtarget.py→TamperError: … modified target.py.cap-evolve --helplistsbenchmarkfrom the generated listing;benchmark --helprenders its own subparsers.