Ship a filled, runnable worked project reference (the scaffold #108 describes is gitignored, not blank) - #244
Ship a filled, runnable worked project reference (the scaffold #108 describes is gitignored, not blank)#244OsherElhadad wants to merge 2 commits into
Conversation
Issue #108 says the checked-in `.capevolve/project/` scaffold is a blank template. It is not — it does not exist. `.capevolve/` is gitignored and no path under it has ever been tracked in this repo's history. The scaffold the issue describes is a LOCAL artifact intake writes; the repo cannot ship a filled version of a path it does not ship at all, and committing one would be committing an ignored run dir. The real gap the issue is pointing at survives that correction: the repo ships `templates/project/` (a blank form) and ZERO filled `PROJECT.md` or commented `capevolve.yaml` anywhere. `examples/toy_calc/` did not even have a spec — `run.sh` copied the blank template. So the first concrete artifact a user meets after intake had no worked counterpart to read. That is what this fills, in the example dir the docs already point at, rather than by resurrecting an ignored path. Added, both genuinely filled: - `examples/toy_calc/PROJECT.md` — what is optimized, how the runner works, how the scorer works and what its feedback says, the data and splits, the optimizer and algorithm, the budget, and which RECOMMENDED inputs were skipped and why. - `examples/toy_calc/capevolve.yaml` — the run spec with a note on WHY for every decision, not a copy of the template's option list. `run.sh` now uses that spec instead of `templates/project/capevolve.yaml`, and runs `cap-evolve check` before the pipeline, so the documented hard gate is exercised rather than described. Both docs LEAD with what the example does not prove, because that is what gets over-read: - `gate_mode: paired` / `gate_k_se: 1.0` are set but never tested here. Both val tasks move 0 -> 1 together, so the spread of the per-task deltas is zero, SE(delta) = 0, and the gate logs a `gate_warning` and takes its documented STRICT fallback. The run says so literally: `paired delta=+1.0000 > 0 (SE=0 -> STRICT fallback, warned; n=2)`. - At `k_se = 1.0`, improving exactly ONE of n val tasks gives mean(delta) = SE(delta) exactly, so the strict `>` rejects it — at every n, and however large that one gain is. Improve >= 2 tasks or bank nothing. - `val = 2` equals the harness floor (MIN_VAL_TASKS), not a recommendation. - `cap-evolve check` passing proves less than it looks: it never calls `run_target`, so a non-deterministic runner passes, and a raising `materialize()` can still yield `{"ok": true}`. - The 0.0 -> 1.0 improvement is engineered by construction. `protected_paths` is deliberately OMITTED, with a comment saying why: an empty list is a HARD ERROR, never a "use defaults" shorthand. The adapter description matches the corrected contract — three required abstract methods (`tasks`, `run_target`, `score`) plus an override of the optional `apply` hook, not "4 methods". Anti-rot: two tests in `core/tests/test_e2e_slice.py` drive the COMMITTED files through the real CLI — `cap-evolve check` -> a full zero-API `cap-evolve run` -> the sealed test number — and assert the spec rules (no declared `protected_paths`, val >= 2, no template placeholder token still present, no pre-rebrand name) and that the gate really does log the `gate_warning` the docs claim. The placeholder check derives its token list from the template at test time, so it cannot go stale against it. A contract change now breaks a test instead of quietly rotting the prose. `templates/project/PROJECT.md` gains one line pointing at the filled example. Its H1 is left untouched so the rebrand PR owns that line with no conflict.
🔬 EvidenceEvery command, verbatim, with full output. Env: macOS, Python 3.14, 0. The correction —
|
|
❌ Automatic Labeling Failed An error occurred while trying to automatically label this pull request. Please check the workflow logs for details and add labels manually. |
🔍 Review — PR #244APPROVE WITH NITS. The premise rejection is correct and evidenced. The contract claims all match their sibling branches' ground truth — I checked each one against the unmerged branch, not against Nothing blocking. BlockingNone. Non-blocking
Nits
Is the premise rejection right?Yes, on all three facts, and I verified each independently: So the issue's factual claim — "the scaffolded example project that ships in the repo, Is filling Does the epic instead want a Is this additive over #233's zoo, or should it be closed?Additive — keep it. But the margin is thinner than the PR's framing implies, and the honest reason to keep it is not the one the PR leads with. I read
The decisive point, though, is that #233's own README asks for this PR to exist: " Now the honest deduction. The PR says "the only thing lost would be the filled
Verdict: additive, keep open. ~370 lines for the repo's only filled reference plus two tests that pin four contract rules is a fair trade, and the alternative leaves a dangling promise in #233. This is not manufactured work. Contract accuracyEvery claim checked against the sibling branch, not
Anti-rot testsReal, with one hole. I broke each asserted property and re-ran. 7 of 8 failed as they must:
The placeholder list is genuinely derived at test time — verified, not taken on trust. Probe 8 is the proof: The subprocess test is honest work: Merge-order noteTrial-merged this branch against all six siblings: #183 auto-merges clean, as the PR claims. Verified the geometry: #183's only hunk is One conflict the PR does not mention. #181 also edits Trivial to resolve — take HEAD (this PR's version is the more specific and equally correct one; it names the three methods and labels Recommended order: Verification I re-ranTests — exactly +2, as claimed: No flakes;
No third copy of toy_calc —
The gate really does degrade to STRICT — reproduced from Three Is the degradation a teaching hazard or a strength here? A strength. The disclosure is adequate and then some — it is the lead item of Honest limits ( #197 control — both halves reproduce on So #195 control on
Raising
Net: every verification claim in the PR description reproduced. The only corrections are the |
…s-ref The anti-rot assertion was `"Honest limits" in project_md`, which matched an incidental cross-reference earlier in PROJECT.md — so deleting the whole section (all 5 caveats) or renaming the heading both left the test green. It now anchors on `## Honest limits`, slices the section body, asserts a distinctive phrase per caveat, and pins the count at 5. Also: - PROJECT.md: qualify "check does not run run_target" — check.py:178-182 does, behind CAPEVOLVE_CHECK_TRIAL_PROBE=1 and CAPEVOLVE_N_TRIALS>1 (warn-only). - PROJECT.md: name LOW_CONFIDENCE_VAL_TASKS = 5 instead of "the threshold". - PROJECT.md / capevolve.yaml: quote the stable prefix of the gate transcript, since #195 appends a suffix after `n=2`. - README.md: forward link to benchmarks/toy_calc/ (the declarative form), making the #233 cross-reference bidirectional.
🔧 Review fixesCommit Finding 1 — the Honest limits guard now asserts the content [FIXED]You were right and the hole was exactly where you said. assert "\n## Honest limits" in project_md, "the reference lost its `## Honest limits` heading"
limits = project_md.split("\n## Honest limits", 1)[1].split("\n## ", 1)[0]
for claim in ("STRICT fallback", "bank nothing", "MIN_VAL_TASKS", "run_target",
"engineered"):
assert claim in limits, f"Honest limits lost its `{claim}` caveat"
caveats = re.findall(r"^\d+\. \*\*", limits, flags=re.M)
assert len(caveats) == 5, f"Honest limits should list 5 caveats, found {len(caveats)}"The three break-probes you asked for — all three now FAIL. Note probe B: one Point taken on this being the seventh instance in the epic of a check matching the wrong The other 7 anti-rot properties — re-run, none weakenedSo it is now 10 of 10 broken properties detected (your 7, plus heading rename, Finding 2 — the
|
Closes #108
What #108 asked for, and the correction
The issue says the checked-in
.capevolve/project/scaffold is an untouched blanktemplate, and offers Option A (fill it) or Option B (delete it).
Neither is available: the path does not exist.
.capevolve/is gitignored(
.gitignore:# cap-evolve run state (per-run, not source)), and no path under it hasever been tracked:
So there is nothing to delete (Option B is already the status quo), and Option A as
literally written would mean committing an ignored run dir.
The real gap survives the correction. The repo ships
templates/project/— a blankform — and zero filled
PROJECT.mdor commentedcapevolve.yamlanywhere:examples/toy_calc/did not even have a spec:run.shcopiedtemplates/project/capevolve.yaml. So the artifact intake produces first had no workedcounterpart to read anywhere in the repo. That is the gap this fills, in the example dir
the docs already point at.
Judgement on #233's benchmark zoo — does it make this redundant?
Partly, and I scoped around the overlap rather than duplicating it.
The zoo genuinely supersedes the hand-written adapter teaching angle for benchmarks
that fit its manifest: it cuts 78 → 36 hand-authored lines, and
benchmark verifyis afar stronger correctness story than anything a doc can assert. If the question is "how do
I add a benchmark", the zoo is now the answer and this PR should not compete with it.
But the zoo's
benchmarks/toy_calc/project/capevolve.yamlisGENERATED from benchmark.yamland says so in its first line: "edit the manifest and re-runcap-evolve benchmark add --refresh, not this file." A generated file is the opposite ofa worked reference — it deliberately explains nothing about why a value was chosen, and it
carries no
PROJECT.mdat all. The zoo also covers only the declarative path; a projectwhose runner does not fit the manifest still writes a
CapabilityAdapterby hand and stillhas nothing filled to read.
So I scoped this to the gap the zoo does not close: the hand-written path, and the
commentary and honesty framing a generated file structurally cannot carry. No new
directory, no third copy of toy_calc — the two files land in the example that already
exists, and
examples/toy_calc/README.mdcross-links the zoo entry and says outright"Prefer the zoo when the manifest fits your benchmark." If a reviewer disagrees and
wants this closed as zoo-covered, the only thing lost is the filled
PROJECT.md, whichexists nowhere else.
What the reference demonstrates
Two genuinely filled files, plus the honesty framing:
examples/toy_calc/PROJECT.md— the filledtemplates/project/PROJECT.md: what isoptimized, how the runner works, how the scorer works and what its feedback says, the
data and splits, the optimizer and algorithm, the budget, and which RECOMMENDED inputs
were skipped and why.
examples/toy_calc/capevolve.yaml— the run spec with a note on why for everydecision, rather than a copy of the template's option list.
run.shnow uses that spec instead of the blank template, and runscap-evolve checkfirst, so the documented hard gate is exercised, not described.Both docs lead with what it does NOT prove
This is the part worth more than the parts that work, because it is what readers over-read:
gate_mode: paired,gate_k_se: 1.0are set, but both val tasks move0 → 1together, so the spread ofthe per-task deltas is zero,
SE(Δ) = 0, and the gate logs agate_warningand takesits documented STRICT fallback. Verbatim from the run:
paired Δ̄=+1.0000 > 0 (SE=0 → STRICT fallback, warned; n=2).k_se = 1.0is stricter than it looks. Improving exactly ONE ofnval tasksgives
mean(Δ) = SE(Δ)exactly, so the strict>rejects it — at everyn,and however large that single gain is. Improve ≥ 2 tasks at
k_se = 1.0, or banknothing.
val = 2is the harness floor (MIN_VAL_TASKS), not a recommendation — and stillunder the low-confidence threshold.
checkpassing proves less than it looks. It never callsrun_target, so anon-deterministic runner passes it; a raising
materialize()can still yield{"ok": true}.0.0 → 1.0improvement is engineered by construction.Correct against the current (unmerged) contracts
@abstractmethods (tasks,run_target,score) plus an override of the optionalapplyhook. Not "4methods". Matches
docs/ADAPTER_CONTRACT.mdonfix/issue-103-adapter-methodsline-for-line, and matches that branch's own edit to this README.
protected_paths— deliberately OMITTED, with a comment saying why: anempty list is a HARD ERROR, never a "use defaults" shorthand. Verified against
Protected-paths tamper guard: verify the optimizer never edited scoring/eval/task files #197's actual code (evidence below).
k_secap — the reference's splits clear the floor(
honest_gate: true) onfix/issue-113-small-samples, and it emits exactly theLOW CONFIDENCE warning
PROJECT.mddocuments.gate_k_se: 1.0is far under the 26.5 cap.k_se = 1.0caveat isstated in full, magnitude- and
n-independent, in both docs and the CHANGELOG.templates/project/PROJECT.mdgains one line at line 3 pointingat the filled example; the
AcapoH1 at line 1 is left untouched so fix: finish the Acapo -> cap-evolve rebrand (#105) #183 owns it.git merge origin/fix/issue-105-rebrand→ "Automatic merge went well."How this is prevented from rotting
Prose asserting a contract is exactly what this epic keeps finding stale (#203's
cap-evolve finalize,maybe_cached_score, "4 methods"). So the claims are tested, notwritten down:
core/tests/test_e2e_slice.pygains two tests that drive the committed files throughthe real CLI:
test_worked_reference_check_and_spec_are_honest— runs the realrun_check, thenasserts the spec rules: no declared
protected_paths(post-Protected-paths tamper guard: verify the optimizer never edited scoring/eval/task files #197),val ≥ 2(post-Guard tiny/empty val splits and add a Student-t small-sample correction to the paired gate #195), no pre-rebrand name, and no template placeholder token still present. The
placeholder list is derived from
templates/project/PROJECT.mdat test time, so itcannot go stale against the template it mirrors — and it fails loudly if the template
stops having placeholders at all.
test_worked_reference_runs_end_to_end— a real zero-APIcap-evolve runsubprocessthrough to the sealed test number, and asserts the gate actually logs the
gate_warningandSTRICT fallbackreason the docs claim. If the gate stops fallingback, the doc claim is stale and this test fails rather than the doc lying.
Expected merge order
Both are one-line-adjacent, not blocking, but land them first for a clean history:
fix/issue-105-rebrand) — ownstemplates/project/PROJECT.md's H1. Verifiedclean auto-merge with this branch.
feat/issue-142-protected-paths) — makesprotected_paths: []a hard error;this reference is already correct for it (omits the key). No conflict either way.
Neither is required for the tests here to pass on
maintoday.Verification
Every number below is pasted literal output. Full commands + transcripts in the
🔬 Evidence comment.
cap-evolve checkon the reference:A real zero-API
cap-evolve runwith themockoptimizer, to a sealed test number:Sealed test = 1.0 from
baseline_val 0.0.iterations: 3— the win lands oniteration 1, then
stall: 2ends the run, exactly ascapevolve.yamldocuments.The gate claim, from that same run's events (proving the honesty caveat is real):
Post-#197
protected_pathscorrectness (run onfeat/issue-142-protected-paths):Post-#195 split floor (run on
fix/issue-113-small-samples):Clears the
MIN_VAL_TASKS = 2floor, and the warning is exactly whatPROJECT.mddocuments as an honest limit.
run.shend to end (the CI gate path):CI's assertion greps
"baseline_val": 0.0+"test_reward": 1.0— both still present.Full test suite:
179 baseline → 181: exactly +2 (my two new tests), 0 failures on either side.
Earlier in the session the same command hit
test_dashboard_launch.py::test_maybe_launch_spawns_when_available— the known port-7878flake tracked in #200 — on the branch and on the unmodified baseline
(
1 failed, 180 passedvs1 failed, 178 passed: the same +2 with the same pre-existingfailure). It is timing-dependent and unrelated to this change.
compileall:Links: all 15 relative links in the two new/edited docs resolve (checked
programmatically; full output in the Evidence comment).
Files touched
examples/toy_calc/PROJECT.mdPROJECT.md, incl. Honest limitsexamples/toy_calc/capevolve.yamlexamples/toy_calc/run.shcap-evolve checkfirstexamples/toy_calc/README.mdcore/tests/test_e2e_slice.pytemplates/project/PROJECT.mdCHANGELOG.md