Reconcile results across README / docs/RESULTS.md / site: make RESULTS.md the superset, publish the 87-task runs everywhere, remove 8 unsubstantiated numbers - #241
Conversation
… runs everywhere with true provenance Issue #100: the source-of-truth relationship was inverted. site/results.html published runs docs/RESULTS.md did not have, while site/README.md told readers to "cross-check against docs/RESULTS.md" — an instruction that was false. Since the issue was filed the Qwen 14B runs landed in docs/RESULTS.md, and PR #236 added a SkillsBench 87-task optimization section there that the site never got. The divergence just moved rather than closing, so this fixes it structurally: - docs/RESULTS.md declares itself the SUPERSET, and a new guard test enforces it by section identity (core/tests/test_published_results_consistency.py). The guard fails if a site run section has no canonical counterpart — the exact #100 bug, verified by reintroducing it. - Adds the missing "SkillsBench — full 87-task baselines" section, which also fixes a dangling "the baseline section above" reference in the 87-task optimization section (it pointed at a section that was never written). - Publishes both 87-task sections on site/results.html with split labels, run-record links, and the⚠️ reported marker (#99/#182's per-section convention), plus TOC rows. - Labels every 87-task figure with its split: "val (87, fit)" and "test (87, fit metric — test == val by construction, not held out)". - docs/RESULTS.md held-out table now shows 0.567/0.300 (56.7%/30.0%) instead of bare 56.7/30.0, matching the site and the page's own units rule. Removes numbers nothing substantiates rather than keeping them: - "EvoSkills' 71.1%" — no citation in sources.bib, no source anywhere in the tree. - "Optimizer spend: ~$32 total (well below the $400 cap)" — no cost field in the run record; the sentence now says spend is not recorded. - presentation/ slide 9's four "Done" rows quoted baseline/optimized/Δ figures that exist on no other surface and have no artifact or run record at any commit, under a speaker note claiming they were "already committed". Numbers blanked to —, note corrected to say the write-ups are not published. Also removes the two blanket-verification claims #182 does not reach: site/benchmarks.html's "hand-verified canonical numbers" and site/README.md's "cross-check against docs/RESULTS.md and the committed run artifacts", the latter replaced with the actual superset rule.
|
❌ Automatic Labeling Failed An error occurred while trying to automatically label this pull request. Please check the workflow logs for details and add labels manually. |
🔬 EvidenceAll commands run in 1. Setup2. Artifacts that DO exist on disk (the ✅ rows)τ²-bench airline — every ✅ number, from the artifact: SkillsBench held-out — every ✅ number, from the artifact: 3. Artifacts that do NOT exist — the ❌ /
|
🔍 Review — PR #241Verdict: APPROVE WITH NITS. I independently re-ran every claim. All 8 removals are correct — I could not substantiate a single one at any of the 599 commits on any branch. All 9 survivors I spot-checked hit their cited artifact field exactly, with correct split labels. The guard is non-vacuous and reads from disk; it also caught a real regression I accidentally introduced during the #196 merge test (below). Nothing blocking. Two non-blocking gaps: the guard's invariant is one-directional (it would not have caught the #236 divergence that this PR is fixing), and the surviving aggregate claim is falsifiable as written. BlockingNone. Non-blocking1. The PR says "the divergence did not close, it swapped direction" — #236 added a Consequence: after merge, the next Fix: one assertion, no allowlist. Add a def test_every_canonical_run_section_is_published_on_the_site():
headings = re.findall(r"^## (.+)$", _read("docs/RESULTS.md"), re.M)
site_anchors = set(re.findall(r'<h2 id="([^"]+)"', _read("site/results.html")))
mapped = {v for k, v in ANCHOR_TO_CANON_HEADING.items() if v and k in site_anchors}
unpublished = [h for h in headings if not any(m in h for m in mapped)]
assert not unpublished, (
f"docs/RESULTS.md sections with no site/results.html section: {unpublished}\n"
"Add the site section, or add the heading to SITE_OPTIONAL with a reason."
)2. 3. Were the 8 removals right?Searched, for each:
Judgement on the presentation: deletion is right here, and the #192 Cross-check that the removals actually landed. Survivor spot-checksNine checked, against the artifact field cited (not the PR's transcription of it).
Plus the
Five τ² runs with no committed artifact — confirmed, exhaustively. For each of And no Qwen file was ever added: Split labels — all correct, including the two the epic previously caught. The Guard sufficiencyReads disk, not a cache. Non-vacuous — fails before, passes after. Reintroduced the original #100 bug (renamed the canonical Test 2 also bites — removing one TOC entry fails What it catches: a site section whose canonical heading is deleted or renamed (the explicit table beats a set-difference here — a rename fails loudly); a new site anchor with no canonical mapping; TOC/section drift; the two blanket claims on the surfaces it owns. Test 3 is not decoration: during my #196 merge test I resolved That is a real merge-regression class, not a hypothetical. What it misses (all six probed, all 3 tests passed):
Verdict on dropping the numeric scrape: correct, and I'd have made the same call. The stated reasons hold up under inspection — But the reason the scrape fails is scope, not the idea. Take the middle path you already named. Numbers inside a declared results table are unambiguous — no fonts, no SVG, no arXiv ids live in The surviving aggregate claim
The PR's defence is right about the reading, wrong about the fact. I agree a reader parses "must correspond" as normative, not as "all numbers are artifact-backed" — the sentence never says verified, and it sits directly above a paragraph saying uncommitted runs are But it is false as written on the merged tree, in two places I can point at:
(A third, softer one: Consequence: a reader who tests the rule finds a counter-example in one grep, on a page whose entire premise is that its claims survive checking. On an honesty PR that is the expensive kind of wrong. And note the guard does not enforce this scope — it reads only Fix — smallest edit that makes it true, scope it to what it governs: **This page is the superset for cap-evolve's own runs.** Every result claim about a
cap-evolve run on any other surface — `README.md`, `site/results.html`,
`site/index.html`, `OUTREACH.md`, `presentation/` — must correspond to a section here,
with the same value, the same split label, and the same evidence marker. If one of our
numbers is not on this page, it is not published. Externally reported results (other
papers' figures) live in [`COMPARISON.md`](COMPARISON.md) with their own citations and
are out of scope for this rule.That covers Coordination boundaries — did anything left behind contradict what changed?Verified untouched ( Contradiction sweep on the merged
#196 compatibilityBuilt Harbor failure — byte-identical in both trees, so pre-existing: With Edits confined to
|
…rs in mapped tables Review fixes for PR #241. 1. The guard's invariant was one-directional. It checked site -> doc, but the divergence this PR fixes (#236) ran doc -> site: a docs/RESULTS.md section the site never got. Deleting the site's skillsbench-87 section with the doc intact passed all 3 tests. Adds the reverse lookup over the existing mapping, with an explicit (currently empty) SITE_OPTIONAL escape hatch so a doc-only section has to be declared rather than appear silently. 2. Adds a numeric scrape confined to <td class="num"> / <td class="gain"> inside mapped sections, matched row-by-row against the same-labelled docs/RESULTS.md row. Editing a published number is a far more likely edit than deleting a section, and both were undetected. The confinement is what keeps the no-growing- allowlist principle: fonts, arXiv ids and SVG coordinates never live inside a results <td>, so this needs zero allowlist entries. Known limits (a new surface, a new results page, evidence-marker parity) are documented as limits rather than implied as coverage. That scrape immediately found a real divergence: site/results.html labelled the agent-mode row "sealed test (20, held-out)" where the canonical doc says "(20, held-out, scored once)". Fixed on the site, per the superset rule. 3. The superset claim was false as written: it named docs/COMPARISON.md and presentation/, both of which publish figures with no RESULTS.md section. Rewritten to scope it to cap-evolve's own run results, to say which part is mechanically enforced (the site-results <-> doc mapping, not the six-surface sentence), and to place external figures in COMPARISON.md explicitly rather than by omission. Reconciles COMPARISON.md's one in-scope row too: its cap-evolve held-out row now carries the same evidence marker its RESULTS.md section does. 4. Slide 9's row-4 null result was a real finding whose explanation lived only in a speaker note. The reasoning is now visible on the slide: what "-" means, and that row 4 was already at the scorer's ceiling on 5 easy tasks, so optimization had no headroom. A saturated benchmark stated in words, not a bare dash a reader can mistake for withheld data.
🔧 Review fixesAll 3 non-blocking findings fixed. New commit Guard probes
D/E/F are now stated as limits in the module docstring rather than left as implied coverage — they are section-registration problems, not consistency problems, and covering them means either a new-file watcher or an emoji-parity rule that would need the growing allowlist we're avoiding. Literal output — before (on After ( Finding 1b — the confined scrape needs zero allowlist entries, and found a real bugImplemented as you described: Two normalisation rules do the work an allowlist would otherwise do, and neither is an exception list:
Proof — zero false positives across all 9 mapped sections, zero allowlist entries: It immediately caught a real divergence in this PR's own branch, which is the best argument for it. Fixed on the site, per the superset rule (the doc is the superset, so the doc's label wins). That is exactly probe C's failure mode occurring for real, undetected until the scrape existed. Finding 2 — the rewritten aggregate sentence
Your paragraph, plus two additions: the tool-API-error-rates clause (so Grep proof. Every Same on the merged
|
| File | Change |
|---|---|
core/tests/test_published_results_consistency.py |
+2 tests (reverse direction, confined numeric scrape), SITE_OPTIONAL, 5 helpers, docstring rewrite incl. stated limits |
site/results.html |
sealed test (20, held-out) → (20, held-out, scored once) — divergence found by the new scrape |
docs/RESULTS.md |
superset paragraph rewritten (scoped + enforcement scope stated) |
docs/COMPARISON.md |
cap-evolve row gains its |
presentation/index.html |
slide-9 .exp-note explaining — and row 4's null result; .exp-note CSS; speaker note updated |
Closes #100
The problem, as it stands today
Issue #100 reported that the source-of-truth relationship was inverted:
site/results.htmlpublished two Qwen 2.5 14B runs (#qwen-tools,#qwen-all) thatdocs/RESULTS.mddid not contain, whilesite/README.mdtold readers to "cross-check againstdocs/RESULTS.md" — an instruction that was false.Since the issue was filed the Qwen runs landed in
docs/RESULTS.md, and merged PR #236 added aSkillsBench — full 87-task optimizationsection todocs/RESULTS.mdthat the site never got. The divergence did not close, it swapped direction. So this PR fixes it structurally rather than patching the specific rows the issue named.Full inventory — every result claim on every surface
Split column: fit =
train == val == test(no holdout) · sealed test = ids the optimizer never saw. Provenance is what actually exists on disk in this repo.Before
0.0 → 1.0core/tests/test_e2e_slice.py0.536run_full/ui/data/runs_run_full.json→baseline_val: 0.536cand_00070.712best_val: 0.712,best_id: cand_00070.694test == val, NOT held outrun_full/final.json→test.reward: 0.6940.584final.json→test.pass_k["2"]: 0.58440.582/0.634/0.670/0.684/0.712runs_run_full.json→evaluations[]+per_iteration[]statuses56.7 → 70.0(doc) /0.567 → 0.700(site)0.475in zero artifact files) — units also disagreed between surfaces30.0 → 47.5(+58.3%)0.500→0.633, test0.400→0.5500.544→0.644, test0.467→0.4000.567, sealed test0.350.200→0.387, test0.170→0.240(+41.2%)git log --allfinds no Qwen file ever added0.273→0.520, test0.120→0.270(+125.0%)docs/runs/only — on NO published surface0.281(23/87) ·0.0396(2/87)docs/RESULTS.md:177-238only — MISSING from the site0.281→0.357(+27.2%), pass@123→28test == val0.357 / 0.325 / 0.170:36-38(4-dp there, 3-dp in RESULTS)~$32 total,$400 cap71.1%sources.bib; string appears nowhere else in the tree0.333 → 0.714(+114%)skillsbench/run_full/baseline.json→val.reward: 0.33330.556 → 0.667(+20.0%)run_full/final.json→test_baseline.reward: 0.5556,test.reward: 0.6667presentation/index.html:964-9660.600 → 0.800(+33%)presentation/index.html:979-9810.533 → 0.933(+75%)presentation/index.html:994-9960.830 → 0.900(+12%)presentation/index.html:1009-10111.000 → 1.000(0%)35.9→39.1,14.4→15.7,+23.3%site/benchmarks.fixture.json0.0→0.5,0.2→0.4site/benchmarks.jsllms.txtAfter
docs/RESULTS.mdnow shows0.567 (56.7%)/0.300 (30.0%), identical tosite/results.html, obeying the page's own "label percentages explicitly" ruledocs/RESULTS.md§SkillsBench — full 87-task baselines, no optimization+ matchingsite/results.htmlsection, bothsite/results.html§skillsbench-87with the same numbers, split labels, splice footnote, iterations, andval (87, fit)andtest (87, fit metric — test == val by construction, not held out)on both surfacesarXiv:2604.01687v1(the only id present in the tree), with "No external score is quoted here: none is cited insources.bib, and an uncited number is not evidence."—; speaker note corrected from "the results already committed" to "their baseline/optimized numbers are deliberately NOT shown, because no run artifact and no run record for them exists anywhere in the repo"docs/RESULTS.mdpreamble now states it explicitly;site/README.md's false "cross-check againstdocs/RESULTS.mdand the committed run artifacts" replaced with the actual rule (add the canonical section first)site/benchmarks.html's "hand-verified canonical numbers" → points at the per-section evidence markerscore/tests/test_published_results_consistency.py— 3 tests, the issue's own acceptance criterionWhat I removed and why
Four categories, 8 numbers, all removed rather than kept:
71.1%(EvoSkills) — the string appears in exactly one place in the whole repo (the claim itself). Nosources.bibentry, no arXiv id, no URL. An uncited external number in an honesty doc is worse than no number.~$32/$400 cap— the run record has no cost field.docs/runs/local-20260730-skillsbench-opus46-optimize.mdrecords wall-clock but not spend.0.600→0.800,0.533→0.933,0.830→0.900,1.000→1.000. Grepped across every published surface: they appear only in that table. Norun_full/, nodocs/runs/record, nothing in git history. They sat under a speaker note asserting they were "already committed" — the most directly false claim I found.site/benchmarks.html) and "cross-check against … the committed run artifacts" (site/README.md) — aggregate claims that are not true and, per fix(honesty): label the τ²-bench held-out headline as reported, artifact not committed #182's reasoning, rot whenever a run is added.Design note: why the guard checks section identity, not numbers
My first attempt scraped decimals out of the HTML and compared sets. It was unusable: SVG coordinates, font weights (
400;500;600;700;800), arXiv fragments (2603.04900) and external papers' figures (35.9,39.1) all look exactly like rewards, and every one needed an allowlist entry. A guard whose allowlist keeps growing stops guarding.So the guard pins the invariant that actually broke: every
site/results.html<h2 id>maps to adocs/RESULTS.md##heading. That is verbatim the issue's acceptance criterion ("everysite/results.htmlrun anchor maps to aRESULTS.mdsection"). Mapping is an explicit table, not derived from #182's<a id>markers, so the guard's result does not depend on merge order — and a renamed section fails loudly where a set-difference would silently accept it. Proven by reintroducing the original bug (evidence comment).Deliberately left to sibling PRs
README.md:102"cross-checked against committed run artifacts" + the whole Results tableREADME.md:110held-out headline30.0 → 47.5site/index.html:78"Every number ships its artifact" pill;:236"Cross-checked against committed run artifacts"; the 4-row hero tablesite/index.htmlat all.site/results.html:7,12,17meta descriptions "Every number cross-checked…"<head>is generator-owned viachrome:head:*sentinels, so editing it would fail--check. Doubly not mine.site/results.html:55lead; the 7 pre-existing<h2>evidence stamps<span class="muted">— ⚠️ reported, artifact not committed</span>form.presentation/index.html:635, 745, 788+ slide-6 cards<aside class="notes">) — zero overlap, confirmed by a test merge that auto-mergedpresentation/index.htmlwith no conflict.CHANGELOG.md(0.536 → 0.712,0.694named separately)OUTREACH.md:46(val/test conflation) and:47<head>,?v=hashes<main>bodies.harbor.html/harbor-openshift.htmlnot registered inPAGESmain; proven identical without my changes (evidence comment).test_dashboard_launch.pyport-7878 flakeorigin/main.Expected merge order
SkillsBench — skill-package optimizationheading; resolution is a union (keep my sections, take fix(honesty): label the τ²-bench held-out headline as reported, artifact not committed #182's heading). Verified by an actual test merge.--checkexits 0 on the merged tree.Verification
Full suite — 181 passed (179 baseline + 3 new), 0 unexpected failures
The single failure is the known port-7878 flake (#200), reproduced on clean
origin/mainwith none of my changes present:Deselecting it: 181 passed, 0 failed.
New guard passes, and catches the #100 bug
Regression-proof — delete the canonical
qwen-toolssection (the exact original bug) and it fails:python -m compileall core skills— cleanEvery 87-task number identical on both surfaces, with the same split label
Zero surviving aggregate claims
On the merged
#182 + #100tree:One hit, and it is TRUE. It is my own line and is not an artifact claim — it is a normative rule about where numbers may be published, and it is the rule
test_published_results_consistency.pymechanically enforces. Every "every number is artifact-backed"-style claim is gone.Removed numbers are gone from every published surface
Links and anchors resolve on disk (CI
docs-linksruns with--include-fragments)Site pages still parse; fixture untouched
site/benchmarks.jsandsite/benchmarks.fixture.jsonare not modified — the fixture is a dev eyeball fixture, not deployed data, andbenchmarks.jshardcodes no results.#196's generator:
--checkexits 0 on the merged treemain+#182+ this PR +#196:And my edits cause zero chrome drift — chrome regions are byte-identical with and without them:
(The
harbor.html/harbor-openshift.htmlregistration failure is pre-existing — byte-identical onmain + #196alone. #196's to resolve; full both-trees comparison in the evidence comment.)Could not substantiate
Nothing survives unsubstantiated. Every number I kept has either a committed artifact (✅) or a committed run record with the artifact's absence stated in-line (⚠️ ). The 8 numbers with neither were removed.
One honest limitation, unchanged and pre-existing: five τ²-bench runs have no committed artifact anywhere (held-out, agent-mode, and both Qwen runs — the reviewer of #182 confirmed⚠️ -labeled, not deleted, because that is #99/#182's scope and its reviewer's explicit decision. My contribution is that the labeling convention now covers all 9 sections instead of 7, and both surfaces carry all 9.
0.475appears in zero artifact files at any commit; I independently confirmed no Qwen file was ever added in git history). They areFiles touched
docs/RESULTS.mdsite/results.htmlsite/benchmarks.htmlsite/README.mdpresentation/index.htmlcore/tests/test_published_results_consistency.py(new)