Durable synthesized priors (INSIGHTS.md) fed to every proposal, all three algorithms - #219
Durable synthesized priors (INSIGHTS.md) fed to every proposal, all three algorithms#219OsherElhadad wants to merge 6 commits into
Conversation
…ll-climb The two flagship algorithms ran blind. `_inject_optimizer_context` was called only from `run_step` (the hill-climb path): GEPA bypasses `run_step` entirely, so its optimizer got no `./trajectories/`, no `./guidance/<cap>/`, no native-skill injection and a hand-rolled 1.4 KB prompt with no capability brief. `skillopt_loop` had no capability/optimizer parameters and called `_focus_instructions` bare, so CAP_BRIEF was empty and the PARALLEL note always claimed a sequential optimizer. On the CLI side `--capabilities --instructions-file --bench-repo --optimizer-name --capability-sources --target-model --target-profile-file` were gated behind `algorithm_name == "hill-climb"` and silently dropped for the other two. Adds `core/cap_evolve/optimizer_context.py` — the one seam every algorithm routes through: `OptimizerContext` (the per-run bundle + its argparse flag set), `inject` (the file side) and `render_instructions` (the prompt side). hill-climb, gepa and skillopt now all take `ctx=` and produce identical context; the CLI passes the flags unconditionally to all three (agent-driven evograph/agent-optimize excluded explicitly, not per-flag). GEPA's `./trajectories/` is scoped to the parent's minibatch tag, so it holds the verbatim untruncated rollouts REFLECTION.md only excerpts. `INJECTED_DIRS/NAMES` gives the snapshot ignore-list, GEPA's component list and the eval-cache hash one shared definition, so the newly injected read-context cannot become an editable component or bust the cache. Closes #109
…joint failure index Review fixes for PR #199 (issue #109). Three blocking findings. 1. GEPA's cross-iteration history channel was permanently EMPTY. _parent_map / _build_ledger / _build_runmap filtered `kind == "step"`, but GEPA bypasses run_step and emits `gepa_val_gate` — so its LEDGER.md, RUNMAP.md and prior_iterations/ stayed empty while the prompt instructed the optimizer to read them, and it was told its best candidate was `seed` even after an accept. dashboard.py already special-cased all three kinds, so the correct set was known in-repo. Factored into ONE definition, rundir.ITERATION_EVENT_KINDS + RunDir.iteration_events(), now read by all four consumers (dashboard, LEDGER, RUNMAP, prior_iterations) so a fourth kind cannot desync a fifth consumer. gepa_loop also publishes the running best on each accept, not only at loop exit. #128 (persist insight), #129 (failure memory) and #130 (plateau detection) all read exactly this channel — it works for GEPA now. 2. Wrong trajectories on cache hits (regression introduced by this PR). The new `tag=` pin fell through to the best/seed/whole-dir chain when a cached minibatch persisted no rollout files, handing the optimizer a previous iteration's traces (including mb_c_* child rollouts) while the prompt claimed they were "the SAME minibatch VERBATIM". The pin is now honoured or omitted LOUDLY: the stale dir is removed, an optimizer_context_warning is logged, and GEPA's prompt block states the minibatch came from the cache with no traces rather than claiming a dir that holds someone else's data. Adjacent to #111 (GEPA eval cache drops output/trace); this fix does not depend on it — #111 makes trajectories available on a cache hit, this makes the absence honest either way. 3. Failure index was always "of 0 tasks" for SkillOpt. render_instructions received a val SplitResult narrowed by train ids — disjoint by construction — so the index was always empty, and this PR newly routed the slow/meta update through the same shape. Fixed at the root in _focus_instructions: zero overlap between focus_ids and the scored result no longer filters, and the prompt says why. The seam parameter is renamed `current` -> `scored_result` so the invariant is visible at every call site instead of only in a docstring. Non-blocking: cache.py no longer mutates module-level sets at import time (plain constant expression, no import-order dependence; same for harness._SNAPSHOT_IGNORE); a global MAX_INSTRUCTIONS_CHARS cap bounds the assembled prompt (every block was bounded, the sum was not); ARCHITECTURE.md names the seam and the shared event-kind set, and all three algorithm SKILL.mds document the context they receive; the render_instructions docstring records that `extra=` is for algorithm-specific tails and shared blocks belong in the body (#128/#129 need no signature change). Tests: +7 (three blocking regressions, the prompt cap, and a real --target-profile-file fixture exercised end to end on all three algorithms). 199 passed, 0 failed.
…y proposal Closes #128. A compact, bounded, framework-synthesized "what we've learned so far" block that survives across iterations independent of the transcript, and reaches the optimizer prompt for ALL THREE deterministic algorithms. What the insight is (three sections, all derived from the run's own numbers): * What HELPED — gate-ACCEPTED iterations, largest val |Δ| first, with the exact tasks each one fixed (and any it broke anyway). * What HURT — gate-REJECTED iterations, largest |Δ| first, with the reject reason and the exact tasks broken. * Still OPEN — the val tasks the current best does NOT pass. ZERO LLM calls. The synthesis is pure Python over events.jsonl + the persisted rollouts, like every other auxiliary step in core (#132/#205 established that profile). It reads RunDir.iteration_events() and _candidate_task_impact() — the same reads LEDGER.md uses, so the two artifacts can never disagree — and #205's aux_model tier is unused: the signal is already fully determined by the run's numbers, so a per-iteration model call on every run would buy prose, not information. Built on the #199 seam exactly as its reviewer predicted: the whole wiring is two lines in harness._augment_instructions, the ONE function whose output reaches the prompt and which all three algorithms route through (the note #212 left in memory.py). Honesty. Every line is labelled a CANDIDATE PRIOR, a hypothesis worth re-testing; nothing bypasses the val gate, and the prompt says so. Only val rewards and val task ids are read — never the sealed test split, never a gold answer. Growth/eviction. The block is RE-DERIVED every iteration, so it does not grow monotonically — but its input does. Eviction keeps the 6 largest |Δ| movers per section (ties → newer iteration) plus 10 open task ids, because a +0.25 accept and a -0.20 regression are the priors worth re-testing while a -0.001 reject carries no signal. MAX_INSIGHT_CHARS = 4,000 is the backstop. Measured: 1,953 chars after 200 iterations with long task ids — 3.3% of #199's MAX_INSTRUCTIONS_CHARS. INSIGHTS.md is framework read-context, not capability, so it is excluded from the candidate snapshot, GEPA's editable components, the eval-cache content hash (or every iteration would miss the cache, since the block changes as the run progresses) and SkillOpt's applied-edit count. Added to optimizer_context.INJECTED_NAMES — the one list #199 created for exactly this — plus the two diff-skip lists, and SkillOpt's two copy-pasted scaffold tuples were folded into one _SCAFFOLD frozenset (they are how this would have silently registered as an applied edit every iteration). Also fixes a #199 defect this surfaced: SkillOpt logs BOTH "step" (via run_step) and its own "skillopt_step" for the SAME candidate, so RunDir.iteration_events returned two rows per SkillOpt iteration — LEDGER.md, RUNMAP.md, the dashboard lineage and the new priors all double-counted it, the second copy missing parent/parent_val and so showing a blank Δ. Deduplicated by candidate id at the root (first occurrence wins, later records merged in for fields it lacks), so all four consumers are fixed at once rather than the one this PR happens to add.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| if "optimizer_context_warning" in json_line and "mb_p_0007" in json_line] | ||
| assert warn, "unmet trajectory pin failed SILENTLY (no warning event)" | ||
| assert "OMITTED" in warn[0] | ||
| assert kinds == kinds # no-op; keeps the reader exercised without asserting count |
| "A compact, continually-updated summary of what this run has LEARNED SO FAR, " | ||
| "re-derived from the objective record every iteration so it survives even when " | ||
| "the transcript does not. Read it BEFORE proposing.", |
| "**These are CANDIDATE PRIORS, not truth.** Each one is a hypothesis worth " | ||
| "re-testing, and every edit you make is still judged by the val significance " | ||
| "gate — a prior can be wrong, and acting on one earns no exemption.", |
|
❌ Automatic Labeling Failed An error occurred while trying to automatically label this pull request. Please check the workflow logs for details and add labels manually. |
🔬 EvidenceAll commands run in 1. Baseline on the #199 base (before my change)2. Fail-before (source change stashed, test kept)3. Full suite after199 base + 6 new = 205. Plus a 7th test (the SkillOpt dedupe regression) added after that run; the merged-tree run below is 207. 4. Real end-to-end, zero API cost — the driver#!/usr/bin/env bash
set -euo pipefail
ALG="$1"; ITERS="${2:-6}"
REPO=/tmp/wt-128
export CAPEVOLVE_CORE="$REPO/core" PYTHONPATH="$REPO/core"
export CAPEVOLVE_SKILLS_DIR="$REPO/skills"
export CAPEVOLVE_TOY_DATA="$REPO/examples/toy_calc"
export CAPEVOLVE_MOCK_SCRIPT="$REPO/examples/toy_calc/mock_script.json"
D="/tmp/e2e128-$ALG"; rm -rf "$D"; mkdir -p "$D/.capevolve/project/adapters"
cp "$REPO/examples/toy_calc/adapter.py" "$D/.capevolve/project/adapters/"
cp -R "$REPO/examples/toy_calc/capability" "$D/seed_capability"
sed -e "s/^algorithm_skill: .*/algorithm_skill: $ALG/" \
-e "s/^max_iterations: .*/max_iterations: $ITERS/" \
"$REPO/templates/project/capevolve.yaml" > "$D/.capevolve/project/capevolve.yaml"
/tmp/ce-venv/bin/python -m cap_evolve.cli run \
--spec "$D/.capevolve/project/capevolve.yaml" \
--project "$D/.capevolve/project" --run-ts "e2e"4a. hill-climbPer-iteration artifact contents (the money evidence — growth): Empty-but-valid (iter 1) → 1 helped (iter 2) → 1 helped + 1 hurt (iter 3). Present in every workdir listing: 4b. GEPA — the algorithm the hand-filtering bug silently brokeProof this went through 4c. skillopt5. The SkillOpt double-logging defect this surfaced (and its fix)First skillopt run, before the dedupe — every SkillOpt iteration appeared twice:
Fixed for all four consumers (dashboard lineage, LEDGER, RUNMAP, priors) at once — pinned by 6. No sealed-test leak (all three algorithms)sp = json.loads((root/"splits.json").read_text())
test = [str(t) for t in sp["test"]]; val = [str(t) for t in sp["val"]]
for f in (root/"work").rglob("*"): # EVERY file in EVERY iteration workdir
txt = f.read_text()
for t in test:
if t in val: continue # only ids UNIQUE to test are a leak signal
if t in txt: print("LEAK", f, t)Also unit-pinned: 7. Prompt stays under the cap on a long runUnit-pinned by 8. compileall9. Merged tree — #199 + #212 + thisMerged-tree GEPA E2E re-verified end to end: 10. Files touched
|
🔍 Review — PR #219Verdict: CHANGES REQUESTED — 3 blocking. The priors are arithmetically true on every run I constructed (attribution verified against independently-computed ground truth), the SkillOpt double-count is real and the dedup picks the right row, and the leak scans reproduce clean. But the dedup's key is unsound on Blocking1.
Reproduced end-to-end (not synthetic — a real And with a real regression (synthetic events, same code path): The regression never reaches LEDGER, RUNMAP, or the priors. On the #199 base this same input yields 4 rows (double-counted but present); the dedup trades a visible double-count for a silent omission, which is strictly worse for an honesty-critical artifact. GEPA is safe ( Fix: dedup on 2.
Same with 8 long-unicode task ids: Fix: 3.
Consequence is not cosmetic: the priors block is pointed at as "read it FIRST for orientation" ( Fix: mirror the Non-blocking4. A forged 5. 6. This edit fixed 3/10 val tasks and hurt nothing; it was rejected for being statistically indistinguishable from noise. Calling it "HURT" tells the optimizer to avoid a direction that may well be right — the exact misdirection risk the review brief flags. The reject reason is present and does disambiguate for a careful reader, but the heading is doing the loud work. Fix: rename to 7. 8. 9. PR body: "199 base + 6 new = 205". I measured 206 on the branch, 199 on the base — 7 new tests. The 7th ( Non-blocking count: 6 (items 4-9). Nits10. 11. 12. Are the priors true?I built a controlled 4-iteration run where different iterations fix different tasks, computed the expected accepts/rejects/Δ/fixed/broke independently in the test harness, and diffed. 4 val tasks, seed fails all. Plan: Per section:
LEDGER/INSIGHTS agreement (claim 5). They agree on all three real E2E runs (hill-climb / skillopt / gepa, The SkillOpt double-countIt was real. Verified by running the same skillopt config on the #199 base worktree ( So #199's fix (broadening from Is the dedup correct? Partly.
What #129 / #130 / #216 must know (none of those branches exist on origin yet, so this is a forward warning, not a merge conflict):
This is a semantic change to a shared hub function landing in a feature PR. It deserves its own commit message at minimum, and ideally the root fix (drop Val-overfitting verdictNaming persistently-failing val task ids does push toward val overfitting, and the current framing does not defend against it. Not blocking — but the honesty framing is doing less work than the PR believes. Reasoning, separating the two things that get conflated: New exposure: essentially none. The optimizer already receives the full val rollouts for the current best ( New emphasis: substantial, and that's the actual risk. What changes is not availability but persistence and salience. A val failure that survives 40 iterations is repeated in the prompt 40 times, at the top, in a framework-authored block the prompt instructs the optimizer to "read FIRST for orientation" ( Why that's a subtle honesty problem specifically. The gate is val-based, so val is doing double duty: model-selection signal and the thing being optimized. cap-evolve's honesty story is that the sealed test split catches val overfitting after the fact. That story holds — the seal is intact and On the "CANDIDATE PRIORS subject to the val gate" label (item 10). Honest and well-placed for the helped/hurt sections — those genuinely are hypotheses and the gate genuinely re-tests them. It does not address the overfitting vector at all, because "Still OPEN" is not a hypothesis: it is a true fact, and re-testing it via the gate is exactly the mechanism that rewards overfitting to it. The label answers "could this prior be false?" (good) and is silent on "should I chase this specific task?" (the real risk). Framing a target list as a hypothesis is a category error, not a lie. What I'd ask for (non-blocking, cheap): one sentence in the "Still OPEN" section — "these are diagnostic, not a target list: fix the general defect these expose, not these tasks specifically; val is the gate, so a task-specific special case will pass the gate and fail the sealed test." That is a genuinely different instruction from the CANDIDATE PRIORS banner and costs ~30 chars of the 4k budget. Would also make me comfortable with the section persisting for 200 iterations. Cap + leak probes
No sealed-test leak found. Cap does not hold under adversarial input (98 chars over, two distinct triggers). Reject-reason text is not sanitized. Merge-order note
207 reproduces, but the author's "2 mechanical conflicts" undercounts: Recommendation: #199 → #212 → #219. Merging #212 (which removes Verification I re-ranAlso re-ran: the resume-collision repro (blocking 1), the two cap-overflow probes (blocking 2), the >8-task LEDGER/INSIGHTS divergence (blocking 3), the reason-injection probe (nit 4), the 200-iteration cap measurement, the whole-run-dir leak scan for all three algorithms, and the merged |
… no dedup) SkillOpt routes through harness.run_step, which already logs "step" for the same candidate carrying parent/Δ/reason; "skillopt_step" only adds epoch/edit_budget/ applied_changes. Counting both double-counted every SkillOpt iteration in LEDGER.md / RUNMAP.md / INSIGHTS.md, the second copy missing parent/parent_val (blank Δ) and poisoning _parent_map. The id-keyed dedup this branch first shipped is REVERTED: SkillOpt mints ids from epoch/step counters that reset on --resume (the algorithm's --resume restores only current_val), so a resumed run re-emits so_e01s01 for a different candidate and first-wins dedup DISCARDED it — regression included. Measured on one live resumed run: 9 rows on the #199 base (double-counted but present), 4 with the dedup (a resumed reject silently gone), 5 correct after this change. iteration_events() is now documented as one row per logged event in log order with no dedup; consumers must key history by position, not candidate id. The dashboard folds skillopt_step's epoch onto the lineage node in a separate pass so the audit metadata is not lost. Also in the durable priors block (#128): - the char cap now RESERVES the truncation notice's length before cutting, so the output is bounded inclusive of the notice (it previously overshot by 98 chars on a long reject reason or long unicode ids), and the pinning test now actually crosses the bound; - broke/fixed task sets render an honest "+N more" count instead of a silent cut at 8, so INSIGHTS no longer under-reports a 20-task regression as 8 while LEDGER shows 20 (LEDGER's own [:20] cut is marked the same way); - gate reasons are flattened and markdown-escaped, so a reason cannot forge a "## What HELPED" section inside a framework-authored block, and are bounded to 200 chars so one reason cannot eat the block's budget; - "Still OPEN" leads with a count (N of M val tasks) and carries an explicit anti-overfit instruction: the names are a diagnostic, not a target list, and a task-specific special case passes the val gate and fails the sealed test; - rejects now render what they FIXED as well as what they broke, and the heading is "What was REJECTED ... a reject is not necessarily a regression" rather than "What HURT", which mislabelled positive-Δ significance rejects; - accepted rows evict by RECENCY (every row already cleared the gate, so |Δ| just froze a top-6 and permanently evicted small-but-real effects); rejected rows still evict by |Δ|, which is the damage signal; - an absent-rollouts "Still OPEN" is reported as UNKNOWN rather than rendering identically to a perfect candidate; - a missing Δ renders "Δ ?" instead of a fake "+0.000"; - the workdir copy is written atomically like the durable one, and the false docstring claim that the dashboard reads INSIGHTS.md is removed.
#212 narrows the signature (drops rejected/history). Passing them positionally made this a THIRD mechanical merge site — and unlike the two in harness.py it surfaced as a test failure git does not flag as a conflict. Fill trailing params reflectively so the test passes on both signatures.
Fourth mechanical merge site for #212's narrowed signatures (two in harness.py, two here) — all in tests, so git flags none of them as conflicts. Also refresh the module docstring for the renamed sections and the new pinned properties.
| lines += ["", "## What was REJECTED by the gate (largest movers first — a reject is " | ||
| "not necessarily a regression; read the reason)"] |
| "**These names are a DIAGNOSTIC of where the capability is weak, not a " | ||
| "target list.** Fix the general defect they expose; your edit must generalize " | ||
| "beyond them. The gate runs on val, so a task-specific special case for these " | ||
| "ids will pass the gate and FAIL the sealed test — that is a val overfit, not " | ||
| "progress.", ""] |
🔧 Review fixesAll three blocking findings fixed, all 6 non-blocking + 3 nits addressed or declined with a reason, and the three verification claims you corrected are restated accurately below. Pushed as three commits on Thank you for the independent 4-iteration ground-truth run and the live Blocking 1 — dedup dropped: I took your root fixI removed the dedup entirely and dropped # core/cap_evolve/rundir.py
ITERATION_EVENT_KINDS = ("step", "gepa_val_gate")
Prove the resume case — live, not syntheticReal Row counts on that exact events file, three ways: Resume now preserves the iteration. Confirmed on the live artifacts too — the fixed reader: Pinned by two tests: No metadata lost
# core/cap_evolve/dashboard.py — after the lineage loop
for ev in events:
if ev.get("kind") == "skillopt_step":
n = nodes.get(_step_candidate(ev) or "")
if n is not None and ev.get("epoch") is not None:
n["epoch"] = ev.get("epoch")
Hill-climb and GEPA re-verified unaffectedBoth were already safe and stay safe — real zero-API toy_calc E2E on the fixed branch: Hill-climb emits only For #129 / #130 / #216 — explicit contract statement
I've messaged #221 (plateau detection) directly — its reviewer found the same root defect from the other side (its plateau ladder fired at half the configured window). #219 owns the
And your correction to my claim — acceptedYou are right: it was three consumers, not four. Blocking 2 — the cap now holds, and the pinning test crosses the boundThe arithmetic was exactly as you diagnosed. Fixed by reserving the notice before cutting: _INSIGHT_TRUNC = ("\n\n... (priors truncated to stay inside the optimizer prompt budget; "
"the full record is LEDGER.md)\n")
...
if len(text) > max_chars:
room = max_chars - len(_INSIGHT_TRUNC)
text = (text[:room].rstrip() + _INSIGHT_TRUNC) if room > 0 else text[:max_chars]
text = text[:max_chars] # rstrip can only shorten, so this is the hard backstopThe Both of your triggers, measuredTrigger A no longer even reaches the cap, because bounding each reason to 200 chars (non-blocking 4) removed the overflow at its source — as you predicted it would. Trigger B lands on exactly 4000, inclusive of the notice. The pinning test is no longer vacuous — for cap in (harness.MAX_INSIGHT_CHARS, 1200, 400, 120, len(harness._INSIGHT_TRUNC), 10):
out = harness._build_insights(Path(tempfile.mkdtemp()), rd2, max_chars=cap)
assert len(out) <= cap, f"cap {cap} overflowed to {len(out)}"
out.encode("utf-8").decode("utf-8") # codepoint-safe, no mojibakeComposed tree (#199 + #219 + #222 + #221) — measured, under the capI merged the real branches: #199 → #212 → #219 → #222 (226 passing), and measured #221's Composed-tree number: 31,617 / 60,000. No block is silently dropped — #222's Blocking 3 — LEDGER and INSIGHTS now agree past 8 tasksYou were right that the docstring claim was false, and right that def _tasks(label: str, ids: list[str]) -> str:
if not ids:
return ""
shown = ", ".join(ids[:_INSIGHT_TASKS])
extra = len(ids) - _INSIGHT_TASKS
return f" — {label} {{{shown}{f', +{extra} more' if extra > 0 else ''}}}"
The >8-task reconciliation, measured8 shown +
I did not raise The val-overfitting finding — acted onYour analysis convinced me. The distinction between exposure (unchanged) and persistence + salience (new, and the actual risk) is the part my honesty framing missed, and you're right that framing a target list as a hypothesis is a category error — "Still OPEN" is a true fact, so "re-test it via the gate" is the mechanism that rewards overfitting rather than a defence against it. The exact sentence added
It sits directly under the "Still OPEN": count AND ids — my reasoningYou asked me to consider a count-and-character instead of an id list, and to argue the choice. I added the count and kept the ids, leading with the count: Reasoning. The count is a strict improvement and I should have had it from the start:
So: count for severity, ids for verifiability, and an explicit instruction about what to do with them. If the predicted failure (a val-passing special case) shows up in a real run, dropping to count-only is a one-line change and I'd take it then rather than speculatively now. Numbered response to all 12 findings1. (blocking) Dedup key unsound on 2. (blocking) Cap overflows by the notice — Fixed. Notice length reserved before truncating, plus a hard backstop for a degenerate cap. Both triggers now hold (2732 and exactly 4000 against a 4000 cap). Pinning test rewritten to actually fire the truncation path — it asserts the notice is present, so it fails if it stops crossing. Composed tree 31,617/60,000 with #222 and #221. 3. (blocking) 4. (non-blocking) Reject reason unsanitized — Fixed, and I went further than the one-liner. The reason text is still fully visible — flattened and escaped, not dropped, because a real gate reason is diagnostic information the optimizer needs. Pinned by 5. (non-blocking) False docstring claim about the dashboard — Fixed. Replaced with the truth: the dashboard does not read it, 6. (non-blocking) Positive-Δ rows under "What HURT" — Fixed, renamed rather than split. Headings are now 7. (non-blocking) Empty rollouts indistinguishable from all-passing — Fixed. Three distinct renderings: 8. (non-blocking) |Δ| eviction wrong for HELPED — Fixed, per-section policy, and I agree with your reasoning. Rejected rows still evict by |Δ| (there, |Δ| is the damage). Accepted rows now evict by recency, because every row already cleared the gate so |Δ| only re-ranks winners and permanently evicts the reproducible small effect — six +0.30 accepts crowding out a +0.02 forever, and the optimizer never learns that direction works. The policy note above 9. (non-blocking) "199 + 6 = 205" is stale — Corrected, see the three claims below. 10. (nit) 11. (nit) missing Δ renders as 12. (nit) Three corrected verification claims1. Test count: 206 on the branch, not 205. You measured correctly. The 7th test was added after the body was written and the headline number was never updated. The number is now 212 (199 base + 13 in 199 + 13 = 212. 2. Fail-before is 13/13, not 6/6 (and you were right that it was 7 at the time). Sources reverted to the #199 base, tests kept: All 13 fail without the source change — including all three blocking fixes, which is the point. 3. The dashboard-fix claim does not reproduce, and I withdraw it. 4. "2 mechanical conflicts" undercounts — and it's worse than 3. You found extra = len(inspect.signature(harness._augment_instructions).parameters) - 3
out = harness._augment_instructions("BASE", wd, rd, *([None] * max(0, extra)))So: 2 conflicts git flags (both in Re-proved after the changesPer-task attribution — your 4-iteration lineage fork, re-runEvery Δ, accept/reject, and per-task attribution still matches ground truth. No sealed-test leak — re-scanned, all three algorithmsZero test-only ids in every workdir file, in the durable copy, in compileallMerge orderAgreed: #199 → #212 → #219. Merging #212 first means #219 rebases onto the narrower Files touched by these fixes
|
Review fixes for PR #221. Two of the four blocking findings made the feature kill or degrade productive runs, which is the failure direction this module exists to prevent. 1. SkillOpt iterations were counted TWICE. `harness.run_step` logs `step` for a candidate and `skillopt.py` then logs `skillopt_step` for the SAME candidate; both were in the iteration-kind list, so a window=6 ladder reached `stop` at iteration 5 of a productive run, and a genuine near-miss was recorded once alive and once dead (the duplicate carries no parent_val). Dropped `skillopt_step` from the pre-#199 `_FALLBACK_KINDS` literal. PR #219 owns the root fix in `rundir.ITERATION_EVENT_KINDS`; no second dedup layer here, and deliberately NOT id-keyed dedup, which drops resumed SkillOpt iterations. 2. `NameError: cannot access local variable 'why'` in `hill_climb_loop` when the loop body never runs (max_iterations=0, or a resume with no budget left). `why` is now initialised next to `steps`. 3. The plateau block escaped the prompt cap and silently lost data. It now travels as `extra=` through `_augment_instructions` / `run_step`, so it is inside #222's MAX_INSTRUCTIONS_CHARS (measured 64941 -> 2625) and lands in the TAIL that truncation preserves — it is the only behavioural block of the three, so it must not be the first casualty. Removed the bare rejected-ids line entirely: it called `RejectedMemory.entries()` (removed by #199) behind a bare `except` that swallowed the AttributeError, and duplicated #222's richer signature channel. `prompt_block` no longer accepts `rejected=`, so a caller passing one now fails loudly. Bounded `exhausted_lineages` to 6 ids, 60 chars each. 4. GEPA was steered away from its BEST lineage. `exhausted_lineages` reused `_dead()`, which counts accepted-but-not-global-best as dead, so a lineage whose children were ALL accepted was dropped from the Pareto sampling pool — with no ratchet on that path and no stop event. Per-lineage exhaustion now uses its own narrower `_lineage_dead()`: an accept is never dead ground for a lineage, because widening the per-instance frontier is GEPA's mechanism. The accepted-but-not-best clause stays at the GLOBAL level, where the ratchet makes it safe. Also, from the non-blocking findings: - The ladder was unreachable at the shipped defaults: `stall: 2` stopped every run at 3 iterations, before `plateau_window: 6`. `stall` now defaults to 0 (off) with the reasoning in the template, so the shipped product actually runs the ladder. All evidence is re-run at the shipped default. - The ratchet now caps escalation at `warn` rather than `diversify`. A run still clearing the honest val gate every iteration gets a warning and no behavioural intervention, so the diversify block can never tell an optimizer a lineage failed when it was accepting. - The reason string no longer claims "no near-miss in that streak" when the streak is all accepts. - `series` carries the real signed delta for `gepa_local_gate` rows instead of hardcoding 0.0, so a tie is distinguishable from a regression. Deadness is unchanged (gepa's pass condition is a strict `>`). - An explicit `plateau_window: 0` now means off instead of silently reverting to 6. - Plateau state reaches the React dashboard: `plateau_level` on the hub row, `plateau` / `exhausted_lineages` typed on the detail summary, and a KPI tile. It is a separate field from `status` on purpose — that one is liveness, this one is progress. - Documented the honest cost of the delta<=0 rule (N regressions then a breakthrough is stopped at N) and that `--resume` deliberately carries the streak. Tests: 215 core (was 207) + 44 dashboard backend. 8 new regression tests, one per blocking finding plus resume, the bounded block, and the local-gate delta. `test_plateau_block_reaches_the_prompt_inside_the_cap` asserts the block reaches the prompt, so the #199 prompt-assembly merge trap fails loudly instead of compiling clean with the feature dead.
Closes #128.
A compact, bounded, framework-synthesized "what we've learned so far" block —
INSIGHTS.md— that survives across iterations independent of the transcript, and reaches the optimizer prompt for all three deterministic algorithms.What the insight actually is
Three sections, all derived from the run's own numbers:
Written to both the run dir (
run_<ts>/INSIGHTS.md— the durable copy, and what the dashboard Insights tab can read) and each iteration's workdir (the copy that reaches the prompt).Is an LLM call involved? No.
Zero LLM calls. The synthesis is pure Python over
events.jsonl+ the persisted rollouts, matching the profile #205/#132 established for every auxiliary step in core (reflection distillation, ledger/runmap, failure clustering, the wholediagnosephase). It readsRunDir.iteration_events()and_candidate_task_impact()— the same readsLEDGER.mduses, so the two artifacts can never disagree.#205's
aux_modeltier is deliberately unused: the signal is already fully determined by the run's own numbers, so a per-iteration model call on every run would buy prose, not information. If a future version wants narrative priors, that is where it belongs.Built on the #199 seam
Its reviewer called #128 a "✅ two-line" extension, and it is: the entire wiring is two lines in
harness._augment_instructions— the ONE function whose output reaches the prompt, and which all three algorithms route through (the note #212 left inmemory.py). No per-algorithm block, no signature change torender_instructions.Critically,
_insight_rowsreadsRunDir.iteration_events(), notkind == "step". That hand-filtering is the bug #199 fixed in three consumers (GEPA emitsgepa_val_gate); a fourth instance is still open as #216. Filtering by hand would have made the priors block permanently empty for the flagship algorithm.Growth / eviction policy and cap
The block is re-derived every iteration, so it does not grow monotonically — but its input does, and an unbounded render would silently eat #199's
MAX_INSTRUCTIONS_CHARSbudget over a long run.|Δ|movers per section (ties → newer iteration) + 10 open task ids.MAX_INSIGHT_CHARS = 4_000.MAX_INSTRUCTIONS_CHARS.Honesty
Every line is labelled a CANDIDATE PRIOR — a hypothesis worth re-testing. Nothing bypasses the val gate, and the prompt block says so explicitly: "a prior can be wrong, and acting on one earns no exemption." Only val rewards and val task ids are read; the sealed test split and gold answers are never touched.
Not capability bytes
INSIGHTS.mdis framework read-context, so it is excluded from the candidate snapshot, GEPA's editable components, the eval-cache content hash (leaving it in would make every iteration miss the cache, since the block changes as the run progresses) and SkillOpt's applied-edit count. Added tooptimizer_context.INJECTED_NAMES— the one list #199 created for exactly this — plus the two diff-skip lists. SkillOpt's two copy-pasted scaffold tuples were folded into one_SCAFFOLDfrozenset; they are precisely how this would have silently registered as an applied edit every iteration.Bonus root-cause fix this surfaced
SkillOpt logs both
step(viaharness.run_step) and its ownskillopt_stepfor the SAME candidate, soRunDir.iteration_events()returned two rows per SkillOpt iteration.LEDGER.md,RUNMAP.md, the dashboard lineage and the new priors all double-counted it — the second copy missingparent/parent_valand therefore showing a blank Δ:Deduplicated by candidate id at the root of
iteration_events(first occurrence wins — it carries the parent edge and gate reason; later records are merged in for fields it lacks, so algorithm metadata is not lost). All four consumers fixed at once, not just the one this PR adds.Expected merge order
#199(the seam) →#212(drops_augment_instructions' unusedrejected/historyparams) → this. Both conflicts are trivial and mechanical (#212's signature drop); I verified the resolved merged tree — see Verification.#210(real output/trace on cache hits) is independent but makes the reflective dataset non-hollow, which is what the priors' per-task lists ultimately distill.Verification
Baseline 199 on the #199 base → 205 passed, 0 failed on this branch (6 new). Merged tree with #199 + #212 applied → 207 passed, 0 failed.
Full suite
Fail-before (source change stashed)
Real end-to-end, zero API cost — the insight GROWING across iterations
examples/toy_calcviacap-evolve runwith themockoptimizer. Per-iterationwork/<cand>/INSIGHTS.md:hill-climb (
baseline_val 0.0 → test_reward 1.0)Empty-but-valid → 1 helped → 1 helped + 1 hurt. And it reaches the prompt every iteration (
grep -c INSIGHTS.md INSTRUCTIONS.md= 1, sizes 21,288 / 20,830 / 20,830 B — all well under the 60,000 cap).GEPA specifically — the algorithm the hand-filtering bug silently broke (
baseline_val 0.0 → test_reward 1.0)Event-kind census for that run confirms GEPA never emits
step, so the priors are populated purely throughgepa_val_gate:Prompt refs 1/1/1, sizes 22,657 / 21,642 / 21,642 B.
skillopt (
baseline_val 0.0 → test_reward 1.0)(each candidate appears exactly once — the dedupe fix above.)
No sealed-test leak
(Scan is every file under
work/**for ids unique to the test split.)Prompt stays under the cap on a long run
compileall
Merged tree (#199 + #212 + this)
Merged-tree GEPA E2E re-verified: priors present and growing across all three iterations.