refactor(core): split the 1,928-line harness.py god module into nine focused modules - #235
refactor(core): split the 1,928-line harness.py god module into nine focused modules#235OsherElhadad wants to merge 32 commits into
Conversation
Two small-sample hazards made the acceptance gate dishonest on small benchmarks: 1. `make_splits` silently produced an empty (n=0) or single-task (n=1) val split for tiny task lists. The gate then had nothing to decide on, or 0 degrees of freedom so SE(Δ) collapsed and it degenerated to "any Δ>0 wins" — unannounced. `check_val_size` now REFUSES both with an actionable `TinyValSplitError`, at split-freeze time AND again at `baseline` (so a hand-written/resumed splits.json can't sneak past). 2 <= n < 5 proceeds but logs a `split_warning` and flags each gate decision LOW CONFIDENCE. `CAPEVOLVE_ALLOW_TINY_VAL=1` opts out. 2. The paired gate's bar was `k_se * SE`, i.e. a *z* multiplier — valid only when SE is known. SE(Δ) is estimated from the same n deltas, so the standardized mean difference is t-distributed with df = n-1 (fatter tails). Using z at small n set the bar too LOW and accepted noise. The bar now uses the t multiplier at the same one-sided significance level. t >= z always, so this can only make the gate STRICTER: 1.32x wider at n=3, 1.14x at n=5, 1.06x at n=10, 1.02x at n=30, converging to the old z bar as n grows. `stats` gains a stdlib-only Student-t CDF (regularized incomplete beta, A&S 1964 §26.7.1 + Numerical Recipes 2e §6.4 `betacf`) and `t_critical` by bisection; cross-checked against published A&S Table 26.10 values. Zero new runtime deps. test_empty_seed's toy adapter goes 4 -> 8 tasks: its 4-task set produced exactly the val=1 split the new guard rightly refuses.
…ader cap-evolve enforces honesty on the evaluation side (seeded splits, sealed test, val-only significance gate). But when the capability is tool code or a skill package, the optimizer is a coding agent with write tools and nothing structural stopped it from "improving" by editing the scorer / eval harness / task data instead of the target. A candidate that rewrites score() is reward hacking and every number after it is fiction. New core/cap_evolve/protect.py records a SHA-256 manifest of the protected paths at baseline (protected.json) and re-verifies it inside evaluate_candidate — the chokepoint EVERY evaluation goes through (baseline, each iteration's val gate, finalize) — plus GEPA's minibatch path, which bypasses it. Any modification, deletion, or newly-added protected file logs a tamper_detected event and raises TamperError naming the file, before the candidate can be scored, snapshotted, become best, or seal the test split. Content hash, not mtime: os.utime is a one-liner, and a size check misses same-length edits. hashlib is stdlib, so zero new deps. Defaults derive from the project layout (adapters/, capevolve.yaml, the spec's dataset_source / split_ids_file, any *gold* file); protected_paths in capevolve.yaml overrides them. The capability dir is never protected — it is the target. __pycache__/*.pyc is excluded because load_adapter execs the adapter during a normal run, and a guard that blocks normal runs is worse than none. The existing PreToolUse honesty hook also denies writes to a protected path, so the model gets actionable feedback instead of burning an iteration; core remains the enforcement. Tests: adversarial (an optimizer editing adapters/adapter.py fails the run, the event is logged, best_id is unchanged, test stays unsealed) and negative (the real mock optimizer's legitimate capability edit runs clean to a sealed test with zero tamper events), plus mtime-spoof, pycache, override, add/delete, dashboard, hook, and no-project no-op cases.
…ll-climb The two flagship algorithms ran blind. `_inject_optimizer_context` was called only from `run_step` (the hill-climb path): GEPA bypasses `run_step` entirely, so its optimizer got no `./trajectories/`, no `./guidance/<cap>/`, no native-skill injection and a hand-rolled 1.4 KB prompt with no capability brief. `skillopt_loop` had no capability/optimizer parameters and called `_focus_instructions` bare, so CAP_BRIEF was empty and the PARALLEL note always claimed a sequential optimizer. On the CLI side `--capabilities --instructions-file --bench-repo --optimizer-name --capability-sources --target-model --target-profile-file` were gated behind `algorithm_name == "hill-climb"` and silently dropped for the other two. Adds `core/cap_evolve/optimizer_context.py` — the one seam every algorithm routes through: `OptimizerContext` (the per-run bundle + its argparse flag set), `inject` (the file side) and `render_instructions` (the prompt side). hill-climb, gepa and skillopt now all take `ctx=` and produce identical context; the CLI passes the flags unconditionally to all three (agent-driven evograph/agent-optimize excluded explicitly, not per-flag). GEPA's `./trajectories/` is scoped to the parent's minibatch tag, so it holds the verbatim untruncated rollouts REFLECTION.md only excerpts. `INJECTED_DIRS/NAMES` gives the snapshot ignore-list, GEPA's component list and the eval-cache hash one shared definition, so the newly injected read-context cannot become an editable component or bust the cache. Closes #109
…joint failure index Review fixes for PR #199 (issue #109). Three blocking findings. 1. GEPA's cross-iteration history channel was permanently EMPTY. _parent_map / _build_ledger / _build_runmap filtered `kind == "step"`, but GEPA bypasses run_step and emits `gepa_val_gate` — so its LEDGER.md, RUNMAP.md and prior_iterations/ stayed empty while the prompt instructed the optimizer to read them, and it was told its best candidate was `seed` even after an accept. dashboard.py already special-cased all three kinds, so the correct set was known in-repo. Factored into ONE definition, rundir.ITERATION_EVENT_KINDS + RunDir.iteration_events(), now read by all four consumers (dashboard, LEDGER, RUNMAP, prior_iterations) so a fourth kind cannot desync a fifth consumer. gepa_loop also publishes the running best on each accept, not only at loop exit. #128 (persist insight), #129 (failure memory) and #130 (plateau detection) all read exactly this channel — it works for GEPA now. 2. Wrong trajectories on cache hits (regression introduced by this PR). The new `tag=` pin fell through to the best/seed/whole-dir chain when a cached minibatch persisted no rollout files, handing the optimizer a previous iteration's traces (including mb_c_* child rollouts) while the prompt claimed they were "the SAME minibatch VERBATIM". The pin is now honoured or omitted LOUDLY: the stale dir is removed, an optimizer_context_warning is logged, and GEPA's prompt block states the minibatch came from the cache with no traces rather than claiming a dir that holds someone else's data. Adjacent to #111 (GEPA eval cache drops output/trace); this fix does not depend on it — #111 makes trajectories available on a cache hit, this makes the absence honest either way. 3. Failure index was always "of 0 tasks" for SkillOpt. render_instructions received a val SplitResult narrowed by train ids — disjoint by construction — so the index was always empty, and this PR newly routed the slow/meta update through the same shape. Fixed at the root in _focus_instructions: zero overlap between focus_ids and the scored result no longer filters, and the prompt says why. The seam parameter is renamed `current` -> `scored_result` so the invariant is visible at every call site instead of only in a docstring. Non-blocking: cache.py no longer mutates module-level sets at import time (plain constant expression, no import-order dependence; same for harness._SNAPSHOT_IGNORE); a global MAX_INSTRUCTIONS_CHARS cap bounds the assembled prompt (every block was bounded, the sum was not); ARCHITECTURE.md names the seam and the shared event-kind set, and all three algorithm SKILL.mds document the context they receive; the render_instructions docstring records that `extra=` is for algorithm-specific tails and shared blocks belong in the body (#128/#129 need no signature change). Tests: +7 (three blocking regressions, the prompt cap, and a real --target-profile-file fixture exercised end to end on all three algorithms). 199 passed, 0 failed.
Review of #197 demonstrated four working bypasses of the guard end to end. Each is now closed with a regression test that pins it. 1. Bytecode cache. A PEP 552 UNCHECKED_HASH pyc planted in the adapter's cache slot executed a hacked score() while adapter.py's SHA-256 stayed identical to the manifest — a sealed test_reward 1.0 against an honest 0.0 with tamper_detected: 0. Unchecked-hash pycs skip mtime, size AND hash validation, so cache invalidation is not a defence. Fixed by stopping the write rather than excluding the file: load_adapter clears the adapter's __pycache__ and sets sys.dont_write_bytecode, so a legitimate run writes no bytecode and the .pyc/__pycache__ exclusion is deleted entirely. A pyc appearing mid-run now reads as an added protected file and aborts the run. 2. Manifest rewrite. protected.json lives in the run dir the optimizer can write, and a torn manifest was silently re-recorded from the current (tampered) tree. Its digest is now logged with the protected_manifest event; a manifest that is missing, unparseable, or no longer matches that digest is a hard TamperError. Re-recording is legitimate only at first creation. The PreToolUse hook additionally denies writes to protected.json, events.jsonl, state.json and best.txt. 3. TOCTOU race. Verification was pre-scoring only, so a detached grandchild of the optimizer subprocess could rewrite ground truth during scoring undetected. Added the post-scoring re-verify issue #142 step 3 asked for, in evaluate_candidate (before aggregate_scores), in GEPA's _eval_minibatch, and once more in finalize before commit_test burns the seal. The residual window is the gap between the scorer's last read and the post-check hash, stated as such in HONEST_EVAL.md. 4. Symlink replacement. A protected path replaced by a symlink de-protected itself, because .resolve() followed the link out of the project dir. Relative paths are now computed from the un-resolved name and a symlink is hashed as a sentinel, so becoming (or ceasing to be) a symlink is a change. Fixed in both resolve_protected and is_protected. Also addressed from the review: - HONEST_EVAL.md guarantee 5 is "tamper-evident", not "tamper-proof", and now states exactly what is guaranteed (detection of byte-level changes to declared protected files between baseline and finalize, before the affected score is recorded, made best, or sealed) and the residual gaps: the scoring-window race, out-of-project ground truth, adapter-less projects, per-run scope, and the hook being advisory rather than enforcement. - A malformed or empty protected_paths is a hard error instead of a silent fallback to the defaults, and the fallback YAML parser now understands block sequences (the idiomatic form previously parsed as {} for every list key). - tamper_events are rendered: a red banner at the top of the dashboard, the ANSI summary, and report.md. - *gold* narrowed to data suffixes so docs/golden-rules.md is not swept in. - A project with no adapters/adapter.py logs protected_manifest_skipped, and a declared glob matching nothing inside the project logs protected_paths_unmatched, so "no protection" is never indistinguishable from "clean". - reuse_baseline refuses a prior run that logged a tamper and inherits its manifest instead of re-recording from the current tree. - is_protected compares case-folded, so a case-varied spelling of a protected path cannot slip past the hook on APFS/NTFS. - The hook's protected-path check prints to stderr on internal error instead of failing open silently. Refs #142
…orrect-or-loud t Review of PR #195 (issue #113) found 5 blocking issues. All five are fixed, plus the 10 non-blocking findings and 4 nits. BLOCKING 1+2 — three unguarded production paths to a gate decision with n<2. The guard sat at ensure_splits + baseline only. reuse_baseline copied a prior splits.json and returned before baseline() (so an escape-hatch run's split became a reusable seed for later runs that nothing marked dishonest), and the baseline --resume / --reuse-baseline fast-paths returned before any check. Fixed at the real chokepoint: gate.decide itself now refuses fewer than 2 matched pairs, which no caller can route around. The three creation paths get check_val_size too (defense in depth, and friendly pre-budget failures). This also closes finding 7: a HEALTHY split whose realized pair count collapsed (candidate errored on most val tasks; _paired_deltas intersects ids) previously got ACCEPTED via the SE=0 strict fallback. BLOCKING 3 — escape-hatch runs were indistinguishable from honest runs. They now carry a durable tiny_val_bypass marker in state.json; final.json gains honest_gate: false plus a warnings array; report.md leads with a NOT AN HONEST GATE banner and retracts its "held-out tasks the optimizer never saw" claim; the dashboard renders a red banner above every number (dashboard.py had no split_warning branch at all). The warning text no longer claims a "Student-t correction (df=0)" was applied at n<2, where none is. Keeping the env var: it is out-of-band, an optimizer editing capevolve.yaml cannot set it, and CI smoke tests need it — the objection was the silent output, which is now loud. BLOCKING 4 — t_critical was SILENTLY wrong for alpha below ~1e-12. It bisected on 1.0 - alpha, exactly 1.0 in float64 below alpha ~1e-16: -5.9% at k_se=8, -29% at 8.3, constant for 10..38, and above 38.5 alpha underflowed so max() silently reverted to the raw z bar — the original bug. Now bisects on a new cancellation-free survival function t_sf, verifies the solved value reproduces alpha before returning, and raises instead of returning sentinels. Verified against an independent closed form at df=2 to 3.8e-13 across k_se 1..26.5; A&S Table 26.10 values unchanged. The max(k_se, t) clamp is gone (bound in 0 of ~30k valid cases; its only live effect was hiding this). betainc now raises on non-convergence. BLOCKING 5 — remediation option 2 did not work. `split_val: 0.4` yields (0.5, 0.4, 0.25), still val=1 at n=3 and n=4. Replaced with the full verified triple 0.25/0.5/0.25 (correct for every n >= 4), a note that option 3 needs test >= 1, and the fact that at exactly 3 tasks no ratio can work. Also: LOW CONFIDENCE now reaches report.md (finding 8); the SE=0 path's exemption from the correction is documented (6); k_se's reinterpretation as a z quantile is documented where users set it — capevolve.yaml template and OPTIMIZE_YOUR_OWN.md (11); CHANGELOG entries for both breaking changes (10); the function-local stats import moved to module level (13); "1 tasks" grammar (14); skillcheck.temp_run_dir default 4 -> 8 ids so skill check.py files don't trip the guard. The committed SkillsBench artifact (n=7, k_se 0.2) re-decides identically — zero flips — so published results still reproduce.
…ean (#110) GEPA's candidate snapshots were dirty: both `run_dir.snapshot()` calls in gepa.py omitted `ignore=`, so every accepted candidate carried FOCUS.md, REFLECTION.md, LEDGER.md, JOURNAL.md and RUNMAP.md alongside the capability — 8 files / 32 KB where hill-climb stored 3 files, and every dashboard candidate-vs-parent diff showed scratch churn instead of the real edit. Root cause is not the missing `ignore=` argument, it is that the scratch-name list was copy-pasted into four modules and desynced. FOCUS.md/REFLECTION.md were already in cache._IGNORE_NAMES and gepa._NON_COMPONENT but never in harness._SNAPSHOT_IGNORE, so even after PR #199 made _SNAPSHOT_IGNORE derived from optimizer_context.INJECTED_* (which fixed LEDGER/JOURNAL/RUNMAP), GEPA snapshots still leaked GEPA's own scratch. Same class of bug as #109's `kind == "step"` filter and #189's counts. Fixed once: rundir.SCRATCH_NAMES is the single definition, at the bottom of the import graph, read by all four consumers — harness._SNAPSHOT_IGNORE, cache._IGNORE_NAMES, gepa._NON_COMPONENT and skillopt._SCAFFOLDING (which still had the literal inline, twice). A newly-injected scratch file now lands in all four automatically. INSTRUCTIONS.md/PROCESS.md stay deliberately OUT of SCRATCH_NAMES: they are snapshotted for explainability and filtered at diff time only. Verified end to end on examples/toy_calc with the mock optimizer: a GEPA candidate went from 8 files / 32 KB to 3 files / 12 KB and now matches hill-climb's snapshot contents exactly (INSTRUCTIONS.md, PROCESS.md, prompt.txt) — 5 stray files per accepted iteration eliminated. Tests: +3 (the shared-constant invariant, plus an end-to-end snapshot-cleanliness check parameterized over gepa and hill-climb so the two can't diverge again). Both fail on origin/main AND on top of #199; hill-climb passes throughout. 182 passed, 0 failed. compileall core skills clean. Zero new runtime deps. Closes #110
…ache docstring
The issue's "memory.py is write-only" premise is only PARTLY right. The
render/entries API and the note/impact fields ARE dead, but the jsonl files
themselves have a live reader the issue told us to verify: the dashboard's
GET /api/runs/{id}/memory (dashboard/backend/capevolve_dashboard/memory.py),
which feeds the Memory panel and the Insights "dead ends" grouping. So the
WRITES stay; only the genuinely-unread surface goes.
Removed (zero readers, proven by grep):
- RejectedMemory.render / .entries, History.render / .entries, _render_impact,
_store_impact — the prompt-facing API. LEDGER/JOURNAL/RUNMAP replaced it.
- the `note=` and `impact=` kwargs and the `note`/`broke`/`fixed` record fields.
- harness._latest_journal_note — its only caller was that dead `note=`.
- the per-iteration _candidate_task_impact call in run_step, which existed only
to populate those dead fields (re-read rollouts from disk every iteration).
The LEDGER and _reconcile_journal paths keep their own computations.
- the unused `rejected` / `history` params on _augment_instructions and
_build_ledger.
_init_memory_store is untouched, so PR #204's algorithm-label stamp and the
dashboard badge are unaffected.
cache.py's docstring claimed wiring into evaluate_candidate was "OFF by default
and gated behind a flag (see maybe_cached_score)" — no such wiring, flag, or
function exists. Replaced with what the cache actually does: GEPA-only, consumed
solely by gepa._eval_minibatch; evaluate_candidate always pays full price.
Test change: test_rejected_memory_roundtrip_and_render tested the removed
render(); replaced by test_memory_jsonl_record_shape_matches_dashboard_contract,
which pins the exact keys the dashboard reads — the contract that actually
matters. Net test count unchanged.
Closes #114
…hot + cache hash Review fix for #211. The four-way unification made harness._SNAPSHOT_IGNORE — the one DESTRUCTIVE consumer — take the full union. Three of the added names (MEMORY.md, STATE.md, REJECTED.md) have no live writer in core/, so their only real-world referent is a capability file that shares the name, and snapshot() silently deleted it from the candidate, from every descendant iteration, and WITHOUT busting the eval-cache key (cache ignored the same name) — a stale hit on a mutilated candidate. Regression vs main. - rundir: SCRATCH_NAMES (live writers) vs LEGACY_SCRATCH_NAMES (no writer, filter-only) + NON_CAPABILITY_NAMES union. Destructive consumer takes the subset; every read-side filter takes the union. - rundir.snapshot: root-anchored ignore callable. shutil.ignore_patterns matches by basename at EVERY depth; every entry is a root-level framework injection, so a nested src/prompts/STATE.md can no longer be caught. - cache.hash_candidate_dir: root-anchored too, so deleting a nested colliding capability file DOES change the key (closes the stale-hit hazard at the root). - dashboard._DIFF_SKIP + harness._CAP_DIFF_SKIP now derive from NON_CAPABILITY_NAMES — they are read-side filters asking the same question, and were the last hardcoded copies (review finding #2). - test_gepa: pin the live/legacy split and that all five read-side filters equal the union; new test_snapshot_ignore_excludes_legacy_names_and_is_root_anchored; drop the IndexError-prone parts[] check subsumed by the exact-set assert.
…mory->prompt framing Review fixes for #212: 1. cache.py — restore main's intro paragraph verbatim (the sentence #211 rewrites) and confine the correction to a separate "Scope: GEPA only" paragraph, so the #211 conflict is a single textual hunk whose wrong resolution can no longer restore the false maybe_cached_score line — that line's removal now auto-merges outside the conflict region. 2. test_w1_engine.py — new guard pinning that no doc under core/ or skills/ cites maybe_cached_score, so a revert of the docstring fix fails a test. 3. skillopt SKILL.md / concepts.md / skillopt.py — the rejected/history jsonl are dashboard audit records, write-only, never prompt input. 4. MemoryPanel.tsx — drop the "do-not-re-propose" framing this PR disproved.
A filter may legitimately add its own read-context names (post-#199 the cache and component lists fold in optimizer_context.INJECTED_NAMES); the invariant that matters is that none of them DROPS a shared name.
…y proposal Closes #128. A compact, bounded, framework-synthesized "what we've learned so far" block that survives across iterations independent of the transcript, and reaches the optimizer prompt for ALL THREE deterministic algorithms. What the insight is (three sections, all derived from the run's own numbers): * What HELPED — gate-ACCEPTED iterations, largest val |Δ| first, with the exact tasks each one fixed (and any it broke anyway). * What HURT — gate-REJECTED iterations, largest |Δ| first, with the reject reason and the exact tasks broken. * Still OPEN — the val tasks the current best does NOT pass. ZERO LLM calls. The synthesis is pure Python over events.jsonl + the persisted rollouts, like every other auxiliary step in core (#132/#205 established that profile). It reads RunDir.iteration_events() and _candidate_task_impact() — the same reads LEDGER.md uses, so the two artifacts can never disagree — and #205's aux_model tier is unused: the signal is already fully determined by the run's numbers, so a per-iteration model call on every run would buy prose, not information. Built on the #199 seam exactly as its reviewer predicted: the whole wiring is two lines in harness._augment_instructions, the ONE function whose output reaches the prompt and which all three algorithms route through (the note #212 left in memory.py). Honesty. Every line is labelled a CANDIDATE PRIOR, a hypothesis worth re-testing; nothing bypasses the val gate, and the prompt says so. Only val rewards and val task ids are read — never the sealed test split, never a gold answer. Growth/eviction. The block is RE-DERIVED every iteration, so it does not grow monotonically — but its input does. Eviction keeps the 6 largest |Δ| movers per section (ties → newer iteration) plus 10 open task ids, because a +0.25 accept and a -0.20 regression are the priors worth re-testing while a -0.001 reject carries no signal. MAX_INSIGHT_CHARS = 4,000 is the backstop. Measured: 1,953 chars after 200 iterations with long task ids — 3.3% of #199's MAX_INSTRUCTIONS_CHARS. INSIGHTS.md is framework read-context, not capability, so it is excluded from the candidate snapshot, GEPA's editable components, the eval-cache content hash (or every iteration would miss the cache, since the block changes as the run progresses) and SkillOpt's applied-edit count. Added to optimizer_context.INJECTED_NAMES — the one list #199 created for exactly this — plus the two diff-skip lists, and SkillOpt's two copy-pasted scaffold tuples were folded into one _SCAFFOLD frozenset (they are how this would have silently registered as an applied edit every iteration). Also fixes a #199 defect this surfaced: SkillOpt logs BOTH "step" (via run_step) and its own "skillopt_step" for the SAME candidate, so RunDir.iteration_events returned two rows per SkillOpt iteration — LEDGER.md, RUNMAP.md, the dashboard lineage and the new priors all double-counted it, the second copy missing parent/parent_val and so showing a blank Δ. Deduplicated by candidate id at the root (first occurrence wins, later records merged in for fields it lacks), so all four consumers are fixed at once rather than the one this PR happens to add.
…lineage exhaustion Closes #130 The engine stopped on budget (iterations/metric-calls/USD) and on `stall` (N consecutive non-accepts). Neither answers "is the search still making progress, or grinding a dead region?" — `stall` counts rejections, and a rejection is not the same thing as a lack of progress. New `core/cap_evolve/plateau.py` adds a third, orthogonal stop condition that escalates warn -> diversify -> stop, plus per-lineage exhaustion. The criterion counts DEAD iterations, not rejections: an iteration is dead when it neither raised the global best nor moved the score in the right direction. A near-miss (rejected but delta>0 — sub-significant, and made strictly more common by #113's Student-t small-sample correction) RESETS the streak, as does a new best. A ratchet caps escalation at `diversify` whenever the streak contains any accept, so a GEPA run still widening its Pareto frontier is nudged, never killed. Reads the per-iteration history through `RunDir.iteration_events()` (#109/PR199), never by hand-filtering `kind == "step"` — GEPA emits `gepa_val_gate` and would otherwise be invisible. Locally-killed GEPA children (`gepa_local_gate`, passed=false) are counted too: they are spent iterations that never reach the val gate. Wired into all three deterministic loops; `plateau` / `lineage_exhausted` events surface in the dashboard (summary + terminal + SPA) and in `--follow`. Config via four `capevolve.yaml` keys threaded to every algorithm through one CLI loop. Zero new runtime deps. 28 new tests in core/tests/test_plateau.py.
…ts (#129) Rejected candidates were persisted to rejected.jsonl (audit + the dashboard's "what not to try" panel) but nothing put them in a proposal prompt, so the optimizer could — and demonstrably did — re-propose an approach the gate had already killed, burning a full-val eval on a known dead end each time. RUN.md nonetheless claimed "rejected approaches are remembered and never re-proposed": false on both halves. This closes the feature gap and the honesty gap together. What changed - harness.approach_signature(parent_dir, cand_dir) — a stable, compact signature of WHAT an edit changed, built from the capability diff (whitespace-collapsed added/removed lines per touched file). Cosmetic variants of the same idea collapse to one signature; a no-op edit yields "". - harness.dead_end_constraints(run_dir) — the "ALREADY TRIED & REJECTED" block: deduped signature + gate reason + a repeat count, with an explicit "do not re-propose, and if you revisit one, state in PROCESS.md what is materially different" instruction. - Wired into _augment_instructions, the ONE function whose output reaches the optimizer prompt (#114) and which all three algorithms route through — so hill-climb, GEPA and SkillOpt get it with no per-algorithm plumbing. - RejectedMemory.add gains an optional `approach` field; every rejection site (run_step, GEPA's local + val gates, GEPA's two merge gates) now records it. Bounding (zero LLM calls — pure Python, per PR #205) - <= 12 most-recent DISTINCT approaches, signature <= 300 chars (capped on write AND on read), reason <= 200 chars -> block stays ~1 KB and is provably < 8 KB even with 50 long rejections. - optimizer_context.cap_instructions is extracted from render_instructions so the FINAL assembled prompt — not just the rendered half — is held under MAX_INSTRUCTIONS_CHARS (60k). Previously these cross-iteration blocks were appended after the cap had already been applied. Enforcement is ADVISORY, stated as such The optimizer is a black-box agent CLI, so cap-evolve cannot forbid it from re-emitting an edit. What is HARD is the val gate — a re-proposed dead end is still rejected, and the repeat is counted in the block ("re-proposed 4x"). RUN.md and docs/COMPARISON.md now say exactly this instead of implying a guarantee. Drive-by root-cause fix: one scratch-file list, four consumers Building the signature exposed that cache._IGNORE_NAMES, gepa._NON_COMPONENT, harness._CAP_DIFF_SKIP and dashboard._DIFF_SKIP each kept their OWN copy of "framework scratch, not capability", and they had drifted — only GEPA's knew about FOCUS.md/REFLECTION.md. So a GEPA candidate's "capability diff" (shown in the dashboard, RUNMAP and prior_iterations) reported its reflective scratch as a real edit. Unified as optimizer_context.SCRATCH_NAMES. Dashboard: the Insights dead-ends panel now shows the rejected edit signature under each reason (same data, one new field). Closes #129
… no dedup) SkillOpt routes through harness.run_step, which already logs "step" for the same candidate carrying parent/Δ/reason; "skillopt_step" only adds epoch/edit_budget/ applied_changes. Counting both double-counted every SkillOpt iteration in LEDGER.md / RUNMAP.md / INSIGHTS.md, the second copy missing parent/parent_val (blank Δ) and poisoning _parent_map. The id-keyed dedup this branch first shipped is REVERTED: SkillOpt mints ids from epoch/step counters that reset on --resume (the algorithm's --resume restores only current_val), so a resumed run re-emits so_e01s01 for a different candidate and first-wins dedup DISCARDED it — regression included. Measured on one live resumed run: 9 rows on the #199 base (double-counted but present), 4 with the dedup (a resumed reject silently gone), 5 correct after this change. iteration_events() is now documented as one row per logged event in log order with no dedup; consumers must key history by position, not candidate id. The dashboard folds skillopt_step's epoch onto the lineage node in a separate pass so the audit metadata is not lost. Also in the durable priors block (#128): - the char cap now RESERVES the truncation notice's length before cutting, so the output is bounded inclusive of the notice (it previously overshot by 98 chars on a long reject reason or long unicode ids), and the pinning test now actually crosses the bound; - broke/fixed task sets render an honest "+N more" count instead of a silent cut at 8, so INSIGHTS no longer under-reports a 20-task regression as 8 while LEDGER shows 20 (LEDGER's own [:20] cut is marked the same way); - gate reasons are flattened and markdown-escaped, so a reason cannot forge a "## What HELPED" section inside a framework-authored block, and are bounded to 200 chars so one reason cannot eat the block's budget; - "Still OPEN" leads with a count (N of M val tasks) and carries an explicit anti-overfit instruction: the names are a diagnostic, not a target list, and a task-specific special case passes the val gate and fails the sealed test; - rejects now render what they FIXED as well as what they broke, and the heading is "What was REJECTED ... a reject is not necessarily a regression" rather than "What HURT", which mislabelled positive-Δ significance rejects; - accepted rows evict by RECENCY (every row already cleared the gate, so |Δ| just froze a top-6 and permanently evicted small-but-real effects); rejected rows still evict by |Δ|, which is the damage signal; - an absent-rollouts "Still OPEN" is reported as UNKNOWN rather than rendering identically to a perfect candidate; - a missing Δ renders "Δ ?" instead of a fake "+0.000"; - the workdir copy is written atomically like the durable one, and the false docstring claim that the dashboard reads INSIGHTS.md is removed.
#212 narrows the signature (drops rejected/history). Passing them positionally made this a THIRD mechanical merge site — and unlike the two in harness.py it surfaced as a test failure git does not flag as a conflict. Fill trailing params reflectively so the test passes on both signatures.
Fourth mechanical merge site for #212's narrowed signatures (two in harness.py, two here) — all in tests, so git flags none of them as conflicts. Also refresh the module docstring for the renamed sections and the new pinned properties.
… edit (#129 review) Review of #222 found the constraint block was actively misdirecting on every REAL agent optimizer: it told the optimizer "you already tried this, re-proposed 2x" about approaches it had never proposed. Four blocking fixes, all reproduced fail-before/pass-after. B1 — the signature was dominated by framework-injected read-context. A capability diff compares a SNAPSHOT parent (INJECTED_* already stripped by _SNAPSHOT_IGNORE) against the LIVE workdir (not stripped), so every injected CLAUDE.md / .claude/skills/<x>/SKILL.md read as a capability ADDITION, sorted to the front, and truncated the real edit away entirely. _CAP_DIFF_SKIP omitted INJECTED_NAMES and _capability_files filtered a hardcoded 3-of-9 subset of INJECTED_DIRS. Both sets are now derived, never enumerated — in harness, dashboard and skillopt alike. Only `mock` (no registry skills_dir) hid this, which is why 13/13 tests passed over a broken path; the new test runs run_step with optimizer_name="claude-code". B2 — head-first truncation made the signature not a function of the edit: two different edits sharing a long prefix (any realistic SKILL.md or system prompt) collapsed to one signature. Overflow now closes with a sha256 digest of the whole normalized body, so it stays stable under cosmetic variation AND distinct whenever the bytes differ. B3 — eviction was by FIRST appearance, so the single most predictive row (a dead end the optimizer JUST re-proposed) was dropped while newer one-offs were kept, contradicting the block's own "12 most recent" wording. A repeat now requeues its row. B4 — skillopt._changed_components was a fifth, pre-drift copy of the scratch list. It now derives from the one shared definition like the other four. SCRATCH_NAMES end state (rebased onto #211): rundir.SCRATCH_NAMES / LEGACY_SCRATCH_NAMES / NON_CAPABILITY_NAMES is the sole definition, split by OPERATION so the one destructive consumer never takes a retired name; optimizer_context.SCRATCH_NAMES is deleted. The five read-side filters compose NON_CAPABILITY_NAMES with INJECTED_NAMES/DIRS. Also: control + bidi-override chars stripped from the signature once (N3); the row shows "(latest <cid>)" so a repeat count cannot be misread as the first candidate's (N2); a frontend test for the new `approaches` field (N4); N5 noted with its upgrade path; the function-body cap_instructions import, the <8 KB bound comment and the 70/30 tail-slice docstring corrected (nits).
Review fixes for PR #221. Two of the four blocking findings made the feature kill or degrade productive runs, which is the failure direction this module exists to prevent. 1. SkillOpt iterations were counted TWICE. `harness.run_step` logs `step` for a candidate and `skillopt.py` then logs `skillopt_step` for the SAME candidate; both were in the iteration-kind list, so a window=6 ladder reached `stop` at iteration 5 of a productive run, and a genuine near-miss was recorded once alive and once dead (the duplicate carries no parent_val). Dropped `skillopt_step` from the pre-#199 `_FALLBACK_KINDS` literal. PR #219 owns the root fix in `rundir.ITERATION_EVENT_KINDS`; no second dedup layer here, and deliberately NOT id-keyed dedup, which drops resumed SkillOpt iterations. 2. `NameError: cannot access local variable 'why'` in `hill_climb_loop` when the loop body never runs (max_iterations=0, or a resume with no budget left). `why` is now initialised next to `steps`. 3. The plateau block escaped the prompt cap and silently lost data. It now travels as `extra=` through `_augment_instructions` / `run_step`, so it is inside #222's MAX_INSTRUCTIONS_CHARS (measured 64941 -> 2625) and lands in the TAIL that truncation preserves — it is the only behavioural block of the three, so it must not be the first casualty. Removed the bare rejected-ids line entirely: it called `RejectedMemory.entries()` (removed by #199) behind a bare `except` that swallowed the AttributeError, and duplicated #222's richer signature channel. `prompt_block` no longer accepts `rejected=`, so a caller passing one now fails loudly. Bounded `exhausted_lineages` to 6 ids, 60 chars each. 4. GEPA was steered away from its BEST lineage. `exhausted_lineages` reused `_dead()`, which counts accepted-but-not-global-best as dead, so a lineage whose children were ALL accepted was dropped from the Pareto sampling pool — with no ratchet on that path and no stop event. Per-lineage exhaustion now uses its own narrower `_lineage_dead()`: an accept is never dead ground for a lineage, because widening the per-instance frontier is GEPA's mechanism. The accepted-but-not-best clause stays at the GLOBAL level, where the ratchet makes it safe. Also, from the non-blocking findings: - The ladder was unreachable at the shipped defaults: `stall: 2` stopped every run at 3 iterations, before `plateau_window: 6`. `stall` now defaults to 0 (off) with the reasoning in the template, so the shipped product actually runs the ladder. All evidence is re-run at the shipped default. - The ratchet now caps escalation at `warn` rather than `diversify`. A run still clearing the honest val gate every iteration gets a warning and no behavioural intervention, so the diversify block can never tell an optimizer a lineage failed when it was accepting. - The reason string no longer claims "no near-miss in that streak" when the streak is all accepts. - `series` carries the real signed delta for `gepa_local_gate` rows instead of hardcoding 0.0, so a tie is distinguishable from a regression. Deadness is unchanged (gepa's pass condition is a strict `>`). - An explicit `plateau_window: 0` now means off instead of silently reverting to 6. - Plateau state reaches the React dashboard: `plateau_level` on the hub row, `plateau` / `exhausted_lineages` typed on the detail summary, and a KPI tile. It is a separate field from `status` on purpose — that one is liveness, this one is progress. - Documented the honest cost of the delta<=0 rule (N regressions then a breakthrough is stopped at N) and that `--resume` deliberately carries the streak. Tests: 215 core (was 207) + 44 dashboard backend. 8 new regression tests, one per blocking finding plus resume, the bounded block, and the local-gate delta. `test_plateau_block_reaches_the_prompt_inside_the_cap` asserts the block reaches the prompt, so the #199 prompt-assembly merge trap fails loudly instead of compiling clean with the feature dead.
… into compose-115
…into compose-115 # Conflicts: # core/cap_evolve/__init__.py # core/cap_evolve/gepa.py # core/cap_evolve/harness.py
…o compose-115 # Conflicts: # core/cap_evolve/dashboard.py # core/cap_evolve/harness.py
…ly-memory' into compose-115 # Conflicts: # core/cap_evolve/gepa.py
…re' into compose-115 # Conflicts: # core/cap_evolve/cache.py # core/cap_evolve/dashboard.py # core/cap_evolve/gepa.py # core/cap_evolve/harness.py # core/cap_evolve/skillopt.py
…nto compose-115 # Conflicts: # core/cap_evolve/cache.py # core/cap_evolve/dashboard.py # core/cap_evolve/harness.py # core/cap_evolve/skillopt.py
…into compose-115 # Conflicts: # core/cap_evolve/dashboard.py # core/cap_evolve/harness.py # core/cap_evolve/skillopt.py
…' into compose-115 # Conflicts: # core/cap_evolve/__init__.py # core/cap_evolve/dashboard.py # core/cap_evolve/gepa.py # core/cap_evolve/harness.py # core/cap_evolve/skillopt.py # skills/algorithms/gepa/scripts/run.py # skills/algorithms/hill-climb/scripts/run.py # skills/algorithms/skillopt/scripts/run.py
`harness.py` had accreted eight unrelated responsibilities behind one import path. On the composed base (six merged-pending PRs) it was 2,526 lines, and it is the module every algorithm change has to touch — which is exactly how the under-wiring bugs #109/#110/#114 hid in it. Split along the seams the module already had, extending the direction #199 (optimizer_context.py) and #221 (plateau.py) started: optimizer_proc.py (110) optimizer subprocess plumbing + cost parsing evaluate.py (323) per-task rollout loop, aggregation, paired deltas capdiff.py (136) capability snapshot reads/diffs + per-task impact insights.py (224) INSIGHTS.md durable synthesized priors (#128) handover.py (520) LEDGER/JOURNAL/PROCESS/RUNMAP + dead-end constraints context_inject.py (316) FILE side of the optimizer-context seam instructions.py (458) prompt templating: failure index, briefs, notes step.py (254) run_step — the SHARED propose->gate step harness.py (482) run lifecycle: splits, baseline, hill-climb loop, finalize The shared/hill-climb-specific boundary the issue asked for is now explicit: `run_step` (all three algorithms drive it) lives in `step.py`; `hill_climb_loop` is hill-climb-only and stays in `harness.py` — deliberately in the same namespace as the `run_step` it calls, because `test_plateau` monkeypatches `harness.run_step` and must still reach the loop. Behaviour-preserving: all 68 pre-split top-level symbols were moved with byte-identical source. No signature changed. Every one of the 97 names previously importable from `cap_evolve.harness` and 55 from `cap_evolve` still imports (re-exported), so no caller in core/, skills/ or dashboard/ changed. Verified: identical test count (357 before, 357 after), compileall clean, no module-level import cycles, `rundir.py` still stdlib + `.splits` only, and toy_calc run end-to-end with the mock optimizer on all three algorithms — every non-timestamp artifact byte-identical.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| assert "MATERIALLY DIFFERENT" in blk | ||
| assert "rejected candidates" not in blk | ||
| with pytest.raises(TypeError): | ||
| plateau.prompt_block(st, rejected=object()) # type: ignore[call-arg] |
| if "optimizer_context_warning" in json_line and "mb_p_0007" in json_line] | ||
| assert warn, "unmet trajectory pin failed SILENTLY (no warning event)" | ||
| assert "OMITTED" in warn[0] | ||
| assert kinds == kinds # no-op; keeps the reader exercised without asserting count |
| "**Constraint:** do NOT propose any of the above again, and do not propose a " | ||
| "cosmetic variation of one (same text, different wording/placement) — it shares " | ||
| "the same hidden assumption and will fail the same way. If you believe a rejected " | ||
| "direction is still right, you MUST state in `PROCESS.md` what is materially " | ||
| "different this time and which specific lesson above it counters. Otherwise pick " | ||
| "a genuinely different hypothesis.", |
| "A compact, continually-updated summary of what this run has LEARNED SO FAR, " | ||
| "re-derived from the objective record every iteration so it survives even when " | ||
| "the transcript does not. Read it BEFORE proposing.", |
| "**These are CANDIDATE PRIORS, not truth.** Each one is a hypothesis worth " | ||
| "re-testing, and every edit you make is still judged by the val significance " | ||
| "gate — a prior can be wrong, and acting on one earns no exemption.", |
| lines += ["", "## What was REJECTED by the gate (largest movers first — a reject is " | ||
| "not necessarily a regression; read the reason)"] |
| "The capability under optimization is composed of these editable artifact(s). " | ||
| "Use the FULL edit space below — do not limit yourself to trivial wording tweaks."] |
| "**These names are a DIAGNOSTIC of where the capability is weak, not a " | ||
| "target list.** Fix the general defect they expose; your edit must generalize " | ||
| "beyond them. The gate runs on val, so a task-specific special case for these " | ||
| "ids will pass the gate and FAIL the sealed test — that is a val overfit, not " | ||
| "progress.", ""] |
| "(A task that merely had one errored trial but still mostly PASSES is NOT listed " | ||
| "here — it is solid/flaky and must be protected, not ignored.)", |
| "# Optimize the capability — analyze this step's trajectories in ./trajectories/, " | ||
| "then fix MANY root causes in this ONE candidate and STOP.", |
|
❌ Automatic Labeling Failed An error occurred while trying to automatically label this pull request. Please check the workflow logs for details and add labels manually. |
🔬 EvidenceEvery command and its full literal output. Run in a worktree at the composed base 1. The composed base2. Baseline test count BEFORE the splitThe one earlier failure on this base was the known #200 flake, confirmed environmental: 3. Test count AFTER the split — identical4. compileall5.
|
🔍 Review — PR #235Verdict: APPROVE WITH NITS The "zero behaviour change" claim holds up under mechanical verification. I rebuilt the composed base independently, extracted and compared all 68 moved symbols myself, probed every new module standalone, and ran all three algorithms pre/post with artifact comparison. One real regression found — a dropped import that Blocking(none) Non-blocking1.
I swept every function and class in all nine modules for this class of defect; this is the only occurrence: Consequence: strictly a regression against the base — Fix — one line in from .loop import SplitResult
from .rundir import SCRATCH_NAMES, RunDirVerified: Worth also considering a cheap guard so the next split cannot repeat this — a test that walks 2. The author's claim checks out. and extra = len(inspect.signature(harness._augment_instructions).parameters) - 3
out = harness._augment_instructions("BASE", wd, rd, *([None] * max(0, extra)))Making Consequence: latent trap for the next refactor. Fix: one line in the Nits3. Cosmetic and pre-existing in the untouched files, but Is it byte-identical?Yes — 68/68, verified by my own AST extraction, not by trusting the claim. I parsed the base No symbol was edited during the move. The one duplicate is pre-existing, not introduced: I also checked the derived import-time sets, since #199's review found an import-time set mutation in exactly this class of code. All module-level work in the new modules is pure constant construction (frozensets, seed strings, Import surface grows 18 → 26 Import-order probesEvery new module imported first, in a fresh interpreter, with nothing else pre-loaded. No cycle manifests under any import order: The import DAG among the new modules is genuinely acyclic and shallow — The one Re-export facade: verified clean. All 97 names still resolve from The 8 additions to Did the traps stay avoided?Yes — verified on running code, not by reading. The trap's signature is that it compiles clean with the feature silently dead, so I captured every prompt actually handed to the optimizer and grepped for #221's literal banner. With an aggressive Identical counts on the base (8 / 3), so the split changed nothing. GEPA never organically reached All three algorithms carry the block, and it lands in the preserved tail — which is the property Seam quality
That is not one responsibility. (2) is already visually a separate module — it shares no state with (1), and its natural home is next to The
Recommended follow-up issue: extract #129's constraint machinery out of The left-behind
|
…lution (#115 review) Review fixes for PR #235. All four changes are non-behavioural. 1. step.py — restore `from .loop import SplitResult`, dropped when `run_step` moved out of harness.py (base had it at harness.py:34). `from __future__ import annotations` hid the break from compileall, from imports, and from all 357 tests: it only surfaces on annotation *resolution* (`typing.get_type_hints(run_step)` -> NameError). 2. test_core.py — new `test_every_public_annotation_resolves`: walks every module in `cap_evolve` and calls `get_type_hints` on every function, class, and method defined there. Verified to FAIL on the unfixed tree and pass after, so it is a real trip-wire, not a tautology. This is the one defect class a byte-identical refactor can introduce that no existing check catches. 3. handover.py — docstring line on `_augment_instructions` naming why `extra` must stay POSITIONAL (test_insights.py fills trailing params by arity), so the constraint is visible from the side someone would edit. Docstring only; AST identical ignoring docstrings. 4. capdiff.py — two stale doc refs to the moved `_SNAPSHOT_IGNORE` (`hillclimb.`/`harness.` -> `step.`). Comment only. Verified: annotation sweep 0 broken across all nine new modules; 68/68 symbols still byte-identical (the one flagged delta is the intended handover docstring); 97/97 facade names resolve; all nine modules import standalone with no reach-back into harness; #221's plateau block reaches the prompt LIVE on hill-climb (1/5) and GEPA (6/6 forced), landing in the preserved tail; compileall clean; no dist/ churn. 357 passed (358 collected; the one failure is the known #200 port-7878 flake). Known follow-ups, deliberately NOT in this PR: protect.manifest_digest's path-dependence (protect.py is not in this diff) and extracting #129's ~160-line constraint machinery out of handover.py. Refs #115
🔧 Review fixesCommit 1.
|
Closes #115
core/cap_evolve/harness.pywas a god module mixing eight unrelated concerns behind oneimport path. It is the module every algorithm change has to touch, which is exactly how
the under-wiring bugs (#109, #110, #114) hid in it: the shared machinery was never cleanly
separated from the hill-climb-specific loop.
This is a pure, behaviour-preserving refactor. No signature changed, no side effect
reordered, no "while I'm here" fix.
Module map — what moved where, and why that seam
optimizer_proc.pyOptimizerFnfrom a shell command; parse the CLI's self-reported cost; render a non-zero exit into an actionable messageevaluate.pysplit_result_from_rollouts,_paired_deltascapdiff.pybroke/fixed)rundir.NON_CAPABILITY_NAMESexists to prevent.insights.pyINSIGHTS.md, the durable synthesized priors (#128)handover._augment_instructions), and its bounds (MAX_INSIGHT_CHARS, the+N moremarkers) are now auditable in one place.handover.pyprior_iterations/, marker-guarded journal reconciliation, #129 dead-end constraints, and_augment_instructions_augment_instructionsis the ONE function whose output reaches the optimizer prompt, and all three algorithms route through it.context_inject.py.claude/skills/,CLAUDE.md, …)optimizer_context.render_instructions(the PROMPT side). Separating the two sides makes #109's "same context for every algorithm" invariant checkable rather than aspirational.instructions.pystep.pyrun_step— ONE propose→gate steprun_stepis SHARED: hill-climb, GEPA and SkillOpt all drive it, so a change here changes all three._SNAPSHOT_IGNORElives here becauserun_stepis its only caller and it is the engine's single DESTRUCTIVE name filter.harness.pyensure_splits,baseline/reuse_baseline,finalize) +hill_climb_loop+ the re-export facadeoptimizer_context.py(#109) andplateau.py(#221) already existed; this extends thatdirection rather than inventing a new architecture.
Why
hill_climb_loopstayed inharness.pyIt is hill-climb-ONLY, so
step.py(shared) is the wrong home — but it also could not moveto a new
hillclimb.py:core/tests/test_plateau.pydoesmonkeypatch.setattr(harness, "run_step", fake_run_step)and then callsharness.hill_climb_loop(...). That patch only reaches the loop if the loop resolvesrun_stepfromharness's own namespace. Keeping the loop inharness.py(next to theother lifecycle entry points) preserves that exactly; a re-export would have silently made
those two tests exercise the real
run_step. Same reasonimport subprocessis retained inharness.py:test_budget_cost.pypatchesharness.subprocess.run, and it must stay thesame module object
optimizer_procuses.Before / after
wc -lNo file is left absurdly large; the largest (
handover.py, 520) is under the issue's<600-line soft guideline, and
harness.pywent 2,526 → 482.Exact composed base + resolutions used
Per epic #127 this lands LAST in the
harness.pycluster, so it is based on the composedtree, not
main. Base:origin/main@e47a8e15, then merged in the agreed order:fix/issue-109-optimizer-context@729a79ad49c539a3feat/issue-142-protected-paths@887349ca3943a9e3fix/issue-113-small-samples@53f1a1f28d26978crefactor/issue-114-drop-write-only-memory@5e31cf409f3dd09dfix/issue-110-gepa-snapshot-ignore@7d6ed65c32bb9980feat/issue-129-failure-memory@6d6cc52d9fddd881feat/issue-128-persist-insight@a814e6bba5af3f84feat/issue-130-plateau-detection@5d530019fad048d8Composed baseline =
69483ee3(tagged locallycompose-base-115). The split is the singlecommit
d67639b9on top of it.Conflict resolutions:
__init__.py,gepa.py,harness.py) — kept bothsides, as the reviews advised.
gepa.py— the obvious "keep both" compiles cleanwith feat(algorithm): plateau/convergence detection with escalation + per-lineage exhaustion #221's plateau feature silently dead. Resolved by keeping
render_instructions(...)(fix(algorithm): give GEPA & SkillOpt the same optimizer context as hill-climb, un-gate the CLI flags #199) and not reintroducing_instructions(...)(deadafter refactor(core): drop dead optimizer-memory API + unused params; fix misleading cache docstring #212), and keeping feat(algorithm): plateau/convergence detection with escalation + per-lineage exhaustion #221's
extra=plateau.prompt_block(pstate)so the plateaublock still reaches the prompt. Verified live:
test_plateau.pyasserts"MATERIALLY DIFFERENT"reaches the assembled instructions and both tests pass._augment_instructions/_build_ledgerlost theirrejected, historyparams; Durable synthesized priors (INSIGHTS.md) fed to every proposal, all three algorithms #219 and feat(algorithm): re-inject rejected approaches as optimizer constraints (#129) #222 both use the narrowed form (they read the rundir instead). feat(algorithm): plateau/convergence detection with escalation + per-lineage exhaustion #221's
extrawas appended to the narrowed signature and keptpositional (
(instructions, workdir, run_dir, extra="")) — making it keyword-onlybroke
test_insights.py's signature-agnostic call, which is a test failure git does notflag as a conflict.
_CAP_DIFF_SKIP— derived asNON_CAPABILITY_NAMES | INJECTED_NAMES, neverhardcoding a single name; the same derived form was applied to the sibling filters
(
dashboard._DIFF_SKIP,skillopt._SCAFFOLDING,cache._IGNORE_NAMES,gepa._NON_COMPONENT) plus their*_DIRScompanions, which is what fix(algorithm): one shared scratch-file list so GEPA snapshots are clean (#110) #211'ssuperset assertion requires.
skills/algorithms/*/scripts/run.py— feat(algorithm): plateau/convergence detection with escalation + per-lineage exhaustion #221 (based onmain) re-added the raw--capabilities/--instructions-file/...flags that fix(algorithm): give GEPA & SkillOpt the same optimizer context as hill-climb, un-gate the CLI flags #199 hadreplaced with
OptimizerContext.add_arguments(p). A naive keep-both registers each flagtwice and argparse raises at runtime. Kept
add_arguments+ feat(algorithm): plateau/convergence detection with escalation + per-lineage exhaustion #221's four--plateau-*flags only; all three
run.py --helpverified.0(tamper) and0b(honesty), preserving the tamper block's trailing blank line.
Findings (reported, deliberately left)
protect.manifest_digesthashesproject_dir(core/cap_evolve/protect.py:266), sothe
protected_manifestevent digest is not reproducible across two runs of the sametree at different paths. Not wrong for its stated purpose (it makes forging
protected.jsonrequire forging the event too) but it does mean the digest is not a purefunction of the protected content. Out of scope for a pure refactor — left as-is, and
it is why the first before/after comparison showed a digest diff until the runs were
re-done at identical paths.
test_dashboard_launch.py::test_maybe_launch_spawns_when_availableis the known test_maybe_launch_spawns_when_available asserts a hard-coded port and fails whenever 7878 is in use #200environmental flake (it fails when any other process holds port 7878). It failed once on
the composed base while a sibling agent held the port and passed in the same tree once
the port freed — no relation to this change.
Verification
Identical test count — the bar for a pure refactor
357 → 357. No test added, removed, skipped or newly xfailed.
Moved bodies are byte-identical
An AST pass over the pre-split file vs. the nine post-split files:
(The raw scan reported one "changed" for
OptimizerFn— a false positive: it matchedgepa.py's own pre-existingOptimizerFnalias first.optimizer_proc.OptimizerFnisbyte-identical to the original.)
compileallcleanPublic API intact
Every
from .harness import/from cap_evolve.harness import/from cap_evolve import harnesssite acrosscore/,skills/anddashboard/was enumerated and re-resolved(49 sites, including
gepa.py's 8-name list andskills/phases/implement-and-check/scripts/pipeline_selftest.py's_focus_instructions).Not one caller file changed.
No import cycle
rundir.pyis untouched and still at the bottom of the graph (stdlib +.splitsonly).Each of the 18 modules also imports cleanly as the FIRST import in a cold interpreter.
Layering (module-level
.imports only):Real end-to-end, zero API cost — all three algorithms
examples/toy_calcviacap-evolve runwith themockoptimizer (deterministic, nomodel calls), before and after the split. Because
protected.jsonrecords the run's ownproject_dir(and the manifest digest hashes it), the comparison was re-done with bothtrees writing to identical paths so the only variable is the code:
splits.json,report.md,INSIGHTS.md,rejected.jsonlandhistory.jsonlarebyte-identical in all three;
final.json/baseline.jsondiffer only inseconds.Same file set, same headline numbers, same winning candidate id. The residual "REAL-DIFF"
entries were then diffed structurally and are wall-clock only — every single field is a
secondsfloat, aplateau.tepoch timestamp, or a git short-SHA (timestamp-derived):skillopt'sevents.jsonlis 44/44 lines with['optimizer_seconds', 't']the onlyfields that differ anywhere in the file.
All
rollouts/and allcandidates/are byte-identical.All the feature blocks the cluster added are still produced after the split:
protected_manifest/protected_paths_unmatched(#142) andlineage_exhausted(#130)both still fire, and GEPA still emits
gepa_local_gate/gepa_select/gepa_val_gate.dashboard/frontend/dist/untouched (#188):git status --short -- dashboard/frontend/distis empty.