Add the cheap first-real-run on-ramp between toy_calc and the ~$148 tau2 run (closes #124) - #249
Add the cheap first-real-run on-ramp between toy_calc and the ~$148 tau2 run (closes #124)#249OsherElhadad wants to merge 2 commits into
Conversation
…au2 run
There was a cliff in onboarding: `toy_calc` is free but calls no model, and the next
advertised step was the tau2 airline run — hours and ~$148. A newcomer who wanted to
see cap-evolve work on a real LLM had nothing between "no model at all" and "commit an
afternoon and $148".
`examples/cheap_real` is the missing rung: a REAL run (real LLM, real paired gate, real
sealed test split) in minutes, for $0 on a local model or cents with a hosted proposer.
The task is date normalization ("the 22nd of November, 1963" -> 1963-11-22) scored by
exact match, so the seed's generic "helpful assistant" prompt answers in prose and
scores 0, and the fix is an output contract — the edit an optimizer is actually good at
proposing. The gain therefore comes from a general rule, not a memorized answer.
MEASURED, both rungs to a sealed test number of 1.0 from a baseline of 0.0:
* free (ollama/llama3.2:3b + mock proposer) 36 s, $0.00
* cheap (same runner + claude-code/Haiku) 321 s, $0.5427
The cheap rung's spend is entirely its 3 proposals ($0.159/$0.199/$0.185 from the run's
own opt_cost_usd); all 30 runner calls are free because the runner is local. The run
reported exactly the 30 metric_calls the documented derivation predicts. The
hosted-runner figure in the docs (~$0.21) is labelled an ESTIMATE with its arithmetic,
because I did not run that variant.
docs/GETTING_STARTED.md and site/getting-started.html now show all three rungs with
cost, runtime and derivation, so free -> cheap-real -> full is one visible ladder.
A standalone example, not a benchmark-zoo entry (#233), and deliberately: `verify`
step 5 runs every val task twice and fails on any rollout-fingerprint drift. That guard
is right and worth keeping, but no real LLM can satisfy it even at temperature 0 with a
fixed seed — and this example exists to call a real model. Admitting it would mean
weakening the zoo's determinism invariant for the one benchmark that structurally
cannot meet it. The reasoning is in the example's README so it is not re-litigated.
No new adapter code: run.sh copies templates/adapters/jsonl_litellm/adapter.py and
model_config.py verbatim, which is exactly what that bundled generic template is for.
No core changes at all.
Composes with the sibling work rather than duplicating it:
* #113 — the 20-task size is a FLOOR, not a preference. val 5 is the smallest split
clearing both MIN_VAL_TASKS=2 (gate.decide refuses below it) and
LOW_CONFIDENCE_VAL_TASKS=5 (below which decisions are branded low-confidence).
Verified against that branch, and pinned by a test rather than a comment.
* #142 — protected_paths is OMITTED, never []. run.sh puts the adapter,
model_config.py and tasks.jsonl inside the project dir and the spec names
dataset_source: tasks.jsonl, so the layout defaults cover all four grader files
without declaring anything (a declared list replaces the defaults).
* #132 — sets proposer_model AND optimizer_model to a cheap tier on purpose: the
latter is what ships today, the former is the tier name and falls back to it, so
both are correct before and after that merge. aux_model is left unset because
every auxiliary step is still pure Python and a tier would route nothing.
* #134 — nothing printed echoes an endpoint. The preflight reports an unreachable
local endpoint WITHOUT the URL, per the rule that a non-default base_url is
confidential.
Three bugs found while proving this works, each of which produced a clean-LOOKING run
whose numbers were meaningless:
1. A missing litellm made a broken run indistinguishable from an honest 0.0 — the
adapter correctly turns a failed call into reward 0.0, so a stopped Ollama yielded
baseline_val 0.0 -> test_reward 0.0 with no visible failure. run.sh now preflights
the interpreter, the endpoint AND whether the model is pulled, before spending.
2. optimizer_max_turns: 12 silently discarded every candidate — claude-code exits
non-zero on `Reached max turns`, correctly reported as a failed iteration, so the
run read as "the optimizer proposed nothing". Now 40, with the measured reason.
3. A relative optimizer_instructions_file silently falls back to the GENERIC
template, because cli.py resolves it against its own cwd. The optimizer got
tau2-flavored "edit the tool code" advice for a project with no tools and
optimized the wrong task entirely. run.sh writes an absolute path; the underlying
sharp edge in cli.py is noted rather than widened into this change.
Tests: 4 new preset invariants (split floors, protected_paths omission, dataset_source
names the real file, budget bounded) — the assertions a future "make it cheaper" edit
would break. Suite 179 -> 183, 0 failed. compileall clean; links resolve.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
|
|
||
| def test_split_clears_the_honest_gate_floors(): | ||
| from cap_evolve.splits import make_splits | ||
| import cap_evolve.splits as splits |
🔬 EvidenceEvery command below was run in a clean worktree of this branch ( 1. Free rung — real local LLM runner,
|
|
❌ Automatic Labeling Failed An error occurred while trying to automatically label this pull request. Please check the workflow logs for details and add labels manually. |
🔍 Review — PR #249CHANGES REQUESTED — one blocking defect, and it is the exact failure mode this PR BlockingB1 —
$ CHEAP_REAL_MODEL=claude-haiku-4-5 bash examples/cheap_real/run.sh
cheap_real preflight: OK
Working directory: /tmp/cr-hosted
Runner model: claude-haiku-4-5 Optimizer: mock
{
"run_dir": ".capevolve/run_cheap",
"best_id": "seed",
"baseline_val": 0.0,
"test_reward": 0.0,
"test_baseline_reward": 0.0,
"test_delta": 0.0,
"test_pass_k": {"1": 0.0, "2": 0.0},
"iterations": 2
}Exit 0. Empty stderr. Zero warnings. The reason is only visible if you go read a rollout $ cat .capevolve/run_cheap/rollouts/val/d05__cand_0001__t0.json
"error": "LLM call failed: litellm.UnsupportedParamsError: anthropic does not support
parameters: ['seed'], for model=claude-haiku-4-5. To drop these, set
`litellm.drop_params=True` ..."Consequence: this is precisely bug (a) from the PR body — "a broken run made Three separate places advertise this exact command as working: Note the PR body says of the hosted variant "I did not run this variant" — that is Fix (cheapest that holds, in export LITELLM_DROP_PARAMS=1 # litellm honours this; unsupported kwargs are droppedor, if you prefer to fail loudly rather than fix silently, extend the preflight past the Non-blockingN1 — N2 — N3 — N4 — N5 — N6 — untested paths. The 4 new tests are all static file reads of the preset (see Nits
Do the cost figures hold up?Yes — every number reproduces, and the accounting model is the right one. This is the 1. The 30-call derivation is arithmetically correct AND matches the spec's actual And I confirmed each of the six evals independently from my own run's $ grep -o '"kind": "evaluate"[^}]*' state events
"split": "val", "tag": "seed", "reward": 0.0
"split": "val", "tag": "cand_0001", "reward": 1.0
"split": "val", "tag": "cand_0002", "reward": 1.0
"split": "val", "tag": "cand_0003", "reward": 1.0
"split": "test", "tag": "FINAL", "reward": 1.0
"split": "test", "tag": "FINAL_seed", "reward": 0.0Six evals × 5 tasks × 1 trial = 30. The "best + baseline seed" term the PR asserts for 2. My own free-rung run reports $ cat /tmp/cr-canary/.capevolve/run_cheap/state.json | jq .spent
{"iterations": 3, "metric_calls": 30, "usd": 0.0, "stall": 2,
"runner_tokens": 3863, "runner_seconds": 29.5, "optimizer_usd": 0.0}3. The three per-proposal costs sum to the total, to 1.5e-07. $ python -c "print(sum([0.159017,0.198835,0.184818]))"
0.54267
$ # state.json optimizer_usd: 0.5426701500000001 → diff = 1.5e-07Exact to float rounding. The three 4. Is $0.5427 the total or only the optimizer share? Both — and that is verifiable, 5. The $0.21 hosted-runner ESTIMATE reproduces from the stated inputs. $ python -c "print(30*(3000*1.0+800*5.0)/1e6)"
0.21Correctly labelled ESTIMATE in the PR body, in 6. The one figure I could not check, and the one that is slightly off. Rung 3's ~$148 Verdict on the figures: they hold. The derivation is independently correct, the Is this a real optimization or a formatting fix?It is a real optimization, and I'll defend that against the obvious objection — but the The objection is fair on its face: the seed scores 0 only because it answers in prose, Why it is nonetheless real:
Where the PR oversells, and what it should have said: The two rejected iterations prove nothing about the gate, and the PR presents them as Compounding it: at Verdict: a real optimization on a task too easy to exercise the interesting half of the Was declining the zoo right?Yes, and the claim is accurate against #233's code — I verified it rather than taking for attempt in (1, 2):
...
got[t.id] = (_rollout_fingerprint(r), round(float(s.reward), 9))
passes.append(got)
drift = sorted(k for k in passes[0] if passes[0][k] != passes[1][k])
if drift:
rep.problems.append(f"NON-DETERMINISTIC: {len(drift)} task(s) produced a different
rollout or reward on an identical re-run with seed=0 ...")Every val task, twice, and Could a tolerance have accommodated it? Not cheaply. The natural knob is Where I'd push slightly: the PR frames this as binary ("the zoo keeps its invariant"), On consistency with #244: both declinations are on the same principle — don't weaken a The relative-instructions-path bugMechanism confirmed on
instr = spec.get("optimizer_instructions_file") or "optimizer/INSTRUCTIONS.md"
instr_p = Path(instr)
if not instr_p.is_absolute() and not instr_p.exists():
instr_p = Path(project) / instr # ← project is RELATIVE: ".capevolve/project"
if instr_p.exists():
alg_cmd += ["--instructions-file", str(instr_p)]
cwd = /private/tmp
workdir = /private/tmp/bugc/proj
project (rel) = .capevolve/project
instr_p abs? False exists from cwd? False
fallback = .capevolve/project/MY_INSTRUCTIONS.md exists? False
=> --instructions-file passed? FalseThen the failure is silent by construction, at two layers:
And Severity beyond what the PR claims: Workaround: acceptable here. File the issue. Suggested fix, one line: resolve against the absolute project dir Canary sweepMine, not the PR's. Six shapes across two secret-looking names, two innocent-looking Full free-rung run with all six exported, then swept every written file plus both $ grep -rIl -E 'CANARY-(AAAA1111|BBBB2222|CCCC3333|DDDD4444|EEEE5555|FFFF6666)|canary-host|pw-CANARY' .
(no output)
$ grep -c -E 'CANARY|canary-host' /tmp/cr-stdout.txt /tmp/cr-stderr.txt
/tmp/cr-stdout.txt:0
/tmp/cr-stderr.txt:0
$ find . -type f | wc -l
116Zero leaks across 116 written files, stdout and stderr. The URL-userinfo shape $ CHEAP_REAL_API_BASE=http://localhost:59999 bash examples/cheap_real/run.sh
cheap_real preflight: the local Ollama endpoint did not answer (URLError). Start it
(`ollama serve`) or set CHEAP_REAL_MODEL to a hosted model. Endpoint withheld
(see docs/TROUBLESHOOTING.md).Type name only, no URL. Correct per #134. I confirm the separate pre-existing committed-hostname finding the PR reports elsewhere in #195 / #197 complianceBoth reproduce on the sibling branches. #195 — against MIN_VAL_TASKS=2 LOW_CONFIDENCE_VAL_TASKS=5
cheap_real realized split: train=10 val=5 test=5
check_val_size warning: Noneval=5, both floors cleared, zero warnings — exactly as claimed. And the control, so The test genuinely pins the floor — it is not a comment. I dropped E AssertionError: cheap_real val split is 3, below 5: every gate decision would be
branded LOW CONFIDENCE. Add tasks rather than lowering this.
E assert 3 >= 5
1 failed, 3 passed in 0.06sFails, on the right assertion, with a message that tells you what to do. This is the One caveat: the test's #197 — against
So the four grader files are covered: The #248 entry pointThe contract holds, including under #217. From my own run: $ python -c "<scan stdout for JSON objects>"
objects on stdout: 1
last object keys: ['baseline_val','best_id','dashboard','iterations','run_dir',
'test_baseline_reward','test_delta','test_pass_k','test_reward']Exactly one JSON object on stdout — #217's two-object bug does not fire here, because A naive whole-stdout Could #248 genuinely call it? Yes — every knob is an env var with a default Test quality4 new tests in
They are cheap (0.03 s, no model) and each one fails for a real reason. Untested: all of On bug (b): Merge-order noteRecommend: #195 → #197 → #233 → #249 → #248.
Verification I re-ran$ gh pr view 249 --json headRefName
{"headRefName":"feat/issue-124-cheap-onramp"} # 8f727e8e
$ PYTHONPATH=/tmp/rv-249/core python -m pytest core/tests -q
........................................................................ [ 39%]
........................................................................ [ 78%]
....................................... [100%]
183 passed in 63.07s (0:01:03)183 passed, 0 failed — matches the claim exactly (179 + 4). $ python -m compileall -q core examples && echo COMPILEALL_OK
COMPILEALL_OK
$ bash -n examples/cheap_real/run.sh && echo BASH_N_OK
BASH_N_OKZero-cost path, end to end, on a real local model: $ CHEAP_REAL_WORKDIR=/tmp/cr-canary CHEAP_REAL_PYTHON=/tmp/ce-venv/bin/python \
time bash examples/cheap_real/run.sh
cheap_real preflight: OK
Working directory: /tmp/cr-canary
Runner model: ollama/llama3.2:3b Optimizer: mock
{
"run_dir": ".capevolve/run_cheap",
"best_id": "cand_0001",
"baseline_val": 0.0,
"test_reward": 1.0,
"test_baseline_reward": 0.0,
"test_delta": 1.0,
"test_pass_k": {"1": 1.0, "2": 0.0},
"iterations": 3,
"dashboard": ".capevolve/run_cheap/dashboard.html"
}
33.947 totalReproduces the PR's free-rung result exactly (same All three preflight branches fire: $ bash examples/cheap_real/run.sh # system python3, no litellm
cheap_real preflight: litellm is not importable by this interpreter. `pip install
litellm`, or point CHEAP_REAL_PYTHON at one that has it.
$ CHEAP_REAL_MODEL=ollama/nonexistent-model bash examples/cheap_real/run.sh
cheap_real preflight: Ollama is up but does not have 'nonexistent-model'. Run
`ollama pull nonexistent-model` (~2 GB), or set CHEAP_REAL_MODEL.
$ CHEAP_REAL_API_BASE=http://localhost:59999 bash examples/cheap_real/run.sh
cheap_real preflight: the local Ollama endpoint did not answer (URLError). ...All three exit non-zero before spending anything. To answer the brief's question Bug (c) mechanism, on #195 floors, #197 defaults, the shrunk-dataset control, the 30-call breakdown, the Docs links: broken relative links: NONE |
B1 (blocking). `CHEAP_REAL_MODEL=claude-haiku-4-5` — the fallback offered in three
docs for users with no local model — produced `preflight: OK`, exit 0, empty stderr
and `test_reward: 0.0` on every task. Two causes, both fixed in run.sh:
* the bundled adapter forwards `seed=`, which Anthropic rejects
(`UnsupportedParamsError`); the adapter correctly turns that into reward 0.0, so
the whole rung scored zero behind a clean-looking run. `export LITELLM_DROP_PARAMS=1`.
* every preflight check sat inside `if MODEL.startswith("ollama/")`, so the one
documented path nobody had exercised was the one path the preflight did not cover —
and it printed OK, actively vouching for the dead run. The hosted branch now does one
real 1-token completion through `model_config.llm_kwargs()` and exits non-zero on any
failure, with the endpoint and every credential redacted from the message.
Secondary trap on the same path: `API_BASE=localhost:11434` was exported
unconditionally and `model_config.py:95` reads the generic `API_BASE` before any
provider special-casing, so a hosted MODEL inherited a localhost endpoint. Now exported
only for an `ollama/` MODEL.
MEASURED, hosted variant, run for real: 98 s, $0.011478, 30 metric_calls,
baseline_val 0.0 -> sealed test_reward 1.0. The six eval costs sum to the total
exactly. That retires the prior $0.21 ESTIMATE, which was 20x high because
pricing.py assumes a 3000-in/800-out rollout and a one-line date is far smaller.
Review findings also addressed:
* Dropped "real paired gate" (5x) for "real accept/reject decision", and added an
Honest-limits entry stating plainly that this run never rejects on merit: val
saturates at 1.0 on iteration 1, so the two later rejections are against a ceiling,
and at num_trials 1 the decision is Δ > 0, not a significance test.
* Three progress lines moved to stderr (#116), and the single-object stdout contract no
longer depends on `--dashboard off` defaulting: run.sh filters the CLI's stdout to the
last object, so `json.loads(stdout)` works under `CAPEVOLVE_DASHBOARD=auto` too (#217).
* Corrected the #197 wording. `tasks.jsonl` matches NO default glob; the answer key is
covered by the `dataset_source` fold-in at protect.py:169, which only happens when
`protected_paths` is omitted. Declaring the key drops it — verified.
* Corrected the call-count comment in the preset (15 -> 30, with the derivation) and
the stale CHEAP_REAL_MAX_USD default in run.sh's header (2.0 -> 3.0).
* Replaced the "36 s" figure, which appeared in neither measurement, with "~30 s" and
the observed spread (30.4 / 33.9 / 34.2 s).
* Softened the zoo line: the determinism guarantee question is deferred, not settled —
named the `determinism: exact|statistical|none` grade #233 will need.
* Noted the pass^k `{"2": 0.0}` artifact as pre-existing #112, and that the local
rung's $0 is a metering artifact (the runner's cost there is time).
* Two new tests: the run.sh YAML-rewrite heredoc over the committed preset (it is a
line-prefix rewriter, so a no-op means the paid rung runs `mock`), and the measured
`optimizer_max_turns >= 40` floor. Both verified to fail on a broken invariant.
Declined, filed instead: the relative `optimizer_instructions_file` bug is #252 (the
absolute-path workaround stays, with the `check`-passes-what-`run`-ignores disagreement
named in the comment); the optimizer stderr discarded on success is #251 — the reason a
warning alone would not have surfaced this bug.
🔧 Review fixesCommit B1 (BLOCKING) — fixed. The hosted variant, run once for real.You were right on both counts, and on the part that stung: the preflight printing
The hosted run. $ CHEAP_REAL_MODEL=claude-haiku-4-5 bash examples/cheap_real/run.sh
cheap_real preflight: OK # ← on stderr now
Working directory: /tmp/fx-hosted-wd # ← on stderr now
Runner model: claude-haiku-4-5 Optimizer: mock # ← on stderr now
{
"run_dir": ".capevolve/run_cheap",
"best_id": "cand_0001",
"baseline_val": 0.0,
"test_reward": 1.0,
"test_baseline_reward": 0.0,
"test_delta": 1.0,
"test_pass_k": {"1": 1.0, "2": 0.0},
"iterations": 3,
"dashboard": ".capevolve/run_cheap/dashboard.html"
}
1:38.13 total
$ jq .spent .capevolve/run_cheap/state.json
{
"iterations": 3, "metric_calls": 30, "usd": 0.011478000000000002, "stall": 2,
"runner_tokens": 3910, "runner_seconds": 91.76, "optimizer_seconds": 0.55,
"optimizer_usd": 0.0, "optimizer_tokens": 0, "intake_usd": 0.0, "intake_seconds": 0.0
}MEASURED: 98 s, $0.011478, 30 $ grep -o '"kind": "evaluate"[^}]*' events.jsonl
"split": "val", "tag": "seed", "reward": 0.0, "cost_usd": 0.003466
"split": "val", "tag": "cand_0001", "reward": 1.0, "cost_usd": 0.000681
"split": "val", "tag": "cand_0002", "reward": 1.0, "cost_usd": 0.000681
"split": "val", "tag": "cand_0003", "reward": 1.0, "cost_usd": 0.000681
"split": "test", "tag": "FINAL", "reward": 1.0, "cost_usd": 0.000687
"split": "test", "tag": "FINAL_seed", "reward": 0.0, "cost_usd": 0.005282
$ python -c "print(0.003466+0.000681*3+0.000687+0.005282)"
0.011478 # state.json usd: 0.011478000000000002And a rollout, to show the call actually landed rather than erroring into a 0: $ cat rollouts/test/d02__FINAL__t0.json
{"input": "the 5th of July 1999",
"rollout": {"output": "1999-07-05", "cost_usd": 0.000138, "tokens": 98, "error": null,
"metadata": {"model": "claude-haiku-4-5", "seed": 0}},
"score": {"reward": 1.0, "feedback": "correct"}}This retires the $0.21 ESTIMATE — and it was 20x high. The preflight fails loudly on the hosted path. A bad model name: $ CHEAP_REAL_MODEL=claude-haiku-4-5-NONEXISTENT bash examples/cheap_real/run.sh
cheap_real preflight: the hosted model 'claude-haiku-4-5-NONEXISTENT' did not answer a
1-token probe (BadRequestError). Every rollout would score 0.0. Check the model name and
its credential; endpoint/keys withheld (see docs/TROUBLESHOOTING.md). Detail:
litellm.BadRequestError: LLM Provider NOT provided. ...A missing credential (run under $ env -i ... CHEAP_REAL_MODEL=anthropic/claude-haiku-4-5 bash examples/cheap_real/run.sh
cheap_real preflight: the hosted model 'anthropic/claude-haiku-4-5' did not answer a
1-token probe (AuthenticationError). Every rollout would score 0.0. ...All five preflight branches, exit code and stdout cleanliness: exit=1 stdout_bytes=0 <- system python3 (no litellm)
exit=1 stdout_bytes=0 <- CHEAP_REAL_API_BASE=http://localhost:59999
exit=1 stdout_bytes=0 <- CHEAP_REAL_MODEL=ollama/nonexistent-model
exit=1 stdout_bytes=0 <- CHEAP_REAL_MODEL=claude-haiku-4-5-NOPE (new)
exit=1 stdout_bytes=0 <- hosted, no credential (new)Note stdout is 0 bytes on every failure. It was not before: litellm writes some of The MODEL=claude-haiku-4-5 API_BASE=<UNSET>
MODEL=ollama/llama3.2:3b API_BASE=http://localhost:11434
The nine other findingsN1 — fixed. # 30 runner calls, plus 3 proposer calls. The full derivation — the baseline val eval
# and both test evals are easy to forget:
# val 5 x trials 1 x (1 baseline + 3 candidates) = 20
# test 5 x (best FINAL + baseline FINAL_seed) = 10
# total = 30 (= state.json metric_calls)N2 — fixed. N3 — fixed. "36 s" is gone from all three places. Replaced with "~30 s" plus the N4 — fixed, and this was the most useful correction in the review. My wording implied === OMITTED key (what the preset ships)
globs : ['adapters', 'capevolve.yaml', '*gold*.json', ..., 'tasks.jsonl']
resolved: ['adapters/adapter.py', 'adapters/model_config.py', 'capevolve.yaml', 'tasks.jsonl']
tasks.jsonl matches a _DEFAULT_GLOB on its own? False
<- it is the dataset_source fold-in that covers the answer key
=== EMPTY list -> TamperError: `protected_paths` ... is an EMPTY list ...
=== DECLARED [adapters] resolved: ['adapters/adapter.py', 'adapters/model_config.py']
<- tasks.jsonl DROPPEDThat third line is the one that makes the point: declaring the key at all, even N5 — acknowledged, no change. A conscious call, as you read it: N6 — fixed, on the path you named. A test that runs the real heredoc (extracted $ # with optimizer_max_turns lowered to 12 and optimizer_skill indented one space
E AssertionError: optimizer_max_turns below 40 was MEASURED to make every iteration
fail with `Reached max turns`, so no candidate survives. Raise it back.
E assert 12 >= 40
E AssertionError: the optimizer_skill flip no-opped
2 failed, 4 passed in 0.04sBoth new tests fail on the real breakage, in the shape you praised on the floor test. Nit 1 — fixed. The README now says the local rung's 30 calls were "metered at $0 Nit 2 — fixed. New Honest-limits entry naming The gate claim — you were right, and here is the corrected wordingI over-claimed and the two "correct rejections" I touted are vacuous, exactly as you
Same line in Zoo line softened. README:75 read as settled; it now names the shape you identified:
Filed, not fixed here
Re-verification$ PYTHONPATH=$PWD/core python -m pytest core/tests -q
........................................................................ [ 38%]
........................................................................ [ 77%]
......................................... [100%]
185 passed in 64.06s (0:01:04)
$ python -m compileall -q core examples && echo OK
OK
$ bash -n examples/cheap_real/run.sh && echo OK
OK185 passed (183 + 2 new), 0 failed. Stdout is exactly one object with $ CAPEVOLVE_DASHBOARD=auto bash examples/cheap_real/run.sh
objects on stdout: 1
whole-stdout json.loads: OK
keys: ['baseline_val','best_id','dashboard','dashboard_server','iterations','run_dir',
'test_baseline_reward','test_delta','test_pass_k','test_reward']Note Free rung, unchanged (33.6 s this run, $0): {"run_dir": ".capevolve/run_cheap", "best_id": "cand_0001", "baseline_val": 0.0,
"test_reward": 1.0, "test_baseline_reward": 0.0, "test_delta": 1.0, "iterations": 3}
metric_calls: 30#195 compliance — against MIN_VAL_TASKS=2 LOW_CONFIDENCE_VAL_TASKS=5
cheap_real realized split: train=10 val=5 test=5
check_val_size warning: None
CONTROL 12 tasks: val=3 warning: val split has only 3 tasks (< 5) — acceptance
decisions are LOW CONFIDENCE. ...#197 compliance — pasted under N4 above, including the new
{"ok": true, "stubs": [], "problems": [],
"notes": ["tasks('val') -> 20 task(s)", "scorer deterministic (probe reward=0.0000)",
"materialize() callable (dry-run into temp copy; host untouched)"]}No canary leaks — including on the newly-added hosted path, which is where it now $ find . -type f | wc -l
116
$ grep -rIl -E 'CANARY-(AAAA1111|...|GGGG7777)|canary-host|pw-CANARY' .
(no output)
$ grep -c -E 'CANARY|canary-host' stdout stderr
stdout:0 stderr:0And two new shapes aimed at the new code, both forcing the preflight to fail so the $ ANTHROPIC_API_KEY=sk-ant-CANARY-HHHH8888 \
ANTHROPIC_API_BASE='https://user:pw-CANARY-IIII9999@gw.canary-host.invalid/v1' ...
cheap_real preflight: the hosted model 'anthropic/claude-haiku-4-5' did not answer a
1-token probe (InternalServerError). ... Detail: litellm.InternalServerError:
AnthropicException - [Errno 8] nodename nor servname provided, or not known.
canary count: stdout:0 stderr:0
$ # reachable endpoint, so the 404 path — the one that usually echoes the URL
$ ANTHROPIC_API_BASE='http://user:pw-CANARY-LLLL5678@127.0.0.1:11434/CANARY-PATH-MMMM9012'
cheap_real preflight: ... (NotFoundError). ... Detail: litellm.NotFoundError:
AnthropicException - 404 page not found
canary count: stdout:0 stderr:0Zero leaks, including the URL-userinfo shape, on the failure path. The preflight replaces Broken relative links: NONE. Every finding is fixed or, in one case (N5), consciously kept with the reason. The one |
Closes #124
There was a cliff between rung 1 and rung 2 of onboarding:
toy_calcis free but callsno model, and the next advertised step was the τ² airline run — hours and ~$148. This
adds the missing rung:
examples/cheap_real, a real run (real LLM, real pairedgate, real sealed test split) that finishes in minutes for $0 on a local model, or
cents with a hosted proposer.
The task is date normalization —
"the 22nd of November, 1963"→1963-11-22, scoredby exact match. The seed prompt is a generic "you are a helpful assistant", so a small
model answers in prose and scores 0; the fix is an output contract, which is exactly the
edit an optimizer is good at proposing. That makes the demonstration honest: the gain
comes from a general rule, not a memorized answer.
The three-rung ladder
All three are now visible in
docs/GETTING_STARTED.mdandsite/getting-started.html,each with its cost, runtime, and derivation.
toy_calccheap_realDerivation (rung 2)
20 tasks → train 10 / val 5 / test 5,
num_trials: 1,max_iterations: 3:ollama/llama3.2:3brunner +mockproposer) — MEASURED: 36 s,$0.00. $0 by construction: a local model is unmetered and
mockmakes no networkcall at all.
claude-code/Haiku proposer) — MEASUREDfrom the run's own accounting (
state.json→spent): $0.5427 total, 321 s(5.4 min) wall clock, and exactly the 30
metric_callsthe formula above predicts.Per-proposal spend was $0.159 / $0.199 / $0.185, so budget ~$0.15–0.35 per
iteration. Proposal latency dominates: 293 s of the 321 s was the optimizer, only
27 s the runner. All 30 runner calls cost $0 because the runner is local.
Haiku rate and cap-evolve's assumed 3 000 in / 800 out tokens per rollout:
30 x (3000 x $1.00 + 800 x $5.00) / 1e6≈ $0.21 — an ESTIMATE fromcore/cap_evolve/pricing.py, explicitly labelled as such in the docs. I did not runthis variant.
max_usd: 3.0/max_optimizer_usd: 2.5are hard stops, so rung 2 cannot quietlybecome rung 3.
Standalone example, not a zoo entry — and why
#233's zoo was the natural home, and I checked it properly rather than assuming. It
cannot host this one:
verifystep 5 runs every val task twice and fails thebenchmark if any rollout fingerprint differs
(
NON-DETERMINISTIC: … cannot produce a reproducible number). That guard is correct andworth keeping, but no real LLM can satisfy it, even at
temperature=0with a fixedseed. This example exists specifically to call a real model, so making it a zoo entry
would mean weakening the zoo's determinism guarantee to admit the one benchmark that
structurally cannot meet it. Standalone example; the zoo keeps its invariant. Reasoning
is recorded in the example's README so the next person does not re-litigate it.
Entry point for #133
bash examples/cheap_real/run.sh. Every knob is an env var with a default, and the lastobject on stdout is the run's summary JSON:
CHEAP_REAL_MODELollama/llama3.2:3b(free, local)CHEAP_REAL_API_BASEhttp://localhost:11434CHEAP_REAL_OPTIMIZERmock(zero-API)CHEAP_REAL_OPT_MODEL""CHEAP_REAL_WORKDIRmktempdirCHEAP_REAL_MAX_USDCHEAP_REAL_PYTHONpython3So #133 can call it with
CHEAP_REAL_WORKDIRpinned and parse the last JSON object; noimport, no new API surface.
How it composes with the sibling PRs
model_config.py, and nothing this PR prints echoes an endpoint: the preflightreports a failed local endpoint without the URL, deliberately, per feat(config): provider-scoped credential resolution + auto provider probing (closes #134) #190's rule that
a non-default
base_urlis confidential.cap-evolve doctorinstall/health diagnostic #193 / Add cap-evolve doctor install/health diagnostic #121 —doctorcovers credential presence; this preflight covers theorthogonal thing (is the model actually reachable and pulled). No overlap.
proposer_modelandoptimizer_modeltoclaude-haiku-4-5on purpose.optimizer_modelis what main reads today;proposer_modelis Model tiering: cheap model for auxiliary steps, strong model for proposals #132's tier name and falls back tooptimizer_modelwhen blank,so setting both is correct before and after that merge.
aux_modelis deliberatelyunset, because every auxiliary step is still pure Python and an aux tier would route
nothing.
smallest split that clears both
MIN_VAL_TASKS = 2(below whichgate.deciderefuses) and
LOW_CONFIDENCE_VAL_TASKS = 5(below which every decision is brandedLOW CONFIDENCE). Verified against Guard tiny/empty val splits and add a Student-t small-sample correction to the paired gate #195's branch, not just described.
protected_pathsis omitted, never[].run.shplaces theadapter,
model_config.pyandtasks.jsonlinside the project dir and the specnames
dataset_source: tasks.jsonl, so Protected-paths tamper guard: verify the optimizer never edited scoring/eval/task files #197's layout defaults cover all four graderfiles without the preset declaring anything (a declared list would replace the
defaults).
Expected merge order: none required — this is additive (one new example dir, one new
test file, three edited docs; zero core changes). It composes with all five siblings in
any order.
Three bugs found while proving this works
Worth recording, because each one produced a clean-looking run whose numbers were
meaningless — the failure mode this epic keeps catching.
litellmmade a broken run look like an honest 0.0. The adaptercorrectly turns a failed model call into
error:+ reward 0.0 (infra noise must notbe optimized against), but that means a stopped Ollama yields
baseline_val 0.0 → test_reward 0.0with no visible failure.run.shnow preflights the interpreter,the endpoint, and whether the model is actually pulled, before spending anything.
optimizer_max_turns: 12silently discarded every candidate. claude-code exitsnon-zero with
Reached max turns (12), which cap-evolve correctly reports as afailed iteration — so the run read as "the optimizer proposed nothing" rather than
"the cap was too tight". Now 40, with the measured reason in a comment.
optimizer_instructions_filesilently falls back to the generictemplate.
cli.pyresolves it against its own cwd and then a cwd-relative.capevolve/project, neither of which is the run's project dir when the workdir iselsewhere. The optimizer consequently received tau2-flavored "edit the tool code"
advice for a project with no tools, and proposed a prompt for the wrong task
(it made the agent a date-trivia assistant).
run.shwrites an absolute path.This is a latent main-branch sharp edge for any out-of-tree project dir; I worked
around it here rather than widen this PR's scope into
cli.py.Verification
Every command and its full output is in the
## 🔬 Evidencecomment below. Summary:cap-evolve checkon the preset —{"ok": true}:{ "ok": true, "stubs": [], "problems": [], "notes": [ "tasks('val') -> 20 task(s)", "scorer deterministic (probe reward=0.0000)", "materialize() callable (dry-run into temp copy; host untouched)" ] }Zero-cost proof first — real local LLM runner,
mockoptimizer, through to a sealedtest number (36 s, $0):
{ "run_dir": ".capevolve/run_cheap", "best_id": "cand_0001", "baseline_val": 0.0, "test_reward": 1.0, "test_baseline_reward": 0.0, "test_delta": 1.0, "iterations": 3 }Then with a REAL agent proposing the edit (claude-code / Haiku) — completed end to
end to a sealed test number, MEASURED:
{ "run_dir": ".capevolve/run_cheap", "best_id": "cand_0001", "baseline_val": 0.0, "test_reward": 1.0, "test_baseline_reward": 0.0, "test_delta": 1.0, "iterations": 3 }Its own spend accounting — note
metric_calls: 30, exactly what the derivation predicts:{"iterations": 3, "metric_calls": 30, "usd": 0.0, "optimizer_usd": 0.5426701500000001, "runner_seconds": 26.98, "optimizer_seconds": 293.47}The gate accepted on iteration 1 and correctly rejected the two that followed (val was
already 1.0, so there was nothing left to gain):
The agent's actual proposed edit — a general rule, no memorized dates:
#195's val floor:
#197's
protected_pathsrule — key omitted, all four grader files covered by thelayout defaults:
Full suite — 183 passed (179 baseline + 4 new), 0 failed:
compileallclean,bash -n run.shclean, every relative link in the touched docsresolves.
Honest limits, stated in the README
SE=0 → STRICT fallbackatnum_trials: 1, so the accept decisionis "Δ > 0", not a significance test. That is documented behavior, not a defect of the
preset — but it is stated plainly rather than glossed.
model, not producing a publishable number.
arithmetic shown.