Part of #244 (full-codebase eval testbed). No dependencies — startable today.
Why
Today only tasks declaring scripted turns are runner-driven: run_task
(src/cli/run/conversation.rs:89-93) errors out if a task has none. Every one-shot task is
dispatched by a human or an agent following the recipe in profiles/shared/runbook.md.
#244 wants real tasks with a dynamic number of turns, run enough times to be statistically
meaningful. That is not drivable by hand, and dynamic turns are impossible without the runner in the
loop. There is also no per-dispatch timeout anywhere — execute_round
(src/cli/run/conversation.rs:367-387) runs to completion, so one hung task hangs the campaign.
Decision taken on #244: the runner drives every dispatch.
Scope
- Drop
dispatch-task's "must declare scripted follow-up turns" precondition so one-shot tasks run
through the same driver, which already handles turn 1, native session resume, transcript parsing,
and the conversation.json completion artifact.
- Add
eval-magic dispatch --iteration N [--jobs K] [--task-index …]:
--jobs concurrency (each env is private, so tasks are independent).
- Per-task timeout with the timeout recorded as a task outcome, not a crash.
- Recorded failures rather than aborting the campaign.
- Resume on rerun — completed tasks are skipped unless
--overwrite.
- Rewrite
profiles/shared/runbook.md and each harness's next_steps / manifest templates around
the single command. Golden artifacts under tests/golden/ pin this output and must be updated.
Files
src/cli/run/conversation.rs — the driver
src/cli/run/dispatch.rs, src/cli/commands/run.rs
src/cli/args.rs, src/cli/help.rs
profiles/shared/runbook.md, harnesses/*.toml
tests/golden/
Acceptance criteria
- A one-shot task and a multi-turn task both complete through
eval-magic dispatch.
--jobs 4 runs four tasks concurrently without cross-task interference.
- A task exceeding its timeout is recorded as timed out; the rest of the campaign completes.
- Rerunning
dispatch skips completed tasks and reruns only failures.
RUNBOOK.md describes one dispatch command; golden tests updated to match.
Note
This is the largest single behavioural change in #244 and it unblocks the dynamic-turn work.
Cross-cutting requirements
- Mode B parity. Acceptance must hold for both
Mode::NewSkill and Mode::Revision
(src/core/types.rs:225), and the snapshot/promote path (src/workspace/snapshot.rs,
src/workspace/promote.rs) must keep working against codebase-backed iterations.
- Provenance reaches the report. Source + resolved SHA must land in
conditions.json, each
run.json, benchmark.json, and the BASELINE.md built by promote.rs:231 — not just in the
workspace.
Verification
cargo fmt --check && cargo build && cargo test && cargo clippy --all-targets -- -D warnings
Implementation requires the slow-powers:working-with-tdd skill.
Parent: #244
Part of #244 (full-codebase eval testbed). No dependencies — startable today.
Why
Today only tasks declaring scripted
turnsare runner-driven:run_task(
src/cli/run/conversation.rs:89-93) errors out if a task has none. Every one-shot task isdispatched by a human or an agent following the recipe in
profiles/shared/runbook.md.#244 wants real tasks with a dynamic number of turns, run enough times to be statistically
meaningful. That is not drivable by hand, and dynamic turns are impossible without the runner in the
loop. There is also no per-dispatch timeout anywhere —
execute_round(
src/cli/run/conversation.rs:367-387) runs to completion, so one hung task hangs the campaign.Decision taken on #244: the runner drives every dispatch.
Scope
dispatch-task's "must declare scripted follow-up turns" precondition so one-shot tasks runthrough the same driver, which already handles turn 1, native session resume, transcript parsing,
and the
conversation.jsoncompletion artifact.eval-magic dispatch --iteration N [--jobs K] [--task-index …]:--jobsconcurrency (each env is private, so tasks are independent).--overwrite.profiles/shared/runbook.mdand each harness'snext_steps/ manifest templates aroundthe single command. Golden artifacts under
tests/golden/pin this output and must be updated.Files
src/cli/run/conversation.rs— the driversrc/cli/run/dispatch.rs,src/cli/commands/run.rssrc/cli/args.rs,src/cli/help.rsprofiles/shared/runbook.md,harnesses/*.tomltests/golden/Acceptance criteria
eval-magic dispatch.--jobs 4runs four tasks concurrently without cross-task interference.dispatchskips completed tasks and reruns only failures.RUNBOOK.mddescribes one dispatch command; golden tests updated to match.Note
This is the largest single behavioural change in #244 and it unblocks the dynamic-turn work.
Cross-cutting requirements
Mode::NewSkillandMode::Revision(
src/core/types.rs:225), and the snapshot/promote path (src/workspace/snapshot.rs,src/workspace/promote.rs) must keep working against codebase-backed iterations.conditions.json, eachrun.json,benchmark.json, and theBASELINE.mdbuilt bypromote.rs:231— not just in theworkspace.
Verification
Implementation requires the
slow-powers:working-with-tddskill.Parent: #244