Skip to content

Runner-driven dispatch for every task #256

Description

@slowdini

Part of #244 (full-codebase eval testbed). No dependencies — startable today.

Why

Today only tasks declaring scripted turns are runner-driven: run_task
(src/cli/run/conversation.rs:89-93) errors out if a task has none. Every one-shot task is
dispatched by a human or an agent following the recipe in profiles/shared/runbook.md.

#244 wants real tasks with a dynamic number of turns, run enough times to be statistically
meaningful. That is not drivable by hand, and dynamic turns are impossible without the runner in the
loop. There is also no per-dispatch timeout anywhere — execute_round
(src/cli/run/conversation.rs:367-387) runs to completion, so one hung task hangs the campaign.

Decision taken on #244: the runner drives every dispatch.

Scope

  • Drop dispatch-task's "must declare scripted follow-up turns" precondition so one-shot tasks run
    through the same driver, which already handles turn 1, native session resume, transcript parsing,
    and the conversation.json completion artifact.
  • Add eval-magic dispatch --iteration N [--jobs K] [--task-index …]:
    • --jobs concurrency (each env is private, so tasks are independent).
    • Per-task timeout with the timeout recorded as a task outcome, not a crash.
    • Recorded failures rather than aborting the campaign.
    • Resume on rerun — completed tasks are skipped unless --overwrite.
  • Rewrite profiles/shared/runbook.md and each harness's next_steps / manifest templates around
    the single command. Golden artifacts under tests/golden/ pin this output and must be updated.

Files

  • src/cli/run/conversation.rs — the driver
  • src/cli/run/dispatch.rs, src/cli/commands/run.rs
  • src/cli/args.rs, src/cli/help.rs
  • profiles/shared/runbook.md, harnesses/*.toml
  • tests/golden/

Acceptance criteria

  • A one-shot task and a multi-turn task both complete through eval-magic dispatch.
  • --jobs 4 runs four tasks concurrently without cross-task interference.
  • A task exceeding its timeout is recorded as timed out; the rest of the campaign completes.
  • Rerunning dispatch skips completed tasks and reruns only failures.
  • RUNBOOK.md describes one dispatch command; golden tests updated to match.

Note

This is the largest single behavioural change in #244 and it unblocks the dynamic-turn work.

Cross-cutting requirements

  • Mode B parity. Acceptance must hold for both Mode::NewSkill and Mode::Revision
    (src/core/types.rs:225), and the snapshot/promote path (src/workspace/snapshot.rs,
    src/workspace/promote.rs) must keep working against codebase-backed iterations.
  • Provenance reaches the report. Source + resolved SHA must land in conditions.json, each
    run.json, benchmark.json, and the BASELINE.md built by promote.rs:231 — not just in the
    workspace.

Verification

cargo fmt --check && cargo build && cargo test && cargo clippy --all-targets -- -D warnings

Implementation requires the slow-powers:working-with-tdd skill.

Parent: #244

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions