Part of #244 (full-codebase eval testbed). Depends on #256 (runner-driven dispatch).
Why
Conversations today are scripted: an eval declares an ordered turns array, each with a fixed
prompt and a deliver_when gate of always or agent_asks
(schema/evals.schema.json, src/cli/run/conversation.rs). The author has to predict what the
agent will ask and in what order.
#244 wants the opposite: a realistic opening prompt, however many turns the task actually needs,
and a run that ends when the task is done. It also names the happy path — "Did the agent ask a
multiple choice question, with a recommended option? Always choose the recommendation."
Decision taken on #244: heuristics first. The LLM answering agent is a separate follow-up ticket
so this loop can be proven cheaply and reproducibly before a second model enters the attribution
picture.
Scope
Replace the fixed turns array with a responder policy:
max_turns bound, plus stop conditions.
- Recommendation picking — detect a multiple-choice question that marks one option as
recommended, and answer with it.
- Completion detection — recognise that the agent considers the task done and stop, rather than
burning turns.
- Every synthesized turn is recorded in
conversation.json with its provenance (which heuristic
fired, and why), so a judge can see exactly what the agent was told and a human can audit whether
the responder distorted the run.
The existing ConversationStopReason vocabulary needs extending; today it carries only
AgentDidNotAsk and AgentResponseMismatch, which are scripted-gate concepts.
Files
src/cli/run/conversation.rs
schema/evals.schema.json, schema/conversation.schema.json
src/core/types.rs
Acceptance criteria
- An eval with a
responder policy and no scripted turns runs to completion.
- A recommended option in a multiple-choice question is selected and the selection is recorded.
- A run that reaches
max_turns stops and is recorded as such, not as a failure.
- A completed task stops before
max_turns.
- Each synthesized turn in
conversation.json names the heuristic that produced it.
- A question the heuristics cannot classify stops the conversation with a distinct, greppable reason
— that reason is the entry point for the follow-up LLM-responder ticket.
Cross-cutting requirements
- Mode B parity. Acceptance must hold for both
Mode::NewSkill and Mode::Revision
(src/core/types.rs:225), and the snapshot/promote path (src/workspace/snapshot.rs,
src/workspace/promote.rs) must keep working against codebase-backed iterations.
- Provenance reaches the report. Source + resolved SHA must land in
conditions.json, each
run.json, benchmark.json, and the BASELINE.md built by promote.rs:231 — not just in the
workspace.
Verification
cargo fmt --check && cargo build && cargo test && cargo clippy --all-targets -- -D warnings
Implementation requires the slow-powers:working-with-tdd skill.
Parent: #244
Part of #244 (full-codebase eval testbed). Depends on #256 (runner-driven dispatch).
Why
Conversations today are scripted: an eval declares an ordered
turnsarray, each with a fixedpromptand adeliver_whengate ofalwaysoragent_asks(
schema/evals.schema.json,src/cli/run/conversation.rs). The author has to predict what theagent will ask and in what order.
#244 wants the opposite: a realistic opening prompt, however many turns the task actually needs,
and a run that ends when the task is done. It also names the happy path — "Did the agent ask a
multiple choice question, with a recommended option? Always choose the recommendation."
Decision taken on #244: heuristics first. The LLM answering agent is a separate follow-up ticket
so this loop can be proven cheaply and reproducibly before a second model enters the attribution
picture.
Scope
Replace the fixed
turnsarray with aresponderpolicy:max_turnsbound, plus stop conditions.recommended, and answer with it.
burning turns.
conversation.jsonwith its provenance (which heuristicfired, and why), so a judge can see exactly what the agent was told and a human can audit whether
the responder distorted the run.
The existing
ConversationStopReasonvocabulary needs extending; today it carries onlyAgentDidNotAskandAgentResponseMismatch, which are scripted-gate concepts.Files
src/cli/run/conversation.rsschema/evals.schema.json,schema/conversation.schema.jsonsrc/core/types.rsAcceptance criteria
responderpolicy and no scripted turns runs to completion.max_turnsstops and is recorded as such, not as a failure.max_turns.conversation.jsonnames the heuristic that produced it.— that reason is the entry point for the follow-up LLM-responder ticket.
Cross-cutting requirements
Mode::NewSkillandMode::Revision(
src/core/types.rs:225), and the snapshot/promote path (src/workspace/snapshot.rs,src/workspace/promote.rs) must keep working against codebase-backed iterations.conditions.json, eachrun.json,benchmark.json, and theBASELINE.mdbuilt bypromote.rs:231— not just in theworkspace.
Verification
Implementation requires the
slow-powers:working-with-tddskill.Parent: #244