Skip to content

Dynamic conversation turns with a heuristic responder #257

Description

@slowdini

Part of #244 (full-codebase eval testbed). Depends on #256 (runner-driven dispatch).

Why

Conversations today are scripted: an eval declares an ordered turns array, each with a fixed
prompt and a deliver_when gate of always or agent_asks
(schema/evals.schema.json, src/cli/run/conversation.rs). The author has to predict what the
agent will ask and in what order.

#244 wants the opposite: a realistic opening prompt, however many turns the task actually needs,
and a run that ends when the task is done. It also names the happy path — "Did the agent ask a
multiple choice question, with a recommended option? Always choose the recommendation."

Decision taken on #244: heuristics first. The LLM answering agent is a separate follow-up ticket
so this loop can be proven cheaply and reproducibly before a second model enters the attribution
picture.

Scope

Replace the fixed turns array with a responder policy:

  • max_turns bound, plus stop conditions.
  • Recommendation picking — detect a multiple-choice question that marks one option as
    recommended, and answer with it.
  • Completion detection — recognise that the agent considers the task done and stop, rather than
    burning turns.
  • Every synthesized turn is recorded in conversation.json with its provenance (which heuristic
    fired, and why), so a judge can see exactly what the agent was told and a human can audit whether
    the responder distorted the run.

The existing ConversationStopReason vocabulary needs extending; today it carries only
AgentDidNotAsk and AgentResponseMismatch, which are scripted-gate concepts.

Files

  • src/cli/run/conversation.rs
  • schema/evals.schema.json, schema/conversation.schema.json
  • src/core/types.rs

Acceptance criteria

  • An eval with a responder policy and no scripted turns runs to completion.
  • A recommended option in a multiple-choice question is selected and the selection is recorded.
  • A run that reaches max_turns stops and is recorded as such, not as a failure.
  • A completed task stops before max_turns.
  • Each synthesized turn in conversation.json names the heuristic that produced it.
  • A question the heuristics cannot classify stops the conversation with a distinct, greppable reason
    — that reason is the entry point for the follow-up LLM-responder ticket.

Cross-cutting requirements

  • Mode B parity. Acceptance must hold for both Mode::NewSkill and Mode::Revision
    (src/core/types.rs:225), and the snapshot/promote path (src/workspace/snapshot.rs,
    src/workspace/promote.rs) must keep working against codebase-backed iterations.
  • Provenance reaches the report. Source + resolved SHA must land in conditions.json, each
    run.json, benchmark.json, and the BASELINE.md built by promote.rs:231 — not just in the
    workspace.

Verification

cargo fmt --check && cargo build && cargo test && cargo clippy --all-targets -- -D warnings

Implementation requires the slow-powers:working-with-tdd skill.

Parent: #244

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions