Skip to content

LLM answering agent for free-form questions #258

Description

@slowdini

Part of #244 (full-codebase eval testbed). Depends on #257 (heuristic responder).

Why

#257 ships deterministic responder heuristics and stops the conversation with a distinct reason
when it meets a question it cannot classify. #244 asks for the general case:

For more complex questions, we may need some kind of "question answering agent", designed to
quickly generate a comprehensible response, always tending towards "choose whatever is
recommended/simplest/easiest".

This ticket fills that gap. It is deliberately separate from #257 so the loop is proven before a
second model is introduced into the attribution picture.

Scope

  • Dispatch a small answering agent through the same harness as the run, prompted to choose the
    recommended, simplest, or easiest path.
  • Trigger it only on the "heuristics could not classify" branch from Dynamic conversation turns with a heuristic responder #257 — the deterministic path
    stays the default, both for cost and for reproducibility.
  • Attribute it separately in conversation.json: a turn produced by the answering agent must be
    distinguishable from a heuristic turn, and the responder model must be recorded alongside the
    agent and judge models already tracked in conditions.json.

Files

  • src/cli/run/conversation.rs
  • schema/conversation.schema.json
  • src/core/types.rs (responder model provenance)

Acceptance criteria

Cross-cutting requirements

  • Mode B parity. Acceptance must hold for both Mode::NewSkill and Mode::Revision
    (src/core/types.rs:225), and the snapshot/promote path (src/workspace/snapshot.rs,
    src/workspace/promote.rs) must keep working against codebase-backed iterations.
  • Provenance reaches the report. Source + resolved SHA must land in conditions.json, each
    run.json, benchmark.json, and the BASELINE.md built by promote.rs:231 — not just in the
    workspace.

Verification

cargo fmt --check && cargo build && cargo test && cargo clippy --all-targets -- -D warnings

Implementation requires the slow-powers:working-with-tdd skill.

Parent: #244

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions