Skip to content

Skill sets as the treatment (multi-skill evals) #264

Description

@slowdini

Part of #244 (full-codebase eval testbed). Interacts with #253 (sourced skills).

Why

An eval has exactly one skill under test. skill_name is a single required string
(schema/evals.schema.json:10); Mode is binary (src/core/types.rs:225); each ConditionEntry
carries one skill_path. Other skills in the directory are staged as ambient siblings into both
arms (stage_sibling_skills, src/cli/run/staging/mod.rs), so they are background, never treatment.

#244 wants the suite itself to be the thing under test:

With a real test case, we could see the full suite of slow-powers skills working together, with the
final code being a useful product by which to judge the plugin as a whole. This is also likely to be
a way to catch places where skills conflict or cause confusion.

Inter-skill conflict is invisible under the current model by construction: conflicting skills are
either both ambient (present in both arms, so they cancel) or only one is treatment.

Scope

  • skill_name grows a set form alongside the single-skill form.
  • ConditionEntry carries a roster rather than one skill_path.
  • The __skill_invoked meta-check becomes per-skill. Both the deterministic transcript check
    (check_skill_invoked_from_transcript, which matches one staged slug) and the LLM fallback rubric
    currently assume a single skill.
  • Sibling staging needs a clear rule for what remains ambient when the treatment is a set.

Once #253 makes skills a sourced, recorded input, the roster is part of that record — extend it rather than
introducing a second way to name skills.

Files

  • schema/evals.schema.json
  • src/core/types.rs
  • src/cli/run/orchestrate/{resolve,stage}.rs
  • src/pipeline/grade/judge_tasks.rs
  • src/cli/args.rs, src/validation/evals.rs

Acceptance criteria

  • An eval naming a set of skills stages all of them in the treatment arm and none in the control arm.
  • __skill_invoked reports per-skill, and a suite where only some skills fired is distinguishable
    from one where all did.
  • Single-skill evals behave exactly as today.
  • The condition roster appears in conditions.json and the run report.
  • Mode B (revision) works with a set — suite-at-revision-A vs suite-at-revision-B.

Cross-cutting requirements

  • Mode B parity. Acceptance must hold for both Mode::NewSkill and Mode::Revision
    (src/core/types.rs:225), and the snapshot/promote path (src/workspace/snapshot.rs,
    src/workspace/promote.rs) must keep working against codebase-backed iterations.
  • Provenance reaches the report. Source + resolved SHA must land in conditions.json, each
    run.json, benchmark.json, and the BASELINE.md built by promote.rs:231 — not just in the
    workspace.

Verification

cargo fmt --check && cargo build && cargo test && cargo clippy --all-targets -- -D warnings

Implementation requires the slow-powers:working-with-tdd skill.

Parent: #244

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions