Part of #244 (full-codebase eval testbed). Interacts with #253 (sourced skills).
Why
An eval has exactly one skill under test. skill_name is a single required string
(schema/evals.schema.json:10); Mode is binary (src/core/types.rs:225); each ConditionEntry
carries one skill_path. Other skills in the directory are staged as ambient siblings into both
arms (stage_sibling_skills, src/cli/run/staging/mod.rs), so they are background, never treatment.
#244 wants the suite itself to be the thing under test:
With a real test case, we could see the full suite of slow-powers skills working together, with the
final code being a useful product by which to judge the plugin as a whole. This is also likely to be
a way to catch places where skills conflict or cause confusion.
Inter-skill conflict is invisible under the current model by construction: conflicting skills are
either both ambient (present in both arms, so they cancel) or only one is treatment.
Scope
skill_name grows a set form alongside the single-skill form.
ConditionEntry carries a roster rather than one skill_path.
- The
__skill_invoked meta-check becomes per-skill. Both the deterministic transcript check
(check_skill_invoked_from_transcript, which matches one staged slug) and the LLM fallback rubric
currently assume a single skill.
- Sibling staging needs a clear rule for what remains ambient when the treatment is a set.
Once #253 makes skills a sourced, recorded input, the roster is part of that record — extend it rather than
introducing a second way to name skills.
Files
schema/evals.schema.json
src/core/types.rs
src/cli/run/orchestrate/{resolve,stage}.rs
src/pipeline/grade/judge_tasks.rs
src/cli/args.rs, src/validation/evals.rs
Acceptance criteria
- An eval naming a set of skills stages all of them in the treatment arm and none in the control arm.
__skill_invoked reports per-skill, and a suite where only some skills fired is distinguishable
from one where all did.
- Single-skill evals behave exactly as today.
- The condition roster appears in
conditions.json and the run report.
- Mode B (revision) works with a set — suite-at-revision-A vs suite-at-revision-B.
Cross-cutting requirements
- Mode B parity. Acceptance must hold for both
Mode::NewSkill and Mode::Revision
(src/core/types.rs:225), and the snapshot/promote path (src/workspace/snapshot.rs,
src/workspace/promote.rs) must keep working against codebase-backed iterations.
- Provenance reaches the report. Source + resolved SHA must land in
conditions.json, each
run.json, benchmark.json, and the BASELINE.md built by promote.rs:231 — not just in the
workspace.
Verification
cargo fmt --check && cargo build && cargo test && cargo clippy --all-targets -- -D warnings
Implementation requires the slow-powers:working-with-tdd skill.
Parent: #244
Part of #244 (full-codebase eval testbed). Interacts with #253 (sourced skills).
Why
An eval has exactly one skill under test.
skill_nameis a single required string(
schema/evals.schema.json:10);Modeis binary (src/core/types.rs:225); eachConditionEntrycarries one
skill_path. Other skills in the directory are staged as ambient siblings into botharms (
stage_sibling_skills,src/cli/run/staging/mod.rs), so they are background, never treatment.#244 wants the suite itself to be the thing under test:
Inter-skill conflict is invisible under the current model by construction: conflicting skills are
either both ambient (present in both arms, so they cancel) or only one is treatment.
Scope
skill_namegrows a set form alongside the single-skill form.ConditionEntrycarries a roster rather than oneskill_path.__skill_invokedmeta-check becomes per-skill. Both the deterministic transcript check(
check_skill_invoked_from_transcript, which matches one staged slug) and the LLM fallback rubriccurrently assume a single skill.
Once #253 makes skills a sourced, recorded input, the roster is part of that record — extend it rather than
introducing a second way to name skills.
Files
schema/evals.schema.jsonsrc/core/types.rssrc/cli/run/orchestrate/{resolve,stage}.rssrc/pipeline/grade/judge_tasks.rssrc/cli/args.rs,src/validation/evals.rsAcceptance criteria
__skill_invokedreports per-skill, and a suite where only some skills fired is distinguishablefrom one where all did.
conditions.jsonand the run report.Cross-cutting requirements
Mode::NewSkillandMode::Revision(
src/core/types.rs:225), and the snapshot/promote path (src/workspace/snapshot.rs,src/workspace/promote.rs) must keep working against codebase-backed iterations.conditions.json, eachrun.json,benchmark.json, and theBASELINE.mdbuilt bypromote.rs:231— not just in theworkspace.
Verification
Implementation requires the
slow-powers:working-with-tddskill.Parent: #244