Separate SkillBuilder evals from runtime skill conformance evals

[karimad]

The current evals/skillbuilder/scenarios.ts scenarios appear to focus on validating the SkillBuilder output: plan structure, tool selection, constraints, and expected steps.

This is useful for evaluating the builder layer, but it is different from validating whether a generated skill actually works when used by a target agent architecture.

A runtime/conformance evaluation layer could validate the full lifecycle:

1. Generate the skill from SkillBuilder
2. Export the generated skill artifact
3. Load the skill into a target agent/runtime
4. Provide required capabilities, tools, and authentication context
5. Execute the task
6. Score the resulting behavior/output

This would help answer questions that builder-level evals cannot:

• Can another runtime consume the generated skill?
• Are required tools and capabilities available?
• Does authentication/configuration work?
• Does the skill achieve the intended task outcome?

One possible direction would be to keep both evaluation types:

• evals/skillbuilder/
  • Validate generated skill plans/specifications
  • Check tool choices, structure, and constraints
• evals/skill-runtime/
  • Validate exported skills in representative target architectures
  • Measure execution success and interoperability

This could also provide a foundation for a skill interoperability/conformance approach across different agent runtimes.

I would be interested in contributing to this direction if it aligns with the project roadmap.