Problem
The claude-sdk provider does not discover workspace-level skills from .claude/skills/. When running skill-trigger evals with provider: claude-sdk, positive test cases always fail because the agent doesn't know the skill exists.
Evidence
Running multi-provider-skill-trigger.EVAL.yaml (from examples/features/agent-skills-evals/) with --target claude-sdk:
| Test |
claude-sdk result |
| should-trigger-direct-request (should_trigger: true) |
FAIL — skill not found |
| should-trigger-casual-phrasing (should_trigger: true) |
FAIL — skill not found |
| should-not-trigger-unrelated (should_trigger: false) |
PASS |
| should-not-trigger-near-miss (should_trigger: false) |
PASS |
The same eval passes all positive cases with claude-cli (claude -p subprocess).
claude-sdk session init — NO workspace skills or plugins:
"skills": ["keybindings-help", "debug", "claude-developer-platform"],
"plugins": []
claude-cli session init — workspace skills discovered automatically:
"skills": ["keybindings-help", "debug", "claude-developer-platform", "csv-analyzer", ...],
"plugins": [{"name": "frontend-design", ...}, ...]
Root Cause
claude -p CLI auto-discovers .claude/skills/ from the workspace cwd. The Claude Agent SDK's query() does not — it needs explicit configuration.
The AgentV ClaudeSdkProvider (in packages/core/src/evaluation/providers/claude-sdk.ts) passes only: model, cwd, systemPrompt, maxTurns, maxBudgetUsd, env. It does NOT pass skills or plugin config.
SDK Capabilities (already available)
The SDK (@anthropic-ai/claude-agent-sdk v0.2.49, types in sdk.d.ts) supports:
Options.plugins?: SdkPluginConfig[] — load local plugins: [{ type: 'local', path: './my-plugin' }]
- BUT
.claude/skills/ are bare skill files (SKILL.md), not plugins with plugin.json manifests
AgentDefinition.skills?: string[] — preload skill names into agent context
Options.allowedTools?: string[] / Options.disallowedTools?: string[] — tool control
Files to Modify
| File |
Change |
packages/core/src/evaluation/providers/claude-sdk.ts |
Pass skills/plugins to SDK query() options |
packages/core/src/evaluation/providers/targets.ts |
Extend ClaudeResolvedConfig if new target fields needed |
examples/features/.agentv/targets.yaml |
Add claude-sdk target definition |
Design Question
.claude/skills/ directories are bare SKILL.md files, not structured plugins. The SDK's plugins option expects { type: 'local', path } pointing to a plugin with plugin.json. Options to bridge this:
- Auto-detect and inject via system prompt — read
SKILL.md files from workspace .claude/skills/ and append to systemPrompt. Pragmatic, works today, no SDK changes needed.
- Wrap as temporary plugin — create an ephemeral plugin manifest pointing to the workspace skills. More complex but uses the SDK's native plugin system.
- Check if SDK auto-discovers from cwd — verify whether setting
cwd to the workspace root triggers built-in skill discovery in newer SDK versions. If so, this might be a version issue.
Acceptance Criteria
Problem
The
claude-sdkprovider does not discover workspace-level skills from.claude/skills/. When running skill-trigger evals withprovider: claude-sdk, positive test cases always fail because the agent doesn't know the skill exists.Evidence
Running
multi-provider-skill-trigger.EVAL.yaml(fromexamples/features/agent-skills-evals/) with--target claude-sdk:The same eval passes all positive cases with
claude-cli(claude -psubprocess).claude-sdk session init — NO workspace skills or plugins:
claude-cli session init — workspace skills discovered automatically:
Root Cause
claude -pCLI auto-discovers.claude/skills/from the workspace cwd. The Claude Agent SDK'squery()does not — it needs explicit configuration.The AgentV
ClaudeSdkProvider(inpackages/core/src/evaluation/providers/claude-sdk.ts) passes only:model,cwd,systemPrompt,maxTurns,maxBudgetUsd,env. It does NOT pass skills or plugin config.SDK Capabilities (already available)
The SDK (
@anthropic-ai/claude-agent-sdkv0.2.49, types insdk.d.ts) supports:Options.plugins?: SdkPluginConfig[]— load local plugins:[{ type: 'local', path: './my-plugin' }].claude/skills/are bare skill files (SKILL.md), not plugins withplugin.jsonmanifestsAgentDefinition.skills?: string[]— preload skill names into agent contextOptions.allowedTools?: string[]/Options.disallowedTools?: string[]— tool controlFiles to Modify
packages/core/src/evaluation/providers/claude-sdk.tsquery()optionspackages/core/src/evaluation/providers/targets.tsClaudeResolvedConfigif new target fields neededexamples/features/.agentv/targets.yamlclaude-sdktarget definitionDesign Question
.claude/skills/directories are bare SKILL.md files, not structured plugins. The SDK'spluginsoption expects{ type: 'local', path }pointing to a plugin withplugin.json. Options to bridge this:SKILL.mdfiles from workspace.claude/skills/and append tosystemPrompt. Pragmatic, works today, no SDK changes needed.cwdto the workspace root triggers built-in skill discovery in newer SDK versions. If so, this might be a version issue.Acceptance Criteria
multi-provider-skill-trigger.EVAL.yamlpasses all 4 tests with--target claude-sdk.claude/skills/are available to the SDK session