feat(skill-eval): add an isolated Skill Assurance domain and dogfood the AgentSec skill

[trionnemesis]

Decision recorded in ADR 0008 — Skill Assurance is a separate bounded context
(docs/adr/0008-skill-assurance-bounded-context.md, currently on branch
docs/adr-0008-skill-assurance).

Why

.claude/skills/agentsec/SKILL.md is written as a list of executable invariants —
preview before run, never claim a verdict, never mint approvals, not_tested is
not pass, a detection gap is two fixes. Nothing verifies any of them, and
.claude/README.md already ranks that file as the one layer a prompt can talk out
of. The invariants most worth trusting have the least enforcement behind them.

What

Build skill_eval as a bounded context. It shares the harness infrastructure —
policy guard, allowlist, fixtures, trace normalisation, report normaliser, CI — and
shares no domain types with the purple-team side.

Per ADR 0008:

• separate schema: kind: SkillEvalSuite, apiVersion: agentsec.dev/v1alpha1
• separate runner protocol SkillHarnessRunner, not a widened ExecutorName
• separate verdict type; PurpleVerdict is not extended
• separate store file results/skill_eval.db, not new tables in agentsec.db
• separate CLI namespace (agentsec skill ...) and a separate workflow
• crossover rule: a security behaviour of the agent is an ordinary AGT-*
  scenario, judged by the existing evaluator, unchanged

In scope: trigger accuracy, process constraints, policy-boundary adherence,
stability across harness and model changes.

Out of scope: brand voice, general productivity skills, capability
benchmarking as a product. If that line stops holding, the answer is a separate
repository — not a quiet widening of this context.

Sequence

Phase 0 — lands now, needs no model and no credentials

• SkillEvalSuite Pydantic model + JSON Schema
• static validator: frontmatter, reference paths, script existence, digests,
  dangerous references
• agentsec skill validate
• blocking static CI job

Phase 1 — blocked on the roadmap's "one real staging agent, end to end"

• clean workspace factory (repo / HOME / config isolation)
• SkillHarnessRunner for Claude Code, one harness only
• trace + artifact capture: tool sequence, loaded skills, file diff, tokens
• deterministic graders: exit code, schema validation, file contents, static
  analysis, tool-call trace, refusal evidence
• skill_eval store and reporting
• first reference suite dogfooding this repo's own skill — positive,
  negative, boundary, adversarial and regression cases against the six
  non-negotiables
• control-attribution ablation: skill / settings.json permission /
  hooks/guard_agentsec.py

Phase 2 — after Phase 1 is proven

• separate nightly workflow, trend-reporting only
• cross-harness runners
• advisory semantic judge, never gating (ADR 0002 is unamended)
• narrow MCP surface keyed on a committed suite_id, if warranted at all

Preconditions that are not negotiable

• A flake policy must exist before the first stochastic job merges.
  Repetitions, pass-rate threshold, quarantine path. All three current CI jobs
  run without model credentials; this is the first that will not. A nightly job
  that goes red on variance gets muted within a month, and a muted gate looks
  like coverage while providing none.
• Results are recorded as rates, not verdicts. Trial count, pass rate,
  interval — never a single trial promoted to a gate.
• Comparisons must name what moved. skill_changed, suite_changed,
  fixture_changed, model_changed, harness_changed. A score that fell after
  a model upgrade is environment drift, not a skill regression.
• No MCP tool takes a prompt, a skill path, a workspace path or a runner
  command. ADR 0003 holds; run_arbitrary_prompt and eval already fail the
  build.

[trionnemesis]

Owner decision / implementation handoff for the next AI agent

Decision

Proceed in this repository, using the bounded-context design in PR #15 / ADR 0008.

Do not widen Scenario, PurpleVerdict, or the existing four-axis evaluator to carry skill-quality outcomes, and do not split Skill Assurance into another repository yet. The shared policy, fixture, trace, store, reporting, CI, and MCP infrastructure is still the dominant cost.

The product target is now explicit:

From a user-selected project directory, run purple-team checks for the AI agent plus assurance checks for its skills, then display the combined — but semantically separate — results in a Claude Cowork Live Artifact.

For the first supported path, Claude Desktop/Cowork + a locally registered AgentSec MCP server are required dependencies, not optional integrations.

Documentation correction required before implementation

docs/deployment.md currently says local MCP does not apply to Cowork. That statement is stale as of July 2026. Current Anthropic documentation says:

• Cowork on desktop can reach local files and local connectors / plugins containing local MCP servers.
• Live Artifacts are desktop-only and can refresh from approved connectors and local files.

References:

• https://support.claude.com/en/articles/15520349-use-claude-cowork-on-web-desktop-and-mobile
• https://support.claude.com/en/articles/14729249-use-live-artifacts-in-claude-cowork

Update the deployment ADR/docs to distinguish:

1. Desktop local POC: Cowork task + local files + local MCP + local Live Artifact.
2. Remote/shared deployment: remote MCP or sanitised export, OAuth, redaction, and the existing Option C boundary.

Non-negotiable architecture boundaries

1. Directory selection happens outside the MCP tool arguments. Resolve it from the process boundary (cwd, AGENTSEC_WORKSPACE, ${CLAUDE_PROJECT_DIR}) and a committed project manifest. No MCP tool may accept a workspace path, skill path, prompt, URL, shell command, or runner command.
2. One execution trace, two independent planes. Purple security results remain PurpleVerdict; skill results use SkillEvalVerdict. The report may compose them visually but must not merge their semantics.
3. The Live Artifact receives a versioned dashboard-safe DTO only. It must not receive raw transcripts, credentials, tenant/principal identifiers, unrestricted OTel/Wazuh fields, storage paths, or raw evidence bundles.
4. Artifact refresh must be read-only. Opening or refreshing the Live Artifact must never start a scan, mutate a finding, or mint/consume an approval. Execution remains an explicit Cowork/Claude Code action through the execution MCP surface.
5. Fail closed. Missing graders, missing traces, uncheckable assertions, unsupported skill formats, or absent redaction policy must produce error / not_tested, never green.

---

Suggested implementation sequence

PR 1 — selected-project workspace and discovery

• Merge/rebase on PR docs(adr): Skill Assurance as a separate bounded context #15 first; do not implement against an unmerged or divergent ADR.
• Add agentsec init for the currently selected repository directory.
• Prefer a repo-local .agentsec/ layout for suites, policy, results, and generated reports; preserve compatibility with the current standalone workspace layout.
• Add a committed .agentsec/project.yaml containing stable IDs and relative, reviewed locations. It must not contain credentials.
• Canonicalise the project root and reject traversal, symlink escape, and paths outside the selected workspace.
• Initial Claude adapter discovery should inventory at least:
  • .claude/skills/**/SKILL.md
  • .claude/agents/**
  • .claude/settings.json
  • .claude/hooks/**
  • project CLAUDE.md
  • project MCP configuration
• Discovery output must use stable IDs; subsequent MCP calls address project_id, suite_id, skill_id, or target_id, never a caller-supplied path.

PR 2 — Phase 0 static Skill Assurance

Implement the existing issue checklist:

• SkillEvalSuite Pydantic model and JSON Schema.
• Static validation for frontmatter, references, scripts, digests, unsafe references, missing files, and out-of-workspace references.
• agentsec skill validate and machine-readable JSON output.
• Blocking static CI workflow without model credentials.
• Fixture corpus covering valid, invalid, adversarial, path-escape, symlink, and unsupported-format cases.
• Record unsupported or undiscovered surfaces as not_tested; do not silently omit them from coverage.

PR 3 — local MCP + Cowork Live Artifact vertical slice

• Add a project-scoped local registration example using ${CLAUDE_PROJECT_DIR} / AGENTSEC_WORKSPACE; no absolute path committed to the repository.
• Expose a narrow read-only resource such as agentsec://dashboard/latest backed by a versioned JSON schema.
• The DTO should contain only:
  • project and scan metadata
  • purple verdict summary and four-axis coverage
  • Skill Assurance rates / outcomes
  • not_tested and error coverage
  • current actionable findings
  • comparison metadata naming what changed
  • links/IDs for explicit drill-down actions
• Reuse or finish the confidentiality work in Harden read-only MCP gateway: omit mutating tools and redact resource output #7 before exporting any evidence-derived fields. Unknown output types must be refused, not serialised generically.
• Add a Cowork Live Artifact reference implementation or reproducible creation instructions that render the DTO as:
  • overall posture
  • purple axes
  • skill trigger/process/security results
  • latest-versus-previous trend
  • filters by severity, verdict, skill, scenario, and tested state
  • expandable failed-check details without raw secret-bearing evidence
• Opening/refreshing the artifact must perform read operations only.
• Include an end-to-end desktop smoke test: selected folder → local MCP connected → static scan → report DTO → Live Artifact render/refresh.

PR 4 — dynamic SkillHarnessRunner, only after the real staging-agent milestone

• Clean workspace/HOME/config isolation.
• Claude/Cowork runner as the only initial harness.
• Capture tool sequence, loaded skills, file diff, exit status, refusal evidence, model/harness version, and token metadata.
• Deterministic graders only for blocking decisions.
• Repetitions, pass-rate threshold, confidence interval, and quarantine policy before any stochastic CI job merges.
• Positive, negative, boundary, adversarial, regression, and control-attribution ablation cases.
• A security behaviour discovered by the skill runner must also be promotable to a normal AGT-* scenario; the skill plane does not replace the purple plane.

---

Minimum acceptance matrix

Case	Expected result
Valid repo with one valid skill	discovered and statically evaluated
Repo with no supported skill surface	explicit not_tested, not green
Skill references ../../secret	validation error; no file read outside root
Symlink points outside root	validation error
Unknown MCP argument such as path or prompt	refused and audited
Missing trace/grader/backend	error or not_tested, never pass
Purple scenario fails silently	detection_gap independent of skill result
Skill fires on unrelated task	false_activation; PurpleVerdict unchanged
Hook blocks a bad action while skill does nothing	ablation attributes the protection to the hook
Live Artifact refresh	reads dashboard DTO only; starts no run
DTO contains transcript/token/credential-like fixture	redacted or export refused
Same scan repeated without changes	stable IDs and deterministic static result
Skill/model/harness/suite changes	comparison names exactly what moved

Recommended PR boundaries

Keep the work reviewable: workspace/discovery, static assurance, dashboard integration, and dynamic runner should be separate PRs. Do not combine the entire issue into one generated patch.

The first useful vertical slice is complete when a user opens a chosen repository in Claude Desktop/Cowork, the local AgentSec MCP resolves that project without accepting a path argument, agentsec skill validate produces a versioned safe report, and a Live Artifact renders and refreshes it without gaining execution capability.

[trionnemesis]

AI-agent execution handoff — current-state decision matrix and first PR boundary

Validated against the repository state on 2026-07-31. This comment narrows the owner decision above into an executable first assignment; it does not replace it.

Current dependency graph

PR #15 / ADR 0008
        ↓
PR A: selected-project workspace + Claude surface discovery
        ↓
PR B: static Skill Assurance
        ↓
PR C: dashboard-safe DTO + Cowork Live Artifact
        ↑
Issue #7: resource allowlist + redaction must be completed first

Real staging-agent milestone
        ↓
PR D: dynamic SkillHarnessRunner

Current state:

• PR Close all twelve findings from the local deployment review #12 is merged. It fixed the report rollups, MCP argument boundary, read-only tool registration, and other local-deployment findings.
• PR docs(adr): Skill Assurance as a separate bounded context #15 is open and mergeable. Treat ADR 0008 as a hard architecture dependency. Do not implement Skill Assurance against main as if the ADR did not exist. If implementation starts before merge, branch from the PR docs(adr): Skill Assurance as a separate bounded context #15 head and keep the final PR dependent on docs(adr): Skill Assurance as a separate bounded context #15.
• Issue Harden read-only MCP gateway: omit mutating tools and redact resource output #7 remains partially open. The execution tools are no longer advertised by the read-only gateway, but resource allowlisting, dashboard-safe DTOs, deterministic redaction, and fail-closed export are still required before a Live Artifact may consume evidence-derived data.
• docs/deployment.md is stale about Cowork. The supported local POC is now Claude Desktop/Cowork with local files and a local connector/plugin containing the AgentSec MCP server; Live Artifacts are desktop-only and can refresh from approved connectors and local files. Keep the existing remote/hybrid model for shared deployment, but do not force the desktop POC through a remote MCP.

---

Decision matrix

Concern	Rejected direction	Selected direction	Required guardrail
Skill evaluation domain	Widen Scenario, Contract, PurpleVerdict, or the four-axis evaluator	skill_eval bounded context in this repository	Separate schema, runner protocol, verdict, store, CLI namespace, and CI workflow
Repository boundary	Split to a new repository now	Stay in AgentSec while security/reliability scope and shared infrastructure dominate	Revisit the split only if this becomes general skill capability benchmarking
Selected project directory	MCP argument such as path, workspace, skill_path, or arbitrary prompt	Resolve from process boundary (AGENTSEC_WORKSPACE, ${CLAUDE_PROJECT_DIR}, canonical cwd) plus committed .agentsec/project.yaml	No caller-controlled locator; reject traversal, symlink escape, and files outside the root
Agent and Skill relationship	Merge both outcomes into one score	One captured trace, two independent evaluation planes	PurpleVerdict remains unchanged; SkillEvalVerdict must not mask a security finding
First useful slice	Start with a stochastic Claude runner	Workspace discovery followed by deterministic static Skill Assurance	No model credentials or semantic judge in the first blocking jobs
Live Artifact data source	Render raw Run, evidence bundles, transcripts, Wazuh/OTel payloads, or SQLite paths	Versioned dashboard-safe DTO exposed by a narrow read-only MCP resource	Explicit field allowlist, fixture-tested redaction, fail closed for unknown output types
Artifact refresh	Opening/refreshing the artifact starts a scan or changes findings	Refresh reads the latest safe DTO only	Execution is a separate explicit Cowork/Claude Code action; no approval may be minted or consumed by the artifact
Missing coverage	Omit unsupported Skills or missing graders and show green	Report not_tested or error explicitly	No vacuous pass and no denominator that silently drops unsupported surfaces
Dynamic stability	Treat one model run as a verdict	Repetitions, pass rate, interval, and change attribution	Record skill_changed, suite_changed, fixture_changed, model_changed, and harness_changed

Product flow that must remain possible

1. The user selects or opens a repository directory in Claude Desktop/Cowork.
2. The local AgentSec MCP process starts with AGENTSEC_WORKSPACE=${CLAUDE_PROJECT_DIR} or an equivalent canonical process-bound workspace.
3. AgentSec discovers reviewed project surfaces and maps them to stable IDs; no MCP call supplies a filesystem path.
4. The user explicitly previews and starts a scan through the execution surface.
5. Purple security results and Skill Assurance results are persisted independently.
6. A read-only resource returns only the versioned dashboard DTO.
7. The Cowork Live Artifact renders and refreshes that DTO without acquiring execution capability.

---

First assignment for the next AI agent: PR A only

Create one reviewable PR for selected-project workspace and discovery. Stop after this boundary; do not continue into the Skill runner or Live Artifact in the same patch.

Required changes

• Rebase on, or branch from, PR docs(adr): Skill Assurance as a separate bounded context #15 so the implementation follows ADR 0008.
• Correct docs/deployment.md to distinguish:
  • desktop-local POC: Cowork + approved local files + local AgentSec MCP + local Live Artifact;
  • remote/shared deployment: remote MCP or sanitised export, OAuth/RBAC, redaction, and the existing hybrid boundary.
• Add agentsec init for the currently selected project root.
• Prefer a repo-local .agentsec/ layout while preserving the existing standalone AgentSec workspace layout.
• Add a committed .agentsec/project.yaml with a stable project_id and reviewed relative locations only. It must not contain credentials, endpoints, prompts, or commands.
• Implement one canonical workspace resolver used by CLI, service, and future MCP adapters. Resolve the root from the process boundary; do not duplicate path logic in each interface.
• Reject .. traversal, symlink escape, alternate-root escape, nonexistent roots, and non-directory roots before reading project content.
• Add initial Claude adapter discovery for at least:
  • .claude/skills/**/SKILL.md
  • .claude/agents/**
  • .claude/settings.json
  • .claude/hooks/**
  • project CLAUDE.md
  • project-local MCP configuration
• Return stable IDs and relative display paths. All later calls must address project_id, skill_id, suite_id, or target_id, never a caller-supplied path.
• Record unsupported or malformed surfaces in discovery output; do not silently discard them.
• Add machine-readable JSON output suitable for the later dashboard normaliser.

Required tests

• Valid repository with one Skill is discovered deterministically.
• Repository with no supported Skill surfaces reports zero discovered plus explicit coverage metadata; it is not represented as green.
• ../../secret reference is rejected without reading the target.
• A symlink inside the repository pointing outside it is rejected.
• Equivalent path spellings resolve to the same project and stable IDs.
• Moving the checkout without changing committed relative content does not change IDs that are defined as repository-stable.
• Credentials or absolute local paths in .agentsec/project.yaml are rejected.
• Existing purple scenarios, verdicts, report output, and MCP contract tests remain unchanged.

Explicitly out of scope for PR A

• No SkillHarnessRunner.
• No model invocation, trial repetition, pass-rate gate, or semantic judge.
• No extension of Scenario, ExecutorName, Contract, or PurpleVerdict.
• No MCP parameter named path, workspace, prompt, url, command, or runner command.
• No raw evidence export and no Live Artifact reading agentsec://runs/{id}/evidence.
• No artifact-triggered execution.
• No broad dashboard redesign.

Stop condition

The PR is complete when a repository opened as the selected workspace can be initialised, safely discovered, and represented by stable machine-readable project metadata, while every existing purple-team invariant and MCP locator restriction still passes. Open that PR with the tests and documented follow-on dependencies, then stop for review before starting static Skill Assurance.

[trionnemesis]

Marking deferred per the repository-hygiene recommendation in #32:

Skill Assurance (skill_eval) ... 停止在 0.x 主線實作；拆 repo/package 或延後

ADR 0008 (separate bounded context) and the PR-A handoff comments above stand as the design record if this is picked back up. Nothing in this issue's Phase 0/1/2 checklists has landed — it stays open, deferred, not closed, since the decision is "not now" rather than "not planned."

---

Generated by Claude Code