foomakers · rucka · Jul 13, 2026 · Jul 12, 2026 · Jul 13, 2026 · Jul 13, 2026
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -63,6 +63,8 @@ jobs:
         run: pnpm hygiene:check
       - name: Run docs-staleness check
         run: pnpm docs:staleness
+      - name: Run skills-conformance check
+        run: pnpm skills:conformance
       - name: Run duplication check
         run: pnpm dup:check
       - name: Install Playwright browsers

diff --git a/package.json b/package.json
@@ -25,8 +25,9 @@
     "a11y:report:html": "turbo a11y:report:html",
     "hygiene:check": "node scripts/code-hygiene-check.js",
     "docs:staleness": "pnpm --filter @pair/website docs:staleness",
+    "skills:conformance": "pnpm --filter @pair/knowledge-hub skills:conformance",
     "dup:check": "jscpd apps packages",
-    "quality-gate": "turbo ts:check test lint && turbo prettier:fix mdlint:fix && ./tools/markdownlint-config/bin/markdownlint-fix.sh '*.md' && pnpm hygiene:check && pnpm docs:staleness && pnpm dup:check",
+    "quality-gate": "turbo ts:check test lint && turbo prettier:fix mdlint:fix && ./tools/markdownlint-config/bin/markdownlint-fix.sh '*.md' && pnpm hygiene:check && pnpm docs:staleness && pnpm skills:conformance && pnpm dup:check",
     "e2e": "pnpm --filter @pair/website e2e",
     "smoke-tests": "./scripts/smoke-tests/run-all.sh --cleanup"
   },

diff --git a/packages/knowledge-hub/eval/README.md b/packages/knowledge-hub/eval/README.md
@@ -0,0 +1,72 @@
+# Skill Trigger Eval
+
+Trigger/effectiveness eval harness for the skill corpus (story #313, T7). It measures whether the
+frontmatter `description` of each skill in `packages/knowledge-hub/dataset/.skills/**` routes an
+LLM executor to the right skill — the experiment behind principle 9 ("Evaluation") of the
+authoring standard (`apps/website/content/docs/contributing/writing-skills.mdx`): a description
+rewrite is a hypothesis; this eval is the experiment.
+
+## Contents
+
+- `trigger-prompts.json` — should-trigger / should-not-trigger prompt sets, one entry per skill
+  family. Every skill has at least one should-trigger prompt; near-miss prompts name the sibling
+  skill they are a decoy for (`near_miss_for`) and the correct destination (`expected`, `none`
+  when no skill applies).
+- `results/` — committed evidence, one file per run, named `YYYY-MM-DD-<label>.md`.
+
+## Procedure
+
+1. **Build the catalog** — extract the current name+description pairs (descriptions only, no
+   skill bodies):
+
+   ```bash
+   for f in packages/knowledge-hub/dataset/.skills/next/SKILL.md \
+            packages/knowledge-hub/dataset/.skills/*/*/SKILL.md; do
+     awk '/^---$/{c++; next} c==1 && (/^name:/ || /^description:/)' "$f"; echo
+   done
+   ```
+
+2. **Run each prompt in a fresh session** — triggering is a cold-start behavior; conversation
+   history invalidates the result. Present the executor with the full catalog and one prompt, and
+   ask: *"Given only these skill names and descriptions, which single skill would you load for
+   this request? Answer `none` if no skill applies. If two or more descriptions claim the prompt
+   equally, name all of them."*
+
+3. **Record per prompt** — id, expected, selected, verdict:
+   - `PASS` — selected matches expected.
+   - `AMBIGUOUS` — executor names two or more candidates (counts against the corpus: two
+     executors could disagree).
+   - `FAIL` — selected differs from expected.
+
+   Record honestly, including near-ties resolved by a judgment call (`PASS` with a note).
+
+4. **Summarize** — pass rate per prompt type (should-trigger / should-not-trigger), list of
+   ambiguities/failures with the competing descriptions, methodology statement (model, passes,
+   executor identity). Commit the file under `results/`.
+
+## Before/after comparison (gating a description rewrite)
+
+When a skill's description changes (e.g. T3 of #313):
+
+1. Run the full set against the **old** descriptions (or reuse the latest committed baseline if
+   descriptions are unchanged since).
+2. Apply the rewrite; run the same set against the **new** descriptions, same procedure, fresh
+   sessions.
+3. Compare per skill. **A regression on any prompt reverts that skill's description** — iterate
+   on the description, not on the eval. Improvements on previously ambiguous/failing prompts are
+   the acceptance evidence.
+4. Commit both result files; reference them in the PR.
+
+## Maintaining the prompt set
+
+- New skill ⇒ add at least one should-trigger prompt and one family near-miss.
+- Keep prompts realistic user phrasing, not description echoes — a prompt that quotes the
+  description verbatim tests nothing.
+- Prompts are grounded in current descriptions; when a trigger branch is added or removed, add or
+  retire the matching prompts in the same PR.
+
+## Related gates
+
+Static conformance (frontmatter portability, size limits, pointer resolution, catalog counts) is
+enforced separately by `pnpm skills:conformance` (`src/tools/skills-conformance-check.ts`, run
+via `ts-node`), wired into `quality-gate` and CI.
diff --git a/packages/knowledge-hub/eval/results/2026-07-12-baseline.md b/packages/knowledge-hub/eval/results/2026-07-12-baseline.md
@@ -0,0 +1,206 @@
+# Trigger Eval — Baseline (2026-07-12)
+
+Baseline run of `../trigger-prompts.json` against the **current** frontmatter descriptions
+(pre-T3 rewrite), corpus at 35 skills. This is the "before" half of the before/after evidence
+required by AC7 of story #313 and principle 9 of the authoring standard.
+
+## Methodology
+
+- **Executor**: single LLM (Claude, Fable 5), single pass per prompt. The executor was shown the
+  full catalog of 35 `name` + `description` pairs (frontmatter only — no skill bodies, no `next`
+  routing tables) and one prompt at a time, cold (no conversation history about the corpus).
+- **Instruction**: pick the single skill you would load, `none` if no skill applies, name all
+  candidates when two or more descriptions claim the prompt equally.
+- **Bias disclosure**: the prompt author and the executor are the same model/session lineage;
+  ambiguity may be *under*-reported. Near-ties resolved by judgment are marked PASS with a note
+  rather than silently passed.
+- **Verdicts**: PASS (selected = expected), AMBIGUOUS (≥2 candidates named — counts as failure:
+  two executors could disagree), FAIL (selected ≠ expected).
+
+## Summary
+
+| Type | Prompts | PASS | AMBIGUOUS | FAIL | Strict pass rate |
+| ---- | ------- | ---- | --------- | ---- | ---------------- |
+| should-trigger | 53 | 51 | 2 | 0 | 96.2% |
+| should-not-trigger | 26 | 25 | 1 | 0 | 96.2% |
+| **Total** | **79** | **76** | **3** | **0** | **96.2%** |
+
+### Ambiguities (all involve the assess-* family)
+
+| Id | Prompt (abridged) | Expected | Candidates named | Root cause |
+| -- | ----------------- | -------- | ---------------- | ---------- |
+| AD-12 | "Recommend where we should track our work items" | assess-pm | assess-pm, setup-pm | Both descriptions claim tool *selection*: assess-pm "proposes a tool choice" vs setup-pm "guiding selection from supported options" |
+| AD-14 | "Add Redis as a dependency — check whether that's consistent with our stack" | assess-stack | assess-stack, verify-adoption | assess-stack's branch is buried in the cryptic clause "implementation (add dependency)"; verify-adoption's "checks code and configuration against project adoption files" claims the same prompt |
+| PB-N2 | "We haven't picked a tracker yet — what should we use?" | assess-pm | assess-pm, setup-pm | Same assess-pm / setup-pm selection overlap as AD-12 |
+
+### assess-* near-clone evidence (motivates T3)
+
+The 10 assess-* descriptions share ~60% of their text verbatim ("resolution cascade (Argument >
+Adoption > Assessment)", "emits rendered adoption content + target (output-only, writes
+nothing)", "the caller persists via /record-decision", "Idempotent"). In this run **all routing
+inside the family was carried by the single domain noun** (architecture, infrastructure,
+observability, …); the shared scaffold contributed zero discrimination while consuming most of
+the description budget. Consequences observed:
+
+- The ambiguities fall into two kinds (AD-12, PB-N2, AD-14) — both sitting exactly where the
+  domain noun stops discriminating (pm vs setup-pm; stack vs verify-adoption).
+- Domain-keyword prompts pass, but only because the eval prompts contain the noun; prompts
+  phrased without it (AD-02 "bring Copilot into our workflow", AD-10 "zero telemetry") were
+  near-ties resolved by judgment (see notes below).
+- SNT decoys aimed at the family (AD-N3 "assess the security posture") produce a felt pull toward
+  the family before resolving to `none` — the word "assess" alone activates 10 near-identical
+  candidates.
+
+### Other description defects noticed during the run (not counted, feed T1/T3)
+
+- `grill` and `checkpoint` descriptions reference non-existent skills (`/brainstorm`, "a future
+  `/publish-pr`") as live composition targets.
+- `next`'s catalog/count issue is router-level (33 vs 35, checkpoint and grill unreachable) and
+  is not exercised by this description-only eval — covered by the conformance gate's
+  warning-only catalog-count rule until T1.
+- Only `next` has when-to-use phrasing; every other description is a capability summary, so
+  passes lean on noun overlap between prompt and description rather than trigger branches.
+
+## Per-prompt results
+
+Verdict key: PASS / AMBIGUOUS / FAIL. Notes flag near-ties resolved by judgment (pull =
+competing description considered and rejected).
+
+### router
+
+| Id | Type | Expected | Selected | Verdict | Note |
+| -- | ---- | -------- | -------- | ------- | ---- |
+| RT-01 | ST | next | next | PASS | |
+| RT-02 | ST | next | next | PASS | |
+| RT-N1 | SNT | refine-story | refine-story | PASS | pull: next ("what to work on"); resolved by refine-story's "Todo to Refined ... selection" |
+
+### assess-decision
+
+| Id | Type | Expected | Selected | Verdict | Note |
+| -- | ---- | -------- | -------- | ------- | ---- |
+| AD-01 | ST | assess-ai | assess-ai | PASS | |
+| AD-02 | ST | assess-ai | assess-ai | PASS | near-tie: assess-stack's "add dependency" branch competes for "bring Copilot into our workflow"; resolved because Copilot is named an AI tool |
+| AD-03 | ST | assess-architecture | assess-architecture | PASS | |
+| AD-04 | ST | assess-architecture | assess-architecture | PASS | |
+| AD-05 | ST | assess-infrastructure | assess-infrastructure | PASS | |
+| AD-06 | ST | assess-infrastructure | assess-infrastructure | PASS | |
+| AD-07 | ST | assess-methodology | assess-methodology | PASS | |
+| AD-08 | ST | assess-methodology | assess-methodology | PASS | "iterations, ceremonies" absent from description; routed on "development process" ≈ methodology |
+| AD-09 | ST | assess-observability | assess-observability | PASS | |
+| AD-10 | ST | assess-observability | assess-observability | PASS | "telemetry" absent from description; routed via monitoring/logging/tracing synonymy |
+| AD-11 | ST | assess-pm | assess-pm | PASS | |
+| AD-12 | ST | assess-pm | assess-pm, setup-pm | AMBIGUOUS | see summary table |
+| AD-13 | ST | assess-stack | assess-stack | PASS | |
+| AD-14 | ST | assess-stack | assess-stack, verify-adoption | AMBIGUOUS | see summary table |
+| AD-15 | ST | assess-testing | assess-testing | PASS | |
+| AD-16 | ST | assess-testing | assess-testing | PASS | keyword collision: "coverage" also in assess-code-quality; resolved by "should we aim for" = strategy |
+| AD-N1 | SNT | setup-pm | setup-pm | PASS | "already decided" defuses assess-pm |
+| AD-N2 | SNT | record-decision | record-decision | PASS | |
+| AD-N3 | SNT | none | none | PASS | felt pull toward assess-* family from the bare word "assess"; no description covers security |
+| AD-N4 | SNT | none | none | PASS | assess-testing proposes strategy, does not write tests |
+
+### assess-report
+
+| Id | Type | Expected | Selected | Verdict | Note |
+| -- | ---- | -------- | -------- | ------- | ---- |
+| AR-01 | ST | assess-code-quality | assess-code-quality | PASS | |
+| AR-02 | ST | assess-code-quality | assess-code-quality | PASS | |
+| AR-03 | ST | assess-debt | assess-debt | PASS | |
+| AR-04 | ST | assess-debt | assess-debt | PASS | |
+| AR-N1 | SNT | verify-quality | verify-quality | PASS | "quality gate" discriminates from assess-code-quality |
+| AR-N2 | SNT | none | none | PASS | pull: assess-debt ("debt" keyword); defused by its "output-only, returns a report" |
+
+### verification
+
+| Id | Type | Expected | Selected | Verdict | Note |
+| -- | ---- | -------- | -------- | ------- | ---- |
+| VF-01 | ST | verify-quality | verify-quality | PASS | |
+| VF-02 | ST | verify-quality | verify-quality | PASS | |
+| VF-03 | ST | verify-done | verify-done | PASS | |
+| VF-04 | ST | verify-adoption | verify-adoption | PASS | |
+| VF-N1 | SNT | none | none | PASS | verify-* descriptions are detection-only |
+| VF-N2 | SNT | review | review | PASS | |
+
+### decision-recording
+
+| Id | Type | Expected | Selected | Verdict | Note |
+| -- | ---- | -------- | -------- | ------- | ---- |
+| DR-01 | ST | record-decision | record-decision | PASS | |
+| DR-02 | ST | record-decision | record-decision | PASS | |
+| DR-N1 | SNT | assess-stack | assess-stack | PASS | |
+
+### pm-and-backlog
+
+| Id | Type | Expected | Selected | Verdict | Note |
+| -- | ---- | -------- | -------- | ------- | ---- |
+| PB-01 | ST | write-issue | write-issue | PASS | |
+| PB-02 | ST | write-issue | write-issue | PASS | |
+| PB-03 | ST | setup-pm | setup-pm | PASS | |
+| PB-04 | ST | estimate | estimate | PASS | |
+| PB-N1 | SNT | plan-tasks | plan-tasks | PASS | |
+| PB-N2 | SNT | assess-pm | assess-pm, setup-pm | AMBIGUOUS | see summary table |
+
+### ddd-mapping
+
+| Id | Type | Expected | Selected | Verdict | Note |
+| -- | ---- | -------- | -------- | ------- | ---- |
+| DM-01 | ST | map-subdomains | map-subdomains | PASS | |
+| DM-02 | ST | map-contexts | map-contexts | PASS | |
+| DM-N1 | SNT | none | none | PASS | |
+| DM-N2 | SNT | none | none | PASS | pull: map-subdomains ("domain" keyword) |
+
+### session-continuity
+
+| Id | Type | Expected | Selected | Verdict | Note |
+| -- | ---- | -------- | -------- | ------- | ---- |
+| SC-01 | ST | checkpoint | checkpoint | PASS | |
+| SC-02 | ST | checkpoint | checkpoint | PASS | "resume mode" named in description |
+| SC-03 | ST | grill | grill | PASS | |
+| SC-04 | ST | grill | grill | PASS | name match carries it |
+| SC-N1 | SNT | none | none | PASS | pull: checkpoint ("so work survives") |
+| SC-N2 | SNT | none | none | PASS | |
+
+### manual-testing
+
+| Id | Type | Expected | Selected | Verdict | Note |
+| -- | ---- | -------- | -------- | ------- | ---- |
+| MT-01 | ST | design-manual-tests | design-manual-tests | PASS | |
+| MT-02 | ST | execute-manual-tests | execute-manual-tests | PASS | |
+| MT-N1 | SNT | assess-testing | assess-testing | PASS | |
+| MT-N2 | SNT | verify-quality | verify-quality | PASS | "automated" defuses execute-manual-tests |
+
+### setup-ops
+
+| Id | Type | Expected | Selected | Verdict | Note |
+| -- | ---- | -------- | -------- | ------- | ---- |
+| SO-01 | ST | setup-gates | setup-gates | PASS | |
+| SO-02 | ST | manage-flags | manage-flags | PASS | |
+| SO-N1 | SNT | verify-quality | verify-quality | PASS | |
+| SO-N2 | SNT | none | none | PASS | config option ≠ feature flag |
+
+### planning-processes
+
+| Id | Type | Expected | Selected | Verdict | Note |
+| -- | ---- | -------- | -------- | ------- | ---- |
+| PP-01 | ST | specify-prd | specify-prd | PASS | |
+| PP-02 | ST | specify-prd | specify-prd | PASS | |
+| PP-03 | ST | plan-initiatives | plan-initiatives | PASS | |
+| PP-04 | ST | plan-epics | plan-epics | PASS | |
+| PP-05 | ST | plan-stories | plan-stories | PASS | |
+| PP-06 | ST | plan-tasks | plan-tasks | PASS | |
+| PP-07 | ST | refine-story | refine-story | PASS | |
+| PP-N1 | SNT | write-issue | write-issue | PASS | pull: plan-epics ("epic" keyword); "from this text I already wrote" defuses breakdown |
+| PP-N2 | SNT | none | none | PASS | |
+| PP-N3 | SNT | estimate | estimate | PASS | |
+
+### execution-orchestrators
+
+| Id | Type | Expected | Selected | Verdict | Note |
+| -- | ---- | -------- | -------- | ------- | ---- |
+| EO-01 | ST | bootstrap | bootstrap | PASS | |
+| EO-02 | ST | bootstrap | bootstrap | PASS | |
+| EO-03 | ST | implement | implement | PASS | |
+| EO-04 | ST | review | review | PASS | |
+| EO-N1 | SNT | none | none | PASS | pull: implement; a one-line typo is not a story |
+| EO-N2 | SNT | none | none | PASS | pull: bootstrap ("project setup"); CLI install is not skill territory |
+| EO-N3 | SNT | verify-quality | verify-quality | PASS | "no full review needed" defuses review |