0.4 — Author capability-error-explanation scenarios + run Eval 4 baseline

[bb-connor]

Phase 0 task 0.4 (PLAN.md, Phase 0 task 0.4).

Build the capability-error-explanation outcome eval (Eval 4, subjective).

Deliverables

• Author 10 scenarios in chio-pack/eval/fixtures/cap-error-explanation/ per the example template.
• Identify 3 raters + 1 alternate; record handles → pseudonyms in RATERS.md.
• Run a Run-0 inter-rater calibration round; record scores in RATERS.md and per-rater adjustments in vault/_meta/dashboards/rater-calibration.md.
• Implement chio-pack/chio_pack/eval/runners/cap_error_explanation.py per PHASE-0.md Eval 4.
• Run baseline for the raw augmentation only (the others may improve in Phase 1+); commit to ADR-0002.

Done when

• make kb-eval-outcomes reports capability-error-explanation as ok.
• RATERS.md Run-0 calibration row is filled.
• ADR-0002's row has the raw-augmentation baseline + sample size + date.

Reference

• Fixture template
• Rubric (4 dimensions × 1–5)
• Rater protocol
• PHASE-0.md Eval 4

[bb-connor]

10 / 10 scenarios committed — fixture-count gate cleared

Ten capability-error-explanation scenarios landed in f6dcbaf, each exercising the four canonical augmentations (raw / kb_brief_feature / +signed_receipt / +episode_link):

Scenario	Failure mode	Actionability lesson
revoked-cap-still-presented	revocation list rejection	retry-not-the-answer
capability-insufficient-scope	scope mismatch, cap still valid	request wider scope OR refactor action
capability-past-expiration	exp claim past with drift window	mint a fresh cap (cheap)
delegation-chain-broken-link	mid-chain signature corruption	ask parent issuer to re-sign
capability-wrong-audience	aud mismatch	request cap bound to correct target
capability-unrecognized-scope-identifier	vocabulary versioning	could be sync, typo, or x- extension
cross-issuer-revocation-attempt	NOT-SUPPORTED in v1	procedural (not bug); ask original issuer
dpop-proof-htu-mismatch	RFC 9449 binding failure	regenerate proof for the new URL
capability-mangled-jwt-signature	base64 vs base64url corruption	transport-layer fix, not kernel
concurrent-revocation-race	RVL propagation staleness	RVL tightening, not application fix

Scenarios deliberately span retry vs no-retry, kernel-side vs transport-side, and protocol-edge cases (RFC 9449, base64url encoding, vocabulary versioning, the documented v1 limitation).

Eval-outcomes report status flipped:

| capability-error-explanation | 10 | BLOCKED — runner | fixtures present; runner is Phase 1 deliverable |

RATERS.md: solo-maintainer fallback documented

PHASE-0.md assumed three named human raters. Reality at v0.0.1-vault-complete: single human (@connor). RATERS.md now documents two paths:

1. Recruit two humans (preferred — preserves cross-rater variance).
2. LLM-judge fallback — rater-B = Sonnet 4.6, rater-C = Haiku 4.5 with rubric variants. Costs noted: rubric drift undetectable if both LLMs share blind spots; subjective dimensions drift toward LLM stylistic preferences.

A new ADR-0004-rater-pool is proposed (in RATERS.md) as the unblocking ADR before ADR-0002 baseline sign-off.

Remaining for this issue

• Choose path 1 or 2 via ADR-0004 (blocks Run-0 calibration).
• Run-0 calibration round (blocks ADR-0002 sign-off).
• Runner implementation (chio_pack/eval/runners/cap_error_explanation.py) — Phase 1.
• Baseline measurement against raw augmentation only.

Fixture work is complete. Open pending the rater-pool decision and the runner.