0.2 — Define repeated-mistake-rate classifier (Eval 2)

[bb-connor]

Phase 0 task 0.2 (PLAN.md, Phase 0 task 0.2).

Build the repeated-mistake-rate outcome eval (Eval 2).

Deliverables

• Implement session-log writer: chio-dev sessions emit ~/.chio-dev/sessions/<session-id>.jsonl per PHASE-0.md "Session log format".
• Implement heuristic mistake classifier (reverted_edit, failed_then_re_edited, superseded_quickly, wrong_capability_scope, bypassed_guard, other) per rubrics.md "Mistake taxonomy".
• Implement LLM-judge layer (Sonnet 4.6) using the prompt skeleton in rubrics.md.
• Log ≥ 20 baseline sessions from current arc workflow.
• Compute baseline rate over rolling 50-session window; commit to ADR-0002.

Done when

• make kb-eval-outcomes reports repeated-mistake-rate as ok.
• ADR-0002's row for this eval has a baseline number, sample size, and date.

Reference

• PHASE-0.md Eval 2
• rubrics.md mistake taxonomy