config: calibration drift is invisible — no signal when normal/flag/ignore become stale

[ooloth]

Problem

normal, flag, and ignore in projects.json are the calibration layer that makes general prompts project-specific. But:

• There's no tooling to validate whether these fields are still accurate
• There's no visibility into which ignore patterns are actively firing vs. obsolete
• There's no signal when flag criteria produce only noise or only nothing

A project can silently drift into one of two bad states:

1. Over-flagging: flag criteria are too broad → scan files issues every run that are not actionable → issue queue fills with noise → backpressure kicks in → legitimate issues stop being filed
2. Under-flagging: flag criteria are too narrow or ignore patterns are too aggressive → real problems go undetected → the scan appears to be working but produces no output

Neither state is detectable without manually inspecting the issue queue and comparing it to the codebase.

What's needed

Calibration audit tooling:

1. Ignore pattern hit rates: Track how often each ignore entry suppresses a finding across runs. An ignore pattern that never fires is dead weight; one that fires 100% of the time may indicate the underlying condition is now always present and should be addressed.

2. Flag criteria yield rates: Track what fraction of scan runs produce findings that pass triage and become posted issues. A yield rate of 0% over 4+ weeks suggests flag criteria are miscalibrated or the project genuinely has no issues in this category.

3. Staleness warnings: If a scan type produces zero posted issues for N consecutive runs, surface a warning (log line, comment on a calibration issue, or a dedicated scan:calibration label on a filed issue).

Related

• The retrospective scan (prompts/agency/history/scans.md) already analyzes cross-run patterns. Calibration health could be an output of the retrospective rather than a separate tool.
• See idea: Axiom ingestion for run telemetry #33 (Axiom ingestion) — if run telemetry were in Axiom, calibration drift would be queryable via APL without any new tooling.

Definition of Done

• The retrospective scan's output schema includes a calibration_warnings[] field
• Each warning identifies: project, scan type, condition (always-zero / always-fires), and number of consecutive runs
• Warnings result in a filed GitHub issue labeled scope:config with the calibration audit as the body

Out of Scope

• Automated calibration self-tuning (modifying projects.json automatically)
• Cross-project calibration comparison