R&R-007: pooled attribute-Kappa gate not ratifiable — S4 matcher scores per-cycle, not per-message (ceiling effect)

[frank001]

Summary

Investigation (directed by Captain, following the R&R-005 citation cleanup, #35) into whether the §9.3/§10 pooled attribute-κ gate is ready for ratification. Full writeup: qa/rr-study/RR-007.md.

Finding: ratification cannot proceed, and not for the reason §9.3 anticipated. The originally-flagged concern (S4's positive population includes sub-threshold SNRs, same class of contamination as R&R-005's S1 ANOVA problem — 21/66 = 31.8% of S4 signals sit below the −12 dB decodable floor) is real, but is not the binding constraint.

The binding problem: harness/matcher.py's S4 truth/matching records one pooled truth row per (part, trial) — not one per injected message — and scores a match as "did the appraiser decode any one of the N pooled messages." Every other correct decode in that cycle is discarded (falls through to Pass 2 as a spurious "false positive", excluded from κ by the false_positive == False filter). This creates a ceiling effect: run 793a298 shows both appraisers at a perfect TP=15/FN=0, κ=1.000, PASS — while the same run, same build's S7 (which correctly matches per-message) shows OpenWSFZ recovering only 40% of co-channel stacks vs WSJT-X's 100%. S4 cannot currently distinguish good QRM handling from bad.

Recommendation

1. Attribute-κ gate remains advisory (no change to current behavior).
2. Scope a harness redesign so S4 records and matches truth per individual message (carrying that message's own SNR), analogous to S7's existing per-message design — a prerequisite for evaluating the decodable-SNR-restricted population §9.3 already calls for.
3. Re-run S4 and re-evaluate pooled κ against real per-message data before revisiting ratification.

See qa/rr-study/RR-007.md §4 for candidate acceptance criteria (draft, not yet ratified).

Scope note

Investigation and recommendation only — implementation not started, pending Captain decision on how to proceed.

[frank001]

Update (2026-07-07): fix implemented and confirmed with real live-audio data

openspec/changes/rr-study-s4-per-message-matching implemented this issue's recommendation in full (tasks 1-5): S4 truth generation now emits one row per injected message (previously one pooled row per cycle), the matcher's dead "match any one of a pooled set" branch was retired, and an informational decodable-SNR-restricted kappa variant was added alongside the unchanged full-population pooled kappa.

Confirmed with a live re-run (results/2026-07-07-df4cc89/, S1+S4+S5), replacing the degenerate 793a298 baseline:

Metric	793a298 (pre-fix)	df4cc89 (post-fix)
WSJT-X TP/FN	15 / 0	75 / 33
OpenWSFZ TP/FN	15 / 0	71 / 37
WSJT-X recovery	100.00%	69.44%
OpenWSFZ recovery	100.00%	65.74%
kappa vs truth (both)	1.000 / 1.000	0.705 / 0.669

The ceiling effect is gone; OpenWSFZ shows a real, differentiated recovery gap against WSJT-X, directionally consistent with S7's co-channel data (not a same-run comparison — see report.md Section 5 caveat). The restricted-population kappa also confirmed the original §9.3 hypothesis: recovery rises to 79.01% for both appraisers identically once sub-threshold-SNR positives are excluded.

Leaving this open, deliberately. The matcher defect itself is fixed and confirmed — that part of this issue is resolved. But whether to ratify the attribute-Kappa gate (either the pooled method, or the decodable-SNR-restricted population, or both) as a hard gate is a separate Captain decision this work does not make. Full account: qa/rr-study/RR-007.md §6 (Outcome).