[#201] Correct the two mislabelled confidence scores in the STT spike - #213
Conversation
…ection Section 1.1 cited 0.9134 and 0.9330 as "legitimate outputs" bracketing the 0.888 hallucination. Both are damaged: 0.9134 (f1-pad / medium) invented an "AT HE" prefix and corrupted "Sarah" to "SBA", and 0.9330 (f3 / small) produced "electrocardogram". docs/CALIBRATION.md:91,106,151-153 is the labelled source, and its own correction-history note records this as the error under review. Because of the mislabelling the section could not state the floor band the same data supports: clean utterances span 0.9557-0.9949 (n=9) and damaged ones 0.8884-0.9936 (n=11), so a floor just under the lowest clean score would reject five of the eleven damaged utterances and none of the nine clean ones. That is a narrower claim than the section implied, and it is now stated. The conclusion is unchanged and deliberately so: six damaged utterances still score above the lowest clean one, so the ranges overlap, confidence does not separate the classes, and #111's floor table stays at its seed values. The Section 2.1.1 open question is untouched. No product code, benchmark, or CALIBRATION.md change - one doc file, verified by git status. no-stub-gate and color-guard pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
RE2 — APPROVEPR #213 (#201) @ Checked (evidence)Riskiest part: the corrected numbers must match the merged calibration table, since the entire point of #201 is that this document drifted from it. I parsed
Every figure matches. Nothing was transcribed from the ticket. The conclusion is correctly preserved. The overlap finding, the "confidence does not separate the classes" statement, and #111's floors staying at seed values all survive verbatim. That matters because the last time this passage was touched, a correct-but-under-evidenced claim was withdrawn as an over-correction; this edit does the opposite of that mistake. Band and overlap are not in tension, and the document now says so explicitly: the band addresses the low tail, the overlap is about the classes. That distinction is what makes both claims true at once, and stating it is what stops the next reader from "fixing" one of them. Scope holds. On the scope deviation @dev flagged for ruling: the edit runs a few lines past 78–80. That reads as necessary rather than creeping — correcting the labels leaves a sentence whose two clauses no longer follow from each other, and the floor band @Head also asked for has nowhere to live inside three lines. It is one contiguous passage in §1.1 of one file. Forcing it back into three lines would produce worse prose for no reduction in blast radius. Raising it rather than quietly widening a scope called durable is the right instinct, and I'd rule for the wider passage — though the call is @Head's, not mine. Worth recordingThe correction names something neither reviewer articulated at the time: the mislabelling did not merely get two labels wrong, it suppressed a finding. With 0.9134 and 0.9330 read as clean, a threshold above 0.888 appeared to reject good output immediately — which is precisely why the section could not state that no clean utterance scored below 0.9557 at all. A wrong label did not just make one sentence false; it made a true and useful sentence unsayable. Gates: all 5 CI green at (The shared bot token cannot file a formal GitHub approval, so this comment plus my chat message is the RE2 verdict of record.) |
project7-interns
left a comment
There was a problem hiding this comment.
Verdict: APPROVE
Epic Alignment: PASS
The documentation correction matches docs/CALIBRATION.md, preserves the overlap conclusion and §2.1.1, and stays within the sanctioned single-file scope.
Checked (evidence)
- Labels verified against source: 0.9134 is damaged (invented “AT HE”, “Sarah”→“SBA”) and 0.9330 is damaged (“electrocardiogram”) (
docs/CALIBRATION.md:91,106). - Derived figures match source: clean 0.9557–0.9949 (n=9), damaged 0.8884–0.9936 (n=11), and 5/11 damaged vs 0/9 clean below 0.9557 (
docs/CALIBRATION.md:118-119,145-155;docs/STT-ENGINE-SPIKE.md:84-91). - Conclusion preserved: six damaged scores remain above the lowest clean score and ranges overlap (
docs/CALIBRATION.md:121-125;docs/STT-ENGINE-SPIKE.md:88-97). - Scope: live diff contains only
docs/STT-ENGINE-SPIKE.md. - Riskiest part: correcting labels without turning the floor band into a recommendation; the edited text explicitly retains the unresolved §2.1.1 question and seed-floor conclusion.
- Kill-list: documentation delta scanned — clean.
- CI:
gh pr checks 213→ app-macos, color-guard, no-stub-gate, packages-linux, and release-invariants all passing.
Findings
None.
Decision
The source labels, counts, preserved conclusion, and file boundary all satisfy #201. Approved.
Fixes #201
§1.1 of the spike cited 0.9134 and 0.9330 as "legitimate outputs" bracketing the 0.888 hallucination. Both are damaged. Because of that mislabelling the section could not state the floor band the same data supports.
EPIC Alignment
docs/CALIBRATION.md, so a reader deciding on [R&D spike] Alternative STT engines (Parakeet/Nemotron/Apple SpeechAnalyzer) feasibility — BACKLOG, post-MVP #132 is not working from two scores that are labelled backwards.CALIBRATION.mdchange.What was wrong, and what it cost
docs/CALIBRATION.mdpublishes the labelled table; the spike had not caught up with it.mediumCALIBRATION.md:91,152)smallCALIBRATION.md:106,153)The cost was not only the two labels. With them read as clean, a threshold above 0.888 looked like it would immediately start rejecting good output — so the section could not say that no clean utterance scored below 0.9557 at all. That is the finding the mislabelling suppressed.
What the section now states
Labelled and sorted (
CALIBRATION.md:118-119,145-155):That is a narrower claim than the section implied, not a softer one, and it is stated as a band rather than a recommendation.
The conclusion is deliberately unchanged
This is the part most at risk of drifting under a correction, so it is explicit: six damaged utterances still score above the lowest clean one (
CALIBRATION.md:121-125), where no threshold reaches them. The ranges overlap, confidence does not separate the classes, and #111's floor table stays at its seed values. §2.1.1's open question — whether these values are the gate's own signal — is untouched and still unresolved.The band and the overlap are not in tension: the band addresses the low tail, the overlap is about the classes.
Self-Verification
git status --shortreports exactly one modified file,docs/STT-ENGINE-SPIKE.md. Nosrc/,crates/,src-tauri/, benchmark, ordocs/CALIBRATION.mdchange.CALIBRATION.md:91(0.9134),:106(0.9330),:118-119(the two ranges and their n),:145-147(5 of 11, 0 of 9),:121-125(the six above the lowest clean).grep -n "0\.913\|0\.933\|legitimate"over the document returns only the corrected passage, so no second instance was left standing.ranges overlapanddoes not cleanly separateboth still present at:88-91; seven§2.1.1cross-references intact.no-stub-gateandcolor-guardpass.Deviations
CALIBRATION.md's correction-history note is left as-is. It records the first draft's mislabelling as a caught error, and that note is still accurate after this change — the spike is what had not caught up. Editing it was also out of scope.