Skip to content

[#201] Correct the two mislabelled confidence scores in the STT spike - #213

Merged
realproject7 merged 1 commit into
mainfrom
task/201-spike-floor-band
Aug 5, 2026
Merged

[#201] Correct the two mislabelled confidence scores in the STT spike#213
realproject7 merged 1 commit into
mainfrom
task/201-spike-floor-band

Conversation

@realproject7

Copy link
Copy Markdown
Owner

Fixes #201

§1.1 of the spike cited 0.9134 and 0.9330 as "legitimate outputs" bracketing the 0.888 hallucination. Both are damaged. Because of that mislabelling the section could not state the floor band the same data supports.

EPIC Alignment

What was wrong, and what it cost

docs/CALIBRATION.md publishes the labelled table; the spike had not caught up with it.

score run what it actually is
0.9134 f1-pad / medium damaged — invented an "AT HE" prefix, corrupted "Sarah" → "SBA" (CALIBRATION.md:91,152)
0.9330 f3 / small damaged — "electrocardogram", "2 weeks" (CALIBRATION.md:106,153)

The cost was not only the two labels. With them read as clean, a threshold above 0.888 looked like it would immediately start rejecting good output — so the section could not say that no clean utterance scored below 0.9557 at all. That is the finding the mislabelling suppressed.

What the section now states

Labelled and sorted (CALIBRATION.md:118-119,145-155):

  • clean utterances span 0.9557–0.9949 (n=9)
  • damaged utterances span 0.8884–0.9936 (n=11)
  • a floor just under the lowest clean score rejects 5 of 11 damaged and 0 of 9 clean

That is a narrower claim than the section implied, not a softer one, and it is stated as a band rather than a recommendation.

The conclusion is deliberately unchanged

This is the part most at risk of drifting under a correction, so it is explicit: six damaged utterances still score above the lowest clean one (CALIBRATION.md:121-125), where no threshold reaches them. The ranges overlap, confidence does not separate the classes, and #111's floor table stays at its seed values. §2.1.1's open question — whether these values are the gate's own signal — is untouched and still unresolved.

The band and the overlap are not in tension: the band addresses the low tail, the overlap is about the classes.

Self-Verification

  • Scope containment is the main risk on a documentation correction, so it was checked mechanically. git status --short reports exactly one modified file, docs/STT-ENGINE-SPIKE.md. No src/, crates/, src-tauri/, benchmark, or docs/CALIBRATION.md change.
  • Every figure traced to source before it was written, not transcribed from the ticket: CALIBRATION.md:91 (0.9134), :106 (0.9330), :118-119 (the two ranges and their n), :145-147 (5 of 11, 0 of 9), :121-125 (the six above the lowest clean).
  • The same error does not appear elsewhere: grep -n "0\.913\|0\.933\|legitimate" over the document returns only the corrected passage, so no second instance was left standing.
  • Preserved claims verified by grep, not by memory: ranges overlap and does not cleanly separate both still present at :88-91; seven §2.1.1 cross-references intact.
  • no-stub-gate and color-guard pass.

Deviations

  • The edit runs slightly past the three lines the assignment named. Correcting the labels made the sentence carry two clauses that no longer followed from each other, and the floor band the assignment also required has nowhere to go inside it. The change is one contiguous passage in §1.1 and touches no other section; flagging it because "lines 78–80" was stated as durable scope.
  • CALIBRATION.md's correction-history note is left as-is. It records the first draft's mislabelling as a caught error, and that note is still accurate after this change — the spike is what had not caught up. Editing it was also out of scope.
  • No new measurement. Every number here comes from STT live verification & per-model floor calibration on real audio (closes the #92/#93/#94 verification gap) #111's existing table; nothing was re-run, and nothing in this PR could change what the data says.

…ection

Section 1.1 cited 0.9134 and 0.9330 as "legitimate outputs" bracketing the
0.888 hallucination. Both are damaged: 0.9134 (f1-pad / medium) invented an
"AT HE" prefix and corrupted "Sarah" to "SBA", and 0.9330 (f3 / small) produced
"electrocardogram". docs/CALIBRATION.md:91,106,151-153 is the labelled source,
and its own correction-history note records this as the error under review.

Because of the mislabelling the section could not state the floor band the same
data supports: clean utterances span 0.9557-0.9949 (n=9) and damaged ones
0.8884-0.9936 (n=11), so a floor just under the lowest clean score would reject
five of the eleven damaged utterances and none of the nine clean ones. That is a
narrower claim than the section implied, and it is now stated.

The conclusion is unchanged and deliberately so: six damaged utterances still
score above the lowest clean one, so the ranges overlap, confidence does not
separate the classes, and #111's floor table stays at its seed values. The
Section 2.1.1 open question is untouched.

No product code, benchmark, or CALIBRATION.md change - one doc file, verified by
git status. no-stub-gate and color-guard pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@realproject7

Copy link
Copy Markdown
Owner Author

RE2 — APPROVE

PR #213 (#201) @ 67f693061a815c4dad96ce3e9c76c873b2a44762 — all 5 CI checks green. One file, +14/−4. This corrects the defect I originally reported on #199, so I re-derived every figure from the merged source rather than from my own earlier review.

Checked (evidence)

Riskiest part: the corrected numbers must match the merged calibration table, since the entire point of #201 is that this document drifted from it. I parsed docs/CALIBRATION.md at origin/main and recomputed each claim independently:

claim in the correction re-derived from merged CALIBRATION.md
clean 0.9557–0.9949 (n=9) clean n=9, 0.9557–0.9949 ✓
damaged 0.8884–0.9936 (n=11) damaged n=11, 0.8884–0.9936 ✓
floor under lowest clean rejects 5 of 11 damaged, 0 of 9 clean 5 damaged below 0.9557 ✓
six damaged score above the lowest clean 6 ✓
0.9134 damaged — invented "AT HE", "Sarah"→"SBA" f1-pad / mediuminvented "AT HE" prefix; "Sarah" → "SBA"
0.9330 damaged — "electrocardogram" f3 / small"electrocardogram"; "2 weeks"

Every figure matches. Nothing was transcribed from the ticket.

The conclusion is correctly preserved. The overlap finding, the "confidence does not separate the classes" statement, and #111's floors staying at seed values all survive verbatim. That matters because the last time this passage was touched, a correct-but-under-evidenced claim was withdrawn as an over-correction; this edit does the opposite of that mistake.

Band and overlap are not in tension, and the document now says so explicitly: the band addresses the low tail, the overlap is about the classes. That distinction is what makes both claims true at once, and stating it is what stops the next reader from "fixing" one of them.

Scope holds. git diff --name-only origin/main..67f6930docs/STT-ENGINE-SPIKE.md alone. No product code, no benchmark, no CALIBRATION.md. §2.1.1 is untouched — zero diff lines reference it, and the 2.1.1 cross-reference count is 7 on both main and this SHA, so nothing was orphaned.

On the scope deviation @dev flagged for ruling: the edit runs a few lines past 78–80. That reads as necessary rather than creeping — correcting the labels leaves a sentence whose two clauses no longer follow from each other, and the floor band @Head also asked for has nowhere to live inside three lines. It is one contiguous passage in §1.1 of one file. Forcing it back into three lines would produce worse prose for no reduction in blast radius. Raising it rather than quietly widening a scope called durable is the right instinct, and I'd rule for the wider passage — though the call is @Head's, not mine.

Worth recording

The correction names something neither reviewer articulated at the time: the mislabelling did not merely get two labels wrong, it suppressed a finding. With 0.9134 and 0.9330 read as clean, a threshold above 0.888 appeared to reject good output immediately — which is precisely why the section could not state that no clean utterance scored below 0.9557 at all. A wrong label did not just make one sentence false; it made a true and useful sentence unsayable.

Gates: all 5 CI green at 67f6930app-macos 2m32s, release-invariants 2m26s, packages-linux 42s, color-guard, no-stub-gate.

(The shared bot token cannot file a formal GitHub approval, so this comment plus my chat message is the RE2 verdict of record.)

@project7-interns project7-interns left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE

Epic Alignment: PASS

The documentation correction matches docs/CALIBRATION.md, preserves the overlap conclusion and §2.1.1, and stays within the sanctioned single-file scope.

Checked (evidence)

  • Labels verified against source: 0.9134 is damaged (invented “AT HE”, “Sarah”→“SBA”) and 0.9330 is damaged (“electrocardiogram”) (docs/CALIBRATION.md:91,106).
  • Derived figures match source: clean 0.9557–0.9949 (n=9), damaged 0.8884–0.9936 (n=11), and 5/11 damaged vs 0/9 clean below 0.9557 (docs/CALIBRATION.md:118-119,145-155; docs/STT-ENGINE-SPIKE.md:84-91).
  • Conclusion preserved: six damaged scores remain above the lowest clean score and ranges overlap (docs/CALIBRATION.md:121-125; docs/STT-ENGINE-SPIKE.md:88-97).
  • Scope: live diff contains only docs/STT-ENGINE-SPIKE.md.
  • Riskiest part: correcting labels without turning the floor band into a recommendation; the edited text explicitly retains the unresolved §2.1.1 question and seed-floor conclusion.
  • Kill-list: documentation delta scanned — clean.
  • CI: gh pr checks 213 → app-macos, color-guard, no-stub-gate, packages-linux, and release-invariants all passing.

Findings

None.

Decision

The source labels, counts, preserved conclusion, and file boundary all satisfy #201. Approved.

@realproject7
realproject7 merged commit 89792f0 into main Aug 5, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[docs][bug] Correct unsupported calibration confidence labels in #199 and #200

2 participants