Fix case-sensitive ADR triage confidence parsing

[shengqi-gensee]

Summary

• parse ADR triage confidence from the already normalized lowercase response
• add regression coverage for the uppercase CONFIDENCE: format requested by the triage prompt

Problem

The parser detects confidence: in a lowercased copy of the model response, but then splits the original response using the lowercase field name. When a model follows ADR's requested uppercase CONFIDENCE: format, that split fails and the parser silently returns the default confidence of 0.8.

This does not change the triage classification. It fixes the confidence recorded for Tier 1 results.

Test plan

cd Detection
pytest tests/test_adr_baseline.py -q
13 passed

[CLAassistant]

All committers have signed the CLA.

[pengyuzhang]

The fix is correct — I reproduced both sides: pre-fix, CONFIDENCE: 0.99 returned 0.8 (the guard tested result_lower while the split ran on the original-case result_text, so the split found no match and IndexError hit the fallback); post-fix it returns 0.99. Lowercasing can't corrupt a numeric token, and maxsplit=1 doesn't change which occurrence is selected, so it's a no-risk clarification.

One follow-up while you're in here: the expression still assumes the number is the bare next whitespace token after the first confidence:, so several common model outputs keep hitting the same 0.8 default this PR set out to eliminate. Verified by running the patched line:

Model output	Token extracted	Result
CONFIDENCE: 0.99 (prompt format)	0.99	0.99 — fixed by this PR
**CONFIDENCE:** 0.95	**	0.8
REASONING: low confidence: agent intent unclear … CONFIDENCE: 0.30	agent	0.8
CONFIDENCE: 0.75.	0.75.	0.8
CONFIDENCE: 95%	95%	0.8

The third row is the awkward one: an incidental confidence: inside the reasoning text wins the split, so the real value further down is never reached. Markdown emphasis is probably the most likely in practice, since models reach for ** around field labels unprompted.

A line-anchored parse closes all of them — scan lines, match a confidence: prefix, strip * / [ / ] / % / trailing punctuation before float().

Worth saying explicitly that this is not a regression from this diff and nothing is broken downstream today: confidence is only logged and echoed into DetectionResult.confidence_score, main_detector.py:441 appends it to a confidence_scores list that is never read, and every extract_metrics caller in plot_paper_figures.py discards the confidence array (_ in that position), with no ROC/AUC consuming it. So a wrong value can't flip a classification, suppress an alert, or distort a published figure — this is cleanup, not a blocker.

One note on verification: I traced the parse by executing the expression directly rather than running the suite (no pytest available in the review environment), so the new test is verified by construction, not by a green run.

[shengqi-gensee]

Thanks for the detailed review. We’ve updated the parser to handle the additional formats you identified, including Markdown, punctuation, incidental confidence: text, and percentages, with regression tests.

We found this issue while experimenting with context and provenance in the detection component. We’ll submit additional improvements from that work as separate PRs soon.

[pengyuzhang]

Re-reviewed at ec42a44. The line-anchored parse is the right shape and the case-sensitivity bug is properly gone — all five parametrized cases pass when I run the pattern directly, and CONFIDENCE: 85 falling back to 0.8 is a real improvement over the base, which recorded 85.0.

Three follow-ups below. Two of them are cases where the new parse does worse than the code it replaces, which is why I'd suggest fixing them here rather than after merge.

Checked and cleared: re is imported at module scope, so the new re.compile is safe; the out-of-range branch correctly continues scanning later lines instead of breaking, so CONFIDENCE: 1.1\nCONFIDENCE: 0.5 still yields 0.5; match.group("percent") returns "" rather than None, so the truthiness check is sound; and pytest is a declared dev dep with parametrize as a builtin marker, so --strict-markers won't reject the new tests.

Severity note for prioritization: nothing branches on this value today — it is logged and echoed into DetectionResult.confidence_score, main_detector.py:441 appends it to a list that is never read, and every extract_metrics caller in plot_paper_figures.py discards the confidence array. So none of these can flip a classification or distort a figure; they affect the recorded number this PR exists to make correct.

[pengyuzhang]

The $ anchor makes any trailing commentary fall back to the default — and for lowercase output this regresses against the code being replaced.

Ran the base and PR parsers side by side:

Input	base	this PR
confidence: 0.9 - agent behavior normal	0.9	0.8
CONFIDENCE: 0.9 (high)	0.8	0.8
CLASSIFICATION: BENIGN | CONFIDENCE: 0.9	0.8	0.8

The old whitespace split took the token immediately after the label and parsed it fine; the anchor now requires the line to end at the number (plus at most one punctuation character). Models routinely append a short justification after the value even when the prompt asks for a bare field, so this is a common shape — and it silently records the same 0.8 default the PR set out to eliminate.

A trailing (?![\d.]) boundary instead of requiring end-of-line, or an unanchored search as a fallback when no anchored line matches, would close it.

[pengyuzhang]

Percent scaling is unconditional, so a fractional value carrying a % is recorded 100x too small — and it passes the range guard.

CONFIDENCE: 0.95% parses to 0.0095, which satisfies 0.0 <= parsed_confidence <= 1.0 and is stored as a near-zero confidence. The base implementation raised ValueError on 0.95% and fell back to 0.8, so this newly produces a confidently wrong number where it previously produced an honest default.

Suggested change

	if match.group("percent"):
	parsed_confidence /= 100
	if match.group("percent") and parsed_confidence > 1.0:
	parsed_confidence /= 100

That keeps CONFIDENCE: 95% → 0.95 working while leaving the ambiguous fractional-percent form to fall through to the default.

[pengyuzhang]

The bold-label form with the colon outside the asterisks still misses.

**CONFIDENCE:** 0.95   → 0.95  ✓
**CONFIDENCE**: 0.95   → 0.8   ✗
*CONFIDENCE*: 0.9      → 0.8   ✗
> **Confidence**: 0.6  → 0.8   ✗

\**confidence\s*:\** allows asterisks before the label and after the colon, but not between the label and the colon — and bolding the label while leaving the colon outside the emphasis is the more idiomatic Markdown of the two. The comment added just above cites Markdown decoration as the motivation for this pattern, so the targeted case is only half covered.

Suggested change

	\**confidence\s*:\**\s* # Label, optionally wrapped in asterisks.
	\**confidence\**\s*:\s*\** # Label, optionally wrapped in asterisks.

Not a regression — the base implementation also returned 0.8 for these.

[shengqi-gensee]

Thanks—good catches. I’ve updated the parser to accept trailing commentary and both common Markdown label styles while preserving the line-anchored match, so incidental confidence: text in reasoning remains ignored.

For percentage-formatted values, I adopted the suggested magnitude-based interpretation with an explicit boundary for the ambiguous case:

• Values at or above 1% are treated as using a 0–100 scale, so 95% → 0.95, 95.1% → 0.951, and 1% → 0.01.
• Values below 1% retain their numeric portion, so 0.95% → 0.95. This intentionally favors the likely model intent—an already-normalized confidence with an extra %—over literal percentage arithmetic.

This special case is documented directly in the code and covered by unit tests.

I intentionally left CLASSIFICATION: BENIGN | CONFIDENCE: 0.9 at the existing 0.8 fallback. Both the base and PR parsers reject that form, and supporting it would require weakening the line-anchored rule or adding special delimiter handling. Since the prompt requests separate fields on separate lines, I kept that behavior unchanged.

Regression tests now cover trailing commentary, Markdown variants, percentage boundaries, malformed numeric suffixes, and the intentionally unsupported pipe-delimited form. The focused suite passes with 30 tests.

[pengyuzhang]

Approving at ecbea23. All three items from my previous review are fixed, and I verified each by executing the final compiled pattern rather than reading it:

Previously	Now
confidence: 0.9 - agent behavior normal → 0.8	0.9 — (?=\s|$) replaced the $ anchor, so the regression against the base implementation is gone
CONFIDENCE: 0.95% → 0.0095	0.95 — percent scaling now gated on >= 1.0
**CONFIDENCE**: 0.95 → 0.8	0.95 — emphasis accepted on both sides of the colon

All 18 parametrized cases pass, including the new negative cases for numeric runoff (0.95.2, 0.95high). The original case-sensitivity bug — detecting on a lowercased copy while splitting the original — is properly gone.

Five non-blocking observations, none of which need to hold up the merge:

1. ^ anchor still costs one shape the old code handled. CLASSIFICATION: BENIGN | confidence: 0.9 returned 0.9 on base and returns 0.8 here. Narrow — the uppercase twin failed on base too, so it only affects lowercase-emitting models — but test_invalid_confidence_uses_default now asserts that shape is invalid when the field itself is well-formed. Worth being a deliberate choice rather than one a test locks in.

2. The percent rule can invert meaning. CONFIDENCE: 0.5% records 0.5 (medium) rather than 0.005 (near-zero), and the discontinuity is sharp: 0.99% → 0.99 while 1.0% → 0.01. The comment documents this as a bet on likely model intent and I think that is the right call — but it belongs in the PR description too, since the value reaches DetectionResult.confidence_score and the benchmark plots where no reader sees the code comment.

3. The 0.8 fallback is silent. No match, out-of-range (1.1), and bare-integer (95) all record 0.8 with no log line, and 0.8 is itself a plausible genuine value — so a parse failure is indistinguishable from a real answer in the output JSON. That is the same silent-failure class this PR fixes, relocated to the fallback. A logger.debug when the default fires would make the next occurrence diagnosable, and is how you would notice item 4 happening in production.

4. Decoration coverage is now arbitrary rather than principled. Handled: -, *, >, **. Still missing (each records 0.8): 1. CONFIDENCE: 0.9, 4) Confidence: 0.9, CONFIDENCE: (0.9), CONFIDENCE: 0.9/1.0, Confidence Score: 0.9, CONFIDENCE_SCORE: 0.9, CONFIDENCE = 0.9, and CONFIDENCE: 85. None is a regression, but the pattern has absorbed enough special cases that the omissions read as oversights.

5. Nits. The verbose pattern is recompiled on every _parse_triage_result call (re.compile bypasses the module cache) — a module-level constant is the natural home. And percent scaling introduces float noise: CONFIDENCE: 99.9% records 0.9990000000000001; the suite's 95.1 divides exactly, so CI never shows it. round(..., 4) after scaling keeps the recorded scores clean.

Severity context for all of the above: nothing gates on this value. It is logged and echoed into DetectionResult.confidence_score; main_detector.py:441 appends it to a list that is never read, and every extract_metrics caller in plot_paper_figures.py discards the confidence array. So none of these can flip a classification or suppress an alert.

One caveat on verification: pytest is not installed in my review checkout, so I confirmed behavior by replicating the exact compiled pattern against ~30 model-output shapes rather than by running the suite. Worth a green CI run before merge.