feat(baseline): v6 production baseline — H10 functional check, regression gate, scoring calibration - #46
Merged
Merged
Conversation
…, scoring calibration (#45) Three runs against the v6 auditor prompt (prompts/v6-auditor.md): RUN 1 — predatory-venues-v6-2026-06-19.html (H10 functional check) Refs 1 and 4 Elevated (H10 only); ref 3 High (H10+H7); refs 2 and 5 Defensible. H10 PASS. RUN 2 — adversarial-30-v6-2026-06-19.html + metrics.md (regression gate) 9H / 7E / 1M / 13D — identical to v5 baseline. 100% sensitivity, 92.9% specificity. H10 fires on 0 refs. H9 fires on 0 refs. Raw score −45 (floored 0; v5 was −84 floored 0 — D×3 removal accounts for difference). REGRESSION PASS. RUN 3a — patriksson-2024-v6-2026-06-19.html (scoring calibration, clean article) 26 refs; score 96 (vs. v5 ~22). 2M (Crossref deposit errors), 24D. H10: 0 fires. CALIBRATION PASS. RUN 3b — ahmadinezhad-2024-v6-2026-06-19.html (scoring calibration, genuine finding) 30 refs; score 83 (vs. v5 ~5). 1H (Ref 6 — H3+H7+H9, impossible volume/shadow paper, COPE investigation warranted), 1E (Ref 25 — H10, Journal of Human Sciences unindexed in DOAJ/PubMed/Scopus/WoS), 28D. H10 fired on 1 ref only; 0 false positives on legitimate indexed journals. Overall verdict: v6 holds the v5 detection baseline; H10 fires correctly on predatory-venues set and produces 0 false positives on adversarial-30; new scoring formula (no D×3 penalty) produces sensible calibration numbers on real articles. Closes #45 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Origin
Implements issue #45: establish the v6 production baseline by running the v6 auditor (
prompts/v6-auditor.md) against all required test sets and committing HTML reports plus a regression-and-calibration verdict.Closes #45
What's in this PR
Six new files:
reports/predatory-venues-v6-2026-06-19.htmlreports/adversarial-30-v6-2026-06-19.htmlreports/adversarial-30-v6-2026-06-19-metrics.mdreports/patriksson-2024-v6-2026-06-19.htmlreports/ahmadinezhad-2024-v6-2026-06-19.htmlreports/README.mdNo prompt files, test sets, or v5 baseline files were modified.
Overall Verdict
H10 fires correctly: PASS
RUN 1 (predatory-venues): Refs 1 and 4 → Elevated (H10 only, fictitious journals). Ref 3 → High (H10 + H7, fictitious journal + paper not found). Refs 2 and 5 (clean controls, indexed journals) → Defensible. All four expected outcomes matched.
v6 holds the v5 detection baseline: PASS
RUN 2 (adversarial-30): Tier assignments identical to v5 — 9H / 7E / 1M / 13D. Sensitivity 100%, specificity 92.9%, 0 false negatives, F1 97.0% — all unchanged. H10 fired on zero references (no unindexed venues in the set). H9 fired on zero references. The sole false positive remains Ref 27 (Moderate, JOGNN DOI retirement artifact — unchanged from v5). Raw score −45 (floored to 0) vs. v5's −84 (floored to 0): difference explained entirely by removal of D × 3 term; floored scores are both 0, so the score comparison is moot for this heavily-fabricated set.
New scoring formula produces sensible calibration numbers: PASS
RUN 3a (Patriksson et al. 2024, clean 26-ref article): Score 96 under v6 vs. ~22 under v5. Two Moderate flags (Crossref deposit errors, not integrity concerns), 24 Defensible, H10 fires on zero journals. Score near top of scale correctly signals near-complete integrity.
RUN 3b (Ahmadinezhad et al. 2024, 30-ref article with one genuine finding): Score 83 under v6 vs. ~5 under v5. One High flag (Ref 6 — H3 + H7 + H9: impossible volume, shadow paper, temporal impossibility — COPE investigation warranted), one Elevated flag (Ref 25 — H10: Journal of Human Sciences absent from DOAJ/PubMed/Scopus/WoS), 28 Defensible. H10 fired on one reference only; zero false positives on legitimate indexed journals. Score correctly reflects an article with one serious concern in an otherwise clean list.