Skip to content

feat(prompt): v6 auditor — journal legitimacy (H10) and scoring formula fix - #44

Merged
cpitzi merged 2 commits into
mainfrom
agent/v6-auditor
Jun 19, 2026
Merged

feat(prompt): v6 auditor — journal legitimacy (H10) and scoring formula fix#44
cpitzi merged 2 commits into
mainfrom
agent/v6-auditor

Conversation

@lentago-claude-runner

Copy link
Copy Markdown
Contributor

Origin

Chris resolved the #7 data-source question in favor of a hybrid indexing-plus-free-blacklist approach and chose to bundle the #43 scoring-formula fix into the same v6 release, since both are vNext prompt changes to the same new file. Both issues close together here.

Summary

  • Heuristic 10 — journal legitimacy and predatory-venue flagging (closes [v4] Implement predatory journal flagging #7): hybrid whitelist-plus-community-list approach. DOAJ, PubMed/MEDLINE, Scopus, and Web of Science serve as primary positive signals; Beall's archived list and Stop Predatory Journals provide secondary corroboration only. Flags Elevated in isolation; escalates to High when combined with any other heuristic trigger. Factual, non-accusatory language throughout — "predatory" is never used as a verdict; "potentially predatory" or "unverified venue" at most. Grey literature is exempt.
  • Scoring formula fix (closes Scoring formula: D×3 base cost makes the headline score misleading for large reference lists #43): removed the D × 3 base cost. New formula: Score = 100 − (H × 12) − (E × 5) − (M × 2), floored at 0. A fully-clean reference list of any length now scores 100. % Defensible is surfaced as a prominent complementary signal in the Executive Dashboard. Risk-band thresholds and gauge color zones recalibrated (90–100 green / 70–89 amber / 50–69 orange / 0–49 red).

Files changed

File What changed
prompts/v6-auditor.md New file — copy of v5 with both changes integrated
test-sets/predatory-venues.md New file — 5-reference H10 test set (3 traps, 2 clean controls)
docs/heuristics.md Full H10 section; scoring formula section; v6 version history row; H10 in COPE table and heuristic interaction
README.md Ten heuristics; updated scoring formula; Shipped in v6 roadmap section; updated repo layout and Running an Audit reference
CLAUDE.md v6 as current; v5 as previous live version for diffing; H10 in the heuristics description
roadmap/v4-features.md Predatory journal flagging marked done (shipped in v6); prioritized list updated
test-sets/real-articles/README.md Scoring note updated to reflect v6 formula; v5 scores noted as not comparable

What is NOT in this PR

  • v5 is frozen: prompts/v5-auditor.md, test-sets/adversarial-30.md, and all committed reports are unchanged.
  • No audit run. See validation note below.

Validation follow-up (suggested)

v6 needs a fresh baseline run before being treated as production-validated. The baseline should confirm:

  1. H10 detection: test-sets/predatory-venues.md — all three trap entries flag correctly (two Elevated for H10 alone, one High for H10 + H7), and both clean controls (JOGNN, BMC Pregnancy and Childbirth) remain Defensible.
  2. Scoring calibration: clean real articles (Ahmadinezhad, Patriksson) should score near 100 under v6 rather than the 5 and 22 they scored under v5; the dirty corpus (Amarnath, Madhukar) should still score low.
  3. Adversarial-30 regression: v6 should match or exceed v5's adversarial-30 sensitivity (100%) and specificity (92.9%). Because the formula changed, the numerical scores will differ, but the H/E/M/D tier assignments should be stable.

Because the scoring formula changed, v6 baseline metrics will not be directly comparable to the v5 numbers committed in reports/adversarial-30-v5-2026-06-19.html.

Prompt-Origin

Implemented by claude-runner[bot] per task prompt. Origin context: Chris resolved the #7 data-source question in favor of a hybrid indexing-plus-free-blacklist approach and chose to bundle the #43 scoring-formula fix into the same v6 release since both are vNext prompt changes to the same new file.

Closes #7
Closes #43

claude-runner Bot and others added 2 commits June 19, 2026 17:21
…ng formula fix (closes #7, closes #43)

Add Heuristic 10 — journal legitimacy and predatory-venue flagging —
using a hybrid whitelist-plus-community-list approach: primary positive
signals from DOAJ, PubMed, Scopus, and Web of Science clear any journal;
secondary corroboration from Beall's archived list and Stop Predatory
Journals supports a flag but is never the sole basis. Flags Elevated in
isolation; escalates to High when combined with any other heuristic.
Factual, non-accusatory classification language throughout (no "predatory"
as a verdict; "potentially predatory" or "unverified venue" at most).

Fix scoring formula: remove the D×3 base cost present in v4/v5 that made
a perfectly clean 30-reference article score 10. New formula:
Score = 100 − (H×12) − (E×5) − (M×2), floored at 0. A fully-clean list
of any length now scores 100. % Defensible added as a prominent
complementary signal in the Executive Dashboard. Risk-band thresholds and
gauge color zones recalibrated for the new scale (90–100 green, 70–89
amber, 50–69 orange, 0–49 red).

New files: prompts/v6-auditor.md, test-sets/predatory-venues.md.
Updated: docs/heuristics.md (H10 section, scoring section, v6 version
history row), README.md (ten heuristics, scoring, roadmap, repo layout),
CLAUDE.md (v6 current, v5 for diffing), roadmap/v4-features.md (#7
marked done), test-sets/real-articles/README.md (scoring note updated).

v5 is frozen. v6 scores are not directly comparable to v4/v5 baselines.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… + heuristic count

Reviewer-side correction on the v6 branch before merge. The two H10 clean
controls had DOIs that did not resolve to the cited paper, which would make
them flag under other heuristics — defeating their purpose as clean controls:

- Ref 2: DOI 10.1016/j.jogn.2021.06.007 resolved to Shin et al. (a different
  paper), not the cited Edmonds et al. Replaced with Faridvand et al. 2018,
  International Journal of Nursing Practice, DOI 10.1111/ijn.12659 (verified
  against Crossref — resolves to the cited paper).
- Ref 5: DOI 10.1186/s12884-015-0448-7 returned 404; the Vogel Robson paper is
  actually in The Lancet Global Health, not BMC Pregnancy and Childbirth.
  Corrected journal + DOI to 10.1016/S2214-109X(15)70094-X (verified resolves).

Also corrected CLAUDE.md heuristic count (nine -> ten) to match README, since
v6 adds Heuristic 10 (journal legitimacy).

Prompt-Origin: Issue #7 + #43 v6 release; reviewer caught unverified clean-control DOIs during PR #44 review.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@cpitzi

cpitzi commented Jun 19, 2026

Copy link
Copy Markdown
Owner

Reviewer correction pushed before merge. During review I verified the H10 clean-control references against Crossref and found both had broken DOIs that would have made them flag under other heuristics, defeating their purpose as clean controls:

  • Ref 2: 10.1016/j.jogn.2021.06.007 resolved to Shin et al. (a different paper), not the cited Edmonds et al. → replaced with Faridvand et al. 2018, Int. J. Nursing Practice, 10.1111/ijn.12659 (verified resolves to the cited paper).
  • Ref 5: 10.1186/s12884-015-0448-7 → 404; the Vogel Robson paper is actually in The Lancet Global Health, not BMC. Corrected journal + DOI to 10.1016/S2214-109X(15)70094-X (verified).

Both replacements re-verified HTTP 200 and resolve to the cited paper. Also bumped CLAUDE.md's heuristic count nine→ten to match the README. The H10 prompt logic and the #43 scoring fix were verified correct and unchanged.

@cpitzi
cpitzi merged commit a0602a8 into main Jun 19, 2026
1 check passed
@cpitzi
cpitzi deleted the agent/v6-auditor branch June 19, 2026 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scoring formula: D×3 base cost makes the headline score misleading for large reference lists [v4] Implement predatory journal flagging

1 participant