feat(prompt): v6 auditor — journal legitimacy (H10) and scoring formula fix - #44
Merged
Conversation
…ng formula fix (closes #7, closes #43) Add Heuristic 10 — journal legitimacy and predatory-venue flagging — using a hybrid whitelist-plus-community-list approach: primary positive signals from DOAJ, PubMed, Scopus, and Web of Science clear any journal; secondary corroboration from Beall's archived list and Stop Predatory Journals supports a flag but is never the sole basis. Flags Elevated in isolation; escalates to High when combined with any other heuristic. Factual, non-accusatory classification language throughout (no "predatory" as a verdict; "potentially predatory" or "unverified venue" at most). Fix scoring formula: remove the D×3 base cost present in v4/v5 that made a perfectly clean 30-reference article score 10. New formula: Score = 100 − (H×12) − (E×5) − (M×2), floored at 0. A fully-clean list of any length now scores 100. % Defensible added as a prominent complementary signal in the Executive Dashboard. Risk-band thresholds and gauge color zones recalibrated for the new scale (90–100 green, 70–89 amber, 50–69 orange, 0–49 red). New files: prompts/v6-auditor.md, test-sets/predatory-venues.md. Updated: docs/heuristics.md (H10 section, scoring section, v6 version history row), README.md (ten heuristics, scoring, roadmap, repo layout), CLAUDE.md (v6 current, v5 for diffing), roadmap/v4-features.md (#7 marked done), test-sets/real-articles/README.md (scoring note updated). v5 is frozen. v6 scores are not directly comparable to v4/v5 baselines. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… + heuristic count Reviewer-side correction on the v6 branch before merge. The two H10 clean controls had DOIs that did not resolve to the cited paper, which would make them flag under other heuristics — defeating their purpose as clean controls: - Ref 2: DOI 10.1016/j.jogn.2021.06.007 resolved to Shin et al. (a different paper), not the cited Edmonds et al. Replaced with Faridvand et al. 2018, International Journal of Nursing Practice, DOI 10.1111/ijn.12659 (verified against Crossref — resolves to the cited paper). - Ref 5: DOI 10.1186/s12884-015-0448-7 returned 404; the Vogel Robson paper is actually in The Lancet Global Health, not BMC Pregnancy and Childbirth. Corrected journal + DOI to 10.1016/S2214-109X(15)70094-X (verified resolves). Also corrected CLAUDE.md heuristic count (nine -> ten) to match README, since v6 adds Heuristic 10 (journal legitimacy). Prompt-Origin: Issue #7 + #43 v6 release; reviewer caught unverified clean-control DOIs during PR #44 review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owner
|
Reviewer correction pushed before merge. During review I verified the H10 clean-control references against Crossref and found both had broken DOIs that would have made them flag under other heuristics, defeating their purpose as clean controls:
Both replacements re-verified HTTP 200 and resolve to the cited paper. Also bumped CLAUDE.md's heuristic count nine→ten to match the README. The H10 prompt logic and the #43 scoring fix were verified correct and unchanged. |
4 tasks
This was referenced Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Origin
Chris resolved the #7 data-source question in favor of a hybrid indexing-plus-free-blacklist approach and chose to bundle the #43 scoring-formula fix into the same v6 release, since both are vNext prompt changes to the same new file. Both issues close together here.
Summary
Score = 100 − (H × 12) − (E × 5) − (M × 2), floored at 0. A fully-clean reference list of any length now scores 100. % Defensible is surfaced as a prominent complementary signal in the Executive Dashboard. Risk-band thresholds and gauge color zones recalibrated (90–100 green / 70–89 amber / 50–69 orange / 0–49 red).Files changed
prompts/v6-auditor.mdtest-sets/predatory-venues.mddocs/heuristics.mdREADME.mdCLAUDE.mdroadmap/v4-features.mdtest-sets/real-articles/README.mdWhat is NOT in this PR
prompts/v5-auditor.md,test-sets/adversarial-30.md, and all committed reports are unchanged.Validation follow-up (suggested)
v6 needs a fresh baseline run before being treated as production-validated. The baseline should confirm:
test-sets/predatory-venues.md— all three trap entries flag correctly (two Elevated for H10 alone, one High for H10 + H7), and both clean controls (JOGNN, BMC Pregnancy and Childbirth) remain Defensible.Because the scoring formula changed, v6 baseline metrics will not be directly comparable to the v5 numbers committed in
reports/adversarial-30-v5-2026-06-19.html.Prompt-Origin
Implemented by claude-runner[bot] per task prompt. Origin context: Chris resolved the #7 data-source question in favor of a hybrid indexing-plus-free-blacklist approach and chose to bundle the #43 scoring-formula fix into the same v6 release since both are vNext prompt changes to the same new file.
Closes #7
Closes #43