diff --git a/README.md b/README.md index 80effd7..bb03aba 100644 --- a/README.md +++ b/README.md @@ -221,6 +221,8 @@ This is the same pattern as writing configuration management (Puppet, Ansible) f Nothing gets added to the spec until it's been tested. The prompt is the artifact. +**Architecture decisions:** [`docs/adr/`](docs/adr/) records the reconstructed rationale behind this repo's key structural choices — prompt-as-product, paired evaluation gates, and the risk-classification scheme. + ## Repository Structure ``` diff --git a/docs/adr/0001-prompt-engineered-product-versioned-prompts.md b/docs/adr/0001-prompt-engineered-product-versioned-prompts.md new file mode 100644 index 0000000..d9d8837 --- /dev/null +++ b/docs/adr/0001-prompt-engineered-product-versioned-prompts.md @@ -0,0 +1,77 @@ +# ADR-0001: The product is a versioned prompt program, not a scripted pipeline + +**Status:** Accepted (2026-03-23; reconstructed 2026-08-13) + +## Context + +The auditor's core logic lives entirely in a single Opus prompt that uses live +web search against Crossref, PubMed, Retraction Watch, publisher sites, and +the NLM Catalog to verify citations. There is no compile step, test runner, +or package manager — as [CLAUDE.md](../../CLAUDE.md) puts it, "the 'build +system' is 'paste the prompt into Claude with a reference list.'" + +At project scaffold (PR [#1](https://github.com/lentago/reference-checker/pull/1), 2026-03-23) the live prompt was +moved into `prompts/v3-auditor.md`, establishing the convention that ships to +this day: each revision lands as the next-integer file +(`prompts/v-auditor.md`), never an in-place edit, so old versions stay +readable for diffing and for reproducing past audit runs. Four versions have +shipped under this convention since: v3 (scaffold), v4 (2026-05-21), v5 +(PR [#39](https://github.com/lentago/reference-checker/pull/39), 2026-06-19), and v6 (PR [#44](https://github.com/lentago/reference-checker/pull/44), +2026-06-19). + +The same scaffold commit added [`docs/architecture.md`](../architecture.md), +a design document for decomposing the monolithic Opus prompt into a +four-stage Haiku/Sonnet/Opus/Haiku pipeline for editorial-scale throughput, +and issue [#11](https://github.com/lentago/reference-checker/issues/11) to track implementing it. Both the design +doc and the issue are still open as of this writing — the pipeline is +deliberately designed but not built. `docs/architecture.md` estimates the +decomposed pipeline only becomes cost-effective above roughly 3 manuscripts/ +month, and the live-web-search verification surface the whole heuristic set +depends on (see the "Live web search is the verification surface" gotcha in +CLAUDE.md) is easiest to reason about as a single adversarial-reasoning +session rather than state handed between model tiers. + +## Decision + +Keep the auditor as a single versioned Opus prompt. Ship every revision as a +new `prompts/v-auditor.md` file rather than mutating the current one. +Treat the multi-model pipeline decomposition as a scoped-but-deferred design +(tracked in `docs/architecture.md` and issue #11), not a roadmap commitment. + +## Alternatives + +**Recorded at the time:** +- **Decompose into a Haiku (parse) → Sonnet (verify) → Opus (interpret) → + Haiku (report) pipeline**, per `docs/architecture.md`. Estimated ~0.3–0.4x + the token cost of the monolithic approach, but requires an orchestration + layer, inter-stage JSON schema contracts, and escalation-criteria tuning + that didn't exist yet. Explicitly deferred pending real throughput demand + ("editorial scale," estimated break-even above ~3 manuscripts/month) that + hasn't materialized against a single-Opus-prompt lab project. + +**Retrospective — not considered at the time:** +- **A scripted tool calling the Crossref/PubMed/Retraction Watch APIs + directly**, with no LLM in the verification loop at all. *Worse fit for + this project's actual purpose*: it would likely be more deterministic on + the mechanical checks (DOI resolution, retraction lookups), but it drops + the exact thing the repo exists to demonstrate — prompt-engineering-as- + product — and it can't do the adversarial pattern-interpretation the + forensic heuristics (homoglyph substitution, author-shifting, shadow-paper + signatures) actually depend on. Live web search *is* the verification + surface by design, not an implementation detail to be optimized away. + +## Consequences + +- Every prompt revision is fully readable and diffable against every prior + version; reproducing a past audit run means pointing at the matching + `prompts/v-auditor.md` file, not reconstructing pipeline state. +- There is no automated regression suite in the software-engineering sense — + quality control runs through the paired evaluation sets described in + [ADR-0002](0002-paired-evaluation-sets-as-regression-gates.md), executed by + hand (or by an agent) against each new version. +- The pipeline decomposition stays a live option, not a dead idea: if + editorial-scale throughput ever becomes a real requirement, `docs/ + architecture.md` and issue #11 are the starting point rather than a + from-scratch design effort. +- Cost and latency scale linearly with reference-list size on every run, + since there's no cheap-tier pre-filtering of obviously-clean references. diff --git a/docs/adr/0002-paired-evaluation-sets-as-regression-gates.md b/docs/adr/0002-paired-evaluation-sets-as-regression-gates.md new file mode 100644 index 0000000..0493155 --- /dev/null +++ b/docs/adr/0002-paired-evaluation-sets-as-regression-gates.md @@ -0,0 +1,88 @@ +# ADR-0002: Paired evaluation sets — adversarial detection + real-article specificity — as the regression gate + +**Status:** Accepted (2026-06-19; reconstructed 2026-08-13) + +## Context + +`test-sets/adversarial-30.md` (30 deliberately-crafted bad citations, +present since project scaffold) exercises detection: the auditor should flag +every entry. On its own, that set can't catch a prompt revision that gets +more aggressive and starts over-flagging *clean* references — a change that +looks like an improvement (more flags!) but is actually just noise. + +Issue [#3](https://github.com/lentago/reference-checker/issues/3) (opened 2026-03-23, same day as scaffold) tracked +closing that gap; PR [#42](https://github.com/lentago/reference-checker/pull/42) (merged 2026-06-19) delivered it by +adding `test-sets/real-articles/` — full reference lists pulled from two +real, published articles (Ahmadinezhad 2024 and Patriksson 2024), with every +citation individually verified against Crossref before being committed. This +set exercises specificity: it should produce near-zero flags on legitimate +scholarship. + +Running the v5 prompt against the new paired sets (PR [#41](https://github.com/lentago/reference-checker/pull/41), +2026-06-19, and the PR #42 real-article run) surfaced a real problem the +detection-only set could never have shown: issue +[#43](https://github.com/lentago/reference-checker/issues/43) found that the scoring formula's `D × 3` base cost +(a per-Defensible-reference deduction) made a *clean* 30-reference article +score only 10, and the clean 26-reference Patriksson article score 22 — both +reading as alarming for articles with no integrity problems. The v6 revision +(PR [#44](https://github.com/lentago/reference-checker/pull/44)) removed the `D × 3` term; the corrected formula +scored the same clean Patriksson article at 96 +(`reports/patriksson-2024-v6-2026-06-19.html`), confirmed by the v6 baseline +run (PR [#46](https://github.com/lentago/reference-checker/pull/46), 2026-06-20). + +CLAUDE.md states the operating principle this issue established directly: "a +prompt revision that improves one [set] without regressing the other is a +real win; a revision that flags more on both is just noisier." + +## Decision + +Every prompt revision is evaluated against both test sets together before +being treated as a production baseline: `adversarial-30.md` for detection +(sensitivity) and `test-sets/real-articles/` for false-positive rate +(specificity). Baselines — HTML reports plus a metrics/verdict writeup — are +committed to `reports/` per version (v4: PR #36/#38; v5: PR #41; v6: PR #46), +so every shipped version has a reproducible, comparable record. A revision +that raises flags on the real-article corpus without a matching detection +gain on the adversarial set is treated as a regression, not an improvement, +regardless of how it scores on the adversarial set alone. + +## Alternatives + +**Recorded at the time:** +- **Detection-only evaluation** (the status quo before PR #42): keep + `adversarial-30.md` as the sole gate. Cheaper to run, but issue #43's + finding — a structural scoring bug that punished clean articles — was + invisible to it by construction; a detection-only set has no clean + reference lists to over-penalize. + +**Retrospective — not considered at the time:** +- **A single blended metric** (e.g., one combined score averaging detection + and specificity results) instead of two separately-reported test-set runs. + *Lateral, arguably worse*: averaging would have partially masked exactly + the failure issue #43 caught — a formula that tanks clean-article scores + could still average out to a passable blended number if adversarial + detection stayed strong. Keeping the two signals separate and both + visible in `reports/README.md` is what let the Patriksson 22→96 correction + register as an unambiguous, specific fix rather than a rounding change. +- **Statistical sampling from a larger corpus of real articles** rather than + two hand-verified ones. *Better for statistical power, worse for this + project's constraints*: each real-article entry requires individually + verifying every citation against Crossref to avoid the auditor's own + anti-fabrication protocol backfiring on its test data (documented in the + #42 PR body); two carefully-verified articles are tractable for a + single-operator lab project in a way a larger sampled corpus isn't yet. + +## Consequences + +- Shipping a new prompt version now implies two evaluation runs, not one — + more work per revision, but it already caught one real, non-obvious defect + (the `D × 3` scoring bug) that pure detection testing would have missed + entirely. +- `reports/` accumulates paired baselines per version (adversarial + both + real articles), which is the reproducibility record referenced in + [ADR-0001](0001-prompt-engineered-product-versioned-prompts.md) — there's + no other regression history for a prompt-only product. +- The real-article corpus is expensive to grow (each addition needs + individual Crossref verification), so it stays small; specificity + measurement has less statistical power than detection measurement on the + 30-entry adversarial set. diff --git a/docs/adr/0003-four-tier-risk-classification-non-accusatory-language.md b/docs/adr/0003-four-tier-risk-classification-non-accusatory-language.md new file mode 100644 index 0000000..e87bc47 --- /dev/null +++ b/docs/adr/0003-four-tier-risk-classification-non-accusatory-language.md @@ -0,0 +1,97 @@ +# ADR-0003: Four-tier risk classification over a continuous score, with non-accusatory language as a design requirement + +**Status:** Accepted (2026-05-21; reconstructed 2026-08-13) + +## Context + +The auditor's earliest recorded form (`prompts/v3-auditor.md`, scaffolded +2026-03-23) already classified each reference into one of four risk levels — +**Low / Moderate / Elevated / High** — rather than emitting a single +continuous confidence number per reference. `prompts/v4-auditor.md` +(committed 2026-05-21) relabeled the bottom tier **Defensible** and gave the +scheme the letter shorthand (D/M/E/H) that the auditor still uses, tying it +to the aggregate scoring formula documented in `docs/heuristics.md` +(`Score = 100 − H×12 − E×5 − M×2`, floored at 0, with tier weights unchanged +since v4 relative to each other). The per-reference tier is the unit +editorial judgment acts on; the aggregate score is a summary of tiers, not +the other way around. + +Heuristic 10 (journal legitimacy / predatory-venue flagging, shipped in v6 +via PR [#44](https://github.com/lentago/reference-checker/pull/44), 2026-06-19) made a second, related requirement +explicit: classification language must be factual and non-accusatory. +`roadmap/v4-features.md` records the resulting rule directly — *"'Predatory' +is not used as a determination; 'potentially predatory' or 'unverified +venue' at most"* — and the same constraint is echoed in the README's H10 +description ("never 'predatory' as a verdict"). This generalizes a +constraint that was implicit from the start: the auditor's job is to +surface verifiable anomalies for an editor to judge, not to hand down a +finding of misconduct itself. + +## Decision + +Classify every reference into one of four discrete risk tiers — Defensible, +Moderate, Elevated, High — instead of a single continuous per-reference +confidence score, and aggregate those tiers into the headline 0–100 score +via fixed per-tier weights. Require classification language to stay +factual and descriptive rather than accusatory at every tier, most +concretely enforced on Heuristic 10: never "predatory" as a verdict, at +most "potentially predatory" or "unverified venue." + +## Alternatives + +**Recorded at the time:** +- No alternative framing to the four-tier scheme itself is recorded in the + repo history — the tiered classification is present from the earliest + captured prompt (v3) onward, so no "before" state exists to contrast + against in-repo. +- For H10 specifically, no separately weighed alternative is recorded; the + contrast is implicit in the shipped design's own recorded constraints + (issue #7, `roadmap/v4-features.md`). The hybrid + whitelist-plus-community-list design treats absence from the primary + indexes (DOAJ, PubMed/MEDLINE, Scopus, Web of Science) plus corroboration + from secondary, non-authoritative lists (Beall's archived list, Stop + Predatory Journals) as sufficient only for the softer + "Elevated"/"unverified venue" framing, never a standalone accusation — + the never-sole-basis constraint is the recorded fact; the rejected pole + (index absence as an outright verdict) is drawn here retrospectively. + +**Retrospective — not considered at the time:** +- **A continuous 0–100 (or 0–1) per-reference confidence score** instead of + four discrete tiers, with the aggregate simply averaging or summing raw + confidences. *Worse fit*: a continuous per-reference score invites false + precision the underlying evidence doesn't support — "this citation is + 73% likely fabricated" implies a calibration the heuristics (pattern + matches against Crossref/PubMed/publisher metadata) can't actually back + up, and it's easier to game or dispute at the margins than a discrete + tier with a stated rationale. It would also have made the exact defect + issue [#43](https://github.com/lentago/reference-checker/issues/43) found — the aggregate formula structurally + punishing clean articles — harder to spot, since a continuous score + smooths over the same distortion a coarse tier table exposes plainly. +- **A binary flagged/not-flagged classification** (no gradation at all). + *Worse fit*: collapses the real editorial difference between "minor, + inconclusive concern" (Moderate) and "strong evidence of fabrication" + (High) into a single bucket, which would force every flag through the + same escalation path regardless of severity — directly at odds with the + graduated COPE-alignment mapping (`docs/heuristics.md`) that routes + High-tier findings to author query / editorial investigation and treats + Elevated-tier venue findings as no dedicated flowchart, editorial + judgment only. + +## Consequences + +- Every reference gets an auditable, discrete tier with a stated rationale + instead of an opaque number, which is what makes the per-tier weight + table (and therefore the headline score) legible and debuggable — it's + what let issue #43 pin the scoring-formula defect to a specific term + (`D × 3`) rather than a vague "scores feel off." +- The non-accusatory language requirement means the auditor's output is + necessarily conservative in its wording even on strong findings — an + editor reading a High-tier H10 flag still has to exercise judgment and + verify independently rather than treat the report as a determination. + This is a deliberate trade of assertiveness for defensibility, consistent + with the report being described throughout the repo as COPE-aligned + guidance, not an accusation. +- Coarse tiers mean two references with meaningfully different underlying + evidence strength can land in the same tier and receive the same weight + in the aggregate score — the tier system trades some resolution for + interpretability and gaming-resistance. diff --git a/docs/adr/README.md b/docs/adr/README.md new file mode 100644 index 0000000..2d22d8a --- /dev/null +++ b/docs/adr/README.md @@ -0,0 +1,21 @@ +# Architecture Decision Records + +This directory was reconstructed 2026-08-13 from repo history and fleet +records — commit history, issues/PRs, `CLAUDE.md`, and other project +artifacts — as part of a fleet-wide ADR reconstruction effort. No decision +here was written contemporaneously as an ADR; each entry's **Status** date is +the original decision date recovered from the evidence, not the date this +file was written. Every anchor (issue/PR number, file, date) cited in these +records was checked against this repo before being asserted; anything that +didn't check out was dropped or hedged rather than copied blind. Each entry's +**Alternatives** section separates what was actually weighed at the time from +options identified only in retrospect — the latter are labeled explicitly and +are not presented as historical. + +## Index + +| ADR | Title | Original decision date | +|---|---|---| +| [0001](0001-prompt-engineered-product-versioned-prompts.md) | The product is a versioned prompt program, not a scripted pipeline | 2026-03-23 | +| [0002](0002-paired-evaluation-sets-as-regression-gates.md) | Paired evaluation sets — adversarial detection + real-article specificity — as the regression gate | 2026-06-19 | +| [0003](0003-four-tier-risk-classification-non-accusatory-language.md) | Four-tier risk classification over a continuous score, with non-accusatory language as a design requirement | 2026-05-21 |