Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,8 @@ This is the same pattern as writing configuration management (Puppet, Ansible) f

Nothing gets added to the spec until it's been tested. The prompt is the artifact.

**Architecture decisions:** [`docs/adr/`](docs/adr/) records the reconstructed rationale behind this repo's key structural choices — prompt-as-product, paired evaluation gates, and the risk-classification scheme.

## Repository Structure

```
Expand Down
77 changes: 77 additions & 0 deletions docs/adr/0001-prompt-engineered-product-versioned-prompts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# ADR-0001: The product is a versioned prompt program, not a scripted pipeline

**Status:** Accepted (2026-03-23; reconstructed 2026-08-13)

## Context

The auditor's core logic lives entirely in a single Opus prompt that uses live
web search against Crossref, PubMed, Retraction Watch, publisher sites, and
the NLM Catalog to verify citations. There is no compile step, test runner,
or package manager — as [CLAUDE.md](../../CLAUDE.md) puts it, "the 'build
system' is 'paste the prompt into Claude with a reference list.'"

At project scaffold (PR [#1](https://github.com/lentago/reference-checker/pull/1), 2026-03-23) the live prompt was
moved into `prompts/v3-auditor.md`, establishing the convention that ships to
this day: each revision lands as the next-integer file
(`prompts/v<N>-auditor.md`), never an in-place edit, so old versions stay
readable for diffing and for reproducing past audit runs. Four versions have
shipped under this convention since: v3 (scaffold), v4 (2026-05-21), v5
(PR [#39](https://github.com/lentago/reference-checker/pull/39), 2026-06-19), and v6 (PR [#44](https://github.com/lentago/reference-checker/pull/44),
2026-06-19).

The same scaffold commit added [`docs/architecture.md`](../architecture.md),
a design document for decomposing the monolithic Opus prompt into a
four-stage Haiku/Sonnet/Opus/Haiku pipeline for editorial-scale throughput,
and issue [#11](https://github.com/lentago/reference-checker/issues/11) to track implementing it. Both the design
doc and the issue are still open as of this writing — the pipeline is
deliberately designed but not built. `docs/architecture.md` estimates the
decomposed pipeline only becomes cost-effective above roughly 3 manuscripts/
month, and the live-web-search verification surface the whole heuristic set
depends on (see the "Live web search is the verification surface" gotcha in
CLAUDE.md) is easiest to reason about as a single adversarial-reasoning
session rather than state handed between model tiers.

## Decision

Keep the auditor as a single versioned Opus prompt. Ship every revision as a
new `prompts/v<next>-auditor.md` file rather than mutating the current one.
Treat the multi-model pipeline decomposition as a scoped-but-deferred design
(tracked in `docs/architecture.md` and issue #11), not a roadmap commitment.

## Alternatives

**Recorded at the time:**
- **Decompose into a Haiku (parse) → Sonnet (verify) → Opus (interpret) →
Haiku (report) pipeline**, per `docs/architecture.md`. Estimated ~0.3–0.4x
the token cost of the monolithic approach, but requires an orchestration
layer, inter-stage JSON schema contracts, and escalation-criteria tuning
that didn't exist yet. Explicitly deferred pending real throughput demand
("editorial scale," estimated break-even above ~3 manuscripts/month) that
hasn't materialized against a single-Opus-prompt lab project.

**Retrospective — not considered at the time:**
- **A scripted tool calling the Crossref/PubMed/Retraction Watch APIs
directly**, with no LLM in the verification loop at all. *Worse fit for
this project's actual purpose*: it would likely be more deterministic on
the mechanical checks (DOI resolution, retraction lookups), but it drops
the exact thing the repo exists to demonstrate — prompt-engineering-as-
product — and it can't do the adversarial pattern-interpretation the
forensic heuristics (homoglyph substitution, author-shifting, shadow-paper
signatures) actually depend on. Live web search *is* the verification
surface by design, not an implementation detail to be optimized away.

## Consequences

- Every prompt revision is fully readable and diffable against every prior
version; reproducing a past audit run means pointing at the matching
`prompts/v<N>-auditor.md` file, not reconstructing pipeline state.
- There is no automated regression suite in the software-engineering sense —
quality control runs through the paired evaluation sets described in
[ADR-0002](0002-paired-evaluation-sets-as-regression-gates.md), executed by
hand (or by an agent) against each new version.
- The pipeline decomposition stays a live option, not a dead idea: if
editorial-scale throughput ever becomes a real requirement, `docs/
architecture.md` and issue #11 are the starting point rather than a
from-scratch design effort.
- Cost and latency scale linearly with reference-list size on every run,
since there's no cheap-tier pre-filtering of obviously-clean references.
88 changes: 88 additions & 0 deletions docs/adr/0002-paired-evaluation-sets-as-regression-gates.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# ADR-0002: Paired evaluation sets — adversarial detection + real-article specificity — as the regression gate

**Status:** Accepted (2026-06-19; reconstructed 2026-08-13)

## Context

`test-sets/adversarial-30.md` (30 deliberately-crafted bad citations,
present since project scaffold) exercises detection: the auditor should flag
every entry. On its own, that set can't catch a prompt revision that gets
more aggressive and starts over-flagging *clean* references — a change that
looks like an improvement (more flags!) but is actually just noise.

Issue [#3](https://github.com/lentago/reference-checker/issues/3) (opened 2026-03-23, same day as scaffold) tracked
closing that gap; PR [#42](https://github.com/lentago/reference-checker/pull/42) (merged 2026-06-19) delivered it by
adding `test-sets/real-articles/` — full reference lists pulled from two
real, published articles (Ahmadinezhad 2024 and Patriksson 2024), with every
citation individually verified against Crossref before being committed. This
set exercises specificity: it should produce near-zero flags on legitimate
scholarship.

Running the v5 prompt against the new paired sets (PR [#41](https://github.com/lentago/reference-checker/pull/41),
2026-06-19, and the PR #42 real-article run) surfaced a real problem the
detection-only set could never have shown: issue
[#43](https://github.com/lentago/reference-checker/issues/43) found that the scoring formula's `D × 3` base cost
(a per-Defensible-reference deduction) made a *clean* 30-reference article
score only 10, and the clean 26-reference Patriksson article score 22 — both
reading as alarming for articles with no integrity problems. The v6 revision
(PR [#44](https://github.com/lentago/reference-checker/pull/44)) removed the `D × 3` term; the corrected formula
scored the same clean Patriksson article at 96
(`reports/patriksson-2024-v6-2026-06-19.html`), confirmed by the v6 baseline
run (PR [#46](https://github.com/lentago/reference-checker/pull/46), 2026-06-20).

CLAUDE.md states the operating principle this issue established directly: "a
prompt revision that improves one [set] without regressing the other is a
real win; a revision that flags more on both is just noisier."

## Decision

Every prompt revision is evaluated against both test sets together before
being treated as a production baseline: `adversarial-30.md` for detection
(sensitivity) and `test-sets/real-articles/` for false-positive rate
(specificity). Baselines — HTML reports plus a metrics/verdict writeup — are
committed to `reports/` per version (v4: PR #36/#38; v5: PR #41; v6: PR #46),
so every shipped version has a reproducible, comparable record. A revision
that raises flags on the real-article corpus without a matching detection
gain on the adversarial set is treated as a regression, not an improvement,
regardless of how it scores on the adversarial set alone.

## Alternatives

**Recorded at the time:**
- **Detection-only evaluation** (the status quo before PR #42): keep
`adversarial-30.md` as the sole gate. Cheaper to run, but issue #43's
finding — a structural scoring bug that punished clean articles — was
invisible to it by construction; a detection-only set has no clean
reference lists to over-penalize.

**Retrospective — not considered at the time:**
- **A single blended metric** (e.g., one combined score averaging detection
and specificity results) instead of two separately-reported test-set runs.
*Lateral, arguably worse*: averaging would have partially masked exactly
the failure issue #43 caught — a formula that tanks clean-article scores
could still average out to a passable blended number if adversarial
detection stayed strong. Keeping the two signals separate and both
visible in `reports/README.md` is what let the Patriksson 22→96 correction
register as an unambiguous, specific fix rather than a rounding change.
- **Statistical sampling from a larger corpus of real articles** rather than
two hand-verified ones. *Better for statistical power, worse for this
project's constraints*: each real-article entry requires individually
verifying every citation against Crossref to avoid the auditor's own
anti-fabrication protocol backfiring on its test data (documented in the
#42 PR body); two carefully-verified articles are tractable for a
single-operator lab project in a way a larger sampled corpus isn't yet.

## Consequences

- Shipping a new prompt version now implies two evaluation runs, not one —
more work per revision, but it already caught one real, non-obvious defect
(the `D × 3` scoring bug) that pure detection testing would have missed
entirely.
- `reports/` accumulates paired baselines per version (adversarial + both
real articles), which is the reproducibility record referenced in
[ADR-0001](0001-prompt-engineered-product-versioned-prompts.md) — there's
no other regression history for a prompt-only product.
- The real-article corpus is expensive to grow (each addition needs
individual Crossref verification), so it stays small; specificity
measurement has less statistical power than detection measurement on the
30-entry adversarial set.
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# ADR-0003: Four-tier risk classification over a continuous score, with non-accusatory language as a design requirement

**Status:** Accepted (2026-05-21; reconstructed 2026-08-13)

## Context

The auditor's earliest recorded form (`prompts/v3-auditor.md`, scaffolded
2026-03-23) already classified each reference into one of four risk levels —
**Low / Moderate / Elevated / High** — rather than emitting a single
continuous confidence number per reference. `prompts/v4-auditor.md`
(committed 2026-05-21) relabeled the bottom tier **Defensible** and gave the
scheme the letter shorthand (D/M/E/H) that the auditor still uses, tying it
to the aggregate scoring formula documented in `docs/heuristics.md`
(`Score = 100 − H×12 − E×5 − M×2`, floored at 0, with tier weights unchanged
since v4 relative to each other). The per-reference tier is the unit
editorial judgment acts on; the aggregate score is a summary of tiers, not
the other way around.

Heuristic 10 (journal legitimacy / predatory-venue flagging, shipped in v6
via PR [#44](https://github.com/lentago/reference-checker/pull/44), 2026-06-19) made a second, related requirement
explicit: classification language must be factual and non-accusatory.
`roadmap/v4-features.md` records the resulting rule directly — *"'Predatory'
is not used as a determination; 'potentially predatory' or 'unverified
venue' at most"* — and the same constraint is echoed in the README's H10
description ("never 'predatory' as a verdict"). This generalizes a
constraint that was implicit from the start: the auditor's job is to
surface verifiable anomalies for an editor to judge, not to hand down a
finding of misconduct itself.

## Decision

Classify every reference into one of four discrete risk tiers — Defensible,
Moderate, Elevated, High — instead of a single continuous per-reference
confidence score, and aggregate those tiers into the headline 0–100 score
via fixed per-tier weights. Require classification language to stay
factual and descriptive rather than accusatory at every tier, most
concretely enforced on Heuristic 10: never "predatory" as a verdict, at
most "potentially predatory" or "unverified venue."

## Alternatives

**Recorded at the time:**
- No alternative framing to the four-tier scheme itself is recorded in the
repo history — the tiered classification is present from the earliest
captured prompt (v3) onward, so no "before" state exists to contrast
against in-repo.
- For H10 specifically, no separately weighed alternative is recorded; the
contrast is implicit in the shipped design's own recorded constraints
(issue #7, `roadmap/v4-features.md`). The hybrid
whitelist-plus-community-list design treats absence from the primary
indexes (DOAJ, PubMed/MEDLINE, Scopus, Web of Science) plus corroboration
from secondary, non-authoritative lists (Beall's archived list, Stop
Predatory Journals) as sufficient only for the softer
"Elevated"/"unverified venue" framing, never a standalone accusation —
the never-sole-basis constraint is the recorded fact; the rejected pole
(index absence as an outright verdict) is drawn here retrospectively.

**Retrospective — not considered at the time:**
- **A continuous 0–100 (or 0–1) per-reference confidence score** instead of
four discrete tiers, with the aggregate simply averaging or summing raw
confidences. *Worse fit*: a continuous per-reference score invites false
precision the underlying evidence doesn't support — "this citation is
73% likely fabricated" implies a calibration the heuristics (pattern
matches against Crossref/PubMed/publisher metadata) can't actually back
up, and it's easier to game or dispute at the margins than a discrete
tier with a stated rationale. It would also have made the exact defect
issue [#43](https://github.com/lentago/reference-checker/issues/43) found — the aggregate formula structurally
punishing clean articles — harder to spot, since a continuous score
smooths over the same distortion a coarse tier table exposes plainly.
- **A binary flagged/not-flagged classification** (no gradation at all).
*Worse fit*: collapses the real editorial difference between "minor,
inconclusive concern" (Moderate) and "strong evidence of fabrication"
(High) into a single bucket, which would force every flag through the
same escalation path regardless of severity — directly at odds with the
graduated COPE-alignment mapping (`docs/heuristics.md`) that routes
High-tier findings to author query / editorial investigation and treats
Elevated-tier venue findings as no dedicated flowchart, editorial
judgment only.

## Consequences

- Every reference gets an auditable, discrete tier with a stated rationale
instead of an opaque number, which is what makes the per-tier weight
table (and therefore the headline score) legible and debuggable — it's
what let issue #43 pin the scoring-formula defect to a specific term
(`D × 3`) rather than a vague "scores feel off."
- The non-accusatory language requirement means the auditor's output is
necessarily conservative in its wording even on strong findings — an
editor reading a High-tier H10 flag still has to exercise judgment and
verify independently rather than treat the report as a determination.
This is a deliberate trade of assertiveness for defensibility, consistent
with the report being described throughout the repo as COPE-aligned
guidance, not an accusation.
- Coarse tiers mean two references with meaningfully different underlying
evidence strength can land in the same tier and receive the same weight
in the aggregate score — the tier system trades some resolution for
interpretability and gaming-resistance.
21 changes: 21 additions & 0 deletions docs/adr/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Architecture Decision Records

This directory was reconstructed 2026-08-13 from repo history and fleet
records — commit history, issues/PRs, `CLAUDE.md`, and other project
artifacts — as part of a fleet-wide ADR reconstruction effort. No decision
here was written contemporaneously as an ADR; each entry's **Status** date is
the original decision date recovered from the evidence, not the date this
file was written. Every anchor (issue/PR number, file, date) cited in these
records was checked against this repo before being asserted; anything that
didn't check out was dropped or hedged rather than copied blind. Each entry's
**Alternatives** section separates what was actually weighed at the time from
options identified only in retrospect — the latter are labeled explicitly and
are not presented as historical.

## Index

| ADR | Title | Original decision date |
|---|---|---|
| [0001](0001-prompt-engineered-product-versioned-prompts.md) | The product is a versioned prompt program, not a scripted pipeline | 2026-03-23 |
| [0002](0002-paired-evaluation-sets-as-regression-gates.md) | Paired evaluation sets — adversarial detection + real-article specificity — as the regression gate | 2026-06-19 |
| [0003](0003-four-tier-risk-classification-non-accusatory-language.md) | Four-tier risk classification over a continuous score, with non-accusatory language as a design requirement | 2026-05-21 |