Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 78 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,70 @@ A prompt-engineered deep-scan verification system for academic reference lists.

**Authorship:** The prompts, documentation, and test sets in this repo are co-written with [Claude](https://claude.ai) (Anthropic). I direct the work and review the output; Claude writes the prompts. I'm an infrastructure operator, not a software engineer β€” please don't read this repo as a portfolio of coding ability.

## πŸ“š Ask this codebase (DeepWiki)

<a href="https://deepwiki.com/lentago/reference-checker"><img src="https://deepwiki.com/badge.svg" alt="Ask DeepWiki" height="32"></a>

[DeepWiki](https://deepwiki.com/lentago/reference-checker) maintains an AI-generated wiki over this
repository β€” architecture pages, diagrams, and a Q&A box grounded in the actual code. Every
public Lentago Labs repo is indexed ([deepwiki.com/lentago](https://deepwiki.com/lentago));
it is the fastest way to orient before reading source. It is AI-generated: trust it to orient
you, verify against the code before you act on it.

**Good first questions:**
- What does Heuristic 10 (journal legitimacy) check and which sources corroborate a predatory-venue flag?
- What is the required process for shipping a new prompt version, e.g. v7-auditor.md, per repo convention?
- What's the difference in purpose between test-sets/adversarial-30.md and test-sets/real-articles/, and how do baseline PRs use both?

## 🧭 What this repo demonstrates

There's no application code here β€” the "product" is a single versioned prompt. That constraint
turns out to be a clean way to see prompt-as-code and everyday operations discipline without
any of it being obscured by application logic.

| Pattern | How it shows up here |
|---|---|
| Prompt-as-code versioning | [`prompts/v3-auditor.md`](prompts/v3-auditor.md) … [`v6-auditor.md`](prompts/v6-auditor.md) β€” [CLAUDE.md](CLAUDE.md) requires every revision to ship as a new `prompts/v<next>-auditor.md` file, never an in-place edit |
| Eval / regression-gate discipline before promoting a change | [`test-sets/adversarial-30.md`](test-sets/adversarial-30.md) (detection) + [`test-sets/real-articles/`](test-sets/real-articles/) (false-positive check), scored and committed to `reports/` in PR [#46](https://github.com/lentago/reference-checker/pull/46) before v6 shipped |
| Thin reusable-workflow wrappers from a central repo | [`.github/workflows/docs-check.yml`](.github/workflows/docs-check.yml), `claude-code-review.yml`, `claude.yml` all `uses: lentago/shared-workflows/.github/workflows/*.yml@main` β€” CI logic lives once, this repo just declares which checks it consumes |
| Required status check enforced via branch ruleset | `main` requires the `docs-check/docs-check` check to pass, squash-merge only β€” the wrapper itself shipped in PR [#52](https://github.com/lentago/reference-checker/pull/52) |
| Issue-linked, traceable roadmap | Every shipped heuristic in the [Roadmap](#roadmap) links its originating issue, e.g. Heuristic 9 β†’ [#6](https://github.com/lentago/reference-checker/issues/6), Heuristic 10 β†’ [#7](https://github.com/lentago/reference-checker/issues/7) |
| Generated brand header, not hand-maintained | The banner comment at the top of this file β€” generated by `lentago/.github` β†’ `brand/generate.py`, shipped in PR [#51](https://github.com/lentago/reference-checker/pull/51) |
| Codified-but-disabled automation, not silently deleted | [`claude-code-review.yml`](.github/workflows/claude-code-review.yml)'s trigger was flipped to `workflow_dispatch`-only with a dated comment explaining why and how to re-enable, in PR [#47](https://github.com/lentago/reference-checker/pull/47) |
| Explicit AI-authorship disclosure | The Authorship note above, and [Credits](#credits) at the bottom |

## πŸ› οΈ Make a change yourself

This is a lab β€” the systems are real, the stakes are not. Pick a vector. Opening a PR doesn't
require org membership (fork it and propose the change); merging on `main` is reserved for
Lentago Labs members.

**Ship a new prompt version with an eval baseline.** Add `prompts/v<N>-auditor.md` as a new
file β€” never edit the live version in place β€” open a PR, and let `docs-check` run as the
required status check. Once that merges, open a companion PR that runs the new prompt against
`test-sets/adversarial-30.md` (detection) and `test-sets/real-articles/` (false-positive check),
committing the resulting HTML reports plus a metrics verdict to `reports/`.

**Proof this works:**
- PR [#44](https://github.com/lentago/reference-checker/pull/44) β€” feat(prompt): v6 auditor β€” journal legitimacy (H10) and scoring formula fix
- PR [#46](https://github.com/lentago/reference-checker/pull/46) β€” feat(baseline): v6 production baseline β€” H10 functional check, regression gate, scoring calibration

**Adopt a fleet-standard required CI check.** Drop a thin workflow file into
`.github/workflows/` that does `uses: lentago/shared-workflows/.github/workflows/<name>.yml@main`.
Open a PR, get it merged, and the org branch ruleset on `main` treats that workflow's check-run
context as a required status check gating future merges.

**Proof this works:**
- PR [#52](https://github.com/lentago/reference-checker/pull/52) β€” Adopt the shared docs-check workflow

**Toggle an automated-agent capability off without deleting it.** Flip a reusable workflow's
trigger (e.g. `on: pull_request` β†’ `on: workflow_dispatch`) to pause fleet-wide automation for
this repo while preserving the repo-specific config for a future re-enable β€” document the change
inline with a dated comment so the next person knows why it's off and how to turn it back on.

**Proof this works:**
- PR [#47](https://github.com/lentago/reference-checker/pull/47) β€” Disable automated Claude PR review (manual-only trigger)

## The Problem

Academic reference lists are a trust surface. Paper mills, AI-generated citations, and increasingly sophisticated metadata manipulation mean that a reference can *look* perfectly formatted while being completely fabricated β€” or worse, a composite of real elements assembled to resist casual verification.
Expand Down Expand Up @@ -95,34 +159,21 @@ The auditor produces a self-contained HTML report with six sections, designed fo
### Running an Audit

1. Provide the prompt (see `prompts/v6-auditor.md`) to Claude with web search enabled.
2. Paste or upload the reference list to be audited.
2. Paste or upload the reference list β€” raw text, extracted from a manuscript PDF/Word document, or mixed formats; the auditor normalizes during processing.
3. The auditor will systematically verify each reference and produce the HTML report.

> **Note:** A single audit of 25–40 references typically requires 5–15 minutes of processing time and significant tool-call volume. This is by design β€” thorough forensic verification is not a quick-check operation.

### Input Formats

The auditor accepts reference lists in:

- Raw text (pasted APA-formatted references)
- Extracted from manuscript PDFs or Word documents
- Mixed formats (the auditor will normalize during processing)

## Testing

The system has been validated against:
The system has been validated against two purpose-built corpora β€” see
[`test-sets/adversarial-30.md`](test-sets/adversarial-30.md) and
[`test-sets/real-articles/`](test-sets/real-articles/) for the raw sets, and the eval /
regression-gate row above for how they gate a prompt promotion.

### Adversarial Test Set
A deliberately constructed 30-reference list containing layered traps:
- Homoglyph substitutions (Cyrillic characters in journal titles)
- Author-shifted citations (real papers with manipulated author lists)
- Shadow papers (fully fabricated but plausible-sounding)
- Double-Real composites (real DOI + metadata from a different paper)
- Pop-culture junk citations (including a fabricated Obi-Wan Kenobi publication)
- Clean references seeded throughout to test false-positive rates
**Adversarial test set** β€” 30 references with layered traps: homoglyph substitutions (Cyrillic characters in journal titles), author-shifted citations, shadow papers, Double-Real composites, pop-culture junk citations (including a fabricated Obi-Wan Kenobi publication), and clean references seeded throughout to test false-positive rates.

### Real Published Articles
Multiple real articles from JOGNN, MCN, and related nursing journals verified to confirm that the auditor correctly classifies legitimate references as Defensible without over-flagging.
**Real published articles** β€” Multiple real articles from JOGNN, MCN, and related nursing journals, verified to confirm the auditor classifies legitimate references as Defensible without over-flagging.

## Roadmap

Expand Down Expand Up @@ -199,3 +250,10 @@ MIT License β€” see [LICENSE](LICENSE).
## Credits

See the Authorship note at the top β€” the prompts in this repo are co-written with [Claude](https://claude.ai) (Anthropic). Chris Pitzi directs the work, brings the editorial and ops context, and reviews the output; Claude writes the prompt text.

---

🌱 **Lentago Labs** is a team learning lab β€” real systems, non-critical stakes, modern
operations patterns demonstrated in the open. Start at the
[org profile](https://github.com/lentago), and read this repo on
[DeepWiki](https://deepwiki.com/lentago/reference-checker).