From 5d47eaa06383596a4a90b8d81f425b1aa8355c55 Mon Sep 17 00:00:00 2001 From: "claude-runner[bot]" Date: Wed, 12 Aug 2026 16:23:07 +0000 Subject: [PATCH] README: reposition as learning-lab exhibit Add DeepWiki orientation section, a patterns table linking each practice (prompt-as-code versioning, eval/regression gates, thin CI wrappers, branch ruleset, issue-linked roadmap, disabled-not-deleted automation) to concrete evidence in this repo, and an operator-vector section showing exactly how to make each kind of change. Lightly condense Input Formats/Testing prose to fold in duplicated detail. Header, banner, badges, and authorship disclosure are untouched. Co-Authored-By: Claude --- README.md | 98 +++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 78 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index 7de05a3..80effd7 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,70 @@ A prompt-engineered deep-scan verification system for academic reference lists. **Authorship:** The prompts, documentation, and test sets in this repo are co-written with [Claude](https://claude.ai) (Anthropic). I direct the work and review the output; Claude writes the prompts. I'm an infrastructure operator, not a software engineer — please don't read this repo as a portfolio of coding ability. +## 📚 Ask this codebase (DeepWiki) + +Ask DeepWiki + +[DeepWiki](https://deepwiki.com/lentago/reference-checker) maintains an AI-generated wiki over this +repository — architecture pages, diagrams, and a Q&A box grounded in the actual code. Every +public Lentago Labs repo is indexed ([deepwiki.com/lentago](https://deepwiki.com/lentago)); +it is the fastest way to orient before reading source. It is AI-generated: trust it to orient +you, verify against the code before you act on it. + +**Good first questions:** +- What does Heuristic 10 (journal legitimacy) check and which sources corroborate a predatory-venue flag? +- What is the required process for shipping a new prompt version, e.g. v7-auditor.md, per repo convention? +- What's the difference in purpose between test-sets/adversarial-30.md and test-sets/real-articles/, and how do baseline PRs use both? + +## 🧭 What this repo demonstrates + +There's no application code here — the "product" is a single versioned prompt. That constraint +turns out to be a clean way to see prompt-as-code and everyday operations discipline without +any of it being obscured by application logic. + +| Pattern | How it shows up here | +|---|---| +| Prompt-as-code versioning | [`prompts/v3-auditor.md`](prompts/v3-auditor.md) … [`v6-auditor.md`](prompts/v6-auditor.md) — [CLAUDE.md](CLAUDE.md) requires every revision to ship as a new `prompts/v-auditor.md` file, never an in-place edit | +| Eval / regression-gate discipline before promoting a change | [`test-sets/adversarial-30.md`](test-sets/adversarial-30.md) (detection) + [`test-sets/real-articles/`](test-sets/real-articles/) (false-positive check), scored and committed to `reports/` in PR [#46](https://github.com/lentago/reference-checker/pull/46) before v6 shipped | +| Thin reusable-workflow wrappers from a central repo | [`.github/workflows/docs-check.yml`](.github/workflows/docs-check.yml), `claude-code-review.yml`, `claude.yml` all `uses: lentago/shared-workflows/.github/workflows/*.yml@main` — CI logic lives once, this repo just declares which checks it consumes | +| Required status check enforced via branch ruleset | `main` requires the `docs-check/docs-check` check to pass, squash-merge only — the wrapper itself shipped in PR [#52](https://github.com/lentago/reference-checker/pull/52) | +| Issue-linked, traceable roadmap | Every shipped heuristic in the [Roadmap](#roadmap) links its originating issue, e.g. Heuristic 9 → [#6](https://github.com/lentago/reference-checker/issues/6), Heuristic 10 → [#7](https://github.com/lentago/reference-checker/issues/7) | +| Generated brand header, not hand-maintained | The banner comment at the top of this file — generated by `lentago/.github` → `brand/generate.py`, shipped in PR [#51](https://github.com/lentago/reference-checker/pull/51) | +| Codified-but-disabled automation, not silently deleted | [`claude-code-review.yml`](.github/workflows/claude-code-review.yml)'s trigger was flipped to `workflow_dispatch`-only with a dated comment explaining why and how to re-enable, in PR [#47](https://github.com/lentago/reference-checker/pull/47) | +| Explicit AI-authorship disclosure | The Authorship note above, and [Credits](#credits) at the bottom | + +## 🛠️ Make a change yourself + +This is a lab — the systems are real, the stakes are not. Pick a vector. Opening a PR doesn't +require org membership (fork it and propose the change); merging on `main` is reserved for +Lentago Labs members. + +**Ship a new prompt version with an eval baseline.** Add `prompts/v-auditor.md` as a new +file — never edit the live version in place — open a PR, and let `docs-check` run as the +required status check. Once that merges, open a companion PR that runs the new prompt against +`test-sets/adversarial-30.md` (detection) and `test-sets/real-articles/` (false-positive check), +committing the resulting HTML reports plus a metrics verdict to `reports/`. + +**Proof this works:** +- PR [#44](https://github.com/lentago/reference-checker/pull/44) — feat(prompt): v6 auditor — journal legitimacy (H10) and scoring formula fix +- PR [#46](https://github.com/lentago/reference-checker/pull/46) — feat(baseline): v6 production baseline — H10 functional check, regression gate, scoring calibration + +**Adopt a fleet-standard required CI check.** Drop a thin workflow file into +`.github/workflows/` that does `uses: lentago/shared-workflows/.github/workflows/.yml@main`. +Open a PR, get it merged, and the org branch ruleset on `main` treats that workflow's check-run +context as a required status check gating future merges. + +**Proof this works:** +- PR [#52](https://github.com/lentago/reference-checker/pull/52) — Adopt the shared docs-check workflow + +**Toggle an automated-agent capability off without deleting it.** Flip a reusable workflow's +trigger (e.g. `on: pull_request` → `on: workflow_dispatch`) to pause fleet-wide automation for +this repo while preserving the repo-specific config for a future re-enable — document the change +inline with a dated comment so the next person knows why it's off and how to turn it back on. + +**Proof this works:** +- PR [#47](https://github.com/lentago/reference-checker/pull/47) — Disable automated Claude PR review (manual-only trigger) + ## The Problem Academic reference lists are a trust surface. Paper mills, AI-generated citations, and increasingly sophisticated metadata manipulation mean that a reference can *look* perfectly formatted while being completely fabricated — or worse, a composite of real elements assembled to resist casual verification. @@ -95,34 +159,21 @@ The auditor produces a self-contained HTML report with six sections, designed fo ### Running an Audit 1. Provide the prompt (see `prompts/v6-auditor.md`) to Claude with web search enabled. -2. Paste or upload the reference list to be audited. +2. Paste or upload the reference list — raw text, extracted from a manuscript PDF/Word document, or mixed formats; the auditor normalizes during processing. 3. The auditor will systematically verify each reference and produce the HTML report. > **Note:** A single audit of 25–40 references typically requires 5–15 minutes of processing time and significant tool-call volume. This is by design — thorough forensic verification is not a quick-check operation. -### Input Formats - -The auditor accepts reference lists in: - -- Raw text (pasted APA-formatted references) -- Extracted from manuscript PDFs or Word documents -- Mixed formats (the auditor will normalize during processing) - ## Testing -The system has been validated against: +The system has been validated against two purpose-built corpora — see +[`test-sets/adversarial-30.md`](test-sets/adversarial-30.md) and +[`test-sets/real-articles/`](test-sets/real-articles/) for the raw sets, and the eval / +regression-gate row above for how they gate a prompt promotion. -### Adversarial Test Set -A deliberately constructed 30-reference list containing layered traps: -- Homoglyph substitutions (Cyrillic characters in journal titles) -- Author-shifted citations (real papers with manipulated author lists) -- Shadow papers (fully fabricated but plausible-sounding) -- Double-Real composites (real DOI + metadata from a different paper) -- Pop-culture junk citations (including a fabricated Obi-Wan Kenobi publication) -- Clean references seeded throughout to test false-positive rates +**Adversarial test set** — 30 references with layered traps: homoglyph substitutions (Cyrillic characters in journal titles), author-shifted citations, shadow papers, Double-Real composites, pop-culture junk citations (including a fabricated Obi-Wan Kenobi publication), and clean references seeded throughout to test false-positive rates. -### Real Published Articles -Multiple real articles from JOGNN, MCN, and related nursing journals verified to confirm that the auditor correctly classifies legitimate references as Defensible without over-flagging. +**Real published articles** — Multiple real articles from JOGNN, MCN, and related nursing journals, verified to confirm the auditor classifies legitimate references as Defensible without over-flagging. ## Roadmap @@ -199,3 +250,10 @@ MIT License — see [LICENSE](LICENSE). ## Credits See the Authorship note at the top — the prompts in this repo are co-written with [Claude](https://claude.ai) (Anthropic). Chris Pitzi directs the work, brings the editorial and ops context, and reviews the output; Claude writes the prompt text. + +--- + +🌱 **Lentago Labs** is a team learning lab — real systems, non-critical stakes, modern +operations patterns demonstrated in the open. Start at the +[org profile](https://github.com/lentago), and read this repo on +[DeepWiki](https://deepwiki.com/lentago/reference-checker).