paper-forge

The front-end of a reproducibility-review loop. It turns the manual Build-with-Paper funnel — scan the weekly firehose → decide what's worth building → write the repro spec — into an agentic step you can run in one command.

The framing matters. Google's Paper Assistant Tool (PAT, arXiv 2606.28277) reviews papers statically: read the manuscript, check the math, flag flaws. paper-forge aims one notch past that — toward empirical / reproducibility review: paper → reproduce → report what actually survives. The triage → brief → scaffold layers decide what to build; the empirical-review lane (Tầng C) reviews the reproduction once it runs.

papers ─▶ Tầng A: triage (rubric) ─▶ Tầng B: build brief ─▶ scaffold ─▶ Tầng C: empirical review

Offline-first and stdlib-only: the whole pipeline runs $0 on a Mac using a transparent keyword heuristic. Add 9router creds and --live to have GPT-5.5 / Claude sharpen the scores and write real briefs. Nothing here touches a GPU.

The rubric (Tầng A)

Every paper is scored 0–5 on the five things that actually decide a Build-with-Paper project, then normalized to 0–100:

Criterion	Weight	Question
mac_zero_cost	3	Can the core effect be reproduced $0 on Apple silicon (MLX/Metal, small models)?
unreleased_code	2	Is a reproduction novel (no official code released)?
vietnamese_frontier	2	Is there a credible Vietnamese-first extension?
substrate_reuse	2	Can it reuse one of your existing repos as substrate?
money_plot	3	Is there a single measurable plot the demo can hit?

Papers you've already shipped (see data/known_builds.json) short-circuit to ALREADY_BUILT, and every new paper is matched against your existing repos to suggest what to build on top of.

The build brief (Tầng B)

For the top BUILD picks it drafts the skeleton you normally write by hand: the falsifiable claim, the smallest experiment that confirms/refutes it, the substrate + lanes, the one money-plot, whether a GPU is needed (weight-updates only), and — the signature move — an honest-finding hypothesis stated up front (where the claim will probably not hold).

The empirical review (Tầng C)

The part that goes one notch past PAT. A scaffolded repro emits a self-describing data/evidence.json after running its lanes; paperforge review adversarially checks each central claim against the numbers it actually produced and drafts the honest finding — HOLDS / BREAKS / INCONCLUSIVE + the regime it holds in. Offline it uses a transparent rule (a near-flat change is not a confirmation); --live routes each claim through a 9router skeptic.

The signature move a static review cannot make is the risk-regime check: if the reproduction never exercised the failure regime that the paper's own honest-finding hypothesis warns about, a HOLDS is downgraded to "holds only in the tested regime" instead of being accepted at face value.

Usage

# Offline heuristic ($0, no creds) — triage this week's DAIR.AI list:
python -m paperforge run --input data/week_2026-07-05.json

# Write the markdown report to a file:
python -m paperforge run --input data/week_2026-07-05.json --out reports/week.md

# Pull fresh abstracts straight from arXiv, then triage:
python -m paperforge fetch --arxiv 2606.28277 2606.26797

# Live mode — route triage + briefs through 9router (needs .env):
python -m paperforge run --input data/week_2026-07-05.json --live

# Scaffold a repo skeleton (with an evidence contract) for a BUILD pick:
python -m paperforge scaffold --arxiv 2606.28277 --input data/week_2026-07-05.json --out ../pat-vn

# Empirical-review a reproduction once its lanes emit data/evidence.json:
python -m paperforge review --evidence data/evidence.example.json   # or: --repo ../pat-vn [--write]

# The weekly loop (M4): triage → maker/checker gate → human gate. Prints the plan;
# --yes is what actually writes scaffolds to disk:
python -m paperforge loop --input data/week_2026-07-05.json --scaffold-top 2
python -m paperforge loop --input data/week_2026-07-05.json --scaffold-top 2 --yes

# Gate CI on reproduced claims (nonzero exit if any claim BREAKS):
python -m paperforge review --repo ../verifier-horizon --ci

For --live, copy .env.example to .env and set NINEROUTER_BASE_URL / NINEROUTER_API_KEY / PAPERFORGE_MODEL.

Example (offline, this week's list)

_10 papers · 5 to build · 5 already shipped_

🟢 BUILD  87  Red Queen Godel Machine          substrate: verifier-horizon
🟢 BUILD  82  Paper Assistant Tool (PAT)       substrate: automem-vn
🟢 BUILD  68  Reasoning Quality Emerges Early  substrate: system-iii-router
🟢 BUILD  65  ASPIRE                           substrate: scalable-robot-demo-curation
🟡 ...        MCP Server Patterns
✅ ALREADY_BUILT  The Verification Horizon     → verifier-horizon
✅ ALREADY_BUILT  Generative Skill Composition → super-agent
✅ ALREADY_BUILT  AutoMem                      → automem-vn
✅ ALREADY_BUILT  RLMF                         → faithful-confidence
✅ ALREADY_BUILT  HORIZON                      → rtl-gauntlet

Where this is going

• M1 (done) — Tầng A triage + Tầng B build brief, offline + 9router.
• M2 (done) — scaffold generation: emit the substrate/lanes/tests skeleton for a BUILD pick, now including the evidence.py contract that feeds M3.
• M3 (done) — the empirical-review lane (paperforge review): check each claim against executed evidence, flag untested risk regimes, auto-draft the honest finding — the part that beats a static PAT-style read.
• M4 (core done) — paperforge loop is the loop-engineering body: an LLM budget (over budget → $0 heuristic fallback), a maker/checker gate (only picks past a stricter promote threshold are eligible to scaffold), a human gate (nothing writes to disk without --yes), and an arxiv-id denylist. review --ci exits nonzero on a BREAK so it can gate CI. Remaining: an actual scheduler (cron/launchd) and a second-opinion LLM checker.

Honest notes / limitations

• The offline scorer is a keyword heuristic — deliberately transparent, and a little generous at the BUILD threshold. It will occasionally suggest a spurious substrate (e.g. matching the verb "document" to a doc-extraction repo). The --live pass is what trims these; the human still owns the final angle.
• Seed abstracts in data/week_2026-07-05.json are short paraphrases of the weekly digest. Use paperforge fetch to pull full arXiv abstracts before a real triage.
• The Vietnamese-frontier score is inherently a judgment call; the heuristic only gives a base from transfer-friendly topics.
• The empirical-review lane is only as honest as the evidence.json it is fed: paper-forge checks the numbers you report, it does not run your lanes for you (yet). Garbage evidence in, garbage verdict out — but the risk-regime check still flags claims whose stated failure mode was never actually tested.

Layout

paperforge/
  criteria.py     # the 5-criteria rubric (single source of truth)
  triage.py       # Tầng A — heuristic + LLM scoring
  brief.py        # Tầng B — build-brief (heuristic template + LLM)
  scaffold.py     # M2 — emit a repo skeleton (+ evidence contract) from a brief
  review.py       # Tầng C / M3 — adversarial empirical review of executed evidence
  runner.py       # M4 — loop budget + maker/checker gate + human gate
  knownbuilds.py  # your build history → dedup + substrate suggestions
  arxiv.py        # stdlib arXiv metadata fetch
  llm.py          # stdlib OpenAI-compatible 9router client (offline-safe)
  pipeline.py     # list → triage → rank → briefs
  report.py       # markdown renderer
  cli.py          # run | fetch | scaffold | loop | review | doctor
data/
  known_builds.json        # arxiv_id → repo, topic, keywords
  week_2026-07-05.json     # seed input (this week's DAIR.AI list)
  evidence.example.json    # illustrative evidence for the review lane
tests/                     # 33 offline tests, $0/Mac

Tests

python -c "import pytest,sys; sys.exit(pytest.main(['-q']))"
# 39 passed