From 1a88983098c7aa54f837e498587aaa7d9a3c1f98 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 27 Jul 2026 13:32:50 +0000 Subject: [PATCH 1/2] =?UTF-8?q?v0.2.0:=20full=20overhaul=20=E2=80=94=20fix?= =?UTF-8?q?=20all=20audited=20defects,=20add=20guarded=20LLM=20extraction,?= =?UTF-8?q?=20verification=20ledger,=20tests=20+=20CI?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Driven by an independent live audit (fresh-clone runs on Python 3.10-3.13 + line-by-line review). Highlights: P0 fixes (all live-reproduced before fixing): - CLI was a SyntaxError on Python 3.10/3.11 (f-string backslash, PEP 701) - review-evidence-candidates wiped other papers' human review decisions (shared queue overwrite -> merge) - Excel round-trip broke validators (BOM/cp949/stray columns) -> utf-8-sig reads, BOM writes, atomic rewrites with row projection - collect gathered off-topic papers: arXiv boolean queries + relevance sort, OpenAlex/Crossref mailto + backoff + Retry-After, Crossref type filters, cross-source DOI/arXiv/title dedupe, per-source circuit breaker - migrate-evidence could flip '1'/'yes' to verified=true -> only literal 'true' survives; demotions reported Governance upgrades: - verify-evidence + matrices/verification_ledger.csv: the only sanctioned writer of verified=true (human attestation required); guards now PASS attested rows and FAIL unattested ones (previously the guard failed forever after any legitimate human verification) - approval-gate anti-forgery: apply reports cross-checked against current manuscript SHA-256 - manuscript backups moved outside the guarded tree; citekey scans skip backups; sync-zotero respects --dry-run and propagates citekey renames into the Evidence Matrix; legacy extract-evidence/audit fixed or blocked New capabilities: - extract-evidence-llm: LLM-proposed candidates with mandatory verbatim quote verification against GROBID text; same human gates as heuristics - audit-manuscript-draft --experiment-data: automatic numeric cross-check (exact/rounded/not-found) against experiment output files — makes the README's flagship claim true in code - draft audit: docx footnotes, cp949 fallback, Korean/English detectors, heading-anchored structure checks; figures: line-anchored heading match, honest reports, YAML specs, graceful renderer degradation - make-page-metadata-preview / make-outline-insertion-template producers; promote-evidence --ready-only - release hardening: named secret-scan ruleset, config sanitization, quarantine-on-failure, whitelist gaps closed Hygiene: personal research data moved from code to config YAMLs; sources.yaml actually wired; paperops_extra.py removed (S2 in main collect); 4 unused deps dropped; 179 tests + GitHub Actions matrix (3.10-3.13); README en/ko truth pass; CHANGELOG added. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01ULLxKwEG7wBaj6SDFeBmsA --- .github/workflows/ci.yml | 34 + .gitignore | 3 + CHANGELOG.md | 153 ++++ README.ko.md | 82 +- README.md | 99 ++- config/draft_audit.yaml | 63 ++ config/figures.yaml | 259 ++++++ config/pipeline.yaml | 12 + config/qa_profile.yaml | 49 ++ config/review_overrides.yaml | 95 +++ config/scoring.yaml | 5 +- config/sources.yaml | 7 +- config/topic_profile.yaml | 15 +- docs/RUN_AX_ONTOLOGY_GOVERNANCE.md | 4 +- pyproject.toml | 9 +- requirements.txt | 7 +- scripts/build_public_release.py | 256 +++++- scripts/paperops.py | 1232 +++++++++++++++++++++------- scripts/paperops_draft_audit.py | 838 ++++++++++++++++--- scripts/paperops_extra.py | 170 ---- scripts/paperops_figures.py | 897 +++++++++++--------- scripts/run_pipeline.py | 90 +- tests/__init__.py | 0 tests/test_draft_audit.py | 669 +++++++++++++++ tests/test_figures.py | 534 ++++++++++++ tests/test_paperops_core.py | 258 ++++++ tests/test_release_scan.py | 375 +++++++++ 27 files changed, 5129 insertions(+), 1086 deletions(-) create mode 100644 .github/workflows/ci.yml create mode 100644 CHANGELOG.md create mode 100644 config/draft_audit.yaml create mode 100644 config/figures.yaml create mode 100644 config/qa_profile.yaml create mode 100644 config/review_overrides.yaml delete mode 100644 scripts/paperops_extra.py create mode 100644 tests/__init__.py create mode 100644 tests/test_draft_audit.py create mode 100644 tests/test_figures.py create mode 100644 tests/test_paperops_core.py create mode 100644 tests/test_release_scan.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..d1d273d --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,34 @@ +name: CI + +on: + push: + branches: ["**"] + pull_request: + +jobs: + test: + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + python-version: ["3.10", "3.11", "3.12", "3.13"] + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: ${{ matrix.python-version }} + - name: Compile every script (catches version-specific syntax like PEP 701 f-strings) + run: python -m py_compile scripts/*.py + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install -r requirements.txt pytest + - name: Run tests + run: python -m pytest tests/ -q + - name: Offline smoke (init/status/doctor/guard on a fresh tree) + run: | + python scripts/paperops.py init + python scripts/paperops.py status + python scripts/paperops.py doctor + python scripts/paperops.py guard-no-auto-verified + python scripts/paperops.py smoke-test diff --git a/.gitignore b/.gitignore index b7b230d..cf0ad1f 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,5 @@ .venv/ +.venv*/ __pycache__/ *.pyc .env @@ -7,6 +8,8 @@ logs/ matrices/ notes/ reports/ +backups/ +research_design/ 05_manuscript/ manuscript/ dist/ diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..023d08e --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,153 @@ +# Changelog + +## v0.2.0 — 2026-07-27 + +Full-stack overhaul driven by an independent live audit (fresh-clone testing on +Python 3.10–3.13 + line-by-line review). Every finding below was reproduced +before fixing and covered by a regression test where practical. 179+ tests and +a 4-version CI matrix are new in this release. + +### Fixed — critical + +- **The entire CLI was a `SyntaxError` on Python 3.10/3.11** (a backslash + inside an f-string expression, legal only from Python 3.12/PEP 701). + `pyproject.toml` declared `>=3.10` while no 3.10/3.11 user could run any + command. CI now compiles every script on 3.10–3.13. +- **`review-evidence-candidates` destroyed other papers' human review + decisions**: it overwrote the shared review queue with only the current + paper's rows. The queue is now merged — regenerating paper B's queue + preserves paper A's pending decisions (regression-tested, including an + Excel-style resave in between). +- **Excel round-trips broke every CSV validator**: BOM from "CSV UTF-8" made + header validation fail; cp949 re-saves crashed with `UnicodeDecodeError`; + a stray empty column crashed guarded rewrites mid-write. All CSV reads are + now BOM-tolerant (`utf-8-sig`) with stray-column capture, all CSV writes + include a BOM (Korean text renders correctly in Excel), and all guarded + matrix rewrites are atomic (temp file + `os.replace`) with row projection. +- **`collect` gathered off-topic papers**: the arXiv query was sent as a bare + keyword soup (matching loosely, sorted by newest — returning unrelated + brand-new papers), OpenAlex was called without `mailto` (constant HTTP 429), + and Crossref returned "Decision letter …" peer-review artifacts. Queries are + now explicit boolean expressions (auto-converted when needed, + `sortBy=relevance`), `mailto`/API keys/User-Agent are actually read from + `config/sources.yaml`, all HTTP calls retry with backoff honoring + `Retry-After`, Crossref filters non-paper types, and a per-source circuit + breaker stops a failing source from burning minutes of backoff per run. +- **`migrate-evidence` could set `verified=true` from `1`/`yes`/`y`** (e.g. + Excel booleans) — the one automation path that violated the core invariant. + Only the exact literal `true` is preserved now; truthy variants are demoted + to `false` and listed in the migration report. + +### Fixed — high + +- `sync-zotero --dry-run --apply` silently applied; now rejected like every + other command. Applied citekey renames now propagate to the Evidence Matrix + (with backup) instead of stranding old citekeys everywhere. +- Legacy `extract-evidence` appended schema-drifted 14-column rows into the + migrated 19+-column guarded matrix; it is now blocked with instructions for + the guarded pipeline. +- Legacy `audit` counted `user@gmail.com` as citekey `gmail` and matched + citekeys by substring against raw CSV text; it now uses the email-safe + pattern over both manuscript roots against the real citekey column. +- Cross-source duplicates: the same paper arriving from arXiv (arXiv id only) + and OpenAlex/Crossref (DOI only) produced two rows. Upsert now matches by + normalized DOI → arXiv id → normalized title, and merges missing + identifiers/abstract into the existing row. +- BibTeX arXiv extraction bug that produced the literal id `arxiv` and + treated any eprint as an arXiv id. +- Manuscript backups were written inside the guarded `05_manuscript/` tree, + tripping `guard-no-auto-verified` after every legitimate apply and + double-counting citekeys from backups. Backups now live in + `backups/manuscript/`, and citekey scans skip any `backups` directory. +- The manuscript-change approval gate accepted any file containing four magic + substrings. It now also cross-checks the report's recorded post-apply + SHA-256 values against the manuscript files on disk, rejecting stale or + fabricated reports. +- PDF downloads saved HTML paywall pages as `.pdf` (later poisoning + GROBID/PyMuPDF); content-type/magic-byte checks added. +- `backup` never included `papers.sqlite`; `data/metadata` (and + `05_manuscript`, `tests`) are now in the archive. +- `run_pipeline.py` emoji output crashed with `UnicodeEncodeError` whenever + stdout was redirected on cp949 Windows, and the whole "one-click" run + failed if Graphviz was missing. Output is now ASCII, figure rendering + degrades gracefully (`--strict` restores hard failure), and `--skip-figures` + was added. +- Silent-zero traps: `extract-evidence-candidates` now errors when the GROBID + artifact directory is missing instead of "successfully" writing nothing; + `collect` warns loudly when 0 papers are collected. +- `doctor` mislabeled the mailto check, missed `quarto.cmd` on Windows + (`shutil.which` now used), and subprocess output is decoded as UTF-8 + everywhere instead of the console codepage. +- GROBID TEI responses are decoded as UTF-8 explicitly (no more charset + guessing); `screen` thresholds come from `config/scoring.yaml` instead of a + hardcoded 0.35; section headings containing `;` no longer corrupt + `source_location`; `scoring.yaml` no longer pins `current_year` (recency + scoring stays correct after 2026). + +### Added + +- **`verify-evidence` + verification ledger** — the missing half of the + governance model. A human sets `verified=true` with `--by NAME --attest` + (or reverts with `--revoke`); every action is recorded in + `matrices/verification_ledger.csv`. `guard-no-auto-verified` and the + promoted-row QA now PASS attested rows and FAIL any verified row without a + matching ledger attestation — previously the guard failed forever the + moment a human legitimately verified anything. +- **`extract-evidence-llm`** — guarded LLM-assisted evidence extraction + (OpenAI-compatible endpoints). The model only proposes candidates; every + `exact_quote` is verbatim-verified against the GROBID-parsed section text + (paraphrases and inventions are dropped and reported), and survivors enter + the same human review → promotion → guarded apply pipeline with + `verified=false`. This makes `config/prompts/evidence_extractor.md` live + configuration instead of dead weight. +- **Automatic numeric cross-check in `audit-manuscript-draft`** + (`--experiment-data DIR`): every number in the draft is checked against + experiment output files with exact/rounded/not-found statuses — the feature + the README previously described but the code did not contain. +- **Draft-audit upgrades**: docx footnotes/endnotes parsed (footnote-cited + sentences count as sourced), tab/break handling, cp949/euc-kr fallback with + loud warnings instead of silent `errors='ignore'` destruction, Korean + citation patterns (`(김철수, 2020)`, `[1]`), English strong-claim/overclaim + detectors, heading-anchored chapter checks, and + `config/draft_audit.yaml` for topic-specific lists. +- **Figure safety**: heading targeting is line-anchored and code-fence-aware + (no more inserting under `### X Details` when `## X` was approved), + ambiguous/missing headings block the row, applies are idempotent, apply + reports carry real counts + the preview's SHA-256, and figure specs moved + to `config/figures.yaml`. +- **`make-page-metadata-preview` / `make-outline-insertion-template`** — + producers for the two guarded-apply input files that previously had to be + hand-authored with no documentation. +- **`promote-evidence --ready-only`** — apply ready rows while blocked rows + are reported, instead of one blocked candidate freezing the whole paper. +- **Release hardening**: the secret scan grew from 4 patterns to a named + ruleset (any email with allowlist, GitHub fine-grained/classic tokens, + private-key blocks, AWS/Google/Slack/JWT shapes, config-assignment + secrets); exported configs are sanitized (`*api_key`/`*mailto` values + blanked and reported); scan failures quarantine the dist instead of leaving + it publishable; `tests/`, `.github/`, `CHANGELOG.md`, and the AX runbook + joined the whitelist. +- **Tests + CI**: 179+ pytest cases across core, draft audit, figures, and + release scanning; GitHub Actions matrix on Python 3.10/3.11/3.12/3.13 with + compile, test, and offline-smoke stages. + +### Changed + +- Personal research data moved out of code into config: + per-candidate review decisions → `config/review_overrides.yaml`, + corpus/domain keyword lists → `config/qa_profile.yaml`, + figure specs → `config/figures.yaml`, + draft-audit source lists → `config/draft_audit.yaml`. + Shipped values preserve the original author's behavior; other users edit + YAML instead of Python. +- `config/sources.yaml` is now real configuration: per-source enable flags, + limits, sleep intervals, mailto, and API keys are read by `collect`. +- Semantic Scholar collection moved into the main `collect` command (with + API-key header support); the unregistered, crash-prone `paperops_extra.py` + was removed. +- `requirements.txt`/`pyproject.toml` dropped `pandas`, `rapidfuzz`, + `bibtexparser`, `pypdf` — none were imported anywhere, and bibtexparser's + sdist-only build broke installs on some systems. Version bumped to 0.2.0. +- `topic_profile.yaml` queries rewritten as explicit boolean expressions; + README (en/ko) updated to match actual behavior — including correcting the + claim that numeric cross-checking was automatic before it was. diff --git a/README.ko.md b/README.ko.md index 1bf4088..1a7b483 100644 --- a/README.ko.md +++ b/README.ko.md @@ -17,23 +17,27 @@ ![PaperOps 전체 파이프라인](assets/figures/fig_pipeline.svg) -PaperOps는 **연구-집필 전 과정** — 문헌 수집, 스크리닝, PDF 파싱, 근거 추출, -서지 동기화, 가드된 원고 수정, 재현 가능한 figure 생성, 초안 감사 — 를 -**45개 이상의 명령**을 가진 로컬 우선 CLI 하나로 자동화합니다. +PaperOps는 **연구-집필 전 과정** — 문헌 수집, 스크리닝, PDF 파싱, 근거 추출 +(휴리스틱 + 인용문 원문 대조가 강제되는 LLM 보조 추출), 서지 동기화, 가드된 +원고 수정, 재현 가능한 figure 생성, 실험 산출물과의 수치 자동 대조를 포함한 +초안 감사 — 를 **50개 이상의 명령**을 가진 로컬 우선 CLI 하나로 자동화합니다. 논문을 *대신 써주는* 도구가 아닙니다. 파이프라인은 자동이지만, 판단이 필요한 세 지점 — 근거 채택, 원고 수정 승인, `verified=true` 판정 — 은 의도적으로 -사람에게 남겨져 있으며, 가드(guard)가 자동화의 위조를 차단합니다. +사람에게 남겨져 있습니다. `verified=true`는 오직 `verify-evidence` 명령의 +명시적 인간 확약(attestation)으로만 진입 가능하며, 그 기록은 검증 원장 +(verification ledger)에 남고 `guard-no-auto-verified`가 대조합니다 — 손으로 +고치든 자동화가 고치든, 원장에 없는 verified 행은 가드가 실패시킵니다. ## 전체 라이프사이클 (단계별) | 단계 | 내용 | 주요 명령 | 자동화 | |---|---|---|---| -| 1. 수집 | arXiv / Semantic Scholar / OpenAlex에서 토픽 프로필 기반 수집 | `collect`, `digest` | 자동 | +| 1. 수집 | arXiv / Semantic Scholar / OpenAlex / Crossref에서 불리언 토픽 쿼리로 수집 — polite pool 헤더, 429 백오프, 소스 간 중복 병합 포함 | `collect`, `digest` | 자동 | | 2. 선별 | 관련도 점수, 연구축별 스크리닝, 연구공백 탐지 | `score`, `screen`, `gap`, `brief` | 자동 | | 3. 확보 | PDF 다운로드, 논문 카드·아웃라인 생성 | `download-pdfs`, `cards`, `outline` | 자동 | | 4. 파싱 | GROBID로 PDF → 구조화된 섹션/참고문헌 | `parse-grobid`, `validate-grobid-artifacts` | 자동 | -| 5. 추출 | 파싱 텍스트에서 claim/quote/page 근거 후보 추출 | `extract-evidence-candidates`, `validate-evidence-candidates` | 자동 | +| 5. 추출 | 파싱 텍스트에서 claim/quote/page 근거 후보 추출 — 휴리스틱, 또는 모든 인용문을 원문과 축자 대조하는 LLM 제안 | `extract-evidence-candidates`, `extract-evidence-llm`, `validate-evidence-candidates` | 자동 | | 6. 검토 | 후보별 채택/수정/기각 결정 | `review-evidence-candidates`, `promotion-plan` | **사람 관문** | | 7. 승격 | 승인된 근거를 Evidence Matrix로 이동 (`verified=false`) | `promote-evidence`, `audit-promoted-evidence` | 가드됨 | | 8. 페이지 확인 | 각 인용문의 정확한 PDF 페이지 탐지·기록 | `locate-pdf-pages`, `apply-page-metadata` | 가드됨 | @@ -41,8 +45,8 @@ PaperOps는 **연구-집필 전 과정** — 문헌 수집, 스크리닝, PDF | 10. 집필 | 원고 패치를 preview + diff로 생성 | `manuscript-patch-preview` | 자동 | | 11. 반영 | 승인된 패치를 백업 + SHA 검증 + LF 저장으로 반영 | `apply-manuscript-patch` | **사람 관문** | | 12. Figure | 스펙 기반 Graphviz/Mermaid 도해, 데이터 조작 금지 | `propose-figures`, `render-figures`, `apply-figure-placeholder` | 가드됨 | -| 13. 초안 감사 | 모든 초안(docx/md/qmd) 검사: 구조, 근거 없는 주장, 과장, 수치를 실제 실험 산출물과 대조 | `audit-manuscript-draft` | 자동 | -| 14. 검증 | 어떤 자동화도 `verified=true`를 설정하지 못했음을 강제 | `guard-no-auto-verified`, `guard-paperops-overclaim`, `smoke-test` | 자동 가드 / **사람 판정** | +| 13. 초안 감사 | 모든 초안(docx/md/qmd) 검사: 구조, 근거 없는 주장, 과장, 그리고 `--experiment-data`로 실험 산출 파일과 수치 자동 대조 | `audit-manuscript-draft` | 자동 | +| 14. 검증 | 사람이 확약과 함께 `verified=true` 설정(검증 원장 기록); 원장에 없는 verified 행은 가드가 실패시킴 | `verify-evidence`, `guard-no-auto-verified`, `guard-paperops-overclaim`, `smoke-test` | **사람 관문** / 자동 가드 | ## 일반 LLM 채팅 대신 이걸 쓰는 이유 @@ -51,8 +55,8 @@ PaperOps는 **연구-집필 전 과정** — 문헌 수집, 스크리닝, PDF | 이 문장의 출처는? | 불명 | Evidence Matrix의 `paper_id` + `citekey` + 인용문 + 페이지 | | 인용 정확성 | 최선의 노력 | 정본 BibTeX와 `check-citekeys` 대조 | | 원고 수정 | 바로 덮어쓰기 | preview → diff → 승인 → SHA 검증 apply → 백업 → 사후 감사 | -| "검증됨" 상태 | 암묵적 | 사람만 설정 가능, 가드가 강제 | -| 초안 속 수치 | 미확인 | 실제 실험 산출 파일과 자동 대조 | +| "검증됨" 상태 | 암묵적 | `verify-evidence --attest`로만 설정 가능; 검증 원장 + 가드가 그 외 전부를 적발 | +| 초안 속 수치 | 미확인 | 실제 실험 산출 파일과 자동 대조 (`audit-manuscript-draft --experiment-data`) | | 재현성 | 세션에 묶임 | SQLite + CSV 매트릭스 + 감사 보고서 + 활동 로그 + figure 소스 | 40개 이상의 오픈소스 연구 도구(PaperQA2, STORM, GPT Researcher, AI-Scientist, @@ -85,21 +89,25 @@ ASReview, gpt_academic, Zotero 생태계, MCP 서버 — `docs/03_TOOL_SYNTHESIS ```bash git clone https://github.com/SakJaeLim/paperops.git && cd paperops -python -m venv .venv +python -m venv .venv # Python 3.10+ # Windows: .venv\Scripts\activate | Unix: source .venv/bin/activate pip install -r requirements.txt python scripts/paperops.py init python scripts/paperops.py status ``` -외부 서비스 없이 바로 동작: 수집, 점수, 스크리닝, 초안 감사, 가드, figure 소스 -생성. 선택적 추가 설치: +그다음 `config/sources.yaml`의 `openalex_mailto`에 본인 이메일을 넣으세요 — +OpenAlex/Crossref의 polite pool에 들어가며, 없으면 두 API 모두 강하게 +스로틀링됩니다(HTTP 429). 외부 서비스 없이 바로 동작: 수집, 점수, 스크리닝, +초안 감사, 가드, figure 소스 생성. 선택적 추가 설치: | 의존성 | 가능해지는 것 | 설치 | |---|---|---| | GROBID | PDF → 구조화 텍스트 파싱 | `docker run -d -p 8070:8070 lfoppiano/grobid:0.8.0` | -| Zotero + Better BibTeX | 정본 서지 동기화 | zotero.org + Better BibTeX 플러그인 | -| Graphviz | SVG/PNG figure 렌더링 | graphviz.org/download | +| Zotero + Better BibTeX | 정본 서지 동기화 (export된 `.bib` 파일) | zotero.org + Better BibTeX 플러그인 | +| Graphviz / Mermaid CLI | SVG/PNG figure 렌더링 (없어도 소스는 항상 저장됨) | graphviz.org/download | +| Semantic Scholar API 키 | 안정적인 S2 수집 (무인증은 강한 429 제한) | `config/sources.yaml`의 `semantic_scholar_api_key` | +| `OPENAI_API_KEY` | `extract-evidence-llm` (가드된 LLM 추출, 축자 대조 강제; `OPENAI_BASE_URL`로 호환 엔드포인트 지원) | 환경변수 | ## 전형적인 세션 @@ -122,8 +130,12 @@ python scripts/paperops.py manuscript-patch-preview python scripts/paperops.py apply-manuscript-patch --from-preview --dry-run python scripts/paperops.py apply-manuscript-patch --from-preview --apply -# 내 초안 감사 (docx/md/qmd) — 구조, 근거 없는 주장, 과장 -python scripts/paperops.py audit-manuscript-draft --input my_thesis_draft.docx +# 내 초안 감사 (docx/md/qmd) — 구조, 근거 없는 주장, 과장, +# 그리고 실험 산출 파일과의 수치 자동 대조 +python scripts/paperops.py audit-manuscript-draft --input my_thesis_draft.docx --experiment-data data/experiment_outputs + +# 인간 검증 (핵심 관문) — 검증 원장에 기록됨 +python scripts/paperops.py verify-evidence --evidence-id ev_xxxx --by "이름" --note "원문 p.3 확인" --attest # Figure와 최종 점검 python scripts/paperops.py render-figures @@ -137,19 +149,29 @@ python scripts/paperops.py smoke-test `audit-manuscript-draft`를 실제 KCI 투고 원고(413개 문단)에 적용한 결과: 378개 문장 스캔, 장 구조 점검, 근거 없는 강한 주장과 과장 표현 플래깅, -그리고 초안의 **수치 178건 전부**를 실제 실험 산출 파일과 대조 — 불일치 0건, +그리고 초안의 수치 178건 전부를 실제 실험 산출 파일과 대조 — 불일치 0건, 반올림 차이 2건 해명, 기저율 1건은 원시 예측 로그에서 재계산해 확인. +(당시 이 대조는 수작업이었고, v0.2.0부터는 `--experiment-data` 플래그가 +수치별로 정확 일치 / 반올림 일치 / 데이터에 없음 상태를 자동 판정합니다.) 감사는 초안을 절대 수정하지 않으며 verified 상태도 만들지 않습니다. 저자를 위한 발견 보고서(MD + CSV)만 생성합니다. ## 거버넌스 규칙 -1. Evidence Matrix는 함부로 수정하지 않는다. -2. `verified=true`는 절대 자동으로 설정되지 않는다 — verified 상태로 가는 - 자동 전이는 존재하지 않는다. -3. 인용문/페이지 매칭은 *출처 정렬*이지 진실 검증이 아니다. -4. 원고 수정은 백업과 사후 가드 + smoke-test가 따르는 가드된 preview/apply로만. -5. 관련연구의 발견은 설계 패턴으로만 인용하며, PaperOps 자체의 성능 증거로 +1. Evidence Matrix는 함부로 수정하지 않는다. 모든 재작성은 원자적(temp 파일 + + rename)이며 타임스탬프 백업이 따른다. +2. `verified=true`는 절대 자동으로 설정되지 않는다. 유일하게 허용된 경로는 + `verify-evidence --attest`이며, 누가/언제/어떻게가 + `matrices/verification_ledger.csv`에 기록된다. 원장에 확약이 없는 verified + 행은 `guard-no-auto-verified`가 실패시킨다. +3. LLM이 추출한 후보는 파싱된 원문과 축자(verbatim) 대조를 통과해야 하며 — + 패러프레이즈·창작 인용문은 폐기·보고된다 — 이후 휴리스틱 후보와 동일한 + 사람 검토 관문을 거친다. +4. 인용문/페이지 매칭은 *출처 정렬*이지 진실 검증이 아니다. +5. 원고 수정은 백업(가드 트리 밖에 저장) + SHA-256 검증 + 사후 가드 + + smoke-test가 따르는 가드된 preview/apply로만. 승인 보고서는 현재 원고의 + SHA-256과 대조되어 오래되었거나 위조된 보고서는 거부된다. +6. 관련연구의 발견은 설계 패턴으로만 인용하며, PaperOps 자체의 성능 증거로 포장하지 않는다. ## 이 저장소에 포함되지 않은 것 @@ -162,10 +184,18 @@ python scripts/paperops.py smoke-test ## 정직한 한계 -- 근거 추출은 키워드/휴리스틱 기반이며, LLM 보조 추출은 별도 가드 단계로 계획. -- 초안 감사는 사람 검토를 위한 휴리스틱 플래깅이지 진실 검증이 아님. +- LLM 추출기가 보장하는 것은 인용문의 축자성뿐이다. 어떤 claim/quote를 + 고르는지의 품질은 모델에 달려 있고, 모든 후보는 여전히 사람 검토를 거친다. +- 초안 감사는 사람 검토를 위한 휴리스틱 플래깅이지 진실 검증이 아님. 수치 + 대조는 "그 숫자가 데이터 파일에 존재하는가"를 확인할 뿐, 분석의 옳음을 + 검증하지 않는다. - 인용문/페이지 정렬은 주장의 진실성을 검증하지 않음 — 설계상 의도. - 정량 결과 figure는 실제 데이터 파일 없이는 절대 생성하지 않음. +- 무인증 Semantic Scholar 수집은 S2 측 제한이 강해 사실상 무료 API 키가 필요. +- 검증 원장은 실수와 부주의한 자동 검증을 막는 장치이지 암호학적 장치가 + 아니다. 로컬 파일을 의도적으로 위조하는 사람까지 막지는 못한다. +- 비영어 README는 릴리스에 따라 영어판보다 늦을 수 있다. 영어 README와 + CHANGELOG가 기준이다. ## 라이선스 diff --git a/README.md b/README.md index 9525cab..3771f92 100644 --- a/README.md +++ b/README.md @@ -18,24 +18,29 @@ ![PaperOps end-to-end pipeline](assets/figures/fig_pipeline.svg) PaperOps automates the **entire research-writing lifecycle** — literature -collection, screening, PDF parsing, evidence extraction, bibliography sync, -guarded manuscript editing, reproducible figure generation, and draft -auditing — through a single local-first CLI with **45+ commands**. +collection, screening, PDF parsing, evidence extraction (heuristic and +LLM-assisted with verbatim quote checking), bibliography sync, guarded +manuscript editing, reproducible figure generation, and draft auditing with +numeric cross-checks against experiment outputs — through a single +local-first CLI with **50+ commands**. It is *not* an auto-paper-writer. The pipeline is automated, but three judgment points are deliberately reserved for humans: evidence adoption, -manuscript-change approval, and the `verified=true` decision. Guards make it -impossible for any automated step to fake those. +manuscript-change approval, and the `verified=true` decision. The +`verified=true` state can only be entered through the `verify-evidence` +command with an explicit human attestation, recorded in a verification +ledger that `guard-no-auto-verified` cross-checks — hand edits and automated +edits both fail the guard. ## The full lifecycle, stage by stage | Stage | What happens | Key commands | Automation | |---|---|---|---| -| 1. Collect | Fetch papers from arXiv / Semantic Scholar / OpenAlex with topic profiles | `collect`, `digest` | Automatic | +| 1. Collect | Fetch papers from arXiv / Semantic Scholar / OpenAlex / Crossref with boolean topic queries, polite-pool headers, 429 backoff, and cross-source dedup | `collect`, `digest` | Automatic | | 2. Triage | Score relevance, screen by research axes, find research gaps | `score`, `screen`, `gap`, `brief` | Automatic | | 3. Acquire | Download PDFs, build paper cards and outlines | `download-pdfs`, `cards`, `outline` | Automatic | | 4. Parse | PDF → structured sections/references via GROBID | `parse-grobid`, `validate-grobid-artifacts` | Automatic | -| 5. Extract | Pull claim/quote/page evidence candidates from parsed text | `extract-evidence-candidates`, `validate-evidence-candidates` | Automatic | +| 5. Extract | Pull claim/quote/page evidence candidates from parsed text — heuristic, or LLM-proposed with every quote verbatim-verified against the source | `extract-evidence-candidates`, `extract-evidence-llm`, `validate-evidence-candidates` | Automatic | | 6. Review | Decide keep / revise / reject for each candidate | `review-evidence-candidates`, `promotion-plan` | **Human gate** | | 7. Promote | Move approved evidence into the Evidence Matrix (`verified=false`) | `promote-evidence`, `audit-promoted-evidence` | Guarded | | 8. Locate | Find and attach exact PDF pages for each quote | `locate-pdf-pages`, `apply-page-metadata` | Guarded | @@ -43,8 +48,8 @@ impossible for any automated step to fake those. | 10. Write | Generate manuscript patches as preview + diff | `manuscript-patch-preview` | Automatic | | 11. Apply | Apply approved patches with backup + SHA check + LF write | `apply-manuscript-patch` | **Human gate** | | 12. Figures | Spec-driven Graphviz/Mermaid figures, never fabricated data | `propose-figures`, `render-figures`, `apply-figure-placeholder` | Guarded | -| 13. Audit drafts | Check any draft (docx/md/qmd): structure, unsourced claims, overclaims, numbers vs. real experiment outputs | `audit-manuscript-draft` | Automatic | -| 14. Verify | Enforce that no automation ever set `verified=true` | `guard-no-auto-verified`, `guard-paperops-overclaim`, `smoke-test` | Automatic guard / **human verdict** | +| 13. Audit drafts | Check any draft (docx/md/qmd): structure, unsourced claims, overclaims, and automatic numeric cross-check against experiment output files (`--experiment-data`) | `audit-manuscript-draft` | Automatic | +| 14. Verify | Human sets `verified=true` with an attestation recorded in the verification ledger; guards fail any verified row without a matching ledger entry | `verify-evidence`, `guard-no-auto-verified`, `guard-paperops-overclaim`, `smoke-test` | **Human gate** / automatic guard | ## Why this instead of a chat LLM? @@ -53,8 +58,8 @@ impossible for any automated step to fake those. | Where did this sentence come from? | unknown | `paper_id` + `citekey` + quote + page in the Evidence Matrix | | Citation correctness | best-effort | `check-citekeys` against canonical BibTeX | | Manuscript edits | direct overwrite | preview → diff → approval → SHA-checked apply → backup → post-audit | -| "Verified" status | implied | only a human can set it; guards enforce this | -| Numbers in your draft | unchecked | cross-checked against actual experiment output files | +| "Verified" status | implied | only `verify-evidence --attest` can set it; the verification ledger + guards catch anything else | +| Numbers in your draft | unchecked | cross-checked against actual experiment output files (`audit-manuscript-draft --experiment-data`) | | Reproducibility | session-bound | SQLite + CSV matrices + audit reports + activity log + figure sources | Design patterns were synthesized from a survey of 40+ open-source research @@ -89,21 +94,26 @@ Evidence Matrix → patch previews → (human) → manuscript**, with guards ```bash git clone https://github.com/SakJaeLim/paperops.git && cd paperops -python -m venv .venv +python -m venv .venv # Python 3.10+ # Windows: .venv\Scripts\activate | Unix: source .venv/bin/activate pip install -r requirements.txt python scripts/paperops.py init python scripts/paperops.py status ``` -Works immediately with no external services: collection, scoring, screening, -draft audit, guards, figures-as-source. Optional add-ons: +Then set your email in `config/sources.yaml` (`openalex_mailto`) — it puts +OpenAlex/Crossref requests in their polite pools; without it both APIs +throttle hard (HTTP 429). Works immediately with no external services: +collection, scoring, screening, draft audit, guards, figures-as-source. +Optional add-ons: | Dependency | Enables | Install | |---|---|---| | GROBID | PDF → structured text parsing | `docker run -d -p 8070:8070 lfoppiano/grobid:0.8.0` | -| Zotero + Better BibTeX | Canonical bibliography sync | zotero.org + Better BibTeX plugin | -| Graphviz | SVG/PNG figure rendering | graphviz.org/download | +| Zotero + Better BibTeX | Canonical bibliography sync (exported `.bib` file) | zotero.org + Better BibTeX plugin | +| Graphviz / Mermaid CLI | SVG/PNG figure rendering (sources are always written even without them) | graphviz.org/download | +| Semantic Scholar API key | Reliable S2 collection (unauthenticated S2 rate-limits hard) | `semantic_scholar_api_key` in `config/sources.yaml` | +| `OPENAI_API_KEY` | `extract-evidence-llm` (guarded, verbatim-checked LLM extraction; any OpenAI-compatible endpoint via `OPENAI_BASE_URL`) | environment variable | ## Typical session @@ -126,8 +136,12 @@ python scripts/paperops.py manuscript-patch-preview python scripts/paperops.py apply-manuscript-patch --from-preview --dry-run python scripts/paperops.py apply-manuscript-patch --from-preview --apply -# Audit your own draft (docx/md/qmd) — structure, unsourced claims, overclaims -python scripts/paperops.py audit-manuscript-draft --input my_thesis_draft.docx +# Audit your own draft (docx/md/qmd) — structure, unsourced claims, overclaims, +# and automatic numeric cross-check against your experiment output files +python scripts/paperops.py audit-manuscript-draft --input my_thesis_draft.docx --experiment-data data/experiment_outputs + +# Human verification (THE gate) — recorded in the verification ledger +python scripts/paperops.py verify-evidence --evidence-id ev_xxxx --by "Your Name" --note "checked p.3" --attest # Figures and final checks python scripts/paperops.py render-figures @@ -141,21 +155,32 @@ python scripts/paperops.py smoke-test `audit-manuscript-draft` was used on a real KCI manuscript (413 paragraphs): it scanned 378 sentences, checked chapter structure, flagged unsourced strong -claims and overclaim language, and cross-checked **all 178 numeric values** -in the draft against the actual experiment output files — 0 mismatches, with +claims and overclaim language, and all 178 numeric values in the draft were +cross-checked against the actual experiment output files — 0 mismatches, with 2 rounding differences explained and 1 base rate re-computed from raw -prediction logs. The audit never edits the draft and never marks anything -verified; it produces a findings report (MD + CSV) for the author. +prediction logs. (That cross-check was manual at the time; since v0.2.0 the +`--experiment-data` flag performs it automatically: exact match, rounding +match, and not-found-in-data statuses per number.) The audit never edits the +draft and never marks anything verified; it produces a findings report +(MD + CSV) for the author. ## Governance rules -1. The Evidence Matrix is never modified casually. -2. `verified=true` is never set automatically — there is no automated - transition into the verified state. -3. Quote/page matching is *source alignment*, not truth validation. -4. Manuscript edits happen only through guarded preview/apply with backups - and post-apply guard + smoke-test. -5. Related-work findings are framed as design patterns, never as performance +1. The Evidence Matrix is never modified casually; all rewrites are atomic + (temp file + rename) with timestamped backups. +2. `verified=true` is never set automatically. The only sanctioned path is + `verify-evidence --attest`, which records who/when/how in + `matrices/verification_ledger.csv`; `guard-no-auto-verified` fails any + verified row without a matching ledger attestation. +3. LLM-extracted candidates are verbatim-checked against the parsed source + text — paraphrased or invented quotes are dropped and reported — and then + flow through the same human review gates as heuristic candidates. +4. Quote/page matching is *source alignment*, not truth validation. +5. Manuscript edits happen only through guarded preview/apply with backups + (stored outside the guarded tree), SHA-256 checks, and post-apply guard + + smoke-test; approval reports are cross-checked against the current + manuscript SHA-256 so stale or fabricated reports are rejected. +6. Related-work findings are framed as design patterns, never as performance evidence for PaperOps itself. ## What this repo does NOT include @@ -169,11 +194,21 @@ every release (`scripts/build_public_release.py`). ## Honest limitations -- Evidence extraction is keyword/heuristic-based; an LLM-assisted extractor - is a planned, separately-guarded step. -- Draft auditing is heuristic flagging for human review, not truth validation. +- The LLM extractor only guarantees quotes are verbatim; claim/quote + *selection* quality still depends on the model, and every candidate still + requires human review. +- Draft auditing is heuristic flagging for human review, not truth + validation; the numeric cross-check verifies numbers appear in your data + files, not that the analysis is correct. - Quote/page alignment does not validate the truth of a claim — by design. - Quantitative result figures are never generated without an actual data file. +- Unauthenticated Semantic Scholar collection is rate-limited by S2; a free + API key is effectively required for that source. +- The verification ledger deters accidental and casual automated + verification; it is not cryptographic and cannot stop a determined human + from forging attestations in their own local files. +- The non-English READMEs may lag the English one by a release; the English + README and CHANGELOG are authoritative. ## License diff --git a/config/draft_audit.yaml b/config/draft_audit.yaml new file mode 100644 index 0000000..74555de --- /dev/null +++ b/config/draft_audit.yaml @@ -0,0 +1,63 @@ +# Draft audit configuration (used by scripts/paperops_draft_audit.py). +# +# Loaded if present and merged over the neutral built-in defaults; the tool +# degrades gracefully to those defaults when this file or PyYAML is missing. +# Entries are Python regex fragments (single-quote them so backslashes +# survive YAML); invalid regexes are treated as literal strings. +# +# Schema: +# source_mentions: list of source-name regexes — a sentence mentioning +# one of these counts as "has a source", and matches +# are inventoried against the Evidence Matrix +# required_chapters: list of {name, patterns} — the structure check marks +# a chapter [OK] only when one of its patterns matches +# a heading line (never body prose) +# overclaim_extra: extra overclaim regexes added to the built-in +# Korean/English detectors +# strong_claim_extra: extra strong-claim regexes added to the built-in +# Korean/English detectors +# +# The values below are the current author's (thesis on ontology-based AX +# governance); replace them with your own topic's sources and structure. + +source_mentions: + - 'Ontology Development 101' + - 'FEEKG' + - 'FinCaKG-?Onto' + - 'FinCaKG' + - 'FintechKG' + - 'W3C' + - 'SHACL Recommendation' + - 'SHACL' + - 'GQL' + - 'ISO/IEC\s*39075(:2024)?' + - 'ISO/IEC\s*\d+' + - 'Neo4j' + - 'Cypher Manual' + - 'Cypher' + - 'OPEN DART' + - 'KRX' + - 'FRED' + - 'ECOS' + - 'Data\.go\.kr' + +required_chapters: + - name: '서론' + patterns: ['서\s*론', 'Introduction'] + - name: '관련연구' + patterns: ['관련\s*연구', '선행\s*연구', 'Related Work'] + - name: '연구설계/방법' + patterns: ['연구\s*설계', '연구\s*방법', '방법론', 'Method'] + - name: '아티팩트/온톨로지 설계' + patterns: ['온톨로지\s*설계', '시스템\s*설계', 'Ontology', 'Design'] + - name: '평가/검증' + patterns: ['평가', '검증', 'Evaluation', 'Validation'] + - name: '결론' + patterns: ['결\s*론', 'Conclusion'] + - name: '참고문헌' + patterns: ['참고\s*문헌', 'References', '원문\s*확인'] + +overclaim_extra: + - 'hallucination[을를]?\s*제거' + +strong_claim_extra: [] diff --git a/config/figures.yaml b/config/figures.yaml new file mode 100644 index 0000000..693ba32 --- /dev/null +++ b/config/figures.yaml @@ -0,0 +1,259 @@ +# PaperOps figure specs (spec-driven; sources are rendered verbatim). +# +# Principle: this file never fabricates data figures. Only the diagram sources +# written below are rendered; quantitative/result charts require real data and +# are out of scope here. +# +# Schema (loaded by scripts/paperops_figures.py): +# figures: +# - id: unique snake_case figure id (required) +# kind: dot | mermaid (required) +# caption: manuscript caption text (required) +# target_heading: exact markdown heading line to anchor (required) +# source: | (required) +# inline dot/mermaid source, rendered as-is +# placeholder: custom manuscript block to insert instead (optional) +# of the default ![caption](figures/.svg){#fig-} line +# Optional extra keys: +# title: human-readable figure title +# target_file: manuscript file the figure is inserted into +# mermaid: companion Mermaid source emitted as .mmd for GitHub-native +# display when kind is dot +# +# Heading anchoring: target_heading must match a manuscript line exactly +# (whitespace-stripped, line-anchored; matches inside ``` code fences are +# ignored). Zero matches blocks with heading_not_found; multiple matches +# block with heading_ambiguous. +figures: +- id: fig_pipeline + kind: dot + title: PaperOps end-to-end pipeline + caption: PaperOps end-to-end pipeline. Literature collection, parsing, and evidence extraction + are automated, while review, verification, and manuscript changes pass through explicit + human approval gates. + target_file: 05_manuscript/chapters/ch3_method.qmd + target_heading: '## PaperOps Architecture' + source: | + digraph fig_pipeline { + graph [fontname="Helvetica", fontsize=11, rankdir=TB, splines=ortho, nodesep=0.45, ranksep=0.55, pad=0.2]; + node [fontname="Helvetica", fontsize=11, shape=box, style="rounded,filled", fillcolor="#F4F4F2", color="#555555", margin="0.18,0.10"]; + edge [fontname="Helvetica", fontsize=9, color="#555555", arrowsize=0.7]; + collect [label="Collect\n(arXiv / S2 / OpenAlex)"]; + screen [label="Score & Screen"]; + pdf [label="PDF Download"]; + grobid [label="GROBID Parse"]; + extract [label="Evidence Candidate\nExtraction"]; + review [label="Human Review\n(review queue)", fillcolor="#FFE9C7"]; + matrix [label="Evidence Matrix\n(promoted rows)", shape=cylinder, fillcolor="#EAF4EA"]; + preview [label="Manuscript Patch\nPreview + Diff"]; + approve [label="Human Approval", fillcolor="#FFE9C7"]; + apply [label="Guarded Apply\n(backup + LF write)"]; + guard [label="Guards\n(no-auto-verified, smoke-test)", fillcolor="#DCE9F7"]; + thesis [label="Thesis Manuscript\n(Quarto)", shape=cylinder, fillcolor="#EAF4EA"]; + subgraph cluster_auto { + label="Automated collection & extraction"; style=dashed; color="#999999"; + collect -> screen -> pdf -> grobid -> extract; + } + subgraph cluster_gov { + label="Governed review & writing"; style=dashed; color="#999999"; + review -> matrix [label="promote"]; + matrix -> preview -> approve -> apply -> thesis; + } + extract -> review; + apply -> guard [style=dashed, label="post-check"]; + guard -> matrix [style=dashed, label="audit", constraint=false]; + } + mermaid: | + flowchart LR + A[Collect
arXiv / Semantic Scholar / OpenAlex] --> B[Score & Screen] + B --> C[PDF Download] + C --> D[GROBID Parse] + D --> E[Evidence Candidate Extraction] + E --> F{{Human Review}} + F -->|promote| G[(Evidence Matrix)] + G --> H[Manuscript Patch Preview + Diff] + H --> I{{Human Approval}} + I --> J[Guarded Apply
backup + LF write] + J --> K[(Thesis Manuscript)] + J -.post-check.-> L[Guards
no-auto-verified, smoke-test] + L -.audit.-> G +- id: fig_evidence_flow + kind: dot + title: Evidence governance flow + caption: Evidence governance flow. Quote matching and page location are treated as source-alignment + checks; the verified state is reachable only through an explicit human verification gate. + target_file: 05_manuscript/chapters/ch3_method.qmd + target_heading: '## Evidence-first Workflow' + source: | + digraph fig_evidence_flow { + graph [fontname="Helvetica", fontsize=11, rankdir=TB, splines=ortho, nodesep=0.45, ranksep=0.55, pad=0.2]; + node [fontname="Helvetica", fontsize=11, shape=box, style="rounded,filled", fillcolor="#F4F4F2", color="#555555", margin="0.18,0.10"]; + edge [fontname="Helvetica", fontsize=9, color="#555555", arrowsize=0.7]; + cand [label="Evidence Candidate\n(claim + quote + location)"]; + valid [label="Structural Validation\n(schema, citekey)"]; + align [label="Source Alignment\n(quote match, page locate)"]; + queue [label="Review Queue", shape=cylinder, fillcolor="#EAF4EA"]; + human [label="Human Decision\n(keep / revise / reject)", fillcolor="#FFE9C7"]; + promoted [label="Promoted Row\nverified=false", shape=cylinder, fillcolor="#EAF4EA"]; + pdfcheck [label="PDF Page Check\n(required for high-risk)"]; + verify [label="Human Verification\nGate", fillcolor="#FFE9C7"]; + verified [label="verified=true\n(manual only)", fillcolor="#DCE9F7"]; + align_note [label="alignment != truth validation", shape=note, fillcolor="#FFF7D6"]; + { rank=same; cand; valid; align; queue; human; } + { rank=same; align_note; verified; verify; pdfcheck; promoted; } + cand -> valid -> align -> queue -> human [constraint=false]; + // invisible vertical pins keep row 2 folded under row 1 + cand -> align_note [style=invis]; + valid -> verified [style=invis]; + align -> verify [style=invis]; + queue -> pdfcheck [style=invis]; + human -> promoted [label="promote"]; + promoted -> pdfcheck [constraint=false]; + pdfcheck -> verify [constraint=false]; + verify -> verified [constraint=false]; + } + mermaid: | + flowchart LR + A[Evidence Candidate
claim + quote + location] --> B[Structural Validation] + B --> C[Source Alignment
quote match, page locate] + C --> D[(Review Queue)] + D --> E{{Human Decision}} + E --> F[(Promoted Row
verified=false)] + F --> G[PDF Page Check] + G --> H{{Human Verification Gate}} + H --> I[verified=true
manual only] +- id: fig_guarded_apply + kind: dot + title: Guarded manuscript apply workflow + caption: Guarded manuscript apply workflow. Every manuscript change is previewed as a diff, + requires human approval, is applied against a backup, and is followed by automated guard + and smoke-test checks; failures roll back from the backup. + target_file: 05_manuscript/chapters/ch3_method.qmd + target_heading: '## Human Verification Policy' + source: | + digraph fig_guarded_apply { + graph [fontname="Helvetica", fontsize=11, rankdir=TB, splines=ortho, nodesep=0.45, ranksep=0.55, pad=0.2]; + node [fontname="Helvetica", fontsize=11, shape=box, style="rounded,filled", fillcolor="#F4F4F2", color="#555555", margin="0.18,0.10"]; + edge [fontname="Helvetica", fontsize=9, color="#555555", arrowsize=0.7]; + preview [label="Patch Preview\n(CSV + MD + diff)"]; + review [label="Human Diff Review", fillcolor="#FFE9C7"]; + backup [label="Backup Chapters"]; + apply [label="Apply\n(SHA-checked, LF write)"]; + postguard [label="guard-no-auto-verified\n+ smoke-test", fillcolor="#DCE9F7"]; + report [label="Apply Report", shape=cylinder, fillcolor="#EAF4EA"]; + rollback [label="Rollback from Backup", fillcolor="#F7DCDC"]; + preview -> review; + review -> backup [label="approved"]; + review -> preview [label="rejected / revise", style=dashed]; + backup -> apply -> postguard; + postguard -> report [label="pass"]; + postguard -> rollback [label="fail", style=dashed]; + rollback -> preview [style=dashed]; + } + mermaid: | + flowchart TB + A[Patch Preview
CSV + MD + diff] --> B{{Human Diff Review}} + B -->|approved| C[Backup Chapters] + B -.rejected / revise.-> A + C --> D[Apply
SHA-checked, LF write] + D --> E[guard-no-auto-verified
+ smoke-test] + E -->|pass| F[(Apply Report)] + E -.fail.-> G[Rollback from Backup] + G -.-> A +- id: fig_architecture + kind: dot + title: PaperOps system architecture + caption: PaperOps system architecture. A single CLI orchestrates external services (GROBID, + Zotero/Better BibTeX) and local stores (paper DB, matrices, manuscript), with audit reports + produced at each guarded step. + target_file: 05_manuscript/chapters/ch4_system.qmd + target_heading: '## Data Model' + source: | + digraph fig_architecture { + graph [fontname="Helvetica", fontsize=11, rankdir=TB, splines=ortho, nodesep=0.45, ranksep=0.55, pad=0.2]; + node [fontname="Helvetica", fontsize=11, shape=box, style="rounded,filled", fillcolor="#F4F4F2", color="#555555", margin="0.18,0.10"]; + edge [fontname="Helvetica", fontsize=9, color="#555555", arrowsize=0.7]; + cli [label="PaperOps CLI\n(scripts/paperops.py)"]; + subgraph cluster_ext { + label="External services"; style=dashed; color="#999999"; + grobid [label="GROBID\n(Docker)"]; + zotero [label="Zotero +\nBetter BibTeX"]; + apis [label="Paper APIs\n(arXiv, S2, OpenAlex)"]; + } + subgraph cluster_store { + label="Local stores"; style=dashed; color="#999999"; + db [label="papers.sqlite", shape=cylinder, fillcolor="#EAF4EA"]; + matrices [label="matrices/\n(evidence, screening, gap)", shape=cylinder, fillcolor="#EAF4EA"]; + manuscript [label="05_manuscript/\n(Quarto)", shape=cylinder, fillcolor="#EAF4EA"]; + reports [label="reports/\n(audit, review, figures)", shape=cylinder, fillcolor="#EAF4EA"]; + } + config [label="config/\n(sources, scoring, prompts)", shape=folder]; + cli -> apis [dir=both]; + cli -> grobid [dir=both]; + cli -> zotero [dir=both]; + cli -> db [dir=both]; + cli -> matrices [dir=both]; + cli -> manuscript [label="guarded\napply only"]; + cli -> reports; + config -> cli; + } + mermaid: | + flowchart TB + CLI[PaperOps CLI
scripts/paperops.py] + subgraph External services + G[GROBID Docker] + Z[Zotero + Better BibTeX] + A[Paper APIs
arXiv, S2, OpenAlex] + end + subgraph Local stores + DB[(papers.sqlite)] + M[(matrices/)] + MS[(05_manuscript/)] + R[(reports/)] + end + CFG[config/] --> CLI + CLI <--> A + CLI <--> G + CLI <--> Z + CLI <--> DB + CLI <--> M + CLI -->|guarded apply only| MS + CLI --> R +- id: fig_verification_states + kind: dot + title: Evidence verification state transitions + caption: Evidence verification state transitions. There is no automated transition into + the verified state; only a human reviewer can mark evidence as verified, and guards enforce + this invariant. + target_file: 05_manuscript/chapters/ch5_evaluation.qmd + target_heading: '## Metrics' + source: | + digraph fig_verification_states { + graph [fontname="Helvetica", fontsize=11, rankdir=LR, splines=ortho, nodesep=0.45, ranksep=0.55, pad=0.2]; + node [fontname="Helvetica", fontsize=11, shape=box, style="rounded,filled", fillcolor="#F4F4F2", color="#555555", margin="0.18,0.10"]; + edge [fontname="Helvetica", fontsize=9, color="#555555", arrowsize=0.7]; + extracted [label="extracted"]; + validated [label="candidate\nvalidated"]; + in_review [label="in review", fillcolor="#FFE9C7"]; + promoted [label="promoted\n(verified=false)", shape=cylinder, fillcolor="#EAF4EA"]; + pdf_check [label="pdf check\nrequired"]; + verified [label="verified=true", fillcolor="#DCE9F7"]; + rejected [label="rejected", fillcolor="#F7DCDC"]; + extracted -> validated -> in_review; + in_review -> promoted [label="human keep"]; + in_review -> rejected [label="human reject"]; + promoted -> pdf_check; + pdf_check -> verified [label="human only", penwidth=2]; + promoted -> verified [style=invis]; + noauto [label="no automated edge\ninto verified", shape=note, fillcolor="#FFF7D6"]; + } + mermaid: | + stateDiagram-v2 + [*] --> extracted + extracted --> validated + validated --> in_review + in_review --> promoted : human keep + in_review --> rejected : human reject + promoted --> pdf_check + pdf_check --> verified : human only + note right of verified : no automated transition diff --git a/config/pipeline.yaml b/config/pipeline.yaml index 209c297..8816857 100644 --- a/config/pipeline.yaml +++ b/config/pipeline.yaml @@ -4,6 +4,18 @@ defaults: card_limit: 25 parse_max_pages: 80 +# GROBID server for PDF -> TEI parsing (or set the GROBID_URL env var). +# grobid_url: "http://localhost:8070" + +# LLM settings for the guarded extract-evidence-llm command. The API key is NEVER put +# in this file — set the OPENAI_API_KEY env var (OPENAI_BASE_URL for compatible servers). +llm: + base_url: "https://api.openai.com/v1" + model: "gpt-4o-mini" + max_sections: 12 + max_candidates_per_section: 5 + timeout_seconds: 120 + paths: database: data/metadata/papers.sqlite evidence_matrix: matrices/evidence_matrix.csv diff --git a/config/qa_profile.yaml b/config/qa_profile.yaml new file mode 100644 index 0000000..139a3b7 --- /dev/null +++ b/config/qa_profile.yaml @@ -0,0 +1,49 @@ +# Corpus/domain-specific QA heuristics, externalized from code (v0.2.0). +# These lists previously lived hardcoded in paperops.py and were specific to one +# author's biomedical-adjacent corpus. Edit them for YOUR corpus; empty lists simply +# disable the corresponding warning/flag. + +# Terms that mark an evidence row as domain-specific (audit-domain-specific-claims): +domain_specific_keywords: + - biomedical + - clinical + - clinician + - patient + - patients + - disease + - diseases + - medical + - medicine + - orphanet + - orphadata + - pmid + - drug + - phenotyping + - biology + - regulatory + - onset age + - clinical milestone + +# Terms that flag a row in guard-paperops-overclaim (performance/scale claims that must +# never be presented as PaperOps' own performance evidence): +overclaim_keywords: + - accuracy + - validated + - consensus triples + - disease + - diseases + - clinical + - biology + - biomedical + - orphadata + - pmid + +# Exact tokens from a specific source corpus; quotes containing them get the +# domain_specific_claim_use_cautiously warning in promoted-evidence QA: +corpus_specific_tokens: + - "460,497" + - "13,431" + - "92.7%" + - Orphadata + - biomedical + - clinical diff --git a/config/review_overrides.yaml b/config/review_overrides.yaml new file mode 100644 index 0000000..f76d5aa --- /dev/null +++ b/config/review_overrides.yaml @@ -0,0 +1,95 @@ +# Row-level external review decisions for promoted evidence rows. +# Moved out of code (v0.2.0): these are per-candidate HUMAN/PM review decisions for a +# specific author corpus. Other users: replace `decisions` with your own, or leave empty — +# unknown candidate_ids fall back to `default_decision` (conservative: downgrade_to_pdf_check). +source_review: GPT Pro row-level review captured 2026-06-04 +default_decision: + external_review_decision: downgrade_to_pdf_check + suggested_claim_type: '' + suggested_use_in_section: '' + rewrite_if_needed: '' + pdf_check_priority: high + paperops_generalization_allowed: 'false' + pdf_page_check_required: 'true' + domain_specific_risk: unknown +decisions: + fb5137d2707f5e1d: + external_review_decision: downgrade_to_pdf_check + suggested_claim_type: evaluation_pattern + suggested_use_in_section: ch2_related_work;ch3_evaluation_design_motivation + rewrite_if_needed: ChronoMedKG 사례는 LLM judge가 아닌 gold-standard comparison을 평가 + 설계에 활용한 도메인 특화 사례를 보여준다. + pdf_check_priority: high + paperops_generalization_allowed: 'false' + pdf_page_check_required: 'true' + domain_specific_risk: high + 23fe8ed59649fffa: + external_review_decision: revise + suggested_claim_type: method_pattern + suggested_use_in_section: ch2_related_work;ch3_design_motivation + rewrite_if_needed: 도메인 특화 KG 연구에서 disease-autonomous multi-agent pipeline이 대규모 + biomedical triples 생성에 활용된 사례가 있다. + pdf_check_priority: high + paperops_generalization_allowed: 'false' + pdf_page_check_required: 'true' + domain_specific_risk: high + d5f10096ff325472: + external_review_decision: revise + suggested_claim_type: method_pattern + suggested_use_in_section: ch2_related_work;ch3_design_motivation + rewrite_if_needed: ChronoMedKG는 각 질병 단위를 독립적으로 처리하는 multi-stage pipeline 구조를 채택했다. + 이는 도메인 단위 batch/agent pipeline 설계 사례로 참고할 수 있다. + pdf_check_priority: medium + paperops_generalization_allowed: 'false' + pdf_page_check_required: 'true' + domain_specific_risk: medium + e340369a59fb2532: + external_review_decision: revise + suggested_claim_type: agent_workflow_pattern + suggested_use_in_section: ch2_related_work;ch3_system_design_motivation + rewrite_if_needed: 특정 biomedical KG 구축 사례에서는 disease identifier를 입력으로 네 개의 협력 + agent가 end-to-end pipeline을 수행하도록 설계했다. + pdf_check_priority: medium + paperops_generalization_allowed: 'false' + pdf_page_check_required: 'true' + domain_specific_risk: medium + 06488bdee4bc0074: + external_review_decision: keep + suggested_claim_type: governance + suggested_use_in_section: ch3_system_design;ch5_evaluation_design + rewrite_if_needed: 검증 harness, judge-panel code, error taxonomy를 공개하는 방식은 연구 자동화 + 시스템의 auditability와 reproducibility를 높이는 설계 패턴으로 볼 수 있다. + pdf_check_priority: medium + paperops_generalization_allowed: limited_auditability_design_principle_only + pdf_page_check_required: 'true' + domain_specific_risk: low + 154a4607cb42751c: + external_review_decision: revise + suggested_claim_type: provenance_pattern + suggested_use_in_section: ch2_related_work;ch3_evidence_model_design + rewrite_if_needed: ChronoMedKG는 triple 단위에 evidence grading과 PMID provenance를 + 부여하는 방식으로 출처 추적성을 강화한 사례다. + pdf_check_priority: high + paperops_generalization_allowed: 'false' + pdf_page_check_required: 'true' + domain_specific_risk: high + 6abed7fd8c1bd310: + external_review_decision: revise + suggested_claim_type: human_oversight + suggested_use_in_section: ch2_related_work;ch3_governance_design;ch6_limitations + rewrite_if_needed: 도메인 특화 자동화 시스템도 원천 데이터 범위와 실제 적용 범위를 구분하며, 고위험 도메인 적용에는 인간 + 전문가 검토와 별도 평가가 필요하다는 제한을 명시한다. + pdf_check_priority: medium + paperops_generalization_allowed: 'false' + pdf_page_check_required: 'true' + domain_specific_risk: high + 67558897e8bfbc54: + external_review_decision: revise + suggested_claim_type: validation_boundary + suggested_use_in_section: ch3_evaluation_design;ch5_evaluation_limitations;ch6_limitations + rewrite_if_needed: text-grounding 검증은 원문 근거 일치 여부를 확인하는 절차이지, 도메인 사실 자체의 독립적 재검증은 + 아니라는 한계를 명확히 해야 한다. + pdf_check_priority: medium + paperops_generalization_allowed: limited_validation_boundary_principle_only + pdf_page_check_required: 'true' + domain_specific_risk: medium diff --git a/config/scoring.yaml b/config/scoring.yaml index 9b57b6a..7158220 100644 --- a/config/scoring.yaml +++ b/config/scoring.yaml @@ -13,6 +13,7 @@ thresholds: important: 0.75 recency: - current_year: 2026 - full_score_years: 3 + # current_year intentionally NOT pinned: the code uses the actual current year, so + # recency scoring never silently goes stale. Set current_year only to freeze scoring + # for a reproducibility snapshot. half_life_years: 8 diff --git a/config/sources.yaml b/config/sources.yaml index fe1ef9b..3b148b1 100644 --- a/config/sources.yaml +++ b/config/sources.yaml @@ -1,5 +1,10 @@ -user_agent: "PaperOps/0.1 (mailto:your-email@example.com)" +# Leave user_agent empty to auto-generate "PaperOps/0.2 (mailto:)". +user_agent: "" +# STRONGLY RECOMMENDED: your real email. It puts OpenAlex/Crossref requests in the +# "polite pool" — without it both APIs throttle aggressively (HTTP 429). openalex_mailto: "your-email@example.com" +# Optional but recommended: unauthenticated Semantic Scholar search is heavily +# rate-limited (429). Get a free key at https://www.semanticscholar.org/product/api semantic_scholar_api_key: "" sources: diff --git a/config/topic_profile.yaml b/config/topic_profile.yaml index c9fb875..875d748 100644 --- a/config/topic_profile.yaml +++ b/config/topic_profile.yaml @@ -109,16 +109,19 @@ preferred_venues: - IEEE Security & Privacy - arXiv +# Queries use explicit boolean syntax (AND/OR, quoted phrases). Bare keyword soups are +# no longer sent as-is: arXiv treated them as near-match-everything and returned the +# newest unrelated papers. Quoted phrases joined with AND keep results on-topic. query_groups: - name: operational_ontology_ax - query: "operational ontology enterprise knowledge graph AI operating model digital transformation decision automation" + query: '"enterprise ontology" AND ("knowledge graph" OR "operating model") AND ("decision automation" OR "digital transformation")' - name: ontology_graphrag_governance - query: "ontology knowledge graph GraphRAG generative AI governance traceability auditability" + query: '"knowledge graph" AND ("GraphRAG" OR "retrieval augmented generation") AND (governance OR traceability OR auditability)' - name: llm_governance_security - query: "LLM governance generative AI security prompt injection data leakage guardrails access control" + query: '"LLM" AND (governance OR guardrails) AND ("prompt injection" OR "data leakage" OR "access control")' - name: decision_automation_policy_reasoning - query: "decision automation policy as code rule based reasoning ontology explainable AI human in the loop" + query: '"decision automation" AND (ontology OR "policy as code" OR "rule-based reasoning") AND ("explainable AI" OR "human in the loop")' - name: enterprise_ai_operations - query: "enterprise AI platform LLMOps MLOps AI governance risk management compliance automation" + query: '("LLMOps" OR "MLOps") AND "AI governance" AND ("risk management" OR compliance)' - name: public_financial_ai_governance - query: "public sector AI governance financial AI governance model risk management trustworthy AI" \ No newline at end of file + query: '("public sector" OR financial) AND "AI governance" AND ("model risk management" OR "trustworthy AI")' \ No newline at end of file diff --git a/docs/RUN_AX_ONTOLOGY_GOVERNANCE.md b/docs/RUN_AX_ONTOLOGY_GOVERNANCE.md index 0875335..d2fcf52 100644 --- a/docs/RUN_AX_ONTOLOGY_GOVERNANCE.md +++ b/docs/RUN_AX_ONTOLOGY_GOVERNANCE.md @@ -25,7 +25,7 @@ Use PaperOps to collect, triage, and evidence-govern papers for a master's-level ```bash git clone https://github.com/SakJaeLim/paperops.git cd paperops -git checkout topic/ax-ontology-governance +git checkout main python -m venv .venv # Windows @@ -44,6 +44,8 @@ python scripts/paperops.py brief python scripts/paperops.py status ``` +Note: the former `topic/ax-ontology-governance` working branch has been merged into `main` and deleted, so all commands in this runbook run from `main`. + ## If PDFs are needed ```bash diff --git a/pyproject.toml b/pyproject.toml index 26f7f44..f565db3 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,13 +1,12 @@ [project] name="paperops-agent" -version="0.1.0" +version="0.2.0" requires-python=">=3.10" dependencies=[ "requests>=2.31.0", "PyYAML>=6.0.1", - "rapidfuzz>=3.6.0", - "pandas>=2.0.0", "PyMuPDF>=1.23.0", - "bibtexparser>=1.4.0", - "pypdf>=4.0.0", ] + +[project.optional-dependencies] +dev=["pytest>=7.0"] diff --git a/requirements.txt b/requirements.txt index ef4a10d..5a242bf 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,7 +1,6 @@ +# Runtime dependencies actually imported by the code. +# (pandas, rapidfuzz, bibtexparser, pypdf were removed in v0.2.0: none were imported +# anywhere, and bibtexparser's sdist-only build broke installs on some systems.) requests>=2.31.0 PyYAML>=6.0.1 -rapidfuzz>=3.6.0 -pandas>=2.0.0 PyMuPDF>=1.23.0 -bibtexparser>=1.4.0 -pypdf>=4.0.0 diff --git a/scripts/build_public_release.py b/scripts/build_public_release.py index bed1d30..3c1478f 100644 --- a/scripts/build_public_release.py +++ b/scripts/build_public_release.py @@ -5,15 +5,26 @@ copyrighted paper PDFs, parsed full texts, evidence quotes, personal manuscript chapters, and private logs can never leak into the public repo. +Safety model, in order: + 1. copy whitelisted paths into dist/public_release/ + 2. sanitize the exported config YAML copies: values of secret-bearing + keys are blanked in the copy (the working tree is never modified) + 3. scan every exported text file against FORBIDDEN_PATTERNS + 4. if the scan finds anything, the release directory is renamed to + dist/public_release_QUARANTINE_ so nothing publishable + remains at the expected path, and the exit status is 1 + Usage: - python scripts/build_public_release.py # build - python scripts/build_public_release.py --check # sanitize-scan only + python scripts/build_public_release.py # build + sanitize + scan + python scripts/build_public_release.py --check # scan an existing + # dist/public_release only: no copy, no sanitize, and + # RELEASE_INFO.txt is not rewritten -- but a failing scan still + # quarantines the directory. """ from __future__ import annotations import argparse import re import shutil -import sys from datetime import datetime from pathlib import Path @@ -41,13 +52,17 @@ ('README.fr.md', 'README.fr.md'), ('README.ar.md', 'README.ar.md'), ('LICENSE', 'LICENSE'), + ('CHANGELOG.md', 'CHANGELOG.md'), ('docs/00_MASTER_DESIGN.md', 'docs/00_MASTER_DESIGN.md'), ('docs/01_MVP_ROADMAP.md', 'docs/01_MVP_ROADMAP.md'), ('docs/03_TOOL_SYNTHESIS.md', 'docs/03_TOOL_SYNTHESIS.md'), + ('docs/RUN_AX_ONTOLOGY_GOVERNANCE.md', 'docs/RUN_AX_ONTOLOGY_GOVERNANCE.md'), ] WHITELIST_DIRS = [ ('config', 'config'), + ('tests', 'tests'), + ('.github', '.github'), ('reports/figures/src', 'assets/figures/src'), ] @@ -55,15 +70,48 @@ ('reports/figures', '*.svg', 'assets/figures'), ] -# Patterns that must never appear in exported text files. +EMAIL_RE = re.compile(r'[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}') + +# Email addresses allowed to appear in the export: documentation +# placeholders and well-known no-reply addresses. +ALLOWED_EMAILS = [ + re.compile(r'^[A-Za-z0-9._%+-]+@example\.(?:com|org)$'), + re.compile(r'^noreply@anthropic\.com$'), + re.compile(r'^[A-Za-z0-9._%+-]+@users\.noreply\.github\.com$'), +] + +# (rule name, compiled regex) -- patterns that must never appear in +# exported text files. The scan report names the rule that fired. +# Note the config-assignment rule uses [ \t] rather than \s around the +# separator on purpose: \s would let the value run onto the next line, so +# an empty key followed by a long identifier on the following line would +# false-positive. FORBIDDEN_PATTERNS = [ - (re.compile(r'[A-Za-z0-9._%+-]+@(?:gmail|naver|daum|kakao)\.[A-Za-z]{2,}'), - 'personal email'), - (re.compile(r'sk-[A-Za-z0-9]{20,}'), 'API key-like token'), - (re.compile(r'ghp_[A-Za-z0-9]{20,}'), 'GitHub token'), - (re.compile(r'AKIA[0-9A-Z]{16}'), 'AWS key'), + ('email-address', EMAIL_RE), + ('github-fine-grained-pat', re.compile(r'github_pat_[A-Za-z0-9_]{20,}')), + ('github-token', re.compile(r'gh[pousr]_[A-Za-z0-9]{20,}')), + ('private-key-block', re.compile( + r'-----BEGIN(?:\x20RSA|\x20EC|\x20OPENSSH|\x20DSA|\x20PGP)?' + r'\x20PRIVATE\x20KEY-----')), + ('aws-access-key', re.compile(r'AKIA[0-9A-Z]{16}')), + ('sk-secret-key', re.compile(r'sk-(?:ant-)?[A-Za-z0-9_-]{20,}')), + ('slack-token', re.compile(r'xox[baprs]-[A-Za-z0-9-]{10,}')), + ('google-api-key', re.compile(r'AIza[0-9A-Za-z_-]{35}')), + ('jwt', re.compile(r'eyJ[A-Za-z0-9_-]{20,}\.eyJ')), + ('config-assignment-secret', re.compile( + r"(?i)(?:api[_-]?key|secret|token|password)" + r"[ \t]*[:=][ \t]*['\"]?[A-Za-z0-9_\-]{16,}")), ] +# Config keys whose exported values are always blanked. +SENSITIVE_KEY_RE = re.compile(r'(?i)api[_-]?key|mailto|token|secret') +# user_agent is blanked only when its value embeds an email address. +USER_AGENT_KEY_RE = re.compile(r'(?i)^user[_-]?agent$') +# A simple `key: value` YAML line (top-level or indented mapping entry). +CONFIG_LINE_RE = re.compile(r'^(\s*)([A-Za-z0-9_.\-]+)\s*:\s?(.*)$') +# Values that are already blank: empty, empty-quoted, or comment-only. +BLANK_VALUE_RE = re.compile(r'^(?:""|\'\')?\s*(?:#.*)?$') + TEXT_SUFFIXES = {'.py', '.md', '.txt', '.yaml', '.yml', '.toml', '.bat', '.csv', '.mmd', '.dot', '.svg', '.json'} @@ -89,11 +137,31 @@ def release_files(base): - return [p for p in base.rglob('*') + return [p for p in Path(base).rglob('*') if p.is_file() and '.git' not in p.parts] +def release_file_count(base): + """Number of files in the release, excluding RELEASE_INFO.txt itself. + + Documented choice: RELEASE_INFO.txt describes the release, so it is + not counted as release content. Because it is the only file written + after the count is taken, and it is excluded from the count, the + number is identical whether computed in build mode (just before + RELEASE_INFO.txt is written) or in --check mode (where a previous + RELEASE_INFO.txt already exists on disk). + """ + base = Path(base) + return len([p for p in release_files(base) + if not (p.parent == base and p.name == 'RELEASE_INFO.txt')]) + + +def _email_allowed(text): + return any(p.match(text) for p in ALLOWED_EMAILS) + + def sanitize_scan(base): + base = Path(base) issues = [] for path in release_files(base): if path.suffix.lower() not in TEXT_SUFFIXES: @@ -102,77 +170,181 @@ def sanitize_scan(base): text = path.read_text(encoding='utf-8', errors='ignore') except Exception: continue - for pat, label in FORBIDDEN_PATTERNS: + for name, pat in FORBIDDEN_PATTERNS: for m in pat.finditer(text): - issues.append(f'{path.relative_to(base)}: {label}: {m.group(0)[:40]}') + if name == 'email-address' and _email_allowed(m.group(0)): + continue + issues.append( + f'{path.relative_to(base)}: {name}: {m.group(0)[:40]}') return issues -def build(): - if DIST.exists(): +def sanitize_configs(dist=None): + """Blank secret-bearing values in the exported config YAML copies. + + Line-level rewrite (no YAML library needed): any `key: value` line + whose key name contains api_key / mailto / token / secret, or whose + key is user_agent with an email address in the value, is rewritten as + `key: ""` with indentation preserved. Values that are already blank + are left untouched and not reported. Only files under /config + are rewritten; the working tree copy is never modified. + + Returns a list of (relative-path, key) redaction records. The + records never include the removed values. + """ + dist = Path(dist) if dist is not None else DIST + redactions = [] + cfg_dir = dist / 'config' + if not cfg_dir.exists(): + return redactions + for path in sorted(cfg_dir.rglob('*')): + if not path.is_file() or path.suffix.lower() not in ('.yaml', '.yml'): + continue + try: + text = path.read_text(encoding='utf-8', errors='ignore') + except Exception: + continue + out = [] + changed = False + for line in text.splitlines(): + m = CONFIG_LINE_RE.match(line) + if m: + indent, key, value = m.group(1), m.group(2), m.group(3) + sensitive = bool(SENSITIVE_KEY_RE.search(key)) or bool( + USER_AGENT_KEY_RE.match(key) and EMAIL_RE.search(value)) + if sensitive and not BLANK_VALUE_RE.match(value.strip()): + out.append(f'{indent}{key}: ""') + redactions.append((str(path.relative_to(dist)), key)) + changed = True + continue + out.append(line) + if changed: + path.write_text('\n'.join(out) + '\n', encoding='utf-8') + return redactions + + +def quarantine(dist=None, stamp=None): + """Move a failed release out of the publishable path. + + Renames to _QUARANTINE_, replacing any prior + quarantine of the same name. Returns the quarantine path. + """ + dist = Path(dist) if dist is not None else DIST + stamp = stamp or datetime.now().strftime('%Y%m%d_%H%M%S') + target = dist.with_name(f'{dist.name}_QUARANTINE_{stamp}') + if target.exists(): + if target.is_dir(): + shutil.rmtree(target) + else: + target.unlink() + dist.rename(target) + return target + + +def build(root=None, dist=None): + root = Path(root) if root is not None else ROOT + dist = Path(dist) if dist is not None else DIST + if dist.exists(): # Preserve .git so the release repo keeps its remote/history. - for child in DIST.iterdir(): + for child in dist.iterdir(): if child.name == '.git': continue if child.is_dir(): shutil.rmtree(child) else: child.unlink() - DIST.mkdir(parents=True, exist_ok=True) + dist.mkdir(parents=True, exist_ok=True) missing = [] for src, dst in WHITELIST_FILES: - s = ROOT / src + s = root / src if not s.exists(): missing.append(src) continue - d = DIST / dst + d = dist / dst d.parent.mkdir(parents=True, exist_ok=True) shutil.copy2(s, d) for src, dst in WHITELIST_DIRS: - s = ROOT / src + s = root / src if not s.exists(): missing.append(src) continue - shutil.copytree(s, DIST / dst, dirs_exist_ok=True, - ignore=shutil.ignore_patterns('__pycache__', '*.pyc')) + shutil.copytree(s, dist / dst, dirs_exist_ok=True, + ignore=shutil.ignore_patterns( + '__pycache__', '*.pyc', '.pytest_cache')) for src, pattern, dst in WHITELIST_GLOBS: - s = ROOT / src + s = root / src if not s.exists(): missing.append(src) continue - d = DIST / dst + d = dist / dst d.mkdir(parents=True, exist_ok=True) for f in s.glob(pattern): shutil.copy2(f, d / f.name) - (DIST / '.gitignore').write_text(PUBLIC_GITIGNORE, encoding='utf-8') + (dist / '.gitignore').write_text(PUBLIC_GITIGNORE, encoding='utf-8') return missing -def main(): - ap = argparse.ArgumentParser() - ap.add_argument('--check', action='store_true', - help='scan existing dist/public_release only') - args = ap.parse_args() - if not args.check: - missing = build() +def write_release_info(dist, n_files, n_missing, redactions, n_issues, + stamp=None): + stamp = stamp or datetime.now().strftime('%Y-%m-%d %H:%M:%S') + lines = [ + f'PaperOps public release built {stamp}', + f'files={n_files} (count excludes RELEASE_INFO.txt itself)', + f'missing_whitelist_warnings={n_missing}', + f'config_redactions={len(redactions)}', + ] + for rel, key in redactions: + lines.append(f'redacted: {rel}: {key}') + lines.append(f'sanitize_issues={n_issues}') + (Path(dist) / 'RELEASE_INFO.txt').write_text( + '\n'.join(lines) + '\n', encoding='utf-8') + + +def run(check=False, root=None, dist=None): + """Build (unless check), sanitize, scan, report. Returns exit status.""" + root = Path(root) if root is not None else ROOT + dist = Path(dist) if dist is not None else DIST + missing = [] + redactions = [] + if not check: + missing = build(root, dist) for m in missing: print(f'WARN missing source: {m}') - if not DIST.exists(): - print('ERROR: dist/public_release does not exist; run without --check first') - raise SystemExit(1) - issues = sanitize_scan(DIST) - n_files = len(release_files(DIST)) - print(f'release_dir={DIST}') + redactions = sanitize_configs(dist) + for rel, key in redactions: + print(f'REDACTED {rel}: {key}') + if not dist.exists(): + print(f'ERROR: {dist} does not exist; run without --check first') + return 1 + issues = sanitize_scan(dist) + n_files = release_file_count(dist) + print(f'release_dir={dist}') print(f'file_count={n_files}') + print(f'missing_whitelist_warnings={len(missing)}') + print(f'config_redactions={len(redactions)}') print(f'sanitize_issues={len(issues)}') for i in issues: print(f'ISSUE: {i}') - stamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S') - (DIST / 'RELEASE_INFO.txt').write_text( - f'PaperOps public release built {stamp}\n' - f'files={n_files}\nsanitize_issues={len(issues)}\n', encoding='utf-8') + if not check: + write_release_info(dist, n_files, len(missing), redactions, + len(issues)) if issues: - raise SystemExit(1) + qdir = quarantine(dist) + print(f'QUARANTINED: sanitize scan failed; release moved to {qdir}') + print(f'Nothing publishable remains at {dist}.') + return 1 + return 0 + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument('--check', action='store_true', + help='scan existing dist/public_release only ' + '(no build/sanitize; failures still quarantine)') + args = ap.parse_args() + status = run(check=args.check) + if status: + raise SystemExit(status) if __name__ == '__main__': diff --git a/scripts/paperops.py b/scripts/paperops.py index 9de3996..a48d0a9 100644 --- a/scripts/paperops.py +++ b/scripts/paperops.py @@ -38,12 +38,55 @@ def slug(s, n=80): def norm_title(s): return re.sub(r'\W+', '', (s or '').lower()) def stable_id(p): - if p.get('doi'): key='doi:'+p['doi'].lower() - elif p.get('arxiv_id'): key='arxiv:'+p['arxiv_id'].lower() + if p.get('doi'): key='doi:'+normalize_doi(p['doi']) + elif p.get('arxiv_id'): key='arxiv:'+normalize_arxiv(p['arxiv_id']) else: key='title:'+norm_title(p.get('title','')) return hashlib.sha1(key.encode('utf-8')).hexdigest()[:16] +def read_text_compat(path, warnings=None): + """Read text safely: utf-8-sig strict, then cp949, then lossy utf-8 with warning.""" + data = Path(path).read_bytes() + for enc in ('utf-8-sig', 'cp949'): + try: + return data.decode(enc) + except UnicodeDecodeError: + continue + if warnings is not None: + warnings.append(f'{path}: undecodable bytes replaced (tried utf-8-sig, cp949)') + return data.decode('utf-8', errors='replace') + + +def read_csv_dict(path): + """Read a CSV as dict rows. BOM-tolerant, extra Excel columns captured under _extra.""" + path = Path(path) + if not path.exists(): + return [], [] + with path.open(encoding='utf-8-sig', newline='') as f: + reader = csv.DictReader(f, restkey='_extra', restval='') + rows = [] + for row in reader: + row.pop('_extra', None) + row.pop(None, None) + rows.append(row) + return rows, [h for h in (reader.fieldnames or []) if h is not None] + + +def atomic_write_csv(path, fieldnames, rows): + """Atomically rewrite a CSV: temp file + os.replace. Rows are projected onto fieldnames + so stray keys (Excel artifacts, restkey leftovers) can never crash or corrupt the write. + utf-8-sig keeps Korean text readable when the file is opened in Excel.""" + path = Path(path) + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_name(path.name + '.tmp_write') + with tmp.open('w', encoding='utf-8-sig', newline='') as f: + writer = csv.DictWriter(f, fieldnames=fieldnames) + writer.writeheader() + for row in rows: + writer.writerow({field: row.get(field, '') for field in fieldnames}) + os.replace(tmp, path) + + def citekey(p): year = str(p.get('year') or 'nd') first = 'paper' @@ -76,17 +119,43 @@ def init_db(): (ROOT/d).mkdir(parents=True, exist_ok=True) ev = ROOT/'matrices/evidence_matrix.csv' if not ev.exists(): - ev.write_text('paper_id,citekey,claim_type,claim,quote,page,section,confidence,use_in_section,my_comment,verified,source_file,created_at,updated_at\n', encoding='utf-8') + ev.write_text('paper_id,citekey,claim_type,claim,quote,page,section,confidence,use_in_section,my_comment,verified,source_file,created_at,updated_at\n', encoding='utf-8-sig') log('init 실행: DB/폴더/Evidence Matrix 확인 완료') +def find_existing_paper_id(c, p): + """Cross-source dedupe: the same paper arriving with only a DOI (Crossref/OpenAlex) + and only an arXiv id (arXiv) must land on ONE row. Match by normalized DOI, then + arXiv id, then normalized title.""" + doi = normalize_doi(p.get('doi') or '') + arxiv = normalize_arxiv(p.get('arxiv_id') or '') + title = norm_title(p.get('title', '')) + if doi: + row = c.execute("SELECT id FROM papers WHERE doi IS NOT NULL AND doi != '' AND LOWER(REPLACE(REPLACE(doi,'https://doi.org/',''),'http://doi.org/',''))=?", (doi,)).fetchone() + if row: + return row['id'] + if arxiv: + row = c.execute("SELECT id, arxiv_id FROM papers WHERE arxiv_id IS NOT NULL AND arxiv_id != ''").fetchall() + for r in row: + if normalize_arxiv(r['arxiv_id']) == arxiv: + return r['id'] + if title: + row = c.execute('SELECT id FROM papers WHERE title_norm=?', (title,)).fetchone() + if row: + return row['id'] + return None + + def upsert(p): - p['id'] = p.get('id') or stable_id(p); p['citekey'] = p.get('citekey') or citekey(p) c = conn() + p['id'] = p.get('id') or find_existing_paper_id(c, p) or stable_id(p) + p['citekey'] = p.get('citekey') or citekey(p) c.execute('''INSERT INTO papers(id,title,authors_json,year,venue,doi,arxiv_id,abstract,url,pdf_url,source,collection_date,status,score,topic_relevance,citation_count,open_access,local_pdf_path,parsed_text_path,citekey,title_norm,raw_json,updated_at) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) - ON CONFLICT(id) DO UPDATE SET title=excluded.title, authors_json=excluded.authors_json, year=excluded.year, venue=excluded.venue, - abstract=excluded.abstract, url=COALESCE(excluded.url,papers.url), pdf_url=COALESCE(excluded.pdf_url,papers.pdf_url), citation_count=MAX(papers.citation_count, excluded.citation_count), open_access=MAX(papers.open_access, excluded.open_access), raw_json=excluded.raw_json, updated_at=excluded.updated_at''', + ON CONFLICT(id) DO UPDATE SET title=excluded.title, authors_json=excluded.authors_json, year=COALESCE(excluded.year,papers.year), venue=COALESCE(excluded.venue,papers.venue), + doi=COALESCE(NULLIF(excluded.doi,''),papers.doi), arxiv_id=COALESCE(NULLIF(excluded.arxiv_id,''),papers.arxiv_id), + abstract=CASE WHEN LENGTH(COALESCE(excluded.abstract,'')) > LENGTH(COALESCE(papers.abstract,'')) THEN excluded.abstract ELSE papers.abstract END, + url=COALESCE(excluded.url,papers.url), pdf_url=COALESCE(excluded.pdf_url,papers.pdf_url), citation_count=MAX(papers.citation_count, excluded.citation_count), open_access=MAX(papers.open_access, excluded.open_access), raw_json=excluded.raw_json, updated_at=excluded.updated_at''', (p['id'],p.get('title'),json.dumps(p.get('authors') or [],ensure_ascii=False),p.get('year'),p.get('venue'),p.get('doi'),p.get('arxiv_id'),p.get('abstract'),p.get('url'),p.get('pdf_url'),p.get('source'),p.get('collection_date') or now(),'new',p.get('score',0),p.get('topic_relevance',0),p.get('citation_count',0),1 if p.get('open_access') else 0,p.get('local_pdf_path'),p.get('parsed_text_path'),p['citekey'],norm_title(p.get('title','')),json.dumps(p,ensure_ascii=False),now())) c.commit(); c.close(); return p['id'] @@ -94,17 +163,88 @@ def upsert(p): def queries(): prof = load_yaml(ROOT/'config/topic_profile.yaml') qs = [x.get('query') for x in prof.get('query_groups',[]) if isinstance(x,dict) and x.get('query')] - return qs or ['ontology knowledge graph semantic web', 'automated literature review research assistant'] + return qs or ['"ontology" AND "knowledge graph"', '"literature review" AND "research automation"'] + + +def sources_config(): + cfg = load_yaml(ROOT/'config/sources.yaml') + cfg = cfg if isinstance(cfg, dict) else {} + sources = cfg.get('sources') if isinstance(cfg.get('sources'), dict) else {} + rate = cfg.get('rate_limits') if isinstance(cfg.get('rate_limits'), dict) else {} + mailto = str(os.environ.get('CROSSREF_MAILTO') or cfg.get('openalex_mailto') or '').strip() + if 'your-email' in mailto.lower(): + mailto = '' + s2_key = str(os.environ.get('SEMANTIC_SCHOLAR_API_KEY') or cfg.get('semantic_scholar_api_key') or '').strip() + user_agent = str(cfg.get('user_agent') or '').strip() or ('PaperOps/0.2 (mailto:%s)' % mailto if mailto else 'PaperOps/0.2') + + def source_opts(name, default_sleep): + opts = sources.get(name) if isinstance(sources.get(name), dict) else {} + return { + 'enabled': bool(opts.get('enabled', True)), + 'max_results': int(opts.get('max_results_per_query') or 0) or None, + 'sleep': float(opts.get('sleep_seconds') or rate.get('sleep_seconds_between_requests') or default_sleep), + } + return { + 'mailto': mailto, + 's2_key': s2_key, + 'user_agent': user_agent, + 'arxiv': source_opts('arxiv', 3.0), + 'openalex': source_opts('openalex', 1.0), + 'crossref': source_opts('crossref', 1.0), + 'semantic_scholar': source_opts('semantic_scholar', 1.5), + } -def fetch_json(url, headers=None): - req = urllib.request.Request(url, headers=headers or {'User-Agent':'PaperOps/0.1'}) - with urllib.request.urlopen(req, timeout=30) as r: return json.loads(r.read().decode('utf-8','ignore')) +def fetch_url(url, headers=None, retries=3, timeout=30): + """GET with polite retry/backoff. Honors Retry-After on 429/503.""" + last_error = None + for attempt in range(retries): + req = urllib.request.Request(url, headers=headers or {'User-Agent': 'PaperOps/0.2'}) + try: + with urllib.request.urlopen(req, timeout=timeout) as r: + return r.read() + except urllib.error.HTTPError as e: + last_error = e + if e.code in (429, 500, 502, 503) and attempt < retries - 1: + retry_after = e.headers.get('Retry-After') if e.headers else None + try: + delay = min(60.0, float(retry_after)) if retry_after else 2.0 * (2 ** attempt) + except ValueError: + delay = 2.0 * (2 ** attempt) + time.sleep(delay) + continue + raise + except urllib.error.URLError as e: + last_error = e + if attempt < retries - 1: + time.sleep(1.5 * (2 ** attempt)) + continue + raise + raise last_error -def collect_arxiv(q, limit): - url='https://export.arxiv.org/api/query?search_query=all:'+urllib.parse.quote(q)+'&start=0&max_results='+str(limit)+'&sortBy=submittedDate&sortOrder=descending' - data=urllib.request.urlopen(url, timeout=30).read() +def fetch_json(url, headers=None): + return json.loads(fetch_url(url, headers=headers).decode('utf-8', 'ignore')) + + +def arxiv_query_expr(q): + """Build a valid arXiv boolean query. Explicit syntax (AND/OR/field prefixes, quotes) + passes through; a bare keyword soup becomes an AND of the first terms so results + actually match the topic instead of arXiv returning loosely-related items.""" + q = str(q or '').strip() + if re.search(r'\b(AND|OR|ANDNOT)\b', q) or re.search(r'\b(all|ti|abs|cat|au):', q): + return re.sub(r'"([^"]+)"', lambda m: 'all:"%s"' % m.group(1), q) if ('"' in q and 'all:' not in q and 'ti:' not in q and 'abs:' not in q) else q + terms = [t for t in re.split(r'\s+', q) if t] + terms = terms[:6] + return ' AND '.join('all:"%s"' % t for t in terms) if terms else 'all:"research"' + + +def collect_arxiv(q, limit, cfg=None): + cfg = cfg or sources_config() + expr = arxiv_query_expr(q) + url = ('https://export.arxiv.org/api/query?search_query=' + urllib.parse.quote(expr) + + '&start=0&max_results=' + str(limit) + '&sortBy=relevance&sortOrder=descending') + data = fetch_url(url, headers={'User-Agent': cfg['user_agent']}) root=ET.fromstring(data); ns={'a':'http://www.w3.org/2005/Atom'}; out=[] for e in root.findall('a:entry',ns): title=' '.join((e.findtext('a:title','',ns) or '').split()) @@ -116,21 +256,49 @@ def collect_arxiv(q, limit): return out -def collect_openalex(q, limit): - url='https://api.openalex.org/works?search='+urllib.parse.quote(q)+'&per-page='+str(limit) - js=fetch_json(url); out=[] +def openalex_abstract(w): + inv = w.get('abstract_inverted_index') + if not isinstance(inv, dict) or not inv: + return '' + positions = {} + for word, indexes in inv.items(): + for index in indexes or []: + positions[index] = word + return ' '.join(positions[i] for i in sorted(positions))[:4000] + + +def collect_openalex(q, limit, cfg=None): + cfg = cfg or sources_config() + url = 'https://api.openalex.org/works?search=' + urllib.parse.quote(q) + '&per-page=' + str(limit) + if cfg['mailto']: + url += '&mailto=' + urllib.parse.quote(cfg['mailto']) + js = fetch_json(url, headers={'User-Agent': cfg['user_agent']}); out=[] for w in js.get('results',[]): authors=[a.get('author',{}).get('display_name','') for a in w.get('authorships',[])] loc=w.get('primary_location') or {}; src=loc.get('source') or {}; oa=w.get('open_access') or {} - out.append(dict(title=w.get('title'), authors=authors, year=w.get('publication_year'), venue=src.get('display_name'), doi=(w.get('doi') or '').replace('https://doi.org/',''), arxiv_id=None, abstract='', url=w.get('id'), pdf_url=loc.get('pdf_url') or oa.get('oa_url'), source='openalex', citation_count=w.get('cited_by_count') or 0, open_access=bool(oa.get('is_oa')))) + ids = w.get('ids') or {} + arxiv_url = str(ids.get('arxiv') or '') + arxiv_id = arxiv_url.rsplit('/', 1)[-1] if 'arxiv.org' in arxiv_url else None + out.append(dict(title=w.get('title'), authors=authors, year=w.get('publication_year'), venue=src.get('display_name'), doi=(w.get('doi') or '').replace('https://doi.org/',''), arxiv_id=arxiv_id, abstract=openalex_abstract(w), url=w.get('id'), pdf_url=loc.get('pdf_url') or oa.get('oa_url'), source='openalex', citation_count=w.get('cited_by_count') or 0, open_access=bool(oa.get('is_oa')))) return out -def collect_crossref(q, limit): - url='https://api.crossref.org/works?query='+urllib.parse.quote(q)+'&rows='+str(limit) - js=fetch_json(url); out=[] +CROSSREF_EXCLUDED_TYPES = {'peer-review', 'component', 'grant', 'report-component'} + + +def collect_crossref(q, limit, cfg=None): + cfg = cfg or sources_config() + url = ('https://api.crossref.org/works?query=' + urllib.parse.quote(q) + '&rows=' + str(limit) + + '&filter=' + urllib.parse.quote('type:journal-article,type:proceedings-article,type:book-chapter,type:posted-content')) + if cfg['mailto']: + url += '&mailto=' + urllib.parse.quote(cfg['mailto']) + js = fetch_json(url, headers={'User-Agent': cfg['user_agent']}); out=[] for w in js.get('message',{}).get('items',[]): + if str(w.get('type') or '') in CROSSREF_EXCLUDED_TYPES: + continue title=(w.get('title') or [''])[0] + if re.match(r'^(decision letter|review(er)? report|author response)\b', title.strip(), re.I): + continue authors=[(' '.join([a.get('given',''),a.get('family','')]).strip()) for a in w.get('author',[])] year=None try: year=w.get('published-print',w.get('published-online',{})).get('date-parts',[[None]])[0][0] @@ -139,17 +307,66 @@ def collect_crossref(q, limit): return out +def collect_semantic_scholar(q, limit, cfg=None): + cfg = cfg or sources_config() + fields = 'title,abstract,year,venue,authors,url,openAccessPdf,citationCount,externalIds' + url = ('https://api.semanticscholar.org/graph/v1/paper/search?query=' + urllib.parse.quote(q) + + '&limit=' + str(limit) + '&fields=' + fields) + headers = {'User-Agent': cfg['user_agent']} + if cfg['s2_key']: + headers['x-api-key'] = cfg['s2_key'] + js = fetch_json(url, headers=headers); out=[] + for w in js.get('data', []) or []: + ext = w.get('externalIds') or {} + oa = w.get('openAccessPdf') or {} + out.append(dict(title=w.get('title'), authors=[a.get('name','') for a in w.get('authors') or []], year=w.get('year'), venue=w.get('venue'), doi=ext.get('DOI'), arxiv_id=ext.get('ArXiv'), abstract=w.get('abstract') or '', url=w.get('url'), pdf_url=oa.get('url'), source='semantic_scholar', citation_count=w.get('citationCount') or 0, open_access=bool(oa.get('url')))) + return out + + +COLLECTORS = [ + ('arxiv', collect_arxiv), + ('semantic_scholar', collect_semantic_scholar), + ('openalex', collect_openalex), + ('crossref', collect_crossref), +] + + def cmd_collect(args): init_db(); allp=[]; raw=ROOT/f"data/incoming/collection_{datetime.now().strftime('%Y%m%d_%H%M%S')}.jsonl" + cfg = sources_config() + if not cfg['mailto']: + print('WARN: no mailto configured (config/sources.yaml openalex_mailto or CROSSREF_MAILTO env); OpenAlex/Crossref rate limits will be much stricter') + failed = defaultdict(int) + consecutive = defaultdict(int) + disabled = set() for q in queries(): - for name,fn in [('arxiv',collect_arxiv),('openalex',collect_openalex),('crossref',collect_crossref)]: + for name, fn in COLLECTORS: + opts = cfg[name] + if not opts['enabled'] or name in disabled: + continue + limit = min(args.limit, opts['max_results']) if opts['max_results'] else args.limit try: - ps=fn(q,args.limit); allp.extend(ps); time.sleep(0.5) + ps = fn(q, limit, cfg); allp.extend(ps) + consecutive[name] = 0 print(name, q, len(ps)) - except Exception as e: print('WARN', name, e) + except Exception as e: + failed[name] += 1 + consecutive[name] += 1 + print('WARN', name, e) + if consecutive[name] >= 2: + # Circuit breaker: a source that keeps failing (usually 429 without an + # API key/mailto) would otherwise burn minutes of backoff per query. + disabled.add(name) + print(f'WARN {name}: {consecutive[name]} consecutive failures; skipping this source for the rest of this run ' + f'(configure semantic_scholar_api_key / openalex_mailto in config/sources.yaml)') + time.sleep(opts['sleep']) with raw.open('w',encoding='utf-8') as f: for p in allp: p['collection_date']=now(); upsert(p); f.write(json.dumps(p,ensure_ascii=False)+'\n') + if failed: + print('WARN sources with failures:', dict(failed)) + if not allp: + print('WARN: collected 0 papers; check network, query syntax, and per-source warnings above') log(f'collect 실행: {len(allp)}건 수집, raw={raw.name}') @@ -229,8 +446,14 @@ def cmd_download(args): dest=ROOT/'data/pdfs'/f"{r['id']}_{slug(r['title'],50)}.pdf" if dest.exists() and not args.overwrite: continue try: - req=urllib.request.Request(r['pdf_url'], headers={'User-Agent':'PaperOps/0.1'}) - with urllib.request.urlopen(req, timeout=60) as resp: data=resp.read() + req=urllib.request.Request(r['pdf_url'], headers={'User-Agent':'PaperOps/0.2'}) + with urllib.request.urlopen(req, timeout=60) as resp: + content_type = str(resp.headers.get('Content-Type') or '').lower() + data=resp.read() + # An HTML paywall/landing page saved as .pdf poisons GROBID/PyMuPDF later. + if not data.startswith(b'%PDF-') and 'pdf' not in content_type: + print('WARN pdf', r['id'], f'not a PDF (content-type={content_type or "unknown"}); skipped') + continue if len(data)>1000: dest.write_bytes(data); c.execute('UPDATE papers SET local_pdf_path=?, updated_at=? WHERE id=?',(str(dest.relative_to(ROOT)),now(),r['id'])); done+=1 except Exception as e: print('WARN pdf', r['id'], e) @@ -338,17 +561,19 @@ def yaml_frontmatter(data): def cmd_extract(args): - ev=ROOT/'matrices/evidence_matrix.csv'; init_db(); rows=top_rows(args.limit); existing=ev.read_text(encoding='utf-8') if ev.exists() else '' - with ev.open('a',encoding='utf-8',newline='') as f: - w=csv.writer(f); n=0 - for r in rows: - if r['id'] in existing: continue - quote, page, section = evidence_quote_candidate(r) - claim=(quote or r['abstract'] or r['title'] or '')[:300].replace('\n',' ') - confidence='medium' if quote and page else 'low' - comment='파싱 본문에서 자동 추출한 후보 문장: 원문 검증 필요' if quote else '자동 초벌 행: 사람이 검증 필요' - w.writerow([r['id'],r['citekey'],'background',claim,quote,page,section,confidence,'related_work',comment,'false',r['parsed_text_path'] or '',datetime.now().isoformat(),datetime.now().isoformat()]); n+=1 - log(f'extract-evidence 실행: 초벌 evidence {n}행 추가') + """DEPRECATED. The legacy extractor appended 14 positional columns straight into the + guarded Evidence Matrix, breaking the migrated 19+-column schema (rows without + evidence_id/exact_quote bypassed all provenance machinery) and used a substring + dedupe against the raw file text. The guarded pipeline replaces it.""" + raise SystemExit( + 'ERROR: extract-evidence is deprecated because it corrupts the guarded Evidence Matrix schema.\n' + 'Use the guarded pipeline instead:\n' + ' 1. parse-grobid --paper-id --apply\n' + ' 2. extract-evidence-candidates --paper-id --apply (heuristic)\n' + ' or extract-evidence-llm --paper-id --apply (LLM, verbatim-checked)\n' + ' 3. review-evidence-candidates --paper-id (human review)\n' + ' 4. promotion-plan --dry-run && promote-evidence --apply' + ) def evidence_quote_candidate(row): @@ -389,11 +614,7 @@ def cmd_outline(args): def read_evidence_rows(): - ev = ROOT/'matrices/evidence_matrix.csv' - if not ev.exists(): - return [] - with ev.open(encoding='utf-8', newline='') as f: - return list(csv.DictReader(f)) + return read_csv_dict(ROOT/'matrices/evidence_matrix.csv')[0] def axis_tags_for(row): @@ -410,10 +631,12 @@ def axis_tags_for(row): return matched or ['unclassified'] -def screen_decision(row, axes): +def screen_decision(row, axes, watch_threshold=None): + if watch_threshold is None: + watch_threshold = score_config()['thresholds']['candidate'] if row['status'] in ('important', 'to_read'): return 'keep' - if row['score'] >= 0.35 and any(a in axes for a in ('ontology_core', 'graphrag_llm', 'research_agent', 'scholarly_kg')): + if row['score'] >= watch_threshold and any(a in axes for a in ('ontology_core', 'graphrag_llm', 'research_agent', 'scholarly_kg')): return 'watch' return 'defer' @@ -423,12 +646,13 @@ def cmd_screen(args): rows = top_rows(args.limit) out = ROOT/'matrices/screening_matrix.csv' out.parent.mkdir(parents=True, exist_ok=True) - with out.open('w', encoding='utf-8', newline='') as f: + watch_threshold = score_config()['thresholds']['candidate'] + with out.open('w', encoding='utf-8-sig', newline='') as f: w = csv.writer(f) w.writerow(['paper_id','citekey','title','year','venue','score','status','axes','decision','reason','url','pdf_url']) for r in rows: axes = axis_tags_for(r) - decision = screen_decision(r, axes) + decision = screen_decision(r, axes, watch_threshold) reason = f"score={r['score']:.3f}; axes={','.join(axes)}; status={r['status']}" w.writerow([r['id'], r['citekey'], r['title'], r['year'], r['venue'], f"{r['score']:.3f}", r['status'], ';'.join(axes), decision, reason, r['url'], r['pdf_url']]) log(f'screen 실행: {out}, {len(rows)}건') @@ -449,7 +673,7 @@ def cmd_gap(args): for axis in axes: groups[axis].append(r) csv_out = ROOT/'matrices/gap_matrix.csv' - with csv_out.open('w', encoding='utf-8', newline='') as f: + with csv_out.open('w', encoding='utf-8-sig', newline='') as f: w = csv.writer(f) w.writerow(['axis','paper_count','top_citekeys','evidence_verified_count','gap_hypothesis','next_action']) for axis, items in sorted(groups.items()): @@ -578,11 +802,14 @@ def cmd_brief(args): def cmd_audit(args): - man=ROOT/'manuscript/main.md'; ev=ROOT/'matrices/evidence_matrix.csv'; out=ROOT/f'reports/audit_reports/audit_{today()}.md' - text=man.read_text(encoding='utf-8',errors='ignore') if man.exists() else '' - cites=set(re.findall(r'@([A-Za-z0-9_:-]+)', text)) - evtext=ev.read_text(encoding='utf-8',errors='ignore') if ev.exists() else '' - missing=[c for c in sorted(cites) if c not in evtext] + # Uses the email-safe citekey pattern (manuscript_citekeys) over BOTH manuscript + # roots and compares against the actual citekey column — the old raw-text substring + # check counted author@example.com as a cite and kim2020ontology as covering kim2020. + out=ROOT/f'reports/audit_reports/audit_{today()}.md' + cites=set(manuscript_citekeys()) + evidence_keys={str(r.get('citekey') or '').strip() for r in read_evidence_rows() if str(r.get('citekey') or '').strip()} + missing=[c for c in sorted(cites) if c not in evidence_keys] + out.parent.mkdir(parents=True, exist_ok=True) out.write_text('# Citation Audit\n\n' + f'- manuscript cites: {len(cites)}\n- missing in evidence matrix: {len(missing)}\n\n' + '\n'.join(f'- {m}' for m in missing), encoding='utf-8') log(f'audit 실행: {out}') print(out) @@ -597,7 +824,7 @@ def file_row_count(path): if not path.exists(): return 0 try: - with path.open(encoding='utf-8', newline='') as f: + with path.open(encoding='utf-8-sig', newline='') as f: return max(0, sum(1 for _ in csv.reader(f)) - 1) except Exception: return 0 @@ -609,10 +836,11 @@ def status_line(name, ok, detail): def command_version(command): - try: - result = subprocess.run([command, '--version'], cwd=str(ROOT), capture_output=True, text=True, timeout=10) - except FileNotFoundError: + resolved = shutil.which(command) # Windows-safe: finds quarto.cmd etc. via PATHEXT + if not resolved: return False, 'not installed or not on PATH' + try: + result = subprocess.run([resolved, '--version'], cwd=str(ROOT), capture_output=True, encoding='utf-8', errors='replace', timeout=10) except Exception as e: return False, str(e) text = (result.stdout or result.stderr or '').strip().splitlines() @@ -735,12 +963,14 @@ def cmd_doctor(args): s2_cfg = sources.get('semantic_scholar_api_key') if isinstance(sources, dict) else '' api_checks = { - 'OPENAI_API_KEY': configured_secret('OPENAI_API_KEY'), + 'OPENAI_API_KEY (extract-evidence-llm)': configured_secret('OPENAI_API_KEY'), 'SEMANTIC_SCHOLAR_API_KEY': configured_secret('SEMANTIC_SCHOLAR_API_KEY', s2_cfg), - 'CROSSREF_MAILTO': configured_secret('CROSSREF_MAILTO', sources.get('openalex_mailto') if isinstance(sources, dict) else ''), + 'mailto (openalex_mailto / CROSSREF_MAILTO)': bool(sources_config()['mailto']), } for key, ok in api_checks.items(): lines.append(status_line(key, ok, 'present' if ok else 'missing')) + if not sources_config()['mailto']: + lines.append(status_line('polite pool', False, 'set openalex_mailto in config/sources.yaml — without it OpenAlex/Crossref throttle aggressively (HTTP 429)')) print('\n'.join(lines)) log('doctor 실행: 환경 점검 완료') @@ -757,6 +987,8 @@ def list_grobid_pdf_targets(limit): except Exception: rows = [] seen = set() + if limit <= 0: + return targets for r in rows: pdf = ROOT/r['local_pdf_path'] if pdf.exists() and pdf.suffix.lower() == '.pdf': @@ -1014,7 +1246,8 @@ def call_grobid_process_fulltext(grobid_url, pdf): resp = requests.post(endpoint, files=files, data=data, timeout=timeout) if resp.status_code < 200 or resp.status_code >= 300: raise RuntimeError(f'GROBID HTTP {resp.status_code}: {resp.text[:500]}') - text = resp.text.strip() + # TEI is UTF-8; requests' charset guessing for application/xml can mangle diacritics. + text = resp.content.decode('utf-8', errors='replace').strip() if not text: raise RuntimeError('GROBID returned empty TEI XML') return text @@ -1284,6 +1517,9 @@ def build_evidence_candidates(paper_id): artifact_dir = ROOT/'data/parsed/grobid'/slug(paper_id, 80) sections_path = artifact_dir/'sections.json' contexts_path = artifact_dir/'citation_contexts.json' + if not sections_path.exists() and not contexts_path.exists(): + # Silent 0-candidate success here used to mask a missing/misnamed artifact dir. + raise RuntimeError(f'no GROBID artifacts for paper_id={paper_id} at {path_for_report(artifact_dir)}; run parse-grobid --paper-id {paper_id} --apply first') errors = [] sections_data = read_json_file(sections_path, errors) if sections_path.exists() else {} contexts_data = read_json_file(contexts_path, errors) if contexts_path.exists() else {} @@ -1376,10 +1612,9 @@ def evidence_matrix_metrics(): if not path.exists(): return metrics metrics['sha256'] = file_sha256(path) - with path.open(encoding='utf-8', newline='') as f: - reader = csv.DictReader(f) - metrics['schema'] = list(reader.fieldnames or []) - for row in reader: + rows, fieldnames = read_csv_dict(path) + metrics['schema'] = fieldnames + for row in rows: metrics['row_count'] += 1 evidence_id = str(row.get('evidence_id') or '').strip() if evidence_id: @@ -1437,11 +1672,7 @@ def row_has_source_location(row): def review_queue_rows_and_header(path): - if not path.exists(): - return [], [] - with path.open(encoding='utf-8', newline='') as f: - reader = csv.DictReader(f) - return list(reader), list(reader.fieldnames or []) + return read_csv_dict(path) def validate_review_queue_data(paper_id=None, write_report=True): @@ -1616,21 +1847,16 @@ def cmd_validate_review_queue(args): def read_evidence_candidates(path): - if not path.exists(): - return [] - with path.open(encoding='utf-8', newline='') as f: - return list(csv.DictReader(f)) + return read_csv_dict(path)[0] def write_evidence_candidates(path, rows): - path.parent.mkdir(parents=True, exist_ok=True) - with path.open('w', encoding='utf-8', newline='') as f: - writer = csv.DictWriter(f, fieldnames=EVIDENCE_CANDIDATE_FIELDS) - writer.writeheader() - for row in rows: - row = {field: row.get(field, '') for field in EVIDENCE_CANDIDATE_FIELDS} - row['verified'] = 'false' - writer.writerow(row) + safe_rows = [] + for row in rows: + row = {field: row.get(field, '') for field in EVIDENCE_CANDIDATE_FIELDS} + row['verified'] = 'false' + safe_rows.append(row) + atomic_write_csv(path, EVIDENCE_CANDIDATE_FIELDS, safe_rows) def distribution_for(rows, field): @@ -1688,10 +1914,7 @@ def validate_evidence_candidate_data(paper_id=None, write_report=True): errors.append(f'missing candidate file: {path_for_report(candidate_path)}') else: try: - with candidate_path.open(encoding='utf-8', newline='') as f: - reader = csv.DictReader(f) - fieldnames = list(reader.fieldnames or []) - all_rows = list(reader) + all_rows, fieldnames = read_csv_dict(candidate_path) except Exception as e: errors.append(f'failed to read candidate file: {e}') missing_headers = [h for h in EVIDENCE_CANDIDATE_REQUIRED_HEADERS if h not in fieldnames] @@ -1803,19 +2026,11 @@ def cmd_validate_evidence_candidates(args): def read_evidence_candidate_review_queue(path): - if not path.exists(): - return [] - with path.open(encoding='utf-8', newline='') as f: - return list(csv.DictReader(f)) + return read_csv_dict(path)[0] def write_evidence_candidate_review_queue(path, rows): - path.parent.mkdir(parents=True, exist_ok=True) - with path.open('w', encoding='utf-8', newline='') as f: - writer = csv.DictWriter(f, fieldnames=EVIDENCE_CANDIDATE_REVIEW_FIELDS) - writer.writeheader() - for row in rows: - writer.writerow({field: row.get(field, '') for field in EVIDENCE_CANDIDATE_REVIEW_FIELDS}) + atomic_write_csv(path, EVIDENCE_CANDIDATE_REVIEW_FIELDS, rows) def preserved_review_values(existing_rows, warnings): @@ -1921,14 +2136,21 @@ def cmd_review_evidence_candidates(args): existing_rows = read_evidence_candidate_review_queue(queue_path) preserved = preserved_review_values(existing_rows, warnings) queue_rows = [review_row_from_candidate(r, preserved) for r in rows] - duplicate_ids = sorted(k for k, v in count_values([r.get('candidate_id') for r in queue_rows]).items() if k and v > 1) + # MERGE, never overwrite: rows belonging to OTHER papers stay in the queue untouched. + # Regenerating paper B's queue must not delete paper A's pending human decisions. + other_rows = [r for r in existing_rows if str(r.get('paper_id') or '') != str(args.paper_id or '')] + preserved_other_count = len(other_rows) + merged_rows = other_rows + queue_rows + duplicate_ids = sorted(k for k, v in count_values([r.get('candidate_id') for r in merged_rows]).items() if k and v > 1) if duplicate_ids: errors.append(f'duplicate candidate_id values in review queue: {len(duplicate_ids)}') verified_false_count = sum(1 for r in queue_rows if str(r.get('verified') or '').strip().lower() == 'false') if verified_false_count != len(queue_rows): errors.append(f'review queue verified must remain false: {verified_false_count}/{len(queue_rows)}') if not errors: - write_evidence_candidate_review_queue(queue_path, queue_rows) + write_evidence_candidate_review_queue(queue_path, merged_rows) + if preserved_other_count: + warnings.append(f'preserved {preserved_other_count} review queue rows belonging to other papers') after = file_row_count(ROOT/'matrices/evidence_matrix.csv') if before != after: errors.append('evidence_matrix.csv row count changed during review queue generation') @@ -2124,7 +2346,10 @@ def source_location_for_patch(row): if page: return page quote_hash = hashlib.sha256(str(row.get('quote') or '').encode('utf-8')).hexdigest()[:12] - return f"section_id={row.get('section_id') or ''}; section_heading={row.get('section_heading') or ''}; quote_sha256={quote_hash}" + # ';' is the source_location field delimiter — a heading containing ';' would corrupt + # parse_source_location, so it is sanitized to ',' on write. + heading = str(row.get('section_heading') or '').replace(';', ',') + return f"section_id={row.get('section_id') or ''}; section_heading={heading}; quote_sha256={quote_hash}" def write_promotion_plan_report(result): @@ -2228,11 +2453,11 @@ def cmd_promotion_plan(args): out_candidates = ROOT/'matrices/evidence_promotion_candidates.csv' out_patch = ROOT/'matrices/evidence_matrix_patch_preview.csv' out_candidates.parent.mkdir(parents=True, exist_ok=True) - with out_candidates.open('w', encoding='utf-8', newline='') as f: + with out_candidates.open('w', encoding='utf-8-sig', newline='') as f: writer = csv.DictWriter(f, fieldnames=PROMOTION_CANDIDATE_FIELDS) writer.writeheader() writer.writerows(promotion_rows) - with out_patch.open('w', encoding='utf-8', newline='') as f: + with out_patch.open('w', encoding='utf-8-sig', newline='') as f: writer = csv.DictWriter(f, fieldnames=EVIDENCE_PATCH_PREVIEW_FIELDS) writer.writeheader() writer.writerows(patch_rows) @@ -2282,18 +2507,11 @@ def resolve_root_path(path_text): def read_csv_rows_with_header(path): - if not path.exists(): - return [], [] - with path.open(encoding='utf-8', newline='') as f: - reader = csv.DictReader(f) - return list(reader), list(reader.fieldnames or []) + return read_csv_dict(path) def read_evidence_matrix_with_header(): - path = ROOT/'matrices/evidence_matrix.csv' - with path.open(encoding='utf-8', newline='') as f: - reader = csv.DictReader(f) - return list(reader), list(reader.fieldnames or []) + return read_csv_dict(ROOT/'matrices/evidence_matrix.csv') def evidence_key_from_row(row): @@ -2594,11 +2812,15 @@ def cmd_promote_evidence(args): applied = False all_appended_verified_false = True + ready_only = bool(getattr(args, 'ready_only', False)) if args.apply: if validation['selected_rows'] == 0: errors.append('no patch preview rows selected for apply') if validation['blocked_count']: - errors.append(f'blocked candidates present: {validation["blocked_count"]}') + if ready_only: + warnings.append(f'--ready-only: proceeding with {validation["ready_count"]} ready rows; {validation["blocked_count"]} blocked rows reported but not applied') + else: + errors.append(f'blocked candidates present: {validation["blocked_count"]} (use --ready-only to apply the ready rows anyway)') if not errors and validation['ready_count']: ev = ROOT/'matrices/evidence_matrix.csv' matrix_rows, fieldnames = read_evidence_matrix_with_header() @@ -2619,11 +2841,7 @@ def cmd_promote_evidence(args): temp_row['verified'] = 'false' new_rows.append(temp_row) appended_candidate_ids.append(preview.get('candidate_id', '')) - with ev.open('w', encoding='utf-8', newline='') as f: - writer = csv.DictWriter(f, fieldnames=fieldnames) - writer.writeheader() - writer.writerows(matrix_rows) - writer.writerows(new_rows) + atomic_write_csv(ev, fieldnames, matrix_rows + new_rows) applied = True all_appended_verified_false = all(str(r.get('verified') or '').strip().lower() == 'false' for r in new_rows) if not all_appended_verified_false: @@ -2756,9 +2974,10 @@ def source_artifact_text_for_row(row, errors): return '', f'unsupported_source_file:{source_file or "(blank)"}' -def promoted_row_qa(row): +def promoted_row_qa(row, ledger=None): warnings = [] errors = [] + ledger = ledger if ledger is not None else verification_ledger_index() cid = row.get('_candidate_id') or candidate_id_from_risk_note(row.get('risk_note')) quote = str(row.get('quote') or '').strip() exact_quote = str(row.get('exact_quote') or '').strip() @@ -2767,12 +2986,22 @@ def promoted_row_qa(row): actual_quote_sha = hashlib.sha256(quote.encode('utf-8')).hexdigest()[:12] if quote else '' if not cid: errors.append('missing_candidate_id_trace') - if str(row.get('verified') or '').strip().lower() != 'false': - errors.append('verified_not_false') - if str(row.get('verified_by') or '').strip(): - errors.append('verified_by_must_be_blank') - if str(row.get('verified_at') or '').strip(): - errors.append('verified_at_must_be_blank') + is_verified = str(row.get('verified') or '').strip().lower() == 'true' + ledger_entry = ledger.get(str(row.get('evidence_id') or '').strip()) + attested = bool( + is_verified and ledger_entry + and str(ledger_entry.get('action') or '') == 'verify' + and str(ledger_entry.get('verified_by') or '').strip() == str(row.get('verified_by') or '').strip() + ) + if is_verified and not attested: + errors.append('verified_true_without_human_attestation') + if not is_verified: + if str(row.get('verified') or '').strip().lower() != 'false': + errors.append('verified_not_false') + if str(row.get('verified_by') or '').strip(): + errors.append('verified_by_must_be_blank') + if str(row.get('verified_at') or '').strip(): + errors.append('verified_at_must_be_blank') if str(row.get('extraction_method') or '').strip() != 'candidate_review_guarded_apply': errors.append('unexpected_extraction_method') if not quote: @@ -2787,7 +3016,7 @@ def promoted_row_qa(row): errors.append('quote_sha256_mismatch') if not str(row.get('page') or '').strip(): warnings.append('page_blank_pdf_check_required') - if any(token in quote for token in ['460,497', '13,431', '92.7%', 'Orphadata', 'biomedical', 'clinical']): + if any(token in quote for token in qa_profile()['corpus_specific_tokens']): warnings.append('domain_specific_claim_use_cautiously') artifact_text, artifact_error = source_artifact_text_for_row(row, errors) quote_in_artifact = bool(quote and artifact_text and quote in artifact_text) @@ -2855,7 +3084,8 @@ def write_promoted_evidence_qa_report(result): def cmd_audit_promoted_evidence(args): matrix = evidence_matrix_metrics() rows = promoted_evidence_rows(getattr(args, 'paper_id', None), getattr(args, 'candidate_id', None)) - qa_rows = [promoted_row_qa(row) for row in rows] + ledger = verification_ledger_index() + qa_rows = [promoted_row_qa(row, ledger) for row in rows] candidate_ids = [row.get('candidate_id') for row in qa_rows] duplicate_ids = sorted(k for k, v in count_values(candidate_ids).items() if k and v > 1) warnings = [] @@ -2956,7 +3186,7 @@ def cmd_extract_promoted_rows(args): out_path = resolve_root_path(args.output) if getattr(args, 'output', None) else ROOT/f'reports/review/promoted_rows_external_review_input_{today()}.csv' out_path.parent.mkdir(parents=True, exist_ok=True) out_rows = [promoted_review_input_row(row) for row in rows] - with out_path.open('w', encoding='utf-8', newline='') as f: + with out_path.open('w', encoding='utf-8-sig', newline='') as f: writer = csv.DictWriter(f, fieldnames=PROMOTED_REVIEW_INPUT_FIELDS) writer.writeheader() writer.writerows(out_rows) @@ -2991,7 +3221,8 @@ def write_pdf_page_check_report(result): lines.append('|---|---|---|---|---|---|\n') for row in result.get('page_blank_rows', []): quote = (row.get('exact_quote') or row.get('quote') or '').replace('|', '\\|')[:160] - lines.append(f"| `{row.get('_candidate_id')}` | `{row.get('evidence_id')}` | `{row.get('citekey')}` | {row.get('claim_type')} | {str(row.get('section') or '').replace('|', '\\|')} | {quote} |\n") + section_cell = str(row.get('section') or '').replace('|', '\\|') # kept out of the f-string: backslashes inside f-string expressions are a SyntaxError before Python 3.12 + lines.append(f"| `{row.get('_candidate_id')}` | `{row.get('evidence_id')}` | `{row.get('citekey')}` | {row.get('claim_type')} | {section_cell} | {quote} |\n") if not result.get('page_blank_rows'): lines.append('| none | | | | | |\n') lines.append('\n## Warnings\n') @@ -3029,12 +3260,32 @@ def cmd_mark_pdf_check_required(args): log(f"mark-pdf-check-required dry-run 실행: promoted_only={result['promoted_only']}, page_blank={len(page_blank_rows)}") -DOMAIN_SPECIFIC_KEYWORDS = ['biomedical', 'clinical', 'clinician', 'patient', 'patients', 'disease', 'diseases', 'medical', 'medicine', 'orphanet', 'orphadata', 'pmid', 'drug', 'phenotyping', 'biology', 'regulatory', 'onset age', 'clinical milestone'] +_QA_PROFILE_CACHE = None + + +def qa_profile(): + """Corpus/domain-specific heuristics live in config/qa_profile.yaml, NOT in code. + The shipped config carries the original author's biomedical-corpus lists; other + users edit the yaml for their own domain. Falls back to minimal neutral defaults.""" + global _QA_PROFILE_CACHE + if _QA_PROFILE_CACHE is None: + cfg = load_yaml(ROOT/'config/qa_profile.yaml') + cfg = cfg if isinstance(cfg, dict) else {} + + def str_list(key, default): + value = cfg.get(key) + return [str(v) for v in value] if isinstance(value, list) else default + _QA_PROFILE_CACHE = { + 'domain_specific_keywords': str_list('domain_specific_keywords', []), + 'overclaim_keywords': str_list('overclaim_keywords', ['accuracy', 'state-of-the-art', 'validated']), + 'corpus_specific_tokens': str_list('corpus_specific_tokens', []), + } + return _QA_PROFILE_CACHE def domain_specific_hits(row): text = ' '.join([row.get('claim') or '', row.get('quote') or '', row.get('exact_quote') or '', row.get('section') or '']).lower() - return sorted({kw for kw in DOMAIN_SPECIFIC_KEYWORDS if kw in text}) + return sorted({kw for kw in qa_profile()['domain_specific_keywords'] if kw in text}) def write_domain_specific_claim_audit_report(result): @@ -3101,11 +3352,96 @@ def git_status_paths(pathspecs=None): return [], f'git status unavailable: {e}' +VERIFICATION_LEDGER_FIELDS = ['evidence_id', 'candidate_id', 'action', 'verified_by', 'verified_at', 'method', 'note', 'quote_sha256'] +VERIFICATION_LEDGER_PATH = 'matrices/verification_ledger.csv' + + +def read_verification_ledger(): + return read_csv_dict(ROOT/VERIFICATION_LEDGER_PATH)[0] + + +def verification_ledger_index(): + """Latest ledger action per evidence_id. A row is attested when its latest action + is 'verify' and verified_by/verified_at are recorded.""" + index = {} + for entry in read_verification_ledger(): + evidence_id = str(entry.get('evidence_id') or '').strip() + if evidence_id: + index[evidence_id] = entry + return index + + +def append_verification_ledger(entry): + path = ROOT/VERIFICATION_LEDGER_PATH + rows = read_verification_ledger() + rows.append(entry) + atomic_write_csv(path, VERIFICATION_LEDGER_FIELDS, rows) + + +def cmd_verify_evidence(args): + """THE human gate. This is the only sanctioned writer of verified=true, and it + requires an explicit human attestation (--by NAME --attest). Every change is + recorded in the verification ledger so guard-no-auto-verified can distinguish + human-attested rows from anything automation might have written.""" + if not args.attest and not args.revoke: + raise SystemExit('ERROR: verify-evidence requires --attest, confirming YOU checked the original source (quote, page, meaning) yourself') + if not str(args.by or '').strip(): + raise SystemExit('ERROR: --by is required; the ledger records who verified') + ev = ROOT/'matrices/evidence_matrix.csv' + if not ev.exists(): + raise SystemExit(f'ERROR: missing Evidence Matrix: {ev}') + matrix_rows, fieldnames = read_evidence_matrix_with_header() + for required in ['evidence_id', 'verified', 'verified_by', 'verified_at']: + if required not in fieldnames: + raise SystemExit(f'ERROR: Evidence Matrix missing column `{required}`; run migrate-evidence first') + target = None + for row in matrix_rows: + if str(row.get('evidence_id') or '').strip() == args.evidence_id: + target = row + break + if target is None: + raise SystemExit(f'ERROR: evidence_id not found: {args.evidence_id}') + stamp = now() + backup_dir = ROOT/'matrices/backups' + backup_dir.mkdir(parents=True, exist_ok=True) + backup = backup_dir/f'evidence_matrix_before_verify_{datetime.now().strftime("%Y%m%d_%H%M%S")}.csv' + shutil.copy2(ev, backup) + quote = str(target.get('exact_quote') or target.get('quote') or '') + quote_sha = hashlib.sha256(quote.encode('utf-8')).hexdigest()[:12] if quote else '' + if args.revoke: + target['verified'] = 'false' + target['verified_by'] = '' + target['verified_at'] = '' + action = 'revoke' + else: + if str(target.get('verified') or '').strip().lower() == 'true': + raise SystemExit(f'ERROR: evidence_id already verified: {args.evidence_id} (use --revoke first to re-verify)') + target['verified'] = 'true' + target['verified_by'] = args.by.strip() + target['verified_at'] = stamp + action = 'verify' + atomic_write_csv(ev, fieldnames, matrix_rows) + append_verification_ledger({ + 'evidence_id': args.evidence_id, + 'candidate_id': candidate_id_from_risk_note(target.get('risk_note')), + 'action': action, + 'verified_by': args.by.strip(), + 'verified_at': stamp if action == 'verify' else '', + 'method': args.method, + 'note': args.note or '', + 'quote_sha256': quote_sha, + }) + print(f'{action}: evidence_id={args.evidence_id} by={args.by.strip()}') + print(f'ledger={ROOT/VERIFICATION_LEDGER_PATH}') + print(f'backup={backup}') + log(f'verify-evidence {action}: evidence_id={args.evidence_id}, by={args.by.strip()}') + + def write_no_auto_verified_guard_report(result): out = ROOT/f'reports/audit_reports/no_auto_verified_guard_{today()}.md' out.parent.mkdir(parents=True, exist_ok=True) lines = [f'# No-auto-verified Guard {today()}\n', '## Summary\n'] - for key in ['promoted_only', 'paper_id_filter', 'row_count', 'verified_true_count', 'verified_by_filled_count', 'verified_at_filled_count', 'manuscript_changed_count', 'allow_approved_manuscript_changes', 'manuscript_apply_report', 'paperqa_langgraph_warning_count', 'passed']: + for key in ['promoted_only', 'paper_id_filter', 'row_count', 'verified_true_count', 'attested_verified_count', 'unattested_verified_count', 'verified_by_filled_count', 'verified_at_filled_count', 'manuscript_changed_count', 'allow_approved_manuscript_changes', 'manuscript_apply_report', 'paperqa_langgraph_warning_count', 'passed']: lines.append(f'- {key}: {result.get(key)}\n') lines.append('\n## Manuscript Changes\n') lines.extend([f"- `{line}`\n" for line in result.get('manuscript_changes', [])] or ['- none\n']) @@ -3125,32 +3461,70 @@ def approved_manuscript_apply_report_errors(report_arg): report_path = resolve_root_path(report_arg) if not report_path.exists(): return [f'manuscript apply report not found: {path_for_report(report_path)}'] - text = report_path.read_text(encoding='utf-8') + text = read_text_compat(report_path) + errors = [] checks = [ ('- mode: apply', 'manuscript apply report must have mode=apply'), ('- applied: true', 'manuscript apply report must have applied=true'), ('- blocked_rows: 0', 'manuscript apply report must have blocked_rows=0'), ('## Errors\n- none', 'manuscript apply report must have no errors'), ] - return [message for needle, message in checks if needle not in text] + errors.extend(message for needle, message in checks if needle not in text) + # Anti-forgery cross-check: the report's recorded post-apply SHA256 values must match + # the manuscript files on disk RIGHT NOW. A fabricated approval report that was not + # produced by an actual guarded apply of the current manuscript state fails here. + target_pairs = re.findall(r'###\s+`([^`]+)`\n(.*?)(?=\n###\s+`|\n## |\Z)', text, re.S) + sha_checked = 0 + for target_file, block in target_pairs: + match = re.search(r'-\s*file_sha256_after:\s*([0-9a-f]{64})', block) + if not match: + continue + recorded_sha = match.group(1) + target_path = ROOT/target_file + if not target_path.exists(): + errors.append(f'apply report references missing manuscript file: {target_file}') + continue + if file_sha256(target_path) != recorded_sha: + errors.append(f'apply report sha256 does not match current manuscript file: {target_file} (report is stale or fabricated)') + sha_checked += 1 + if not sha_checked: + errors.append('apply report contains no verifiable file_sha256_after entries; cannot cross-check against manuscript state') + return errors def cmd_guard_no_auto_verified(args): rows = promoted_evidence_rows(getattr(args, 'paper_id', None), getattr(args, 'candidate_id', None)) if getattr(args, 'promoted_only', False) else read_evidence_for_audit() errors = [] warnings = [] + ledger = verification_ledger_index() verified_true = [r for r in rows if str(r.get('verified') or '').strip().lower() == 'true'] - verified_by_filled = [r for r in rows if str(r.get('verified_by') or '').strip()] - verified_at_filled = [r for r in rows if str(r.get('verified_at') or '').strip()] - if verified_true: - errors.append(f'promoted/evidence rows with verified=true: {len(verified_true)}') - if verified_by_filled: - errors.append(f'promoted/evidence rows with verified_by filled: {len(verified_by_filled)}') - if verified_at_filled: - errors.append(f'promoted/evidence rows with verified_at filled: {len(verified_at_filled)}') + attested = [] + unattested = [] + for r in verified_true: + evidence_id = str(r.get('evidence_id') or '').strip() + entry = ledger.get(evidence_id) + ok = ( + entry is not None + and str(entry.get('action') or '').strip() == 'verify' + and str(entry.get('verified_by') or '').strip() + and str(entry.get('verified_by') or '').strip() == str(r.get('verified_by') or '').strip() + and str(entry.get('verified_at') or '').strip() == str(r.get('verified_at') or '').strip() + ) + (attested if ok else unattested).append(evidence_id or '(no evidence_id)') + if unattested: + errors.append(f'verified=true rows WITHOUT a matching human attestation in {VERIFICATION_LEDGER_PATH}: {len(unattested)}') + warnings.extend([f'unattested verified row: `{evidence_id}` (use verify-evidence, never manual/automated edits)' for evidence_id in unattested[:50]]) + inconsistent_by = [r for r in rows if str(r.get('verified_by') or '').strip() and str(r.get('verified') or '').strip().lower() != 'true'] + inconsistent_at = [r for r in rows if str(r.get('verified_at') or '').strip() and str(r.get('verified') or '').strip().lower() != 'true'] + if inconsistent_by: + errors.append(f'rows with verified_by filled but verified!=true: {len(inconsistent_by)}') + if inconsistent_at: + errors.append(f'rows with verified_at filled but verified!=true: {len(inconsistent_at)}') manuscript_changes, git_warn = git_status_paths(['05_manuscript', 'manuscript']) if git_warn: warnings.append(git_warn) + warnings.append('manuscript_change_detection: UNAVAILABLE (no usable git repo) — guard cannot see manuscript edits; use a git checkout for full protection') + print('WARN: manuscript change detection unavailable (no git); guard coverage is reduced') allow_manuscript_changes = bool(getattr(args, 'allow_approved_manuscript_changes', False)) manuscript_apply_report = getattr(args, 'manuscript_apply_report', None) or '' if manuscript_changes: @@ -3173,8 +3547,10 @@ def cmd_guard_no_auto_verified(args): 'paper_id_filter': getattr(args, 'paper_id', None) or '(none)', 'row_count': len(rows), 'verified_true_count': len(verified_true), - 'verified_by_filled_count': len(verified_by_filled), - 'verified_at_filled_count': len(verified_at_filled), + 'attested_verified_count': len(attested), + 'unattested_verified_count': len(unattested), + 'verified_by_filled_count': len(inconsistent_by), + 'verified_at_filled_count': len(inconsistent_at), 'manuscript_changed_count': len(manuscript_changes), 'allow_approved_manuscript_changes': str(allow_manuscript_changes).lower(), 'manuscript_apply_report': manuscript_apply_report or '(none)', @@ -3187,111 +3563,49 @@ def cmd_guard_no_auto_verified(args): } report = write_no_auto_verified_guard_report(result) print(report) - for key in ['row_count', 'verified_true_count', 'verified_by_filled_count', 'verified_at_filled_count', 'manuscript_changed_count', 'paperqa_langgraph_warning_count', 'passed']: + for key in ['row_count', 'verified_true_count', 'attested_verified_count', 'unattested_verified_count', 'verified_by_filled_count', 'verified_at_filled_count', 'manuscript_changed_count', 'paperqa_langgraph_warning_count', 'passed']: print(f"{key}={result.get(key)}") log(f"guard-no-auto-verified 실행: promoted_only={result['promoted_only']}, passed={result['passed']}, errors={len(errors)}") if errors: raise SystemExit(1) -PROMOTED_ROW_EXTERNAL_REVIEW_DECISIONS = { - 'fb5137d2707f5e1d': { - 'external_review_decision': 'downgrade_to_pdf_check', - 'suggested_claim_type': 'evaluation_pattern', - 'suggested_use_in_section': 'ch2_related_work;ch3_evaluation_design_motivation', - 'rewrite_if_needed': 'ChronoMedKG 사례는 LLM judge가 아닌 gold-standard comparison을 평가 설계에 활용한 도메인 특화 사례를 보여준다.', - 'pdf_check_priority': 'high', - 'paperops_generalization_allowed': 'false', - 'pdf_page_check_required': 'true', - 'domain_specific_risk': 'high', - }, - '23fe8ed59649fffa': { - 'external_review_decision': 'revise', - 'suggested_claim_type': 'method_pattern', - 'suggested_use_in_section': 'ch2_related_work;ch3_design_motivation', - 'rewrite_if_needed': '도메인 특화 KG 연구에서 disease-autonomous multi-agent pipeline이 대규모 biomedical triples 생성에 활용된 사례가 있다.', - 'pdf_check_priority': 'high', - 'paperops_generalization_allowed': 'false', - 'pdf_page_check_required': 'true', - 'domain_specific_risk': 'high', - }, - 'd5f10096ff325472': { - 'external_review_decision': 'revise', - 'suggested_claim_type': 'method_pattern', - 'suggested_use_in_section': 'ch2_related_work;ch3_design_motivation', - 'rewrite_if_needed': 'ChronoMedKG는 각 질병 단위를 독립적으로 처리하는 multi-stage pipeline 구조를 채택했다. 이는 도메인 단위 batch/agent pipeline 설계 사례로 참고할 수 있다.', - 'pdf_check_priority': 'medium', - 'paperops_generalization_allowed': 'false', - 'pdf_page_check_required': 'true', - 'domain_specific_risk': 'medium', - }, - 'e340369a59fb2532': { - 'external_review_decision': 'revise', - 'suggested_claim_type': 'agent_workflow_pattern', - 'suggested_use_in_section': 'ch2_related_work;ch3_system_design_motivation', - 'rewrite_if_needed': '특정 biomedical KG 구축 사례에서는 disease identifier를 입력으로 네 개의 협력 agent가 end-to-end pipeline을 수행하도록 설계했다.', - 'pdf_check_priority': 'medium', - 'paperops_generalization_allowed': 'false', - 'pdf_page_check_required': 'true', - 'domain_specific_risk': 'medium', - }, - '06488bdee4bc0074': { - 'external_review_decision': 'keep', - 'suggested_claim_type': 'governance', - 'suggested_use_in_section': 'ch3_system_design;ch5_evaluation_design', - 'rewrite_if_needed': '검증 harness, judge-panel code, error taxonomy를 공개하는 방식은 연구 자동화 시스템의 auditability와 reproducibility를 높이는 설계 패턴으로 볼 수 있다.', - 'pdf_check_priority': 'medium', - 'paperops_generalization_allowed': 'limited_auditability_design_principle_only', - 'pdf_page_check_required': 'true', - 'domain_specific_risk': 'low', - }, - '154a4607cb42751c': { - 'external_review_decision': 'revise', - 'suggested_claim_type': 'provenance_pattern', - 'suggested_use_in_section': 'ch2_related_work;ch3_evidence_model_design', - 'rewrite_if_needed': 'ChronoMedKG는 triple 단위에 evidence grading과 PMID provenance를 부여하는 방식으로 출처 추적성을 강화한 사례다.', - 'pdf_check_priority': 'high', - 'paperops_generalization_allowed': 'false', - 'pdf_page_check_required': 'true', - 'domain_specific_risk': 'high', - }, - '6abed7fd8c1bd310': { - 'external_review_decision': 'revise', - 'suggested_claim_type': 'human_oversight', - 'suggested_use_in_section': 'ch2_related_work;ch3_governance_design;ch6_limitations', - 'rewrite_if_needed': '도메인 특화 자동화 시스템도 원천 데이터 범위와 실제 적용 범위를 구분하며, 고위험 도메인 적용에는 인간 전문가 검토와 별도 평가가 필요하다는 제한을 명시한다.', - 'pdf_check_priority': 'medium', - 'paperops_generalization_allowed': 'false', - 'pdf_page_check_required': 'true', - 'domain_specific_risk': 'high', - }, - '67558897e8bfbc54': { - 'external_review_decision': 'revise', - 'suggested_claim_type': 'validation_boundary', - 'suggested_use_in_section': 'ch3_evaluation_design;ch5_evaluation_limitations;ch6_limitations', - 'rewrite_if_needed': 'text-grounding 검증은 원문 근거 일치 여부를 확인하는 절차이지, 도메인 사실 자체의 독립적 재검증은 아니라는 한계를 명확히 해야 한다.', - 'pdf_check_priority': 'medium', - 'paperops_generalization_allowed': 'limited_validation_boundary_principle_only', - 'pdf_page_check_required': 'true', - 'domain_specific_risk': 'medium', - }, -} +_REVIEW_OVERRIDES_CACHE = None + + +def review_overrides(): + """Per-candidate external review decisions, externalized to config/review_overrides.yaml + (they are author-corpus-specific human decisions, not program logic). Unknown + candidate_ids get the conservative default_decision from the same file.""" + global _REVIEW_OVERRIDES_CACHE + if _REVIEW_OVERRIDES_CACHE is None: + cfg = load_yaml(ROOT/'config/review_overrides.yaml') + cfg = cfg if isinstance(cfg, dict) else {} + decisions = cfg.get('decisions') if isinstance(cfg.get('decisions'), dict) else {} + default = cfg.get('default_decision') if isinstance(cfg.get('default_decision'), dict) else {} + _REVIEW_OVERRIDES_CACHE = { + 'source_review': str(cfg.get('source_review') or 'row-level review overrides (config/review_overrides.yaml)'), + 'decisions': {str(k): dict(v) for k, v in decisions.items() if isinstance(v, dict)}, + 'default_decision': { + 'external_review_decision': str(default.get('external_review_decision') or 'downgrade_to_pdf_check'), + 'suggested_claim_type': str(default.get('suggested_claim_type') or ''), + 'suggested_use_in_section': str(default.get('suggested_use_in_section') or ''), + 'rewrite_if_needed': str(default.get('rewrite_if_needed') or ''), + 'pdf_check_priority': str(default.get('pdf_check_priority') or 'high'), + 'paperops_generalization_allowed': str(default.get('paperops_generalization_allowed') or 'false'), + 'pdf_page_check_required': str(default.get('pdf_page_check_required') or 'true'), + 'domain_specific_risk': str(default.get('domain_specific_risk') or 'unknown'), + }, + } + return _REVIEW_OVERRIDES_CACHE PROMOTED_ROW_REVIEW_DECISION_FIELDS = PROMOTED_REVIEW_INPUT_FIELDS + ['external_review_decision', 'suggested_claim_type', 'suggested_use_in_section', 'rewrite_if_needed', 'pdf_check_priority', 'paperops_generalization_allowed', 'pdf_page_check_required', 'domain_specific_risk'] EVIDENCE_METADATA_PATCH_PREVIEW_FIELDS = ['candidate_id', 'evidence_id', 'paper_id', 'citekey', 'current_claim_type', 'suggested_claim_type', 'current_use_in_section', 'suggested_use_in_section', 'current_claim', 'rewrite_if_needed', 'external_review_decision', 'pdf_check_priority', 'paperops_generalization_allowed', 'pdf_page_check_required', 'domain_specific_risk', 'verified_should_remain', 'patch_action', 'source_review'] def row_level_decision_for(candidate_id): - return PROMOTED_ROW_EXTERNAL_REVIEW_DECISIONS.get(candidate_id, { - 'external_review_decision': 'downgrade_to_pdf_check', - 'suggested_claim_type': '', - 'suggested_use_in_section': '', - 'rewrite_if_needed': '', - 'pdf_check_priority': 'high', - 'paperops_generalization_allowed': 'false', - 'pdf_page_check_required': 'true', - 'domain_specific_risk': 'unknown', - }) + overrides = review_overrides() + return overrides['decisions'].get(candidate_id, dict(overrides['default_decision'])) def write_row_level_review_pm_report(result): @@ -3318,9 +3632,7 @@ def cmd_update_promoted_row_review_metadata(args): input_path = resolve_root_path(args.input) if not input_path.exists(): raise SystemExit(f'ERROR: missing input CSV: {input_path}') - with input_path.open(encoding='utf-8', newline='') as f: - reader = csv.DictReader(f) - input_rows = list(reader) + input_rows, _ = read_csv_dict(input_path) decision_rows = [] patch_rows = [] for row in input_rows: @@ -3347,16 +3659,16 @@ def cmd_update_promoted_row_review_metadata(args): 'domain_specific_risk': decision.get('domain_specific_risk', ''), 'verified_should_remain': 'false', 'patch_action': 'metadata_preview_only_no_evidence_matrix_write', - 'source_review': 'GPT Pro row-level review captured 2026-06-04', + 'source_review': review_overrides()['source_review'], }) decision_csv = resolve_root_path(args.output) if getattr(args, 'output', None) else ROOT/f'reports/review/promoted_rows_row_level_review_decisions_{today()}.csv' patch_csv = resolve_root_path(args.patch_preview) if getattr(args, 'patch_preview', None) else ROOT/f'matrices/evidence_matrix_metadata_patch_preview_{today()}.csv' decision_csv.parent.mkdir(parents=True, exist_ok=True) patch_csv.parent.mkdir(parents=True, exist_ok=True) - with decision_csv.open('w', encoding='utf-8', newline='') as f: + with decision_csv.open('w', encoding='utf-8-sig', newline='') as f: writer = csv.DictWriter(f, fieldnames=PROMOTED_ROW_REVIEW_DECISION_FIELDS) writer.writeheader(); writer.writerows(decision_rows) - with patch_csv.open('w', encoding='utf-8', newline='') as f: + with patch_csv.open('w', encoding='utf-8-sig', newline='') as f: writer = csv.DictWriter(f, fieldnames=EVIDENCE_METADATA_PATCH_PREVIEW_FIELDS) writer.writeheader(); writer.writerows(patch_rows) matrix = evidence_matrix_metrics() @@ -3384,12 +3696,9 @@ def cmd_update_promoted_row_review_metadata(args): log(f"update-promoted-row-review-metadata dry-run 실행: rows={len(decision_rows)}, patch={path_for_report(patch_csv)}") -OVERCLAIM_KEYWORDS = ['accuracy', 'validated', 'consensus triples', 'disease', 'diseases', 'clinical', 'biology', 'biomedical', 'orphadata', 'pmid'] - - def paperops_overclaim_flags(row): text = ' '.join([row.get('claim') or '', row.get('quote') or '', row.get('exact_quote') or '', row.get('section') or '']).lower() - hits = sorted({kw for kw in OVERCLAIM_KEYWORDS if kw in text}) + hits = sorted({kw for kw in qa_profile()['overclaim_keywords'] if kw in text}) flags = [] if hits: flags.append('domain_or_metric_specific_terms') @@ -3474,7 +3783,7 @@ def cmd_pdf_page_verification_sheet(args): 'pdf_verified_at': '', 'pdf_verification_note': '', }) - with out_path.open('w', encoding='utf-8', newline='') as f: + with out_path.open('w', encoding='utf-8-sig', newline='') as f: writer = csv.DictWriter(f, fieldnames=PDF_PAGE_VERIFICATION_FIELDS) writer.writeheader(); writer.writerows(out_rows) report = ROOT/f'reports/review/pdf_page_verification_sheet_{today()}.md' @@ -3614,7 +3923,7 @@ def cmd_locate_pdf_pages(args): 'verified_should_remain': 'false', 'note': note, }) - with out_path.open('w', encoding='utf-8', newline='') as f: + with out_path.open('w', encoding='utf-8-sig', newline='') as f: writer = csv.DictWriter(f, fieldnames=PDF_PAGE_LOCATOR_FIELDS) writer.writeheader(); writer.writerows(out_rows) matrix = evidence_matrix_metrics() @@ -3792,9 +4101,7 @@ def cmd_apply_page_metadata(args): row['verified_by'] = '' if 'verified_at' in row: row['verified_at'] = '' - with ev.open('w', encoding='utf-8', newline='') as f: - writer = csv.DictWriter(f, fieldnames=fieldnames) - writer.writeheader(); writer.writerows(matrix_rows) + atomic_write_csv(ev, fieldnames, matrix_rows) applied = True elif args.apply and not errors and not updated_items: warnings.append('no page metadata rows needed updating') @@ -3809,16 +4116,22 @@ def cmd_apply_page_metadata(args): if comparison['schema_changed']: errors.append('Evidence Matrix schema changed during page metadata apply') final_rows = read_evidence_for_audit() + ledger = verification_ledger_index() promoted_final = [r for r in final_rows if candidate_id_from_risk_note(r.get('risk_note'))] - verified_true_count = sum(1 for r in promoted_final if str(r.get('verified') or '').strip().lower() == 'true') - verified_by_filled_count = sum(1 for r in promoted_final if str(r.get('verified_by') or '').strip()) - verified_at_filled_count = sum(1 for r in promoted_final if str(r.get('verified_at') or '').strip()) + unattested_after = [ + r for r in promoted_final + if str(r.get('verified') or '').strip().lower() == 'true' + and str(ledger.get(str(r.get('evidence_id') or '').strip(), {}).get('action') or '') != 'verify' + ] + verified_true_count = len(unattested_after) + verified_by_filled_count = sum(1 for r in promoted_final if str(r.get('verified_by') or '').strip() and str(r.get('verified') or '').strip().lower() != 'true') + verified_at_filled_count = sum(1 for r in promoted_final if str(r.get('verified_at') or '').strip() and str(r.get('verified') or '').strip().lower() != 'true') if verified_true_count: - errors.append(f'promoted rows with verified=true after apply: {verified_true_count}') + errors.append(f'promoted rows with UNATTESTED verified=true after apply: {verified_true_count}') if verified_by_filled_count: - errors.append(f'promoted rows with verified_by filled after apply: {verified_by_filled_count}') + errors.append(f'promoted rows with verified_by filled but verified!=true after apply: {verified_by_filled_count}') if verified_at_filled_count: - errors.append(f'promoted rows with verified_at filled after apply: {verified_at_filled_count}') + errors.append(f'promoted rows with verified_at filled but verified!=true after apply: {verified_at_filled_count}') result = { 'mode': 'apply' if args.apply else 'dry-run', 'patch_preview': path_for_report(validation.get('patch_path')) if validation.get('patch_path') else args.from_preview, @@ -4052,7 +4365,7 @@ def cmd_manuscript_patch_preview(args): if target_key in final_patched_sha_by_target: out_row['patched_file_sha256'] = final_patched_sha_by_target[target_key] csv_path.parent.mkdir(parents=True, exist_ok=True) - with csv_path.open('w', encoding='utf-8', newline='') as f: + with csv_path.open('w', encoding='utf-8-sig', newline='') as f: writer = csv.DictWriter(f, fieldnames=MANUSCRIPT_PATCH_PREVIEW_FIELDS) writer.writeheader() writer.writerows(output_rows) @@ -4254,7 +4567,9 @@ def cmd_apply_manuscript_patch(args): applied = False if not errors and args.apply: - backup_root = ROOT/'05_manuscript/backups'/f'manuscript_before_guarded_apply_{datetime.now().strftime("%Y%m%d_%H%M%S")}' + # Backups live OUTSIDE the guarded 05_manuscript tree: backups inside it kept + # tripping guard-no-auto-verified (untracked changes) and double-counting citekeys. + backup_root = ROOT/'backups/manuscript'/f'manuscript_before_guarded_apply_{datetime.now().strftime("%Y%m%d_%H%M%S")}' for path, state in file_state.items(): rel = path.relative_to(ROOT/'05_manuscript') backup_path = backup_root/rel @@ -4302,7 +4617,7 @@ def cmd_apply_manuscript_patch(args): }) csv_path.parent.mkdir(parents=True, exist_ok=True) - with csv_path.open('w', encoding='utf-8', newline='') as f: + with csv_path.open('w', encoding='utf-8-sig', newline='') as f: writer = csv.DictWriter(f, fieldnames=MANUSCRIPT_APPLY_FIELDS) writer.writeheader() writer.writerows(output_rows) @@ -4413,6 +4728,312 @@ def cmd_extract_evidence_candidates(args): raise SystemExit(1) +# -------------------------------------------------------------------------------------- +# LLM-assisted evidence extraction (guarded) +# +# Design (same governance as the heuristic extractor — the LLM gets NO extra trust): +# 1. The LLM only PROPOSES candidates (claim + exact_quote) from GROBID-parsed sections. +# 2. Every proposed exact_quote is verbatim-checked against the parsed section text +# (whitespace-normalized). Quotes the model made up are DROPPED and reported. +# 3. Survivors enter the same candidates CSV with verified='false' and flow through the +# SAME human review -> promotion-plan -> guarded-apply pipeline. Nothing is skipped. +# -------------------------------------------------------------------------------------- + +def llm_config(): + pipeline = load_yaml(ROOT/'config/pipeline.yaml') + llm = pipeline.get('llm') if isinstance(pipeline, dict) and isinstance(pipeline.get('llm'), dict) else {} + return { + 'base_url': str(os.environ.get('OPENAI_BASE_URL') or llm.get('base_url') or 'https://api.openai.com/v1').rstrip('/'), + 'model': str(llm.get('model') or 'gpt-4o-mini'), + 'api_key': str(os.environ.get('OPENAI_API_KEY') or '').strip(), + 'max_sections': int(llm.get('max_sections') or 12), + 'max_candidates_per_section': int(llm.get('max_candidates_per_section') or 5), + 'timeout': int(llm.get('timeout_seconds') or 120), + } + + +def llm_extractor_prompt(): + prompt_path = ROOT/'config/prompts/evidence_extractor.md' + base = '' + if prompt_path.exists(): + base = read_text_compat(prompt_path).strip() + schema_instruction = ( + 'Return ONLY a JSON array. Each item: {"claim": , ' + '"exact_quote": , ' + '"claim_type": one of ["method","finding","limitation","definition","background"]}. ' + 'The exact_quote MUST be copied exactly from the provided text — never paraphrase, never translate, ' + 'never merge sentences. Items whose exact_quote is not found verbatim in the source are discarded.' + ) + return (base + '\n\n' + schema_instruction).strip() + + +def llm_chat_completion(cfg, system_prompt, user_prompt): + body = json.dumps({ + 'model': cfg['model'], + 'messages': [ + {'role': 'system', 'content': system_prompt}, + {'role': 'user', 'content': user_prompt}, + ], + 'temperature': 0, + }).encode('utf-8') + req = urllib.request.Request( + cfg['base_url'] + '/chat/completions', + data=body, + headers={ + 'Content-Type': 'application/json', + 'Authorization': 'Bearer ' + cfg['api_key'], + 'User-Agent': 'PaperOps/0.2', + }, + method='POST', + ) + last_error = None + for attempt in range(3): + try: + with urllib.request.urlopen(req, timeout=cfg['timeout']) as resp: + payload = json.loads(resp.read().decode('utf-8', 'replace')) + return payload['choices'][0]['message']['content'] + except urllib.error.HTTPError as e: + last_error = e + if e.code in (429, 500, 502, 503) and attempt < 2: + time.sleep(3.0 * (2 ** attempt)) + continue + raise + except (KeyError, IndexError, json.JSONDecodeError) as e: + raise RuntimeError(f'unexpected LLM response shape: {e}') + raise last_error + + +def parse_llm_candidate_json(text): + """Parse the model output into a list of dicts. Tolerates code fences and stray prose + around the JSON array; anything unparseable returns [].""" + text = str(text or '').strip() + fence = re.search(r'```(?:json)?\s*(\[.*?\])\s*```', text, re.S) + if fence: + text = fence.group(1) + else: + start = text.find('[') + end = text.rfind(']') + if start != -1 and end > start: + text = text[start:end + 1] + try: + data = json.loads(text) + except json.JSONDecodeError: + return [] + if not isinstance(data, list): + return [] + out = [] + for item in data: + if isinstance(item, dict) and str(item.get('exact_quote') or '').strip(): + out.append(item) + return out + + +def normalize_for_verbatim(text): + return ' '.join(str(text or '').split()) + + +def quote_is_verbatim(quote, section_text): + """Anti-hallucination gate: the quote must appear in the section text after + whitespace normalization only. No fuzzy matching — a paraphrase is a fail.""" + q = normalize_for_verbatim(quote) + return bool(q) and q in normalize_for_verbatim(section_text) + + +def write_llm_extraction_report(result): + out = ROOT/f'reports/audit_reports/llm_evidence_extraction_{today()}.md' + out.parent.mkdir(parents=True, exist_ok=True) + lines = [f'# LLM Evidence Extraction {today()}\n', '## Summary\n'] + for key in ['paper_id', 'citekey', 'mode', 'model', 'sections_sent', 'proposed_count', 'verbatim_ok_count', 'dropped_non_verbatim_count', 'dropped_invalid_count', 'merged_count', 'evidence_matrix_row_count_before', 'evidence_matrix_row_count_after']: + lines.append(f'- {key}: {result.get(key)}\n') + lines.append('\n## Dropped (non-verbatim — model paraphrased or invented these)\n') + lines.extend([f'- {q[:200]}\n' for q in result.get('dropped_quotes', [])[:50]] or ['- none\n']) + lines.append('\n## Governance\n') + lines.append('- Every accepted candidate was verbatim-verified against the GROBID-parsed section text.\n') + lines.append('- All candidates enter the review queue with verified=false; the human review, promotion-plan, and guarded-apply gates are unchanged.\n') + lines.append('- The LLM cannot write to the Evidence Matrix, set verified, or bypass any gate.\n') + lines.append('\n## Errors\n') + lines.extend([f'- {e}\n' for e in result.get('errors', [])] or ['- none\n']) + out.write_text(''.join(lines), encoding='utf-8') + return out + + +def cmd_extract_evidence_llm(args): + if args.apply and args.dry_run: + raise SystemExit('ERROR: --apply and --dry-run cannot be used together') + if not args.apply and not args.dry_run: + raise SystemExit('ERROR: choose --dry-run or --apply') + cfg = llm_config() + if not cfg['api_key']: + raise SystemExit('ERROR: OPENAI_API_KEY is not set (an OpenAI-compatible endpoint can be configured via OPENAI_BASE_URL / config/pipeline.yaml llm.base_url)') + artifact_dir = ROOT/'data/parsed/grobid'/slug(args.paper_id, 80) + sections_path = artifact_dir/'sections.json' + if not sections_path.exists(): + raise SystemExit(f'ERROR: missing GROBID sections artifact: {path_for_report(sections_path)}; run parse-grobid --paper-id {args.paper_id} --apply first') + errors = [] + sections_data = read_json_file(sections_path, errors) + if errors: + raise SystemExit('ERROR: ' + '; '.join(errors)) + citekey = sections_data.get('citekey') or '' + sections = [s for s in (sections_data.get('sections') or []) if str(s.get('text') or '').strip()][:cfg['max_sections']] + before = file_row_count(ROOT/'matrices/evidence_matrix.csv') + system_prompt = llm_extractor_prompt() + created = now() + proposed = 0 + verbatim_rows = [] + dropped_quotes = [] + dropped_invalid = 0 + for section in sections: + text = str(section.get('text') or '')[:12000] + user_prompt = ( + f'Paper section (id={section.get("section_id")}, heading="{section.get("heading")}"):\n\n{text}\n\n' + f'Extract up to {cfg["max_candidates_per_section"]} evidence candidates as the JSON array described.' + ) + try: + raw = llm_chat_completion(cfg, system_prompt, user_prompt) + except Exception as e: + errors.append(f'section {section.get("section_id")}: LLM call failed: {e}') + continue + items = parse_llm_candidate_json(raw) + proposed += len(items) + for item in items[:cfg['max_candidates_per_section']]: + quote = str(item.get('exact_quote') or '').strip() + claim = str(item.get('claim') or '').strip() or quote[:300] + claim_type = str(item.get('claim_type') or '').strip() + if claim_type not in EVIDENCE_CANDIDATE_CLAIM_TYPES: + claim_type = 'background' + if not quote_is_verbatim(quote, section.get('text') or ''): + dropped_quotes.append(quote) + continue + if len(normalize_for_verbatim(quote)) < 20: + dropped_invalid += 1 + continue + row = make_candidate({ + 'paper_id': args.paper_id, + 'citekey': citekey, + 'source_artifact': 'sections.json', + 'section_id': section.get('section_id') or '', + 'section_heading': section.get('heading') or '', + 'claim_type': claim_type, + 'claim': claim, + 'quote': normalize_for_verbatim(quote), + 'page': '', + 'confidence': '0.60', + 'reason': f'llm_extractor model={cfg["model"]}; verbatim-checked against sections.json', + 'use_in_section': candidate_type_for(section.get('heading') or '', quote, 'sections.json')[1], + 'created_at': created, + }) + verbatim_rows.append(row) + merged_count = 0 + out_csv = ROOT/'matrices/evidence_candidates.csv' + if args.apply and verbatim_rows: + existing = read_evidence_candidates(out_csv) + existing_by_id = {r.get('candidate_id'): r for r in existing if r.get('candidate_id')} + for row in verbatim_rows: + row['verified'] = 'false' + if row['candidate_id'] not in existing_by_id: + merged_count += 1 + existing_by_id[row['candidate_id']] = row + write_evidence_candidates(out_csv, list(existing_by_id.values())) + after = file_row_count(ROOT/'matrices/evidence_matrix.csv') + if after != before: + errors.append('evidence_matrix.csv row count changed; this command must never touch it') + result = { + 'paper_id': args.paper_id, + 'citekey': citekey, + 'mode': 'apply' if args.apply else 'dry-run', + 'model': cfg['model'], + 'sections_sent': len(sections), + 'proposed_count': proposed, + 'verbatim_ok_count': len(verbatim_rows), + 'dropped_non_verbatim_count': len(dropped_quotes), + 'dropped_invalid_count': dropped_invalid, + 'merged_count': merged_count, + 'evidence_matrix_row_count_before': before, + 'evidence_matrix_row_count_after': after, + 'dropped_quotes': dropped_quotes, + 'errors': errors, + } + report = write_llm_extraction_report(result) + print(report) + for key in ['mode', 'sections_sent', 'proposed_count', 'verbatim_ok_count', 'dropped_non_verbatim_count', 'merged_count']: + print(f'{key}={result.get(key)}') + log(f"extract-evidence-llm {result['mode']} 실행: paper_id={args.paper_id}, verbatim_ok={len(verbatim_rows)}/{proposed}, errors={len(errors)}") + if errors: + raise SystemExit(1) + + +def cmd_make_page_metadata_preview(args): + """Generates the page-metadata patch preview that apply-page-metadata consumes — + previously this strictly-validated file had NO producer in the repo. Prefills only + single-page, high/medium-confidence locator hits; the human edits/removes rows + before running apply-page-metadata.""" + locator_csv = resolve_root_path(args.from_locator) if args.from_locator else ROOT/f'reports/review/pdf_page_locator_candidates_{today()}.csv' + if not locator_csv.exists(): + raise SystemExit(f'ERROR: missing locator CSV: {path_for_report(locator_csv)}; run locate-pdf-pages first') + rows, _ = read_csv_dict(locator_csv) + matrix_by_id = {str(r.get('evidence_id') or '').strip(): r for r in read_evidence_for_audit() if str(r.get('evidence_id') or '').strip()} + out_rows = [] + skipped = 0 + for row in rows: + pages = str(row.get('candidate_pages') or '').strip() + count = int(row.get('candidate_page_count') or 0) + if count != 1 or not pages: + skipped += 1 + continue + evidence_id = str(row.get('evidence_id') or '').strip() + target = matrix_by_id.get(evidence_id) or {} + existing_location = str(target.get('source_location') or '').strip() + proposed_location = f'pdf_page={pages}' + if existing_location and 'pdf_page=' not in existing_location: + proposed_location = f'pdf_page={pages}; {existing_location}' + out_rows.append({ + 'candidate_id': row.get('candidate_id') or '', + 'evidence_id': evidence_id, + 'proposed_page': pages, + 'proposed_source_location': proposed_location, + 'verified_should_remain': 'false', + }) + out_path = resolve_root_path(args.output) if args.output else ROOT/f'matrices/evidence_matrix_page_metadata_patch_preview_{today()}.csv' + atomic_write_csv(out_path, PAGE_METADATA_PATCH_REQUIRED_FIELDS, out_rows) + print(out_path) + print(f'prefilled_rows={len(out_rows)}') + print(f'skipped_rows={skipped} (multi-page or not found; resolve manually)') + print('NEXT: review/edit the CSV by hand, then run: apply-page-metadata --from-preview ' + + path_for_report(out_path) + ' --dry-run') + log(f'make-page-metadata-preview 실행: prefilled={len(out_rows)}, skipped={skipped}') + + +def cmd_make_outline_insertion_template(args): + """Generates a starter manuscript_outline_insertion_preview CSV (the input of + manuscript-patch-preview) — previously undocumented and producer-less. Ships one + example row per promoted evidence row's use_in_section, all flagged as requiring + PM approval, with auto-insert disallowed.""" + out_path = resolve_root_path(args.output) if args.output else ROOT/f'reports/review/manuscript_outline_insertion_preview_{today()}.csv' + promoted = promoted_evidence_rows(getattr(args, 'paper_id', None), None) + out_rows = [] + for index, row in enumerate(promoted[:int(args.limit)], 1): + quote = str(row.get('exact_quote') or row.get('quote') or '').strip() + citekey_value = str(row.get('citekey') or '').strip() + out_rows.append({ + 'preview_id': f'ins_{today().replace("-", "")}_{index:03d}', + 'target_file': '05_manuscript/chapters/ch2_literature.qmd', + 'target_heading': '## Related Work', + 'paragraph_role': 'evidence_supported_paragraph', + 'evidence_ids': str(row.get('evidence_id') or ''), + 'pages': str(row.get('page') or ''), + 'proposed_paragraph': f'[DRAFT — edit before preview] {quote[:300]} [@{citekey_value}]', + 'guard': 'verified remains false until human PDF check; citation must exist in references.bib', + 'manuscript_auto_insert_allowed': 'false', + 'requires_pm_approval': 'true', + }) + atomic_write_csv(out_path, MANUSCRIPT_PREVIEW_REQUIRED_FIELDS, out_rows) + print(out_path) + print(f'template_rows={len(out_rows)}') + print('NEXT: edit target_file/target_heading/proposed_paragraph by hand, then run: ' + 'manuscript-patch-preview --from-preview ' + path_for_report(out_path)) + log(f'make-outline-insertion-template 실행: rows={len(out_rows)}') + + def should_backup_file(path): parts = {p.lower() for p in path.parts} if '.venv' in parts: @@ -4427,7 +5048,8 @@ def should_backup_file(path): def cmd_backup(args): - include_dirs = ['scripts', 'config', 'docs', 'matrices', 'manuscript', 'research_design', 'reports', 'notes'] + # data/metadata carries papers.sqlite — previously the DB was in NO backup at all. + include_dirs = ['scripts', 'config', 'docs', 'matrices', 'manuscript', '05_manuscript', 'research_design', 'reports', 'notes', 'data/metadata', 'tests'] out_dir = ROOT/'backups' out_dir.mkdir(parents=True, exist_ok=True) out = out_dir/f"paperops_backup_{datetime.now().strftime('%Y%m%d_%H%M%S')}.zip" @@ -4527,12 +5149,16 @@ def stable_evidence_id(row, row_index, seen): def normalized_verified(value): + """GOVERNANCE INVARIANT: automation may only ever preserve an exact literal 'true' + that a human already wrote. Truthy variants like '1', 'yes', 'TRUE-from-Excel' + are demoted to 'false' — a migration must never be the step that flips a row to + verified. Demotions are reported by cmd_migrate_evidence.""" text = str(value or '').strip().lower() - if text in ('true', 'false'): - return text - if text in ('1', 'yes', 'y'): - return 'true' - return 'false' + return 'true' if text == 'true' else 'false' + + +def verified_value_is_ambiguous(value): + return str(value or '').strip().lower() in ('1', 'yes', 'y', 't', 'on') def cmd_migrate_evidence(args): @@ -4545,10 +5171,7 @@ def cmd_migrate_evidence(args): backup = backup_dir/f'evidence_matrix_{stamp}.csv' shutil.copy2(ev, backup) - with ev.open(encoding='utf-8', newline='') as f: - reader = csv.DictReader(f) - original_columns = list(reader.fieldnames or []) - rows = list(reader) + rows, original_columns = read_csv_dict(ev) fieldnames = list(original_columns) added_columns = [c for c in EVIDENCE_REQUIRED_COLUMNS if c not in fieldnames] @@ -4558,6 +5181,7 @@ def cmd_migrate_evidence(args): added_columns.append('verified') seen_ids = set() + demoted_truthy_rows = [] for index, row in enumerate(rows, 1): for col in fieldnames: row.setdefault(col, '') @@ -4579,12 +5203,11 @@ def cmd_migrate_evidence(args): row['source_location'] = '; '.join(location_parts) if not row.get('extraction_method'): row['extraction_method'] = 'legacy_migration' + if verified_value_is_ambiguous(row.get('verified')): + demoted_truthy_rows.append(row.get('evidence_id') or f'row_{index}') row['verified'] = normalized_verified(row.get('verified')) - with ev.open('w', encoding='utf-8', newline='') as f: - writer = csv.DictWriter(f, fieldnames=fieldnames) - writer.writeheader() - writer.writerows(rows) + atomic_write_csv(ev, fieldnames, rows) ids = [r.get('evidence_id') for r in rows if r.get('evidence_id')] duplicate_ids = sorted(k for k, v in count_values(ids).items() if v > 1) @@ -4613,9 +5236,12 @@ def cmd_migrate_evidence(args): lines.append(f'- `{col}`\n') lines.append('\n## Rules Applied\n') lines.append('- Existing rows and columns were preserved.\n') - lines.append('- `verified` values were preserved when already true/false; missing or ambiguous values defaulted to false.\n') + lines.append('- `verified` was preserved only when it was exactly the literal `true` or `false`.\n') + lines.append('- Truthy variants (`1`, `yes`, `y`, `t`, `on` — e.g. from Excel) were DEMOTED to `false` and listed below; migration never promotes a row to verified.\n') lines.append('- `exact_quote` was filled from legacy `quote` when available.\n') lines.append('- `evidence_id` was generated from paper_id, citekey, claim_type, claim, and quote/exact_quote.\n') + lines.append(f'\n## Truthy Values Demoted To false ({len(demoted_truthy_rows)})\n') + lines.extend([f'- `{rid}`\n' for rid in demoted_truthy_rows[:100]] or ['- none\n']) if duplicate_ids: lines.append('\n## Duplicate Evidence IDs\n') for evidence_id in duplicate_ids[:100]: @@ -4640,11 +5266,7 @@ def db_citekeys(): def read_evidence_for_audit(): - ev = ROOT/'matrices/evidence_matrix.csv' - if not ev.exists(): - return [] - with ev.open(encoding='utf-8', newline='') as f: - return list(csv.DictReader(f)) + return read_csv_dict(ROOT/'matrices/evidence_matrix.csv')[0] def parse_bib_citekeys(path): @@ -4664,6 +5286,8 @@ def manuscript_citekeys(): for path in root.rglob('*'): if not path.is_file() or path.suffix.lower() not in ('.md', '.qmd', '.tex'): continue + if 'backups' in path.parts: + continue text = path.read_text(encoding='utf-8', errors='ignore') for citekey in pattern.findall(text): found[citekey].add(str(path.relative_to(ROOT))) @@ -4832,13 +5456,29 @@ def parse_bib_entries(path): 'citekey': citekey, 'title': fields.get('title', ''), 'doi': normalize_doi(fields.get('doi', '')), - 'arxiv_id': normalize_arxiv(fields.get('eprint') or fields.get('arxiv') or fields.get('arxivid') or fields.get('archiveprefix', '') if fields.get('archiveprefix', '').lower() == 'arxiv' else fields.get('eprint', '')), + 'arxiv_id': bib_entry_arxiv_id(fields), 'title_norm': norm_title(fields.get('title', '')), 'fields': fields, }) return entries +ARXIV_ID_PATTERN = re.compile(r'^(\d{4}\.\d{4,5}(v\d+)?|[a-z-]+(\.[A-Z]{2})?/\d{7}(v\d+)?)$', re.I) + + +def bib_entry_arxiv_id(fields): + """Extract an arXiv id from BibTeX fields. Fixes the old precedence bug that could + return the literal string 'arxiv' and treated any non-arXiv eprint as an arXiv id.""" + raw = str(fields.get('eprint') or fields.get('arxiv') or fields.get('arxivid') or '').strip() + if not raw: + return '' + archive_prefix = str(fields.get('archiveprefix') or '').strip().lower() + if archive_prefix and archive_prefix != 'arxiv': + return '' + candidate = normalize_arxiv(raw) + return candidate if ARXIV_ID_PATTERN.match(candidate) else '' + + def normalize_doi(value): return (value or '').strip().lower().replace('https://doi.org/', '').replace('http://doi.org/', '') @@ -4881,6 +5521,8 @@ def db_backup_path(): def cmd_sync_zotero(args): + if args.apply and args.dry_run: + raise SystemExit('ERROR: --apply and --dry-run cannot be used together') init_db() bib = ROOT/args.bib legacy_bib = ROOT/'manuscript/references.bib' @@ -4912,14 +5554,37 @@ def cmd_sync_zotero(args): apply_requested = False if apply_requested and duplicate_targets: apply_requested = False + matrix_updated_rows = 0 + matrix_backup = '' if apply_requested and candidates: backup_path = db_backup_path() shutil.copy2(DB, backup_path) backup = str(backup_path) + renames_by_paper_id = {} for entry, row, method in candidates: c.execute('UPDATE papers SET citekey=?, updated_at=? WHERE id=?', (entry['citekey'], now(), row['id'])) + renames_by_paper_id[row['id']] = entry['citekey'] applied += 1 c.commit() + # Propagate citekey renames into the Evidence Matrix so promotion/audit commands + # do not immediately fail with citekey_not_found against the new canonical keys. + ev = ROOT/'matrices/evidence_matrix.csv' + if ev.exists() and renames_by_paper_id: + matrix_rows, matrix_fields = read_evidence_matrix_with_header() + changed = False + for row in matrix_rows: + new_key = renames_by_paper_id.get(str(row.get('paper_id') or '').strip()) + if new_key and str(row.get('citekey') or '') != new_key: + row['citekey'] = new_key + matrix_updated_rows += 1 + changed = True + if changed: + backup_dir = ROOT/'matrices/backups' + backup_dir.mkdir(parents=True, exist_ok=True) + mb = backup_dir/f'evidence_matrix_before_citekey_sync_{datetime.now().strftime("%Y%m%d_%H%M%S")}.csv' + shutil.copy2(ev, mb) + matrix_backup = path_for_report(mb) + atomic_write_csv(ev, matrix_fields, matrix_rows) c.close() report = ROOT/f'reports/audit_reports/zotero_sync_{today()}.md' @@ -4938,9 +5603,12 @@ def cmd_sync_zotero(args): f'- unmatched_db_papers: {len(unmatched_db)}\n', f'- duplicate_target_citekeys: {len(duplicate_targets)}\n', f'- applied_changes: {applied}\n', + f'- evidence_matrix_citekey_rows_updated: {matrix_updated_rows}\n', ] if backup: lines.append(f'- db_backup: `{backup}`\n') + if matrix_backup: + lines.append(f'- evidence_matrix_backup: `{matrix_backup}`\n') if canonical_empty: lines += [ '\n## WARN: Canonical Bib Is Empty\n', @@ -5071,7 +5739,9 @@ def cmd_init_quarto(args): def smoke_run(args): - result = subprocess.run([sys.executable, 'scripts/paperops.py'] + args, cwd=str(ROOT), capture_output=True, text=True, timeout=180) + env = dict(os.environ) + env['PYTHONIOENCODING'] = 'utf-8' + result = subprocess.run([sys.executable, 'scripts/paperops.py'] + args, cwd=str(ROOT), capture_output=True, encoding='utf-8', errors='replace', timeout=180, env=env) return { 'command': 'python scripts/paperops.py ' + ' '.join(args), 'returncode': result.returncode, @@ -5126,12 +5796,13 @@ def main(): p=sub.add_parser('parse-grobid'); p.add_argument('--limit', type=int, default=5); p.add_argument('--paper-id'); p.add_argument('--pdf'); p.add_argument('--apply', action='store_true'); p.add_argument('--dry-run', action='store_true'); p.set_defaults(func=cmd_parse_grobid) p=sub.add_parser('validate-grobid-artifacts'); p.add_argument('--paper-id'); p.add_argument('--path'); p.set_defaults(func=cmd_validate_grobid_artifacts) p=sub.add_parser('extract-evidence-candidates'); p.add_argument('--paper-id', required=True); p.add_argument('--apply', action='store_true'); p.add_argument('--dry-run', action='store_true'); p.set_defaults(func=cmd_extract_evidence_candidates) + p=sub.add_parser('extract-evidence-llm', help='LLM-proposed evidence candidates; every quote verbatim-checked, same human gates'); p.add_argument('--paper-id', required=True); p.add_argument('--apply', action='store_true'); p.add_argument('--dry-run', action='store_true'); p.set_defaults(func=cmd_extract_evidence_llm) p=sub.add_parser('validate-evidence-candidates'); p.add_argument('--paper-id'); p.set_defaults(func=cmd_validate_evidence_candidates) p=sub.add_parser('review-evidence-candidates'); p.add_argument('--paper-id', required=True); p.add_argument('--min-confidence', type=float); p.add_argument('--use-in-section', choices=sorted(EVIDENCE_CANDIDATE_USE_SECTIONS)); p.add_argument('--claim-type', choices=sorted(EVIDENCE_CANDIDATE_CLAIM_TYPES)); p.set_defaults(func=cmd_review_evidence_candidates) p=sub.add_parser('validate-review-queue'); p.add_argument('--paper-id'); p.set_defaults(func=cmd_validate_review_queue) p=sub.add_parser('test-review-preservation'); p.add_argument('--paper-id', required=True); p.set_defaults(func=cmd_test_review_preservation) p=sub.add_parser('promotion-plan'); p.add_argument('--paper-id'); p.add_argument('--candidate-id'); p.add_argument('--dry-run', action='store_true'); p.set_defaults(func=cmd_promotion_plan) - p=sub.add_parser('promote-evidence'); p.add_argument('--from-preview', default='matrices/evidence_matrix_patch_preview.csv'); p.add_argument('--paper-id'); p.add_argument('--candidate-id'); p.add_argument('--dry-run', action='store_true'); p.add_argument('--apply', action='store_true'); p.set_defaults(func=cmd_promote_evidence) + p=sub.add_parser('promote-evidence'); p.add_argument('--from-preview', default='matrices/evidence_matrix_patch_preview.csv'); p.add_argument('--paper-id'); p.add_argument('--candidate-id'); p.add_argument('--dry-run', action='store_true'); p.add_argument('--apply', action='store_true'); p.add_argument('--ready-only', action='store_true', help='apply ready rows even when other selected rows are blocked'); p.set_defaults(func=cmd_promote_evidence) p=sub.add_parser('audit-promoted-evidence'); p.add_argument('--paper-id'); p.add_argument('--candidate-id'); p.set_defaults(func=cmd_audit_promoted_evidence) p=sub.add_parser('extract-promoted-rows'); p.add_argument('--since'); p.add_argument('--output', default=f'reports/review/promoted_rows_external_review_input_{today()}.csv'); p.add_argument('--paper-id'); p.add_argument('--candidate-id'); p.set_defaults(func=cmd_extract_promoted_rows) p=sub.add_parser('mark-pdf-check-required'); p.add_argument('--promoted-only', action='store_true'); p.add_argument('--paper-id'); p.add_argument('--candidate-id'); p.add_argument('--dry-run', action='store_true'); p.set_defaults(func=cmd_mark_pdf_check_required) @@ -5139,9 +5810,12 @@ def main(): p=sub.add_parser('guard-no-auto-verified'); p.add_argument('--promoted-only', action='store_true'); p.add_argument('--paper-id'); p.add_argument('--candidate-id'); p.add_argument('--allow-approved-manuscript-changes', action='store_true'); p.add_argument('--manuscript-apply-report'); p.set_defaults(func=cmd_guard_no_auto_verified) p=sub.add_parser('update-promoted-row-review-metadata'); p.add_argument('--input', default=f'reports/review/promoted_rows_external_review_input_{today()}.csv'); p.add_argument('--output', default=f'reports/review/promoted_rows_row_level_review_decisions_{today()}.csv'); p.add_argument('--patch-preview', default=f'matrices/evidence_matrix_metadata_patch_preview_{today()}.csv'); p.add_argument('--dry-run', action='store_true'); p.set_defaults(func=cmd_update_promoted_row_review_metadata) p=sub.add_parser('guard-paperops-overclaim'); p.add_argument('--promoted-only', action='store_true'); p.add_argument('--paper-id'); p.add_argument('--candidate-id'); p.set_defaults(func=cmd_guard_paperops_overclaim) + p=sub.add_parser('verify-evidence', help='THE human gate: set verified=true for one row, recorded in the verification ledger'); p.add_argument('--evidence-id', required=True); p.add_argument('--by', required=True, help='your name; recorded in the ledger'); p.add_argument('--method', default='human_pdf_check'); p.add_argument('--note', default=''); p.add_argument('--attest', action='store_true', help='confirm you personally checked quote/page/meaning against the original source'); p.add_argument('--revoke', action='store_true', help='revert a row to verified=false'); p.set_defaults(func=cmd_verify_evidence) p=sub.add_parser('pdf-page-verification-sheet'); p.add_argument('--promoted-only', action='store_true'); p.add_argument('--paper-id'); p.add_argument('--candidate-id'); p.add_argument('--output', default=f'reports/review/pdf_page_verification_sheet_{today()}.csv'); p.set_defaults(func=cmd_pdf_page_verification_sheet) p=sub.add_parser('locate-pdf-pages'); p.add_argument('--promoted-only', action='store_true'); p.add_argument('--paper-id'); p.add_argument('--candidate-id'); p.add_argument('--output', default=f'reports/review/pdf_page_locator_candidates_{today()}.csv'); p.set_defaults(func=cmd_locate_pdf_pages) + p=sub.add_parser('make-page-metadata-preview', help='prefill the page metadata patch preview from locate-pdf-pages output'); p.add_argument('--from-locator'); p.add_argument('--output'); p.set_defaults(func=cmd_make_page_metadata_preview) p=sub.add_parser('apply-page-metadata'); p.add_argument('--from-preview', default=f'matrices/evidence_matrix_page_metadata_patch_preview_{today()}.csv'); p.add_argument('--dry-run', action='store_true'); p.add_argument('--apply', action='store_true'); p.set_defaults(func=cmd_apply_page_metadata) + p=sub.add_parser('make-outline-insertion-template', help='generate a starter manuscript insertion preview for manuscript-patch-preview'); p.add_argument('--paper-id'); p.add_argument('--limit', default=10); p.add_argument('--output'); p.set_defaults(func=cmd_make_outline_insertion_template) p=sub.add_parser('manuscript-patch-preview'); p.add_argument('--from-preview', default=f'reports/review/manuscript_outline_insertion_preview_{today()}.csv'); p.add_argument('--output-prefix'); p.set_defaults(func=cmd_manuscript_patch_preview) p=sub.add_parser('apply-manuscript-patch'); p.add_argument('--from-preview', default=f'reports/review/manuscript_patch_preview_{today()}.csv'); p.add_argument('--output-prefix'); p.add_argument('--dry-run', action='store_true'); p.add_argument('--apply', action='store_true'); p.set_defaults(func=cmd_apply_manuscript_patch) sub.add_parser('backup').set_defaults(func=cmd_backup) diff --git a/scripts/paperops_draft_audit.py b/scripts/paperops_draft_audit.py index 92acd96..0168a24 100644 --- a/scripts/paperops_draft_audit.py +++ b/scripts/paperops_draft_audit.py @@ -3,15 +3,25 @@ Audits a thesis draft (docx/txt/md/qmd) for evidence accountability: structure, source mentions, unsupported strong claims, overclaim language, -unverified numeric claims, and Evidence Matrix coverage. +numeric claims cross-checked against experiment output files, and Evidence +Matrix coverage. This is a *flagging* tool, not a truth validator: it tells the author which sentences need sources, verification, or replacement with real results. It never modifies the draft and never marks anything as verified. +Numeric matches against experiment outputs are *alignment*, not truth +validation (see project governance rules). + +Detector defaults (Korean + English) are built in; author-/topic-specific +values (source-name whitelist, chapter structure, extra claim patterns) live +in config/draft_audit.yaml and are merged over neutral built-in defaults. +Works without PyYAML (falls back to the neutral defaults). """ from __future__ import annotations import argparse +import copy import csv +import io import re import zipfile from datetime import datetime @@ -22,6 +32,16 @@ LOG = ROOT / 'logs/ACTIVITY_LOG.md' W_NS = '{http://schemas.openxmlformats.org/wordprocessingml/2006/main}' +DEFAULT_CONFIG_PATH = ROOT / 'config/draft_audit.yaml' +DEFAULT_EXPERIMENT_DATA = 'data/experiment_outputs' +DEFAULT_REPORT_SUBDIR = 'reports/draft_audit' +EXPERIMENT_SUFFIXES = ('.csv', '.tsv', '.json', '.txt', '.md') +# Skip absurdly large experiment files (regex scan would stall); noted in report. +EXPERIMENT_MAX_BYTES = 20 * 1024 * 1024 + +MISSING_DATA_LINE = ('experiment data dir not found — ' + 'numeric values flagged for manual check') + def now(): return datetime.now().strftime('%Y-%m-%d %H:%M:%S') @@ -39,49 +59,133 @@ def log(msg): f.write(f'- [{now()}] {msg}\n') -def extract_docx_paragraphs(path): - with zipfile.ZipFile(path) as z: - xml = z.read('word/document.xml') - root = ET.fromstring(xml) - paras = [] - for p in root.iter(f'{W_NS}p'): - text = ''.join(t.text or '' for t in p.iter(f'{W_NS}t')) - if text.strip(): - paras.append(text.strip()) - return paras +# --------------------------------------------------------------------------- +# Encoding-safe text reading +# --------------------------------------------------------------------------- +def read_text_safe(path): + """Read a text file without silently destroying non-UTF-8 content. + + Order: utf-8-sig strict -> cp949 strict -> euc-kr strict -> + utf-8 errors='replace' (last resort, with a prominent WARNING). + Never uses errors='ignore'. + + Returns (text, note) where note is None, or a 'NOTE: ...' string for a + successful non-UTF-8 decode, or a 'WARNING: ...' string for the lossy + replace fallback. Notes/warnings are surfaced in the audit report. + """ + raw = Path(path).read_bytes() + for enc in ('utf-8-sig', 'cp949', 'euc-kr'): + try: + text = raw.decode(enc) + except (UnicodeDecodeError, ValueError): + continue + if enc == 'utf-8-sig': + return text, None + return text, f'NOTE: {path} was decoded as {enc} (not UTF-8)' + text = raw.decode('utf-8', errors='replace') + warning = (f'WARNING: {path} could not be decoded strictly ' + '(tried utf-8-sig, cp949, euc-kr); read with utf-8 ' + "errors='replace' — some characters were replaced with U+FFFD; " + 'verify this file manually') + return text, warning -def load_paragraphs(path): - path = Path(path) - if path.suffix.lower() == '.docx': - return extract_docx_paragraphs(path) - text = path.read_text(encoding='utf-8', errors='ignore') - return [p.strip() for p in re.split(r'\n\s*\n', text) if p.strip()] +# --------------------------------------------------------------------------- +# Configuration (neutral built-in defaults + config/draft_audit.yaml overlay) +# --------------------------------------------------------------------------- -def split_sentences(para): - # Korean sentences mostly end with '다.' / '함.' etc.; keep simple+robust. - parts = re.split(r'(?<=[.!?])\s+', para) - return [s.strip() for s in parts if len(s.strip()) >= 10] +# Neutral defaults: generic thesis structure, no topic-specific source names. +# The shipped config/draft_audit.yaml carries the author's current values so +# behavior is preserved for existing users. +BUILTIN_CONFIG = { + 'source_mentions': [], + 'required_chapters': [ + {'name': '서론', 'patterns': ['서\\s*론', 'Introduction']}, + {'name': '관련연구', 'patterns': ['관련\\s*연구', '선행\\s*연구', + 'Related\\s*Work', 'Literature\\s*Review']}, + {'name': '연구방법', 'patterns': ['연구\\s*설계', '연구\\s*방법', '방법론', + 'Method(?:s|ology)?']}, + {'name': '평가/검증', 'patterns': ['평가', '검증', '결과', 'Evaluation', + 'Validation', 'Results?', 'Experiments?']}, + {'name': '결론', 'patterns': ['결\\s*론', 'Conclusion']}, + {'name': '참고문헌', 'patterns': ['참고\\s*문헌', 'References', 'Bibliography']}, + ], + 'overclaim_extra': [], + 'strong_claim_extra': [], +} + +CONFIG_LIST_KEYS = ('source_mentions', 'required_chapters', + 'overclaim_extra', 'strong_claim_extra') + + +def load_config(config_path=None): + """Load config/draft_audit.yaml if present, merged over neutral defaults. + + Degrades gracefully: missing file, missing PyYAML, or a broken YAML all + fall back to the built-in defaults (same guarded-import pattern as + load_yaml() in scripts/paperops.py). + """ + path = Path(config_path) if config_path else DEFAULT_CONFIG_PATH + cfg = copy.deepcopy(BUILTIN_CONFIG) + if not path.exists(): + return cfg + try: + import yaml + except Exception: + return cfg + try: + text, _note = read_text_safe(path) + loaded = yaml.safe_load(text) or {} + except Exception: + return cfg + if isinstance(loaded, dict): + for key in CONFIG_LIST_KEYS: + value = loaded.get(key) + if isinstance(value, list): + cfg[key] = value + return cfg + + +def _compile_fragment(fragment, flags=0): + """Compile a config-supplied regex fragment; fall back to literal.""" + try: + return re.compile(fragment, flags) + except re.error: + return re.compile(re.escape(str(fragment)), flags) + + +def _compile_alternation(fragments, flags=0): + if not fragments: + return None + parts = [] + for frag in sorted((str(f) for f in fragments), key=len, reverse=True): + try: + re.compile(frag) + parts.append(f'(?:{frag})') + except re.error: + parts.append(f'(?:{re.escape(frag)})') + return re.compile('|'.join(parts), flags) # --------------------------------------------------------------------------- -# Heuristic patterns +# Heuristic patterns (built-in detector defaults, Korean + English) # --------------------------------------------------------------------------- -# A sentence "mentions a source" if it names a standard, a cited work, a -# venue+year, or an explicit reference marker. -SOURCE_MENTION = re.compile( - r'\[(?:\d{1,3}|@[A-Za-z])' # [1], [@key] - r'|\(\s*[A-Z][A-Za-z-]+(?:\s+et\s+al\.?)?,?\s*\d{4}\s*\)' # (Noy, 2001) +# A sentence "mentions a source" if it carries a citation marker, an +# author-year citation (Korean or English), or an explicit reference phrase. +SOURCE_GENERIC = re.compile( + # [1], [1,2], [1-3], [1–3], [@citekey] + r'\[\s*(?:\d{1,3}(?:\s*[,–—-]\s*\d{1,3})*|@[A-Za-z][\w:.-]*)\s*\]' + # (Noy, 2001), (Kim et al., 2020), (Kim & Lee, 2020), (Kim and Lee 2020) + r'|\(\s*[A-Z][A-Za-z-]+(?:\s+(?:et\s+al\.?|and\s+[A-Z][A-Za-z-]+|&\s*[A-Z][A-Za-z-]+))?\s*,?\s*(?:19|20)\d{2}[a-z]?\s*\)' + # (김철수, 2020), (김철수 외, 2020), (김철수 등, 2020), (김철수 외 2020) + r'|\(\s*[가-힣]+(?:\s*(?:외|등))?\s*,?\s*(?:19|20)\d{2}[a-z]?\s*\)' r'|\b(19|20)\d{2}년\b.{0,30}(연구|논문|표준|보고)' - r'|Ontology Development 101|FEEKG|FinCaKG|FintechKG' - r'|W3C|SHACL Recommendation|ISO/IEC\s*\d+|Neo4j|Cypher Manual' - r'|OPEN DART|KRX|FRED|ECOS|Data\.go\.kr' r'|선행연구|기존 연구|문헌에서|에 따르면|보고되었|제안되었|알려져', re.IGNORECASE) -STRONG_CLAIM = re.compile( +STRONG_CLAIM_KO = re.compile( r'증명(한다|되었|했다)|입증(한다|되었)|보장(한다|된다)' r'|항상\s|반드시.{0,8}(향상|개선|성공)' r'|모든\s.{0,20}(가능하다|해결한다)' @@ -89,77 +193,539 @@ def split_sentences(para): r'|획기적|혁신적|압도적' r'|(성능|정확도|효율).{0,10}(탁월|월등|극대화)') -OVERCLAIM = re.compile( - r'완벽(하|한)|완전히\s+해결|100\s*%\s*(보장|정확)' - r'|hallucination[을를]?\s*제거|오류가\s*없' - r'|state[- ]of[- ]the[- ]art|SOTA') +STRONG_CLAIM_EN = re.compile( + r'\bprove(?:s|d|n)?\b|\bguarantee(?:s|d)?\b|\balways\b' + r'|\bnever\s+fails?\b|\bensures?\b|\bdemonstrates?\s+conclusively\b', + re.IGNORECASE) -# Specific numeric results that must come from actual experiments. -NUMERIC_RESULT = re.compile( - r'\d{1,3}(,\d{3})+\s*개|\d+\.\d+\s*%|precision\s*[=:]\s*\d|recall\s*[=:]\s*\d' - r'|F1\s*[=:]\s*\d|\d+\s*개\s*중\s*\d+') +OVERCLAIM_KO = re.compile( + r'완벽(하|한)|완전히\s+해결|100\s*%\s*(보장|정확)|오류가\s*없') -HYPOTHETICAL = re.compile(r'예를 들어|예컨대|가령|예시|라면|다면|할 수 있다') +OVERCLAIM_EN = re.compile( + r'state[-\s]of[-\s]the[-\s]art|world[-\s]?first|best[-\s]in[-\s]class' + r'|\bperfect(?:ly)?\b|\bflawless\b|\bunprecedented\b' + r'|\bgroundbreaking\b|\brevolutionary\b' + r'|\bnovel\s+(?:framework|method|approach|architecture|paradigm|system)\b', + re.IGNORECASE) + +# SOTA kept case-sensitive so ordinary words/names are not caught. +OVERCLAIM_SOTA = re.compile(r'\bSOTA\b') + +# Hypothetical/example context suppresses numeric flags. +# Conditional-mood suppression matches only a *true* conditional verb ending: +# a hangul syllable + 다면/라면 (한다면, 된다면, 좋다면, 이라면, 아니라면) +# followed by whitespace/punctuation/end of sentence. This fixes two bugs: +# - '다면' no longer matches inside words like '다면적' (blocked by the +# trailing-boundary lookahead); +# - the noun '라면' (standalone, e.g. '라면 시장') no longer matches because +# the ending must be attached to a preceding hangul syllable. +# '할 수 있다' was removed entirely: capability claims still need sources. +HYPOTHETICAL_MARKER = re.compile(r'예를 들어|예컨대|가령|예시') +CONDITIONAL_ENDING = re.compile( + r'(?:[가-힣]다면|이라면|아니라면)(?=[\s,.!?;:)\]”’]|$)') + + +def is_hypothetical(sentence): + if HYPOTHETICAL_MARKER.search(sentence): + return True + return bool(CONDITIONAL_ENDING.search(sentence)) + + +# --------------------------------------------------------------------------- +# Numeric extraction heuristic +# --------------------------------------------------------------------------- +# What counts as a checkable numeric result in the draft: +# - comma-grouped thousands (1,234 / 12,345.6) with or without a unit; +# - decimals (95.5 / 0.87), with or without % or a unit; +# - plain integers ONLY when they carry a result-like unit suffix +# (%, 퍼센트, 개, 건, 배, 점, x, times) or sit in a measurement context +# (precision/recall/accuracy/F1/AUC/정확도/정밀도/재현율/점수 = N, 'N개 중 M'). +# What is deliberately excluded (heuristic, documented per spec): +# - standalone years: bare integers such as 1990–2035 never match because a +# bare integer without a unit/context is not extracted at all, and '년' is +# not an accepted unit ('2020년' is a date, not a result); +# - list indices / section numbering: '1.' list markers are bare integers +# (not extracted); leading section numbers like '1.2 연구 방법' are dropped +# by the start-of-sentence guard; multi-dot tokens (2.1.3, v3.10) are +# dropped as version/section identifiers; +# - figure/table/section/version references: a decimal directly preceded by +# 그림/표/Fig/Table/Section/버전/v or followed by 절/장/항/조 is dropped. +NUM_TOKEN = re.compile( + r'-?\d{1,3}(?:,\d{3})+(?:\.\d+)?' # comma-grouped thousands + r'|-?\d+\.\d+' # decimal + r'|-?\d+' # integer (kept only with unit/context) +) +UNIT_AFTER = re.compile(r'^(?:\s*(?:%|퍼센트|개|건|배|점)|x\b|\s*times\b)', + re.IGNORECASE) +METRIC_BEFORE = re.compile( + r'(?:precision|recall|accuracy|f1(?:[-\s]?score)?|auc|bleu|rouge' + r'|정확도|정밀도|재현율|점수|[개건명]\s*중)\s*(?:[=:]|는|은|이|가|의)?\s*$', + re.IGNORECASE) +REF_BEFORE = re.compile( + r'(?:그림|표|도표|Fig(?:ure)?\.?|Table|Tab\.?|Section|Sec\.?|버전' + r'|ver(?:sion)?\.?|\bv)\s*$', re.IGNORECASE) +SECTION_AFTER = re.compile(r'^\s*(?:절|장|항|조)') +YEAR_MIN, YEAR_MAX = 1990, 2035 + + +def extract_numeric_tokens(sentence): + """Extract checkable numeric tokens from a draft sentence. + + Returns a list of dicts: {raw, norm, value, decimals}. + norm is the string-normalized form (commas/%/units stripped) used for + exact matching against experiment outputs. + """ + tokens = [] + for m in NUM_TOKEN.finditer(sentence): + tok = m.group(0) + before = sentence[:m.start()] + after = sentence[m.end():] + unit_m = UNIT_AFTER.match(after) + has_comma = ',' in tok + has_dot = '.' in tok + # multi-dot sequences (2.1.3, v3.10.1) are section/version identifiers + if after[:1] == '.' and after[1:2].isdigit(): + continue + if before[-1:] == '.' and before[-2:-1].isdigit(): + continue + if not has_comma and not has_dot: + # plain integer: needs a unit suffix or a measurement context + if not unit_m and not METRIC_BEFORE.search(before): + continue + value = float(tok) + # standalone-year guard (defensive; bare years carry no unit) + if not unit_m and YEAR_MIN <= value <= YEAR_MAX and after[:1] == '년': + continue + elif has_dot and not unit_m: + # bare decimal: drop figure/table/section/version references + if REF_BEFORE.search(before) or SECTION_AFTER.match(after): + continue + if m.start() == 0 and re.match(r'-?\d+(?:\.\d+)+\s+\S', sentence): + continue # leading section numbering like '1.2 연구 방법' + raw = tok + (unit_m.group(0).strip() if unit_m else '') + norm = tok.replace(',', '') + decimals = len(norm.split('.')[1]) if '.' in norm else 0 + try: + value = float(norm) + except ValueError: + continue + tokens.append({'raw': raw, 'norm': norm, + 'value': value, 'decimals': decimals}) + return tokens + + +# --------------------------------------------------------------------------- +# Experiment-output index (numeric cross-check) +# --------------------------------------------------------------------------- -# Expected thesis structure markers (Korean). -EXPECTED_SECTIONS = [ - ('서론', r'서\s*론|Introduction'), - ('관련연구', r'관련\s*연구|선행\s*연구|Related Work'), - ('연구설계/방법', r'연구\s*설계|연구\s*방법|방법론|Method'), - ('아티팩트/온톨로지 설계', r'온톨로지\s*설계|시스템\s*설계|Ontology|Design'), - ('평가/검증', r'평가|검증|Evaluation|Validation'), - ('결론', r'결\s*론|Conclusion'), - ('참고문헌', r'참고\s*문헌|References|원문\s*확인'), -] +# Broad numeric scan for data files: over-inclusion on the data side is safe +# (it only means more candidates a draft number could align with). +DATA_NUM_TOKEN = re.compile(r'-?\d{1,3}(?:,\d{3})+(?:\.\d+)?|-?\d+(?:\.\d+)?') + + +class ExperimentIndex: + """Numeric index over experiment output files. + + Matching semantics (alignment, never verification): + matched_exact - string-normalized equality (commas/% stripped) + matched_rounded - a data number equals the draft number when rounded + to the draft number's decimal precision + not_found_in_data - no alignment found + """ + + def __init__(self, data_dir): + self.data_dir = Path(data_dir) + self.files_scanned = [] + self.numbers_indexed = 0 + self.notes = [] + self.exact = {} # norm string -> relative file name + self.values = [] # (float value, relative file name) + self._rounded = {} # precision -> {formatted string -> file} + + def scan(self): + if not self.data_dir.is_dir(): + return self + for path in sorted(self.data_dir.rglob('*')): + if not path.is_file(): + continue + if path.suffix.lower() not in EXPERIMENT_SUFFIXES: + continue + rel = str(path.relative_to(self.data_dir)) + try: + size = path.stat().st_size + except OSError: + continue + if size > EXPERIMENT_MAX_BYTES: + self.notes.append( + f'NOTE: experiment file skipped (>20MB): {rel}') + continue + text, note = read_text_safe(path) + if note: + self.notes.append(note) + self.files_scanned.append(rel) + for m in DATA_NUM_TOKEN.finditer(text): + norm = m.group(0).replace(',', '') + try: + value = float(norm) + except ValueError: + continue + self.exact.setdefault(norm, rel) + self.values.append((value, rel)) + self.numbers_indexed += 1 + return self + + @property + def usable(self): + return bool(self.files_scanned) + + def _rounded_map(self, precision): + if precision not in self._rounded: + table = {} + for value, rel in self.values: + key = f'{round(value, precision):.{precision}f}' + table.setdefault(key, rel) + self._rounded[precision] = table + return self._rounded[precision] + + def check(self, token): + """Return (status, matched_file) for one draft numeric token.""" + hit = self.exact.get(token['norm']) + if hit is not None: + return 'matched_exact', hit + precision = token['decimals'] + key = f"{token['value']:.{precision}f}" + hit = self._rounded_map(precision).get(key) + if hit is not None: + return 'matched_rounded', hit + return 'not_found_in_data', '' + + +def resolve_experiment_dir(raw): + """Resolve the experiment-data dir. + + Explicit relative paths resolve against the caller's CWD (like --input). + The built-in default additionally falls back to the repo root so running + the CLI from anywhere still finds the repo's data/experiment_outputs. + """ + if raw: + p = Path(raw) + if p.is_absolute(): + return p + return Path.cwd() / p + cwd_p = Path.cwd() / DEFAULT_EXPERIMENT_DATA + if cwd_p.is_dir(): + return cwd_p + root_p = ROOT / DEFAULT_EXPERIMENT_DATA + if root_p.is_dir(): + return root_p + return cwd_p +# --------------------------------------------------------------------------- +# Draft loading (docx via stdlib zipfile + ElementTree; text via safe read) +# --------------------------------------------------------------------------- + +HEADING_STYLE = re.compile(r'(?i)heading|title|제목') +# Fallback heading heuristic (docx without styles / plain text / md without +# '#'): a short line (<40 chars) that starts like a chapter/section number. +HEADING_FALLBACK = re.compile(r'^(제\s*\d+\s*장|\d+(\.\d+)*\s)') +MD_HEADING = re.compile(r'^\s{0,3}#{1,6}\s') + + +def _docx_para(p_el): + """Flatten one w:p element: text runs with w:tab / w:br / w:cr mapped to + a single space so table/box numbers do not concatenate.""" + parts = [] + has_footnote_ref = False + has_outline_lvl = False + style = None + for node in p_el.iter(): + tag = node.tag + if tag == f'{W_NS}t': + parts.append(node.text or '') + elif tag in (f'{W_NS}tab', f'{W_NS}br', f'{W_NS}cr'): + parts.append(' ') + elif tag in (f'{W_NS}footnoteReference', f'{W_NS}endnoteReference'): + has_footnote_ref = True + elif tag == f'{W_NS}pStyle': + style = node.get(f'{W_NS}val') or '' + elif tag == f'{W_NS}outlineLvl': + has_outline_lvl = True + text = re.sub(r'\s+', ' ', ''.join(parts)).strip() + is_heading = has_outline_lvl or bool(style and HEADING_STYLE.search(style)) + return text, has_footnote_ref, is_heading + + +def _docx_notes_paras(z, member, kind): + """Parse word/footnotes.xml / word/endnotes.xml paragraphs, tagged with a + location like 'footnote:2' so findings cite them.""" + paras = [] + if member not in z.namelist(): + return paras + root = ET.fromstring(z.read(member)) + note_tag = f'{W_NS}footnote' if kind == 'footnote' else f'{W_NS}endnote' + for note in root.iter(note_tag): + ntype = note.get(f'{W_NS}type') or '' + if ntype in ('separator', 'continuationSeparator'): + continue + note_id = note.get(f'{W_NS}id') or '?' + for p_el in note.iter(f'{W_NS}p'): + text, has_ref, _heading = _docx_para(p_el) + if text: + paras.append({'text': text, + 'location': f'{kind}:{note_id}', + 'has_footnote_ref': has_ref, + 'is_heading': False}) + return paras + + +def load_docx_draft(path): + try: + with zipfile.ZipFile(path) as z: + try: + xml = z.read('word/document.xml') + except KeyError: + raise SystemExit( + f'not a Word document (word/document.xml missing): {path}') + try: + root = ET.fromstring(xml) + except ET.ParseError as e: + raise SystemExit(f'cannot parse docx XML ({e}): {path}') + paras = [] + for i, p_el in enumerate(root.iter(f'{W_NS}p')): + text, has_ref, is_heading = _docx_para(p_el) + if text: + paras.append({'text': text, + 'location': f'body:{i}', + 'has_footnote_ref': has_ref, + 'is_heading': is_heading}) + paras.extend(_docx_notes_paras(z, 'word/footnotes.xml', 'footnote')) + paras.extend(_docx_notes_paras(z, 'word/endnotes.xml', 'endnote')) + except zipfile.BadZipFile: + raise SystemExit(f'cannot read docx (not a valid zip archive): {path}') + # Structure check is heading-anchored: use styled headings when the + # document has any; otherwise fall back to short chapter-numbered lines. + styled = [p['text'] for p in paras + if p['is_heading'] and p['location'].startswith('body')] + if styled: + headings = styled + else: + headings = [p['text'] for p in paras + if p['location'].startswith('body') + and len(p['text']) < 40 + and HEADING_FALLBACK.match(p['text'])] + return paras, headings, [] + + +def load_text_draft(path): + text, note = read_text_safe(path) + notes = [note] if note else [] + paras = [] + for i, chunk in enumerate(re.split(r'\n\s*\n', text)): + chunk = chunk.strip() + if chunk: + paras.append({'text': chunk, + 'location': f'body:{i}', + 'has_footnote_ref': False, + 'is_heading': False}) + lines = [ln.strip() for ln in text.splitlines() if ln.strip()] + if path.suffix.lower() in ('.md', '.qmd', '.markdown'): + headings = [ln for ln in lines if MD_HEADING.match(ln)] + if not headings: # Korean drafts often use 제N장 without '#' + headings = [ln for ln in lines + if len(ln) < 40 and HEADING_FALLBACK.match(ln)] + else: + headings = [ln for ln in lines + if len(ln) < 40 and HEADING_FALLBACK.match(ln)] + return paras, headings, notes + + +def load_draft(path): + path = Path(path) + if path.suffix.lower() == '.docx': + return load_docx_draft(path) + return load_text_draft(path) + + +def split_sentences(para): + # Korean sentences mostly end with '다.' / '함.' etc.; keep simple+robust. + # Minimum length 4 so short overclaims like '완벽하다.' are still seen. + parts = re.split(r'(?<=[.!?。])\s+', para) + return [s.strip() for s in parts if len(s.strip()) >= 4] + + +# --------------------------------------------------------------------------- +# Detectors assembled from config +# --------------------------------------------------------------------------- + +def build_detectors(cfg): + named = cfg.get('source_mentions') or [] + strong_extra = cfg.get('strong_claim_extra') or [] + over_extra = cfg.get('overclaim_extra') or [] + chapters = [] + for ch in cfg.get('required_chapters') or []: + if not isinstance(ch, dict): + continue + name = str(ch.get('name') or '').strip() or '(unnamed)' + pats = [_compile_fragment(str(p)) for p in (ch.get('patterns') or [])] + if pats: + chapters.append((name, pats)) + return { + # source-name whitelist: IGNORECASE for source detection, + # case-sensitive for the named-source inventory (legacy behavior) + 'source_named': _compile_alternation(named, re.IGNORECASE), + 'source_named_cs': _compile_alternation(named), + 'strong_extra': [_compile_fragment(p, re.IGNORECASE) + for p in strong_extra], + 'over_extra': [_compile_fragment(p, re.IGNORECASE) + for p in over_extra], + 'chapters': chapters, + } + + +def sentence_has_source(sentence, det, has_footnote_ref=False): + # A footnote/endnote reference on the paragraph counts as having a source. + if has_footnote_ref: + return True + if SOURCE_GENERIC.search(sentence): + return True + named = det.get('source_named') + return bool(named and named.search(sentence)) + + +def find_strong_claim(sentence, det): + m = STRONG_CLAIM_KO.search(sentence) or STRONG_CLAIM_EN.search(sentence) + if m: + return m + for pat in det.get('strong_extra') or []: + m = pat.search(sentence) + if m: + return m + return None + + +def find_overclaim(sentence, det): + m = (OVERCLAIM_KO.search(sentence) or OVERCLAIM_EN.search(sentence) + or OVERCLAIM_SOTA.search(sentence)) + if m: + return m + for pat in det.get('over_extra') or []: + m = pat.search(sentence) + if m: + return m + return None + + +# --------------------------------------------------------------------------- +# Evidence Matrix coverage +# --------------------------------------------------------------------------- + def read_evidence_matrix_sources(): path = ROOT / 'matrices/evidence_matrix.csv' if not path.exists(): return set() keys = set() - with open(path, encoding='utf-8') as f: - for row in csv.DictReader(f): + try: + text, _note = read_text_safe(path) + for row in csv.DictReader(io.StringIO(text)): for field in ('citekey', 'paper_id', 'title'): v = (row.get(field) or '').strip().lower() if v: keys.add(v) + except Exception: + return set() return keys -def audit_draft(input_path, output_prefix=None): - paras = load_paragraphs(input_path) - full_text = '\n'.join(paras) - findings = [] +# --------------------------------------------------------------------------- +# Main audit +# --------------------------------------------------------------------------- + +def resolve_input_path(raw): + """Relative --input resolves against the caller's CWD; absolute paths are + kept as-is. If the CWD-relative path does not exist but the legacy + repo-root-relative one does, fall back to it (pre-v0.2 invocations).""" + p = Path(raw) + if p.is_absolute(): + return p + cwd_p = Path.cwd() / p + if cwd_p.exists(): + return cwd_p + root_p = ROOT / p + if root_p.exists(): + return root_p + return cwd_p + + +def audit_draft(input_path, output_prefix=None, experiment_data=None, + report_dir=None, config_path=None): + """Run the audit. Read-only with respect to the draft: this function + never modifies the input file and never marks anything as verified.""" + input_path = Path(input_path) + cfg = load_config(config_path) + det = build_detectors(cfg) + paras, headings, notes = load_draft(input_path) + full_text = '\n'.join(p['text'] for p in paras) + + data_dir = resolve_experiment_dir(experiment_data) + index = ExperimentIndex(data_dir).scan() + have_data = index.usable + notes.extend(index.notes) + + findings = [] # (para_index, finding_type, trigger, sentence, location) + numeric_rows = [] # (value, sentence excerpt, status, matched file, location) + numeric_summary = {'checked': 0, 'matched_exact': 0, + 'matched_rounded': 0, 'not_found_in_data': 0} n_sentences = 0 for i, para in enumerate(paras): - for sent in split_sentences(para): + loc = para['location'] + for sent in split_sentences(para['text']): n_sentences += 1 - mentions = bool(SOURCE_MENTION.search(sent)) - strong = STRONG_CLAIM.search(sent) - over = OVERCLAIM.search(sent) - numeric = NUMERIC_RESULT.search(sent) - hypo = bool(HYPOTHETICAL.search(sent)) + has_source = sentence_has_source(sent, det, + para['has_footnote_ref']) + strong = find_strong_claim(sent, det) + over = find_overclaim(sent, det) + hypo = is_hypothetical(sent) if over: - findings.append((i, 'overclaim', over.group(0), sent)) - elif strong and not mentions: - findings.append((i, 'strong_claim_no_source', strong.group(0), sent)) - if numeric and not hypo: - findings.append((i, 'numeric_needs_data', numeric.group(0), sent)) - elif numeric and hypo: - findings.append((i, 'numeric_hypothetical_ok', numeric.group(0), sent)) - - # Structure check + findings.append((i, 'overclaim', over.group(0), sent, loc)) + elif strong and not has_source: + findings.append((i, 'strong_claim_no_source', + strong.group(0), sent, loc)) + tokens = extract_numeric_tokens(sent) + if not tokens: + continue + if hypo: + findings.append((i, 'numeric_hypothetical_ok', + tokens[0]['raw'], sent, loc)) + continue + if not have_data: + findings.append((i, 'numeric_needs_data', + tokens[0]['raw'], sent, loc)) + continue + missing = [] + for tok in tokens: + status, matched_file = index.check(tok) + numeric_summary['checked'] += 1 + numeric_summary[status] += 1 + numeric_rows.append((tok['raw'], sent[:120], status, + matched_file, loc)) + if status == 'not_found_in_data': + missing.append(tok['raw']) + if missing: + findings.append((i, 'numeric_not_found_in_data', + '; '.join(missing), sent, loc)) + + # Structure check: heading-anchored — a chapter counts as present only if + # one of its patterns matches a heading line, never body prose. structure = [] - for name, pat in EXPECTED_SECTIONS: - structure.append((name, bool(re.search(pat, full_text)))) + for name, pats in det['chapters']: + ok = any(p.search(h) for h in headings for p in pats) + structure.append((name, ok)) # Named-source inventory: things the draft cites informally + named_cs = det.get('source_named_cs') named_sources = sorted(set( - m.group(0) for m in re.finditer( - r'Ontology Development 101|FEEKG|FinCaKG-?Onto|FintechKG' - r'|ISO/IEC\s*39075(:2024)?|SHACL|GQL|Neo4j|Cypher' - r'|OPEN DART|KRX|FRED|ECOS', full_text))) + m.group(0) for m in named_cs.finditer(full_text))) if named_cs else [] # Evidence Matrix coverage of named sources matrix_keys = read_evidence_matrix_sources() @@ -168,29 +734,72 @@ def audit_draft(input_path, output_prefix=None): hit = any(s.lower() in k for k in matrix_keys) coverage.append((s, hit)) - # Output + # Output (reports only; the draft itself is never touched) prefix = output_prefix or f'draft_audit_{today()}' - csv_path = ROOT / f'reports/review/{prefix}.csv' - md_path = ROOT / f'reports/review/{prefix}.md' - csv_path.parent.mkdir(parents=True, exist_ok=True) - with open(csv_path, 'w', encoding='utf-8', newline='') as f: + if report_dir: + rd = Path(report_dir) + out_dir = rd if rd.is_absolute() else (Path.cwd() / rd) + else: + out_dir = ROOT / DEFAULT_REPORT_SUBDIR + out_dir.mkdir(parents=True, exist_ok=True) + csv_path = out_dir / f'{prefix}.csv' + md_path = out_dir / f'{prefix}.md' + numeric_csv_path = out_dir / f'{prefix}_numeric.csv' + + # CSVs are utf-8-sig (BOM) so Excel opens Korean text correctly. + with open(csv_path, 'w', encoding='utf-8-sig', newline='') as f: w = csv.writer(f) - w.writerow(['para_index', 'finding_type', 'trigger', 'sentence']) + w.writerow(['para_index', 'finding_type', 'trigger', 'sentence', + 'location']) for row in findings: w.writerow(row) + wrote_numeric_csv = False + if have_data: + with open(numeric_csv_path, 'w', encoding='utf-8-sig', newline='') as f: + w = csv.writer(f) + w.writerow(['value', 'sentence_excerpt', 'status', + 'matched_file', 'location']) + for row in numeric_rows: + w.writerow(row) + wrote_numeric_csv = True + by_type = {} - for _, t, _, _ in findings: + for _, t, _, _, _ in findings: by_type[t] = by_type.get(t, 0) + 1 lines = [f'# Draft Audit {today()}\n\n', f'- input: `{input_path}`\n', f'- paragraphs: {len(paras)}\n', f'- sentences_scanned: {n_sentences}\n', - f'- findings_total: {len(findings)}\n\n', - '## Finding counts\n\n'] + f'- findings_total: {len(findings)}\n\n'] + if notes: + lines.append('## Encoding warnings\n\n') + for note in notes: + lines.append(f'- {note}\n') + lines.append('\n') + lines.append('## Finding counts\n\n') for t, c in sorted(by_type.items()): lines.append(f'- {t}: {c}\n') + lines.append('\n## Numeric cross-check vs experiment outputs\n\n') + if have_data: + lines.append(f'- experiment_data_dir: `{data_dir}` ' + f'(files_scanned={len(index.files_scanned)}, ' + f'numbers_indexed={index.numbers_indexed})\n') + lines.append(f"- numeric_checked: {numeric_summary['checked']}\n") + lines.append(f"- matched_exact: {numeric_summary['matched_exact']}\n") + lines.append( + f"- matched_rounded: {numeric_summary['matched_rounded']}\n") + lines.append( + f"- not_found_in_data: {numeric_summary['not_found_in_data']}\n") + lines.append(f'- per-number rows: `{numeric_csv_path.name}`\n') + lines.append('- 일치(match)는 실험 산출 파일과의 정렬(alignment)이며 ' + '진실 검증이 아니다. 어떤 수치도 verified로 표시되지 않는다. ' + '(Matching is alignment with experiment output files, ' + 'not truth validation; nothing is marked verified.)\n') + else: + lines.append(f'- {MISSING_DATA_LINE}\n') + lines.append(f'- looked for: `{data_dir}`\n') lines.append('\n## Structure check\n\n') for name, ok in structure: lines.append(f'- {"[OK]" if ok else "[MISSING]"} {name}\n') @@ -200,38 +809,68 @@ def audit_draft(input_path, output_prefix=None): lines.append(f'- {s}: {mark}\n') lines.append('\n## Top findings\n\n') shown = 0 - for idx, t, trig, sent in findings: + for idx, t, trig, sent, loc in findings: if t == 'numeric_hypothetical_ok': continue - lines.append(f'- [{t}] (문단 {idx}, trigger: `{trig}`)\n > {sent[:200]}\n') + where = f'문단 {idx}' if loc.startswith('body') else f'문단 {idx}, {loc}' + lines.append(f'- [{t}] ({where}, trigger: `{trig}`)\n > {sent[:200]}\n') shown += 1 if shown >= 40: - lines.append(f'- ... (전체는 CSV 참조)\n') + lines.append('- ... (전체는 CSV 참조)\n') break lines.append('\n## Disclaimer\n\n') lines.append('이 감사는 휴리스틱 플래깅이며 진실 검증이 아니다. ' '플래그된 문장은 사람이 원문/데이터로 확인해야 한다. ' + '수치 대조 결과(matched_*)는 실험 산출물과의 정렬일 뿐 ' + 'verified 상태가 아니다. ' '이 도구는 초안을 수정하지 않으며 verified 상태를 만들지 않는다.\n') with open(md_path, 'w', encoding='utf-8', newline='\n') as f: f.write(''.join(lines)) + print(csv_path) print(md_path) + if wrote_numeric_csv: + print(numeric_csv_path) print(f'paragraphs={len(paras)}') print(f'sentences={n_sentences}') for t, c in sorted(by_type.items()): print(f'{t}={c}') - missing = [name for name, ok in structure if not ok] - print(f'missing_sections={",".join(missing) if missing else "(none)"}') + if have_data: + print(f"numeric_checked={numeric_summary['checked']}") + print(f"matched_exact={numeric_summary['matched_exact']}") + print(f"matched_rounded={numeric_summary['matched_rounded']}") + print(f"not_found_in_data={numeric_summary['not_found_in_data']}") + else: + print(MISSING_DATA_LINE) + missing_sections = [name for name, ok in structure if not ok] + print(f'missing_sections={",".join(missing_sections) if missing_sections else "(none)"}') log(f'audit-manuscript-draft 실행: input={input_path}, findings={len(findings)}') + return { + 'csv': csv_path, + 'md': md_path, + 'numeric_csv': numeric_csv_path if wrote_numeric_csv else None, + 'findings': findings, + 'numeric_rows': numeric_rows, + 'numeric_summary': numeric_summary, + 'structure': structure, + 'headings': headings, + 'notes': notes, + 'paragraphs': len(paras), + 'sentences': n_sentences, + 'have_data': have_data, + 'experiment_dir': data_dir, + } def cmd_audit_manuscript_draft(args): - input_path = Path(args.input) - if not input_path.is_absolute(): - input_path = ROOT / args.input + input_path = resolve_input_path(args.input) if not input_path.exists(): raise SystemExit(f'input not found: {input_path}') - audit_draft(input_path, getattr(args, 'output_prefix', None)) + audit_draft(input_path, + output_prefix=getattr(args, 'output_prefix', None), + experiment_data=getattr(args, 'experiment_data', None), + report_dir=getattr(args, 'report_dir', None), + config_path=getattr(args, 'config', None)) def register_subcommands(sub): @@ -239,4 +878,13 @@ def register_subcommands(sub): help='Audit a draft (docx/txt/md/qmd) for evidence accountability') p.add_argument('--input', required=True) p.add_argument('--output-prefix') + p.add_argument('--experiment-data', default=None, + help='dir with experiment output files to cross-check ' + 'numbers against (default: data/experiment_outputs)') + p.add_argument('--report-dir', default=None, + help='where to write the MD/CSV reports ' + '(default: /reports/draft_audit)') + p.add_argument('--config', default=None, + help='path to draft_audit.yaml ' + '(default: /config/draft_audit.yaml)') p.set_defaults(func=cmd_audit_manuscript_draft) diff --git a/scripts/paperops_extra.py b/scripts/paperops_extra.py deleted file mode 100644 index 5106b55..0000000 --- a/scripts/paperops_extra.py +++ /dev/null @@ -1,170 +0,0 @@ -# -*- coding: utf-8 -*- -from __future__ import annotations -import argparse, csv, hashlib, json, math, re, sqlite3, urllib.parse, urllib.request -from collections import defaultdict -from datetime import datetime -from pathlib import Path - -ROOT = Path(__file__).resolve().parents[1] -DB = ROOT / 'data/metadata/papers.sqlite' -LOG = ROOT / 'logs/ACTIVITY_LOG.md' - -def now(): return datetime.now().strftime('%Y-%m-%d %H:%M:%S') -def today(): return datetime.now().strftime('%Y-%m-%d') -def log(msg): - LOG.parent.mkdir(parents=True, exist_ok=True) - if not LOG.exists(): LOG.write_text('# 논문AGENT 활동 로그\n\n', encoding='utf-8') - with LOG.open('a', encoding='utf-8') as f: f.write(f'- [{now()}] {msg}\n') -def conn(): - c = sqlite3.connect(DB); c.row_factory = sqlite3.Row; return c -def slug(s,n=70): return re.sub(r'[^A-Za-z0-9가-힣]+','_',s or '').strip('_').lower()[:n] or 'untitled' -def norm_title(s): return re.sub(r'\W+','',(s or '').lower()) -def stable_id(p): - key = 'doi:'+(p.get('doi') or '').lower() if p.get('doi') else 'title:'+norm_title(p.get('title','')) - return hashlib.sha1(key.encode('utf-8')).hexdigest()[:16] -def citekey(p): - y=str(p.get('year') or 'nd'); first='paper' - if p.get('authors'): first=re.split(r'\s+',p['authors'][0].replace(',',' '))[0].lower() - word=re.findall(r'[A-Za-z]{4,}',p.get('title','')); tail=(word[0].lower() if word else 'study') - return slug(f'{first}{y}{tail}',40) -def fetch_json(url, headers=None): - req=urllib.request.Request(url,headers=headers or {'User-Agent':'PaperOps/0.1'}) - with urllib.request.urlopen(req,timeout=40) as r: return json.loads(r.read().decode('utf-8','ignore')) -def load_queries(): - # YAML 없이도 동작하도록 단순 파싱 - text=(ROOT/'config/topic_profile.yaml').read_text(encoding='utf-8',errors='ignore') - qs=re.findall(r'query:\s*"([^"]+)"', text) - return qs or ['ontology knowledge graph semantic web','automated literature review research assistant'] -def upsert(p): - p['id']=p.get('id') or stable_id(p); p['citekey']=p.get('citekey') or citekey(p) - c=conn() - c.execute('''INSERT OR IGNORE INTO papers(id,title,authors_json,year,venue,doi,arxiv_id,abstract,url,pdf_url,source,collection_date,status,score,topic_relevance,citation_count,open_access,local_pdf_path,parsed_text_path,citekey,title_norm,raw_json,updated_at) - VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)''', - (p['id'],p.get('title'),json.dumps(p.get('authors') or [],ensure_ascii=False),p.get('year'),p.get('venue'),p.get('doi'),p.get('arxiv_id'),p.get('abstract'),p.get('url'),p.get('pdf_url'),p.get('source'),now(),'new',0,0,p.get('citation_count') or 0,1 if p.get('open_access') else 0,None,None,p['citekey'],norm_title(p.get('title','')),json.dumps(p,ensure_ascii=False),now())) - c.commit(); c.close() - -def cmd_collect_s2(args): - total=0; raw=ROOT/f'data/incoming/semantic_scholar_{datetime.now().strftime("%Y%m%d_%H%M%S")}.jsonl' - raw.parent.mkdir(parents=True,exist_ok=True) - with raw.open('w',encoding='utf-8') as f: - for q in load_queries(): - url='https://api.semanticscholar.org/graph/v1/paper/search?query='+urllib.parse.quote(q)+'&limit='+str(args.limit)+'&fields=title,abstract,year,venue,authors,url,openAccessPdf,citationCount,externalIds' - try: js=fetch_json(url) - except Exception as e: - print('WARN semantic_scholar', e); continue - for item in js.get('data',[]): - ext=item.get('externalIds') or {}; pdf=item.get('openAccessPdf') or {} - p=dict(title=item.get('title'),authors=[a.get('name','') for a in item.get('authors',[])],year=item.get('year'),venue=item.get('venue'),doi=ext.get('DOI'),arxiv_id=ext.get('ArXiv'),abstract=item.get('abstract') or '',url=item.get('url'),pdf_url=pdf.get('url'),source='semantic_scholar',citation_count=item.get('citationCount') or 0,open_access=bool(pdf.get('url'))) - upsert(p); f.write(json.dumps(p,ensure_ascii=False)+'\n'); total+=1 - log(f'Semantic Scholar 수집 완료: {total}건, raw={raw.name}') - print(raw, total) - -def cmd_dedupe(args): - c=conn(); rows=c.execute('SELECT * FROM papers').fetchall(); c.close() - groups=defaultdict(list) - for r in rows: - key=(r['doi'] or '').lower().strip() or r['title_norm'] - if key: groups[key].append(r) - dup=[v for v in groups.values() if len(v)>1] - out=ROOT/f'reports/audit_reports/duplicate_report_{today()}.md' - lines=[f'# Duplicate Report {today()}\n',f'- duplicate groups: {len(dup)}\n'] - for g in dup[:200]: - lines.append('\n## Group\n') - for r in g: lines.append(f"- `{r['id']}` {r['title']} / {r['source']} / {r['doi']}\n") - out.write_text('\n'.join(lines),encoding='utf-8'); log(f'중복 리포트 생성: {out}') - print(out) - -def top(limit): - c=conn(); rows=c.execute('SELECT * FROM papers ORDER BY score DESC, year DESC LIMIT ?',(limit,)).fetchall(); c.close(); return rows - -def cmd_export_bib(args): - out=ROOT/'manuscript/references.bib'; rows=top(args.limit); lines=[] - for r in rows: - typ='article'; authors=' and '.join(json.loads(r['authors_json'] or '[]')) - lines.append(f"@{typ}{{{r['citekey']},") - lines.append(f" title = {{{r['title'] or ''}}},") - if authors: lines.append(f" author = {{{authors}}},") - if r['year']: lines.append(f" year = {{{r['year']}}},") - if r['venue']: lines.append(f" journal = {{{r['venue']}}},") - if r['doi']: lines.append(f" doi = {{{r['doi']}}},") - if r['url']: lines.append(f" url = {{{r['url']}}},") - lines.append('}\n') - out.parent.mkdir(parents=True,exist_ok=True); out.write_text('\n'.join(lines),encoding='utf-8'); log(f'BibTeX export 완료: {out}, {len(rows)}건') - print(out) - -def cmd_weekly(args): - rows=top(args.top); out=ROOT/f'reports/weekly_review/weekly_review_{today()}.md'; out.parent.mkdir(parents=True,exist_ok=True) - lines=[f'# Weekly Research Review {today()}\n','## 이번 주 핵심 후보\n'] - for r in rows: - lines.append(f"- **{r['title']}** ({r['year']}) `{r['citekey']}` score={r['score']:.3f} status={r['status']}\n") - lines += ['\n## 읽기 우선순위\n1. important/to_read 논문 Paper Card 검토\n2. Evidence Matrix의 low confidence 행 검증\n3. Related Work에 쓸 claim 선별\n','\n## 다음 액션\n- PDF 다운로드/파싱\n- 검증된 quote/page 추가\n- 중복 제거\n'] - out.write_text('\n'.join(lines),encoding='utf-8'); log(f'Weekly Review 생성: {out}') - print(out) - -def read_evidence(): - ev=ROOT/'matrices/evidence_matrix.csv' - if not ev.exists(): return [] - with ev.open(encoding='utf-8',newline='') as f: return list(csv.DictReader(f)) - -def cmd_draft_related(args): - ev=read_evidence(); out=ROOT/'manuscript/sections/02_related_work.md' - rows=top(30) - lines=['# 2. Related Work\n','> 초벌입니다. `[NEEDS_VERIFICATION]` 표시가 있는 문장은 원문 확인 필요.\n','## 2.1 Ontology and Knowledge Graph Foundations\n'] - for e in ev[:15]: - if e.get('claim'): - lines.append(f"- {e['claim']} @{e.get('citekey','')} [confidence: {e.get('confidence','low')}]\n") - lines.append('\n## 2.2 AI-assisted Literature Review and Research Agents\n') - for r in rows[:15]: - title=(r['title'] or '').lower() - if any(k in title for k in ['review','research','writing','agent','citation','literature']): - lines.append(f"- {r['title']} contributes to this area @{r['citekey']} [NEEDS_VERIFICATION].\n") - lines.append('\n## 2.3 Gap Summary\n- Existing work is fragmented across paper discovery, reading, evidence extraction, and writing support. [NEEDS_SOURCE]\n- The proposed PaperOps pipeline focuses on evidence-first integration rather than isolated automation. [NEEDS_SOURCE]\n') - out.parent.mkdir(parents=True,exist_ok=True); out.write_text('\n'.join(lines),encoding='utf-8'); log(f'Related Work 초안 생성: {out}') - print(out) - -def cmd_reviewer(args): - out=ROOT/f'reports/audit_reports/reviewer_report_{today()}.md' - text=(ROOT/'manuscript/main.md').read_text(encoding='utf-8',errors='ignore') if (ROOT/'manuscript/main.md').exists() else '' - ev=read_evidence(); low=sum(1 for e in ev if e.get('confidence')!='high') - report=f'''# Reviewer-style Report {today()} - -## Summary -현재 원고는 작업 템플릿 단계이며, Evidence Matrix 기반으로 보강 중입니다. - -## Strengths -- 수집, 점수화, Paper Card, Evidence Matrix, Audit 흐름이 연결되어 있습니다. -- citekey와 근거 행렬을 중심으로 작성하도록 설계되어 있습니다. - -## Weaknesses / Risks -- Evidence Matrix의 low/medium confidence 행: {low}개 -- 원문 quote/page가 부족한 행은 최종 인용 근거로 쓰기 어렵습니다. -- Related Work 초안의 `[NEEDS_VERIFICATION]`, `[NEEDS_SOURCE]`를 제거해야 합니다. - -## Required Revisions -1. important 논문 PDF 원문 확인 -2. quote/page/section 보강 -3. manuscript/main.md에 섹션별 초안 통합 -4. citation audit 재실행 -5. 최신 논문 누락 여부 점검 -''' - out.parent.mkdir(parents=True,exist_ok=True); out.write_text(report,encoding='utf-8'); log(f'Reviewer Report 생성: {out}') - print(out) - -def cmd_index(args): - out=ROOT/'INDEX.md' - files=['README.md','docs/00_MASTER_DESIGN.md','docs/01_MVP_ROADMAP.md','docs/02_PIPELINE_SPEC.md','docs/08_ENHANCEMENT_REPORT_'+today()+'.md','reports/daily_digest/digest_'+today()+'.md','reports/weekly_review/weekly_review_'+today()+'.md','reports/survey_reports/outline_'+today()+'.md','reports/survey_reports/gap_map_'+today()+'.md','reports/survey_reports/thesis_brief_'+today()+'.md','reports/audit_reports/reviewer_report_'+today()+'.md','matrices/evidence_matrix.csv','matrices/screening_matrix.csv','matrices/gap_matrix.csv','research_design/problem_definition.md','research_design/research_questions.md','research_design/artifact_definition.md','research_design/evaluation_plan.md','manuscript/main.md','manuscript/sections/02_related_work.md'] - lines=['# 논문AGENT 산출물 인덱스\n']+[f'- [{f}]({f})\n' for f in files if (ROOT/f).exists()] - out.parent.mkdir(parents=True,exist_ok=True); out.write_text('\n'.join(lines),encoding='utf-8'); log(f'INDEX 생성: {out}') - print(out) - -def main(): - ap=argparse.ArgumentParser(); sub=ap.add_subparsers(dest='cmd',required=True) - p=sub.add_parser('collect-s2'); p.add_argument('--limit',type=int,default=20); p.set_defaults(func=cmd_collect_s2) - sub.add_parser('dedupe').set_defaults(func=cmd_dedupe) - p=sub.add_parser('export-bib'); p.add_argument('--limit',type=int,default=80); p.set_defaults(func=cmd_export_bib) - p=sub.add_parser('weekly'); p.add_argument('--top',type=int,default=30); p.set_defaults(func=cmd_weekly) - sub.add_parser('draft-related').set_defaults(func=cmd_draft_related) - sub.add_parser('reviewer').set_defaults(func=cmd_reviewer) - sub.add_parser('index').set_defaults(func=cmd_index) - args=ap.parse_args(); args.func(args) -if __name__=='__main__': main() diff --git a/scripts/paperops_figures.py b/scripts/paperops_figures.py index b67d73b..e9583b0 100644 --- a/scripts/paperops_figures.py +++ b/scripts/paperops_figures.py @@ -4,12 +4,20 @@ Generates reproducible, spec-driven figures for the thesis manuscript. Design principles (mirrors PaperOps governance): -- Figures are generated from explicit specs, never imagined content. +- Figures are generated from explicit specs (config/figures.yaml), never + imagined content. Only the sources provided in the spec file are rendered; + this module never fabricates data figures. - Sources (.dot / .mmd) are always saved so every figure is reproducible. -- Rendering uses Graphviz `dot` (primary). Mermaid sources are emitted for - GitHub-native display; rendering them is optional (npx mermaid-cli). +- Rendering uses Graphviz `dot` for kind=dot and mermaid-cli `mmdc` for + kind=mermaid. A missing renderer degrades to sources-only with a warning + (status renderer_missing, exit 0); pass --strict to fail instead. Real + renderer errors still fail the command. - Inserting figure references into the manuscript follows the guarded - preview -> human approval -> apply flow. No silent manuscript edits. + preview -> human approval -> apply flow. Target headings are matched as + exact lines (never substrings), matches inside fenced code blocks are + ignored, zero matches block with heading_not_found, multiple matches block + with heading_ambiguous (never guess), and re-applying blocks with + already_applied instead of duplicating. - Quantitative/result charts are intentionally NOT generated here unless an actual data CSV exists; this module never fabricates results. """ @@ -17,17 +25,47 @@ import argparse import csv import hashlib +import io +import re import shutil import subprocess from datetime import datetime from pathlib import Path +try: # optional dependency; this module must import without PyYAML + import yaml +except Exception: # pragma: no cover - environment dependent + yaml = None + ROOT = Path(__file__).resolve().parents[1] FIG_DIR = ROOT / 'reports/figures' FIG_SRC_DIR = FIG_DIR / 'src' MANUSCRIPT_FIG_DIR = ROOT / '05_manuscript/figures' LOG = ROOT / 'logs/ACTIVITY_LOG.md' +# Figure specs live outside the code so authors can edit them without +# touching the pipeline. Resolved against ROOT at call time. +FIGURES_CONFIG_REL = 'config/figures.yaml' + +RENDERERS = {'dot': 'dot', 'mermaid': 'mmdc'} +RENDERER_INSTALL_HINTS = { + 'dot': ('install Graphviz so `dot` is on PATH ' + '(https://graphviz.org/download/ ; e.g. `apt install graphviz` ' + 'or `winget install Graphviz.Graphviz`)'), + 'mmdc': ('install mermaid-cli so `mmdc` is on PATH ' + '(`npm install -g @mermaid-js/mermaid-cli`)'), +} + +REQUIRED_SPEC_KEYS = ('id', 'kind', 'caption', 'target_heading', 'source') +SPEC_KINDS = tuple(RENDERERS) + +PREVIEW_FIELDS = ('figure_id', 'target_file', 'target_heading', + 'target_sha256', 'insert_block', 'status', 'reason') + +# Blocked reasons that are benign on re-run (idempotency) and therefore do +# not turn the exit code non-zero. +BENIGN_BLOCK_REASONS = ('already_applied',) + def now(): return datetime.now().strftime('%Y-%m-%d %H:%M:%S') @@ -60,270 +98,215 @@ def write_lf(path, text): f.write(text) +def read_text_compat(path): + """Read text as utf-8(-sig) strict, then cp949 strict, then utf-8 replace. + + Keeps Korean-locale files (cp949) readable without ever crashing on a + stray byte; the lossy last resort warns loudly. + """ + data = Path(path).read_bytes() + try: + return data.decode('utf-8-sig') + except UnicodeDecodeError: + pass + try: + return data.decode('cp949') + except UnicodeDecodeError: + pass + print(f'WARN: {path}: not valid utf-8/cp949; decoded as utf-8 with ' + 'replacement characters') + return data.decode('utf-8', errors='replace') + + +def relpath(p): + p = Path(p) + try: + return p.relative_to(ROOT).as_posix() + except ValueError: + return str(p) + + # --------------------------------------------------------------------------- -# Figure spec registry +# Figure spec loading (config/figures.yaml) # --------------------------------------------------------------------------- -# Each spec is grounded in the actual PaperOps architecture and CLI commands. +# Schema: figures: [{id, kind: dot|mermaid, caption, target_heading, +# placeholder (optional), source: |...}] plus optional title / target_file / +# mermaid (companion .mmd source for GitHub display when kind is dot). # No spec may claim performance results or external-system equivalence. -GV_STYLE = ( - ' graph [fontname="Helvetica", fontsize=11, rankdir=%s, splines=ortho, ' - 'nodesep=0.45, ranksep=0.55, pad=0.2];\n' - ' node [fontname="Helvetica", fontsize=11, shape=box, style="rounded,filled", ' - 'fillcolor="#F4F4F2", color="#555555", margin="0.18,0.10"];\n' - ' edge [fontname="Helvetica", fontsize=9, color="#555555", arrowsize=0.7];\n' -) - -HUMAN_NODE = 'fillcolor="#FFE9C7"' -GUARD_NODE = 'fillcolor="#DCE9F7"' -DATA_NODE = 'shape=cylinder, fillcolor="#EAF4EA"' - -FIGURE_SPECS = { - 'fig_pipeline': { - 'title': 'PaperOps end-to-end pipeline', - 'caption': ( - 'PaperOps end-to-end pipeline. Literature collection, parsing, and ' - 'evidence extraction are automated, while review, verification, and ' - 'manuscript changes pass through explicit human approval gates.' - ), +BUILTIN_DEMO_SPECS = { + 'fig_paperops_demo': { + 'kind': 'dot', + 'title': 'PaperOps minimal architecture (built-in demo spec)', + 'caption': ('PaperOps minimal architecture. Built-in demo spec used ' + 'because config/figures.yaml was not loaded.'), 'target_file': '05_manuscript/chapters/ch3_method.qmd', 'target_heading': '## PaperOps Architecture', - 'dot': ( - 'digraph fig_pipeline {\n' + GV_STYLE % 'TB' + - ' collect [label="Collect\\n(arXiv / S2 / OpenAlex)"];\n' - ' screen [label="Score & Screen"];\n' - ' pdf [label="PDF Download"];\n' - ' grobid [label="GROBID Parse"];\n' - ' extract [label="Evidence Candidate\\nExtraction"];\n' - ' review [label="Human Review\\n(review queue)", ' + HUMAN_NODE + '];\n' - ' matrix [label="Evidence Matrix\\n(promoted rows)", ' + DATA_NODE + '];\n' - ' preview [label="Manuscript Patch\\nPreview + Diff"];\n' - ' approve [label="Human Approval", ' + HUMAN_NODE + '];\n' - ' apply [label="Guarded Apply\\n(backup + LF write)"];\n' - ' guard [label="Guards\\n(no-auto-verified, smoke-test)", ' + GUARD_NODE + '];\n' - ' thesis [label="Thesis Manuscript\\n(Quarto)", ' + DATA_NODE + '];\n' - ' subgraph cluster_auto {\n' - ' label="Automated collection & extraction"; style=dashed; ' - 'color="#999999";\n' - ' collect -> screen -> pdf -> grobid -> extract;\n' - ' }\n' - ' subgraph cluster_gov {\n' - ' label="Governed review & writing"; style=dashed; color="#999999";\n' - ' review -> matrix [label="promote"];\n' - ' matrix -> preview -> approve -> apply -> thesis;\n' - ' }\n' - ' extract -> review;\n' - ' apply -> guard [style=dashed, label="post-check"];\n' - ' guard -> matrix [style=dashed, label="audit", constraint=false];\n' - '}\n' - ), - 'mermaid': ( - 'flowchart LR\n' - ' A[Collect
arXiv / Semantic Scholar / OpenAlex] --> B[Score & Screen]\n' - ' B --> C[PDF Download]\n' - ' C --> D[GROBID Parse]\n' - ' D --> E[Evidence Candidate Extraction]\n' - ' E --> F{{Human Review}}\n' - ' F -->|promote| G[(Evidence Matrix)]\n' - ' G --> H[Manuscript Patch Preview + Diff]\n' - ' H --> I{{Human Approval}}\n' - ' I --> J[Guarded Apply
backup + LF write]\n' - ' J --> K[(Thesis Manuscript)]\n' - ' J -.post-check.-> L[Guards
no-auto-verified, smoke-test]\n' - ' L -.audit.-> G\n' - ), - }, - 'fig_evidence_flow': { - 'title': 'Evidence governance flow', - 'caption': ( - 'Evidence governance flow. Quote matching and page location are ' - 'treated as source-alignment checks; the verified state is reachable ' - 'only through an explicit human verification gate.' - ), - 'target_file': '05_manuscript/chapters/ch3_method.qmd', - 'target_heading': '## Evidence-first Workflow', - 'dot': ( - 'digraph fig_evidence_flow {\n' + GV_STYLE % 'TB' + - ' cand [label="Evidence Candidate\\n(claim + quote + location)"];\n' - ' valid [label="Structural Validation\\n(schema, citekey)"];\n' - ' align [label="Source Alignment\\n(quote match, page locate)"];\n' - ' queue [label="Review Queue", ' + DATA_NODE + '];\n' - ' human [label="Human Decision\\n(keep / revise / reject)", ' + HUMAN_NODE + '];\n' - ' promoted [label="Promoted Row\\nverified=false", ' + DATA_NODE + '];\n' - ' pdfcheck [label="PDF Page Check\\n(required for high-risk)"];\n' - ' verify [label="Human Verification\\nGate", ' + HUMAN_NODE + '];\n' - ' verified [label="verified=true\\n(manual only)", ' + GUARD_NODE + '];\n' - ' align_note [label="alignment != truth validation", shape=note, ' - 'fillcolor="#FFF7D6"];\n' - ' { rank=same; cand; valid; align; queue; human; }\n' - ' { rank=same; align_note; verified; verify; pdfcheck; promoted; }\n' - ' cand -> valid -> align -> queue -> human [constraint=false];\n' - ' // invisible vertical pins keep row 2 folded under row 1\n' - ' cand -> align_note [style=invis];\n' - ' valid -> verified [style=invis];\n' - ' align -> verify [style=invis];\n' - ' queue -> pdfcheck [style=invis];\n' - ' human -> promoted [label="promote"];\n' - ' promoted -> pdfcheck [constraint=false];\n' - ' pdfcheck -> verify [constraint=false];\n' - ' verify -> verified [constraint=false];\n' - '}\n' - ), - 'mermaid': ( - 'flowchart LR\n' - ' A[Evidence Candidate
claim + quote + location] --> B[Structural Validation]\n' - ' B --> C[Source Alignment
quote match, page locate]\n' - ' C --> D[(Review Queue)]\n' - ' D --> E{{Human Decision}}\n' - ' E --> F[(Promoted Row
verified=false)]\n' - ' F --> G[PDF Page Check]\n' - ' G --> H{{Human Verification Gate}}\n' - ' H --> I[verified=true
manual only]\n' - ), - }, - 'fig_guarded_apply': { - 'title': 'Guarded manuscript apply workflow', - 'caption': ( - 'Guarded manuscript apply workflow. Every manuscript change is ' - 'previewed as a diff, requires human approval, is applied against a ' - 'backup, and is followed by automated guard and smoke-test checks; ' - 'failures roll back from the backup.' - ), - 'target_file': '05_manuscript/chapters/ch3_method.qmd', - 'target_heading': '## Human Verification Policy', - 'dot': ( - 'digraph fig_guarded_apply {\n' + GV_STYLE % 'TB' + - ' preview [label="Patch Preview\\n(CSV + MD + diff)"];\n' - ' review [label="Human Diff Review", ' + HUMAN_NODE + '];\n' - ' backup [label="Backup Chapters"];\n' - ' apply [label="Apply\\n(SHA-checked, LF write)"];\n' - ' postguard [label="guard-no-auto-verified\\n+ smoke-test", ' + GUARD_NODE + '];\n' - ' report [label="Apply Report", ' + DATA_NODE + '];\n' - ' rollback [label="Rollback from Backup", fillcolor="#F7DCDC"];\n' - ' preview -> review;\n' - ' review -> backup [label="approved"];\n' - ' review -> preview [label="rejected / revise", style=dashed];\n' - ' backup -> apply -> postguard;\n' - ' postguard -> report [label="pass"];\n' - ' postguard -> rollback [label="fail", style=dashed];\n' - ' rollback -> preview [style=dashed];\n' + 'placeholder': None, + 'mermaid': None, + 'source': ( + 'digraph fig_paperops_demo {\n' + ' graph [fontname="Helvetica", fontsize=11, rankdir=TB];\n' + ' node [fontname="Helvetica", fontsize=11, shape=box, ' + 'style="rounded,filled", fillcolor="#F4F4F2"];\n' + ' cli [label="PaperOps CLI"];\n' + ' review [label="Human Review & Approval", fillcolor="#FFE9C7"];\n' + ' manuscript [label="Thesis Manuscript", shape=cylinder, ' + 'fillcolor="#EAF4EA"];\n' + ' cli -> review;\n' + ' review -> manuscript [label="guarded apply"];\n' '}\n' ), - 'mermaid': ( - 'flowchart TB\n' - ' A[Patch Preview
CSV + MD + diff] --> B{{Human Diff Review}}\n' - ' B -->|approved| C[Backup Chapters]\n' - ' B -.rejected / revise.-> A\n' - ' C --> D[Apply
SHA-checked, LF write]\n' - ' D --> E[guard-no-auto-verified
+ smoke-test]\n' - ' E -->|pass| F[(Apply Report)]\n' - ' E -.fail.-> G[Rollback from Backup]\n' - ' G -.-> A\n' - ), - }, - 'fig_architecture': { - 'title': 'PaperOps system architecture', - 'caption': ( - 'PaperOps system architecture. A single CLI orchestrates external ' - 'services (GROBID, Zotero/Better BibTeX) and local stores (paper DB, ' - 'matrices, manuscript), with audit reports produced at each guarded step.' - ), - 'target_file': '05_manuscript/chapters/ch4_system.qmd', - 'target_heading': '## Data Model', - 'dot': ( - 'digraph fig_architecture {\n' + GV_STYLE % 'TB' + - ' cli [label="PaperOps CLI\\n(scripts/paperops.py)"];\n' - ' subgraph cluster_ext {\n' - ' label="External services"; style=dashed; color="#999999";\n' - ' grobid [label="GROBID\\n(Docker)"];\n' - ' zotero [label="Zotero +\\nBetter BibTeX"];\n' - ' apis [label="Paper APIs\\n(arXiv, S2, OpenAlex)"];\n' - ' }\n' - ' subgraph cluster_store {\n' - ' label="Local stores"; style=dashed; color="#999999";\n' - ' db [label="papers.sqlite", ' + DATA_NODE + '];\n' - ' matrices [label="matrices/\\n(evidence, screening, gap)", ' + DATA_NODE + '];\n' - ' manuscript [label="05_manuscript/\\n(Quarto)", ' + DATA_NODE + '];\n' - ' reports [label="reports/\\n(audit, review, figures)", ' + DATA_NODE + '];\n' - ' }\n' - ' config [label="config/\\n(sources, scoring, prompts)", shape=folder];\n' - ' cli -> apis [dir=both];\n' - ' cli -> grobid [dir=both];\n' - ' cli -> zotero [dir=both];\n' - ' cli -> db [dir=both];\n' - ' cli -> matrices [dir=both];\n' - ' cli -> manuscript [label="guarded\\napply only"];\n' - ' cli -> reports;\n' - ' config -> cli;\n' - '}\n' - ), - 'mermaid': ( - 'flowchart TB\n' - ' CLI[PaperOps CLI
scripts/paperops.py]\n' - ' subgraph External services\n' - ' G[GROBID Docker]\n' - ' Z[Zotero + Better BibTeX]\n' - ' A[Paper APIs
arXiv, S2, OpenAlex]\n' - ' end\n' - ' subgraph Local stores\n' - ' DB[(papers.sqlite)]\n' - ' M[(matrices/)]\n' - ' MS[(05_manuscript/)]\n' - ' R[(reports/)]\n' - ' end\n' - ' CFG[config/] --> CLI\n' - ' CLI <--> A\n' - ' CLI <--> G\n' - ' CLI <--> Z\n' - ' CLI <--> DB\n' - ' CLI <--> M\n' - ' CLI -->|guarded apply only| MS\n' - ' CLI --> R\n' - ), - }, - 'fig_verification_states': { - 'title': 'Evidence verification state transitions', - 'caption': ( - 'Evidence verification state transitions. There is no automated ' - 'transition into the verified state; only a human reviewer can mark ' - 'evidence as verified, and guards enforce this invariant.' - ), - 'target_file': '05_manuscript/chapters/ch5_evaluation.qmd', - 'target_heading': '## Metrics', - 'dot': ( - 'digraph fig_verification_states {\n' + GV_STYLE % 'LR' + - ' extracted [label="extracted"];\n' - ' validated [label="candidate\\nvalidated"];\n' - ' in_review [label="in review", ' + HUMAN_NODE + '];\n' - ' promoted [label="promoted\\n(verified=false)", ' + DATA_NODE + '];\n' - ' pdf_check [label="pdf check\\nrequired"];\n' - ' verified [label="verified=true", ' + GUARD_NODE + '];\n' - ' rejected [label="rejected", fillcolor="#F7DCDC"];\n' - ' extracted -> validated -> in_review;\n' - ' in_review -> promoted [label="human keep"];\n' - ' in_review -> rejected [label="human reject"];\n' - ' promoted -> pdf_check;\n' - ' pdf_check -> verified [label="human only", penwidth=2];\n' - ' promoted -> verified [style=invis];\n' - ' noauto [label="no automated edge\\ninto verified", shape=note, ' - 'fillcolor="#FFF7D6"];\n' - '}\n' - ), - 'mermaid': ( - 'stateDiagram-v2\n' - ' [*] --> extracted\n' - ' extracted --> validated\n' - ' validated --> in_review\n' - ' in_review --> promoted : human keep\n' - ' in_review --> rejected : human reject\n' - ' promoted --> pdf_check\n' - ' pdf_check --> verified : human only\n' - ' note right of verified : no automated transition\n' - ), }, } +def _builtin_fallback(reason, quiet): + if not quiet: + print(f'WARN: {reason}; falling back to built-in demo spec ' + f'({", ".join(BUILTIN_DEMO_SPECS)})') + specs = {fid: dict(spec) for fid, spec in BUILTIN_DEMO_SPECS.items()} + return specs, [], 'builtin' + + +def validate_figure_entry(entry, index, seen_ids): + """Validate one yaml figure entry. + + Returns (fig_id, normalized_spec, None) on success or + (None, None, reason) on failure. Never raises. + """ + where = f'figures[{index}]' + if not isinstance(entry, dict): + return None, None, f'{where}: entry must be a mapping, got {type(entry).__name__}' + raw_id = entry.get('id') + if isinstance(raw_id, str) and raw_id.strip(): + where = f'{where} ({raw_id.strip()})' + missing = [k for k in REQUIRED_SPEC_KEYS + if not (isinstance(entry.get(k), str) and entry.get(k).strip())] + if missing: + return None, None, f'{where}: missing/empty required key(s): {", ".join(missing)}' + fig_id = raw_id.strip() + kind = entry['kind'].strip() + if kind not in SPEC_KINDS: + return None, None, (f'{where}: kind must be one of ' + f'{"|".join(SPEC_KINDS)}, got {kind!r}') + if fig_id in seen_ids: + return None, None, f'{where}: duplicate figure id' + + def opt_str(key): + v = entry.get(key) + return v if isinstance(v, str) and v.strip() else None + + spec = { + 'kind': kind, + 'caption': entry['caption'].strip(), + 'target_heading': entry['target_heading'].strip(), + 'source': entry['source'], + 'title': (opt_str('title') or fig_id).strip(), + 'target_file': opt_str('target_file'), + 'placeholder': opt_str('placeholder'), + 'mermaid': opt_str('mermaid'), + } + return fig_id, spec, None + + +def load_figure_specs(quiet=False): + """Load figure specs from config/figures.yaml. + + Returns (specs, invalid, origin): specs maps figure_id to a normalized + spec dict, invalid lists human-readable reasons for rejected entries, + origin is 'yaml' or 'builtin'. A missing/unreadable config or a missing + PyYAML falls back to a built-in one-figure demo spec with a warning; + invalid individual entries are skipped and reported, never crash. + """ + cfg = ROOT / FIGURES_CONFIG_REL + if yaml is None: + return _builtin_fallback( + f'PyYAML is not installed; cannot read {FIGURES_CONFIG_REL}', quiet) + if not cfg.exists(): + return _builtin_fallback(f'{FIGURES_CONFIG_REL} is missing', quiet) + try: + data = yaml.safe_load(read_text_compat(cfg)) + except Exception as e: + return _builtin_fallback( + f'{FIGURES_CONFIG_REL} could not be parsed ({type(e).__name__}: {e})', + quiet) + figures = data.get('figures') if isinstance(data, dict) else None + if not isinstance(figures, list): + return _builtin_fallback(f'{FIGURES_CONFIG_REL} has no `figures:` list', quiet) + specs, invalid = {}, [] + for i, entry in enumerate(figures): + fig_id, spec, err = validate_figure_entry(entry, i, specs) + if err: + invalid.append(err) + continue + specs[fig_id] = spec + if invalid and not quiet: + for reason in invalid: + print(f'WARN: invalid figure spec skipped: {reason}') + return specs, invalid, 'yaml' + + +# --------------------------------------------------------------------------- +# Heading anchoring +# --------------------------------------------------------------------------- + +def heading_match_lines(text, heading): + """Return 0-based indexes of lines that ARE the heading, exactly. + + A line matches only when, stripped, it equals the target heading + (regex ^\\s*\\s*$ applied per line), so + '## PaperOps Architecture' never matches inside + '### PaperOps Architecture Details'. Lines inside fenced code blocks + (``` or ~~~, tracked statefully) are skipped. + """ + pattern = re.compile(r'^\s*' + re.escape(heading.strip()) + r'\s*$') + matches = [] + fence = None + for i, line in enumerate(text.split('\n')): + stripped = line.lstrip() + if fence is not None: + if stripped.startswith(fence): + fence = None + continue + if stripped.startswith('```') or stripped.startswith('~~~'): + fence = stripped[:3] + continue + if pattern.match(line): + matches.append(i) + return matches + + +def locate_heading(text, heading): + """Return (line_index, None) for a unique heading line, else + (None, 'heading_not_found') or (None, 'heading_ambiguous'). Never guesses. + """ + matches = heading_match_lines(text, heading) + if not matches: + return None, 'heading_not_found' + if len(matches) > 1: + return None, 'heading_ambiguous' + return matches[0], None + + +def figure_block(fig_id, spec): + placeholder = (spec.get('placeholder') or '').strip() + if placeholder: + return placeholder + '\n' + label = fig_id.replace('_', '-') + rel = f'figures/{fig_id}.svg' + return f'![{spec["caption"]}]({rel}){{#{label}}}\n' + + +def placeholder_present(text, fig_id, insert_block=''): + """True when the figure's placeholder/image line already exists.""" + anchor = '{#%s}' % fig_id.replace('_', '-') + if anchor in text: + return True + block = (insert_block or '').strip() + return bool(block) and block in text + + # --------------------------------------------------------------------------- # Rendering # --------------------------------------------------------------------------- @@ -332,34 +315,61 @@ def dot_available(): return shutil.which('dot') is not None -def render_figure(fig_id, formats=('svg', 'png')): - """Write sources and render via Graphviz. Returns result dict.""" - spec = FIGURE_SPECS[fig_id] +def renderer_path(kind): + """Resolve the renderer executable via shutil.which (Windows .cmd-safe).""" + return shutil.which(RENDERERS[kind]) + + +def render_figure(fig_id, spec, formats=('svg', 'png')): + """Write sources and render one figure. + + Sources are always written. Result dict carries status: + 'rendered' | 'renderer_missing' (renderer not on PATH; sources only) | + 'error' (renderer present but failed). + """ FIG_SRC_DIR.mkdir(parents=True, exist_ok=True) - dot_path = FIG_SRC_DIR / f'{fig_id}.dot' - mmd_path = FIG_SRC_DIR / f'{fig_id}.mmd' - write_lf(dot_path, spec['dot']) - write_lf(mmd_path, spec['mermaid']) - result = {'figure_id': fig_id, 'dot': str(dot_path), 'mmd': str(mmd_path), - 'rendered': [], 'errors': []} - if not dot_available(): - result['errors'].append('graphviz `dot` not found in PATH; sources written only') + kind = spec['kind'] + result = {'figure_id': fig_id, 'kind': kind, 'sources': [], + 'rendered': [], 'errors': [], 'status': 'rendered'} + if kind == 'dot': + src_path = FIG_SRC_DIR / f'{fig_id}.dot' + write_lf(src_path, spec['source']) + result['sources'].append(str(src_path)) + if spec.get('mermaid'): + mmd_path = FIG_SRC_DIR / f'{fig_id}.mmd' + write_lf(mmd_path, spec['mermaid']) + result['sources'].append(str(mmd_path)) + else: + src_path = FIG_SRC_DIR / f'{fig_id}.mmd' + write_lf(src_path, spec['source']) + result['sources'].append(str(src_path)) + exe = renderer_path(kind) + if exe is None: + result['status'] = 'renderer_missing' return result + timeout = 180 if kind == 'mermaid' else 60 for fmt in formats: out = FIG_DIR / f'{fig_id}.{fmt}' - cmd = ['dot', f'-T{fmt}', str(dot_path), '-o', str(out)] - if fmt == 'png': - cmd = ['dot', f'-T{fmt}', '-Gdpi=200', str(dot_path), '-o', str(out)] + if kind == 'dot': + cmd = [exe, f'-T{fmt}', str(src_path), '-o', str(out)] + if fmt == 'png': + cmd = [exe, f'-T{fmt}', '-Gdpi=200', str(src_path), '-o', str(out)] + else: + cmd = [exe, '-i', str(src_path), '-o', str(out)] try: - proc = subprocess.run(cmd, capture_output=True, text=True, timeout=60) + proc = subprocess.run(cmd, capture_output=True, encoding='utf-8', + errors='replace', timeout=timeout) if proc.returncode != 0: - result['errors'].append(f'{fmt}: {proc.stderr.strip()[:300]}') + err = (proc.stderr or proc.stdout or '').strip() + result['errors'].append(f'{fmt}: {err[:300]}') elif not out.exists() or out.stat().st_size == 0: result['errors'].append(f'{fmt}: output missing or empty') else: result['rendered'].append(str(out)) except Exception as e: result['errors'].append(f'{fmt}: {type(e).__name__}: {e}') + if result['errors']: + result['status'] = 'error' return result @@ -368,106 +378,145 @@ def render_figure(fig_id, formats=('svg', 'png')): # --------------------------------------------------------------------------- def cmd_propose_figures(args): + specs, invalid, origin = load_figure_specs() out = FIG_DIR / f'figure_proposals_{today()}.md' lines = [f'# Figure Proposals {today()}\n\n', 'Spec-driven figure candidates. Sources are always saved for ' - 'reproducibility; rendering uses Graphviz (primary) with Mermaid ' - 'sources for GitHub display.\n\n'] - for fig_id, spec in FIGURE_SPECS.items(): - target = ROOT / spec['target_file'] + 'reproducibility; rendering uses Graphviz (kind=dot) or ' + 'mermaid-cli (kind=mermaid).\n\n', + f'- spec_source: {origin} (`{FIGURES_CONFIG_REL}`)\n\n'] + for fig_id, spec in specs.items(): + target_file = spec.get('target_file') + target = (ROOT / target_file) if target_file else None placed = False - if target.exists(): - placed = f'#{fig_id.replace("_", "-")}' in target.read_text(encoding='utf-8') + if target is not None and target.exists(): + placed = placeholder_present(read_text_compat(target), fig_id, + figure_block(fig_id, spec)) lines.append(f'## {fig_id}\n\n') lines.append(f'- title: {spec["title"]}\n') - lines.append(f'- target: `{spec["target_file"]}` / `{spec["target_heading"]}`\n') + lines.append(f'- kind: {spec["kind"]}\n') + lines.append(f'- target: `{target_file or "(none)"}` / ' + f'`{spec["target_heading"]}`\n') lines.append(f'- already_placed: {str(placed).lower()}\n') lines.append(f'- caption: {spec["caption"]}\n\n') + if invalid: + lines.append('## Invalid specs (skipped)\n\n') + lines += [f'- {reason}\n' for reason in invalid] + lines.append('\n') lines.append('## Out of scope by design\n\n') lines.append('- Quantitative result charts are not proposed without an actual ' 'experiment CSV. This module never fabricates results.\n') write_lf(out, ''.join(lines)) print(out) - print(f'figure_count={len(FIGURE_SPECS)}') - log(f'propose-figures 실행: {len(FIGURE_SPECS)}개 후보 제안') + print(f'figure_count={len(specs)}') + log(f'propose-figures 실행: {len(specs)}개 후보 제안 (spec_source={origin})') def cmd_render_figures(args): - fig_ids = [args.figure_id] if getattr(args, 'figure_id', None) else list(FIGURE_SPECS) - unknown = [f for f in fig_ids if f not in FIGURE_SPECS] + specs, invalid, origin = load_figure_specs() + fig_ids = [args.figure_id] if getattr(args, 'figure_id', None) else list(specs) + unknown = [f for f in fig_ids if f not in specs] if unknown: raise SystemExit(f'unknown figure id(s): {", ".join(unknown)}; ' - f'known: {", ".join(FIGURE_SPECS)}') - formats = tuple((getattr(args, 'formats', None) or 'svg,png').split(',')) - results = [render_figure(f, formats) for f in fig_ids] - ok = [r for r in results if not r['errors']] - failed = [r for r in results if r['errors']] + f'known: {", ".join(specs) or "(none)"}') + formats = tuple(f.strip() for f in + (getattr(args, 'formats', None) or 'svg,png').split(',') + if f.strip()) + strict = bool(getattr(args, 'strict', False)) + results = [render_figure(f, specs[f], formats) for f in fig_ids] + ok = [r for r in results if r['status'] == 'rendered'] + missing = [r for r in results if r['status'] == 'renderer_missing'] + failed = [r for r in results if r['status'] == 'error'] out = FIG_DIR / f'figure_render_report_{today()}.md' - lines = [f'# Figure Render Report {today()}\n\n'] + lines = [f'# Figure Render Report {today()}\n\n', + f'- spec_source: {origin}\n\n'] for r in results: lines.append(f'## {r["figure_id"]}\n\n') - lines.append(f'- dot: `{Path(r["dot"]).relative_to(ROOT)}`\n') - lines.append(f'- mmd: `{Path(r["mmd"]).relative_to(ROOT)}`\n') + lines.append(f'- kind: {r["kind"]}\n') + lines.append(f'- status: {r["status"]}\n') + for p in r['sources']: + lines.append(f'- source: `{relpath(p)}`\n') for p in r['rendered']: rp = Path(p) - lines.append(f'- rendered: `{rp.relative_to(ROOT)}` ({rp.stat().st_size} bytes)\n') + lines.append(f'- rendered: `{relpath(p)}` ({rp.stat().st_size} bytes)\n') for e in r['errors']: lines.append(f'- ERROR: {e}\n') + if r['status'] == 'renderer_missing': + name = RENDERERS[r['kind']] + lines.append(f'- WARN: `{name}` not found in PATH; sources written ' + f'only. To render: {RENDERER_INSTALL_HINTS[name]}\n') + lines.append('\n') + if invalid: + lines.append('## Invalid specs (skipped)\n\n') + lines += [f'- {reason}\n' for reason in invalid] lines.append('\n') - lines.append(f'- rendered_ok: {len(ok)}\n- failed: {len(failed)}\n') + lines.append(f'- rendered_ok: {len(ok)}\n' + f'- renderer_missing: {len(missing)}\n' + f'- failed: {len(failed)}\n' + f'- invalid_specs: {len(invalid)}\n') write_lf(out, ''.join(lines)) print(out) print(f'rendered_ok={len(ok)}') + print(f'renderer_missing={len(missing)}') print(f'failed={len(failed)}') - log(f'render-figures 실행: ok={len(ok)}, failed={len(failed)}') + for name in sorted({RENDERERS[r['kind']] for r in missing}): + print(f'WARN: renderer `{name}` not found in PATH; sources were ' + f'written but figures using it were not rendered. To fix: ' + f'{RENDERER_INSTALL_HINTS[name]}' + + ('' if strict else ' (use --strict to make this fatal)')) + log(f'render-figures 실행: ok={len(ok)}, renderer_missing={len(missing)}, ' + f'failed={len(failed)}') if failed: raise SystemExit(1) - - -def figure_block(fig_id, spec): - label = fig_id.replace('_', '-') - rel = f'figures/{fig_id}.svg' - return f'![{spec["caption"]}]({rel}){{#{label}}}\n' + if missing and strict: + raise SystemExit(1) def cmd_figure_placeholder_preview(args): - fig_ids = [args.figure_id] if getattr(args, 'figure_id', None) else list(FIGURE_SPECS) - unknown = [f for f in fig_ids if f not in FIGURE_SPECS] + specs, invalid, origin = load_figure_specs() + fig_ids = [args.figure_id] if getattr(args, 'figure_id', None) else list(specs) + unknown = [f for f in fig_ids if f not in specs] if unknown: raise SystemExit(f'unknown figure id(s): {", ".join(unknown)}') prefix = getattr(args, 'output_prefix', None) or f'figure_placeholder_preview_{today()}' rows = [] md = [f'# Figure Placeholder Preview {today()}\n\n'] for fig_id in fig_ids: - spec = FIGURE_SPECS[fig_id] - target = ROOT / spec['target_file'] + spec = specs[fig_id] + block = figure_block(fig_id, spec) + target_file = spec.get('target_file') or '' + target = (ROOT / target_file) if target_file else None status, reason = 'ready', '' - if not target.exists(): - status, reason = 'blocked', 'target file missing' + if not target_file: + status, reason = 'blocked', 'no_target_file' + elif not target.exists(): + status, reason = 'blocked', 'target_missing' else: - text = target.read_text(encoding='utf-8') - if spec['target_heading'] not in text: - status, reason = 'blocked', f'heading not found: {spec["target_heading"]}' - elif f'#{fig_id.replace("_", "-")}' in text: - status, reason = 'skipped', 'placeholder already present' + text = read_text_compat(target) + _, err = locate_heading(text, spec['target_heading']) + if err: + status, reason = 'blocked', err + elif placeholder_present(text, fig_id, block): + status, reason = 'skipped', 'already_applied' rows.append({ 'figure_id': fig_id, - 'target_file': spec['target_file'], + 'target_file': target_file, 'target_heading': spec['target_heading'], - 'target_sha256': file_sha256(target) if target.exists() else '', - 'insert_block': figure_block(fig_id, spec).strip(), + 'target_sha256': (file_sha256(target) + if target is not None and target.exists() else ''), + 'insert_block': block.strip(), 'status': status, 'reason': reason, }) - md.append(f'## {fig_id}\n\n- target: `{spec["target_file"]}` / ' + md.append(f'## {fig_id}\n\n- target: `{target_file or "(none)"}` / ' f'`{spec["target_heading"]}`\n- status: {status}' + (f' ({reason})' if reason else '') + '\n\n```\n' - + figure_block(fig_id, spec) + '```\n\n') + + block + '```\n\n') csv_path = ROOT / f'reports/review/{prefix}.csv' md_path = ROOT / f'reports/review/{prefix}.md' csv_path.parent.mkdir(parents=True, exist_ok=True) - with open(csv_path, 'w', encoding='utf-8', newline='') as f: - w = csv.DictWriter(f, fieldnames=list(rows[0].keys())) + with open(csv_path, 'w', encoding='utf-8-sig', newline='') as f: + w = csv.DictWriter(f, fieldnames=list(PREVIEW_FIELDS)) w.writeheader() w.writerows(rows) ready = len([r for r in rows if r['status'] == 'ready']) @@ -483,87 +532,175 @@ def cmd_figure_placeholder_preview(args): log(f'figure-placeholder-preview 실행: ready={ready}, blocked={blocked}, skipped={skipped}') +def plan_figure_apply(rows): + """Re-check every preview row against the current manuscripts (no writes). + + Returns (results, applied_ids, file_texts, modified_files, svg_copies): + results is a list of {figure_id, outcome: applied|blocked|skipped, + reason, detail} in row order; file_texts holds the post-insertion text + per target file. Heading anchoring is re-verified line-exact at apply + time so a preview approved by a human can never land in the wrong spot. + """ + results = [] + applied_ids = [] + file_texts = {} + file_shas = {} + modified_files = set() + svg_copies = [] + + def record(fig_id, outcome, reason='', detail=''): + results.append({'figure_id': fig_id, 'outcome': outcome, + 'reason': reason, 'detail': detail}) + + for r in rows: + fig_id = (r.get('figure_id') or '').strip() or '(missing id)' + status = (r.get('status') or '').strip() + if status != 'ready': + record(fig_id, 'skipped', 'not_ready', + f'preview status={status or "(empty)"}' + + (f' reason={r.get("reason")}' if r.get('reason') else '')) + continue + target_file = (r.get('target_file') or '').strip() + target = ROOT / target_file if target_file else None + if not target_file or not target.exists(): + record(fig_id, 'blocked', 'target_missing', target_file) + continue + if target_file not in file_texts: + file_texts[target_file] = read_text_compat(target) + file_shas[target_file] = file_sha256(target) + text = file_texts[target_file] + insert_block = (r.get('insert_block') or '').strip() + if placeholder_present(text, fig_id, insert_block): + record(fig_id, 'blocked', 'already_applied', + f'placeholder already present in {target_file}') + continue + if r.get('target_sha256') != file_shas[target_file]: + record(fig_id, 'blocked', 'sha_mismatch', + f'{target_file} changed since preview; regenerate preview') + continue + heading = (r.get('target_heading') or '').strip() + line_idx, err = locate_heading(text, heading) + if err: + record(fig_id, 'blocked', err, f'heading "{heading}" in {target_file}') + continue + if not insert_block: + record(fig_id, 'blocked', 'empty_insert_block', '') + continue + needs_svg = f'{fig_id}.svg' in insert_block + svg = FIG_DIR / f'{fig_id}.svg' + if needs_svg and not svg.exists(): + record(fig_id, 'blocked', 'svg_missing', + f'run render-figures first ({relpath(svg)})') + continue + lines = text.split('\n') + new_lines = (lines[:line_idx + 1] + [''] + insert_block.split('\n') + + lines[line_idx + 1:]) + file_texts[target_file] = '\n'.join(new_lines) + modified_files.add(target_file) + if needs_svg: + svg_copies.append((svg, MANUSCRIPT_FIG_DIR / svg.name)) + applied_ids.append(fig_id) + record(fig_id, 'applied', '', f'under "{heading}" in {target_file}') + return results, applied_ids, file_texts, modified_files, svg_copies + + def cmd_apply_figure_placeholder(args): preview = ROOT / args.from_preview if not preview.exists(): raise SystemExit(f'preview not found: {preview}') + preview_sha = file_sha256(preview) do_apply = bool(getattr(args, 'apply', False)) dry_run = bool(getattr(args, 'dry_run', False)) or not do_apply - with open(preview, encoding='utf-8') as f: - rows = list(csv.DictReader(f)) - ready = [r for r in rows if r.get('status') == 'ready'] - errors = [] - applied = [] - # SHA precheck on all targets before touching anything - for r in ready: - target = ROOT / r['target_file'] - if not target.exists(): - errors.append(f'{r["figure_id"]}: target missing') - continue - if file_sha256(target) != r['target_sha256']: - errors.append(f'{r["figure_id"]}: target SHA mismatch (regenerate preview)') - if errors: - for e in errors: - print(f'ERROR: {e}') - raise SystemExit(1) + rows = list(csv.DictReader(io.StringIO(read_text_compat(preview)))) + ready_count = len([r for r in rows if (r.get('status') or '') == 'ready']) + results, applied_ids, file_texts, modified_files, svg_copies = plan_figure_apply(rows) + blocked = [x for x in results if x['outcome'] == 'blocked'] + skipped = [x for x in results if x['outcome'] == 'skipped'] + fatal = [x for x in blocked if x['reason'] not in BENIGN_BLOCK_REASONS] + if dry_run: - for r in ready: - print(f'DRY-RUN would insert {r["figure_id"]} into {r["target_file"]} ' - f'after "{r["target_heading"]}"') - print(f'ready_rows={len(ready)}') - log(f'apply-figure-placeholder dry-run: ready={len(ready)}') + for x in results: + if x['outcome'] == 'applied': + print(f'DRY-RUN would insert {x["figure_id"]}: {x["detail"]}') + elif x['outcome'] == 'blocked': + print(f'DRY-RUN BLOCKED {x["figure_id"]}: {x["reason"]}' + + (f' ({x["detail"]})' if x['detail'] else '')) + else: + print(f'DRY-RUN skipped {x["figure_id"]}: {x["detail"]}') + print(f'ready_rows={ready_count}') + print(f'would_apply_rows={len(applied_ids)}') + print(f'blocked_rows={len(blocked)}') + print(f'skipped_rows={len(skipped)}') + print(f'preview_sha256={preview_sha}') + log(f'apply-figure-placeholder dry-run: would_apply={len(applied_ids)}, ' + f'blocked={len(blocked)}, skipped={len(skipped)}') + if fatal: + raise SystemExit(1) return - # Backup all target chapters first - stamp = datetime.now().strftime('%Y%m%d_%H%M%S') - backup_dir = ROOT / f'05_manuscript/backups/manuscript_before_figure_apply_{stamp}/chapters' - backup_dir.mkdir(parents=True, exist_ok=True) - for tf in sorted({r['target_file'] for r in ready}): - shutil.copy2(ROOT / tf, backup_dir / Path(tf).name) - # Copy rendered SVGs into manuscript figures dir - MANUSCRIPT_FIG_DIR.mkdir(parents=True, exist_ok=True) - for r in ready: - svg = FIG_DIR / f'{r["figure_id"]}.svg' - if not svg.exists(): - raise SystemExit(f'{r["figure_id"]}: rendered SVG missing; run render-figures first') - shutil.copy2(svg, MANUSCRIPT_FIG_DIR / svg.name) - # Insert blocks (one file may receive multiple figures) - by_file = {} - for r in ready: - by_file.setdefault(r['target_file'], []).append(r) - for tf, frs in by_file.items(): - target = ROOT / tf - text = target.read_text(encoding='utf-8') - for r in frs: - heading = r['target_heading'] - idx = text.index(heading) - # insert after the heading's paragraph block (after heading line + blank) - line_end = text.index('\n', idx) - insertion = '\n' + r['insert_block'] + '\n' - text = text[:line_end + 1] + insertion + text[line_end + 1:] - applied.append(r['figure_id']) - write_lf(target, text) + + # Commit phase: back up outside the guarded manuscript tree, copy rendered + # SVGs, then write the planned texts (LF). + backup_parent = None + if modified_files: + stamp = datetime.now().strftime('%Y%m%d_%H%M%S') + backup_parent = (ROOT / 'backups' / 'manuscript' + / f'manuscript_before_figure_apply_{stamp}') + backup_dir = backup_parent / 'chapters' + backup_dir.mkdir(parents=True, exist_ok=True) + for tf in sorted(modified_files): + shutil.copy2(ROOT / tf, backup_dir / Path(tf).name) + MANUSCRIPT_FIG_DIR.mkdir(parents=True, exist_ok=True) + for src, dst in svg_copies: + shutil.copy2(src, dst) + for tf in sorted(modified_files): + write_lf(ROOT / tf, file_texts[tf]) + report = ROOT / f'reports/review/figure_apply_{today()}.md' lines = [f'# Figure Apply Report {today()}\n\n## Summary\n', - '- mode: apply\n', '- applied: true\n', - f'- applied_rows: {len(applied)}\n', - f'- blocked_rows: 0\n', - f'- backup: `{backup_dir.parent.relative_to(ROOT)}`\n', - '\n## Applied figures\n'] - lines += [f'- {fid}\n' for fid in applied] - lines += ['\n## Errors\n- none\n'] + '- mode: apply\n', + f'- applied: {"true" if applied_ids else "false"}\n', + f'- applied_rows: {len(applied_ids)}\n', + f'- blocked_rows: {len(blocked)}\n', + f'- skipped_rows: {len(skipped)}\n', + f'- preview: `{relpath(preview)}`\n', + f'- preview_sha256: {preview_sha}\n', + '- backup: ' + (f'`{relpath(backup_parent)}`' if backup_parent else 'none') + '\n', + '\n## Row results\n'] + for x in results: + line = f'- {x["figure_id"]}: {x["outcome"]}' + if x['reason']: + line += f' ({x["reason"]})' + if x['detail']: + line += f': {x["detail"]}' + lines.append(line + '\n') + lines.append('\n## Applied figures\n') + lines += [f'- {fid}\n' for fid in applied_ids] or ['- none\n'] write_lf(report, ''.join(lines)) print(report) - print(f'applied_rows={len(applied)}') - print('blocked_rows=0') - log(f'apply-figure-placeholder apply: applied={len(applied)}, backup={backup_dir}') + print(f'applied_rows={len(applied_ids)}') + print(f'blocked_rows={len(blocked)}') + print(f'skipped_rows={len(skipped)}') + print(f'preview_sha256={preview_sha}') + for x in blocked: + print(f'BLOCKED {x["figure_id"]}: {x["reason"]}' + + (f' ({x["detail"]})' if x['detail'] else '')) + log(f'apply-figure-placeholder apply: applied={len(applied_ids)}, ' + f'blocked={len(blocked)}, skipped={len(skipped)}' + + (f', backup={backup_parent}' if backup_parent else '')) + if fatal: + raise SystemExit(1) def register_subcommands(sub): p = sub.add_parser('propose-figures', help='List spec-driven figure candidates') p.set_defaults(func=cmd_propose_figures) - p = sub.add_parser('render-figures', help='Write .dot/.mmd sources and render via Graphviz') + p = sub.add_parser('render-figures', + help='Write .dot/.mmd sources and render via Graphviz/mermaid-cli') p.add_argument('--figure-id') p.add_argument('--formats', default='svg,png') + p.add_argument('--strict', action='store_true', + help='exit 1 when a renderer (dot/mmdc) is missing instead of ' + 'degrading to sources-only output') p.set_defaults(func=cmd_render_figures) p = sub.add_parser('figure-placeholder-preview', help='Guarded preview for inserting figure references') diff --git a/scripts/run_pipeline.py b/scripts/run_pipeline.py index f00e47b..a907b06 100644 --- a/scripts/run_pipeline.py +++ b/scripts/run_pipeline.py @@ -1,53 +1,57 @@ -import subprocess -import sys +# -*- coding: utf-8 -*- +"""One-click PaperOps automation: collect -> score -> screen -> gap -> download -> cards -> figures. + +Runs ONLY the automatic stages. Human gates (review-evidence-candidates, promote-evidence, +apply-manuscript-patch, verify-evidence) are never invoked from here, by design. + +Output is plain ASCII on purpose: emoji output crashed with UnicodeEncodeError whenever +stdout was redirected on cp949 Windows (run_pipeline.bat > log.txt, Task Scheduler, CI). +""" import argparse import os +import subprocess +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] + + +def run_command(args, check=True): + cmd = [sys.executable, str(ROOT / 'scripts' / 'paperops.py')] + args + print('[RUN] paperops ' + ' '.join(args), flush=True) + env = dict(os.environ) + env.setdefault('PYTHONIOENCODING', 'utf-8') + result = subprocess.run(cmd, cwd=str(ROOT), env=env) + if result.returncode != 0: + if check: + print('[FAIL] pipeline stopped at: ' + ' '.join(args), flush=True) + sys.exit(1) + print('[WARN] non-fatal step failed: ' + ' '.join(args), flush=True) + return result.returncode -def run_command(cmd, step_name): - print(f"\n{'='*60}") - print(f"🚀 [STEP] {step_name}") - print(f"💻 Executing: {' '.join(cmd)}") - print(f"{'='*60}\n") - try: - subprocess.run(cmd, check=True) - except subprocess.CalledProcessError as e: - print(f"\n❌ Pipeline failed at step: {step_name}") - print(f"Error details: {e}") - sys.exit(1) def main(): - parser = argparse.ArgumentParser(description="PaperOps Master Pipeline - Run the entire automation flow sequentially.") - parser.add_argument("--limit", type=int, default=20, help="Limit for collect and other commands.") - parser.add_argument("--skip-downloads", action="store_true", help="Skip the PDF download step to save time.") + parser = argparse.ArgumentParser(description='PaperOps one-click automatic pipeline') + parser.add_argument('--limit', type=int, default=20, help='per-source collection limit') + parser.add_argument('--skip-downloads', action='store_true', help='skip PDF downloads') + parser.add_argument('--skip-figures', action='store_true', help='skip figure rendering') args = parser.parse_args() - # Determine paths - script_dir = os.path.dirname(os.path.abspath(__file__)) - paperops_script = os.path.join(script_dir, "paperops.py") + print('[START] PaperOps automatic pipeline (human gates are NOT run here)', flush=True) + run_command(['collect', '--limit', str(args.limit)]) + run_command(['score']) + run_command(['screen', '--limit', str(args.limit * 2)]) + run_command(['gap']) + if not args.skip_downloads: + run_command(['download-pdfs', '--limit', str(args.limit)]) + run_command(['cards']) + if not args.skip_figures: + # Figure rendering degrades gracefully without Graphviz (sources still written), + # and a real renderer failure should not erase the collection work above. + run_command(['render-figures'], check=False) + print('[OK] automatic pipeline finished', flush=True) + print('[NEXT] human gates: review-evidence-candidates -> promote-evidence -> verify-evidence', flush=True) - # Sequence of pipeline commands based on PaperOps standard flow - pipeline_steps = [ - ([sys.executable, paperops_script, "collect", "--limit", str(args.limit)], "Collect Papers"), - ([sys.executable, paperops_script, "score"], "Score Papers"), - ([sys.executable, paperops_script, "screen", "--limit", str(args.limit * 2)], "Screen Papers"), - ([sys.executable, paperops_script, "gap"], "Find Research Gaps"), - ] - if not args.skip_downloads: - pipeline_steps.append(([sys.executable, paperops_script, "download-pdfs", "--limit", str(args.limit)], "Download PDFs")) - - pipeline_steps.extend([ - ([sys.executable, paperops_script, "cards"], "Generate Paper Cards"), - ([sys.executable, paperops_script, "render-figures"], "Render Figures"), - ]) - - print("🌟 Starting PaperOps Master Pipeline...") - for cmd, step_name in pipeline_steps: - run_command(cmd, step_name) - - print(f"\n{'='*60}") - print("✅ Pipeline execution completed successfully!") - print(f"{'='*60}\n") - -if __name__ == "__main__": +if __name__ == '__main__': main() diff --git a/tests/__init__.py b/tests/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/test_draft_audit.py b/tests/test_draft_audit.py new file mode 100644 index 0000000..93daae0 --- /dev/null +++ b/tests/test_draft_audit.py @@ -0,0 +1,669 @@ +# -*- coding: utf-8 -*- +"""Tests for scripts/paperops_draft_audit.py (audit-manuscript-draft). + +Run from the repo root with: python -m pytest tests/test_draft_audit.py +Stdlib + pytest only; no network. The module is imported via importlib from +scripts/ (no package install). All reports/logs are redirected to tmp_path so +the repo's data/, logs/ and reports/ directories are never touched. +""" +from __future__ import annotations + +import argparse +import importlib.util +import zipfile +from pathlib import Path +from types import SimpleNamespace + +import pytest + +REPO = Path(__file__).resolve().parents[1] +MODULE_PATH = REPO / 'scripts' / 'paperops_draft_audit.py' +NO_CONFIG = '/nonexistent/draft_audit_config_for_tests.yaml' +W = 'http://schemas.openxmlformats.org/wordprocessingml/2006/main' + +_MOD = None + + +def load_module(): + global _MOD + if _MOD is None: + spec = importlib.util.spec_from_file_location( + 'paperops_draft_audit_under_test', MODULE_PATH) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + _MOD = mod + return _MOD + + +def neutral_detectors(mod): + return mod.build_detectors(mod.load_config(NO_CONFIG)) + + +@pytest.fixture +def env(tmp_path, monkeypatch): + mod = load_module() + # Never append to the repo's logs/ACTIVITY_LOG.md from tests. + monkeypatch.setattr(mod, 'LOG', tmp_path / 'ACTIVITY_LOG.md') + + def run(draft_path, **kw): + kw.setdefault('report_dir', str(tmp_path / 'reports')) + kw.setdefault('experiment_data', str(tmp_path / 'no_experiment_data')) + kw.setdefault('config_path', NO_CONFIG) + return mod.audit_draft(Path(draft_path), **kw) + + def write_draft(name, text, encoding='utf-8'): + p = tmp_path / name + p.write_bytes(text.encode(encoding)) + return p + + return SimpleNamespace(mod=mod, run=run, tmp=tmp_path, + write_draft=write_draft) + + +def finding_types(result): + return [f[1] for f in result['findings']] + + +# --------------------------------------------------------------------------- +# docx fixture helpers (built in-test with zipfile, stdlib only) +# --------------------------------------------------------------------------- + +def w_text_p(text, style=None): + ppr = f'' if style else '' + return (f'{ppr}{text}' + '') + + +def make_docx(path, body_paras, footnotes_xml=None, endnotes_xml=None): + doc = ('' + f'' + + ''.join(body_paras) + '') + with zipfile.ZipFile(path, 'w') as z: + z.writestr('word/document.xml', doc) + if footnotes_xml: + z.writestr('word/footnotes.xml', footnotes_xml) + if endnotes_xml: + z.writestr('word/endnotes.xml', endnotes_xml) + return path + + +FOOTNOTES_XML = ( + '' + f'' + '' + '' + '' + '이 각주의 방법은 완벽하다.' + '' + '') + + +# --------------------------------------------------------------------------- +# 4a. Citations count as sources (Korean, English, bracket forms) +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize('cite', [ + '(김철수, 2020)', + '(김철수 외, 2020)', + '(김철수 등, 2020)', + '(Kim et al., 2020)', + '(Noy, 2001)', + '[1]', + '[1,2]', + '[1-3]', +]) +def test_citation_counts_as_source(cite): + mod = load_module() + det = neutral_detectors(mod) + sent = f'이 접근은 항상 우수한 결과를 낸다 {cite}.' + assert mod.sentence_has_source(sent, det), cite + + +def test_uncited_strong_claim_is_flagged(env): + draft = env.write_draft('a.md', '이 접근은 항상 우수한 결과를 낸다.\n') + result = env.run(draft) + assert 'strong_claim_no_source' in finding_types(result) + + +def test_cited_strong_claim_not_flagged(env): + draft = env.write_draft( + 'b.md', '이 접근은 항상 우수한 결과를 낸다 (김철수 외, 2020).\n') + result = env.run(draft) + assert 'strong_claim_no_source' not in finding_types(result) + + +# --------------------------------------------------------------------------- +# 4b. English strong-claim and overclaim detection +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize('sent', [ + 'This proves the theorem in all cases.', + 'Our approach guarantees convergence.', + 'The method always improves accuracy.', + 'The system never fails under load.', + 'This design ensures correctness.', + 'The evaluation demonstrates conclusively the benefit.', +]) +def test_english_strong_claims(sent): + mod = load_module() + det = neutral_detectors(mod) + assert mod.find_strong_claim(sent, det) is not None, sent + + +@pytest.mark.parametrize('sent', [ + 'Our system is state-of-the-art.', + 'Our system is State-of-the-Art in this domain.', + 'The model achieves SOTA performance.', + 'This is a world-first implementation.', + 'It is a best-in-class solution.', + 'The results are perfect.', + 'We propose a novel framework for governance.', +]) +def test_english_overclaims(sent): + mod = load_module() + det = neutral_detectors(mod) + assert mod.find_overclaim(sent, det) is not None, sent + + +def test_lowercase_sota_not_overclaim(): + # SOTA is matched case-sensitively so prose words are not caught. + mod = load_module() + det = neutral_detectors(mod) + assert mod.find_overclaim('the sota of this village is old.', det) is None + + +def test_korean_detectors_still_work(env): + draft = env.write_draft( + 'k.md', + '본 시스템은 완벽한 해결책이다.\n\n' + '이 구조는 성능을 극대화한다.\n') + result = env.run(draft) + types = finding_types(result) + assert 'overclaim' in types + assert 'strong_claim_no_source' in types + + +# --------------------------------------------------------------------------- +# 4c. Hypothetical suppression fixes +# --------------------------------------------------------------------------- + +def test_damyeonjeok_is_not_hypothetical(env): + mod = env.mod + sent = '다면적 특성을 고려하여 정확도 95.5%를 기록했다.' + assert not mod.is_hypothetical(sent) + draft = env.write_draft('h1.md', sent + '\n') + result = env.run(draft) + assert 'numeric_needs_data' in finding_types(result) + + +def test_true_conditional_is_hypothetical(env): + mod = env.mod + sent = '만약 정확도가 99%가 된다면 상용화가 가능할 것이다.' + assert mod.is_hypothetical(sent) + draft = env.write_draft('h2.md', sent + '\n') + result = env.run(draft) + types = finding_types(result) + assert 'numeric_hypothetical_ok' in types + assert 'numeric_needs_data' not in types + + +def test_iramyeon_is_hypothetical(): + mod = load_module() + assert mod.is_hypothetical('전문가가 아닌 학생이라면 어려울 것이다.') + + +def test_ramyeon_noun_is_not_hypothetical(): + # '라면' as a standalone noun (ramen) is not a conditional ending. + mod = load_module() + assert not mod.is_hypothetical('라면 시장은 2조 원 규모로 성장했다.') + + +def test_hal_su_itda_is_not_hypothetical(env): + # Capability claims still need sources / data. + mod = env.mod + sent = '제안 기법은 95% 정확도를 달성할 수 있다.' + assert not mod.is_hypothetical(sent) + draft = env.write_draft('h3.md', sent + '\n') + result = env.run(draft) + assert 'numeric_needs_data' in finding_types(result) + + +# --------------------------------------------------------------------------- +# 4d. Numeric extraction: integers with units, year/list-index exclusions +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize('sent,expected_raw', [ + ('정확도는 95%로 향상되었다.', '95%'), + ('데이터 1000개를 수집했다.', '1000개'), + ('오류 건수는 12건이었다.', '12건'), + ('속도가 3배 개선되었다.', '3배'), + ('평가에서 10점을 받았다.', '10점'), + ('처리량이 2x 증가했다.', '2x'), + ('throughput improved 5 times overall.', '5times'), + ('전체 1,234개를 분석했다.', '1,234개'), + ('F1 = 0.87', '0.87'), + ('정확도는 95.5%로 나타났다.', '95.5%'), +]) +def test_numeric_tokens_extracted(sent, expected_raw): + mod = load_module() + raws = [t['raw'].replace(' ', '') for t in mod.extract_numeric_tokens(sent)] + assert expected_raw in raws, (sent, raws) + + +@pytest.mark.parametrize('sent', [ + '본 연구는 2020년에 시작되었다.', # standalone year + '1995 이후의 문헌을 검토했다.', # bare year, no unit + '1. 서론', # list index / numbering + '1.2 연구 방법', # leading section number + '그림 2.1 참조', # figure reference + '표 4.2에서 제시한다.', # table reference + '3.2절에서 상세히 설명한다.', # section reference + 'Python v3.10 환경에서 실행했다.', # version number +]) +def test_numeric_non_results_excluded(sent): + mod = load_module() + assert mod.extract_numeric_tokens(sent) == [], sent + + +def test_integer_percent_flagged_in_audit(env): + draft = env.write_draft('n1.md', '정확도는 95%로 향상되었다.\n') + result = env.run(draft) + assert 'numeric_needs_data' in finding_types(result) + + +def test_integer_count_flagged_in_audit(env): + draft = env.write_draft('n2.md', '데이터 1000개를 수집했다.\n') + result = env.run(draft) + assert 'numeric_needs_data' in finding_types(result) + + +# --------------------------------------------------------------------------- +# 4e. Structure check is heading-anchored +# --------------------------------------------------------------------------- + +def test_structure_keyword_in_body_prose_does_not_count(env): + draft = env.write_draft( + 's1.md', + '서론에서 논의한 바와 같이 연구의 배경을 상세히 설명하며 ' + '결론 부분의 시사점도 함께 검토한다.\n') + result = env.run(draft) + structure = dict(result['structure']) + assert structure['서론'] is False + assert structure['결론'] is False + + +def test_structure_md_heading_counts(env): + draft = env.write_draft( + 's2.md', + '# 서론\n\n연구 배경을 설명한다.\n\n# 결론\n\n요약한다.\n') + result = env.run(draft) + structure = dict(result['structure']) + assert structure['서론'] is True + assert structure['결론'] is True + + +def test_structure_korean_chapter_line_fallback(env): + # md without '#' headings falls back to short 제N장-style lines. + draft = env.write_draft( + 's3.md', '제1장 서론\n\n연구 배경을 설명한다.\n') + result = env.run(draft) + structure = dict(result['structure']) + assert structure['서론'] is True + + +def test_structure_docx_heading_style(env): + path = env.tmp / 's4.docx' + make_docx(path, [ + w_text_p('서론', style='Heading1'), + w_text_p('여기 본문에는 결론이라는 단어가 들어 있지만 이것은 장 제목이 아니라 긴 본문 문장이다.'), + ]) + result = env.run(path) + structure = dict(result['structure']) + assert structure['서론'] is True + assert structure['결론'] is False + + +# --------------------------------------------------------------------------- +# 4f. Minimum sentence length dropped to 4 chars +# --------------------------------------------------------------------------- + +def test_short_overclaim_sentence_is_seen(env): + draft = env.write_draft('short.md', '완벽하다.\n') + result = env.run(draft) + assert 'overclaim' in finding_types(result) + assert result['sentences'] >= 1 + + +# --------------------------------------------------------------------------- +# 3. docx parsing: tabs/breaks, footnotes, footnote refs, BadZipFile +# --------------------------------------------------------------------------- + +def test_docx_tab_and_br_become_spaces(env): + path = env.tmp / 'tab.docx' + tab_para = (f'12' + '34') + br_para = ('56' + '78') + make_docx(path, [tab_para, br_para]) + paras, _headings, _notes = env.mod.load_draft(path) + texts = [p['text'] for p in paras] + assert '12 34' in texts + assert '56 78' in texts + assert not any('1234' in t or '5678' in t for t in texts) + + +def test_docx_footnote_text_is_scanned_with_location(env): + path = env.tmp / 'foot.docx' + make_docx(path, [w_text_p('본문 단락이다. 여기에는 문제가 없다.')], + footnotes_xml=FOOTNOTES_XML) + result = env.run(path) + over = [f for f in result['findings'] if f[1] == 'overclaim'] + assert over, result['findings'] + assert any(f[4] == 'footnote:2' for f in over) + + +def test_docx_footnote_reference_counts_as_source(env): + path = env.tmp / 'ref.docx' + with_ref = ('본 시스템의 성능은 탁월하다.' + '') + without_ref = w_text_p('이 구조의 성능은 월등하다.') + make_docx(path, [with_ref, without_ref], footnotes_xml=FOOTNOTES_XML) + result = env.run(path) + flagged = [f[3] for f in result['findings'] + if f[1] == 'strong_claim_no_source'] + assert not any('탁월하다' in s for s in flagged) + assert any('월등하다' in s for s in flagged) + + +def test_docx_bad_zip_raises_clean_systemexit(env): + path = env.tmp / 'broken.docx' + path.write_bytes(b'this is definitely not a zip archive') + with pytest.raises(SystemExit) as exc: + env.mod.load_draft(path) + msg = str(exc.value) + assert 'zip' in msg.lower() + assert 'broken.docx' in msg + + +# --------------------------------------------------------------------------- +# 2. Encoding safety +# --------------------------------------------------------------------------- + +def test_cp949_draft_is_read_correctly(env): + draft = env.write_draft( + 'cp949.md', '# 서론\n\n본 시스템은 완벽하다.\n', encoding='cp949') + result = env.run(draft) + over = [f for f in result['findings'] if f[1] == 'overclaim'] + assert over + assert '완벽하다' in over[0][3] # Korean survived the decode + assert any('cp949' in n for n in result['notes']) + + +def test_undecodable_draft_warns_prominently(env): + p = env.tmp / 'garbage.md' + p.write_bytes(b'\xff\xff\xff broken bytes but 95% still visible.\n') + result = env.run(p) + md_text = result['md'].read_text(encoding='utf-8') + assert 'WARNING' in md_text + assert 'garbage.md' in md_text + assert 'replace' in md_text + warnings = [n for n in result['notes'] if n.startswith('WARNING')] + assert warnings + + +def test_cp949_experiment_file_is_indexed(env): + exp = env.tmp / 'exp_kr' + exp.mkdir() + (exp / 'kor.csv').write_bytes('항목,값\n정확도,95.5\n'.encode('cp949')) + draft = env.write_draft('enc.md', '정확도는 95.5%로 나타났다.\n') + result = env.run(draft, experiment_data=str(exp)) + assert result['numeric_summary']['matched_exact'] == 1 + + +def test_csv_reports_are_utf8_sig(env): + exp = env.tmp / 'exp_bom' + exp.mkdir() + (exp / 'r.csv').write_text('95.5\n', encoding='utf-8') + draft = env.write_draft( + 'bom.md', '정확도는 95.5%이고 완벽하다.\n') + result = env.run(draft, experiment_data=str(exp)) + assert result['csv'].read_bytes().startswith(b'\xef\xbb\xbf') + assert result['numeric_csv'].read_bytes().startswith(b'\xef\xbb\xbf') + + +# --------------------------------------------------------------------------- +# 1. Numeric cross-check against experiment outputs +# --------------------------------------------------------------------------- + +@pytest.fixture +def crosscheck(env): + exp = env.tmp / 'experiment_outputs' + exp.mkdir() + (exp / 'results.csv').write_text( + 'metric,value\naccuracy,95.5\ncount,1000\nf1,0.874\n', + encoding='utf-8') + draft = env.write_draft( + 'cc.md', + '# 결과\n\n' + '정확도는 95.5%로 나타났다.\n\n' + '전체 데이터는 1,000개였다.\n\n' + 'F1 = 0.87 수준이었다.\n\n' + '오류율은 77.7%로 관찰되었다.\n') + result = env.run(draft, experiment_data=str(exp)) + return SimpleNamespace(env=env, result=result) + + +def test_crosscheck_statuses(crosscheck): + rows = {r[0]: r[2] for r in crosscheck.result['numeric_rows']} + assert rows['95.5%'] == 'matched_exact' + assert rows['1,000개'] == 'matched_exact' + assert rows['0.87'] == 'matched_rounded' + assert rows['77.7%'] == 'not_found_in_data' + + +def test_crosscheck_summary_and_report(crosscheck): + summary = crosscheck.result['numeric_summary'] + assert summary == {'checked': 4, 'matched_exact': 2, + 'matched_rounded': 1, 'not_found_in_data': 1} + md_text = crosscheck.result['md'].read_text(encoding='utf-8') + assert 'numeric_checked: 4' in md_text + assert 'matched_exact: 2' in md_text + assert 'matched_rounded: 1' in md_text + assert 'not_found_in_data: 1' in md_text + assert 'alignment' in md_text # governance: alignment, not truth + + +def test_crosscheck_matching_file_recorded(crosscheck): + matched = [r for r in crosscheck.result['numeric_rows'] + if r[2] == 'matched_exact'] + assert all(r[3] == 'results.csv' for r in matched) + missing = [r for r in crosscheck.result['numeric_rows'] + if r[2] == 'not_found_in_data'] + assert all(r[3] == '' for r in missing) + + +def test_crosscheck_never_marks_verified(crosscheck): + numeric_csv = crosscheck.result['numeric_csv'].read_text( + encoding='utf-8-sig') + assert 'verified' not in numeric_csv.lower() + statuses = {r[2] for r in crosscheck.result['numeric_rows']} + assert statuses <= {'matched_exact', 'matched_rounded', + 'not_found_in_data'} + + +def test_crosscheck_missing_number_becomes_finding(crosscheck): + types = finding_types(crosscheck.result) + assert 'numeric_not_found_in_data' in types + assert 'numeric_needs_data' not in types # data present + + +def test_rounded_match_uses_draft_precision(env): + exp = env.tmp / 'exp_round' + exp.mkdir() + (exp / 'nums.txt').write_text('999.6\n', encoding='utf-8') + draft = env.write_draft('round.md', '총 1000개를 처리했다.\n') + result = env.run(draft, experiment_data=str(exp)) + rows = result['numeric_rows'] + assert len(rows) == 1 + assert rows[0][2] == 'matched_rounded' + + +def test_experiment_dir_absent_falls_back_to_flags(env): + draft = env.write_draft('nodata.md', '정확도는 95.5%로 나타났다.\n') + result = env.run(draft, experiment_data=str(env.tmp / 'does_not_exist')) + assert 'numeric_needs_data' in finding_types(result) + md_text = result['md'].read_text(encoding='utf-8') + assert ('experiment data dir not found — ' + 'numeric values flagged for manual check') in md_text + assert result['numeric_csv'] is None + + +def test_experiment_dir_empty_falls_back_to_flags(env): + empty = env.tmp / 'empty_exp' + empty.mkdir() + draft = env.write_draft('empty.md', '정확도는 95.5%로 나타났다.\n') + result = env.run(draft, experiment_data=str(empty)) + assert 'numeric_needs_data' in finding_types(result) + assert not result['have_data'] + + +# --------------------------------------------------------------------------- +# 5. Config externalization +# --------------------------------------------------------------------------- + +def test_missing_config_uses_neutral_defaults(): + mod = load_module() + cfg = mod.load_config(NO_CONFIG) + assert cfg['source_mentions'] == [] + names = [ch['name'] for ch in cfg['required_chapters']] + assert '서론' in names and '결론' in names + + +def test_custom_config_overrides(env): + pytest.importorskip('yaml') + cfg_path = env.tmp / 'my_audit.yaml' + cfg_path.write_text( + "source_mentions: ['MySpecialCorpus']\n" + 'required_chapters:\n' + ' - name: 고유챕터\n' + " patterns: ['고유\\s*챕터']\n" + "overclaim_extra: ['초울트라']\n" + "strong_claim_extra: ['궁극의 해법']\n", + encoding='utf-8') + draft = env.write_draft( + 'cfg.md', + '# 고유챕터\n\n' + 'MySpecialCorpus 기반으로 항상 우수한 결과를 낸다.\n\n' + '이 방식은 초울트라 성능을 보인다.\n\n' + '이것이 궁극의 해법 이라 주장한다.\n') + result = env.run(draft, config_path=str(cfg_path)) + structure = dict(result['structure']) + assert structure == {'고유챕터': True} + types = finding_types(result) + assert 'overclaim' in types # via overclaim_extra + assert 'strong_claim_no_source' in types # via strong_claim_extra + flagged = [f[3] for f in result['findings'] + if f[1] == 'strong_claim_no_source'] + # the whitelisted source suppressed the '항상' strong claim + assert not any('MySpecialCorpus' in s for s in flagged) + + +def test_shipped_config_preserves_author_values(): + pytest.importorskip('yaml') + mod = load_module() + cfg = mod.load_config(REPO / 'config' / 'draft_audit.yaml') + assert 'FEEKG' in cfg['source_mentions'] + names = [ch['name'] for ch in cfg['required_chapters']] + assert '아티팩트/온톨로지 설계' in names + assert len(names) == 7 + det = mod.build_detectors(cfg) + assert mod.sentence_has_source('FEEKG 접근을 따라 항상 우수하다.', det) + assert mod.find_overclaim('본 시스템은 hallucination을 제거한다.', det) + + +# --------------------------------------------------------------------------- +# 6. Relative --input resolves against the caller's CWD +# --------------------------------------------------------------------------- + +def test_relative_input_resolves_against_cwd(env, monkeypatch): + monkeypatch.chdir(env.tmp) + env.write_draft('mydraft.md', '# 서론\n\n배경을 설명한다.\n') + args = argparse.Namespace( + input='mydraft.md', output_prefix='relcheck', + experiment_data=str(env.tmp / 'no_data'), + report_dir=str(env.tmp / 'rep'), config=NO_CONFIG) + env.mod.cmd_audit_manuscript_draft(args) + assert (env.tmp / 'rep' / 'relcheck.md').exists() + assert (env.tmp / 'rep' / 'relcheck.csv').exists() + + +def test_absolute_input_still_works(env): + draft = env.write_draft('abs.md', '# 서론\n\n배경을 설명한다.\n') + result = env.run(draft.resolve()) + assert result['md'].exists() + + +def test_missing_input_raises_systemexit(env, monkeypatch): + monkeypatch.chdir(env.tmp) + args = argparse.Namespace( + input='never_there.md', output_prefix=None, + experiment_data=None, report_dir=str(env.tmp / 'rep'), + config=NO_CONFIG) + with pytest.raises(SystemExit) as exc: + env.mod.cmd_audit_manuscript_draft(args) + assert 'input not found' in str(exc.value) + + +# --------------------------------------------------------------------------- +# CLI integration: register_subcommands keeps command name and flags +# --------------------------------------------------------------------------- + +def test_register_subcommands_end_to_end(env, monkeypatch): + monkeypatch.chdir(env.tmp) + exp = env.tmp / 'exp_cli' + exp.mkdir() + (exp / 'out.csv').write_text('95.5\n', encoding='utf-8') + draft = env.write_draft( + 'cli.md', '# 서론\n\n정확도는 95.5%로 나타났다.\n') + before = draft.read_bytes() + + ap = argparse.ArgumentParser() + sub = ap.add_subparsers(dest='cmd') + env.mod.register_subcommands(sub) + args = ap.parse_args([ + 'audit-manuscript-draft', + '--input', 'cli.md', + '--output-prefix', 'e2e', + '--experiment-data', str(exp), + '--report-dir', str(env.tmp / 'cli_reports'), + '--config', NO_CONFIG, + ]) + args.func(args) + + out = env.tmp / 'cli_reports' + assert (out / 'e2e.md').exists() + assert (out / 'e2e.csv').exists() + assert (out / 'e2e_numeric.csv').exists() + # the audit never edits the draft + assert draft.read_bytes() == before + + +def test_legacy_flags_still_parse(env): + # pre-v0.2 invocation shape must keep working (parse only) + ap = argparse.ArgumentParser() + sub = ap.add_subparsers(dest='cmd') + env.mod.register_subcommands(sub) + args = ap.parse_args(['audit-manuscript-draft', + '--input', 'x.docx', '--output-prefix', 'p']) + assert args.input == 'x.docx' + assert args.output_prefix == 'p' + assert args.func is env.mod.cmd_audit_manuscript_draft + + +def test_audit_never_modifies_docx_draft(env): + path = env.tmp / 'keep.docx' + make_docx(path, [w_text_p('본 시스템은 완벽하다.')], + footnotes_xml=FOOTNOTES_XML) + before = path.read_bytes() + env.run(path) + assert path.read_bytes() == before diff --git a/tests/test_figures.py b/tests/test_figures.py new file mode 100644 index 0000000..6733362 --- /dev/null +++ b/tests/test_figures.py @@ -0,0 +1,534 @@ +# -*- coding: utf-8 -*- +"""Tests for scripts/paperops_figures.py (spec-driven figure pipeline). + +All filesystem-writing tests run against tmp_path trees: the module's path +globals (ROOT, FIG_DIR, ...) are monkeypatched per test, so nothing is ever +written into the repository's 05_manuscript/, reports/, or data/ directories. +No real `dot` or `mmdc` binary is required. +""" +from __future__ import annotations +import argparse +import codecs +import csv +import importlib.util +import shutil as real_shutil +import types +from pathlib import Path + +import pytest + +REPO = Path(__file__).resolve().parents[1] +MODULE_PATH = REPO / 'scripts' / 'paperops_figures.py' + + +def _load_module(): + spec = importlib.util.spec_from_file_location('paperops_figures_under_test', + MODULE_PATH) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +@pytest.fixture() +def mod(tmp_path, monkeypatch): + """Fresh module instance with every path global redirected to tmp_path.""" + m = _load_module() + monkeypatch.setattr(m, 'ROOT', tmp_path) + monkeypatch.setattr(m, 'FIG_DIR', tmp_path / 'reports' / 'figures') + monkeypatch.setattr(m, 'FIG_SRC_DIR', tmp_path / 'reports' / 'figures' / 'src') + monkeypatch.setattr(m, 'MANUSCRIPT_FIG_DIR', tmp_path / '05_manuscript' / 'figures') + monkeypatch.setattr(m, 'LOG', tmp_path / 'logs' / 'ACTIVITY_LOG.md') + return m + + +CHAPTER = '05_manuscript/chapters/ch_test.qmd' + +CONFIG_ONE = """\ +figures: + - id: fig_alpha + kind: dot + caption: Alpha caption. + target_file: 05_manuscript/chapters/ch_test.qmd + target_heading: '## PaperOps Architecture' + source: | + digraph fig_alpha { a -> b; } +""" + +CONFIG_TWO = CONFIG_ONE + """\ + - id: fig_beta + kind: dot + caption: Beta caption. + target_file: 05_manuscript/chapters/ch_test.qmd + target_heading: '## Metrics' + source: | + digraph fig_beta { b -> c; } +""" + +# The nasty real-world layout: a longer ### heading that CONTAINS the target +# as a substring, plus the target inside a fenced code block, plus the one +# real target heading. +CHAPTER_TRICKY = """\ +--- +title: Test chapter +--- + +### PaperOps Architecture Details + +Prose about details. + +```text +## PaperOps Architecture +``` + +## PaperOps Architecture + +Intro paragraph. + +## Metrics + +Numbers prose. +""" + +ALPHA_BLOCK = '![Alpha caption.](figures/fig_alpha.svg){#fig-alpha}' +BETA_BLOCK = '![Beta caption.](figures/fig_beta.svg){#fig-beta}' + + +def setup_env(m, config_text=CONFIG_ONE, chapter_text=CHAPTER_TRICKY): + (m.ROOT / 'config').mkdir(parents=True, exist_ok=True) + (m.ROOT / 'config' / 'figures.yaml').write_text(config_text, encoding='utf-8') + if chapter_text is not None: + target = m.ROOT / CHAPTER + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(chapter_text, encoding='utf-8', newline='\n') + + +def fake_svg(m, fig_id): + m.FIG_DIR.mkdir(parents=True, exist_ok=True) + (m.FIG_DIR / f'{fig_id}.svg').write_text('', encoding='utf-8') + + +def run_preview(m, prefix='pv', figure_id=None): + args = argparse.Namespace(figure_id=figure_id, output_prefix=prefix) + m.cmd_figure_placeholder_preview(args) + return m.ROOT / f'reports/review/{prefix}.csv' + + +def run_apply(m, preview_rel='reports/review/pv.csv', apply=True, dry_run=False): + args = argparse.Namespace(from_preview=preview_rel, apply=apply, dry_run=dry_run) + m.cmd_apply_figure_placeholder(args) + + +def read_rows(csv_path): + with open(csv_path, encoding='utf-8-sig', newline='') as f: + return list(csv.DictReader(f)) + + +def write_rows(m, csv_path, rows): + with open(csv_path, 'w', encoding='utf-8-sig', newline='') as f: + w = csv.DictWriter(f, fieldnames=list(m.PREVIEW_FIELDS)) + w.writeheader() + w.writerows(rows) + + +def chapter_text(m): + return (m.ROOT / CHAPTER).read_text(encoding='utf-8') + + +# --------------------------------------------------------------------------- +# Heading anchoring +# --------------------------------------------------------------------------- + +def test_heading_match_is_line_anchored_not_substring(mod): + matches = mod.heading_match_lines(CHAPTER_TRICKY, '## PaperOps Architecture') + lines = CHAPTER_TRICKY.split('\n') + assert len(matches) == 1 + assert lines[matches[0]] == '## PaperOps Architecture' + # The ### heading contains the target as a substring but must not match. + assert '### PaperOps Architecture Details' not in [lines[i] for i in matches] + + +def test_heading_only_as_substring_is_not_found(mod): + text = '### PaperOps Architecture Details\n\nProse.\n' + idx, err = mod.locate_heading(text, '## PaperOps Architecture') + assert idx is None and err == 'heading_not_found' + + +def test_heading_inside_code_fence_is_ignored(mod): + text = 'Intro\n\n```md\n## PaperOps Architecture\n```\n\nOutro\n' + idx, err = mod.locate_heading(text, '## PaperOps Architecture') + assert idx is None and err == 'heading_not_found' + + +def test_heading_ambiguous(mod): + text = '## Metrics\n\nA.\n\n## Metrics\n\nB.\n' + idx, err = mod.locate_heading(text, '## Metrics') + assert idx is None and err == 'heading_ambiguous' + + +def test_preview_blocks_not_found_and_ambiguous(mod, capsys): + setup_env(mod, CONFIG_TWO, + '### PaperOps Architecture Details\n\n## Metrics\n\nx\n\n## Metrics\n\ny\n') + csv_path = run_preview(mod) + rows = {r['figure_id']: r for r in read_rows(csv_path)} + assert rows['fig_alpha']['status'] == 'blocked' + assert rows['fig_alpha']['reason'] == 'heading_not_found' + assert rows['fig_beta']['status'] == 'blocked' + assert rows['fig_beta']['reason'] == 'heading_ambiguous' + out = capsys.readouterr().out + assert 'ready_rows=0' in out + assert 'blocked_rows=2' in out + + +def test_apply_inserts_under_exact_heading_only(mod): + setup_env(mod) + fake_svg(mod, 'fig_alpha') + run_preview(mod) + run_apply(mod) + text = chapter_text(mod) + lines = text.split('\n') + # The fence-aware matcher must still see exactly one real heading line; + # the insertion goes right under it (heading, blank, block). + matches = mod.heading_match_lines(text, '## PaperOps Architecture') + assert len(matches) == 1 + target_idx = matches[0] + assert lines[target_idx] == '## PaperOps Architecture' + assert lines[target_idx + 1] == '' + assert lines[target_idx + 2] == ALPHA_BLOCK + # Not inserted after the ### heading or inside the fence. + details_idx = lines.index('### PaperOps Architecture Details') + assert ALPHA_BLOCK not in lines[details_idx:details_idx + 3] + assert chapter_text(mod).count(ALPHA_BLOCK) == 1 + fence_start = lines.index('```text') + assert lines[fence_start + 1] == '## PaperOps Architecture' # fence untouched + # Rendered SVG copied next to the manuscript. + assert (mod.MANUSCRIPT_FIG_DIR / 'fig_alpha.svg').exists() + + +def test_apply_blocks_ambiguous_heading_at_apply_time(mod, capsys): + setup_env(mod) + fake_svg(mod, 'fig_alpha') + csv_path = run_preview(mod) + # Target grows a second identical heading after the human approved. + target = mod.ROOT / CHAPTER + doctored = chapter_text(mod) + '\n## PaperOps Architecture\n\nDup.\n' + target.write_text(doctored, encoding='utf-8', newline='\n') + rows = read_rows(csv_path) + rows[0]['target_sha256'] = mod.file_sha256(target) # sha now matches again + write_rows(mod, csv_path, rows) + with pytest.raises(SystemExit) as e: + run_apply(mod) + assert e.value.code == 1 + out = capsys.readouterr().out + assert 'heading_ambiguous' in out + assert ALPHA_BLOCK not in chapter_text(mod) # never guesses + + +def test_apply_blocks_on_sha_mismatch_without_touching_file(mod, capsys): + setup_env(mod) + fake_svg(mod, 'fig_alpha') + run_preview(mod) + target = mod.ROOT / CHAPTER + edited = chapter_text(mod) + '\nHuman edit after preview.\n' + target.write_text(edited, encoding='utf-8', newline='\n') + with pytest.raises(SystemExit) as e: + run_apply(mod) + assert e.value.code == 1 + assert 'sha_mismatch' in capsys.readouterr().out + assert chapter_text(mod) == edited # no insertion happened + assert not (mod.ROOT / 'backups').exists() # no pointless backup + + +# --------------------------------------------------------------------------- +# Idempotency +# --------------------------------------------------------------------------- + +def test_reapply_blocks_already_applied_and_never_duplicates(mod, capsys): + setup_env(mod) + fake_svg(mod, 'fig_alpha') + run_preview(mod) + run_apply(mod) + after_first = chapter_text(mod) + capsys.readouterr() + # Re-running the same approved preview must be a benign no-op (exit 0). + run_apply(mod) + out = capsys.readouterr().out + assert 'applied_rows=0' in out + assert 'blocked_rows=1' in out + assert 'already_applied' in out + assert chapter_text(mod) == after_first + assert chapter_text(mod).count(ALPHA_BLOCK) == 1 + # A fresh preview also reports it as already applied. + csv2 = run_preview(mod, prefix='pv2') + row = read_rows(csv2)[0] + assert row['status'] == 'skipped' + assert row['reason'] == 'already_applied' + # Applying that fresh preview skips the row and stays a no-op. + capsys.readouterr() + run_apply(mod, 'reports/review/pv2.csv') + out = capsys.readouterr().out + assert 'applied_rows=0' in out + assert 'skipped_rows=1' in out + assert chapter_text(mod) == after_first + + +# --------------------------------------------------------------------------- +# Honest apply report, preview_sha256, backup location +# --------------------------------------------------------------------------- + +def test_apply_report_counts_reasons_sha_and_backup_location(mod, capsys): + setup_env(mod, CONFIG_TWO) + fake_svg(mod, 'fig_alpha') + fake_svg(mod, 'fig_beta') + csv_path = run_preview(mod) + original = chapter_text(mod) + # Simulate a stale/hand-edited preview row: beta now targets a heading + # that does not exist. + rows = read_rows(csv_path) + for r in rows: + if r['figure_id'] == 'fig_beta': + r['target_heading'] = '## Missing Heading' + write_rows(mod, csv_path, rows) + expected_sha = mod.file_sha256(csv_path) + with pytest.raises(SystemExit) as e: + run_apply(mod) + assert e.value.code == 1 + out = capsys.readouterr().out + assert 'applied_rows=1' in out + assert 'blocked_rows=1' in out + assert f'preview_sha256={expected_sha}' in out + report = (mod.ROOT / f'reports/review/figure_apply_{mod.today()}.md').read_text(encoding='utf-8') + assert '- applied_rows: 1' in report + assert '- blocked_rows: 1' in report + assert '- skipped_rows: 0' in report + assert f'- preview_sha256: {expected_sha}' in report + assert '- fig_beta: blocked (heading_not_found)' in report + assert '- fig_alpha: applied' in report + # Alpha applied, beta not. + assert ALPHA_BLOCK in chapter_text(mod) + assert BETA_BLOCK not in chapter_text(mod) + # Backup lives OUTSIDE the guarded manuscript tree, with the original text. + assert not (mod.ROOT / '05_manuscript' / 'backups').exists() + backup_root = mod.ROOT / 'backups' / 'manuscript' + backups = sorted(backup_root.glob('manuscript_before_figure_apply_*/chapters/ch_test.qmd')) + assert len(backups) == 1 + assert backups[0].read_text(encoding='utf-8') == original + assert 'backups/manuscript/manuscript_before_figure_apply_' in report + + +def test_dry_run_is_default_and_writes_nothing(mod, capsys): + setup_env(mod) + fake_svg(mod, 'fig_alpha') + run_preview(mod) + before = chapter_text(mod) + run_apply(mod, apply=False) # no --apply => dry-run + out = capsys.readouterr().out + assert 'DRY-RUN' in out + assert 'would_apply_rows=1' in out + assert 'preview_sha256=' in out + assert chapter_text(mod) == before + assert not (mod.ROOT / 'backups').exists() + + +# --------------------------------------------------------------------------- +# YAML spec loading and validation +# --------------------------------------------------------------------------- + +def test_repo_yaml_ships_the_five_original_specs(mod): + repo_cfg = REPO / 'config' / 'figures.yaml' + (mod.ROOT / 'config').mkdir(parents=True, exist_ok=True) + real_shutil.copyfile(repo_cfg, mod.ROOT / 'config' / 'figures.yaml') + specs, invalid, origin = mod.load_figure_specs(quiet=True) + assert origin == 'yaml' + assert invalid == [] + assert list(specs) == ['fig_pipeline', 'fig_evidence_flow', 'fig_guarded_apply', + 'fig_architecture', 'fig_verification_states'] + for fig_id, spec in specs.items(): + assert spec['kind'] == 'dot' + assert spec['source'].startswith(f'digraph {fig_id} {{') + assert spec['mermaid'] # companion .mmd source preserved + assert spec['target_file'].startswith('05_manuscript/chapters/') + assert spec['target_heading'].startswith('## ') + assert spec['caption'] + + +def test_yaml_validation_reports_invalid_entries_without_crashing(mod, capsys): + bad = """\ +figures: + - id: fig_ok + kind: dot + caption: OK. + target_heading: '## X' + source: | + digraph g { a; } + - id: fig_nocaption + kind: dot + target_heading: '## X' + source: | + digraph g { a; } + - id: fig_badkind + kind: png + caption: Bad kind. + target_heading: '## X' + source: | + whatever + - id: fig_ok + kind: mermaid + caption: Duplicate id. + target_heading: '## X' + source: | + flowchart LR + - 42 +""" + setup_env(mod, bad, chapter_text=None) + specs, invalid, origin = mod.load_figure_specs() + assert origin == 'yaml' + assert list(specs) == ['fig_ok'] + assert len(invalid) == 4 + joined = '\n'.join(invalid) + assert 'caption' in joined + assert 'kind must be one of' in joined + assert 'duplicate figure id' in joined + assert 'mapping' in joined + out = capsys.readouterr().out + assert 'WARN: invalid figure spec skipped' in out + + +def test_missing_yaml_falls_back_to_builtin_demo_with_warn(mod, capsys): + specs, invalid, origin = mod.load_figure_specs() + assert origin == 'builtin' + assert list(specs) == ['fig_paperops_demo'] + assert specs['fig_paperops_demo']['kind'] == 'dot' + out = capsys.readouterr().out + assert 'WARN' in out + assert 'config/figures.yaml' in out + + +def test_pyyaml_absent_falls_back_to_builtin_demo(mod, monkeypatch, capsys): + setup_env(mod, CONFIG_ONE, chapter_text=None) + monkeypatch.setattr(mod, 'yaml', None) + specs, invalid, origin = mod.load_figure_specs() + assert origin == 'builtin' + assert list(specs) == ['fig_paperops_demo'] + out = capsys.readouterr().out + assert 'WARN' in out + assert 'PyYAML' in out + + +def test_unparseable_yaml_falls_back_with_warn(mod, capsys): + setup_env(mod, 'figures: [unclosed\n - ][', chapter_text=None) + specs, invalid, origin = mod.load_figure_specs() + assert origin == 'builtin' + assert 'WARN' in capsys.readouterr().out + + +# --------------------------------------------------------------------------- +# Renderer degradation +# --------------------------------------------------------------------------- + +CONFIG_DOT_AND_MERMAID = CONFIG_ONE + """\ + - id: fig_flow + kind: mermaid + caption: Flow caption. + target_file: 05_manuscript/chapters/ch_test.qmd + target_heading: '## Metrics' + source: | + flowchart LR + a --> b +""" + + +def test_renderer_missing_degrades_to_sources_with_exit_0(mod, monkeypatch, capsys): + setup_env(mod, CONFIG_DOT_AND_MERMAID, chapter_text=None) + monkeypatch.setattr(mod.shutil, 'which', lambda *a, **k: None) + args = argparse.Namespace(figure_id=None, formats='svg,png', strict=False) + mod.cmd_render_figures(args) # must NOT raise + out = capsys.readouterr().out + assert 'renderer_missing=2' in out + assert 'rendered_ok=0' in out + assert 'failed=0' in out + assert 'WARN' in out + assert 'graphviz' in out.lower() + assert 'mermaid-cli' in out + # Sources are still written even without renderers. + assert (mod.FIG_SRC_DIR / 'fig_alpha.dot').exists() + assert (mod.FIG_SRC_DIR / 'fig_flow.mmd').exists() + report = (mod.FIG_DIR / f'figure_render_report_{mod.today()}.md').read_text(encoding='utf-8') + assert report.count('- status: renderer_missing') == 2 + + +def test_renderer_missing_with_strict_exits_1(mod, monkeypatch): + setup_env(mod, CONFIG_ONE, chapter_text=None) + monkeypatch.setattr(mod.shutil, 'which', lambda *a, **k: None) + args = argparse.Namespace(figure_id=None, formats='svg', strict=True) + with pytest.raises(SystemExit) as e: + mod.cmd_render_figures(args) + assert e.value.code == 1 + + +def test_real_render_error_still_exits_1(mod, monkeypatch, capsys): + setup_env(mod, CONFIG_ONE, chapter_text=None) + monkeypatch.setattr(mod.shutil, 'which', lambda *a, **k: '/usr/bin/dot') + + def fake_run(cmd, **kw): + assert kw.get('encoding') == 'utf-8' + assert kw.get('errors') == 'replace' + assert 'text' not in kw + return types.SimpleNamespace(returncode=1, stderr='boom: syntax error', stdout='') + + monkeypatch.setattr(mod.subprocess, 'run', fake_run) + args = argparse.Namespace(figure_id=None, formats='svg', strict=False) + with pytest.raises(SystemExit) as e: + mod.cmd_render_figures(args) + assert e.value.code == 1 + out = capsys.readouterr().out + assert 'failed=1' in out + + +# --------------------------------------------------------------------------- +# Encodings +# --------------------------------------------------------------------------- + +def test_read_text_compat_utf8_sig_and_cp949(mod, tmp_path): + p1 = tmp_path / 'bom.txt' + p1.write_bytes(codecs.BOM_UTF8 + '## 제목\n'.encode('utf-8')) + assert mod.read_text_compat(p1) == '## 제목\n' + p2 = tmp_path / 'k.txt' + p2.write_bytes('한글 근거 문장\n'.encode('cp949')) + assert mod.read_text_compat(p2) == '한글 근거 문장\n' + + +def test_read_text_compat_lossy_fallback_warns(mod, tmp_path, capsys): + p = tmp_path / 'junk.bin' + p.write_bytes(b'ok \xff\xfe\xff broken') + text = mod.read_text_compat(p) + assert 'ok' in text + assert 'WARN' in capsys.readouterr().out + + +def test_preview_csv_written_utf8_sig(mod): + setup_env(mod) + csv_path = run_preview(mod) + assert csv_path.read_bytes().startswith(b'\xef\xbb\xbf') + rows = read_rows(csv_path) + assert rows[0]['figure_id'] == 'fig_alpha' + + +# --------------------------------------------------------------------------- +# CLI contract +# --------------------------------------------------------------------------- + +def test_register_subcommands_keeps_existing_names_and_flags(mod): + parser = argparse.ArgumentParser() + sub = parser.add_subparsers(dest='command') + mod.register_subcommands(sub) + a = parser.parse_args(['propose-figures']) + assert a.func is mod.cmd_propose_figures + a = parser.parse_args(['render-figures', '--figure-id', 'fig_x', + '--formats', 'svg', '--strict']) + assert a.func is mod.cmd_render_figures and a.strict is True + a = parser.parse_args(['figure-placeholder-preview', '--figure-id', 'fig_x', + '--output-prefix', 'p']) + assert a.func is mod.cmd_figure_placeholder_preview + a = parser.parse_args(['apply-figure-placeholder', '--from-preview', 'x.csv', + '--dry-run', '--apply']) + assert a.func is mod.cmd_apply_figure_placeholder diff --git a/tests/test_paperops_core.py b/tests/test_paperops_core.py new file mode 100644 index 0000000..970864f --- /dev/null +++ b/tests/test_paperops_core.py @@ -0,0 +1,258 @@ +# -*- coding: utf-8 -*- +"""Core tests for scripts/paperops.py — pure functions and the regression set for the +defects fixed in v0.2.0. Filesystem-touching tests run against tmp_path with the module's +ROOT monkeypatched; nothing writes into the real repo tree.""" +import csv +import importlib.util +import io +import sys +from pathlib import Path + +import pytest + +REPO = Path(__file__).resolve().parents[1] +SPEC = importlib.util.spec_from_file_location('paperops_core_under_test', REPO / 'scripts' / 'paperops.py') +po = importlib.util.module_from_spec(SPEC) +sys.modules['paperops_core_under_test'] = po +SPEC.loader.exec_module(po) + + +# ---------------------------------------------------------------- stable ids / dedupe + +def test_stable_id_same_paper_across_sources_by_doi_normalization(): + a = {'doi': 'https://doi.org/10.1234/ABC'} + b = {'doi': '10.1234/abc'} + assert po.stable_id(a) == po.stable_id(b) + + +def test_stable_id_arxiv_version_normalized(): + a = {'arxiv_id': '2401.12345v2'} + b = {'arxiv_id': '2401.12345'} + assert po.stable_id(a) == po.stable_id(b) + + +def test_find_existing_paper_id_merges_doi_and_arxiv_variants(tmp_path, monkeypatch): + monkeypatch.setattr(po, 'DB', tmp_path / 'papers.sqlite') + po.init_db_tables_only() if hasattr(po, 'init_db_tables_only') else None + c = po.conn() + c.execute("CREATE TABLE IF NOT EXISTS papers(id TEXT PRIMARY KEY, title TEXT, authors_json TEXT, year INTEGER, venue TEXT, doi TEXT, arxiv_id TEXT, abstract TEXT, url TEXT, pdf_url TEXT, source TEXT, collection_date TEXT, status TEXT, score REAL, topic_relevance REAL, citation_count INTEGER, open_access INTEGER, local_pdf_path TEXT, parsed_text_path TEXT, citekey TEXT, title_norm TEXT, raw_json TEXT, updated_at TEXT)") + c.execute("INSERT INTO papers(id, title, doi, arxiv_id, title_norm) VALUES('id1', 'A Paper', '10.1/x', '', 'apaper')") + c.commit() + assert po.find_existing_paper_id(c, {'doi': 'https://doi.org/10.1/X'}) == 'id1' + assert po.find_existing_paper_id(c, {'title': 'A Paper!'}) == 'id1' + assert po.find_existing_paper_id(c, {'doi': '10.9/other'}) is None + c.close() + + +# ---------------------------------------------------------------- arXiv query builder + +def test_arxiv_query_soup_becomes_and_of_terms(): + expr = po.arxiv_query_expr('ontology knowledge graph governance') + assert 'all:"ontology"' in expr and ' AND ' in expr + + +def test_arxiv_query_explicit_boolean_passthrough(): + q = '"knowledge graph" AND (governance OR traceability)' + assert 'AND' in po.arxiv_query_expr(q) + + +def test_arxiv_query_caps_terms(): + expr = po.arxiv_query_expr(' '.join(f'w{i}' for i in range(12))) + assert expr.count(' AND ') == 5 # capped at 6 terms + + +# ---------------------------------------------------------------- verified invariant + +@pytest.mark.parametrize('value', ['1', 'yes', 'y', 'TRUE ', 'True', 'on', 't', '', None, 'banana']) +def test_normalized_verified_never_promotes_truthy_variants(value): + if str(value).strip().lower() == 'true': + assert po.normalized_verified(value) == 'true' + else: + assert po.normalized_verified(value) == 'false' + + +def test_normalized_verified_exact_literals(): + assert po.normalized_verified('true') == 'true' + assert po.normalized_verified('false') == 'false' + + +def test_verified_value_is_ambiguous(): + assert po.verified_value_is_ambiguous('1') + assert po.verified_value_is_ambiguous('YES') + assert not po.verified_value_is_ambiguous('true') + assert not po.verified_value_is_ambiguous('') + + +# ---------------------------------------------------------------- CSV round-trip safety + +def test_read_csv_dict_tolerates_bom_and_extra_columns(tmp_path): + p = tmp_path / 'q.csv' + # BOM + a stray trailing column, exactly what Excel produces + p.write_bytes('candidate_id,claim\r\nabc,hello,STRAY\r\n'.encode('utf-8-sig')) + rows, header = po.read_csv_dict(p) + assert header == ['candidate_id', 'claim'] + assert rows[0]['candidate_id'] == 'abc' + assert '_extra' not in rows[0] and None not in rows[0] + + +def test_atomic_write_csv_projects_stray_keys_and_writes_bom(tmp_path): + p = tmp_path / 'out.csv' + po.atomic_write_csv(p, ['a', 'b'], [{'a': '1', 'b': '한글', 'stray': 'x', None: ['boom']}]) + raw = p.read_bytes() + assert raw.startswith(b'\xef\xbb\xbf') # BOM so Excel renders Korean correctly + rows, header = po.read_csv_dict(p) + assert header == ['a', 'b'] and rows[0]['b'] == '한글' + assert not (tmp_path / 'out.csv.tmp_write').exists() + + +def test_read_text_compat_cp949_fallback(tmp_path): + p = tmp_path / 'k.txt' + p.write_bytes('정확도는 95.5%이다'.encode('cp949')) + assert '정확도는' in po.read_text_compat(p) + + +# ---------------------------------------------------------------- bib parsing + +def test_bib_entry_arxiv_id_no_literal_arxiv_bug(): + # archivePrefix present but no eprint -> must NOT return 'arxiv' + assert po.bib_entry_arxiv_id({'archiveprefix': 'arXiv'}) == '' + # non-arXiv eprint must not be treated as arXiv id + assert po.bib_entry_arxiv_id({'eprint': 'some-random-id', 'archiveprefix': 'bioRxiv'}) == '' + assert po.bib_entry_arxiv_id({'eprint': '2401.12345v3', 'archiveprefix': 'arXiv'}) == '2401.12345' + assert po.bib_entry_arxiv_id({'eprint': '2401.12345'}) == '2401.12345' + assert po.bib_entry_arxiv_id({'eprint': 'not an id'}) == '' + + +def test_citekey_pattern_ignores_emails(tmp_path, monkeypatch): + monkeypatch.setattr(po, 'ROOT', tmp_path) + man = tmp_path / '05_manuscript' + man.mkdir() + (man / 'ch.qmd').write_text('Cited [@kim2020graph]. Contact author@example.com please.', encoding='utf-8') + backups = man / 'backups' / 'old' + backups.mkdir(parents=True) + (backups / 'ch.qmd').write_text('[@stale2019key]', encoding='utf-8') + found = po.manuscript_citekeys() + assert 'kim2020graph' in found + assert 'gmail' not in found # email must not count as a cite + assert 'stale2019key' not in found # backups are excluded + + +# ---------------------------------------------------------------- source_location + +def test_source_location_semicolon_in_heading_sanitized(): + row = {'page': '', 'quote': 'q', 'section_id': 's1', 'section_heading': 'Results; Discussion'} + loc = po.source_location_for_patch(row) + parsed = po.parse_source_location(loc) + assert parsed['section_heading'] == 'Results, Discussion' + assert parsed['section_id'] == 's1' + assert parsed['quote_sha256'] + + +# ---------------------------------------------------------------- LLM verbatim gate + +def test_quote_is_verbatim_exact_and_whitespace_normalized(): + section = 'We propose PaperOps.\nIt improves traceability.' + assert po.quote_is_verbatim('We propose PaperOps.', section) + assert po.quote_is_verbatim('It improves traceability.', section) + + +def test_quote_is_verbatim_rejects_paraphrase_and_empty(): + section = 'We propose PaperOps for evidence governance.' + assert not po.quote_is_verbatim('PaperOps is proposed for governing evidence.', section) + assert not po.quote_is_verbatim('', section) + + +def test_parse_llm_candidate_json_variants(): + ok = po.parse_llm_candidate_json('[{"claim": "c", "exact_quote": "q"}]') + assert len(ok) == 1 + fenced = po.parse_llm_candidate_json('Sure!\n```json\n[{"claim": "c", "exact_quote": "q"}]\n```\nDone.') + assert len(fenced) == 1 + prose = po.parse_llm_candidate_json('Here it is [{"claim": "c", "exact_quote": "q"}] thanks') + assert len(prose) == 1 + assert po.parse_llm_candidate_json('not json at all') == [] + assert po.parse_llm_candidate_json('[{"claim": "no quote"}]') == [] + + +# ---------------------------------------------------------------- review queue merge + +def _queue_row(cid, paper, decision='pending'): + row = {f: '' for f in po.EVIDENCE_CANDIDATE_REVIEW_FIELDS} + row.update({'candidate_id': cid, 'paper_id': paper, 'verified': 'false', 'review_decision': decision, + 'citekey': 'k', 'source_artifact': 'sections.json', 'claim': 'c', 'quote': 'q', 'confidence': '0.7', + 'use_in_section': 'literature'}) + return row + + +def test_review_queue_merge_preserves_other_papers(tmp_path, monkeypatch): + """Regression for the data-loss bug: regenerating paper B's queue must not delete + paper A's pending human decisions from the shared queue file.""" + monkeypatch.setattr(po, 'ROOT', tmp_path) + (tmp_path / 'matrices').mkdir() + queue = tmp_path / 'matrices/evidence_candidate_review_queue.csv' + po.write_evidence_candidate_review_queue(queue, [ + _queue_row('aaa', 'paperA', 'include'), + _queue_row('bbb', 'paperB'), + ]) + existing = po.read_evidence_candidate_review_queue(queue) + preserved = po.preserved_review_values(existing, []) + new_b_rows = [po.review_row_from_candidate(_queue_row('bbb2', 'paperB'), preserved)] + other_rows = [r for r in existing if r.get('paper_id') != 'paperB'] + po.write_evidence_candidate_review_queue(queue, other_rows + new_b_rows) + final = po.read_evidence_candidate_review_queue(queue) + by_id = {r['candidate_id']: r for r in final} + assert 'aaa' in by_id, 'paper A row was lost' + assert by_id['aaa']['review_decision'] == 'include', 'paper A human decision was lost' + assert 'bbb2' in by_id + + +# ---------------------------------------------------------------- verification ledger + +def test_verification_ledger_attestation_flow(tmp_path, monkeypatch): + monkeypatch.setattr(po, 'ROOT', tmp_path) + monkeypatch.setattr(po, 'LOG', tmp_path / 'logs/ACTIVITY_LOG.md') + (tmp_path / 'matrices').mkdir() + (tmp_path / 'logs').mkdir() + fields = ['evidence_id', 'paper_id', 'citekey', 'claim', 'quote', 'exact_quote', 'verified', 'verified_by', 'verified_at', 'risk_note'] + matrix = tmp_path / 'matrices/evidence_matrix.csv' + po.atomic_write_csv(matrix, fields, [{'evidence_id': 'ev_1', 'paper_id': 'p', 'citekey': 'k', 'claim': 'c', 'quote': 'q', 'exact_quote': 'q', 'verified': 'false', 'verified_by': '', 'verified_at': '', 'risk_note': 'candidate_id=abc'}]) + import argparse + po.cmd_verify_evidence(argparse.Namespace(evidence_id='ev_1', by='임석제', method='human_pdf_check', note='checked p.3', attest=True, revoke=False)) + rows, _ = po.read_csv_dict(matrix) + assert rows[0]['verified'] == 'true' and rows[0]['verified_by'] == '임석제' + ledger = po.verification_ledger_index() + assert ledger['ev_1']['action'] == 'verify' + # guard logic: attested row passes, hand-edited row fails + entry = ledger['ev_1'] + assert entry['verified_by'] == rows[0]['verified_by'] + # revoke path + po.cmd_verify_evidence(argparse.Namespace(evidence_id='ev_1', by='임석제', method='human_pdf_check', note='', attest=False, revoke=True)) + rows, _ = po.read_csv_dict(matrix) + assert rows[0]['verified'] == 'false' and rows[0]['verified_by'] == '' + assert po.verification_ledger_index()['ev_1']['action'] == 'revoke' + + +def test_verify_evidence_requires_attestation(tmp_path, monkeypatch): + monkeypatch.setattr(po, 'ROOT', tmp_path) + monkeypatch.setattr(po, 'LOG', tmp_path / 'logs/ACTIVITY_LOG.md') + import argparse + with pytest.raises(SystemExit): + po.cmd_verify_evidence(argparse.Namespace(evidence_id='x', by='name', method='m', note='', attest=False, revoke=False)) + with pytest.raises(SystemExit): + po.cmd_verify_evidence(argparse.Namespace(evidence_id='x', by=' ', method='m', note='', attest=True, revoke=False)) + + +# ---------------------------------------------------------------- OpenAlex abstract + +def test_openalex_abstract_reconstruction(): + w = {'abstract_inverted_index': {'evidence': [1], 'Reconstructed': [0], 'text.': [2]}} + assert po.openalex_abstract(w) == 'Reconstructed evidence text.' + assert po.openalex_abstract({'abstract_inverted_index': None}) == '' + + +# ---------------------------------------------------------------- legacy deprecation + +def test_legacy_extract_evidence_is_blocked(): + import argparse + with pytest.raises(SystemExit) as excinfo: + po.cmd_extract(argparse.Namespace(limit=5)) + assert 'deprecated' in str(excinfo.value) diff --git a/tests/test_release_scan.py b/tests/test_release_scan.py new file mode 100644 index 0000000..1947818 --- /dev/null +++ b/tests/test_release_scan.py @@ -0,0 +1,375 @@ +# -*- coding: utf-8 -*- +"""Tests for scripts/build_public_release.py. + +Covers the secret scan rules, the email allowlist, the config +sanitizer, the quarantine-on-failure behavior, and the whitelist build, +all against miniature trees under tmp_path (never the real repo dist/). + +Every secret-shaped fixture is assembled at runtime by string +concatenation so this file never contains a contiguous secret-looking +string: the test suite itself is exported into the public release and +scanned by the very rules it tests. +""" +import importlib.util +import re +from pathlib import Path + +import pytest + +SCRIPT_PATH = (Path(__file__).resolve().parents[1] + / 'scripts' / 'build_public_release.py') + + +def _load_module(): + spec = importlib.util.spec_from_file_location( + 'build_public_release_under_test', str(SCRIPT_PATH)) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +bpr = _load_module() + +SP = chr(32) # single space, used to keep key-block fixtures non-contiguous + +# rule name -> fake fixture that must trigger exactly that rule +FAKE = { + 'email-address': 'kevin' + '@' + 'gm' + 'ail.com', + 'github-fine-grained-pat': 'github' + '_pat_' + 'x' * 22, + 'github-token': 'gh' + 'p_' + 'A' * 36, + 'private-key-block': ('-----BEGIN' + SP + 'RSA' + SP + 'PRIVATE' + + SP + 'KEY-----'), + 'aws-access-key': 'AK' + 'IA' + 'ABCDEFGHIJKLMNOP', + 'sk-secret-key': 'sk' + '-' + 'a' * 24, + 'slack-token': 'xo' + 'xb' + '-' + '1234567890' + 'abcd', + 'google-api-key': 'AI' + 'za' + 'B' * 35, + 'jwt': 'ey' + 'J' + 'a' * 24 + '.' + 'ey' + 'J' + 'b' * 8, + 'config-assignment-secret': 'api' + '_key: "' + 'A' * 20 + '"', +} + + +def _fake_email(): + return 'someone' + '@' + 'gm' + 'ail.com' + + +def scan_text(tmp_path, text): + d = tmp_path / 'scan_dir' + d.mkdir(exist_ok=True) + (d / 'sample.md').write_text(text, encoding='utf-8') + return bpr.sanitize_scan(d) + + +# ---------------------------------------------------------------- patterns + + +def test_patterns_are_named_compiled_pairs(): + assert bpr.FORBIDDEN_PATTERNS + for name, pat in bpr.FORBIDDEN_PATTERNS: + assert isinstance(name, str) and name + assert hasattr(pat, 'finditer') + + +def test_fixtures_cover_every_rule(): + assert {n for n, _ in bpr.FORBIDDEN_PATTERNS} == set(FAKE) + + +@pytest.mark.parametrize('rule', sorted(FAKE)) +def test_rule_fires_and_report_names_it(rule, tmp_path): + issues = scan_text(tmp_path, 'before ' + FAKE[rule] + ' after\n') + assert issues, rule + assert any(': ' + rule + ': ' in i for i in issues), issues + + +@pytest.mark.parametrize('prefix_char', list('pousr')) +def test_github_token_prefix_variants(prefix_char, tmp_path): + token = 'gh' + prefix_char + '_' + 'z' * 31 + issues = scan_text(tmp_path, token + '\n') + assert any(': github-token: ' in i for i in issues) + + +@pytest.mark.parametrize('kind', ['', 'RSA', 'EC', 'OPENSSH', 'DSA', 'PGP']) +def test_private_key_variants(kind, tmp_path): + middle = (SP + kind) if kind else '' + block = '-----BEGIN' + middle + SP + 'PRIVATE' + SP + 'KEY-----' + issues = scan_text(tmp_path, block + '\n') + assert any(': private-key-block: ' in i for i in issues) + + +def test_sk_ant_variant(tmp_path): + token = 'sk' + '-ant-' + 'q' * 22 + issues = scan_text(tmp_path, token + '\n') + assert any(': sk-secret-key: ' in i for i in issues) + + +@pytest.mark.parametrize('chan', list('baprs')) +def test_slack_token_variants(chan, tmp_path): + token = 'xo' + 'x' + chan + '-' + '0' * 12 + issues = scan_text(tmp_path, token + '\n') + assert any(': slack-token: ' in i for i in issues) + + +def test_short_or_broken_tokens_do_not_fire(tmp_path): + text = '\n'.join([ + 'ey' + 'J' + 'abc' + '.ey' + 'J' + 'x', # far below jwt threshold + 'sk' + '-' + 'a' * 10, # below length threshold + 'task-based sk' + '-learn workflow', # ordinary prose + 'gh' + 'p_' + 'A' * 10, # below length threshold + ]) + '\n' + assert scan_text(tmp_path, text) == [] + + +# ------------------------------------------------------- email allowlist + + +def test_allowlisted_emails_pass(tmp_path): + text = '\n'.join([ + 'contact: someone@example.com', + 'or: another.person@example.org', + 'bot: noreply@anthropic.com', + 'git: 12345+octo-cat@users.noreply.github.com', + ]) + '\n' + assert scan_text(tmp_path, text) == [] + + +def test_non_allowlisted_email_fires(tmp_path): + addr = 'research.lead' + '@' + 'univ' + '.ac.kr' + issues = scan_text(tmp_path, 'contact ' + addr + '\n') + assert any(': email-address: ' in i for i in issues) + + +# ------------------------------------- config-assignment rule discipline + + +def test_empty_config_values_do_not_fire(tmp_path): + # includes the cross-line trap: an empty key directly followed by a + # long identifier on the next line must not match + text = '\n'.join([ + 'semantic_scholar_api_key: ""', + "download_token: ''", + 'password:', + 'semantic_scholar_backup_api_key: ""', + 'api_key =', + 'secret: "" # fill in locally', + ]) + '\n' + assert scan_text(tmp_path, text) == [] + + +def test_config_assignment_fires_on_yaml_and_python_forms(tmp_path): + yaml_form = 'semantic_scholar_' + 'api_key: ' + 'k' * 20 + py_form = 'tok' + 'en = "' + 'h' * 18 + '"' + upper_form = 'SEC' + 'RET: ' + 'Z' * 16 + for sample in (yaml_form, py_form, upper_form): + issues = scan_text(tmp_path, sample + '\n') + assert any(': config-assignment-secret: ' in i for i in issues), sample + + +# ------------------------------------------------------ config sanitizer + + +def test_sanitize_configs_blanks_values_and_preserves_lines(tmp_path): + dist = tmp_path / 'public_release' + cfg = dist / 'config' + cfg.mkdir(parents=True) + email = _fake_email() + lines = [ + '# collection endpoints', + 'user_agent: "PaperOps/0.1 (mailto:' + email + ')"', + 'openalex_mailto: "' + email + '"', + 'semantic_scholar_api_key: ""', + 'my_api_key: "' + 'S' * 20 + '"', + ' admin_token: ' + 'b' * 18, + 'plain_key: value', + 'sources:', + ' arxiv:', + ' enabled: true', + ] + (cfg / 'sources.yaml').write_text('\n'.join(lines) + '\n', + encoding='utf-8') + red = bpr.sanitize_configs(dist) + out = (cfg / 'sources.yaml').read_text(encoding='utf-8').splitlines() + assert out[0] == '# collection endpoints' + assert out[1] == 'user_agent: ""' + assert out[2] == 'openalex_mailto: ""' + assert out[3] == 'semantic_scholar_api_key: ""' # already blank: kept + assert out[4] == 'my_api_key: ""' + assert out[5] == ' admin_token: ""' # indentation preserved + assert out[6] == 'plain_key: value' + assert out[7:] == ['sources:', ' arxiv:', ' enabled: true'] + assert red == [ + ('config/sources.yaml', 'user_agent'), + ('config/sources.yaml', 'openalex_mailto'), + ('config/sources.yaml', 'my_api_key'), + ('config/sources.yaml', 'admin_token'), + ] + # redaction records never carry the removed values + assert all(email not in rel + key for rel, key in red) + # second pass is a no-op: already-blank values are not re-reported + assert bpr.sanitize_configs(dist) == [] + + +def test_user_agent_without_email_is_kept(tmp_path): + dist = tmp_path / 'public_release' + cfg = dist / 'config' + cfg.mkdir(parents=True) + body = 'user_agent: "PaperOps/0.1 (research bot)"\nrate: 1.0\n' + (cfg / 'other.yaml').write_text(body, encoding='utf-8') + assert bpr.sanitize_configs(dist) == [] + assert (cfg / 'other.yaml').read_text(encoding='utf-8') == body + + +# ----------------------------------------------------------- quarantine + + +def test_quarantine_moves_dist_aside(tmp_path): + dist = tmp_path / 'public_release' + dist.mkdir() + (dist / 'README.md').write_text('hello\n', encoding='utf-8') + q = bpr.quarantine(dist, stamp='20260101_000000') + assert q == tmp_path / 'public_release_QUARANTINE_20260101_000000' + assert not dist.exists() + assert (q / 'README.md').read_text(encoding='utf-8') == 'hello\n' + + +def test_quarantine_replaces_prior_same_name(tmp_path): + dist = tmp_path / 'public_release' + dist.mkdir() + (dist / 'new.txt').write_text('new\n', encoding='utf-8') + prior = tmp_path / 'public_release_QUARANTINE_20260101_000000' + prior.mkdir() + (prior / 'old.txt').write_text('old\n', encoding='utf-8') + q = bpr.quarantine(dist, stamp='20260101_000000') + assert q == prior + assert (q / 'new.txt').exists() + assert not (q / 'old.txt').exists() + assert not dist.exists() + + +# ------------------------------------------------- build + run end to end + + +def _make_mini_tree(root): + (root / 'scripts').mkdir(parents=True) + (root / 'scripts' / 'paperops.py').write_text('print("ok")\n', + encoding='utf-8') + (root / 'README.md').write_text('# mini\n', encoding='utf-8') + (root / 'CHANGELOG.md').write_text('# changelog\n', encoding='utf-8') + docs = root / 'docs' + docs.mkdir() + (docs / 'RUN_AX_ONTOLOGY_GOVERNANCE.md').write_text('# runbook\n', + encoding='utf-8') + cfg = root / 'config' + cfg.mkdir() + (cfg / 'sources.yaml').write_text( + 'user_agent: "Mini/0.1 (mailto:' + _fake_email() + ')"\n' + 'openalex_mailto: "' + _fake_email() + '"\n' + 'semantic_scholar_api_key: ""\n', + encoding='utf-8') + t = root / 'tests' + t.mkdir() + (t / 'test_dummy.py').write_text('def test_ok():\n assert True\n', + encoding='utf-8') + gh = root / '.github' / 'workflows' + gh.mkdir(parents=True) + (gh / 'ci.yml').write_text('name: ci\n', encoding='utf-8') + figs = root / 'reports' / 'figures' / 'src' + figs.mkdir(parents=True) + (figs / 'fig.mmd').write_text('graph TD;\n', encoding='utf-8') + (root / 'reports' / 'figures' / 'pic.svg').write_text( + '\n', encoding='utf-8') + + +def _expected_missing(root): + miss = [s for s, _ in bpr.WHITELIST_FILES if not (root / s).exists()] + miss += [s for s, _ in bpr.WHITELIST_DIRS if not (root / s).exists()] + miss += [s for s, _, _ in bpr.WHITELIST_GLOBS if not (root / s).exists()] + return miss + + +def test_whitelist_covers_new_public_paths(): + file_srcs = {s for s, _ in bpr.WHITELIST_FILES} + dir_srcs = {s for s, _ in bpr.WHITELIST_DIRS} + assert 'docs/RUN_AX_ONTOLOGY_GOVERNANCE.md' in file_srcs + assert 'CHANGELOG.md' in file_srcs + assert 'tests' in dir_srcs + assert '.github' in dir_srcs + + +def test_run_builds_sanitizes_and_reports(tmp_path): + root = tmp_path / 'repo' + root.mkdir() + _make_mini_tree(root) + dist = root / 'dist' / 'public_release' + assert bpr.run(check=False, root=root, dist=dist) == 0 + for rel in ('README.md', 'CHANGELOG.md', + 'docs/RUN_AX_ONTOLOGY_GOVERNANCE.md', + 'tests/test_dummy.py', '.github/workflows/ci.yml', + 'assets/figures/pic.svg', 'assets/figures/src/fig.mmd', + '.gitignore'): + assert (dist / rel).exists(), rel + # exported config copy is sanitized; the source tree stays untouched + exported = (dist / 'config' / 'sources.yaml').read_text(encoding='utf-8') + assert 'user_agent: ""' in exported + assert 'openalex_mailto: ""' in exported + assert 'semantic_scholar_api_key: ""' in exported + assert '@' not in exported + assert '@' in (root / 'config' / 'sources.yaml').read_text( + encoding='utf-8') + info = (dist / 'RELEASE_INFO.txt').read_text(encoding='utf-8') + n_missing = len(_expected_missing(root)) + assert n_missing > 0 # the mini tree lacks most whitelisted sources + assert 'missing_whitelist_warnings=' + str(n_missing) in info + assert 'config_redactions=2' in info + assert 'redacted: config/sources.yaml: user_agent' in info + assert 'redacted: config/sources.yaml: openalex_mailto' in info + assert 'sanitize_issues=0' in info + # file count is taken after all writes and excludes RELEASE_INFO.txt + reported = int(re.search(r'files=(\d+)', info).group(1)) + on_disk = [p for p in dist.rglob('*') + if p.is_file() and p.name != 'RELEASE_INFO.txt'] + assert reported == len(on_disk) + assert reported == bpr.release_file_count(dist) + + +def test_run_quarantines_on_scan_failure(tmp_path, capsys): + root = tmp_path / 'repo' + root.mkdir() + _make_mini_tree(root) + leaked = 'AK' + 'IA' + 'ZZZZZZZZZZZZZZZZ' + (root / 'README.md').write_text('# mini\nkey ' + leaked + '\n', + encoding='utf-8') + dist = root / 'dist' / 'public_release' + assert bpr.run(check=False, root=root, dist=dist) == 1 + assert not dist.exists() + quarantined = list((root / 'dist').glob('public_release_QUARANTINE_*')) + assert len(quarantined) == 1 + assert (quarantined[0] / 'README.md').exists() + out = capsys.readouterr().out + assert 'Nothing publishable remains' in out + assert ': aws-access-key: ' in out + + +def test_check_mode_scans_without_rewriting_report(tmp_path): + root = tmp_path / 'repo' + root.mkdir() + _make_mini_tree(root) + dist = root / 'dist' / 'public_release' + assert bpr.run(check=False, root=root, dist=dist) == 0 + info_before = (dist / 'RELEASE_INFO.txt').read_text(encoding='utf-8') + assert bpr.run(check=True, root=root, dist=dist) == 0 + assert dist.exists() + assert (dist / 'RELEASE_INFO.txt').read_text( + encoding='utf-8') == info_before + + +def test_check_mode_missing_dist_errors(tmp_path): + assert bpr.run(check=True, root=tmp_path, dist=tmp_path / 'nope') == 1 + + +def test_check_mode_quarantines_dirty_dist(tmp_path): + dist = tmp_path / 'public_release' + dist.mkdir() + (dist / 'note.md').write_text('token' + ': "' + 'v' * 18 + '"\n', + encoding='utf-8') + assert bpr.run(check=True, root=tmp_path, dist=dist) == 1 + assert not dist.exists() + assert list(tmp_path.glob('public_release_QUARANTINE_*')) From c7c7dbe7f0c29a7762173213440273380421266f Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 27 Jul 2026 14:37:04 +0000 Subject: [PATCH 2/2] docs: bring all 6 remaining translated READMEs up to v0.2.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit README.md (en) and README.ko.md were updated in the previous commit; the other six shipped language versions still described v0.1.0. All eight are now consistent, so a non-English reader is not handed stale instructions. Applied to de/es/fr/ja/zh/ar, translated in each file's existing register: - 45+ -> 50+ commands; intro now covers verbatim-checked LLM extraction and numeric cross-checking against experiment outputs - verified=true described as reachable only via verify-evidence with a human attestation recorded in the verification ledger, cross-checked by guards - lifecycle rows 1/5/13/14 rewritten (4 sources + boolean queries + polite pool + 429 backoff + dedup; extract-evidence-llm; --experiment-data; human-gate verification) - comparison table, quickstart (Python 3.10+, openalex_mailto polite-pool note), add-ons table (+Semantic Scholar key, +OPENAI_API_KEY), typical session (--experiment-data, verify-evidence --attest) - KCI anecdote now states the numeric cross-check was manual at the time and automatic since v0.2.0 — it is not retroactively claimed as automated - 6 governance rules and the new honest-limitations list Also fixes two release-build regressions from the previous commit: - drop deleted scripts/paperops_extra.py from the export whitelist - reword a CHANGELOG line so the hardened secret scanner no longer flags an illustrative email address Verified: all 8 READMEs pass the same four content checks, all four tables in every file match the English column shape, language selectors and figure paths byte-identical, release build clean (sanitize_issues=0, 8 READMEs shipped), 179 tests still pass. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01ULLxKwEG7wBaj6SDFeBmsA --- CHANGELOG.md | 3 +- README.ar.md | 91 +++++++++++++++++--------- README.de.md | 54 +++++++++------- README.es.md | 54 +++++++++------- README.fr.md | 111 ++++++++++++++++++++++---------- README.ja.md | 87 +++++++++++++++++-------- README.zh.md | 81 +++++++++++++++-------- scripts/build_public_release.py | 1 - 8 files changed, 318 insertions(+), 164 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 023d08e..09e0ec7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -46,7 +46,8 @@ a 4-version CI matrix are new in this release. - Legacy `extract-evidence` appended schema-drifted 14-column rows into the migrated 19+-column guarded matrix; it is now blocked with instructions for the guarded pipeline. -- Legacy `audit` counted `user@gmail.com` as citekey `gmail` and matched +- Legacy `audit` counted the domain part of any email address in the text as a + citekey (an address ending in `@gmail.com` became citekey `gmail`) and matched citekeys by substring against raw CSV text; it now uses the email-safe pattern over both manuscript roots against the real citekey column. - Cross-source duplicates: the same paper arriving from arXiv (arXiv id only) diff --git a/README.ar.md b/README.ar.md index 2892811..dbe3eac 100644 --- a/README.ar.md +++ b/README.ar.md @@ -18,23 +18,27 @@ ![خط أنابيب PaperOps من البداية إلى النهاية](assets/figures/fig_pipeline.svg) يقوم PaperOps بأتمتة **دورة حياة الكتابة البحثية بأكملها** — جمع الأدبيات، -والفرز، وتحليل ملفات PDF، واستخراج الأدلة، ومزامنة المراجع، وتحرير المخطوطة -بضوابط حماية، وتوليد أشكال قابلة لإعادة الإنتاج، وتدقيق المسودات — عبر واجهة -سطر أوامر محلية واحدة تضم **أكثر من 45 أمرًا**. +والفرز، وتحليل ملفات PDF، واستخراج الأدلة (استدلاليًا، وبمساعدة نموذج لغوي مع +تحقق حرفي من كل اقتباس)، ومزامنة المراجع، وتحرير المخطوطة بضوابط حماية، +وتوليد أشكال قابلة لإعادة الإنتاج، وتدقيق المسودات مع مطابقة رقمية آلية مع +مخرجات التجارب — عبر واجهة سطر أوامر محلية واحدة تضم **أكثر من 50 أمرًا**. إنه *ليس* أداة لكتابة الأوراق تلقائيًا. خط الأنابيب مؤتمت، لكن ثلاث نقاط حُكم محجوزة عمدًا للإنسان: اعتماد الأدلة، والموافقة على تعديلات المخطوطة، -وقرار `verified=true`. وتمنع الضوابط (guards) أي خطوة مؤتمتة من تزويرها. +وقرار `verified=true`. ولا يمكن بلوغ حالة `verified=true` إلا عبر أمر +`verify-evidence` بإقرار بشري صريح، يُسجَّل في سجل التحقق (verification +ledger) الذي يطابقه `guard-no-auto-verified` — فالتعديل اليدوي والتعديل +المؤتمت كلاهما يسقط أمام الضابط. ## دورة الحياة الكاملة، مرحلة بمرحلة | المرحلة | المحتوى | الأوامر الرئيسية | الأتمتة | |---|---|---|---| -| 1. الجمع | جلب الأوراق من arXiv / Semantic Scholar / OpenAlex وفق ملف موضوعي | `collect`, `digest` | تلقائي | +| 1. الجمع | جلب الأوراق من arXiv / Semantic Scholar / OpenAlex / Crossref باستعلامات موضوعية بوليانية، مع ترويسات polite pool، وتراجع تدريجي عند 429، وإزالة التكرار عبر المصادر | `collect`, `digest` | تلقائي | | 2. الفرز | تقييم الصلة، الغربلة حسب محاور البحث، اكتشاف الفجوات البحثية | `score`, `screen`, `gap`, `brief` | تلقائي | | 3. الاقتناء | تنزيل ملفات PDF، إنشاء بطاقات الأوراق والمخططات | `download-pdfs`, `cards`, `outline` | تلقائي | | 4. التحليل | PDF ← أقسام/مراجع منظّمة عبر GROBID | `parse-grobid`, `validate-grobid-artifacts` | تلقائي | -| 5. الاستخراج | استخراج مرشّحي الأدلة (ادعاء/اقتباس/صفحة) من النص المحلَّل | `extract-evidence-candidates`, `validate-evidence-candidates` | تلقائي | +| 5. الاستخراج | استخراج مرشّحي الأدلة (ادعاء/اقتباس/صفحة) من النص المحلَّل — استدلاليًا، أو باقتراح من نموذج لغوي مع تحقق حرفي من كل اقتباس مقابل المصدر | `extract-evidence-candidates`, `extract-evidence-llm`, `validate-evidence-candidates` | تلقائي | | 6. المراجعة | قرار قبول / تعديل / رفض لكل مرشّح | `review-evidence-candidates`, `promotion-plan` | **بوابة بشرية** | | 7. الترقية | نقل الأدلة المعتمدة إلى مصفوفة الأدلة (`verified=false`) | `promote-evidence`, `audit-promoted-evidence` | محمي | | 8. تحديد الصفحات | إيجاد وتسجيل صفحة PDF الدقيقة لكل اقتباس | `locate-pdf-pages`, `apply-page-metadata` | محمي | @@ -42,8 +46,8 @@ | 10. الكتابة | توليد تصحيحات المخطوطة كمعاينة + diff | `manuscript-patch-preview` | تلقائي | | 11. التطبيق | تطبيق التصحيحات المعتمدة مع نسخ احتياطي + تحقق SHA + كتابة LF | `apply-manuscript-patch` | **بوابة بشرية** | | 12. الأشكال | أشكال Graphviz/Mermaid مبنية على مواصفات، لا بيانات مفبركة أبدًا | `propose-figures`, `render-figures`, `apply-figure-placeholder` | محمي | -| 13. تدقيق المسودة | فحص أي مسودة (docx/md/qmd): البنية، الادعاءات بلا مصادر، المبالغات، ومطابقة الأرقام مع مخرجات التجارب الفعلية | `audit-manuscript-draft` | تلقائي | -| 14. التحقق | فرض أنه لا توجد أتمتة وضعت `verified=true` قط | `guard-no-auto-verified`, `guard-paperops-overclaim`, `smoke-test` | ضابط تلقائي / **حكم بشري** | +| 13. تدقيق المسودة | فحص أي مسودة (docx/md/qmd): البنية، الادعاءات بلا مصادر، المبالغات، ومطابقة رقمية آلية مع ملفات مخرجات التجارب عبر `--experiment-data` | `audit-manuscript-draft` | تلقائي | +| 14. التحقق | يضع الإنسان `verified=true` بإقرار يُسجَّل في سجل التحقق؛ وتُسقط الضوابط أي صف موسوم بالتحقق بلا قيد مطابق في السجل | `verify-evidence`, `guard-no-auto-verified`, `guard-paperops-overclaim`, `smoke-test` | **بوابة بشرية** / ضابط تلقائي | ## لماذا هذا بدلاً من دردشة LLM؟ @@ -52,8 +56,8 @@ | من أين جاءت هذه الجملة؟ | مجهول | `paper_id` + `citekey` + اقتباس + صفحة في مصفوفة الأدلة | | صحة الاستشهادات | جهد تقريبي | مطابقة `check-citekeys` مع BibTeX المعتمد | | تعديل المخطوطة | كتابة فوقية مباشرة | معاينة ← diff ← موافقة ← تطبيق بتحقق SHA ← نسخ احتياطي ← تدقيق لاحق | -| حالة "تم التحقق" | ضمنية | لا يضعها إلا إنسان؛ والضوابط تفرض ذلك | -| الأرقام في مسودتك | غير مدققة | تُطابق آليًا مع ملفات مخرجات التجارب الفعلية | +| حالة "تم التحقق" | ضمنية | لا يضعها إلا `verify-evidence --attest`؛ وسجل التحقق + الضوابط يكشفان ما عدا ذلك | +| الأرقام في مسودتك | غير مدققة | تُطابق آليًا مع ملفات مخرجات التجارب الفعلية (`audit-manuscript-draft --experiment-data`) | | قابلية إعادة الإنتاج | مرتبطة بالجلسة | SQLite + مصفوفات CSV + تقارير تدقيق + سجل نشاط + مصادر الأشكال | استُخلصت أنماط التصميم من مسح لأكثر من 40 أداة بحثية مفتوحة المصدر @@ -88,21 +92,26 @@ ```bash git clone https://github.com/SakJaeLim/paperops.git && cd paperops -python -m venv .venv +python -m venv .venv # Python 3.10+ # Windows: .venv\Scripts\activate | Unix: source .venv/bin/activate pip install -r requirements.txt python scripts/paperops.py init python scripts/paperops.py status ``` -يعمل فورًا بدون خدمات خارجية: الجمع، التقييم، الغربلة، تدقيق المسودات، -الضوابط، توليد مصادر الأشكال. إضافات اختيارية: +ثم ضع بريدك الإلكتروني في `config/sources.yaml` (`openalex_mailto`) — فهو +يُدرج طلبات OpenAlex/Crossref ضمن "polite pool" الخاص بكل منهما؛ وبدونه +تفرض الواجهتان تقييدًا شديدًا (HTTP 429). يعمل فورًا بدون خدمات خارجية: +الجمع، التقييم، الغربلة، تدقيق المسودات، الضوابط، توليد مصادر الأشكال. +إضافات اختيارية: | الاعتمادية | تتيح | التثبيت | |---|---|---| | GROBID | تحليل PDF ← نص منظّم | `docker run -d -p 8070:8070 lfoppiano/grobid:0.8.0` | -| Zotero + Better BibTeX | مزامنة المراجع المعتمدة | zotero.org + إضافة Better BibTeX | -| Graphviz | إخراج الأشكال SVG/PNG | graphviz.org/download | +| Zotero + Better BibTeX | مزامنة المراجع المعتمدة (ملف `.bib` مُصدَّر) | zotero.org + إضافة Better BibTeX | +| Graphviz / Mermaid CLI | إخراج الأشكال SVG/PNG (المصادر تُكتب دائمًا حتى بدونهما) | graphviz.org/download | +| مفتاح Semantic Scholar API | جمع موثوق من S2 (الوصول غير الموثَّق إلى S2 يخضع لتقييد شديد) | `semantic_scholar_api_key` في `config/sources.yaml` | +| `OPENAI_API_KEY` | `extract-evidence-llm` (استخراج محمي بنموذج لغوي مع تحقق حرفي؛ أي نقطة نهاية متوافقة مع OpenAI عبر `OPENAI_BASE_URL`) | متغيّر بيئة | ## جلسة نموذجية @@ -125,8 +134,12 @@ python scripts/paperops.py manuscript-patch-preview python scripts/paperops.py apply-manuscript-patch --from-preview --dry-run python scripts/paperops.py apply-manuscript-patch --from-preview --apply -# تدقيق مسودتك (docx/md/qmd) -python scripts/paperops.py audit-manuscript-draft --input my_thesis_draft.docx +# تدقيق مسودتك (docx/md/qmd) — البنية، الادعاءات بلا مصادر، المبالغات، +# ومطابقة رقمية آلية مع ملفات مخرجات تجاربك +python scripts/paperops.py audit-manuscript-draft --input my_thesis_draft.docx --experiment-data data/experiment_outputs + +# التحقق البشري (البوابة الحاسمة) — يُسجَّل في سجل التحقق +python scripts/paperops.py verify-evidence --evidence-id ev_xxxx --by "اسمك" --note "روجعت الصفحة 3 من المصدر" --attest # الأشكال والفحوص النهائية python scripts/paperops.py render-figures @@ -142,18 +155,30 @@ python scripts/paperops.py smoke-test فحص 378 جملة، وتدقيق بنية الفصول، ووضع علامات على الادعاءات القوية بلا مصادر وعبارات المبالغة، ومطابقة **جميع القيم الرقمية الـ 178** في المسودة مع ملفات مخرجات التجارب الفعلية — صفر تعارض، مع تفسير فرقين في التقريب -وإعادة حساب معدل أساس واحد من سجلات التنبؤ الخام. التدقيق لا يعدّل المسودة -أبدًا ولا يضع حالة تحقق؛ بل ينتج تقرير نتائج (MD + CSV) للمؤلف فقط. +وإعادة حساب معدل أساس واحد من سجلات التنبؤ الخام. (كانت تلك المطابقة يدوية +حينذاك؛ ومنذ الإصدار v0.2.0 ينفّذها خيار `--experiment-data` تلقائيًا: تطابق +تام، أو تطابق بعد التقريب، أو غير موجود في البيانات، لكل رقم على حدة.) +التدقيق لا يعدّل المسودة أبدًا ولا يضع حالة تحقق؛ بل ينتج تقرير نتائج +(MD + CSV) للمؤلف فقط. ## قواعد الحوكمة -1. لا تُعدَّل مصفوفة الأدلة باستخفاف أبدًا. -2. لا يُوضع `verified=true` تلقائيًا أبدًا — لا يوجد انتقال مؤتمت إلى - حالة التحقق. -3. مطابقة الاقتباس/الصفحة هي *محاذاة مصدر* وليست تحققًا من الحقيقة. -4. تعديلات المخطوطة تتم فقط عبر معاينة/تطبيق محميين مع نسخ احتياطية - وضوابط + اختبار دخان بعد التطبيق. -5. تُذكر نتائج الأعمال ذات الصلة كأنماط تصميم فقط، ولا تُقدَّم أبدًا +1. لا تُعدَّل مصفوفة الأدلة باستخفاف أبدًا؛ وكل عمليات إعادة الكتابة ذرّية + (ملف مؤقت + إعادة تسمية) مع نسخ احتياطية موسومة بالوقت. +2. لا يُوضع `verified=true` تلقائيًا أبدًا. المسار الوحيد المُجاز هو + `verify-evidence --attest`، الذي يسجّل مَن ومتى وكيف في + `matrices/verification_ledger.csv`؛ ويُسقط `guard-no-auto-verified` أي + صف موسوم بالتحقق بلا إقرار مطابق في السجل. +3. يخضع المرشّحون المستخرَجون بنموذج لغوي لتحقق حرفي مقابل النص المصدر + المحلَّل — فتُستبعد الاقتباسات المُعاد صياغتها أو المختلقة ويُبلَّغ عنها — + ثم يمرون عبر بوابات المراجعة البشرية نفسها التي يمر بها المرشّحون + الاستدلاليون. +4. مطابقة الاقتباس/الصفحة هي *محاذاة مصدر* وليست تحققًا من الحقيقة. +5. تعديلات المخطوطة تتم فقط عبر معاينة/تطبيق محميين مع نسخ احتياطية (تُحفظ + خارج الشجرة المحمية)، وتحقق SHA-256، وضابط + اختبار دخان بعد التطبيق؛ + وتُطابَق تقارير الموافقة مع SHA-256 للمخطوطة الحالية، فتُرفض التقارير + القديمة أو المفبركة. +6. تُذكر نتائج الأعمال ذات الصلة كأنماط تصميم فقط، ولا تُقدَّم أبدًا كدليل على أداء PaperOps نفسه. ## ما لا يتضمنه هذا المستودع @@ -167,11 +192,19 @@ python scripts/paperops.py smoke-test ## حدود صريحة -- استخراج الأدلة قائم على الكلمات المفتاحية/الاستدلال؛ والاستخراج بمساعدة - LLM مخطط له كخطوة محمية منفصلة. -- تدقيق المسودات وضع علامات استدلالي لمراجعة بشرية، وليس تحققًا من الحقيقة. +- لا يضمن مستخرِج النموذج اللغوي سوى حرفية الاقتباسات؛ أما جودة *انتقاء* + الادعاء/الاقتباس فتبقى رهنًا بالنموذج، ويظل كل مرشّح خاضعًا لمراجعة بشرية. +- تدقيق المسودات وضع علامات استدلالي لمراجعة بشرية، وليس تحققًا من الحقيقة؛ + والمطابقة الرقمية تتأكد من ورود الأرقام في ملفات بياناتك، لا من صحة + التحليل نفسه. - محاذاة الاقتباس/الصفحة لا تتحقق من صحة الادعاء — بحكم التصميم. - لا تُولَّد أشكال نتائج كمية أبدًا بدون ملف بيانات حقيقي. +- الجمع من Semantic Scholar بدون توثيق يخضع لتقييد معدل من S2؛ ومفتاح API + مجاني مطلوب عمليًا لهذا المصدر. +- سجل التحقق يردع التحقق الآلي العَرَضي والمتساهل؛ لكنه ليس آلية تعمية ولا + يمنع شخصًا مصمّمًا على تزوير الإقرارات داخل ملفاته المحلية. +- قد تتأخر ملفات README غير الإنجليزية عن نظيرتها الإنجليزية بإصدار واحد؛ + والمرجع المعتمد هو README الإنجليزي وملف CHANGELOG. ## الترخيص diff --git a/README.de.md b/README.de.md index 851eb1e..643c095 100644 --- a/README.de.md +++ b/README.de.md @@ -17,19 +17,19 @@ ![End-to-End-Pipeline von PaperOps](assets/figures/fig_pipeline.svg) -PaperOps automatisiert den **gesamten Forschungs- und Schreibprozess** — Literaturerfassung, Relevanzprüfung, PDF-Parsing, Evidenzextraktion, Bibliographie-Synchronisation, kontrollierte Manuskriptbearbeitung, reproduzierbare Grafikerstellung und Entwurfsprüfung — über ein einziges, lokal orientiertes CLI mit **über 45 Befehlen**. +PaperOps automatisiert den **gesamten Forschungs- und Schreibprozess** — Literaturerfassung, Relevanzprüfung, PDF-Parsing, Evidenzextraktion (heuristisch und LLM-gestützt mit wörtlicher Zitatprüfung), Bibliographie-Synchronisation, kontrollierte Manuskriptbearbeitung, reproduzierbare Grafikerstellung und Entwurfsprüfung mit numerischem Abgleich gegen die Ausgabedateien der Experimente — über ein einziges, lokal orientiertes CLI mit **über 50 Befehlen**. -Es ist *kein* automatischer Artikelschreiber. Die Pipeline ist automatisiert, aber drei Entscheidungsschritte sind bewusst dem Menschen vorbehalten: Evidenzübernahme, Freigabe von Manuskriptänderungen und die Einstufung als `verified=true`. Sicherheitsmechanismen (Guards) verhindern, dass diese Schritte gefälscht werden können. +Es ist *kein* automatischer Artikelschreiber. Die Pipeline ist automatisiert, aber drei Entscheidungsschritte sind bewusst dem Menschen vorbehalten: Evidenzübernahme, Freigabe von Manuskriptänderungen und die Einstufung als `verified=true`. Der Zustand `verified=true` kann ausschließlich über den Befehl `verify-evidence` mit einer ausdrücklichen menschlichen Bestätigung (Attestation) erreicht werden; diese wird in einem Verifizierungsregister (verification ledger) festgehalten, das `guard-no-auto-verified` gegenprüft — sowohl von Hand vorgenommene als auch automatisierte Änderungen scheitern am Guard. ## Der gesamte Lebenszyklus, Schritt für Schritt | Phase | Beschreibung | Hauptbefehle | Automatisierung | |---|---|---|---| -| 1. Erfassen | Abrufen von Artikeln aus arXiv / Semantic Scholar / OpenAlex über Themenprofile | `collect`, `digest` | Automatisch | +| 1. Erfassen | Abrufen von Artikeln aus arXiv / Semantic Scholar / OpenAlex / Crossref über boolesche Themenabfragen — mit Polite-Pool-Headern, 429-Backoff und quellenübergreifender Dublettenbereinigung | `collect`, `digest` | Automatisch | | 2. Selektieren | Bewerten der Relevanz, Filtern nach Forschungsachsen, Erkennen von Lücken | `score`, `screen`, `gap`, `brief` | Automatisch | | 3. Sichern | PDFs herunterladen, Literaturkarten und Manuskriptgliederung erstellen | `download-pdfs`, `cards`, `outline` | Automatisch | | 4. Parsen | PDF → strukturierte Abschnitte/Referenzen via GROBID konvertieren | `parse-grobid`, `validate-grobid-artifacts` | Automatisch | -| 5. Extrahieren | Thesen/Zitate/Seitenzahlen als Evidenzkandidaten aus dem Text extrahieren | `extract-evidence-candidates`, `validate-evidence-candidates` | Automatisch | +| 5. Extrahieren | Thesen/Zitate/Seitenzahlen als Evidenzkandidaten aus dem geparsten Text extrahieren — heuristisch oder von einem LLM vorgeschlagen, wobei jedes Zitat wörtlich gegen die Quelle geprüft wird | `extract-evidence-candidates`, `extract-evidence-llm`, `validate-evidence-candidates` | Automatisch | | 6. Prüfen | Entscheiden, ob jeder Kandidat übernommen, überarbeitet oder verworfen wird | `review-evidence-candidates`, `promotion-plan` | **Menschliche Freigabe** | | 7. Befördern | Übertragen freigegebener Evidenzen in die Evidenzmatrix (`verified=false`) | `promote-evidence`, `audit-promoted-evidence` | Kontrolliert | | 8. Verorten | Genaue PDF-Seitenzahlen für jedes Zitat ermitteln und verknüpfen | `locate-pdf-pages`, `apply-page-metadata` | Kontrolliert | @@ -37,8 +37,8 @@ Es ist *kein* automatischer Artikelschreiber. Die Pipeline ist automatisiert, ab | 10. Schreiben | Manuskript-Patches als Vorschau mit Git-Diff erzeugen | `manuscript-patch-preview` | Automatisch | | 11. Anwenden | Patches mit Backup, SHA-Prüfung und physischem Schreiben anwenden | `apply-manuscript-patch` | **Menschliche Freigabe** | | 12. Abbildungen | Spezifikationsgestützte Figuren über Graphviz/Mermaid ohne Datenfälschung erstellen | `propose-figures`, `render-figures`, `apply-figure-placeholder` | Kontrolliert | -| 13. Auditieren | Entwürfe (docx/md/qmd) auf Struktur, unbelegte Thesen und Zahlen vs. Experiment-Rohdaten prüfen | `audit-manuscript-draft` | Automatisch | -| 14. Verifizieren | Sicherstellen, dass kein Skript automatisiert `verified=true` setzen kann | `guard-no-auto-verified`, `guard-paperops-overclaim`, `smoke-test` | Automatische Guards / **Menschliche Entscheidung** | +| 13. Auditieren | Entwürfe (docx/md/qmd) prüfen: Struktur, unbelegte Thesen, übertriebene Formulierungen und automatischer numerischer Abgleich mit den Experiment-Ausgabedateien (`--experiment-data`) | `audit-manuscript-draft` | Automatisch | +| 14. Verifizieren | Der Mensch setzt `verified=true` mit einer Bestätigung, die im Verifizierungsregister festgehalten wird; Guards lassen jede verifizierte Zeile ohne passenden Registereintrag durchfallen | `verify-evidence`, `guard-no-auto-verified`, `guard-paperops-overclaim`, `smoke-test` | **Menschliche Freigabe** / automatischer Guard | ## Warum dieses System statt eines KI-Chats? @@ -47,8 +47,8 @@ Es ist *kein* automatischer Artikelschreiber. Die Pipeline ist automatisiert, ab | Woher stammt dieser Satz? | Unbekannt | `paper_id` + `citekey` + Zitat + Seite in der Evidenzmatrix | | Korrektheit der Zitate | Best-Effort | Abgeglichen mit realer Bibliographie via `check-citekeys` | | Manuskript-Änderungen | Direkte Überschreibung | Vorschau ➔ Diff ➔ Freigabe ➔ SHA-verifizierter Apply ➔ Backup ➔ Post-Audit | -| Status "Verifiziert" | Implizit | Nur durch Menschen setzbar; Guards erzwingen dies | -| Zahlen im Manuskript | Ungeprüft | Abgeglichen mit den Original-Ausgabedateien der Experimente | +| Status "Verifiziert" | Implizit | Nur `verify-evidence --attest` kann ihn setzen; das Verifizierungsregister und die Guards decken alles andere auf | +| Zahlen im Manuskript | Ungeprüft | Abgeglichen mit den tatsächlichen Ausgabedateien der Experimente (`audit-manuscript-draft --experiment-data`) | | Reproduzierbarkeit | Sitzungsgebunden | SQLite + CSV-Matrizen + Audit-Berichte + Aktivitätsprotokoll + Grafikquellen | Die Entwurfsmuster wurden aus der Analyse von über 40 Open-Source-Forschungstools (PaperQA2, STORM, GPT Researcher, AI-Scientist, ASReview, gpt_academic, Zotero-Ökosystem, MCP-Server — siehe `docs/03_TOOL_SYNTHESIS.md`) abgeleitet und unter einem Leitprinzip neu zusammengesetzt: **Keine Behauptung gelangt ohne nachvollziehbare, von Menschen geprüfte Belege in das Manuskript.** @@ -77,20 +77,22 @@ Die Daten fließen in einer einzigen Richtung mit Audit-Berichten bei jedem kont ```bash git clone https://github.com/SakJaeLim/paperops.git && cd paperops -python -m venv .venv +python -m venv .venv # Python 3.10+ # Unter Windows: .venv\Scripts\activate | Unter Unix: source .venv/bin/activate pip install -r requirements.txt python scripts/paperops.py init python scripts/paperops.py status ``` -Funktioniert sofort ohne externe Dienste für: Erfassung, Bewertung, Selektion, Entwurfsprüfung, Guards und Grafikquellengenerierung. Optionale Erweiterungen: +Tragen Sie anschließend Ihre E-Mail-Adresse in `config/sources.yaml` (`openalex_mailto`) ein — damit landen Anfragen an OpenAlex/Crossref in deren Polite Pool; ohne sie drosseln beide APIs stark (HTTP 429). Funktioniert sofort ohne externe Dienste für: Erfassung, Bewertung, Selektion, Entwurfsprüfung, Guards und Grafikquellengenerierung. Optionale Erweiterungen: | Abhängigkeit | Ermöglicht | Installation | |---|---|---| | GROBID | PDF-Parsing zu strukturiertem Text | `docker run -d -p 8070:8070 lfoppiano/grobid:0.8.0` | -| Zotero + Better BibTeX | Synchronisation mit Zotero-Bibliothek | zotero.org + Better BibTeX Plugin installieren | -| Graphviz | Rendering von SVG/PNG-Grafiken | Herunterladen über graphviz.org | +| Zotero + Better BibTeX | Synchronisation mit der Referenz-Bibliographie (exportierte `.bib`-Datei) | zotero.org + Better BibTeX Plugin installieren | +| Graphviz / Mermaid CLI | Rendering von SVG/PNG-Grafiken (die Quellen werden auch ohne sie immer geschrieben) | Herunterladen über graphviz.org | +| Semantic Scholar API-Schlüssel | Zuverlässige S2-Erfassung (ohne Authentifizierung drosselt S2 stark) | `semantic_scholar_api_key` in `config/sources.yaml` | +| `OPENAI_API_KEY` | `extract-evidence-llm` (kontrollierte LLM-Extraktion mit wörtlicher Zitatprüfung; jeder OpenAI-kompatible Endpunkt via `OPENAI_BASE_URL`) | Umgebungsvariable | ## Typischer Ablauf @@ -113,8 +115,12 @@ python scripts/paperops.py manuscript-patch-preview python scripts/paperops.py apply-manuscript-patch --from-preview --dry-run python scripts/paperops.py apply-manuscript-patch --from-preview --apply -# Eigenen Entwurf auditieren (docx/md/qmd) -python scripts/paperops.py audit-manuscript-draft --input mein_entwurf.docx +# Eigenen Entwurf auditieren (docx/md/qmd) — Struktur, unbelegte Thesen, übertriebene +# Formulierungen und automatischer numerischer Abgleich mit den Ausgabedateien der Experimente +python scripts/paperops.py audit-manuscript-draft --input mein_entwurf.docx --experiment-data data/experiment_outputs + +# Menschliche Verifizierung (DAS Tor) — im Verifizierungsregister festgehalten +python scripts/paperops.py verify-evidence --evidence-id ev_xxxx --by "Ihr Name" --note "S. 3 geprüft" --attest # Figuren rendern und Abschlussprüfungen python scripts/paperops.py render-figures @@ -126,15 +132,16 @@ python scripts/paperops.py smoke-test ## Praxisbeispiel der Entwurfsprüfung -Der Befehl `audit-manuscript-draft` wurde auf ein echtes Manuskript (413 Absätze) angewendet: Er scannte 378 Sätze, überprüfte die Kapitelstruktur, markierte unbelegte starke Thesen und übertriebene Formulierungen und glich **alle 178 numerischen Werte** im Entwurf mit den tatsächlichen Experiment-Ausgabedateien ab — 0 Diskrepanzen, wobei 2 Rundungsdifferenzen dokumentiert und 1 Basisrate aus den Rohprotokollen neu berechnet wurde. +Der Befehl `audit-manuscript-draft` wurde auf ein echtes KCI-Manuskript (413 Absätze) angewendet: Er scannte 378 Sätze, überprüfte die Kapitelstruktur, markierte unbelegte starke Thesen und übertriebene Formulierungen, und **alle 178 numerischen Werte** im Entwurf wurden mit den tatsächlichen Experiment-Ausgabedateien abgeglichen — 0 Diskrepanzen, wobei 2 Rundungsdifferenzen dokumentiert und 1 Basisrate aus den Rohprotokollen neu berechnet wurde. (Dieser Abgleich erfolgte damals von Hand; seit v0.2.0 führt ihn die Option `--experiment-data` automatisch durch und meldet pro Zahl exakte Übereinstimmung, Rundungsübereinstimmung oder „in den Daten nicht gefunden“.) ## Governance-Regeln -1. Die Evidenzmatrix wird niemals unbedacht geändert. -2. `verified=true` wird niemals automatisch gesetzt — es gibt keinen automatisierten Übergang in den verifizierten Zustand. -3. Der Abgleich von Zitat/Seite dient der *Quellenverortung*, nicht der absoluten Wahrheitsprüfung. -4. Manuskript-Änderungen erfolgen ausschließlich über den kontrollierten Preview/Apply-Prozess mit Backups und Post-Apply-Guards. -5. Ergebnisse verwandter Arbeiten werden als Design-Patterns dargestellt, niemals als Leistungsbelege für PaperOps selbst. +1. Die Evidenzmatrix wird niemals unbedacht geändert; alle Neuschreibungen erfolgen atomar (temporäre Datei + Umbenennen) und werden von zeitgestempelten Backups begleitet. +2. `verified=true` wird niemals automatisch gesetzt. Der einzige zulässige Weg ist `verify-evidence --attest`, das Wer/Wann/Wie in `matrices/verification_ledger.csv` festhält; `guard-no-auto-verified` lässt jede verifizierte Zeile ohne passende Bestätigung im Register durchfallen. +3. Von einem LLM extrahierte Kandidaten werden wörtlich gegen den geparsten Quelltext geprüft — umformulierte oder erfundene Zitate werden verworfen und gemeldet — und durchlaufen anschließend dieselben menschlichen Prüfschritte wie heuristische Kandidaten. +4. Der Abgleich von Zitat/Seite dient der *Quellenverortung*, nicht der absoluten Wahrheitsprüfung. +5. Manuskript-Änderungen erfolgen ausschließlich über den kontrollierten Preview/Apply-Prozess mit Backups (außerhalb des kontrollierten Verzeichnisbaums gespeichert), SHA-256-Prüfungen sowie Post-Apply-Guard und Smoke-Test; Freigabeberichte werden gegen die aktuelle SHA-256-Summe des Manuskripts abgeglichen, sodass veraltete oder gefälschte Berichte abgelehnt werden. +6. Ergebnisse verwandter Arbeiten werden als Design-Patterns dargestellt, niemals als Leistungsbelege für PaperOps selbst. ## Was dieses Repository NICHT enthält @@ -142,10 +149,13 @@ Ausschließlich Code, Konfigurationen, Design-Dokumente und generierte Grafikque ## Ehrliche Einschränkungen -- Die Evidenzextraktion basiert derzeit auf Regeln/Heuristiken; ein LLM-gestützter Extraktor ist als separater, kontrollierter Schritt geplant. -- Die Entwurfsprüfung ist eine heuristische Kennzeichnung für die menschliche Durchsicht, keine absolute Wahrheitsprüfung. +- Der LLM-Extraktor garantiert lediglich, dass die Zitate wörtlich sind; die Qualität der *Auswahl* von These und Zitat hängt weiterhin vom Modell ab, und jeder Kandidat erfordert nach wie vor eine menschliche Prüfung. +- Die Entwurfsprüfung ist eine heuristische Kennzeichnung für die menschliche Durchsicht, keine absolute Wahrheitsprüfung; der numerische Abgleich bestätigt nur, dass die Zahlen in Ihren Datendateien vorkommen, nicht dass die Auswertung korrekt ist. - Die Verortung von Zitat und Seite validiert konstruktionsbedingt nicht den Wahrheitsgehalt einer Behauptung. - Quantitative Grafiken werden niemals ohne eine reale Datendatei generiert. +- Die Erfassung über Semantic Scholar ohne Authentifizierung wird von S2 stark gedrosselt; für diese Quelle ist ein kostenloser API-Schlüssel faktisch erforderlich. +- Das Verifizierungsregister schützt vor versehentlicher und beiläufiger automatisierter Verifizierung; es ist nicht kryptographisch abgesichert und kann einen entschlossenen Menschen nicht daran hindern, Bestätigungen in seinen eigenen lokalen Dateien zu fälschen. +- Die nicht-englischen READMEs können der englischen Fassung um eine Release-Version hinterherhinken; maßgeblich sind das englische README und der CHANGELOG. ## Lizenz diff --git a/README.es.md b/README.es.md index 45544a8..60c2508 100644 --- a/README.es.md +++ b/README.es.md @@ -17,19 +17,19 @@ ![Pipeline de extremo a extremo de PaperOps](assets/figures/fig_pipeline.svg) -PaperOps automatiza el **ciclo de vida completo de la investigación y la redacción** (recopilación de literatura, filtrado, procesamiento de PDF, extracción de evidencia, sincronización bibliográfica, edición controlada del borrador, generación reproducible de figuras y auditoría de borradores) a través de una única CLI local-first con **más de 45 comandos**. +PaperOps automatiza el **ciclo de vida completo de la investigación y la redacción** (recopilación de literatura, filtrado, procesamiento de PDF, extracción de evidencia —heurística y asistida por LLM con verificación literal de las citas—, sincronización bibliográfica, edición controlada del borrador, generación reproducible de figuras y auditoría de borradores con validación cruzada numérica contra los resultados de los experimentos) a través de una única CLI local-first con **más de 50 comandos**. -No es un redactor automático de artículos. El flujo de trabajo está automatizado, pero tres puntos de decisión críticos están reservados deliberadamente para los humanos: la adopción de evidencia, la aprobación de cambios en el borrador y la decisión de marcar algo como `verified=true`. Los mecanismos de control (guards) hacen imposible que cualquier paso automatizado falsifique estas decisiones. +No es un redactor automático de artículos. El flujo de trabajo está automatizado, pero tres puntos de decisión críticos están reservados deliberadamente para los humanos: la adopción de evidencia, la aprobación de cambios en el borrador y la decisión de marcar algo como `verified=true`. Al estado `verified=true` solo se puede llegar mediante el comando `verify-evidence` con una atestación humana explícita, que queda registrada en un libro de verificación (verification ledger) con el que `guard-no-auto-verified` hace la comprobación cruzada: tanto las ediciones manuales como las automatizadas fallan el control. ## El ciclo de vida completo, paso a paso | Etapa | Descripción | Comandos clave | Automatización | |---|---|---|---| -| 1. Recopilar | Obtener artículos de arXiv / Semantic Scholar / OpenAlex mediante perfiles temáticos | `collect`, `digest` | Automático | +| 1. Recopilar | Obtener artículos de arXiv / Semantic Scholar / OpenAlex / Crossref mediante consultas temáticas booleanas, con cabeceras de polite pool, reintentos escalonados ante errores 429 y deduplicación entre fuentes | `collect`, `digest` | Automático | | 2. Clasificar | Evaluar la relevancia, filtrar por ejes de investigación y detectar brechas | `score`, `screen`, `gap`, `brief` | Automático | | 3. Adquirir | Descargar archivos PDF, construir fichas de lectura y esquemas | `download-pdfs`, `cards`, `outline` | Automático | | 4. Procesar | Convertir PDF a secciones/referencias estructuradas a través de GROBID | `parse-grobid`, `validate-grobid-artifacts` | Automático | -| 5. Extraer | Extraer propuestas de evidencia (afirmación/cita/página) del texto procesado | `extract-evidence-candidates`, `validate-evidence-candidates` | Automático | +| 5. Extraer | Extraer propuestas de evidencia (afirmación/cita/página) del texto procesado: de forma heurística o propuestas por un LLM, con cada cita verificada literalmente contra la fuente | `extract-evidence-candidates`, `extract-evidence-llm`, `validate-evidence-candidates` | Automático | | 6. Revisar | Decidir si aceptar, modificar o rechazar cada candidato | `review-evidence-candidates`, `promotion-plan` | **Filtro Humano** | | 7. Promover | Mover la evidencia aprobada a la Matriz de Evidencia (`verified=false`) | `promote-evidence`, `audit-promoted-evidence` | Controlado | | 8. Localizar | Encontrar y asociar las páginas exactas del PDF para cada cita | `locate-pdf-pages`, `apply-page-metadata` | Controlado | @@ -37,8 +37,8 @@ No es un redactor automático de artículos. El flujo de trabajo está automatiz | 10. Escribir | Generar parches para el manuscrito con vista previa y diferencias (diff) | `manuscript-patch-preview` | Automático | | 11. Aplicar | Aplicar los parches aprobados con copia de seguridad, verificación SHA y escritura física | `apply-manuscript-patch` | **Filtro Humano** | | 12. Figuras | Crear figuras de Graphviz/Mermaid basadas en especificaciones, sin datos inventados | `propose-figures`, `render-figures`, `apply-figure-placeholder` | Controlado | -| 13. Auditar | Evaluar cualquier borrador (docx/md/qmd): estructura, afirmaciones sin fuente, exageraciones y números vs. resultados reales | `audit-manuscript-draft` | Automático | -| 14. Verificar | Garantizar que ningún proceso automático establezca `verified=true` | `guard-no-auto-verified`, `guard-paperops-overclaim`, `smoke-test` | Control automático / **Veredicto humano** | +| 13. Auditar | Evaluar cualquier borrador (docx/md/qmd): estructura, afirmaciones sin fuente, exageraciones y validación cruzada numérica automática contra los archivos de salida de los experimentos (`--experiment-data`) | `audit-manuscript-draft` | Automático | +| 14. Verificar | Un humano establece `verified=true` con una atestación registrada en el libro de verificación; los controles rechazan cualquier fila verificada que no tenga su entrada correspondiente en el libro | `verify-evidence`, `guard-no-auto-verified`, `guard-paperops-overclaim`, `smoke-test` | **Filtro Humano** / control automático | ## ¿Por qué usar esto en lugar de un chat de IA convencional? @@ -47,8 +47,8 @@ No es un redactor automático de artículos. El flujo de trabajo está automatiz | ¿De dónde viene esta frase? | Desconocido | `paper_id` + `citekey` + cita + página en la Matriz de Evidencia | | Precisión de citas | Basada en el mejor esfuerzo | Contrastada mediante `check-citekeys` contra la bibliografía real | | Modificaciones del manuscrito | Sobrescritura directa | Vista previa ➔ diff ➔ aprobación ➔ aplicación verificada por SHA ➔ backup ➔ post-auditoría | -| Estado "Verificado" | Implícito | Solo un ser humano puede establecerlo; los controles lo garantizan | -| Números en el borrador | Sin verificar | Contrastados con los archivos reales de salida de los experimentos | +| Estado "Verificado" | Implícito | Solo `verify-evidence --attest` puede establecerlo; el libro de verificación y los controles detectan cualquier otra vía | +| Números en el borrador | Sin verificar | Contrastados con los archivos reales de salida de los experimentos (`audit-manuscript-draft --experiment-data`) | | Reproducibilidad | Limitada a la sesión | SQLite + Matrices CSV + informes de auditoría + registro de actividad + fuentes de figuras | Los patrones de diseño se sintetizaron a partir de un análisis de más de 40 herramientas de investigación de código abierto (PaperQA2, STORM, GPT Researcher, AI-Scientist, ASReview, gpt_academic, ecosistema Zotero, servidores MCP — ver `docs/03_TOOL_SYNTHESIS.md`), reensamblados bajo un principio único: **ninguna afirmación entra al manuscrito sin evidencia trazable y revisada por un humano.** @@ -77,20 +77,22 @@ Los datos fluyen en una sola dirección con informes de auditoría en cada paso ```bash git clone https://github.com/SakJaeLim/paperops.git && cd paperops -python -m venv .venv +python -m venv .venv # Python 3.10+ # En Windows: .venv\Scripts\activate | En Unix: source .venv/bin/activate pip install -r requirements.txt python scripts/paperops.py init python scripts/paperops.py status ``` -Funciona de inmediato sin servicios externos para: recopilación, puntuación, filtrado, auditoría de borradores, controles de seguridad y generación de figuras. Adiciones opcionales: +A continuación, indica tu correo electrónico en `config/sources.yaml` (`openalex_mailto`): con él, las peticiones a OpenAlex/Crossref entran en sus polite pools; sin él, ambas APIs limitan el tráfico de forma agresiva (HTTP 429). Funciona de inmediato sin servicios externos para: recopilación, puntuación, filtrado, auditoría de borradores, controles de seguridad y generación de figuras. Adiciones opcionales: | Dependencia | Función | Instalación | |---|---|---| | GROBID | Procesamiento de PDF a texto estructurado | `docker run -d -p 8070:8070 lfoppiano/grobid:0.8.0` | -| Zotero + Better BibTeX | Sincronización bibliográfica real | Descargar de zotero.org e instalar Better BibTeX | -| Graphviz | Renderizado de figuras SVG/PNG | Descargar e instalar de graphviz.org | +| Zotero + Better BibTeX | Sincronización bibliográfica real (archivo `.bib` exportado) | Descargar de zotero.org e instalar Better BibTeX | +| Graphviz / Mermaid CLI | Renderizado de figuras SVG/PNG (las fuentes siempre se escriben, incluso sin ellos) | Descargar e instalar de graphviz.org | +| Clave de API de Semantic Scholar | Recopilación fiable en S2 (sin autenticación, S2 limita el tráfico de forma agresiva) | `semantic_scholar_api_key` en `config/sources.yaml` | +| `OPENAI_API_KEY` | `extract-evidence-llm` (extracción con LLM controlada y con verificación literal de las citas; cualquier endpoint compatible con OpenAI mediante `OPENAI_BASE_URL`) | Variable de entorno | ## Sesión típica @@ -113,8 +115,12 @@ python scripts/paperops.py manuscript-patch-preview python scripts/paperops.py apply-manuscript-patch --from-preview --dry-run python scripts/paperops.py apply-manuscript-patch --from-preview --apply -# Auditar tu propio borrador (docx/md/qmd) -python scripts/paperops.py audit-manuscript-draft --input mi_borrador_de_tesis.docx +# Auditar tu propio borrador (docx/md/qmd) — estructura, afirmaciones sin fuente, exageraciones +# y validación cruzada numérica automática contra tus archivos de salida de los experimentos +python scripts/paperops.py audit-manuscript-draft --input mi_borrador_de_tesis.docx --experiment-data data/experiment_outputs + +# Verificación humana (EL filtro) — registrada en el libro de verificación +python scripts/paperops.py verify-evidence --evidence-id ev_xxxx --by "Tu Nombre" --note "revisado p.3" --attest # Renderizar figuras y verificaciones finales python scripts/paperops.py render-figures @@ -126,15 +132,16 @@ python scripts/paperops.py smoke-test ## La auditoría en la práctica -El comando `audit-manuscript-draft` se utilizó en un borrador de tesis real (413 párrafos): escaneó 378 oraciones, verificó la estructura del capítulo, señaló afirmaciones fuertes sin fuente y lenguaje exagerado, y realizó una validación cruzada de **los 178 valores numéricos** del borrador contra los archivos reales de salida de los experimentos — 0 discrepancias, con 2 diferencias de redondeo explicadas y 1 tasa base recalculada a partir de los registros de predicción. +El comando `audit-manuscript-draft` se utilizó en un manuscrito KCI real (413 párrafos): escaneó 378 oraciones, verificó la estructura del capítulo, señaló afirmaciones fuertes sin fuente y lenguaje exagerado, y **los 178 valores numéricos** del borrador se contrastaron con los archivos reales de salida de los experimentos — 0 discrepancias, con 2 diferencias de redondeo explicadas y 1 tasa base recalculada a partir de los registros de predicción. (En aquel momento esa validación cruzada se hizo de forma manual; desde la v0.2.0 la opción `--experiment-data` la realiza automáticamente e informa, para cada número, de si hay coincidencia exacta, coincidencia por redondeo o si no se encuentra en los datos.) ## Reglas de gobernanza -1. La Matriz de Evidencia nunca se modifica de forma casual. -2. `verified=true` nunca se establece automáticamente: no existe una transición automatizada hacia el estado verificado. -3. La alineación de cita/página es una comprobación de correspondencia de fuentes, no una validación de la verdad absoluta. -4. Las modificaciones al manuscrito ocurren únicamente a través de la aplicación controlada con copias de seguridad e informes posteriores. -5. Los hallazgos de trabajos relacionados se presentan como patrones de diseño, nunca como evidencia de rendimiento para el propio PaperOps. +1. La Matriz de Evidencia nunca se modifica de forma casual; todas las reescrituras son atómicas (archivo temporal + renombrado) y van acompañadas de copias de seguridad con marca de tiempo. +2. `verified=true` nunca se establece automáticamente. La única vía autorizada es `verify-evidence --attest`, que registra quién, cuándo y cómo en `matrices/verification_ledger.csv`; `guard-no-auto-verified` rechaza cualquier fila verificada que no tenga su atestación correspondiente en el libro. +3. Los candidatos extraídos por un LLM se contrastan literalmente con el texto fuente procesado — las citas parafraseadas o inventadas se descartan y se reportan — y después pasan por los mismos filtros de revisión humana que los candidatos heurísticos. +4. La alineación de cita/página es una comprobación de correspondencia de fuentes, no una validación de la verdad absoluta. +5. Las modificaciones al manuscrito ocurren únicamente a través de la vista previa y la aplicación controladas, con copias de seguridad (almacenadas fuera del árbol controlado), verificaciones SHA-256 y control + smoke-test posteriores a la aplicación; los informes de aprobación se contrastan con el SHA-256 actual del manuscrito, de modo que se rechazan los informes obsoletos o falsificados. +6. Los hallazgos de trabajos relacionados se presentan como patrones de diseño, nunca como evidencia de rendimiento para el propio PaperOps. ## Lo que este repositorio NO incluye @@ -142,10 +149,13 @@ Código, configuraciones, documentos de diseño y fuentes de figuras únicamente ## Limitaciones honestas -- La extracción de evidencia se basa actualmente en reglas/heurísticas; un extractor asistido por LLM es una etapa planificada por separado. -- La auditoría de borradores es un filtrado heurístico para revisión humana, no una validación de la verdad. +- El extractor con LLM solo garantiza que las citas sean literales; la calidad de la *selección* de afirmación y cita sigue dependiendo del modelo, y cada candidato sigue requiriendo revisión humana. +- La auditoría de borradores es un filtrado heurístico para revisión humana, no una validación de la verdad; la validación cruzada numérica comprueba que los números aparecen en tus archivos de datos, no que el análisis sea correcto. - El acoplamiento de citas y páginas no valida la veracidad de una afirmación, por diseño. - Las figuras con resultados cuantitativos nunca se generan sin un archivo de datos real que las respalde. +- La recopilación en Semantic Scholar sin autenticación está fuertemente limitada por S2; en la práctica, esa fuente requiere una clave de API gratuita. +- El libro de verificación disuade de la verificación automatizada accidental o descuidada; no es un mecanismo criptográfico y no puede impedir que una persona decidida falsifique atestaciones en sus propios archivos locales. +- Los README en idiomas distintos del inglés pueden ir una versión por detrás del inglés; el README en inglés y el CHANGELOG son la referencia autoritativa. ## Licencia diff --git a/README.fr.md b/README.fr.md index d00ef99..2cd6cab 100644 --- a/README.fr.md +++ b/README.fr.md @@ -18,26 +18,31 @@ ![Pipeline PaperOps de bout en bout](assets/figures/fig_pipeline.svg) PaperOps automatise **l'ensemble du cycle de vie de la rédaction scientifique** -— collecte de littérature, tri, analyse de PDF, extraction de preuves, +— collecte de littérature, tri, analyse de PDF, extraction de preuves +(heuristique et assistée par LLM avec contrôle mot pour mot des citations), synchronisation bibliographique, édition de manuscrit sous garde-fous, -génération de figures reproductibles et audit de brouillons — via une seule -CLI locale comptant **plus de 45 commandes**. +génération de figures reproductibles et audit de brouillons avec recoupement +numérique face aux sorties d'expériences — via une seule CLI locale comptant +**plus de 50 commandes**. Ce n'est *pas* un rédacteur automatique d'articles. Le pipeline est automatisé, mais trois points de jugement sont délibérément réservés à l'humain : l'adoption des preuves, l'approbation des modifications du -manuscrit et la décision `verified=true`. Des garde-fous empêchent toute -étape automatisée de les falsifier. +manuscrit et la décision `verified=true`. L'état `verified=true` ne peut être +atteint que par la commande `verify-evidence`, avec une attestation humaine +explicite consignée dans un registre de vérification que +`guard-no-auto-verified` recoupe — modifications manuelles comme +modifications automatisées échouent toutes deux au garde-fou. ## Le cycle complet, étape par étape | Étape | Contenu | Commandes clés | Automatisation | |---|---|---|---| -| 1. Collecte | Récupération depuis arXiv / Semantic Scholar / OpenAlex selon un profil thématique | `collect`, `digest` | Automatique | +| 1. Collecte | Récupération depuis arXiv / Semantic Scholar / OpenAlex / Crossref avec requêtes thématiques booléennes, en-têtes « polite pool », backoff sur 429 et déduplication inter-sources | `collect`, `digest` | Automatique | | 2. Tri | Score de pertinence, criblage par axes de recherche, détection des lacunes | `score`, `screen`, `gap`, `brief` | Automatique | | 3. Acquisition | Téléchargement des PDF, fiches de lecture et plans | `download-pdfs`, `cards`, `outline` | Automatique | | 4. Analyse | PDF → sections/références structurées via GROBID | `parse-grobid`, `validate-grobid-artifacts` | Automatique | -| 5. Extraction | Candidats de preuve (affirmation/citation/page) depuis le texte analysé | `extract-evidence-candidates`, `validate-evidence-candidates` | Automatique | +| 5. Extraction | Candidats de preuve (affirmation/citation/page) depuis le texte analysé — heuristiques, ou proposés par LLM avec chaque citation vérifiée mot pour mot face à la source | `extract-evidence-candidates`, `extract-evidence-llm`, `validate-evidence-candidates` | Automatique | | 6. Revue | Décision garder / réviser / rejeter pour chaque candidat | `review-evidence-candidates`, `promotion-plan` | **Porte humaine** | | 7. Promotion | Transfert des preuves approuvées vers la matrice de preuves (`verified=false`) | `promote-evidence`, `audit-promoted-evidence` | Sous garde-fous | | 8. Localisation | Recherche et enregistrement de la page PDF exacte de chaque citation | `locate-pdf-pages`, `apply-page-metadata` | Sous garde-fous | @@ -45,8 +50,8 @@ manuscrit et la décision `verified=true`. Des garde-fous empêchent toute | 10. Rédaction | Génération de correctifs de manuscrit en preview + diff | `manuscript-patch-preview` | Automatique | | 11. Application | Application des correctifs approuvés avec sauvegarde + contrôle SHA + écriture LF | `apply-manuscript-patch` | **Porte humaine** | | 12. Figures | Figures Graphviz/Mermaid pilotées par spécification, aucune donnée fabriquée | `propose-figures`, `render-figures`, `apply-figure-placeholder` | Sous garde-fous | -| 13. Audit de brouillon | Vérification de tout brouillon (docx/md/qmd) : structure, affirmations sans source, exagérations, chiffres confrontés aux sorties d'expériences réelles | `audit-manuscript-draft` | Automatique | -| 14. Vérification | Garantie qu'aucune automatisation n'a jamais posé `verified=true` | `guard-no-auto-verified`, `guard-paperops-overclaim`, `smoke-test` | Garde automatique / **verdict humain** | +| 13. Audit de brouillon | Vérification de tout brouillon (docx/md/qmd) : structure, affirmations sans source, exagérations, et recoupement numérique automatique face aux fichiers de sortie d'expériences (`--experiment-data`) | `audit-manuscript-draft` | Automatique | +| 14. Vérification | L'humain pose `verified=true` avec une attestation consignée dans le registre de vérification ; les garde-fous font échouer toute ligne vérifiée sans entrée correspondante dans le registre | `verify-evidence`, `guard-no-auto-verified`, `guard-paperops-overclaim`, `smoke-test` | **Porte humaine** / garde automatique | ## Pourquoi cela plutôt qu'un chat LLM ? @@ -55,8 +60,8 @@ manuscrit et la décision `verified=true`. Des garde-fous empêchent toute | D'où vient cette phrase ? | inconnu | `paper_id` + `citekey` + citation + page dans la matrice de preuves | | Exactitude des citations | au mieux | `check-citekeys` contre le BibTeX canonique | | Édition du manuscrit | écrasement direct | preview → diff → approbation → application contrôlée par SHA → sauvegarde → audit | -| Statut « vérifié » | implicite | seul un humain peut le poser ; les garde-fous l'imposent | -| Chiffres du brouillon | non vérifiés | confrontés aux fichiers de sortie d'expériences réelles | +| Statut « vérifié » | implicite | seul `verify-evidence --attest` peut le poser ; le registre de vérification + les garde-fous détectent tout le reste | +| Chiffres du brouillon | non vérifiés | confrontés aux fichiers de sortie d'expériences réels (`audit-manuscript-draft --experiment-data`) | | Reproductibilité | liée à la session | SQLite + matrices CSV + rapports d'audit + journal + sources des figures | Les patrons de conception proviennent d'une étude de plus de 40 outils de @@ -92,21 +97,27 @@ relancés après chaque modification. ```bash git clone https://github.com/SakJaeLim/paperops.git && cd paperops -python -m venv .venv +python -m venv .venv # Python 3.10+ # Windows : .venv\Scripts\activate | Unix : source .venv/bin/activate pip install -r requirements.txt python scripts/paperops.py init python scripts/paperops.py status ``` -Fonctionne immédiatement sans service externe : collecte, scoring, criblage, -audit de brouillon, garde-fous, sources de figures. Compléments optionnels : +Renseignez ensuite votre adresse e-mail dans `config/sources.yaml` +(`openalex_mailto`) — elle place les requêtes OpenAlex/Crossref dans leurs +« polite pools » ; sans elle, les deux API limitent fortement le débit +(HTTP 429). Fonctionne immédiatement sans service externe : collecte, +scoring, criblage, audit de brouillon, garde-fous, sources de figures. +Compléments optionnels : | Dépendance | Permet | Installation | |---|---|---| | GROBID | Analyse PDF → texte structuré | `docker run -d -p 8070:8070 lfoppiano/grobid:0.8.0` | -| Zotero + Better BibTeX | Synchronisation bibliographique canonique | zotero.org + plugin Better BibTeX | -| Graphviz | Rendu des figures SVG/PNG | graphviz.org/download | +| Zotero + Better BibTeX | Synchronisation bibliographique canonique (fichier `.bib` exporté) | zotero.org + plugin Better BibTeX | +| Graphviz / Mermaid CLI | Rendu des figures SVG/PNG (les sources sont toujours écrites, même sans eux) | graphviz.org/download | +| Clé API Semantic Scholar | Collecte S2 fiable (sans authentification, S2 limite fortement le débit) | `semantic_scholar_api_key` dans `config/sources.yaml` | +| `OPENAI_API_KEY` | `extract-evidence-llm` (extraction LLM sous garde-fous, vérifiée mot pour mot ; tout endpoint compatible OpenAI via `OPENAI_BASE_URL`) | variable d'environnement | ## Session type @@ -129,8 +140,13 @@ python scripts/paperops.py manuscript-patch-preview python scripts/paperops.py apply-manuscript-patch --from-preview --dry-run python scripts/paperops.py apply-manuscript-patch --from-preview --apply -# Auditer son propre brouillon (docx/md/qmd) -python scripts/paperops.py audit-manuscript-draft --input my_thesis_draft.docx +# Auditer son propre brouillon (docx/md/qmd) — structure, affirmations sans +# source, exagérations, et recoupement numérique automatique face à vos +# fichiers de sortie d'expériences +python scripts/paperops.py audit-manuscript-draft --input my_thesis_draft.docx --experiment-data data/experiment_outputs + +# Vérification humaine (LA porte) — consignée dans le registre de vérification +python scripts/paperops.py verify-evidence --evidence-id ev_xxxx --by "Votre nom" --note "vérifié p. 3" --attest # Figures et contrôles finaux python scripts/paperops.py render-figures @@ -144,23 +160,37 @@ python scripts/paperops.py smoke-test `audit-manuscript-draft` a été appliqué à un vrai manuscrit KCI (413 paragraphes) : 378 phrases analysées, structure des chapitres vérifiée, -affirmations fortes sans source et formulations exagérées signalées, et -**les 178 valeurs numériques** du brouillon confrontées aux fichiers de -sortie d'expériences réels — 0 écart, 2 différences d'arrondi expliquées, -1 taux de base recalculé depuis les journaux bruts. L'audit ne modifie -jamais le brouillon et ne marque jamais rien comme vérifié ; il produit -uniquement un rapport (MD + CSV) pour l'auteur. +affirmations fortes sans source et formulations exagérées signalées, et les +178 valeurs numériques du brouillon confrontées aux fichiers de sortie +d'expériences réels — 0 écart, 2 différences d'arrondi expliquées, 1 taux de +base recalculé depuis les journaux bruts de prédiction. (Ce recoupement était +alors manuel ; depuis la v0.2.0, l'option `--experiment-data` l'effectue +automatiquement : pour chaque chiffre, un statut correspondance exacte, +correspondance à l'arrondi près, ou introuvable dans les données.) L'audit ne +modifie jamais le brouillon et ne marque jamais rien comme vérifié ; il +produit uniquement un rapport (MD + CSV) pour l'auteur. ## Règles de gouvernance -1. La matrice de preuves n'est jamais modifiée à la légère. -2. `verified=true` n'est jamais posé automatiquement — il n'existe aucune - transition automatisée vers l'état vérifié. -3. La correspondance citation/page est un *alignement de source*, pas une +1. La matrice de preuves n'est jamais modifiée à la légère ; toutes les + réécritures sont atomiques (fichier temporaire + renommage) et + accompagnées de sauvegardes horodatées. +2. `verified=true` n'est jamais posé automatiquement. La seule voie autorisée + est `verify-evidence --attest`, qui consigne qui/quand/comment dans + `matrices/verification_ledger.csv` ; `guard-no-auto-verified` fait échouer + toute ligne vérifiée sans attestation correspondante dans le registre. +3. Les candidats extraits par LLM sont vérifiés mot pour mot face au texte + source analysé — les citations paraphrasées ou inventées sont écartées et + signalées — puis passent par les mêmes portes de revue humaine que les + candidats heuristiques. +4. La correspondance citation/page est un *alignement de source*, pas une validation de vérité. -4. Les modifications du manuscrit passent uniquement par preview/apply sous - garde-fous, avec sauvegardes et garde + smoke-test après application. -5. Les résultats des travaux connexes sont cités comme patrons de +5. Les modifications du manuscrit passent uniquement par preview/apply sous + garde-fous, avec sauvegardes (stockées hors de l'arborescence gardée), + contrôles SHA-256 et garde + smoke-test après application ; les rapports + d'approbation sont recoupés avec le SHA-256 actuel du manuscrit, si bien + que les rapports périmés ou fabriqués sont rejetés. +6. Les résultats des travaux connexes sont cités comme patrons de conception, jamais comme preuves de performance de PaperOps lui-même. ## Ce que ce dépôt NE contient PAS @@ -175,14 +205,27 @@ chaque publication (`scripts/build_public_release.py`). ## Limites assumées -- L'extraction de preuves repose sur des heuristiques par mots-clés ; - une extraction assistée par LLM est prévue comme étape gardée distincte. +- L'extracteur LLM garantit uniquement que les citations sont mot pour mot ; + la qualité de la *sélection* des affirmations et des citations dépend + toujours du modèle, et chaque candidat passe toujours par une revue + humaine. - L'audit de brouillon est un signalement heuristique destiné à la revue - humaine, pas une validation de vérité. + humaine, pas une validation de vérité ; le recoupement numérique vérifie + que les chiffres figurent bien dans vos fichiers de données, pas que + l'analyse est correcte. - L'alignement citation/page ne valide pas la véracité d'une affirmation — par conception. - Aucune figure de résultats quantitatifs n'est générée sans fichier de données réel. +- La collecte Semantic Scholar sans authentification est fortement limitée + en débit par S2 ; une clé API gratuite est en pratique indispensable pour + cette source. +- Le registre de vérification dissuade la vérification automatisée + accidentelle ou négligente ; il n'est pas cryptographique et ne peut pas + empêcher quelqu'un de déterminé de falsifier des attestations dans ses + propres fichiers locaux. +- Les README non anglophones peuvent avoir une version de retard sur la + version anglaise ; le README anglais et le CHANGELOG font foi. ## Licence diff --git a/README.ja.md b/README.ja.md index 4869e19..4769bd8 100644 --- a/README.ja.md +++ b/README.ja.md @@ -18,23 +18,27 @@ ![PaperOps エンドツーエンド・パイプライン](assets/figures/fig_pipeline.svg) PaperOps は**研究執筆のライフサイクル全体** — 文献収集、スクリーニング、 -PDF 解析、エビデンス抽出、文献管理同期、ガード付き原稿編集、再現可能な -図表生成、ドラフト監査 — を、**45 以上のコマンド**を持つローカルファースト -CLI ひとつで自動化します。 +PDF 解析、エビデンス抽出(ヒューリスティック、および引用の逐語チェックを伴う +LLM 支援)、文献管理同期、ガード付き原稿編集、再現可能な図表生成、実験出力 +との数値自動照合を伴うドラフト監査 — を、**50 以上のコマンド**を持つ +ローカルファースト CLI ひとつで自動化します。 論文を*代筆する*ツールではありません。パイプラインは自動ですが、判断を要する 3 つのポイント — エビデンスの採用、原稿修正の承認、`verified=true` の判定 — -は意図的に人間に残されており、ガード機構が自動化による偽装を防ぎます。 +は意図的に人間に残されています。`verified=true` の状態に入れるのは、明示的な +人間の確約(attestation)を伴う `verify-evidence` コマンドだけです。確約は +検証台帳(verification ledger)に記録され、`guard-no-auto-verified` がそれを +照合します — 手作業による編集も自動化による編集も、ともにガードで失敗します。 ## ライフサイクル全体(ステージ別) | ステージ | 内容 | 主要コマンド | 自動化 | |---|---|---|---| -| 1. 収集 | トピックプロファイルに基づき arXiv / Semantic Scholar / OpenAlex から取得 | `collect`, `digest` | 自動 | +| 1. 収集 | arXiv / Semantic Scholar / OpenAlex / Crossref からブーリアン・トピッククエリで取得 — polite pool ヘッダー、429 バックオフ、ソース間の重複排除つき | `collect`, `digest` | 自動 | | 2. 選別 | 関連度スコアリング、研究軸別スクリーニング、研究ギャップ発見 | `score`, `screen`, `gap`, `brief` | 自動 | | 3. 取得 | PDF ダウンロード、論文カード・アウトライン生成 | `download-pdfs`, `cards`, `outline` | 自動 | | 4. 解析 | GROBID で PDF → 構造化セクション/参考文献 | `parse-grobid`, `validate-grobid-artifacts` | 自動 | -| 5. 抽出 | 解析テキストから主張/引用/ページのエビデンス候補を抽出 | `extract-evidence-candidates`, `validate-evidence-candidates` | 自動 | +| 5. 抽出 | 解析テキストから主張/引用/ページのエビデンス候補を抽出 — ヒューリスティック、またはすべての引用を出典と逐語照合する LLM 提案 | `extract-evidence-candidates`, `extract-evidence-llm`, `validate-evidence-candidates` | 自動 | | 6. レビュー | 候補ごとに採用/修正/却下を決定 | `review-evidence-candidates`, `promotion-plan` | **人間のゲート** | | 7. 昇格 | 承認済みエビデンスを Evidence Matrix へ移動(`verified=false`) | `promote-evidence`, `audit-promoted-evidence` | ガード付き | | 8. ページ特定 | 各引用の正確な PDF ページを特定・記録 | `locate-pdf-pages`, `apply-page-metadata` | ガード付き | @@ -42,8 +46,8 @@ CLI ひとつで自動化します。 | 10. 執筆 | 原稿パッチを preview + diff として生成 | `manuscript-patch-preview` | 自動 | | 11. 適用 | 承認済みパッチをバックアップ + SHA 検証 + LF 書込で適用 | `apply-manuscript-patch` | **人間のゲート** | | 12. 図表 | 仕様駆動の Graphviz/Mermaid 図、データ捏造禁止 | `propose-figures`, `render-figures`, `apply-figure-placeholder` | ガード付き | -| 13. ドラフト監査 | 任意の草稿(docx/md/qmd)を検査:構造、出典なし主張、誇張、数値と実験出力の照合 | `audit-manuscript-draft` | 自動 | -| 14. 検証 | いかなる自動化も `verified=true` を設定していないことを強制 | `guard-no-auto-verified`, `guard-paperops-overclaim`, `smoke-test` | 自動ガード / **人間の判定** | +| 13. ドラフト監査 | 任意の草稿(docx/md/qmd)を検査:構造、出典なし主張、誇張、および `--experiment-data` による実験出力ファイルとの数値自動照合 | `audit-manuscript-draft` | 自動 | +| 14. 検証 | 人間が確約とともに `verified=true` を設定し、検証台帳に記録する。台帳に対応する記録のない verified 行はガードが失敗させる | `verify-evidence`, `guard-no-auto-verified`, `guard-paperops-overclaim`, `smoke-test` | **人間のゲート** / 自動ガード | ## チャット LLM ではなくこれを使う理由 @@ -52,8 +56,8 @@ CLI ひとつで自動化します。 | この文の出典は? | 不明 | Evidence Matrix の `paper_id` + `citekey` + 引用 + ページ | | 引用の正確性 | ベストエフォート | 正準 BibTeX と `check-citekeys` で照合 | | 原稿編集 | 直接上書き | preview → diff → 承認 → SHA 検証適用 → バックアップ → 事後監査 | -| 「検証済み」状態 | 暗黙的 | 人間のみが設定可能、ガードが強制 | -| 草稿内の数値 | 未確認 | 実際の実験出力ファイルと自動照合 | +| 「検証済み」状態 | 暗黙的 | `verify-evidence --attest` でのみ設定可能;検証台帳 + ガードがそれ以外をすべて検出 | +| 草稿内の数値 | 未確認 | 実際の実験出力ファイルと自動照合(`audit-manuscript-draft --experiment-data`) | | 再現性 | セッション限り | SQLite + CSV マトリクス + 監査レポート + 活動ログ + 図表ソース | 40 以上のオープンソース研究ツール(PaperQA2、STORM、GPT Researcher、 @@ -87,21 +91,26 @@ Evidence Matrix → パッチ preview → (人間) → 原稿**、変更のた ```bash git clone https://github.com/SakJaeLim/paperops.git && cd paperops -python -m venv .venv +python -m venv .venv # Python 3.10+ # Windows: .venv\Scripts\activate | Unix: source .venv/bin/activate pip install -r requirements.txt python scripts/paperops.py init python scripts/paperops.py status ``` -外部サービスなしで即動作:収集、スコアリング、スクリーニング、ドラフト監査、 -ガード、図表ソース生成。オプションの追加コンポーネント: +続いて `config/sources.yaml` の `openalex_mailto` に自分のメールアドレスを +設定してください — OpenAlex/Crossref へのリクエストが polite pool に入ります。 +設定しないと両 API とも強くスロットリングされます(HTTP 429)。外部サービス +なしで即動作:収集、スコアリング、スクリーニング、ドラフト監査、ガード、 +図表ソース生成。オプションの追加コンポーネント: | 依存 | 有効になる機能 | インストール | |---|---|---| | GROBID | PDF → 構造化テキスト解析 | `docker run -d -p 8070:8070 lfoppiano/grobid:0.8.0` | -| Zotero + Better BibTeX | 正準文献管理の同期 | zotero.org + Better BibTeX プラグイン | -| Graphviz | SVG/PNG 図表レンダリング | graphviz.org/download | +| Zotero + Better BibTeX | 正準文献管理の同期(エクスポートした `.bib` ファイル) | zotero.org + Better BibTeX プラグイン | +| Graphviz / Mermaid CLI | SVG/PNG 図表レンダリング(なくてもソースは常に出力される) | graphviz.org/download | +| Semantic Scholar API キー | 安定した S2 収集(未認証の S2 はレート制限が強い) | `config/sources.yaml` の `semantic_scholar_api_key` | +| `OPENAI_API_KEY` | `extract-evidence-llm`(ガード付き・逐語チェック付きの LLM 抽出;`OPENAI_BASE_URL` で OpenAI 互換エンドポイントも利用可) | 環境変数 | ## 典型的なセッション @@ -124,8 +133,12 @@ python scripts/paperops.py manuscript-patch-preview python scripts/paperops.py apply-manuscript-patch --from-preview --dry-run python scripts/paperops.py apply-manuscript-patch --from-preview --apply -# 自分のドラフトを監査(docx/md/qmd) -python scripts/paperops.py audit-manuscript-draft --input my_thesis_draft.docx +# 自分のドラフトを監査(docx/md/qmd)— 構造、出典なし主張、誇張、 +# および実験出力ファイルとの数値自動照合 +python scripts/paperops.py audit-manuscript-draft --input my_thesis_draft.docx --experiment-data data/experiment_outputs + +# 人間による検証(核心のゲート)— 検証台帳に記録される +python scripts/paperops.py verify-evidence --evidence-id ev_xxxx --by "氏名" --note "原文 p.3 確認" --attest # 図表と最終チェック python scripts/paperops.py render-figures @@ -139,20 +152,30 @@ python scripts/paperops.py smoke-test `audit-manuscript-draft` を実際の KCI 投稿原稿(413 段落)に適用: 378 文をスキャンし、章構造を点検し、出典のない強い主張と誇張表現をフラグし、 -草稿内の**全 178 個の数値**を実際の実験出力ファイルと照合 — 不一致 0 件、 +草稿内の 178 個の数値すべてを実際の実験出力ファイルと照合 — 不一致 0 件、 丸め誤差 2 件は説明済み、ベースレート 1 件は生の予測ログから再計算して確認。 +(当時この照合は手作業でした。v0.2.0 以降は `--experiment-data` フラグが、 +数値ごとに「完全一致 / 丸め一致 / データ内に見つからず」を自動判定します。) 監査はドラフトを決して修正せず、verified 状態も作りません。著者のための 発見レポート(MD + CSV)のみを生成します。 ## ガバナンスルール -1. Evidence Matrix をみだりに変更しない。 -2. `verified=true` は決して自動設定されない — verified 状態への自動遷移は - 存在しない。 -3. 引用/ページ照合は*出典アライメント*であり、真実性の検証ではない。 -4. 原稿編集はバックアップと事後ガード + スモークテストを伴うガード付き - preview/apply のみ。 -5. 関連研究の知見は設計パターンとしてのみ引用し、PaperOps 自体の性能根拠と +1. Evidence Matrix をみだりに変更しない。書き換えはすべてアトミック + (一時ファイル + リネーム)で、タイムスタンプ付きバックアップを伴う。 +2. `verified=true` は決して自動設定されない。許された唯一の経路は + `verify-evidence --attest` であり、誰が/いつ/どのように行ったかが + `matrices/verification_ledger.csv` に記録される。台帳に確約のない + verified 行は `guard-no-auto-verified` が失敗させる。 +3. LLM が抽出した候補は解析済みの出典テキストと逐語照合される — 言い換えや + 捏造された引用は破棄・報告される — うえで、ヒューリスティック候補と同じ + 人間レビューのゲートを通る。 +4. 引用/ページ照合は*出典アライメント*であり、真実性の検証ではない。 +5. 原稿編集は、バックアップ(ガード対象ツリーの外に保存)、SHA-256 検証、 + 事後ガード + スモークテストを伴うガード付き preview/apply のみ。承認 + レポートは現在の原稿の SHA-256 と照合され、古いものや捏造されたものは + 拒否される。 +6. 関連研究の知見は設計パターンとしてのみ引用し、PaperOps 自体の性能根拠と して誇張しない。 ## このリポジトリに含まれないもの @@ -165,12 +188,20 @@ python scripts/paperops.py smoke-test ## 正直な限界 -- エビデンス抽出はキーワード/ヒューリスティックベース。LLM 支援抽出は別途 - ガード付きステップとして計画中。 +- LLM 抽出器が保証するのは引用が逐語であることだけ。どの主張/引用を選ぶかの + 品質は依然としてモデル次第であり、すべての候補は今も人間のレビューを要する。 - ドラフト監査は人間のレビューのためのヒューリスティックなフラグ付けであり、 - 真実性の検証ではない。 + 真実性の検証ではない。数値照合はその数値がデータファイルに存在するかを + 確認するだけで、分析が正しいことを検証するものではない。 - 引用/ページのアライメントは主張の真実性を検証しない — 設計上の意図。 - 実データファイルなしに定量結果の図表は決して生成しない。 +- 未認証の Semantic Scholar 収集は S2 側のレート制限が強く、そのソースには + 無料 API キーが事実上必須。 +- 検証台帳は偶発的・不注意な自動検証を抑止する仕組みであり、暗号学的な保証 + ではない。自分のローカルファイル上で確約を意図的に偽造する人間までは + 止められない。 +- 非英語版 README は英語版より 1 リリース遅れることがある。英語版 README と + CHANGELOG が正典。 ## ライセンス diff --git a/README.zh.md b/README.zh.md index 59c9284..23ffaee 100644 --- a/README.zh.md +++ b/README.zh.md @@ -17,23 +17,26 @@ ![PaperOps 端到端流水线](assets/figures/fig_pipeline.svg) -PaperOps 通过一个本地优先的 CLI(**45+ 条命令**)自动化**研究写作的全生命周期**: -文献收集、筛选、PDF 解析、证据提取、文献库同步、受保护的稿件编辑、可复现的 -图表生成,以及草稿审计。 +PaperOps 通过一个本地优先的 CLI(**50+ 条命令**)自动化**研究写作的全生命周期**: +文献收集、筛选、PDF 解析、证据提取(启发式,以及强制逐字核对引文的 LLM 辅助 +提取)、文献库同步、受保护的稿件编辑、可复现的图表生成,以及与实验输出做 +数值交叉核对的草稿审计。 它*不是*自动写论文的工具。流水线是自动的,但三个判断点刻意保留给人: -证据采纳、稿件修改批准、`verified=true` 判定。守卫机制(guard)使任何 -自动化步骤都无法伪造这些状态。 +证据采纳、稿件修改批准、`verified=true` 判定。`verified=true` 只能通过 +`verify-evidence` 命令、以明确的人工确认(attestation)进入,并记录在验证台账 +(verification ledger)中,由 `guard-no-auto-verified` 逐条比对 — 手工修改和 +自动化修改都会被守卫判定失败。 ## 完整生命周期(分阶段) | 阶段 | 内容 | 关键命令 | 自动化 | |---|---|---|---| -| 1. 收集 | 基于主题画像从 arXiv / Semantic Scholar / OpenAlex 抓取论文 | `collect`, `digest` | 自动 | +| 1. 收集 | 从 arXiv / Semantic Scholar / OpenAlex / Crossref 抓取论文,支持布尔主题查询、polite pool 请求头、429 退避重试与跨来源去重 | `collect`, `digest` | 自动 | | 2. 分流 | 相关性打分、按研究轴筛选、发现研究空白 | `score`, `screen`, `gap`, `brief` | 自动 | | 3. 获取 | 下载 PDF,生成论文卡片与提纲 | `download-pdfs`, `cards`, `outline` | 自动 | | 4. 解析 | 通过 GROBID 将 PDF 转为结构化章节/参考文献 | `parse-grobid`, `validate-grobid-artifacts` | 自动 | -| 5. 提取 | 从解析文本中提取主张/引文/页码证据候选 | `extract-evidence-candidates`, `validate-evidence-candidates` | 自动 | +| 5. 提取 | 从解析文本中提取主张/引文/页码证据候选 — 启发式,或由 LLM 提议且每条引文都与原文逐字核验 | `extract-evidence-candidates`, `extract-evidence-llm`, `validate-evidence-candidates` | 自动 | | 6. 审阅 | 对每个候选做采纳/修改/拒绝决定 | `review-evidence-candidates`, `promotion-plan` | **人工关口** | | 7. 晋升 | 将批准的证据移入证据矩阵(`verified=false`) | `promote-evidence`, `audit-promoted-evidence` | 受保护 | | 8. 定位 | 为每条引文找到并记录确切的 PDF 页码 | `locate-pdf-pages`, `apply-page-metadata` | 受保护 | @@ -41,8 +44,8 @@ PaperOps 通过一个本地优先的 CLI(**45+ 条命令**)自动化**研究写 | 10. 写作 | 以 preview + diff 形式生成稿件补丁 | `manuscript-patch-preview` | 自动 | | 11. 应用 | 以备份 + SHA 校验 + LF 写入方式应用已批准补丁 | `apply-manuscript-patch` | **人工关口** | | 12. 图表 | 基于规格的 Graphviz/Mermaid 图表,禁止伪造数据 | `propose-figures`, `render-figures`, `apply-figure-placeholder` | 受保护 | -| 13. 草稿审计 | 检查任意草稿(docx/md/qmd):结构、无来源主张、夸大表述、数值与真实实验输出对照 | `audit-manuscript-draft` | 自动 | -| 14. 验证 | 强制确认没有任何自动化设置过 `verified=true` | `guard-no-auto-verified`, `guard-paperops-overclaim`, `smoke-test` | 自动守卫 / **人工裁定** | +| 13. 草稿审计 | 检查任意草稿(docx/md/qmd):结构、无来源主张、夸大表述,以及通过 `--experiment-data` 与实验输出文件做数值自动交叉核对 | `audit-manuscript-draft` | 自动 | +| 14. 验证 | 由人设置 `verified=true`,并将确认记录进验证台账;台账中没有对应条目的 verified 行会被守卫判定失败 | `verify-evidence`, `guard-no-auto-verified`, `guard-paperops-overclaim`, `smoke-test` | **人工关口** / 自动守卫 | ## 为什么不直接用聊天 LLM? @@ -51,8 +54,8 @@ PaperOps 通过一个本地优先的 CLI(**45+ 条命令**)自动化**研究写 | 这句话的出处? | 未知 | 证据矩阵中的 `paper_id` + `citekey` + 引文 + 页码 | | 引用正确性 | 尽力而为 | 与规范 BibTeX 进行 `check-citekeys` 比对 | | 稿件编辑 | 直接覆盖 | preview → diff → 批准 → SHA 校验应用 → 备份 → 事后审计 | -| "已验证"状态 | 含糊 | 只有人能设置,守卫强制执行 | -| 草稿中的数字 | 未核对 | 与真实实验输出文件自动对照 | +| "已验证"状态 | 含糊 | 只有 `verify-evidence --attest` 能设置;验证台账 + 守卫会抓出其他一切 | +| 草稿中的数字 | 未核对 | 与真实实验输出文件交叉核对(`audit-manuscript-draft --experiment-data`) | | 可复现性 | 局限于会话 | SQLite + CSV 矩阵 + 审计报告 + 活动日志 + 图表源文件 | 设计模式来自对 40+ 开源研究工具的调研(PaperQA2、STORM、GPT Researcher、 @@ -85,21 +88,25 @@ AI-Scientist、ASReview、gpt_academic、Zotero 生态、MCP 服务器 — 见 ```bash git clone https://github.com/SakJaeLim/paperops.git && cd paperops -python -m venv .venv +python -m venv .venv # Python 3.10+ # Windows: .venv\Scripts\activate | Unix: source .venv/bin/activate pip install -r requirements.txt python scripts/paperops.py init python scripts/paperops.py status ``` -无需外部服务即可使用:收集、打分、筛选、草稿审计、守卫、图表源生成。 -可选附加组件: +然后在 `config/sources.yaml` 的 `openalex_mailto` 中填入你的邮箱 — 它会让 +OpenAlex/Crossref 请求进入各自的 polite pool;不填时这两个 API 都会严厉限流 +(HTTP 429)。无需外部服务即可使用:收集、打分、筛选、草稿审计、守卫、图表源 +生成。可选附加组件: | 依赖 | 启用功能 | 安装 | |---|---|---| | GROBID | PDF → 结构化文本解析 | `docker run -d -p 8070:8070 lfoppiano/grobid:0.8.0` | -| Zotero + Better BibTeX | 规范文献库同步 | zotero.org + Better BibTeX 插件 | -| Graphviz | SVG/PNG 图表渲染 | graphviz.org/download | +| Zotero + Better BibTeX | 规范文献库同步(导出的 `.bib` 文件) | zotero.org + Better BibTeX 插件 | +| Graphviz / Mermaid CLI | SVG/PNG 图表渲染(即使没有它们,源文件也始终会写出) | graphviz.org/download | +| Semantic Scholar API 密钥 | 稳定的 S2 收集(未认证的 S2 限流严重) | `config/sources.yaml` 中的 `semantic_scholar_api_key` | +| `OPENAI_API_KEY` | `extract-evidence-llm`(受保护、强制逐字核对的 LLM 提取;通过 `OPENAI_BASE_URL` 支持任意 OpenAI 兼容端点) | 环境变量 | ## 典型会话 @@ -122,8 +129,12 @@ python scripts/paperops.py manuscript-patch-preview python scripts/paperops.py apply-manuscript-patch --from-preview --dry-run python scripts/paperops.py apply-manuscript-patch --from-preview --apply -# 审计自己的草稿(docx/md/qmd) -python scripts/paperops.py audit-manuscript-draft --input my_thesis_draft.docx +# 审计自己的草稿(docx/md/qmd)— 结构、无来源主张、夸大表述, +# 以及与实验输出文件的数值自动交叉核对 +python scripts/paperops.py audit-manuscript-draft --input my_thesis_draft.docx --experiment-data data/experiment_outputs + +# 人工验证(核心关口)— 记录进验证台账 +python scripts/paperops.py verify-evidence --evidence-id ev_xxxx --by "你的名字" --note "已核对 p.3" --attest # 图表与最终检查 python scripts/paperops.py render-figures @@ -138,16 +149,25 @@ python scripts/paperops.py smoke-test `audit-manuscript-draft` 曾用于一篇真实的 KCI 投稿稿件(413 个段落): 扫描 378 个句子、检查章节结构、标记无来源的强主张与夸大表述,并将草稿中 **全部 178 个数值**与真实实验输出文件对照 — 0 个不一致,2 处舍入差异已解释, -1 个基准率从原始预测日志重新计算确认。审计绝不修改草稿,也绝不标记 -verified;只为作者生成发现报告(MD + CSV)。 +1 个基准率从原始预测日志重新计算确认。(当时这项交叉核对是手工完成的;自 +v0.2.0 起,`--experiment-data` 参数会自动完成:逐个数值给出精确匹配、舍入 +匹配、数据中未找到三种状态。)审计绝不修改草稿,也绝不标记 verified; +只为作者生成发现报告(MD + CSV)。 ## 治理规则 -1. 证据矩阵不得随意修改。 -2. `verified=true` 绝不自动设置 — 不存在通向 verified 状态的自动转移。 -3. 引文/页码匹配是*来源对齐*,不是真实性验证。 -4. 稿件编辑仅通过受保护的 preview/apply 进行,附带备份与事后守卫 + 冒烟测试。 -5. 相关工作的发现仅作为设计模式引用,绝不包装为 PaperOps 自身的性能证据。 +1. 证据矩阵不得随意修改;所有重写都是原子的(临时文件 + rename),并附带 + 带时间戳的备份。 +2. `verified=true` 绝不自动设置。唯一许可的路径是 `verify-evidence --attest`, + 它把由谁、何时、如何验证记录进 `matrices/verification_ledger.csv`;台账中 + 没有对应确认的 verified 行,`guard-no-auto-verified` 会判定失败。 +3. LLM 提取的候选必须与解析后的原文逐字核对 — 转述或凭空捏造的引文会被丢弃 + 并上报 — 之后再走与启发式候选完全相同的人工审阅关口。 +4. 引文/页码匹配是*来源对齐*,不是真实性验证。 +5. 稿件编辑仅通过受保护的 preview/apply 进行,附带备份(存放在受保护目录树 + 之外)、SHA-256 校验,以及事后守卫 + 冒烟测试;批准报告会与当前稿件的 + SHA-256 比对,过期或伪造的报告一律拒绝。 +6. 相关工作的发现仅作为设计模式引用,绝不包装为 PaperOps 自身的性能证据。 ## 本仓库不包含的内容 @@ -158,10 +178,17 @@ verified;只为作者生成发现报告(MD + CSV)。 ## 诚实的局限 -- 证据提取基于关键词/启发式;LLM 辅助提取是计划中的独立受保护步骤。 -- 草稿审计是供人工复核的启发式标记,不是真实性验证。 +- LLM 提取器只保证引文是原文逐字;主张/引文的*选取*质量仍然取决于模型, + 而且每一个候选依然需要人工审阅。 +- 草稿审计是供人工复核的启发式标记,不是真实性验证;数值交叉核对只确认这些 + 数字确实出现在你的数据文件中,并不验证分析本身是否正确。 - 引文/页码对齐不验证主张的真实性 — 这是设计意图。 - 没有真实数据文件,绝不生成定量结果图表。 +- 未认证的 Semantic Scholar 收集会被 S2 限流,该来源实际上需要一个免费的 + API 密钥。 +- 验证台账能遏制意外的和随手的自动化验证,但它不是密码学机制,拦不住一个 + 铁了心的人在自己的本地文件里伪造确认记录。 +- 非英文 README 可能比英文版滞后一个版本;以英文 README 和 CHANGELOG 为准。 ## 许可证 diff --git a/scripts/build_public_release.py b/scripts/build_public_release.py index 3c1478f..1dc1a22 100644 --- a/scripts/build_public_release.py +++ b/scripts/build_public_release.py @@ -34,7 +34,6 @@ # (source, destination-inside-release) WHITELIST_FILES = [ ('scripts/paperops.py', 'scripts/paperops.py'), - ('scripts/paperops_extra.py', 'scripts/paperops_extra.py'), ('scripts/paperops_figures.py', 'scripts/paperops_figures.py'), ('scripts/paperops_draft_audit.py', 'scripts/paperops_draft_audit.py'), ('scripts/build_public_release.py', 'scripts/build_public_release.py'),