v0.2.0: full overhaul - audited defects fixed, guarded LLM extraction, verification ledger, tests + CI - #5
Conversation
…action, verification ledger, tests + CI Driven by an independent live audit (fresh-clone runs on Python 3.10-3.13 + line-by-line review). Highlights: P0 fixes (all live-reproduced before fixing): - CLI was a SyntaxError on Python 3.10/3.11 (f-string backslash, PEP 701) - review-evidence-candidates wiped other papers' human review decisions (shared queue overwrite -> merge) - Excel round-trip broke validators (BOM/cp949/stray columns) -> utf-8-sig reads, BOM writes, atomic rewrites with row projection - collect gathered off-topic papers: arXiv boolean queries + relevance sort, OpenAlex/Crossref mailto + backoff + Retry-After, Crossref type filters, cross-source DOI/arXiv/title dedupe, per-source circuit breaker - migrate-evidence could flip '1'/'yes' to verified=true -> only literal 'true' survives; demotions reported Governance upgrades: - verify-evidence + matrices/verification_ledger.csv: the only sanctioned writer of verified=true (human attestation required); guards now PASS attested rows and FAIL unattested ones (previously the guard failed forever after any legitimate human verification) - approval-gate anti-forgery: apply reports cross-checked against current manuscript SHA-256 - manuscript backups moved outside the guarded tree; citekey scans skip backups; sync-zotero respects --dry-run and propagates citekey renames into the Evidence Matrix; legacy extract-evidence/audit fixed or blocked New capabilities: - extract-evidence-llm: LLM-proposed candidates with mandatory verbatim quote verification against GROBID text; same human gates as heuristics - audit-manuscript-draft --experiment-data: automatic numeric cross-check (exact/rounded/not-found) against experiment output files — makes the README's flagship claim true in code - draft audit: docx footnotes, cp949 fallback, Korean/English detectors, heading-anchored structure checks; figures: line-anchored heading match, honest reports, YAML specs, graceful renderer degradation - make-page-metadata-preview / make-outline-insertion-template producers; promote-evidence --ready-only - release hardening: named secret-scan ruleset, config sanitization, quarantine-on-failure, whitelist gaps closed Hygiene: personal research data moved from code to config YAMLs; sources.yaml actually wired; paperops_extra.py removed (S2 in main collect); 4 unused deps dropped; 179 tests + GitHub Actions matrix (3.10-3.13); README en/ko truth pass; CHANGELOG added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ULLxKwEG7wBaj6SDFeBmsA
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1a88983098
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| duplicate_ids = sorted(k for k, v in count_values([r.get('candidate_id') for r in queue_rows]).items() if k and v > 1) | ||
| # MERGE, never overwrite: rows belonging to OTHER papers stay in the queue untouched. | ||
| # Regenerating paper B's queue must not delete paper A's pending human decisions. | ||
| other_rows = [r for r in existing_rows if str(r.get('paper_id') or '') != str(args.paper_id or '')] |
There was a problem hiding this comment.
When review-evidence-candidates is run with --min-confidence, --claim-type, or --use-in-section, this predicate discards every existing row for the current paper, including rows excluded only by the filter. Any human decisions on those rows are consequently erased and revert to pending when the full queue is regenerated. Merge by candidate ID and retain same-paper rows that were not selected by the current filtered invocation.
Useful? React with 👍 / 👎.
| if ready_only: | ||
| warnings.append(f'--ready-only: proceeding with {validation["ready_count"]} ready rows; {validation["blocked_count"]} blocked rows reported but not applied') |
There was a problem hiding this comment.
--ready-only still cannot process the normal mixed preview produced by promotion-plan: validate_promotion_preview() has already added global errors for each row whose promotion_ready is false or whose promotion_blockers is nonempty. This branch merely changes the blocked-count error into a warning, so the existing errors make the subsequent if not errors false and no ready rows are written. In ready-only mode, validation failures attributable solely to blocked rows must remain row-level blockers rather than global apply errors.
Useful? React with 👍 / 👎.
| entry is not None | ||
| and str(entry.get('action') or '').strip() == 'verify' | ||
| and str(entry.get('verified_by') or '').strip() | ||
| and str(entry.get('verified_by') or '').strip() == str(r.get('verified_by') or '').strip() | ||
| and str(entry.get('verified_at') or '').strip() == str(r.get('verified_at') or '').strip() |
There was a problem hiding this comment.
After a row is verified, changing its quote/exact_quote while leaving verified_by and verified_at unchanged still makes this guard classify it as attested. verify-evidence records quote_sha256 in the ledger specifically for this purpose, but the guard never compares it with the current row, so CI's guard-no-auto-verified can pass evidence that is no longer what the human checked. Include the current quote hash in the attestation match, and similarly invalidate verification when other attested source metadata changes.
Useful? React with 👍 / 👎.
| for sent in split_sentences(para['text']): | ||
| n_sentences += 1 | ||
| mentions = bool(SOURCE_MENTION.search(sent)) | ||
| strong = STRONG_CLAIM.search(sent) | ||
| over = OVERCLAIM.search(sent) | ||
| numeric = NUMERIC_RESULT.search(sent) | ||
| hypo = bool(HYPOTHETICAL.search(sent)) | ||
| has_source = sentence_has_source(sent, det, | ||
| para['has_footnote_ref']) |
There was a problem hiding this comment.
For a multi-sentence DOCX paragraph containing one footnote reference, has_footnote_ref is a paragraph-wide boolean and is passed to every sentence. A citation attached to one sentence therefore suppresses strong_claim_no_source for unrelated strong claims elsewhere in that paragraph. Preserve the reference position while flattening the DOCX runs, or split/map references before invoking sentence_has_source so only the cited sentence is treated as sourced.
Useful? React with 👍 / 👎.
README.md (en) and README.ko.md were updated in the previous commit; the other six shipped language versions still described v0.1.0. All eight are now consistent, so a non-English reader is not handed stale instructions. Applied to de/es/fr/ja/zh/ar, translated in each file's existing register: - 45+ -> 50+ commands; intro now covers verbatim-checked LLM extraction and numeric cross-checking against experiment outputs - verified=true described as reachable only via verify-evidence with a human attestation recorded in the verification ledger, cross-checked by guards - lifecycle rows 1/5/13/14 rewritten (4 sources + boolean queries + polite pool + 429 backoff + dedup; extract-evidence-llm; --experiment-data; human-gate verification) - comparison table, quickstart (Python 3.10+, openalex_mailto polite-pool note), add-ons table (+Semantic Scholar key, +OPENAI_API_KEY), typical session (--experiment-data, verify-evidence --attest) - KCI anecdote now states the numeric cross-check was manual at the time and automatic since v0.2.0 — it is not retroactively claimed as automated - 6 governance rules and the new honest-limitations list Also fixes two release-build regressions from the previous commit: - drop deleted scripts/paperops_extra.py from the export whitelist - reword a CHANGELOG line so the hardened secret scanner no longer flags an illustrative email address Verified: all 8 READMEs pass the same four content checks, all four tables in every file match the English column shape, language selectors and figure paths byte-identical, release build clean (sanitize_issues=0, 8 READMEs shipped), 179 tests still pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ULLxKwEG7wBaj6SDFeBmsA
2 participants
v0.2.0: Full overhaul — every audited defect fixed, guarded LLM extraction, verification ledger, 179 tests + CI
독립 감사(신규 클론을 Python 3.10~3.13에서 실측 + 전체 코드 리뷰)에서 발견된 결함 전체를 수정하고, "유명 오픈소스 논문 자동화 도구들의 장점을 취합한 마스터 파이프라인"이라는 원래 취지에 맞는 업그레이드를 더한 릴리스입니다. 모든 결함은 수정 전 재현했고, 수정 후 실측으로 재검증했습니다.
P0 — 사용 자체를 막던 결함 (전부 실측 재현 후 수정)
requires-python>=3.10선언과 모순review-evidence-candidates가 다른 논문의 사람 리뷰 결정을 삭제 (공유 큐 덮어쓰기)utf-8-sig+잉여열 흡수, 쓰기 BOM 포함(한글 정상 표시), 매트릭스 재작성 전부 원자적(temp+rename)collect가 주제 무관 논문 수집 (arXiv 쿼리 오구성→오늘자 무관 논문, OpenAlex 전 요청 429, Crossref "Decision letter" 수집)migrate-evidence가'1'/'yes'를verified=true로 승격 — 핵심 불변식의 유일한 구멍true만 보존, truthy 변형은 false로 강등+보고서 명시거버넌스 — 시스템의 존재 이유를 완성
verify-evidence+ 검증 원장 (matrices/verification_ledger.csv):verified=true로 가는 유일한 공인 경로.--by 이름 --attest확약 필수, 모든 행위(verify/revoke)가 원장에 기록.guard-no-auto-verified는 이제 원장에 확약이 있는 verified 행은 통과시키고(기존: 정당한 인간 검증 후 가드가 영구 실패), 원장에 없는 verified 행은 실패시킴 — 자동화든 손 편집이든. 공격 시뮬레이션으로 실측 검증.backups/manuscript/)으로 이동 — 가드 자가 발동 및 citekey 이중 집계 해소sync-zotero:--dry-run --apply조합 차단, citekey 리네임을 Evidence Matrix까지 전파(백업 동반)extract-evidence(스키마 파괴 append) 차단, 레거시audit의 이메일 오탐(@gmail)·substring 오판정 수정신규 기능 (마스터 파이프라인 업그레이드)
extract-evidence-llm— LLM 보조 근거 추출(가드됨). 모델은 후보를 제안만 하고, 모든exact_quote는 GROBID 파싱 원문과 축자(verbatim) 대조 — 패러프레이즈·창작 인용문은 폐기·보고. 생존 후보는 휴리스틱과 동일한 사람 검토→승격→가드 apply를 거침. OpenAI 호환 엔드포인트 지원. (config/prompts/evidence_extractor.md가 죽은 설정에서 실설정으로)audit-manuscript-draft --experiment-data DIR: 초안의 모든 수치를 실험 산출 파일과 자동 대조(정확 일치/반올림 일치/데이터에 없음). README가 약속했지만 코드에 없던 대표 기능의 실구현.(김철수, 2020)·[1]인식, 영어 강주장/과장 탐지, 헤딩 앵커 장 구조 검사,config/draft_audit.yaml## X승인이### X Details아래에 삽입되던 버그), 모호/부재 헤딩은 차단, 멱등 apply, 정직한 보고서+preview SHA,config/figures.yaml, 렌더러 부재 시 우아한 강등(--strict로 엄격 모드)make-page-metadata-preview/make-outline-insertion-template— 생성자가 없어 dead end였던 두 가드 입력 파일의 생성기promote-evidence --ready-only, PDF 다운로드 content-type 검사(HTML 유료벽 저장 차단),backup에 papers.sqlite 포함, 원클릭 파이프라인 ASCII화+우아한 강등재사용성·품질
review_overrides.yaml, 코퍼스 키워드→qa_profile.yaml, figure 스펙→figures.yaml(저자 동작은 그대로 보존)collect에 통합(API 키 헤더), 미등록·크래시하던paperops_extra.py제거검증 요약 (전부 이 브랜치에서 실측)
verify-evidence→ 가드 통과 → 무단 verified 편집 → 가드 실패 확인 → 원복 → 가드 통과run_pipeline.py원클릭 완주(exit 0),build_public_release.py클린(sanitize_issues=0)Merge 후 권장: GitHub Actions 첫 실행 확인, 로컬에서
python scripts/paperops.py doctor로 mailto/API 키 설정.🤖 Generated with Claude Code
https://claude.ai/code/session_01ULLxKwEG7wBaj6SDFeBmsA