ScopeProof

See which acceptance criteria have credible PR evidence—and which still need review.

ScopeProof is a reviewer-controlled acceptance-coverage assistant for public GitHub pull requests. It maps each confirmed criterion to inspectable implementation or test candidates, makes missing evidence visible, records attributable human decisions, and exports a reproducible review. The primary workflow has a product target of reaching an inspectable coverage report in under five minutes. That target has not yet been independently observed and is separate from the Stage 1 under-ten-minute completed-review gate.

ScopeProof is an evidence assistant. It does not replace QA, engineering review, runtime testing, or human acceptance.

The R-001 Microsoft public-PR research record shows a compact, hash-bound engineering rerun: observed CI and static candidates remain distinct from runtime verification and human acceptance. It is public engineering research only and does not advance Stage 1.

The R-002 SWE-bench Verified static engineering benchmark adds a frozen 20-case, 12-repository deterministic baseline for candidate matching and immutable references. It executes no target code and contributes zero Stage 1 validation credit.

Controlled demo screenshot—not a customer case. It shows deterministic candidate evidence and missing-evidence states; it is not runtime verification or proof of correctness. See the deliberately constructed demo for the expected findings.

Why this exists

AI coding agents can produce pull requests quickly, but a green CI check does not establish that every ticket promise was implemented. Product reviewers still need to answer questions such as:

• Did export include every active filter?
• Is the failure state visible to the user?
• Was the required analytics event added?
• Does a test exercise the requested behavior, or does a similarly named test merely exist?
• Did the pull request expand scope beyond the approved requirement?

ScopeProof turns that review into a requirement-to-evidence matrix. It shows why each candidate matched and what remains unverified.

MVP boundaries

• The repository is published under an evaluation-only use policy and does not grant an open-source license.
• No paid LLM API and no model-generated verdicts.
• Supports public repositories only.
• Anonymous GitHub access works without a token.
• An optional GitHub token can increase free rate limits; it remains in Streamlit session memory and is never exported or saved.
• Users author and confirm criteria. ScopeProof does not invent product requirements.
• Pull-request code is never executed.
• Static candidates cannot be presented as runtime verification.
• General bug review, security scanning, automatic fixes, private repositories, Jira, billing, and team accounts are outside this release.

Evidence and review language

Level	Meaning in this MVP
E0	No candidate evidence found
E1	Candidate implementation or contract evidence
E2	Candidate test evidence that still requires reviewer confirmation
E3	Runtime verification recorded manually from an external check
E4	Explicit human acceptance

The workbench describes criterion evidence as Strong candidate, Weak candidate, No candidate, Analysis incomplete, Reviewer verified, or Rejected. Candidate strength is not correctness. Implementation, test, runtime, documentation, and contract evidence remain separate types.

The release gate uses explicit precedence:

1. Action required for failed checks, change-required decisions, or unresolved must-have gaps.
2. Review incomplete for unconfirmed criteria, partial ingestion, unavailable checks, ambiguous evidence, or unresolved decisions.
3. Accepted with exceptions for explicitly accepted exceptions.
4. Review complete only after complete ingestion, passing observed CI, current decisions for every criterion, and final human acceptance.

The earlier post-merge v0.2.3 audit found two core integrity defects in the intended flow. PR #177 repaired those findings, PR #179 merged the completed workbench UX change, and PR #180 merged the exact-head runtime-evidence hardening as product commit 2a320df966eff30c05a2b1dce607a247201fa165.

The merged v0.2.3 exact-head runtime-evidence hardening binds each effective E3/E4 decision to exactly one runtime record for the same repository, pull request, reviewed head, criterion, reviewer, and evidence level. Version 1 and 2 local records migrate to version 3 without inventing links: legacy-unlinked manual decisions remain visible but become Needs Review until verification is re-recorded at the active head. It also exposes immutable provenance in the workbench and every export, shows skipped-check limits outside collapsed details, and provides a safe draft-clear recovery for revised bundle-less reviews. Independently verified PR head ed9f9c0cf6b7cf7cc25403d6138e7a8391f55e0f has the same product tree, add81a2d0ba7e64f8e4318a1959bbe7e6e4acfc8, as the merge commit. Stage 0's engineering foundation is restored. PR #183 merged the later integrity and reviewer-loop source work as cd362a85a558645a0f56d6540f6bf035e5821809; exact-main CI run 30847416893, CodeQL run 30847415556, and Pages run 30847417705 succeeded. Those runs remain historical source-integration evidence, not correctness evidence, customer validation, or Stage 1 credit.

ScopeProof v0.2.3 is published. PR #184 merged the release integration at 448c42758ea139bf9203cbf1bb04b02b02ae412c; the peeled v0.2.3 tag resolves to that release baseline. At the 2026-08-08 branch-start snapshot, origin/main also resolved to that commit. Exact-main CI run 30854382641, CodeQL run 30854382413, and Pages run 30854382659 succeeded. The release and those checks are engineering evidence only and do not advance Stage 1, which remains at zero. The exact-head verification audit records the product-tree engineering evidence and remaining gaps.

GitHub exposes visible check runs but does not reliably expose every repository's required-check policy to anonymous clients. ScopeProof therefore labels this value Observed CI state and counts only explicit success as passing.

Quickstart

Python 3.11 or newer is required.

The commands below are the v0.2.3 post-publication install path and do not require cloning the repository. First check the GitHub Releases page: https://github.com/YuzeJ21/Scope-Proof/releases.

Use the v0.2.3 asset URLs only when the GitHub Releases page shows v0.2.3 with scopeproof-0.2.3-py3-none-any.whl and SHA256SUMS.txt; otherwise, do not use an unpublished branch or candidate. Release publication and its exact-tree checks are engineering evidence only; they do not establish correctness, customer validation, or Stage 1 progress.

See the v0.2.3 status and next-stage audit for the implemented feature ledger, current gaps, and evidence-gated Stage 1–4 roadmap.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install \
  https://github.com/YuzeJ21/Scope-Proof/releases/download/v0.2.3/scopeproof-0.2.3-py3-none-any.whl
scopeproof benchmark
scopeproof-web --host 127.0.0.1 --port 8501

To verify the release bytes before installation, download the wheel and its checksum:

curl -LO https://github.com/YuzeJ21/Scope-Proof/releases/download/v0.2.3/scopeproof-0.2.3-py3-none-any.whl
curl -LO https://github.com/YuzeJ21/Scope-Proof/releases/download/v0.2.3/SHA256SUMS.txt

Select the wheel entry from the published manifest, use the command for your platform, then install the verified local file:

# macOS
grep " scopeproof-0.2.3-py3-none-any.whl$" SHA256SUMS.txt | shasum -a 256 -c -

# Linux
grep " scopeproof-0.2.3-py3-none-any.whl$" SHA256SUMS.txt | sha256sum -c -

python -m pip install ./scopeproof-0.2.3-py3-none-any.whl

A matching checksum verifies the downloaded bytes against the digest published with this release. It does not provide code-signing or product-correctness assurance.

The offline benchmark verifies that the installed package and bundled regression corpus execute. It is not runtime evidence for any pull request. scopeproof-web starts the packaged local workbench; stop it with Ctrl+C. Continue with the public-PR CLI workflow below to review reviewer-confirmed criteria against a real public PR.

Public PR CLI workflow

The commands below describe the 0.2.3 source workflow. From a current source checkout, the CLI provides the same read-only public-PR ingestion and deterministic core without starting Streamlit. Live ingestion labels a review public only when GitHub returns matching, unambiguous repository metadata (private: false and visibility: public). A token-accessible private repository and missing, malformed, or contradictory visibility metadata fail closed before review data is saved. Historical records without that verified fact remain readable but cannot qualify for Stage 1. First create requirements.txt with one atomic criterion per line. A human requirements owner or authorized role must inspect and approve the exact file before creating the confirmation record. --confirmed-by is that human's explicit attestation; ScopeProof does not verify their identity, authority, or the criteria's correctness. Every retained line is therefore reviewer-confirmed criteria, not criteria inferred or approved by ScopeProof.

uv run scopeproof prepare-requirements-confirmation \
  --requirements requirements.txt \
  --source-uri https://github.com/OWNER/REPOSITORY/blob/FULL_SHA/path/to/requirements.txt \
  --source-revision FULL_SHA \
  --confirmed-by "Requirements owner or authorized role" \
  --output requirements-confirmation.json

uv run scopeproof validate-requirements-confirmation \
  --requirements requirements.txt \
  --confirmation requirements-confirmation.json

uv run scopeproof review --pr https://github.com/OWNER/REPOSITORY/pull/123 \
  --requirements requirements.txt \
  --confirmation requirements-confirmation.json \
  --storage-dir .scopeproof/reviews \
  --report scopeproof-review.md

The preparation command performs no network request, computes both hashes from the exact UTF-8 bytes and ordered normalized criteria, and refuses to overwrite an existing confirmation file. The optional report path may end in .md, .json, .csv, or .html; ScopeProof validates the selected export and refuses to overwrite an existing file.

The command prints JSON containing the local review_id, record path, head SHA, and provisional gate verdict, plus the requested report path. Use the review identifier later to repeat the export or choose another format:

scopeproof export REVIEW_ID \
  --storage-dir .scopeproof/reviews \
  --format markdown

Available repeat-export formats are json, markdown, csv, and html.

Continue the same validated local review through the CLI with explicit human decisions. Store reviewer notes in UTF-8 files when they are needed:

scopeproof resolve REVIEW_ID \
  --criterion-id AC-01 \
  --decision accepted \
  --reviewer "Reviewer name or role" \
  --comment-file reviewer-note.txt \
  --storage-dir .scopeproof/reviews

scopeproof verify-runtime REVIEW_ID \
  --criterion-id AC-02 \
  --level E3 \
  --reviewer "Runtime reviewer" \
  --artifact-reference https://example.test/artifacts/run-123 \
  --scenario "Exercise the confirmed acceptance scenario" \
  --environment "Owner-operated staging environment" \
  --result "Observed result supplied by the reviewer" \
  --comment-file runtime-note.txt \
  --storage-dir .scopeproof/reviews

scopeproof final-acceptance REVIEW_ID \
  --accept \
  --reviewer "Final reviewer" \
  --comment-file final-note.txt \
  --storage-dir .scopeproof/reviews

scopeproof compare PREVIOUS_REVIEW_ID CURRENT_REVIEW_ID \
  --format markdown \
  --output comparison.md \
  --storage-dir .scopeproof/reviews

resolve records one human criterion decision and never executes PR code. Static candidates never become runtime evidence through resolve; accepting below a criterion's required evidence level requires a non-empty reviewer note. verify-runtime is the only CLI command above that atomically links a human-supplied E3/E4 runtime record to its manual-verification decision. It does not run or independently verify the cited artifact. Final acceptance remains fail-closed until the deterministic prerequisites are satisfied; use --revoke to append a revocation. compare validates both saved reviews, reports candidate changes without carrying decisions forward, and refuses to overwrite an existing output file.

CSV exports neutralize leading spreadsheet-formula characters in scalar text cells. Fields that can contain multiple values (ingestion_warnings, skipped_files, evidence_links, missing_evidence, runtime_artifacts, and runtime_result) are JSON arrays inside their CSV cells so delimiters in repository or reviewer text do not destroy provenance.

Anonymous public-repository access is the default. --token is optional and can increase GitHub's free rate limit, but it is not required or persisted. The CLI never comments on the pull request, executes its code, or converts static candidates into runtime verification.

Contributor setup

For a fully pinned setup, use the reproducible development environment. The standard editable setup remains available below.

Read the evaluation-only use policy before cloning or proposing a contribution. Public visibility does not grant permission beyond evaluation and review.

Clone the repository to run the Streamlit workbench or contribute changes:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dev]'
streamlit run apps/web/app.py

Open the displayed local URL, then choose either path:

1. Select Load deliberately constructed demo for a complete offline walkthrough.
2. Enter a public GitHub PR URL, fetch it, paste one criterion per line, and confirm the criteria. Session-only tokens and bounded unchanged candidate paths are under Advanced source options.

The five review steps are:

1. Start Review.
2. Confirm Criteria.
3. Evidence Matrix.
4. Criterion Detail, external verification, and human resolution.
5. Summary and Markdown, JSON, or CSV export.

Durable local review workflow

The workbench keeps criteria revisions and append-only resolution history. Adding, removing, splitting, reordering, or editing criteria invalidates the previous analysis and requires explicit reconfirmation before another review run. Human decisions and final acceptance are recorded as history rather than silently replacing earlier decisions.

Local review storage

The workbench stores versioned JSON records under ~/.scopeproof/reviews. The reopen panel lists safe local record IDs in deterministic order, while an empty store retains manual ID recovery. The app validates the selected record when it is opened and refuses a configured review path that is a symbolic link or another existing non-directory. This app-owned local directory prevents a browser input from selecting arbitrary file paths. Records preserve the review SHAs, criteria revisions, evidence, findings, resolution history, and gate decision. They never contain the optional GitHub token. A reopened review prepares its public PR URL and bounded unchanged-candidate paths for a one-click current-head check rather than silently reusing old evidence. Records also preserve whether public repository visibility was verified; legacy records without that fact remain explicitly unverified. After a new analysis, the workbench compares previous and current heads, candidates, finding states, reviewer decisions, and review status without mutating either bundle. Candidate evidence is classified as Unchanged, Relocated, Modified, Added, or Removed. Changed candidates show both the previous and current immutable location and excerpt so the reviewer can inspect what moved or changed before recording a new decision. Exact unchanged candidates remain inspectable in a collapsed section, and the validated comparison can be downloaded as Markdown or JSON. This comparison does not prove criterion satisfaction or carry a prior human decision forward.

From the CLI, run scopeproof list to return the safe local review IDs in the default .scopeproof/reviews directory; add --storage-dir PATH only when earlier CLI commands used that same alternative directory. The command lists identifiers in deterministic order and does not parse review contents. This local inventory is not review, test, runtime, or correctness evidence.

To delete one saved review in the workbench, select its listed ID, then check Permanently delete the selected local review. From the CLI, run scopeproof delete REVIEW_ID; add --storage-dir PATH only for a review saved outside the default CLI directory. Deletion removes only that app-owned JSON record. Exported reports remain user-owned and are not removed, and deletion is not secure erasure of storage media or backups. An open deleted review remains available as unsaved session work until it is replaced or the session ends.

Deliberately constructed demo

The bundled CSV export case is a deliberately constructed demo, not a real incident. Its PR-shaped fixture implements CSV export, one active filter, and a happy-path test. It intentionally omits another filter, the error state, and the research_exported event.

Expected output:

Criterion	Expected finding
AC-01 Export CSV	Strong candidate
AC-02 Respect all active filters	Weak candidate
AC-03 Show export failure	No candidate
AC-04 Record research_exported	No candidate

The review status is Action required because must-have gaps remain.

Verification

Run lint and all offline tests:

python -m ruff check .
python -m pytest -q

Run the labeled regression benchmark:

uv run scopeproof benchmark
uv run scopeproof comparison-benchmark

The first command executes 12 executable benchmark cases for acceptance coverage, rather than treating a static category list as coverage. The equivalent module entry point is python -m scopeproof_core.evals.runner. It reports executed case and criterion counts, False Ready, False Blocker, case-level mismatches, immutable evidence-link errors, and unexecuted required categories. It exits nonzero when a known must-have False Ready, label mismatch, evidence-link error, or unexecuted category is present.

The second command executes two paired previous/current review cases and checks deterministic re-review classification, including exact Unchanged coverage and conservative handling of ambiguous duplicate candidates. This corpus is deliberately constructed engineering evidence. It does not advance Stage 1. It does not prove correctness. It does not constitute customer validation. It does not show external use. Both benchmark commands execute local JSON inputs only; they do not run fixture repository code.

Run the opt-in live public GitHub smoke test:

RUN_LIVE_GITHUB_TESTS=1 python -m pytest tests/github/test_live_public_pr.py -q

Check the running Streamlit server:

curl --fail http://127.0.0.1:8501/_stcore/health

Architecture

Public GitHub PR
      ↓
Read-only ingestion
      ↓
User-confirmed criteria
      ↓
Deterministic candidate retrieval
      ↓
Provisional findings + human resolution
      ↓
Deterministic gate
      ↓
Markdown / JSON / CSV / HTML

scopeproof_core contains Pydantic contracts, ingestion, retrieval, verification, gates, reporting, fixtures, and evaluation. It has no Streamlit dependency. apps/web/app.py is a thin local interface over those core services.

Every evidence item contains a file, line, immutable head SHA, GitHub permalink, excerpt, matching rule, relevance reason, deterministic score, and limitations. Deleted lines cannot become current implementation evidence. Partial ingestion cannot produce Ready.

GitHub file and commit ingestion follows pagination and has explicit file, patch, and total-diff limits. If GitHub does not provide inspectable patch text for a changed path, ScopeProof excludes that path, names it as skipped, and marks ingestion partial; it never treats missing diff content as a complete empty file. ScopeProof can also inspect a bounded unchanged candidate file when a caller explicitly justifies it. This evidence is labeled unchanged_candidate, anchored to the head SHA, and never means that the repository was scanned broadly.

Repository layout

apps/web/                 Streamlit review workbench
scopeproof_core/github/   Public GitHub ingestion
scopeproof_core/criteria/ Manual criterion preparation
scopeproof_core/retrieval/Deterministic evidence candidates
scopeproof_core/verification/Provisional findings
scopeproof_core/gates/    Release truth table
scopeproof_core/reporting/Markdown, JSON, CSV, and HTML exports
scopeproof_core/evals/    Regression runner
evals/                    Controlled fixtures and labels
tests/                    Unit, regression, AppTest, and live smoke tests

Privacy and trust

The application does not persist credentials or execute repository code. Review exports contain the repository, PR number, head SHA, ruleset version, criteria, evidence, findings, resolutions, and gate reasons. They do not contain the optional GitHub token.

Large or truncated diffs are labeled partial and force human review. Missing GitHub checks are represented as unavailable, never passing. A parser or retrieval failure must fail safely rather than produce Ready.

The privacy-readiness design documents current local-only retention, deletion responsibility, fork protection, and the future read-only private-repository boundary. It does not claim private support exists.

Product status

This is a public-repository alpha for validating the requirement-to-evidence workflow. The next product decision must be based on repeat use with real pull requests and confirmed gaps found before merge—not release activity or vanity metrics.

The public roadmap defines the evidence-gated path from engineering-complete public alpha to limited beta. The changelog summarizes the active development line and links to authoritative published release notes. Neither document substitutes for genuine public-PR use or human acceptance.

Standard reviews create no research record. To volunteer genuine feedback, submit the inbound public-alpha case form before starting locally:

https://github.com/YuzeJ21/Scope-Proof/issues/new?template=public-alpha-case.yml

Then explicitly enable the Alpha feedback session and follow the public-alpha participant quickstart. Qualification and consent stay local and off by default; the constructed demo never counts as participant validation.

ScopeProof is an acceptance-coverage assistant, not an AI code reviewer. The optional free design-partner review is public-repository-only. No paid product or billing is active, and a pricing question is optional research after product use—not a purchase requirement. The 30-day commercial-validation guide defines the inbound workflow, evidence gates, stop rules, and capabilities that remain deferred. The market-positioning research compares adjacent traceability, test-management, and code-review workflows and records only hypotheses that still require genuine participant evidence.

The launch evidence matrix and archived LinkedIn preparation in the draft and playbook preserve prior launch research and its constructed-demo boundaries. The current policy is passive GitHub intake; those LinkedIn materials are not authorized for publication without a separate owner decision.

GitHub Action advanced preview

The default product is the local workbench. The checked-in GitHub Actions guide covers an advanced, non-blocking preview for repositories that make a separate operator decision to adopt it. It requires checked-in, hash-confirmed requirements and never checks out or executes pull-request head code. It is not part of first use and does not replace the reviewer-controlled workflow.

Explicit local Definition-of-Done packs

scopeproof_core.rule_packs offers opt-in local prompts for error, loading, empty, analytics, authorization, API documentation, and migration coverage. They are never inferred or added to a review automatically. Each is labelled Implicit local rule and has an implicit_rule_pack source so it cannot masquerade as a user-confirmed source requirement; a reviewer must explicitly include and confirm it before analysis.

Evidence-quality metrics

scopeproof benchmark emits evidence_quality_metrics alongside its executed fixture results: immutable evidence-link precision, incorrect-line-citation rate, criterion agreement, and False Ready/False Blocker totals. These measure only the checked-in, executed labels; they do not prove runtime correctness or market validation. Human override, accepted-exception, and unresolved-ambiguity rates are intentionally null unless calculated from selected persisted review histories.