Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,10 @@ reviewer-confirmed criteria against a real public PR.

The commands below describe the `0.2.3` source workflow. From a current source checkout, the CLI
provides the same read-only public-PR ingestion and deterministic core without starting Streamlit.
Live ingestion labels a review public only when GitHub returns matching, unambiguous repository
metadata (`private: false` and `visibility: public`). A token-accessible private repository and
missing, malformed, or contradictory visibility metadata fail closed before review data is saved.
Historical records without that verified fact remain readable but cannot qualify for Stage 1.
First create `requirements.txt` with
one atomic criterion per line. A human requirements owner or authorized role must inspect and
approve the exact file before creating the confirmation record. `--confirmed-by` is that human's
Expand Down Expand Up @@ -314,7 +318,9 @@ is a symbolic link or another existing non-directory. This app-owned local direc
browser input from selecting arbitrary file paths. Records preserve the review SHAs, criteria
revisions, evidence, findings, resolution history, and gate decision. They never contain the
optional GitHub token. A reopened review prepares its public PR URL and bounded unchanged-candidate
paths for a one-click current-head check rather than silently reusing old evidence. After a new
paths for a one-click current-head check rather than silently reusing old evidence. Records also
preserve whether public repository visibility was verified; legacy records without that fact
remain explicitly unverified. After a new
analysis, the workbench compares previous and current heads, candidates,
finding states, reviewer decisions, and review status without mutating either bundle. Candidate
evidence is classified as **Unchanged**, **Relocated**, **Modified**, **Added**, or **Removed**.
Expand Down
2 changes: 2 additions & 0 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ advances Stage 1, which remains at zero.

- [x] Acceptance-coverage vocabulary separates candidate strength from reviewer decisions.
- [x] Standard flow is public PR → confirmed criteria → coverage → decisions → export.
- [x] Live reviews and genuine-alpha records require an explicit verified-public repository fact;
private, ambiguous, and legacy-unverified sources fail closed for Stage 1.
- [x] Enforce at the core boundary that manual verification records runtime
evidence and its decision atomically, and reject unpaired reconstructed
bundles or states at every trusted boundary.
Expand Down
36 changes: 30 additions & 6 deletions apps/web/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
AlphaFrictionStage,
AlphaOutcome,
AlphaQualification,
AlphaQualificationInput,
ParticipantRole,
)
from scopeproof_core.alpha.service import ensure_alpha_case, record_alpha_outcome
Expand Down Expand Up @@ -91,6 +92,7 @@
ReviewInputOrigin,
ReviewState,
RuntimeEvidence,
require_verified_public_origin,
)
from scopeproof_core.storage.json_store import (
JsonReviewStore,
Expand Down Expand Up @@ -675,8 +677,15 @@ def _hydrate_reopened_review(state: ReviewState) -> None:
def _analyze() -> ReviewBundle:
snapshot = st.session_state["snapshot"]
criteria = st.session_state["criteria"]
input_origin = (
ReviewInputOrigin.CONSTRUCTED_DEMO
if st.session_state["criteria_source_mode"] == "demo"
else ReviewInputOrigin.LIVE_PUBLIC_GITHUB
)
require_verified_public_origin(snapshot.repository_visibility, input_origin)
review = Review(
repository=snapshot.repository,
repository_visibility=snapshot.repository_visibility,
pr_number=snapshot.pr_number,
base_sha=snapshot.base_sha,
head_sha=snapshot.head_sha,
Expand All @@ -687,11 +696,7 @@ def _analyze() -> ReviewBundle:
ingestion_state=snapshot.ingestion_state,
ingestion_warnings=snapshot.warnings,
skipped_files=snapshot.skipped_files,
input_origin=(
ReviewInputOrigin.CONSTRUCTED_DEMO
if st.session_state["criteria_source_mode"] == "demo"
else ReviewInputOrigin.LIVE_PUBLIC_GITHUB
),
input_origin=input_origin,
)
retrieval_result = retrieve_evidence_with_diagnostics(
snapshot, criteria, unchanged_files=st.session_state["candidate_files"]
Expand Down Expand Up @@ -1201,6 +1206,7 @@ def _render_ingestion_limitations(source: PullRequestSnapshot | Review | None) -
key="no_confidential_information",
)
alpha_qualification_ready = True
alpha_qualification_input: AlphaQualificationInput | None = None
alpha_qualification: AlphaQualification | None = None
if alpha_feedback_mode:
alpha_qualification_ready = False
Expand All @@ -1211,7 +1217,7 @@ def _render_ingestion_limitations(source: PullRequestSnapshot | Review | None) -
and no_confidential_information
):
try:
alpha_qualification = AlphaQualification(
alpha_qualification_input = AlphaQualificationInput(
public_pr_url=pr_url,
requirements_source_url=requirements_source_url,
participant_role=ParticipantRole(participant_role),
Expand All @@ -1222,6 +1228,23 @@ def _render_ingestion_limitations(source: PullRequestSnapshot | Review | None) -
st.warning("Use a public HTTPS requirements source and a canonical public PR URL.")
else:
alpha_qualification_ready = True
loaded_for_alpha = st.session_state["snapshot"]
if alpha_qualification_input is not None and loaded_for_alpha is not None:
try:
alpha_owner, alpha_repository, alpha_pr_number = parse_pr_url(
alpha_qualification_input.public_pr_url
)
if (
f"{alpha_owner}/{alpha_repository}" != loaded_for_alpha.repository
or alpha_pr_number != loaded_for_alpha.pr_number
):
raise ValueError("alpha qualification must match the loaded public PR")
alpha_qualification = AlphaQualification(
**alpha_qualification_input.model_dump(mode="python"),
repository_visibility=loaded_for_alpha.repository_visibility,
)
except ValueError:
alpha_qualification = None
else:
st.caption("Standard review mode does not create participant research records.")

Expand Down Expand Up @@ -1756,6 +1779,7 @@ def _render_ingestion_limitations(source: PullRequestSnapshot | Review | None) -
confirmed_criteria=[item.text for item in edited_criteria],
confirmed_criterion_snapshot=edited_criteria,
criteria_source_provenance=provenance,
repository_visibility=alpha_qualification.repository_visibility,
)
except ValueError:
st.error(
Expand Down
6 changes: 4 additions & 2 deletions docs/alpha/outcome-form.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,10 @@ scopeproof alpha outcome CASE_ID \

ScopeProof loads and revalidates that saved review, then derives its exact head SHA. The outcome
is rejected unless the review came from live public GitHub ingestion and matches the qualified
public PR, ordered criteria snapshot, and source provenance. Fixture, demo, research, and legacy
unknown review origins cannot complete genuine alpha evidence.
public PR, verified-public repository fact, ordered criteria snapshot, and source provenance.
The alpha case itself must preserve the same verified-public fact. Fixture, demo, research, and legacy
or unverified review origins cannot complete genuine alpha evidence; a historical record
without visibility evidence must be re-fetched and reconfirmed rather than silently migrated.

Report consent and quotation consent are independent. Both default to no. Add `--report-consent` only to allow the reduced public summary; add `--quote-consent` only to permit a quotation. The public summary excludes local notes and permission fields.

Expand Down
3 changes: 2 additions & 1 deletion docs/alpha/participant-quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,8 @@ They do not establish correctness, customer validation, or target-repository run
3. **Minute 3 — enter public sources.** Paste the canonical public PR URL and public HTTPS
requirements source. Select only your role; ScopeProof does not collect names or contact data.
4. **Minute 4 — load the PR.** Confirm source authority and no confidential information, then fetch
the PR. A session-only token is optional under Advanced source options.
the PR. ScopeProof confirms public visibility from GitHub metadata; a session-only token is
optional under Advanced source options but cannot make a private repository eligible.
5. **Minute 5 — confirm criteria.** Prepare one independently judgeable behavior per line, review
the normalized set, and explicitly confirm it. This creates one validated local alpha case.
6. **Minute 6 — review coverage.** Run analysis and inspect every Strong candidate, Weak candidate,
Expand Down
181 changes: 181 additions & 0 deletions docs/superpowers/plans/2026-08-09-public-provenance-boundary.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,181 @@
# Verified Public Provenance Boundary Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Ensure only pull requests with verified public GitHub repository metadata can be labeled live-public or counted as genuine alpha evidence.

**Architecture:** GitHub ingestion produces the only trusted verified-public visibility fact. Shared Pydantic models persist that fact through snapshots, reviews, alpha cases, storage, and exports; CLI and Streamlit copy it only through guarded review construction, and alpha transitions independently require it.

**Tech Stack:** Python 3.11+, Pydantic v2, httpx, Typer CLI, Streamlit, pytest

## Global Constraints

- ScopeProof remains an evidence assistant, not a correctness oracle.
- Never execute target-repository code.
- `verified_public` requires unambiguous GitHub metadata; missing or contradictory metadata fails closed.
- Tokens remain session-only and absent from records, errors, logs, and exports.
- Legacy records without the fact remain readable as `unverified` but Stage 1-ineligible.
- Every persisted or exported object remains Pydantic-validated.
- Use named-file staging only and preserve `.coverage 2` untouched.

---

### Task 1: GitHub ingestion establishes verified visibility

**Files:**
- Modify: `scopeproof_core/schemas/models.py`
- Modify: `scopeproof_core/github/client.py`
- Test: `tests/github/test_client.py`

**Interfaces:**
- Produces: `RepositoryVisibility.UNVERIFIED` and `RepositoryVisibility.VERIFIED_PUBLIC`
- Produces: `PullRequestSnapshot.repository_visibility: RepositoryVisibility`
- Produces: `RepositoryVisibilityUnverified`, a bounded ingestion error

- [ ] **Step 1: Write failing public/private/malformed/mismatched metadata tests**

Add literal GitHub response fixtures asserting that only matching `private: false` plus
`visibility: public` returns a snapshot with `verified_public`; explicit private and
all ambiguous variants raise bounded errors before secondary fetches.

- [ ] **Step 2: Run the focused tests and confirm the intended failures**

Run: `pytest tests/github/test_client.py -q`

Expected: failures because the visibility enum, field, and fail-closed validation do
not exist.

- [ ] **Step 3: Implement the minimal typed ingestion boundary**

Add the enum and snapshot field. Validate `base.repo.full_name`, `private`, and
`visibility` immediately after the PR response, before fetching files, commits, or
checks. Return `verified_public` only for the unambiguous public case.

- [ ] **Step 4: Run the focused tests and confirm they pass**

Run: `pytest tests/github/test_client.py -q`

- [ ] **Step 5: Commit the ingestion contract with named files**

Stage only the schema, client, and GitHub client test files.

### Task 2: Review construction, persistence, and exports retain the fact

**Files:**
- Modify: `scopeproof_core/schemas/models.py`
- Modify: `scopeproof_core/cli.py`
- Modify: `apps/web/app.py`
- Test: `tests/schemas/test_models.py`
- Test: `tests/cli/test_cli.py`
- Test: `tests/apps/test_streamlit_app.py`
- Test: `tests/storage/test_json_store.py`
- Test: `tests/reporting/test_exporters.py`

**Interfaces:**
- Produces: `Review.repository_visibility: RepositoryVisibility`
- Consumes: `PullRequestSnapshot.repository_visibility`
- Enforces: current construction cannot pair `LIVE_PUBLIC_GITHUB` with `unverified`

- [ ] **Step 1: Write failing schema, CLI, Streamlit, reopen, and export regressions**

Assert live review construction rejects unverified snapshots, verified snapshots keep
the fact across CLI/web analysis, save/reopen, and JSON export, and historical payloads
without the field load as `unverified` without receiving invented evidence.

- [ ] **Step 2: Run the focused tests and confirm the intended failures**

Run the named test modules with `pytest -q` and verify each failure names the absent
guard or field.

- [ ] **Step 3: Implement minimal propagation and guards**

Copy snapshot visibility into every review constructor. Guard shared CLI bundle
construction and Streamlit analysis before assigning `LIVE_PUBLIC_GITHUB`. Preserve
legacy readability through the enum default.

- [ ] **Step 4: Run all focused modules and confirm they pass**

Run: `pytest tests/schemas/test_models.py tests/cli/test_cli.py tests/apps/test_streamlit_app.py tests/storage/test_json_store.py tests/reporting/test_exporters.py -q`

- [ ] **Step 5: Commit the propagation contract with named files**

Stage only the touched production and test files.

### Task 3: Genuine alpha requires verified public provenance

**Files:**
- Modify: `scopeproof_core/alpha/models.py`
- Modify: `scopeproof_core/alpha/service.py`
- Modify: `scopeproof_core/alpha/__init__.py`
- Modify: `apps/web/app.py`
- Test: `tests/alpha/test_models.py`
- Test: `tests/alpha/test_service.py`
- Test: `tests/alpha/test_storage.py`
- Test: `tests/apps/test_streamlit_app.py`

**Interfaces:**
- Produces: `AlphaQualificationInput` for pre-fetch session validation
- Produces: `AlphaQualification.repository_visibility == verified_public`
- Produces: `AlphaCaseRecord.repository_visibility`, defaulting legacy records to `unverified`
- Consumes: the loaded snapshot's verified visibility during alpha-case creation

- [ ] **Step 1: Write failing qualification, legacy, outcome, and UI regressions**

Assert verified visibility is required to create a new alpha case, legacy cases remain
readable but cannot record outcomes, unverified reviews cannot record outcomes, and
the Streamlit alpha flow binds the qualification to the loaded verified snapshot.

- [ ] **Step 2: Run the focused tests and confirm the intended failures**

Run: `pytest tests/alpha/test_models.py tests/alpha/test_service.py tests/alpha/test_storage.py tests/apps/test_streamlit_app.py -q`

- [ ] **Step 3: Implement the minimal alpha boundary**

Separate session intake from verified qualification, persist the verified fact in new
alpha cases, add both case and review checks to outcome and public-summary transitions,
and pass the loaded snapshot fact through Streamlit.

- [ ] **Step 4: Run the focused tests and confirm they pass**

Run the same named modules and confirm no token or private metadata enters output.

- [ ] **Step 5: Commit the alpha contract with named files**

Stage only the touched alpha, app, and test files.

### Task 4: Full verification and owner-ready pull request

**Files:**
- Modify if needed: authoritative trust/status documentation directly affected by this repair
- Test: all repository checks and installed-wheel browser workflow

**Interfaces:**
- Consumes: the completed verified-public implementation
- Produces: a ready-for-review PR with all available checks resolved or classified

- [ ] **Step 1: Run Ruff and the complete suite with combined coverage**

Require at least 95% combined coverage and preserve intentional skip classifications.

- [ ] **Step 2: Run repository contracts and both deterministic benchmarks**

Require zero mismatches, zero must-have False Ready outcomes, zero false blockers, and
zero unexecuted declared categories.

- [ ] **Step 3: Build two wheels and compare SHA-256 values**

Require byte-identical wheel hashes, validate clean dependencies, installed version
equality, both CLI versions, installed benchmarks, and exact loopback workbench health.

- [ ] **Step 4: Run the explicit installed-wheel browser regression**

Require loopback-only networking and zero console or page errors.

- [ ] **Step 5: Audit the diff and request independent review**

Resolve every actionable Critical or Important finding without unrelated refactoring.

- [ ] **Step 6: Commit remaining intentional files, push, and open a ready PR**

Monitor CI, CodeQL, Pages, and every available check. Do not merge; stop only at the
clean owner merge decision.
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# Verified Public Provenance Boundary

## Problem

ScopeProof currently treats any successful non-fixture GitHub pull-request fetch as
`live_public_github`. An optional token can make a private repository readable, so a
successful response is not evidence that the repository is public. That unsupported
label can flow into saved reviews, exports, and genuine-alpha outcomes.

## Decision

Introduce a typed `RepositoryVisibility` fact with two persisted states:

- `verified_public`: GitHub returned an unambiguous public visibility assertion for
the repository that owns the requested pull request.
- `unverified`: no current verified-public assertion exists. This is the default for
historical records and local/demo fixtures.

GitHub ingestion is the only adapter allowed to produce `verified_public`. It must
require a complete repository object whose `full_name` matches the requested
repository, whose `private` field is exactly `false`, and whose `visibility` field is
exactly `public`. An explicit private/internal value is rejected as private or
inaccessible. Missing, malformed, mismatched, or contradictory metadata is rejected
with a bounded visibility-unverified error. No snapshot is returned on either path.

## Data flow

`PullRequestSnapshot.repository_visibility` carries the ingestion result. Review
construction copies it to `Review.repository_visibility`. A review may be labeled
`LIVE_PUBLIC_GITHUB` only when the snapshot is `verified_public`; current CLI and web
paths fail closed otherwise. The field is included automatically in validated saved
records and JSON exports.

Historical saved reviews remain readable because missing fields validate as
`unverified`. They are not silently migrated to `verified_public`. A fresh GitHub
fetch and normal criteria reconfirmation are required to create a newly verified
review.

Genuine-alpha qualification is split into a session-only intake and a verified
qualification. `AlphaQualificationInput` validates the pre-fetch human inputs.
`AlphaQualification` adds a required `verified_public` fact obtained from the loaded
snapshot. `AlphaCaseRecord` persists that fact; legacy cases default to `unverified`
and cannot record outcomes or produce public summaries. Outcome recording requires
both the case and the matching review to be verified public.

## Security and privacy invariants

- Tokens remain HTTP-client state only and are never added to a model, exception,
log, saved record, or export.
- Private, internal, missing, malformed, contradictory, and repository-mismatched
visibility responses fail closed before files, commits, or checks are fetched.
- Local fixtures and the constructed demo remain usable but never become live-public
evidence.
- All persisted and exported data remains Pydantic-validated.
- Legacy records remain inspectable while carrying an explicit `unverified` fact.
- No target-repository code is executed.

## Verification

Regression coverage spans GitHub ingestion, schema validation, CLI review, Streamlit
loading and analysis, alpha qualification and outcomes, local save/reopen, and JSON
export. Focused tests are run red before the implementation. The final branch must
also pass the full repository verification, reproducible wheel, installed-package,
workbench, browser, and GitHub check gates defined by the owner objective.
Loading