Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 69 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,74 @@ jobs:
(cd "$RUNNER_TEMP" && scopeproof-web --version)
(cd "$RUNNER_TEMP" && python -m scopeproof_core.evals.r002_swebench --help)

compatibility-python-313:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
fetch-depth: 0
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: "3.13"
cache: pip
- name: Upgrade packaging tools
run: python -m pip install --upgrade pip
- name: Install
run: python -m pip install -e '.[dev,research]'
- name: Test
run: python -m pytest -q
- name: Deterministic benchmarks
run: |
python -m scopeproof_core.evals.runner
scopeproof comparison-benchmark
- name: Installed wheel compatibility smoke
run: |
wheel_dir="$RUNNER_TEMP/scopeproof-wheel-python-313"
python -m pip wheel . --no-deps --wheel-dir "$wheel_dir"
wheel="$(find "$wheel_dir" -type f -name 'scopeproof-*.whl' -print -quit)"
python -m pip install --force-reinstall --no-deps "$wheel"
python -m pip check
(cd "$RUNNER_TEMP" && python -c 'from importlib.metadata import version; from scopeproof_core import __version__; assert version("scopeproof") == __version__')
(cd "$RUNNER_TEMP" && scopeproof --version)
(cd "$RUNNER_TEMP" && scopeproof-web --version)
(cd "$RUNNER_TEMP" && scopeproof benchmark)
(cd "$RUNNER_TEMP" && scopeproof comparison-benchmark)

web_log="$RUNNER_TEMP/scopeproof-web-python-313.log"
web_pid=""
cleanup() {
if [ -n "$web_pid" ]; then
kill -- -"$web_pid" 2>/dev/null || true
wait "$web_pid" 2>/dev/null || true
fi
}
trap cleanup EXIT

setsid scopeproof-web --host 127.0.0.1 --port 8513 > "$web_log" 2>&1 &
web_pid=$!

healthy=0
for attempt in $(seq 1 30); do
if ! kill -0 "$web_pid" 2>/dev/null; then
echo "Installed Python 3.13 workbench exited before becoming healthy."
cat "$web_log"
exit 1
fi
if response="$(curl --fail --silent http://127.0.0.1:8513/_stcore/health 2>/dev/null)"; then
if [ "$response" = "ok" ]; then
healthy=1
break
fi
fi
sleep 1
done

if [ "$healthy" -ne 1 ]; then
echo "Installed Python 3.13 workbench did not become healthy within 30 seconds."
cat "$web_log"
exit 1
fi

locked-environment:
runs-on: ubuntu-latest
steps:
Expand All @@ -59,7 +127,7 @@ jobs:
run: python -m uv run scopeproof comparison-benchmark

verify:
needs: [compatibility-python-311, locked-environment]
needs: [compatibility-python-311, compatibility-python-313, locked-environment]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
Expand Down
134 changes: 134 additions & 0 deletions docs/audits/accessibility-platform-evidence/verification.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
# Accessibility and platform evidence verification

## Evidence boundary

- Date: 2026-08-09 (America/Toronto).
- Branch: `codex/accessibility-platform-evidence`.
- Final executable/test verification HEAD: `1a3defcfec5f9f2231a67e8f0a4450d43a63b9b4`
with tree `b01e358b5aefefd7eaf8abf28e0bf7502cf962af`.
- Base: `origin/main` at `30177733ef312ced22e6a2e57e3df6fdb1e92507`.
- Host: macOS 26.5.1 build 25F80 on Apple silicon.

This is engineering evidence for ScopeProof's own installed workflow. It does not establish WCAG
conformance, correctness, target-repository runtime behavior, authenticated reviewer authority,
customer use, production use, accessibility by every user, or broad platform support. It does not
advance Stage 1.

The canonical Stage 1 counts remain:

- 0/5 qualifying reviews;
- 0/3 independent practitioners;
- 0/3 public repositories;
- 0/3 independently observed under-ten-minute completions;
- 0/2 reuse-intent signals.

## Implemented evidence slice

- The installed-wheel Playwright regression now reaches the complete constructed primary path
with page-level Tab navigation, keyboard typing, Enter, and Space only. It no longer clicks the
demo disclosure or moves focus programmatically.
- The regression requires the demo disclosure, demo-load button, reviewer input, criteria-confirm
button, and deterministic-analysis button to be enabled and intersect the viewport. It compares
unfocused and focused computed styles and requires a changed, nontransparent outline or shadow,
so always-on decoration and transparent focus styling cannot satisfy the assertion.
- Existing 1280-by-720 and 390-by-844 contexts, isolated temporary home, loopback-only network
guard, browser-error assertions, evidence content, export controls, width checks, and process
cleanup remain in force.
- CI now has a separate `compatibility-python-313` job. It does not rename the existing Python
3.11 or `verify` jobs. The new job runs the complete suite, both deterministic benchmarks, wheel
build/reinstallation, `pip check`, both CLI version commands, installed benchmarks, and exact
loopback workbench health on CPython 3.13. `verify` depends on the new job.
- ScopeProof product and deterministic-gate source code did not change in this slice.

## Keyboard-only installed-wheel evidence

Command:

```text
uv run pytest -q -m browser tests/browser/test_packaged_workbench.py
```

Result on the final executable/test verification tree: `3 passed in 26.11s`. This includes the
full installed-wheel browser journey plus pure regressions proving that unchanged always-on
decoration is rejected and a changed visible outline is accepted. The complete suite on the same
tree passed with `1,956 passed, 2 skipped` and 95.06% combined coverage. The hosted `verify` job
also passed the packaged browser regression on this exact head.

Earlier in the slice, the first new journey run failed because the helper required the Confirm
button to be enabled before Tab blurred and committed the reviewer input. The DOM already
contained the full reviewer value. Moving the enabled assertion to the point where keyboard focus
actually reaches the target fixed that ordering defect. A subsequent independent review found
that the original focus predicate could accept unchanged or transparent decoration; the final
helper and the two pure regressions above close that false-pass path. No ScopeProof product change
was required for either repair.

The final regression repeated the full path in fresh 1280-by-720 and 390-by-844 contexts. It
opened `Try ScopeProof`, loaded the deliberately constructed demo, entered the asserted reviewer
label, confirmed normalized criteria, and ran deterministic analysis without pointer input or
programmatic focus. Every primary target met the visible-focus and in-viewport assertions. The
matrix retained missing-evidence guidance, the summary remained fail-closed at `Action required`,
and Markdown, JSON, and CSV controls were visible and enabled. The final browser and external-
request lists were empty.

This proves the tested primary path. It does not prove every control or all possible tab orders.

## Actual 200% Chrome zoom evidence

- Browser: installed Google Chrome 151.0.7922.77.
- Physical outer window: 3440 by 1328 pixels in both observations.
- At reset 100%: device pixel ratio 1; CSS viewport 3440 by 1151.
- At actual 200%: Chrome's toolbar exposed `Zoom: 200%`; device pixel ratio was 2; CSS viewport
was 1720 by 575; document horizontal scroll width was 1720.

At 200%, a second exact-implementation-head journey used only keyboard navigation. Focus reached
the demo disclosure, load button, reviewer input, confirmation button, and analysis button after
8, 1, 13, 36, and 9 bounded Tab presses respectively. Each target was enabled, inside the current
viewport, and showed either the three-pixel focus outline or the explicit focus shadow. The full
evidence matrix, `5 · Summary & Export`, all three download controls, and `Review status: Action
required` appeared. No named visible button, link, input, textarea, select, summary, or button-role
element extended beyond the horizontal viewport. Chrome was restored to device pixel ratio 1 and
the 3440-by-1151 CSS viewport after the run.

This is real native-zoom evidence for the exact Chrome/macOS/display configuration above. It is
not a claim about other browsers, displays, platforms, zoom levels, assistive technologies, or
accessibility conformance. Responsive viewport resizing was not used as zoom evidence.

## Python 3.13 evidence

A genuine uv-managed `cpython-3.13.14-macos-aarch64-none` interpreter was installed from uv's
managed Python distribution. On the implementation tree, a fresh wheel was built and installed
with runtime dependencies into an isolated Python 3.13 virtual environment outside the checkout.

The following checks passed:

- `Python 3.13.14` interpreter identity;
- wheel build and clean runtime-dependency installation;
- `pip check` with no broken requirements;
- installed package metadata and `scopeproof_core.__version__` equality at `0.2.3`;
- `scopeproof --version` and `scopeproof-web --version` at `0.2.3`;
- deterministic benchmark: 12 cases, 13 criteria, zero mismatches, zero must-have False Ready
outcomes, zero false blockers, and zero unexecuted declared categories;
- comparison benchmark: 2 cases and zero mismatches; and
- installed `scopeproof-web` exact loopback health response `ok`.

The local result establishes Python 3.13 package, CLI, deterministic-gate, and health
compatibility on this macOS host. The separate hosted job must pass on the PR before Linux-runner
Python 3.13 evidence is claimed. Neither result is Linux desktop evidence.

## Unsupported evidence rows

| Row | Classification | Exact boundary |
| --- | --- | --- |
| VoiceOver or another real screen reader | Unsupported current evidence | macOS exposed the installed `com.apple.VoiceOver` service, but two bounded attempts to obtain a controllable, observable VoiceOver state timed out. No speech, caption-panel, focus-announcement, or reading-order evidence was available, so no screen-reader result is claimed. |
| Windows desktop | Unavailable environment | No real Windows environment was available for build, installation, launch, or representative workflow completion. |
| Linux desktop | Unavailable environment | No real Linux desktop environment was available. The headless Ubuntu CI job is package/runtime evidence only. |
| Accessibility conformance | Unsupported claim | Keyboard, focus, native-zoom, DOM, and automated evidence cannot by themselves establish WCAG conformance or usability for every user. |

## Verification status

The focused browser regression, repository contracts, complete local suite, and local Python 3.13
package/CLI/health checks passed on the final executable/test verification tree. Hosted CI passed
the Python 3.11, Python 3.13, locked-environment, full verification, packaged-browser, and CodeQL
checks on the same head. The opt-in informational ScopeProof evidence-review job skipped by design.
The audit-only commit that records these results necessarily follows the tested head and does not
change executable or test content.
12 changes: 9 additions & 3 deletions docs/releases/v0.2.3-status-and-next-stages.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,10 +162,11 @@ remains zero independent use.
| Missing GitHub patch integrity | Repaired on merged source | Absent, null, and empty patches make ingestion partial and identify the excluded path; malformed patch types fail with a bounded adapter error |
| Reviewer-loop friction | Reduced on merged source | Reopened source identity, direct matrix navigation, insufficient-evidence rationale, same-head comparison, and comparison downloads have regression coverage |
| R-003 holdout | Owner decision | Approve the outcome-blind cohort and criteria proposal before cohort generation; approve labels separately before scoring |
| Keyboard and assistive technology | Unsupported current evidence | The installed flow completed with pointer automation, but the browser harness could not reliably demonstrate native keyboard activation; complete the packaged flow keyboard-only, with visible focus and a real screen reader |
| 200% zoom | Unsupported current evidence | A 640 px responsive check showed no horizontal overflow, but the browser harness could not apply reliable page zoom; run the packaged workbench at actual 200% browser zoom without loss of content or operation |
| Keyboard-only primary path | Supported local engineering evidence | The installed-wheel regression now completes the constructed path at 1280×720 and 390×844 with Tab, keyboard typing, Enter, and Space only; five primary targets must expose visible focus in the viewport. This does not prove every control or accessibility conformance. |
| 200% zoom | Supported bounded local evidence | Installed Chrome 151.0.7922.77 reported native `Zoom: 200%`, device pixel ratio changed from 1 to 2, the exact-head keyboard path completed, and no named control was horizontally clipped on the tested macOS/display configuration. This is not responsive resizing or a broad browser/platform claim. |
| VoiceOver or another real screen reader | Unsupported current evidence | The installed VoiceOver service was detectable, but bounded control attempts exposed no observable screen-reader state, speech, caption, focus announcement, or reading-order result. |
| Python 3.11 | Remote CI evidence only | Python 3.11 CI passed at the exact merged PR head; no local Python 3.11 desktop flow was executed |
| Python 3.13 | Unavailable environment | Run current-head gates on an actual Python 3.13 interpreter |
| Python 3.13 | Supported local engineering evidence | A genuine CPython 3.13.14 environment passed clean wheel installation, `pip check`, both versioned CLIs, both deterministic benchmarks, and exact workbench health. The new hosted 3.13 job must pass before its Linux-runner result is claimed. |
| Windows and Linux desktop | Unavailable environment | Build, install, launch, and complete representative workflows on those operating systems |
| Stage 1 | External evidence gate | A non-owner supplies a real public PR, public requirements, source-owner confirmation, exact head SHA, saved review, and genuine outcome |
| Stage 2 | Stage gate | Every Stage 1 target passes before repeat-use and commercial-discovery evidence is collected |
Expand All @@ -191,6 +192,11 @@ lock check, both deterministic benchmarks, clean installation, installed
benchmarks, workbench health, browser journey, and responsive-site checks.
Engineering work does not advance Stage 1.

The current [accessibility and platform evidence audit](../audits/accessibility-platform-evidence/verification.md)
adds a durable installed-wheel keyboard/focus regression, bounded native Chrome 200% evidence, and
genuine local Python 3.13 package/CLI/health evidence. VoiceOver and real Windows/Linux desktop
workflows remain unsupported. These engineering checks do not advance Stage 1.

### Stage 1 — genuine public alpha

Status: `waiting_for_inbound_public_alpha_submission`.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# Accessibility and platform evidence implementation plan

> Execute continuously on `codex/accessibility-platform-evidence`. Do not merge, release, tag,
> begin outreach, generate R-003, retune R-002, or claim Stage 1 progress.

**Goal:** Add durable installed-wheel keyboard/focus evidence and genuine Python 3.13 compatibility
evidence while recording native zoom and unsupported environments without overclaiming.

**Architecture:** Keep product and gate code unchanged unless a reproduced defect requires a
test-first repair. Extend the packaged Playwright boundary for keyboard interaction, add an
isolated Python 3.13 CI compatibility job, and update evidence documents only from observed runs.

---

## Task 1: Establish the bounded environment evidence

- Verify branch, base, status, GitHub authentication, checks, available browsers, Python
interpreters, VoiceOver availability, and absent Windows/Linux desktop environments.
- Run the constructed primary path in real Chrome at native 200% zoom using keyboard input.
- Require Chrome's visible 200% indicator plus matching pixel-ratio/viewport metrics, inspect
horizontal control clipping, and restore 100% zoom.
- Attempt VoiceOver only through a bounded observable control path. If no trustworthy state is
exposed, record the exact limitation and stop that probe.

## Task 2: Replace partial keyboard evidence with a durable regression

**File:** `tests/browser/test_packaged_workbench.py`

- Add a helper that reaches a locator with page-level Tab presses only.
- Assert the active target is enabled, intersects the viewport, and has visible focus styling.
- Open the demo disclosure and activate Load, Confirm, and Run using keyboard input only.
- Enter the reviewer field through the keyboard path.
- Preserve the existing installed-wheel, isolated-home, network, error, viewport, content, export,
and teardown assertions.
- Run the focused packaged-browser test. If it exposes a product defect, preserve the failing test,
apply the narrowest repair, and rerun it.

## Task 3: Add genuine Python 3.13 compatibility coverage

**File:** `.github/workflows/ci.yml`

- Add a separate `compatibility-python-313` job without renaming existing protected jobs.
- Install on CPython 3.13, run the complete suite and both deterministic benchmarks, build and
reinstall the wheel, check imports and both versioned entry points, rerun both installed
benchmarks, launch the installed workbench on loopback, require exact health `ok`, and clean up.
- Make `verify` depend on the new compatibility job.
- Obtain a genuine managed local CPython 3.13 interpreter and repeat the package, CLI, health, and
deterministic-gate checks locally.

## Task 4: Update exact evidence documents

**Files:**

- `docs/audits/accessibility-platform-evidence/verification.md`
- `docs/releases/v0.2.3-status-and-next-stages.md`

- Record the exact tested host, Chrome zoom signals, keyboard path, focus assertions, Python 3.13
interpreter, commands, and results.
- Classify VoiceOver as unsupported unless observable screen-reader evidence was obtained.
- Keep Windows and Linux desktop unsupported; distinguish the Linux headless CI job.
- Preserve every trust boundary and the exact Stage 1 zero counts.

## Task 5: Run full verification

- Run Ruff.
- Run the complete suite with combined coverage at or above 95%.
- Run repository contracts and both deterministic benchmarks.
- Build the wheel twice and require matching SHA-256 values.
- Install the final wheel into fresh Python 3.12 and 3.13 environments; run `pip check`, package
imports, both version commands, both benchmarks, and exact loopback health.
- Run the installed-wheel browser regression with the pinned Playwright driver.
- Check distribution inventory, `git diff --check`, status, and preservation of `.coverage 2`.

## Task 6: Publish for owner review

- Stage named files only and commit intentionally.
- Push the feature branch and open a ready-for-review PR against `main`.
- Monitor CI, CodeQL, and review feedback to completion.
- Diagnose failures systematically; fix only confirmed in-scope defects with regression coverage.
- Recheck the final diff, commits, mergeability, exact head, check conclusions, unsupported
environments, and owner decision required.
Loading