From 0d47873bc6e5da57f129add74a3278bf57a833ab Mon Sep 17 00:00:00 2001 From: John McChesney TenEyck Jr Date: Sun, 24 May 2026 01:56:48 +0100 Subject: [PATCH 1/2] Add Phase 3 hardware validation harness --- docs/NATIVE_ONLY_REDESIGN.md | 6 +- docs/PHASE3_HARDWARE_VALIDATION.md | 57 +++++ docs/SECURITY_POSTURE_AND_TESTING.md | 8 + ...se3-hardware-validation-report.template.md | 48 ++++ scripts/validate-phase3-hardware.sh | 220 ++++++++++++++++++ 5 files changed, 337 insertions(+), 2 deletions(-) create mode 100644 docs/PHASE3_HARDWARE_VALIDATION.md create mode 100644 docs/phase3-hardware-validation-report.template.md create mode 100755 scripts/validate-phase3-hardware.sh diff --git a/docs/NATIVE_ONLY_REDESIGN.md b/docs/NATIVE_ONLY_REDESIGN.md index dda9dc0..a33e545 100644 --- a/docs/NATIVE_ONLY_REDESIGN.md +++ b/docs/NATIVE_ONLY_REDESIGN.md @@ -226,8 +226,10 @@ Deliverables: - The integration is unverified against a notarized build with associated-domain entitlements wired (the macOS build cannot be - exercised from CI on Linux). A follow-up validation pass on a real - macOS host is required before declaring Phase 3 complete. + exercised from CI on Linux). Run + `scripts/validate-phase3-hardware.sh` on a real macOS host and attach a + completed `docs/phase3-hardware-validation-report.md` before declaring + Phase 3 complete. - Domain expansion beyond `example.com` is tracked in issue #8. ### Phase 4: command migration and deprecation diff --git a/docs/PHASE3_HARDWARE_VALIDATION.md b/docs/PHASE3_HARDWARE_VALIDATION.md new file mode 100644 index 0000000..5d0d3d7 --- /dev/null +++ b/docs/PHASE3_HARDWARE_VALIDATION.md @@ -0,0 +1,57 @@ +# Phase 3 notarized hardware validation + +Issue: #43 + +Phase 3 is not complete until APW.app has been exercised on real macOS +hardware as a Developer ID signed, notarized, stapled bundle with associated +domain entitlements. CI can build and unit test the broker, but it cannot prove +that Apple's credential picker appears for a notarized app on a user's machine. + +## Validation command + +Run this on the real validation host: + +```bash +./scripts/validate-phase3-hardware.sh \ + --app /path/to/APW.app \ + --apw /path/to/apw \ + --url https://example.com \ + --report docs/phase3-hardware-validation-report.md +``` + +Use a test associated domain that has a valid AASA file and an iCloud Keychain +credential already saved for that domain. The script intentionally does not +persist returned usernames or passwords. + +## What the script proves + +The script fails closed unless all of these checks pass: + +- host is macOS +- `APW.app` exists and contains `Contents/MacOS/APW` +- the app bundle passes `codesign --deep --strict --verify` +- the app bundle passes `spctl --assess --type execute` +- the app bundle passes `xcrun stapler validate` +- bundle entitlements include at least one `webcredentials:` associated domain +- `apw app install` succeeds +- `apw app launch` succeeds +- `apw status --json` reports the app installed and the broker running +- `apw login ` exits successfully +- the operator confirms the native iCloud Keychain picker appeared +- the operator confirms the selected credential was returned by APW + +The operator confirmations are required because the picker is a user-mediated +OS UI flow and the script must not scrape or save credential values. + +## Error paths to record + +After a successful run, record the documented error paths in the generated +report: + +- cancel: dismiss the credential picker and record the broker error code +- denied: deny the APW approval prompt, when that prompt is present +- timeout: stop or block the broker and record the CLI timeout code +- unsupported domain: request a domain outside the app entitlement set + +Do not remove the Phase 3 exit blocker in `docs/NATIVE_ONLY_REDESIGN.md` until +the report captures success plus the required error paths on a notarized host. diff --git a/docs/SECURITY_POSTURE_AND_TESTING.md b/docs/SECURITY_POSTURE_AND_TESTING.md index 9b5a30f..ba5d58f 100644 --- a/docs/SECURITY_POSTURE_AND_TESTING.md +++ b/docs/SECURITY_POSTURE_AND_TESTING.md @@ -64,6 +64,12 @@ cargo build --manifest-path rust/Cargo.toml --release ./scripts/build-native-app.sh ``` +Before claiming Phase 3 complete for a public release, run the real-hardware +notarized broker validation in +[PHASE3_HARDWARE_VALIDATION.md](PHASE3_HARDWARE_VALIDATION.md). This check is +manual because CI cannot prove that the native iCloud Keychain picker appears +for a notarized app with associated-domain entitlements. + ## Security-focused regression coverage The Rust test suite covers: @@ -77,6 +83,8 @@ The Rust test suite covers: - native app diagnostics and `APW_DEMO=1` bootstrap credential file initialization - end-to-end v2 app install, launch, status, doctor, and login flows - direct-exec fallback, unsupported-domain handling, denial handling, and malformed broker response mapping +- a manual notarized-hardware validation contract for the Phase 3 + AuthenticationServices broker flow ## Archive policy diff --git a/docs/phase3-hardware-validation-report.template.md b/docs/phase3-hardware-validation-report.template.md new file mode 100644 index 0000000..b64809b --- /dev/null +++ b/docs/phase3-hardware-validation-report.template.md @@ -0,0 +1,48 @@ +# Phase 3 hardware validation report + +Issue: #43 + +Status: not yet validated + +## Host + +- Date: +- macOS version: +- Hardware model: +- Architecture: +- APW.app version: +- APW CLI version: +- Test associated domain: +- Release tag or commit: + +## Automated checks + +- [ ] `codesign --deep --strict --verify APW.app` +- [ ] `spctl --assess --type execute --verbose APW.app` +- [ ] `xcrun stapler validate APW.app` +- [ ] Associated-domain entitlement contains `webcredentials:` +- [ ] `apw app install` +- [ ] `apw app launch` +- [ ] `apw status --json` reports installed app and running broker +- [ ] `apw login ` exits successfully + +## Operator-observed flow + +- [ ] Native iCloud Keychain credential picker appeared +- [ ] Operator selected the expected test credential +- [ ] APW returned a credential response without saving it to disk + +## Error paths + +| Path | Expected result | Observed result | +| --- | --- | --- | +| Success | credential response with `userMediated: true` | | +| Cancel | stable canceled/denied broker error | | +| Denied | stable denied broker error | | +| Timeout | communication timeout error | | +| Unsupported domain | no-results or unsupported-domain error | | + +## Notes + +- Do not paste real usernames, passwords, session tokens, or credential payloads + into this report. diff --git a/scripts/validate-phase3-hardware.sh b/scripts/validate-phase3-hardware.sh new file mode 100755 index 0000000..b847e38 --- /dev/null +++ b/scripts/validate-phase3-hardware.sh @@ -0,0 +1,220 @@ +#!/usr/bin/env bash +set -euo pipefail + +usage() { + cat <<'USAGE' +Usage: ./scripts/validate-phase3-hardware.sh --app PATH --apw PATH --url URL [--report PATH] + +Validate the Phase 3 APW.app credential-broker flow on real macOS hardware. +The script checks code signing, Gatekeeper, notarization stapling, associated +domain entitlements, app install/launch/status, and a user-mediated login. + +Options: + --app PATH Path to the notarized APW.app bundle. + --apw PATH Path to the matching apw CLI binary. + --url URL HTTPS URL for the associated-domain credential test. + --report PATH Markdown report output path. + -h, --help Show this help. +USAGE +} + +APP_PATH="" +APW_BIN="" +TEST_URL="" +REPORT_PATH="docs/phase3-hardware-validation-report.md" + +while [ "$#" -gt 0 ]; do + case "$1" in + --app) + APP_PATH="${2:-}" + shift 2 + ;; + --apw) + APW_BIN="${2:-}" + shift 2 + ;; + --url) + TEST_URL="${2:-}" + shift 2 + ;; + --report) + REPORT_PATH="${2:-}" + shift 2 + ;; + -h|--help) + usage + exit 0 + ;; + *) + echo "Unknown argument: $1" >&2 + usage >&2 + exit 2 + ;; + esac +done + +fail() { + echo "phase3 validation failed: $*" >&2 + exit 1 +} + +require_command() { + if ! command -v "$1" >/dev/null 2>&1; then + fail "missing required command: $1" + fi +} + +run_step() { + label="$1" + shift + echo "==> $label" + "$@" +} + +json_field_true() { + json="$1" + field="$2" + printf '%s' "$json" | /usr/bin/python3 -c ' +import json +import sys + +field = sys.argv[1].split(".") +payload = json.load(sys.stdin) +value = payload +for part in field: + value = value[part] +if value is not True: + raise SystemExit(1) +' "$field" +} + +if [ "$(uname -s)" != "Darwin" ]; then + fail "real-hardware validation must run on macOS" +fi + +[ -n "$APP_PATH" ] || fail "--app is required" +[ -n "$APW_BIN" ] || fail "--apw is required" +[ -n "$TEST_URL" ] || fail "--url is required" + +case "$TEST_URL" in + https://*) ;; + *) fail "--url must be an https URL" ;; +esac + +[ -d "$APP_PATH" ] || fail "APW.app bundle not found: $APP_PATH" +[ -x "$APP_PATH/Contents/MacOS/APW" ] || fail "APW.app executable missing or not executable" +[ -x "$APW_BIN" ] || fail "apw CLI missing or not executable: $APW_BIN" + +APP_PARENT="$(cd "$(dirname "$APP_PATH")" && pwd -P)" || + fail "failed to resolve APW.app parent directory" +APP_NAME="$(basename "$APP_PATH")" +if [ "$APP_NAME" != "APW.app" ]; then + fail "--app must point to a bundle named APW.app so apw app install uses the validated bundle" +fi +APP_PATH="$APP_PARENT/$APP_NAME" +APW_BIN="$(cd "$(dirname "$APW_BIN")" && pwd -P)/$(basename "$APW_BIN")" + +require_command codesign +require_command spctl +require_command xcrun +require_command plutil +require_command /usr/bin/python3 + +run_step "Verify Developer ID code signature" \ + codesign --deep --strict --verify "$APP_PATH" + +run_step "Assess Gatekeeper execution policy" \ + spctl --assess --type execute --verbose "$APP_PATH" + +run_step "Validate notarization staple" \ + xcrun stapler validate "$APP_PATH" + +entitlements_file="$(mktemp "${TMPDIR:-/tmp}/apw-entitlements.XXXXXX")" +trap 'rm -f "$entitlements_file"' EXIT + +codesign -d --entitlements :- "$APP_PATH" >"$entitlements_file" 2>/dev/null || + fail "failed to read APW.app entitlements" + +if ! grep -q "webcredentials:" "$entitlements_file"; then + fail "APW.app entitlements do not include a webcredentials associated domain" +fi + +run_step "Install APW.app with matching CLI" \ + sh -c 'cd "$1" && "$2" app install' sh "$APP_PARENT" "$APW_BIN" + +run_step "Launch APW.app broker" \ + "$APW_BIN" app launch + +status_json="$("$APW_BIN" status --json)" +json_field_true "$status_json" "payload.app.installed" || + fail "status JSON did not report installed app" +json_field_true "$status_json" "payload.app.service.running" || + fail "status JSON did not report running broker" + +echo "==> Running user-mediated login request" +echo "The next command should show the native iCloud Keychain credential picker." +echo "Do not paste credential values into the generated report." +if ! "$APW_BIN" login "$TEST_URL" >/dev/null; then + fail "apw login failed for $TEST_URL" +fi + +printf "Did the native iCloud Keychain credential picker appear? [yes/no] " +read -r picker_seen +[ "$picker_seen" = "yes" ] || fail "operator did not confirm credential picker" + +printf "Did APW return the selected test credential? [yes/no] " +read -r credential_returned +[ "$credential_returned" = "yes" ] || fail "operator did not confirm credential response" + +mkdir -p "$(dirname "$REPORT_PATH")" +{ + echo "# Phase 3 hardware validation report" + echo + echo "Issue: #43" + echo + echo "Status: success path validated; error paths require manual entries below" + echo + echo "## Host" + echo + echo "- Date: $(date -u +%Y-%m-%dT%H:%M:%SZ)" + echo "- macOS version: $(sw_vers -productVersion)" + echo "- Hardware model: $(sysctl -n hw.model)" + echo "- Architecture: $(uname -m)" + echo "- APW.app path: $APP_PATH" + echo "- APW CLI path: $APW_BIN" + echo "- Test associated domain URL: $TEST_URL" + echo + echo "## Automated checks" + echo + echo "- [x] codesign strict verification" + echo "- [x] Gatekeeper execution assessment" + echo "- [x] notarization staple validation" + echo "- [x] associated-domain entitlement contains webcredentials" + echo "- [x] apw app install" + echo "- [x] apw app launch" + echo "- [x] apw status --json reports installed app and running broker" + echo "- [x] apw login exits successfully" + echo + echo "## Operator-observed flow" + echo + echo "- [x] Native iCloud Keychain credential picker appeared" + echo "- [x] Operator selected the expected test credential" + echo "- [x] APW returned a credential response without saving it to disk" + echo + echo "## Error paths" + echo + echo "| Path | Expected result | Observed result |" + echo "| --- | --- | --- |" + echo "| Success | credential response with userMediated true | PASS |" + echo "| Cancel | stable canceled/denied broker error | TODO |" + echo "| Denied | stable denied broker error | TODO |" + echo "| Timeout | communication timeout error | TODO |" + echo "| Unsupported domain | no-results or unsupported-domain error | TODO |" + echo + echo "## Notes" + echo + echo "- No credential values were written by this script." +} >"$REPORT_PATH" + +chmod 0600 "$REPORT_PATH" +echo "Wrote validation report: $REPORT_PATH" From 5228eaa93315afe8d0e37c8c585341880791df33 Mon Sep 17 00:00:00 2001 From: John McChesney TenEyck Jr Date: Sun, 24 May 2026 05:28:15 +0100 Subject: [PATCH 2/2] Require Phase 3 error-path observations --- docs/PHASE3_HARDWARE_VALIDATION.md | 9 +- ...se3-hardware-validation-report.template.md | 3 +- scripts/validate-phase3-hardware.sh | 84 ++++++++++++++++--- 3 files changed, 82 insertions(+), 14 deletions(-) diff --git a/docs/PHASE3_HARDWARE_VALIDATION.md b/docs/PHASE3_HARDWARE_VALIDATION.md index 5d0d3d7..55b61d9 100644 --- a/docs/PHASE3_HARDWARE_VALIDATION.md +++ b/docs/PHASE3_HARDWARE_VALIDATION.md @@ -16,6 +16,7 @@ Run this on the real validation host: --app /path/to/APW.app \ --apw /path/to/apw \ --url https://example.com \ + --unsupported-url https://unsupported.invalid \ --report docs/phase3-hardware-validation-report.md ``` @@ -39,19 +40,23 @@ The script fails closed unless all of these checks pass: - `apw login ` exits successfully - the operator confirms the native iCloud Keychain picker appeared - the operator confirms the selected credential was returned by APW +- the operator records cancel, denied, and timeout observations +- an unsupported-domain credential request fails with a domain/no-credential + error The operator confirmations are required because the picker is a user-mediated OS UI flow and the script must not scrape or save credential values. ## Error paths to record -After a successful run, record the documented error paths in the generated -report: +During a successful run, the script requires observations for the documented +error paths before it writes the generated report: - cancel: dismiss the credential picker and record the broker error code - denied: deny the APW approval prompt, when that prompt is present - timeout: stop or block the broker and record the CLI timeout code - unsupported domain: request a domain outside the app entitlement set + (automated by `--unsupported-url`) Do not remove the Phase 3 exit blocker in `docs/NATIVE_ONLY_REDESIGN.md` until the report captures success plus the required error paths on a notarized host. diff --git a/docs/phase3-hardware-validation-report.template.md b/docs/phase3-hardware-validation-report.template.md index b64809b..67afb00 100644 --- a/docs/phase3-hardware-validation-report.template.md +++ b/docs/phase3-hardware-validation-report.template.md @@ -13,6 +13,7 @@ Status: not yet validated - APW.app version: - APW CLI version: - Test associated domain: +- Unsupported-domain test URL: - Release tag or commit: ## Automated checks @@ -40,7 +41,7 @@ Status: not yet validated | Cancel | stable canceled/denied broker error | | | Denied | stable denied broker error | | | Timeout | communication timeout error | | -| Unsupported domain | no-results or unsupported-domain error | | +| Unsupported domain | no-results or unsupported-domain error from `--unsupported-url` | | ## Notes diff --git a/scripts/validate-phase3-hardware.sh b/scripts/validate-phase3-hardware.sh index b847e38..700c9c8 100755 --- a/scripts/validate-phase3-hardware.sh +++ b/scripts/validate-phase3-hardware.sh @@ -3,24 +3,27 @@ set -euo pipefail usage() { cat <<'USAGE' -Usage: ./scripts/validate-phase3-hardware.sh --app PATH --apw PATH --url URL [--report PATH] +Usage: ./scripts/validate-phase3-hardware.sh --app PATH --apw PATH --url URL [--unsupported-url URL] [--report PATH] Validate the Phase 3 APW.app credential-broker flow on real macOS hardware. The script checks code signing, Gatekeeper, notarization stapling, associated -domain entitlements, app install/launch/status, and a user-mediated login. +domain entitlements, app install/launch/status, a user-mediated login, and +the required manual error-path observations. Options: - --app PATH Path to the notarized APW.app bundle. - --apw PATH Path to the matching apw CLI binary. - --url URL HTTPS URL for the associated-domain credential test. - --report PATH Markdown report output path. - -h, --help Show this help. + --app PATH Path to the notarized APW.app bundle. + --apw PATH Path to the matching apw CLI binary. + --url URL HTTPS URL for the associated-domain credential test. + --unsupported-url URL HTTPS URL outside the app entitlement set. + --report PATH Markdown report output path. + -h, --help Show this help. USAGE } APP_PATH="" APW_BIN="" TEST_URL="" +UNSUPPORTED_URL="https://unsupported.invalid" REPORT_PATH="docs/phase3-hardware-validation-report.md" while [ "$#" -gt 0 ]; do @@ -37,6 +40,10 @@ while [ "$#" -gt 0 ]; do TEST_URL="${2:-}" shift 2 ;; + --unsupported-url) + UNSUPPORTED_URL="${2:-}" + shift 2 + ;; --report) REPORT_PATH="${2:-}" shift 2 @@ -101,6 +108,11 @@ case "$TEST_URL" in *) fail "--url must be an https URL" ;; esac +case "$UNSUPPORTED_URL" in + https://*) ;; + *) fail "--unsupported-url must be an https URL" ;; +esac + [ -d "$APP_PATH" ] || fail "APW.app bundle not found: $APP_PATH" [ -x "$APP_PATH/Contents/MacOS/APW" ] || fail "APW.app executable missing or not executable" [ -x "$APW_BIN" ] || fail "apw CLI missing or not executable: $APW_BIN" @@ -151,6 +163,25 @@ json_field_true "$status_json" "payload.app.installed" || json_field_true "$status_json" "payload.app.service.running" || fail "status JSON did not report running broker" +prompt_observation() { + label="$1" + instructions="$2" + expected="$3" + + echo >&2 + echo "==> Manual error-path check: $label" >&2 + echo "$instructions" >&2 + echo "Expected result: $expected" >&2 + printf "Observed result for %s: " "$label" >&2 + read -r observed + [ -n "$observed" ] || fail "$label observation is required" + printf '%s' "$observed" +} + +markdown_cell() { + printf '%s' "$1" | tr '\n' ' ' | sed 's/|/\\|/g; s/[[:space:]][[:space:]]*/ /g' +} + echo "==> Running user-mediated login request" echo "The next command should show the native iCloud Keychain credential picker." echo "Do not paste credential values into the generated report." @@ -166,6 +197,36 @@ printf "Did APW return the selected test credential? [yes/no] " read -r credential_returned [ "$credential_returned" = "yes" ] || fail "operator did not confirm credential response" +cancel_observed="$(prompt_observation \ + "Cancel" \ + "Run: $APW_BIN login $TEST_URL, then cancel the native credential picker." \ + "stable canceled or denied broker error")" + +denied_observed="$(prompt_observation \ + "Denied" \ + "Run: $APW_BIN login $TEST_URL and deny any APW-owned approval prompt if it appears." \ + "stable denied broker error")" + +timeout_observed="$(prompt_observation \ + "Timeout" \ + "Stop or block the broker, run: $APW_BIN login $TEST_URL, then restore the broker before continuing." \ + "communication timeout error")" + +echo +echo "==> Running unsupported-domain request" +unsupported_output="$("$APW_BIN" login "$UNSUPPORTED_URL" 2>&1)" && { + echo "$unsupported_output" >&2 + fail "unsupported-domain request unexpectedly succeeded for $UNSUPPORTED_URL" +} +case "$unsupported_output" in + *unsupported*|*notHandled*|*no_credential_source*|*No*credential*|*domain*) ;; + *) + echo "$unsupported_output" >&2 + fail "unsupported-domain request did not emit an expected domain/no-credential error" + ;; +esac +unsupported_observed="$(markdown_cell "$unsupported_output")" + mkdir -p "$(dirname "$REPORT_PATH")" { echo "# Phase 3 hardware validation report" @@ -183,6 +244,7 @@ mkdir -p "$(dirname "$REPORT_PATH")" echo "- APW.app path: $APP_PATH" echo "- APW CLI path: $APW_BIN" echo "- Test associated domain URL: $TEST_URL" + echo "- Unsupported-domain test URL: $UNSUPPORTED_URL" echo echo "## Automated checks" echo @@ -206,10 +268,10 @@ mkdir -p "$(dirname "$REPORT_PATH")" echo "| Path | Expected result | Observed result |" echo "| --- | --- | --- |" echo "| Success | credential response with userMediated true | PASS |" - echo "| Cancel | stable canceled/denied broker error | TODO |" - echo "| Denied | stable denied broker error | TODO |" - echo "| Timeout | communication timeout error | TODO |" - echo "| Unsupported domain | no-results or unsupported-domain error | TODO |" + echo "| Cancel | stable canceled/denied broker error | $(markdown_cell "$cancel_observed") |" + echo "| Denied | stable denied broker error | $(markdown_cell "$denied_observed") |" + echo "| Timeout | communication timeout error | $(markdown_cell "$timeout_observed") |" + echo "| Unsupported domain | no-results or unsupported-domain error | $unsupported_observed |" echo echo "## Notes" echo