Skip to content

feat: add one-shot operator-safe AX press - #9

Draft
saariuslystoned wants to merge 5 commits into
mainfrom
codex/operator-safe-ax-phase1
Draft

feat: add one-shot operator-safe AX press#9
saariuslystoned wants to merge 5 commits into
mainfrom
codex/operator-safe-ax-phase1

Conversation

@saariuslystoned

@saariuslystoned saariuslystoned commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Summary

  • add computer_use_ax_action as the tenth MCP tool
  • require explicit-app AX inspection and issue short-lived one-shot element leases
  • bind action authority to exact process birth, topology, retained window/ancestry, semantic fingerprints, and macOS combined-session input counters
  • dispatch only advertised semantic press, with zero global-HID fallback
  • require fresh computer_use_ax_tree after dispatch, stale/intervention errors, or uncertain outcomes
  • preserve clean-room Computer Use behavior research and defer the exact-window operator HUD to research(preview): build an AGY-native isolated live operator HUD #8

Safety boundary

operator-safe is intentionally narrow in this phase: exact retained-element AXPress, no global HID posts, no physical pointer movement by this route, one-shot consumption, and fail-closed stale/intervention handling. Legacy coordinate and keyboard tools remain global-HID surfaces and are only policy-scoped to exclusive GUI sessions in this PR.

Proof

  • source candidate: bdaee43ffb6c51cb3600e9df2ef01fb9058cc265
  • proof-only lineage: 137b554, wording correction 5985731
  • tracked proof: proof/operator_safe_ax_phase1_20260726.md
  • native strict-concurrency authority: 42/42
  • MCP/Node authority: 55/55
  • TypeScript check: pass
  • production configured stdio: exact ten-tool inventory; minimal-PATH positive and missing/mismatched toolchain negatives pass
  • both skill mirrors validate and match recursively
  • independent read-only re-review: commit-ready, no remaining P0/P1 code blockers; final proof wording corrected additively
  • exact-source GitHub Actions: pass on run 30211472441 attempt 2

The first hosted attempt hit the same pre-existing RL-3 SIGINT cleanup race already present on the unchanged main base; the focused test passed locally and the exact-source rerun passed every job step. No lifecycle assertion was changed or weakened.

Deliberate residual scope

This advances #7 but does not close it. Still open: staged-host live background-app fixture proof, concurrent human-input dogfood, set_value and additional action classes, code-enforced exclusive-session gating for global HID, broad app/framework coverage, durable signing/notarization, and the operator HUD in #8.

@saariuslystoned

Copy link
Copy Markdown
Owner Author

Operator-safe AX live dogfood closeout

Exact PR head: d29b20b6abcbac1964047db48932e927b7b24a31

Live verdict

The isolated staged host qualified the bounded Phase 1 semantic press path against the real Calculator app while Bobby continued using the three-display Mac:

  • exact app-scoped AX selection and one retained-element action attempt per run;
  • global_hid_posts=0, with no physical-pointer movement or focus theft;
  • read-only USER_INTERVENED inspections retried safely;
  • no automatic action retry after dispatch or an action-side error;
  • independent bundled Computer Use observation confirmed visible 0 -> 1 and 1 -> 0 results; and
  • exact isolated PID 63159, socket, and tmux session agy-ax-live-p9 were stopped without broad cleanup.

Dogfood caught and repaired a real unsafe retry that had produced 0 -> 11, plus a vacuous replay probe, selector drift, perception-continuity overclaims, timeout/recovery ordering, and explicit Accessibility enrollment authority. The receipt preserves those negative results.

Exact-head gates

  • Push CI: PASS
  • PR CI: PASS
  • Local native: 42/42; Node/MCP: 56/56; hermetic enrollment: 5/5; both skill mirrors and production/canary readiness: PASS.
  • On source head 95aa013, an initial run hit the known pre-existing RL-3 SIGINT socket-cleanup race; the unchanged rerun passed. No lifecycle assertion was weakened.

Explicit residuals / separate approval

This qualifies semantic press only. Node continuity remains perception-based; uncertain native dispatch can still end in an unknown outcome; the ad-hoc app is not yet a durable TCC principal; legacy coordinate/keyboard/scroll/drag tools remain global-HID/exclusive-session paths. Exact-window thumbnails/HUD remain deferred to #8.

A mistakenly broad, non-hermetic lifecycle test moved pre-existing canonical generation 8679120f-1e76-40e4-8fa6-2220503b2689 to stopping. Its exact daemon/LaunchServices/native PIDs 70423/70497/70500 remained alive and were deliberately left untouched. Recovery still requires explicit approval for those exact processes.

PR remains open and draft.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant