Skip to content

feat(bridge-macos): Phase 0e2' + 0e3' — list_windows, capture, execute (real AXAPI driving real macOS apps) - #17

Merged
rrader26 merged 3 commits into
mainfrom
feat/bridge-macos-axapi-capture
May 11, 2026
Merged

feat(bridge-macos): Phase 0e2' + 0e3' — list_windows, capture, execute (real AXAPI driving real macOS apps)#17
rrader26 merged 3 commits into
mainfrom
feat/bridge-macos-axapi-capture

Conversation

@rrader26

Copy link
Copy Markdown
Contributor

Summary

Brings the macOS bridge from "scaffolded" to "actually drives macOS
software." Includes list_windows, capture, and execute via the
Accessibility framework (AXAPI). Live-demo PASSED against real
TextEdit on Apple Silicon arm64
— bridge captured the editor
element, typed a message via AXUIElementSetAttributeValue,
re-captured to verify the text landed.

Stacks on PR #16 (the macOS scaffold). Same architecture as the
Windows PR #13 — by design, the Node-side MacosAxapiBackend (Phase
0f') will be a near-exact mirror of WindowsUiaBackend (PR #14).

Validation

End-to-end demo on a real Mac:

Step 2 -- list_windows: 22 windows across 13 apps
  found TextEdit window: axapi:21232:0 (Untitled 5)

Step 3 -- capture TextEdit...
  treeDepth=3  elementCount=47
  editor element_id: First Text View

Step 4 -- execute type...
  ok: True
  newValue: Hello from AgentMark Desktop -- macOS bridge phase 0e3'

Step 5 -- re-capture and verify...
  editor value after type: "Hello from AgentMark Desktop -- macOS bridge phase 0e3'"

LIVE DEMO PASSED -- AgentMark Desktop drove real TextEdit end-to-end on macOS.

A separate list_windows-only test enumerated 22 real windows across
13 apps (Finder x3, Cursor x3, Chrome x5, VS Code, Claude Desktop,
Teams, Messages, Outlook, Parallels, Terminal, etc.) with focused
window correctly marked.

What's in this PR

Sources/AgentMarkBridgeMacos/Axapi.swift

  • AccessibilityPermission.isGranted — probes AXIsProcessTrusted()
  • AxElement — defensive wrapper around AXUIElement. Every
    attribute read returns nil on AXAPI errors so a single stale
    element doesn't abort the capture.
  • RoleMapper — AXAPI roles → normalised DesktopRole vocab.
    Handles subroles for password fields, search fields, dialogs.
    Unknown → "other".

Sources/AgentMarkBridgeMacos/AxapiCapturer.swift

  • listWindows() — walks NSWorkspace.runningApplications, filters
    to .regular policy, creates per-app AX refs, enumerates
    kAXWindowsAttribute. hasFocus marked by cross-referencing
    NSWorkspace.frontmostApplication.
  • capture(req) — resolves target by windowId /
    processId / processName (substring, .app-tolerant) /
    windowTitle, or falls back to focused window. Walks tree
    depth-first with depth + element-count + deadline caps. Stashes
    every walked AX element in a per-session lookup map so execute
    can find it later.
  • execute(req) — routes 8 action types through the right AXAPI
    pattern (AXPress / AXSetValue / AXScrollToVisible /
    AXShowMenu / etc.) with sensible fallbacks (keyboard simulation
    via CGEvent.keyboardSetUnicodeString for text typing when
    AXValue isn't writable; modifier-aware CGEvent for key
    actions).
  • requirePermission() surfaces a clear accessibilityNotGranted
    JSON-RPC error (code -32020) pointing the user at System Settings.

Sources/AgentMarkBridgeMacos/Dispatcher.swift

Routes list_windows / capture / execute to AxapiCapturer
(lazy init). Adds accessibilityGranted to capabilities response
so clients can detect permission state without waiting for first
capture.

scripts/textedit-demo.py

Python end-to-end demo. Same shape as the Windows
notepad-demo.ps1: opens TextEdit, drives capture → execute →
re-capture, asserts the typed text in element.value.

Protocol compatibility with the Windows bridge

Same JSON shape on the wire. Same error code numbering. The
Node-side MacosAxapiBackend (Phase 0f', future PR) shares 90% of
its code with WindowsUiaBackend (PR #14) — only difference is
which binary it spawns and where it looks for it.

Accessibility permission gotcha

macOS guards AXAPI behind System Settings → Privacy & Security →
Accessibility. The parent process that launches the bridge must
be granted:

  • For Claude Desktop integration: grant Claude Desktop
  • For dev / Cursor integration: grant the terminal or Cursor

Without permission the bridge returns
{ "error": { "code": -32020, "message": "Accessibility permission not granted..." } }
on the first list_windows / capture / execute call.

Stack order

  1. Merge PR feat(bridge-macos): Phase 0e1' scaffold + stdio JSON-RPC (macOS sidecar foundation) #16 first (macOS bridge scaffold)
  2. Rebase this PR on the new main, then merge
  3. Phase 0f': Node-side MacosAxapiBackend (mirror of
    WindowsUiaBackend from PR feat(desktop): WindowsUiaBackend — Node-side bridge client (Phase 0f) #14)
  4. Live macOS demo via Claude Desktop driving real macOS apps —
    no PowerShell scripts, just "Capture TextEdit and type into it."

🤖 Generated with Claude Code

rrader26-sys and others added 3 commits May 11, 2026 16:30
Adds the macOS-side sidecar process for AgentMark Desktop, mirroring
the Windows bridge architecture exactly. Same stdio JSON-RPC 2.0
protocol; same DesktopCaptureBackend contract on the Node side. Uses
Apple's Accessibility framework (AXAPI) as the capture / execute
substrate when Phase 0e2'/0e3' land.

This first commit ships the protocol substrate and two no-AXAPI
methods so we can validate the cross-language pipeline before adding
the AXAPI walking + action dispatch logic:

  - ping         -> { pong, version, arch, processId }
  - capabilities -> supported methods + AXAPI provider info

Built natively on Apple Silicon (arm64). Smoke test runs locally on
the developer Mac in seconds.

Layout (apps/agent-runner/bridges/macos/):

  Package.swift                            -- SPM manifest (macOS 13+)
  Sources/AgentMarkBridgeMacos/
    main.swift                             -- entry + stdin read loop +
                                              BOM stripping + dispatch
    JsonRpc.swift                          -- request/response types +
                                              JSON-RPC 2.0 envelope
                                              encoder/decoder + error
                                              code catalog
    Dispatcher.swift                       -- method routing (mirrors
                                              the C# bridge's
                                              RpcDispatcher class)
  scripts/smoke-test.sh                    -- shell harness driving the
                                              binary via stdin; grep-
                                              based assertions, no jq
                                              dependency
  README.md                                -- build + protocol docs

The protocol -- request envelope, error codes (parseError -32700 ..
internalError -32603 plus bridge-specific 32010+), stderr discipline,
BOM stripping -- is byte-identical to the Windows bridge. This is
deliberate: the Node-side `MacosAxapiBackend` (Phase 0f') will be a
near-exact mirror of `WindowsUiaBackend`, sharing 90% of the code
through the abstract DesktopCaptureBackend interface.

.gitignore extended for Swift Package Manager output (.build/,
.swiftpm/, Packages/) alongside the existing .NET ignores.

Why now: Windows-side debugging (Claude Desktop MCP config) is blocked
on the user; macOS work unblocks local end-to-end validation while
that resolves. After Phase 0e2'/0e3' and Phase 0f' ship, the user can
demo AgentMark Desktop driving Pages / Excel-for-Mac / Numbers from
Claude Desktop on their development machine -- no Windows VM
dependency for the demo.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the real AXAPI-driven methods to the macOS bridge. Builds clean
on Apple Silicon arm64. Validated end-to-end against a live Mac:
list_windows returned 22 real windows across 13 apps (Finder, Cursor,
Chrome, VS Code, Claude Desktop, Teams, Outlook, Parallels, Terminal,
etc.); capture against Finder's Copy progress dialog returned 11
elements at depth 2 with correct role mapping (window -> pane ->
image / static_text / progress_bar / button) and live property values
(progress=0.132, copy status text, traffic-light buttons).

Mirrors the Windows bridge UiaCapturer pattern: defensive everywhere
(every AXAPI read wrapped, swallows mid-walk failures), depth +
element-count + deadline caps, per-capture session cache for
element_id -> AXUIElement lookup (Phase 0e3' execute will resolve
against this), same JSON wire format the Node-side body-builder
already consumes.

## New module Sources/AgentMarkBridgeMacos/

  Axapi.swift          -- AccessibilityPermission probe
                          (AXIsProcessTrusted), AxElement wrapper
                          around AXUIElement with defensive attribute
                          readers (string/bool/value/children/bounds),
                          content-derived stableId() fallback (macOS
                          has no AutomationId equivalent so we hash
                          role+title+value+position), RoleMapper for
                          AXAPI roles -> normalised DesktopRole vocab
                          (AXButton -> button, AXTextField subrole
                          AXSecureTextField -> password_input,
                          AXScrollArea -> pane, AXOutline -> tree,
                          AXOutlineRow -> tree_item, etc.)

  AxapiCapturer.swift  -- ListWindows() walks
                          NSWorkspace.runningApplications, filters to
                          .regular activation policy, AXUIElementCreate-
                          Application(pid) -> kAXWindowsAttribute,
                          marks hasFocus by cross-referencing
                          NSWorkspace.frontmostApplication.
                          Capture(req) resolves target by windowId
                          (axapi:<pid>:<index>), processId,
                          processName (substring, .app-tolerant),
                          windowTitle, or falls back to focused window.
                          Walks tree depth-first with depth+count+
                          deadline caps, extracts value via the
                          per-role correct attribute, screen bounds via
                          kAXPosition + kAXSize, focused-element id via
                          kAXFocusedUIElementAttribute + identity
                          comparison against captured handles.
                          requirePermission() surfaces a clear
                          accessibilityNotGranted error (-32020) when
                          the parent process hasn't been granted
                          Accessibility in System Settings.

## Dispatcher.swift

  Routes list_windows + capture through AxapiCapturer (lazy init so
  ping-only smoke tests don't pay AXAPI startup cost). Adds
  accessibilityGranted bool to capabilities response so clients can
  detect permission issues without waiting for first capture.

## Validation

Built locally on Apple Silicon arm64; tested against real Mac:

  list_windows -> 22 windows across 13 real apps (Finder, Cursor x3,
  Chrome x5 including the actual agentmark PR view, VS Code with
  claude_desktop_config.json open, Claude Desktop, Teams, Messages,
  Outlook, Parallels Desktop, Terminal, etc.). hasFocus correctly
  marked on the active Cursor window.

  capture processName=Finder -> Copy progress dialog with 11 elements,
  depth 2. Live progress value (0.132 -> 13%) extracted. Real screen
  coordinates. Correct role mapping. Traffic-light buttons surfaced
  with bounds at x=78/98/118 (close/minimize/maximize).

  Same JSON wire format as the Windows bridge -- the Node-side body
  builder we built for the FixtureBackend renders this transparently
  once Phase 0f' (MacosAxapiBackend) lands.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the `execute` JSON-RPC method to the macOS bridge. Routes
element-targeted actions through the right AXAPI pattern. Mirrors the
Windows bridge's UiaCapturer.Execute, same JSON shape on the wire.

## Action types

  - click       -> AXUIElementPerformAction(AXPress) with AXToggle /
                   AXPick / AXConfirm / AXShowMenu fallbacks
  - type        -> AXUIElementSetAttributeValue(kAXValueAttribute) with
                   focus+keystrokes (CGEvent) fallback when AXValue is
                   not writable; supports clearFirst (Cmd+A then Delete)
  - select      -> AXPress on a selection item, or kAXValueAttribute
                   set directly for popup buttons
  - check       -> Toggle until state matches (capped at 3 iterations)
                   via AXPress or AXToggle, mapping into kAXValue bool
  - expand      -> kAXExpandedAttribute set or AXShowMenu / AXPress
                   fallback for outline rows / disclosure triangles
  - focus       -> kAXFocusedAttribute = true
  - scroll_to   -> AXUIElementPerformAction(AXScrollToVisible) with
                   focus fallback
  - key         -> CGEvent keyboard simulation. Modifier-aware (ctrl /
                   option / shift / cmd). Named keys: enter, tab, esc,
                   space, delete, arrows, pgup/pgdn, home/end, F1..F12.
                   Unknown names fall through to literal text typing.

## Element lookup

Capture stashes each walked AXUIElement in a per-session
[element_id -> AXUIElement] map. Execute resolves the request's
elementId against that map. Errors are specific:
  - "No capture session active" -> caller must call capture first
  - "Unknown element_id" -> stale binding, re-capture
  - "Accessibility permission not granted" -> system settings error

## CGEvent typing strategy

For the keystroke fallback we use
CGEventKeyboardSetUnicodeString rather than per-character keyCode
lookup. Handles Latin + accented + most BMP characters without
maintaining a layout-specific table. Combined with AXSetValue as the
primary path, this covers TextEdit / Mail / Cursor / VS Code /
Pages / Numbers cleanly.

## Live demo (scripts/textedit-demo.py)

Python harness that:
  1. Launches TextEdit with a new document via osascript
  2. list_windows -> find TextEdit
  3. capture     -> locate the text_area element (the AXTextArea
                    inside the AXScrollArea inside the AXSplitGroup
                    inside the AXWindow)
  4. execute type -> "Hello from AgentMark Desktop -- macOS bridge
                    phase 0e3'"
  5. capture     -> assert the typed text is in element.value

Tested live on Apple Silicon arm64. PASSED first run:

  Step 2 -- list_windows: found TextEdit window axapi:21232:0
  Step 3 -- capture: 47 elements, depth 3, editor id="First Text View"
  Step 4 -- execute type: ok=true, newValue echoes the sent text
  Step 5 -- re-capture: editor value matches "Hello from AgentMark..."
  LIVE DEMO PASSED

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@rrader26
rrader26 merged commit 777a463 into main May 11, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants