feat(bridge-macos): Phase 0e2' + 0e3' — list_windows, capture, execute (real AXAPI driving real macOS apps) - #17
Merged
Conversation
Adds the macOS-side sidecar process for AgentMark Desktop, mirroring
the Windows bridge architecture exactly. Same stdio JSON-RPC 2.0
protocol; same DesktopCaptureBackend contract on the Node side. Uses
Apple's Accessibility framework (AXAPI) as the capture / execute
substrate when Phase 0e2'/0e3' land.
This first commit ships the protocol substrate and two no-AXAPI
methods so we can validate the cross-language pipeline before adding
the AXAPI walking + action dispatch logic:
- ping -> { pong, version, arch, processId }
- capabilities -> supported methods + AXAPI provider info
Built natively on Apple Silicon (arm64). Smoke test runs locally on
the developer Mac in seconds.
Layout (apps/agent-runner/bridges/macos/):
Package.swift -- SPM manifest (macOS 13+)
Sources/AgentMarkBridgeMacos/
main.swift -- entry + stdin read loop +
BOM stripping + dispatch
JsonRpc.swift -- request/response types +
JSON-RPC 2.0 envelope
encoder/decoder + error
code catalog
Dispatcher.swift -- method routing (mirrors
the C# bridge's
RpcDispatcher class)
scripts/smoke-test.sh -- shell harness driving the
binary via stdin; grep-
based assertions, no jq
dependency
README.md -- build + protocol docs
The protocol -- request envelope, error codes (parseError -32700 ..
internalError -32603 plus bridge-specific 32010+), stderr discipline,
BOM stripping -- is byte-identical to the Windows bridge. This is
deliberate: the Node-side `MacosAxapiBackend` (Phase 0f') will be a
near-exact mirror of `WindowsUiaBackend`, sharing 90% of the code
through the abstract DesktopCaptureBackend interface.
.gitignore extended for Swift Package Manager output (.build/,
.swiftpm/, Packages/) alongside the existing .NET ignores.
Why now: Windows-side debugging (Claude Desktop MCP config) is blocked
on the user; macOS work unblocks local end-to-end validation while
that resolves. After Phase 0e2'/0e3' and Phase 0f' ship, the user can
demo AgentMark Desktop driving Pages / Excel-for-Mac / Numbers from
Claude Desktop on their development machine -- no Windows VM
dependency for the demo.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the real AXAPI-driven methods to the macOS bridge. Builds clean
on Apple Silicon arm64. Validated end-to-end against a live Mac:
list_windows returned 22 real windows across 13 apps (Finder, Cursor,
Chrome, VS Code, Claude Desktop, Teams, Outlook, Parallels, Terminal,
etc.); capture against Finder's Copy progress dialog returned 11
elements at depth 2 with correct role mapping (window -> pane ->
image / static_text / progress_bar / button) and live property values
(progress=0.132, copy status text, traffic-light buttons).
Mirrors the Windows bridge UiaCapturer pattern: defensive everywhere
(every AXAPI read wrapped, swallows mid-walk failures), depth +
element-count + deadline caps, per-capture session cache for
element_id -> AXUIElement lookup (Phase 0e3' execute will resolve
against this), same JSON wire format the Node-side body-builder
already consumes.
## New module Sources/AgentMarkBridgeMacos/
Axapi.swift -- AccessibilityPermission probe
(AXIsProcessTrusted), AxElement wrapper
around AXUIElement with defensive attribute
readers (string/bool/value/children/bounds),
content-derived stableId() fallback (macOS
has no AutomationId equivalent so we hash
role+title+value+position), RoleMapper for
AXAPI roles -> normalised DesktopRole vocab
(AXButton -> button, AXTextField subrole
AXSecureTextField -> password_input,
AXScrollArea -> pane, AXOutline -> tree,
AXOutlineRow -> tree_item, etc.)
AxapiCapturer.swift -- ListWindows() walks
NSWorkspace.runningApplications, filters to
.regular activation policy, AXUIElementCreate-
Application(pid) -> kAXWindowsAttribute,
marks hasFocus by cross-referencing
NSWorkspace.frontmostApplication.
Capture(req) resolves target by windowId
(axapi:<pid>:<index>), processId,
processName (substring, .app-tolerant),
windowTitle, or falls back to focused window.
Walks tree depth-first with depth+count+
deadline caps, extracts value via the
per-role correct attribute, screen bounds via
kAXPosition + kAXSize, focused-element id via
kAXFocusedUIElementAttribute + identity
comparison against captured handles.
requirePermission() surfaces a clear
accessibilityNotGranted error (-32020) when
the parent process hasn't been granted
Accessibility in System Settings.
## Dispatcher.swift
Routes list_windows + capture through AxapiCapturer (lazy init so
ping-only smoke tests don't pay AXAPI startup cost). Adds
accessibilityGranted bool to capabilities response so clients can
detect permission issues without waiting for first capture.
## Validation
Built locally on Apple Silicon arm64; tested against real Mac:
list_windows -> 22 windows across 13 real apps (Finder, Cursor x3,
Chrome x5 including the actual agentmark PR view, VS Code with
claude_desktop_config.json open, Claude Desktop, Teams, Messages,
Outlook, Parallels Desktop, Terminal, etc.). hasFocus correctly
marked on the active Cursor window.
capture processName=Finder -> Copy progress dialog with 11 elements,
depth 2. Live progress value (0.132 -> 13%) extracted. Real screen
coordinates. Correct role mapping. Traffic-light buttons surfaced
with bounds at x=78/98/118 (close/minimize/maximize).
Same JSON wire format as the Windows bridge -- the Node-side body
builder we built for the FixtureBackend renders this transparently
once Phase 0f' (MacosAxapiBackend) lands.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the `execute` JSON-RPC method to the macOS bridge. Routes
element-targeted actions through the right AXAPI pattern. Mirrors the
Windows bridge's UiaCapturer.Execute, same JSON shape on the wire.
## Action types
- click -> AXUIElementPerformAction(AXPress) with AXToggle /
AXPick / AXConfirm / AXShowMenu fallbacks
- type -> AXUIElementSetAttributeValue(kAXValueAttribute) with
focus+keystrokes (CGEvent) fallback when AXValue is
not writable; supports clearFirst (Cmd+A then Delete)
- select -> AXPress on a selection item, or kAXValueAttribute
set directly for popup buttons
- check -> Toggle until state matches (capped at 3 iterations)
via AXPress or AXToggle, mapping into kAXValue bool
- expand -> kAXExpandedAttribute set or AXShowMenu / AXPress
fallback for outline rows / disclosure triangles
- focus -> kAXFocusedAttribute = true
- scroll_to -> AXUIElementPerformAction(AXScrollToVisible) with
focus fallback
- key -> CGEvent keyboard simulation. Modifier-aware (ctrl /
option / shift / cmd). Named keys: enter, tab, esc,
space, delete, arrows, pgup/pgdn, home/end, F1..F12.
Unknown names fall through to literal text typing.
## Element lookup
Capture stashes each walked AXUIElement in a per-session
[element_id -> AXUIElement] map. Execute resolves the request's
elementId against that map. Errors are specific:
- "No capture session active" -> caller must call capture first
- "Unknown element_id" -> stale binding, re-capture
- "Accessibility permission not granted" -> system settings error
## CGEvent typing strategy
For the keystroke fallback we use
CGEventKeyboardSetUnicodeString rather than per-character keyCode
lookup. Handles Latin + accented + most BMP characters without
maintaining a layout-specific table. Combined with AXSetValue as the
primary path, this covers TextEdit / Mail / Cursor / VS Code /
Pages / Numbers cleanly.
## Live demo (scripts/textedit-demo.py)
Python harness that:
1. Launches TextEdit with a new document via osascript
2. list_windows -> find TextEdit
3. capture -> locate the text_area element (the AXTextArea
inside the AXScrollArea inside the AXSplitGroup
inside the AXWindow)
4. execute type -> "Hello from AgentMark Desktop -- macOS bridge
phase 0e3'"
5. capture -> assert the typed text is in element.value
Tested live on Apple Silicon arm64. PASSED first run:
Step 2 -- list_windows: found TextEdit window axapi:21232:0
Step 3 -- capture: 47 elements, depth 3, editor id="First Text View"
Step 4 -- execute type: ok=true, newValue echoes the sent text
Step 5 -- re-capture: editor value matches "Hello from AgentMark..."
LIVE DEMO PASSED
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Brings the macOS bridge from "scaffolded" to "actually drives macOS
software." Includes
list_windows,capture, andexecutevia theAccessibility framework (AXAPI). Live-demo PASSED against real
TextEdit on Apple Silicon arm64 — bridge captured the editor
element, typed a message via
AXUIElementSetAttributeValue,re-captured to verify the text landed.
Stacks on PR #16 (the macOS scaffold). Same architecture as the
Windows PR #13 — by design, the Node-side
MacosAxapiBackend(Phase0f') will be a near-exact mirror of
WindowsUiaBackend(PR #14).Validation
End-to-end demo on a real Mac:
A separate
list_windows-only test enumerated 22 real windows across13 apps (Finder x3, Cursor x3, Chrome x5, VS Code, Claude Desktop,
Teams, Messages, Outlook, Parallels, Terminal, etc.) with focused
window correctly marked.
What's in this PR
Sources/AgentMarkBridgeMacos/Axapi.swiftAccessibilityPermission.isGranted— probesAXIsProcessTrusted()AxElement— defensive wrapper aroundAXUIElement. Everyattribute read returns nil on AXAPI errors so a single stale
element doesn't abort the capture.
RoleMapper— AXAPI roles → normalisedDesktopRolevocab.Handles subroles for password fields, search fields, dialogs.
Unknown →
"other".Sources/AgentMarkBridgeMacos/AxapiCapturer.swiftlistWindows()— walksNSWorkspace.runningApplications, filtersto
.regularpolicy, creates per-app AX refs, enumerateskAXWindowsAttribute.hasFocusmarked by cross-referencingNSWorkspace.frontmostApplication.capture(req)— resolves target bywindowId/processId/processName(substring, .app-tolerant) /windowTitle, or falls back to focused window. Walks treedepth-first with depth + element-count + deadline caps. Stashes
every walked AX element in a per-session lookup map so
executecan find it later.
execute(req)— routes 8 action types through the right AXAPIpattern (
AXPress/AXSetValue/AXScrollToVisible/AXShowMenu/ etc.) with sensible fallbacks (keyboard simulationvia
CGEvent.keyboardSetUnicodeStringfor text typing whenAXValueisn't writable; modifier-awareCGEventforkeyactions).
requirePermission()surfaces a clearaccessibilityNotGrantedJSON-RPC error (code -32020) pointing the user at System Settings.
Sources/AgentMarkBridgeMacos/Dispatcher.swiftRoutes
list_windows/capture/executetoAxapiCapturer(lazy init). Adds
accessibilityGrantedtocapabilitiesresponseso clients can detect permission state without waiting for first
capture.
scripts/textedit-demo.pyPython end-to-end demo. Same shape as the Windows
notepad-demo.ps1: opens TextEdit, drives capture → execute →re-capture, asserts the typed text in element.value.
Protocol compatibility with the Windows bridge
Same JSON shape on the wire. Same error code numbering. The
Node-side
MacosAxapiBackend(Phase 0f', future PR) shares 90% ofits code with
WindowsUiaBackend(PR #14) — only difference iswhich binary it spawns and where it looks for it.
Accessibility permission gotcha
macOS guards AXAPI behind System Settings → Privacy & Security →
Accessibility. The parent process that launches the bridge must
be granted:
Without permission the bridge returns
{ "error": { "code": -32020, "message": "Accessibility permission not granted..." } }on the first
list_windows/capture/executecall.Stack order
MacosAxapiBackend(mirror ofWindowsUiaBackendfrom PR feat(desktop): WindowsUiaBackend — Node-side bridge client (Phase 0f) #14)no PowerShell scripts, just "Capture TextEdit and type into it."
🤖 Generated with Claude Code