feat(desktop): MacosAxapiBackend — Node-side bridge client for real macOS AXAPI (Phase 0f') - #18
Merged
Merged
Conversation
Adds the macOS-side sidecar process for AgentMark Desktop, mirroring
the Windows bridge architecture exactly. Same stdio JSON-RPC 2.0
protocol; same DesktopCaptureBackend contract on the Node side. Uses
Apple's Accessibility framework (AXAPI) as the capture / execute
substrate when Phase 0e2'/0e3' land.
This first commit ships the protocol substrate and two no-AXAPI
methods so we can validate the cross-language pipeline before adding
the AXAPI walking + action dispatch logic:
- ping -> { pong, version, arch, processId }
- capabilities -> supported methods + AXAPI provider info
Built natively on Apple Silicon (arm64). Smoke test runs locally on
the developer Mac in seconds.
Layout (apps/agent-runner/bridges/macos/):
Package.swift -- SPM manifest (macOS 13+)
Sources/AgentMarkBridgeMacos/
main.swift -- entry + stdin read loop +
BOM stripping + dispatch
JsonRpc.swift -- request/response types +
JSON-RPC 2.0 envelope
encoder/decoder + error
code catalog
Dispatcher.swift -- method routing (mirrors
the C# bridge's
RpcDispatcher class)
scripts/smoke-test.sh -- shell harness driving the
binary via stdin; grep-
based assertions, no jq
dependency
README.md -- build + protocol docs
The protocol -- request envelope, error codes (parseError -32700 ..
internalError -32603 plus bridge-specific 32010+), stderr discipline,
BOM stripping -- is byte-identical to the Windows bridge. This is
deliberate: the Node-side `MacosAxapiBackend` (Phase 0f') will be a
near-exact mirror of `WindowsUiaBackend`, sharing 90% of the code
through the abstract DesktopCaptureBackend interface.
.gitignore extended for Swift Package Manager output (.build/,
.swiftpm/, Packages/) alongside the existing .NET ignores.
Why now: Windows-side debugging (Claude Desktop MCP config) is blocked
on the user; macOS work unblocks local end-to-end validation while
that resolves. After Phase 0e2'/0e3' and Phase 0f' ship, the user can
demo AgentMark Desktop driving Pages / Excel-for-Mac / Numbers from
Claude Desktop on their development machine -- no Windows VM
dependency for the demo.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the real AXAPI-driven methods to the macOS bridge. Builds clean
on Apple Silicon arm64. Validated end-to-end against a live Mac:
list_windows returned 22 real windows across 13 apps (Finder, Cursor,
Chrome, VS Code, Claude Desktop, Teams, Outlook, Parallels, Terminal,
etc.); capture against Finder's Copy progress dialog returned 11
elements at depth 2 with correct role mapping (window -> pane ->
image / static_text / progress_bar / button) and live property values
(progress=0.132, copy status text, traffic-light buttons).
Mirrors the Windows bridge UiaCapturer pattern: defensive everywhere
(every AXAPI read wrapped, swallows mid-walk failures), depth +
element-count + deadline caps, per-capture session cache for
element_id -> AXUIElement lookup (Phase 0e3' execute will resolve
against this), same JSON wire format the Node-side body-builder
already consumes.
## New module Sources/AgentMarkBridgeMacos/
Axapi.swift -- AccessibilityPermission probe
(AXIsProcessTrusted), AxElement wrapper
around AXUIElement with defensive attribute
readers (string/bool/value/children/bounds),
content-derived stableId() fallback (macOS
has no AutomationId equivalent so we hash
role+title+value+position), RoleMapper for
AXAPI roles -> normalised DesktopRole vocab
(AXButton -> button, AXTextField subrole
AXSecureTextField -> password_input,
AXScrollArea -> pane, AXOutline -> tree,
AXOutlineRow -> tree_item, etc.)
AxapiCapturer.swift -- ListWindows() walks
NSWorkspace.runningApplications, filters to
.regular activation policy, AXUIElementCreate-
Application(pid) -> kAXWindowsAttribute,
marks hasFocus by cross-referencing
NSWorkspace.frontmostApplication.
Capture(req) resolves target by windowId
(axapi:<pid>:<index>), processId,
processName (substring, .app-tolerant),
windowTitle, or falls back to focused window.
Walks tree depth-first with depth+count+
deadline caps, extracts value via the
per-role correct attribute, screen bounds via
kAXPosition + kAXSize, focused-element id via
kAXFocusedUIElementAttribute + identity
comparison against captured handles.
requirePermission() surfaces a clear
accessibilityNotGranted error (-32020) when
the parent process hasn't been granted
Accessibility in System Settings.
## Dispatcher.swift
Routes list_windows + capture through AxapiCapturer (lazy init so
ping-only smoke tests don't pay AXAPI startup cost). Adds
accessibilityGranted bool to capabilities response so clients can
detect permission issues without waiting for first capture.
## Validation
Built locally on Apple Silicon arm64; tested against real Mac:
list_windows -> 22 windows across 13 real apps (Finder, Cursor x3,
Chrome x5 including the actual agentmark PR view, VS Code with
claude_desktop_config.json open, Claude Desktop, Teams, Messages,
Outlook, Parallels Desktop, Terminal, etc.). hasFocus correctly
marked on the active Cursor window.
capture processName=Finder -> Copy progress dialog with 11 elements,
depth 2. Live progress value (0.132 -> 13%) extracted. Real screen
coordinates. Correct role mapping. Traffic-light buttons surfaced
with bounds at x=78/98/118 (close/minimize/maximize).
Same JSON wire format as the Windows bridge -- the Node-side body
builder we built for the FixtureBackend renders this transparently
once Phase 0f' (MacosAxapiBackend) lands.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the `execute` JSON-RPC method to the macOS bridge. Routes
element-targeted actions through the right AXAPI pattern. Mirrors the
Windows bridge's UiaCapturer.Execute, same JSON shape on the wire.
## Action types
- click -> AXUIElementPerformAction(AXPress) with AXToggle /
AXPick / AXConfirm / AXShowMenu fallbacks
- type -> AXUIElementSetAttributeValue(kAXValueAttribute) with
focus+keystrokes (CGEvent) fallback when AXValue is
not writable; supports clearFirst (Cmd+A then Delete)
- select -> AXPress on a selection item, or kAXValueAttribute
set directly for popup buttons
- check -> Toggle until state matches (capped at 3 iterations)
via AXPress or AXToggle, mapping into kAXValue bool
- expand -> kAXExpandedAttribute set or AXShowMenu / AXPress
fallback for outline rows / disclosure triangles
- focus -> kAXFocusedAttribute = true
- scroll_to -> AXUIElementPerformAction(AXScrollToVisible) with
focus fallback
- key -> CGEvent keyboard simulation. Modifier-aware (ctrl /
option / shift / cmd). Named keys: enter, tab, esc,
space, delete, arrows, pgup/pgdn, home/end, F1..F12.
Unknown names fall through to literal text typing.
## Element lookup
Capture stashes each walked AXUIElement in a per-session
[element_id -> AXUIElement] map. Execute resolves the request's
elementId against that map. Errors are specific:
- "No capture session active" -> caller must call capture first
- "Unknown element_id" -> stale binding, re-capture
- "Accessibility permission not granted" -> system settings error
## CGEvent typing strategy
For the keystroke fallback we use
CGEventKeyboardSetUnicodeString rather than per-character keyCode
lookup. Handles Latin + accented + most BMP characters without
maintaining a layout-specific table. Combined with AXSetValue as the
primary path, this covers TextEdit / Mail / Cursor / VS Code /
Pages / Numbers cleanly.
## Live demo (scripts/textedit-demo.py)
Python harness that:
1. Launches TextEdit with a new document via osascript
2. list_windows -> find TextEdit
3. capture -> locate the text_area element (the AXTextArea
inside the AXScrollArea inside the AXSplitGroup
inside the AXWindow)
4. execute type -> "Hello from AgentMark Desktop -- macOS bridge
phase 0e3'"
5. capture -> assert the typed text is in element.value
Tested live on Apple Silicon arm64. PASSED first run:
Step 2 -- list_windows: found TextEdit window axapi:21232:0
Step 3 -- capture: 47 elements, depth 3, editor id="First Text View"
Step 4 -- execute type: ok=true, newValue echoes the sent text
Step 5 -- re-capture: editor value matches "Hello from AgentMark..."
LIVE DEMO PASSED
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… AXAPI Phase 0f'. Mirrors WindowsUiaBackend (PR #14) for the macOS side. Once this merges, Claude Desktop / Cursor / any MCP client on macOS can drive real Mac software through the same agentmark_desktop_* MCP tools, no per-AI-tool integration work. ## What it does `MacosAxapiBackend` implements the `DesktopCaptureBackend` interface: - Lazily spawns agentmark-bridge-macos as a child process - Speaks JSON-RPC 2.0 over stdio (one JSON message per line) - Performs a ping handshake on first use; tears down + retries on handshake failure - Correlates concurrent calls by request id (Map<id, PendingCall>); each call has its own timeout - Forwards bridge stderr to a structured logger - Closes cleanly: end stdin, wait 2s, SIGKILL if still alive - Re-throws bridge error responses as typed JS errors Wire-format mapping is identical to WindowsUiaBackend (the two bridges speak the same JSON protocol by design). ## Bridge resolution Search order: 1. options.bridgePath (explicit) 2. AGENTMARK_BRIDGE_PATH env var (if file exists) 3. Walk up from __dirname probing common SPM output paths: apps/agent-runner/bridges/macos/.build/{release,debug}/ apps/agent-runner/bridges/macos/.build/arm64-apple-macosx/... apps/agent-runner/bridges/macos/.build/x86_64-apple-macosx/... 4. Throws with the swift-build command and env-var instruction ## MCP dispatcher wiring `agentmark_desktop_open` with `backend: "macos_axapi"` now actually spawns the bridge (instead of returning "not yet bundled"). The optional `bridge_path` arg lets callers override auto-detection. On non-macOS hosts it returns a clear error pointing to fixture for testing. ## Tests `test/desktop/macos-axapi-backend.test.ts` (9 tests) reuse the same fake-bridge.cjs fixture as the Windows backend tests -- the wire protocol is byte-identical between bridges, so one fake substitutes for both: - construct refuses on non-macOS without allowNonMac - backend name is 'macos_axapi' - capture maps response to DesktopCapture shape - capture forwards target fields to bridge - execute type maps newValue -> new_value - same bridge process is reused across calls - handshake failure surfaces with a clear error - close rejects pending + future calls - missing bridge path defers until first call Plus the existing dispatcher test `agentmark_desktop_open with macos_axapi returns "not yet bundled"` updated to reflect the new behaviour (refuses on non-macOS with a clear "requires macOS" message; real spawn path tested separately). Full suite: 313 passed, 10 skipped, 0 failed. ## Stacking Goes on top of PR #16 (macOS scaffold) + PR #17 (macOS capture + execute). Order to merge: 16 -> 17 -> this PR. Once all three land, the macOS path is complete and Claude Desktop on macOS can drive real apps like Pages / Numbers / Excel-for-Mac / TextEdit through the protocol. ## Code-sharing note WindowsUiaBackend + MacosAxapiBackend share ~90% of their code (process spawn, JSON-RPC lifecycle, request correlation, close logic). A follow-up refactor can extract a `SubprocessBridgeBackend` base class. Keeping them parallel for now so this PR is reviewable in isolation and the diff is easy to read. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacks on PR #16 (macOS scaffold) + PR #17 (macOS capture/execute). Closes the four-PR cross-platform stack: once these merge, Claude Desktop / Cursor / Codex / any MCP client on macOS can drive real Mac software through the same `agentmark_desktop_*` MCP tools, with zero per-AI-tool integration.
Mirror of `WindowsUiaBackend` from the recently-merged PR #14. Same JSON-RPC protocol, same spawn/handshake/correlate-by-id pattern, same close lifecycle. The two backends share ~90% of their code; a future cleanup pass can extract a shared `SubprocessBridgeBackend` base.
What it does
`MacosAxapiBackend` implements the `DesktopCaptureBackend` interface:
Wire-format mapping is identical to `WindowsUiaBackend` (bridges share the same JSON protocol by design).
Bridge resolution
Search order:
MCP dispatcher wiring
`agentmark_desktop_open` with `backend: "macos_axapi"` now actually spawns the bridge (instead of returning "not yet bundled"). The optional `bridge_path` arg lets callers override auto-detection. On non-macOS hosts it returns a clear error pointing to `fixture` for testing.
Tests
`test/desktop/macos-axapi-backend.test.ts` (9 tests) reuse the same `fake-bridge.cjs` fixture as the Windows tests — the wire protocol is byte-identical between bridges, so one fake substitutes for both:
Plus the existing dispatcher test `agentmark_desktop_open with macos_axapi returns "not yet bundled"` updated to reflect the new behaviour (refuses on non-macOS with a clear "requires macOS" message; real spawn path tested separately).
`npm test`: 313 passed, 10 skipped, 0 failed.
After this + PR #16 + PR #17 merge
The full local-test path on macOS:
```jsonc
// %APPDATA%/Claude/claude_desktop_config.json (or Cursor / Codex equivalent)
{
"mcpServers": {
"agentmark": {
"command": "node",
"args": ["/Users/you/Dev/agentmark/dist/src/mcp/cli.js"]
}
}
}
```
Then ask the AI:
Real software, real AXAPI, real round-trip.
Code-sharing note
`WindowsUiaBackend` + `MacosAxapiBackend` share ~90% of their code (process spawn, JSON-RPC lifecycle, request correlation, close logic). A follow-up refactor can extract a `SubprocessBridgeBackend` base class. Keeping them parallel for now so this PR is reviewable in isolation and the diff is easy to read.
🤖 Generated with Claude Code