Skip to content

feat(desktop): MacosAxapiBackend — Node-side bridge client for real macOS AXAPI (Phase 0f') - #18

Merged
rrader26 merged 4 commits into
mainfrom
feat/desktop-macos-axapi-backend
May 11, 2026
Merged

feat(desktop): MacosAxapiBackend — Node-side bridge client for real macOS AXAPI (Phase 0f')#18
rrader26 merged 4 commits into
mainfrom
feat/desktop-macos-axapi-backend

Conversation

@rrader26

Copy link
Copy Markdown
Contributor

Summary

Stacks on PR #16 (macOS scaffold) + PR #17 (macOS capture/execute). Closes the four-PR cross-platform stack: once these merge, Claude Desktop / Cursor / Codex / any MCP client on macOS can drive real Mac software through the same `agentmark_desktop_*` MCP tools, with zero per-AI-tool integration.

Mirror of `WindowsUiaBackend` from the recently-merged PR #14. Same JSON-RPC protocol, same spawn/handshake/correlate-by-id pattern, same close lifecycle. The two backends share ~90% of their code; a future cleanup pass can extract a shared `SubprocessBridgeBackend` base.

What it does

`MacosAxapiBackend` implements the `DesktopCaptureBackend` interface:

Wire-format mapping is identical to `WindowsUiaBackend` (bridges share the same JSON protocol by design).

Bridge resolution

Search order:

  1. `options.bridgePath` (explicit)
  2. `AGENTMARK_BRIDGE_PATH` env var
  3. Walk up from `__dirname` probing common SPM output paths:
    • `apps/agent-runner/bridges/macos/.build/{release,debug}/`
    • Architecture-specific subpaths (`.build/arm64-apple-macosx/...`, `.build/x86_64-apple-macosx/...`)
  4. Throws with the `swift build` command + env-var instruction

MCP dispatcher wiring

`agentmark_desktop_open` with `backend: "macos_axapi"` now actually spawns the bridge (instead of returning "not yet bundled"). The optional `bridge_path` arg lets callers override auto-detection. On non-macOS hosts it returns a clear error pointing to `fixture` for testing.

Tests

`test/desktop/macos-axapi-backend.test.ts` (9 tests) reuse the same `fake-bridge.cjs` fixture as the Windows tests — the wire protocol is byte-identical between bridges, so one fake substitutes for both:

  • construct refuses on non-macOS without `allowNonMac`
  • backend name is `'macos_axapi'`
  • capture maps response to `DesktopCapture` shape
  • capture forwards target fields to bridge
  • execute `type` maps `newValue` → `new_value`
  • same bridge process is reused across calls
  • handshake failure surfaces with a clear error
  • close rejects pending + future calls
  • missing bridge path defers until first call

Plus the existing dispatcher test `agentmark_desktop_open with macos_axapi returns "not yet bundled"` updated to reflect the new behaviour (refuses on non-macOS with a clear "requires macOS" message; real spawn path tested separately).

`npm test`: 313 passed, 10 skipped, 0 failed.

After this + PR #16 + PR #17 merge

The full local-test path on macOS:

```jsonc
// %APPDATA%/Claude/claude_desktop_config.json (or Cursor / Codex equivalent)
{
"mcpServers": {
"agentmark": {
"command": "node",
"args": ["/Users/you/Dev/agentmark/dist/src/mcp/cli.js"]
}
}
}
```

Then ask the AI:

  • "Open a desktop session with backend macos_axapi"
  • "List my windows"
  • "Capture TextEdit"
  • "Type 'Hello from Claude' into the editor"

Real software, real AXAPI, real round-trip.

Code-sharing note

`WindowsUiaBackend` + `MacosAxapiBackend` share ~90% of their code (process spawn, JSON-RPC lifecycle, request correlation, close logic). A follow-up refactor can extract a `SubprocessBridgeBackend` base class. Keeping them parallel for now so this PR is reviewable in isolation and the diff is easy to read.

🤖 Generated with Claude Code

rrader26-sys and others added 4 commits May 11, 2026 16:30
Adds the macOS-side sidecar process for AgentMark Desktop, mirroring
the Windows bridge architecture exactly. Same stdio JSON-RPC 2.0
protocol; same DesktopCaptureBackend contract on the Node side. Uses
Apple's Accessibility framework (AXAPI) as the capture / execute
substrate when Phase 0e2'/0e3' land.

This first commit ships the protocol substrate and two no-AXAPI
methods so we can validate the cross-language pipeline before adding
the AXAPI walking + action dispatch logic:

  - ping         -> { pong, version, arch, processId }
  - capabilities -> supported methods + AXAPI provider info

Built natively on Apple Silicon (arm64). Smoke test runs locally on
the developer Mac in seconds.

Layout (apps/agent-runner/bridges/macos/):

  Package.swift                            -- SPM manifest (macOS 13+)
  Sources/AgentMarkBridgeMacos/
    main.swift                             -- entry + stdin read loop +
                                              BOM stripping + dispatch
    JsonRpc.swift                          -- request/response types +
                                              JSON-RPC 2.0 envelope
                                              encoder/decoder + error
                                              code catalog
    Dispatcher.swift                       -- method routing (mirrors
                                              the C# bridge's
                                              RpcDispatcher class)
  scripts/smoke-test.sh                    -- shell harness driving the
                                              binary via stdin; grep-
                                              based assertions, no jq
                                              dependency
  README.md                                -- build + protocol docs

The protocol -- request envelope, error codes (parseError -32700 ..
internalError -32603 plus bridge-specific 32010+), stderr discipline,
BOM stripping -- is byte-identical to the Windows bridge. This is
deliberate: the Node-side `MacosAxapiBackend` (Phase 0f') will be a
near-exact mirror of `WindowsUiaBackend`, sharing 90% of the code
through the abstract DesktopCaptureBackend interface.

.gitignore extended for Swift Package Manager output (.build/,
.swiftpm/, Packages/) alongside the existing .NET ignores.

Why now: Windows-side debugging (Claude Desktop MCP config) is blocked
on the user; macOS work unblocks local end-to-end validation while
that resolves. After Phase 0e2'/0e3' and Phase 0f' ship, the user can
demo AgentMark Desktop driving Pages / Excel-for-Mac / Numbers from
Claude Desktop on their development machine -- no Windows VM
dependency for the demo.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the real AXAPI-driven methods to the macOS bridge. Builds clean
on Apple Silicon arm64. Validated end-to-end against a live Mac:
list_windows returned 22 real windows across 13 apps (Finder, Cursor,
Chrome, VS Code, Claude Desktop, Teams, Outlook, Parallels, Terminal,
etc.); capture against Finder's Copy progress dialog returned 11
elements at depth 2 with correct role mapping (window -> pane ->
image / static_text / progress_bar / button) and live property values
(progress=0.132, copy status text, traffic-light buttons).

Mirrors the Windows bridge UiaCapturer pattern: defensive everywhere
(every AXAPI read wrapped, swallows mid-walk failures), depth +
element-count + deadline caps, per-capture session cache for
element_id -> AXUIElement lookup (Phase 0e3' execute will resolve
against this), same JSON wire format the Node-side body-builder
already consumes.

## New module Sources/AgentMarkBridgeMacos/

  Axapi.swift          -- AccessibilityPermission probe
                          (AXIsProcessTrusted), AxElement wrapper
                          around AXUIElement with defensive attribute
                          readers (string/bool/value/children/bounds),
                          content-derived stableId() fallback (macOS
                          has no AutomationId equivalent so we hash
                          role+title+value+position), RoleMapper for
                          AXAPI roles -> normalised DesktopRole vocab
                          (AXButton -> button, AXTextField subrole
                          AXSecureTextField -> password_input,
                          AXScrollArea -> pane, AXOutline -> tree,
                          AXOutlineRow -> tree_item, etc.)

  AxapiCapturer.swift  -- ListWindows() walks
                          NSWorkspace.runningApplications, filters to
                          .regular activation policy, AXUIElementCreate-
                          Application(pid) -> kAXWindowsAttribute,
                          marks hasFocus by cross-referencing
                          NSWorkspace.frontmostApplication.
                          Capture(req) resolves target by windowId
                          (axapi:<pid>:<index>), processId,
                          processName (substring, .app-tolerant),
                          windowTitle, or falls back to focused window.
                          Walks tree depth-first with depth+count+
                          deadline caps, extracts value via the
                          per-role correct attribute, screen bounds via
                          kAXPosition + kAXSize, focused-element id via
                          kAXFocusedUIElementAttribute + identity
                          comparison against captured handles.
                          requirePermission() surfaces a clear
                          accessibilityNotGranted error (-32020) when
                          the parent process hasn't been granted
                          Accessibility in System Settings.

## Dispatcher.swift

  Routes list_windows + capture through AxapiCapturer (lazy init so
  ping-only smoke tests don't pay AXAPI startup cost). Adds
  accessibilityGranted bool to capabilities response so clients can
  detect permission issues without waiting for first capture.

## Validation

Built locally on Apple Silicon arm64; tested against real Mac:

  list_windows -> 22 windows across 13 real apps (Finder, Cursor x3,
  Chrome x5 including the actual agentmark PR view, VS Code with
  claude_desktop_config.json open, Claude Desktop, Teams, Messages,
  Outlook, Parallels Desktop, Terminal, etc.). hasFocus correctly
  marked on the active Cursor window.

  capture processName=Finder -> Copy progress dialog with 11 elements,
  depth 2. Live progress value (0.132 -> 13%) extracted. Real screen
  coordinates. Correct role mapping. Traffic-light buttons surfaced
  with bounds at x=78/98/118 (close/minimize/maximize).

  Same JSON wire format as the Windows bridge -- the Node-side body
  builder we built for the FixtureBackend renders this transparently
  once Phase 0f' (MacosAxapiBackend) lands.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the `execute` JSON-RPC method to the macOS bridge. Routes
element-targeted actions through the right AXAPI pattern. Mirrors the
Windows bridge's UiaCapturer.Execute, same JSON shape on the wire.

## Action types

  - click       -> AXUIElementPerformAction(AXPress) with AXToggle /
                   AXPick / AXConfirm / AXShowMenu fallbacks
  - type        -> AXUIElementSetAttributeValue(kAXValueAttribute) with
                   focus+keystrokes (CGEvent) fallback when AXValue is
                   not writable; supports clearFirst (Cmd+A then Delete)
  - select      -> AXPress on a selection item, or kAXValueAttribute
                   set directly for popup buttons
  - check       -> Toggle until state matches (capped at 3 iterations)
                   via AXPress or AXToggle, mapping into kAXValue bool
  - expand      -> kAXExpandedAttribute set or AXShowMenu / AXPress
                   fallback for outline rows / disclosure triangles
  - focus       -> kAXFocusedAttribute = true
  - scroll_to   -> AXUIElementPerformAction(AXScrollToVisible) with
                   focus fallback
  - key         -> CGEvent keyboard simulation. Modifier-aware (ctrl /
                   option / shift / cmd). Named keys: enter, tab, esc,
                   space, delete, arrows, pgup/pgdn, home/end, F1..F12.
                   Unknown names fall through to literal text typing.

## Element lookup

Capture stashes each walked AXUIElement in a per-session
[element_id -> AXUIElement] map. Execute resolves the request's
elementId against that map. Errors are specific:
  - "No capture session active" -> caller must call capture first
  - "Unknown element_id" -> stale binding, re-capture
  - "Accessibility permission not granted" -> system settings error

## CGEvent typing strategy

For the keystroke fallback we use
CGEventKeyboardSetUnicodeString rather than per-character keyCode
lookup. Handles Latin + accented + most BMP characters without
maintaining a layout-specific table. Combined with AXSetValue as the
primary path, this covers TextEdit / Mail / Cursor / VS Code /
Pages / Numbers cleanly.

## Live demo (scripts/textedit-demo.py)

Python harness that:
  1. Launches TextEdit with a new document via osascript
  2. list_windows -> find TextEdit
  3. capture     -> locate the text_area element (the AXTextArea
                    inside the AXScrollArea inside the AXSplitGroup
                    inside the AXWindow)
  4. execute type -> "Hello from AgentMark Desktop -- macOS bridge
                    phase 0e3'"
  5. capture     -> assert the typed text is in element.value

Tested live on Apple Silicon arm64. PASSED first run:

  Step 2 -- list_windows: found TextEdit window axapi:21232:0
  Step 3 -- capture: 47 elements, depth 3, editor id="First Text View"
  Step 4 -- execute type: ok=true, newValue echoes the sent text
  Step 5 -- re-capture: editor value matches "Hello from AgentMark..."
  LIVE DEMO PASSED

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… AXAPI

Phase 0f'. Mirrors WindowsUiaBackend (PR #14) for the macOS side.
Once this merges, Claude Desktop / Cursor / any MCP client on macOS
can drive real Mac software through the same agentmark_desktop_*
MCP tools, no per-AI-tool integration work.

## What it does

`MacosAxapiBackend` implements the `DesktopCaptureBackend` interface:

  - Lazily spawns agentmark-bridge-macos as a child process
  - Speaks JSON-RPC 2.0 over stdio (one JSON message per line)
  - Performs a ping handshake on first use; tears down + retries on
    handshake failure
  - Correlates concurrent calls by request id (Map<id, PendingCall>);
    each call has its own timeout
  - Forwards bridge stderr to a structured logger
  - Closes cleanly: end stdin, wait 2s, SIGKILL if still alive
  - Re-throws bridge error responses as typed JS errors

Wire-format mapping is identical to WindowsUiaBackend (the two
bridges speak the same JSON protocol by design).

## Bridge resolution

Search order:
  1. options.bridgePath (explicit)
  2. AGENTMARK_BRIDGE_PATH env var (if file exists)
  3. Walk up from __dirname probing common SPM output paths:
       apps/agent-runner/bridges/macos/.build/{release,debug}/
       apps/agent-runner/bridges/macos/.build/arm64-apple-macosx/...
       apps/agent-runner/bridges/macos/.build/x86_64-apple-macosx/...
  4. Throws with the swift-build command and env-var instruction

## MCP dispatcher wiring

`agentmark_desktop_open` with `backend: "macos_axapi"` now actually
spawns the bridge (instead of returning "not yet bundled"). The
optional `bridge_path` arg lets callers override auto-detection. On
non-macOS hosts it returns a clear error pointing to fixture for
testing.

## Tests

`test/desktop/macos-axapi-backend.test.ts` (9 tests) reuse the same
fake-bridge.cjs fixture as the Windows backend tests -- the wire
protocol is byte-identical between bridges, so one fake substitutes
for both:

  - construct refuses on non-macOS without allowNonMac
  - backend name is 'macos_axapi'
  - capture maps response to DesktopCapture shape
  - capture forwards target fields to bridge
  - execute type maps newValue -> new_value
  - same bridge process is reused across calls
  - handshake failure surfaces with a clear error
  - close rejects pending + future calls
  - missing bridge path defers until first call

Plus the existing dispatcher test `agentmark_desktop_open with
macos_axapi returns "not yet bundled"` updated to reflect the new
behaviour (refuses on non-macOS with a clear "requires macOS"
message; real spawn path tested separately).

Full suite: 313 passed, 10 skipped, 0 failed.

## Stacking

Goes on top of PR #16 (macOS scaffold) + PR #17 (macOS capture +
execute). Order to merge: 16 -> 17 -> this PR. Once all three land,
the macOS path is complete and Claude Desktop on macOS can drive
real apps like Pages / Numbers / Excel-for-Mac / TextEdit through
the protocol.

## Code-sharing note

WindowsUiaBackend + MacosAxapiBackend share ~90% of their code
(process spawn, JSON-RPC lifecycle, request correlation, close
logic). A follow-up refactor can extract a `SubprocessBridgeBackend`
base class. Keeping them parallel for now so this PR is reviewable
in isolation and the diff is easy to read.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@rrader26
rrader26 merged commit accd556 into main May 11, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants