Skip to content

feat(desktop): WindowsUiaBackend — Node-side bridge client (Phase 0f) - #14

Merged
rrader26 merged 1 commit into
mainfrom
feat/desktop-windows-uia-backend
May 11, 2026
Merged

feat(desktop): WindowsUiaBackend — Node-side bridge client (Phase 0f)#14
rrader26 merged 1 commit into
mainfrom
feat/desktop-windows-uia-backend

Conversation

@rrader26

Copy link
Copy Markdown
Contributor

Summary

Stacks on PR #13 (Phase 0e2 + 0e3 of the Windows bridge).

Once both PRs land, the AgentMark MCP server can drive real Windows
software via the C# UIA bridge — Claude Code / Cursor / Codex / any
MCP client gets real desktop access through the same
`agentmark_desktop_*` MCP tools they already use against the
FixtureBackend. Zero per-AI-tool integration work.

What it does

`WindowsUiaBackend` implements the `DesktopCaptureBackend` interface:

  • Lazily spawns `agentmark-bridge-windows.exe` as a child process
  • Speaks JSON-RPC 2.0 over stdio (one JSON message per line)
  • Performs a ping handshake on first use; tears down + retries on failure
  • Correlates concurrent calls by request id; each call has its own timeout
  • Forwards bridge stderr to a structured logger
  • Closes cleanly: end stdin → wait 2s → SIGKILL if still alive
  • Re-throws bridge error responses as typed JS errors

Wire-format mapping (camelCase on the wire, snake_case in JS):

  • `capture(opts.target.window_id)` → `bridge.capture({ windowId })`
  • response `windowTitle` → `DesktopCapture.window_title`
  • `execute(action)` flattens the discriminated union into the bridge's
    flat payload (`actionType` + only the relevant fields)
  • response `newValue` → `ExecuteDesktopResult.new_value`

Bridge resolution

Search order:

  1. `options.bridgePath` (explicit)
  2. `AGENTMARK_BRIDGE_PATH` env var (if file exists)
  3. Walk up from `__dirname` looking for
    `apps/agent-runner/bridges/windows/bin/{Release,Debug}/net8.0-windows/agentmark-bridge-windows.exe`
    — works in dev checkouts and the Parallels shared-folder layout
  4. Throws with the `dotnet build` command and env-var instruction

MCP dispatcher wiring

`agentmark_desktop_open` with `backend: "windows_uia"` now actually
spawns the bridge (instead of returning "not yet bundled"). New
optional `bridge_path` arg lets a caller override the auto-detected
exe path. On non-Windows hosts it returns a clear error pointing to
`fixture` for testing.

`macos_axapi` still reports not-yet-bundled (Phase 4 future work).

Tests

`test/desktop/windows-uia-backend.test.ts` (8 tests) uses a tiny
`fake-bridge.cjs` Node script that speaks the same stdio JSON-RPC
protocol as the real C# bridge. Tests run cross-platform on
macOS/Linux without needing the .NET runtime:

  • construct refuses on non-Windows without `allowNonWindows`
  • capture maps bridge response to `DesktopCapture` shape
  • capture forwards target fields to bridge
  • execute `type` maps `newValue` → `new_value`
  • same bridge process is reused across calls
  • handshake failure surfaces with a clear error
  • close rejects pending + future calls cleanly
  • missing bridge path defers until first call

`npm test`: 304 passed, 10 skipped, 0 failed.

After this merges

The full local-test path:

```jsonc
// On a Windows machine, Claude Desktop / Cursor / Codex config:
{
"mcpServers": {
"agentmark": {
"command": "npx",
"args": ["-y", "@thinkfleet/agentmark", "mcp"]
}
}
}
```

Then ask the AI:

  • "Open a desktop session with backend windows_uia"
  • "List my open windows"
  • "Capture the Excel window"
  • "Type 'Beta Industries' into cell A1"
  • "Re-capture to confirm the change"

Every request goes: AI → MCP → AgentMark MCP server →
WindowsUiaBackend → agentmark-bridge-windows.exe → UIA → real software.

Stack order

  1. Merge PR feat(bridge-windows): Phase 0e2 + 0e3 — list_windows, capture, execute (real UIA driving real Windows) #13 first (bridge: list_windows + capture + execute)
  2. Rebase this PR on the new main, then merge
  3. Phase 0e4: live Excel demo via Claude Code (no more PowerShell scripts)

🤖 Generated with Claude Code

…ws UIA

Phase 0f. This is the piece that connects the AgentMark MCP server
running on a user's machine to the Windows UIA bridge (Phase 0e2/0e3,
PR #13). Once the user merges PR #13 and stacks this on top, Claude
Code / Cursor / Codex / any MCP client gets real Windows desktop
access through the same agentmark_desktop_* MCP tools they already
use against the FixtureBackend.

## What it does

`WindowsUiaBackend` implements the `DesktopCaptureBackend` interface:

  - Lazily spawns agentmark-bridge-windows.exe as a child process
  - Speaks JSON-RPC 2.0 over stdio (one JSON message per line)
  - Performs a ping handshake on first use; tears down + retries on
    handshake failure
  - Correlates concurrent calls by request id (Map<id, PendingCall>);
    each call has its own timeout
  - Forwards bridge stderr to a structured logger
  - Closes cleanly: end stdin, wait 2s, SIGKILL if still alive
  - Re-throws bridge error responses as typed JS errors

Wire format mapping (camelCase on the wire, snake_case in JS):
  - capture(opts.target.window_id) -> bridge.capture({windowId})
  - capture response.windowTitle -> DesktopCapture.window_title
  - execute(action) flattens the discriminated union into the
    bridge's flat payload (actionType + only the relevant fields)
  - execute response.newValue -> ExecuteDesktopResult.new_value

## Bridge resolution

Search order:
  1. options.bridgePath (explicit)
  2. AGENTMARK_BRIDGE_PATH env var (if file exists)
  3. Walk up from __dirname looking for
     apps/agent-runner/bridges/windows/bin/{Release,Debug}/net8.0-windows/
     agentmark-bridge-windows.exe -- works in dev checkouts and the
     Parallels-shared-folder layout
  4. Throws with the dotnet-build command and env-var instruction

## MCP dispatcher wiring

`agentmark_desktop_open` with `backend: "windows_uia"` now actually
spawns the bridge (instead of returning "not yet bundled"). New
optional `bridge_path` arg lets a caller override the auto-detected
exe path. On non-Windows hosts it returns a clear error pointing to
fixture for testing.

`macos_axapi` still reports not-yet-bundled (Phase 4 future work).

## Tests

`test/desktop/windows-uia-backend.test.ts` (8 tests) uses a tiny
`fake-bridge.cjs` Node script that speaks the same stdio JSON-RPC
protocol as the real C# bridge. Tests run cross-platform on
macOS/Linux without needing the .NET runtime:

  - construct refuses on non-Windows without allowNonWindows
  - capture maps bridge response to DesktopCapture shape
  - capture forwards target fields to bridge
  - execute type maps newValue -> new_value
  - same bridge process is reused across calls
  - handshake failure surfaces with a clear error
  - close rejects pending + future calls cleanly
  - missing bridge path defers until first call

Existing test `agentmark_desktop_open with windows_uia returns "not
yet bundled"` updated to reflect the new behaviour (now: refuses to
spawn on non-Windows with a clear "requires Windows" message; real
spawn path is tested in the windows-uia-backend suite).

`npm test`: 304 passed, 10 skipped, 0 failed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants