feat(desktop): WindowsUiaBackend — Node-side bridge client (Phase 0f) - #14
Merged
Conversation
…ws UIA Phase 0f. This is the piece that connects the AgentMark MCP server running on a user's machine to the Windows UIA bridge (Phase 0e2/0e3, PR #13). Once the user merges PR #13 and stacks this on top, Claude Code / Cursor / Codex / any MCP client gets real Windows desktop access through the same agentmark_desktop_* MCP tools they already use against the FixtureBackend. ## What it does `WindowsUiaBackend` implements the `DesktopCaptureBackend` interface: - Lazily spawns agentmark-bridge-windows.exe as a child process - Speaks JSON-RPC 2.0 over stdio (one JSON message per line) - Performs a ping handshake on first use; tears down + retries on handshake failure - Correlates concurrent calls by request id (Map<id, PendingCall>); each call has its own timeout - Forwards bridge stderr to a structured logger - Closes cleanly: end stdin, wait 2s, SIGKILL if still alive - Re-throws bridge error responses as typed JS errors Wire format mapping (camelCase on the wire, snake_case in JS): - capture(opts.target.window_id) -> bridge.capture({windowId}) - capture response.windowTitle -> DesktopCapture.window_title - execute(action) flattens the discriminated union into the bridge's flat payload (actionType + only the relevant fields) - execute response.newValue -> ExecuteDesktopResult.new_value ## Bridge resolution Search order: 1. options.bridgePath (explicit) 2. AGENTMARK_BRIDGE_PATH env var (if file exists) 3. Walk up from __dirname looking for apps/agent-runner/bridges/windows/bin/{Release,Debug}/net8.0-windows/ agentmark-bridge-windows.exe -- works in dev checkouts and the Parallels-shared-folder layout 4. Throws with the dotnet-build command and env-var instruction ## MCP dispatcher wiring `agentmark_desktop_open` with `backend: "windows_uia"` now actually spawns the bridge (instead of returning "not yet bundled"). New optional `bridge_path` arg lets a caller override the auto-detected exe path. On non-Windows hosts it returns a clear error pointing to fixture for testing. `macos_axapi` still reports not-yet-bundled (Phase 4 future work). ## Tests `test/desktop/windows-uia-backend.test.ts` (8 tests) uses a tiny `fake-bridge.cjs` Node script that speaks the same stdio JSON-RPC protocol as the real C# bridge. Tests run cross-platform on macOS/Linux without needing the .NET runtime: - construct refuses on non-Windows without allowNonWindows - capture maps bridge response to DesktopCapture shape - capture forwards target fields to bridge - execute type maps newValue -> new_value - same bridge process is reused across calls - handshake failure surfaces with a clear error - close rejects pending + future calls cleanly - missing bridge path defers until first call Existing test `agentmark_desktop_open with windows_uia returns "not yet bundled"` updated to reflect the new behaviour (now: refuses to spawn on non-Windows with a clear "requires Windows" message; real spawn path is tested in the windows-uia-backend suite). `npm test`: 304 passed, 10 skipped, 0 failed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacks on PR #13 (Phase 0e2 + 0e3 of the Windows bridge).
Once both PRs land, the AgentMark MCP server can drive real Windows
software via the C# UIA bridge — Claude Code / Cursor / Codex / any
MCP client gets real desktop access through the same
`agentmark_desktop_*` MCP tools they already use against the
FixtureBackend. Zero per-AI-tool integration work.
What it does
`WindowsUiaBackend` implements the `DesktopCaptureBackend` interface:
Wire-format mapping (camelCase on the wire, snake_case in JS):
flat payload (`actionType` + only the relevant fields)
Bridge resolution
Search order:
`apps/agent-runner/bridges/windows/bin/{Release,Debug}/net8.0-windows/agentmark-bridge-windows.exe`
— works in dev checkouts and the Parallels shared-folder layout
MCP dispatcher wiring
`agentmark_desktop_open` with `backend: "windows_uia"` now actually
spawns the bridge (instead of returning "not yet bundled"). New
optional `bridge_path` arg lets a caller override the auto-detected
exe path. On non-Windows hosts it returns a clear error pointing to
`fixture` for testing.
`macos_axapi` still reports not-yet-bundled (Phase 4 future work).
Tests
`test/desktop/windows-uia-backend.test.ts` (8 tests) uses a tiny
`fake-bridge.cjs` Node script that speaks the same stdio JSON-RPC
protocol as the real C# bridge. Tests run cross-platform on
macOS/Linux without needing the .NET runtime:
`npm test`: 304 passed, 10 skipped, 0 failed.
After this merges
The full local-test path:
```jsonc
// On a Windows machine, Claude Desktop / Cursor / Codex config:
{
"mcpServers": {
"agentmark": {
"command": "npx",
"args": ["-y", "@thinkfleet/agentmark", "mcp"]
}
}
}
```
Then ask the AI:
Every request goes: AI → MCP → AgentMark MCP server →
WindowsUiaBackend → agentmark-bridge-windows.exe → UIA → real software.
Stack order
🤖 Generated with Claude Code