feat(desktop): Node producer + FixtureBackend + MCP tools + Windows bridge IPC - #12
Merged
Conversation
Implements the spec-v0.4 producer side. Pure-Node and OS-agnostic — OS
work is hidden behind the new DesktopCaptureBackend interface. Real OS
bridges (Windows UIA via FlaUI, macOS AXAPI via Swift) ship in
subsequent PRs as separate sidecar processes that implement this
interface.
New module: src/desktop/
- types.ts — DesktopCaptureBackend interface contract,
DesktopCapture (the tree shape backends
return), DesktopElement (one node), DesktopRole
(normalised UIA/AXAPI role vocabulary),
ExecuteDesktopAction (click | type | select |
check | expand | focus | scroll_to | key)
- body-builder.ts — walks the captured tree → emits markdown +
Records<actionId, ActionDefinition>. Headings
for windows/panes/dialogs, lists for menus and
tree controls, native tables for grids,
[ACTION:]/[INPUT:] tags for interactive
elements, [WINDOW:]/[ELEMENT:] for structural
markers
- desktop-converter.ts — convertDesktop() entry point. Backend agnostic.
Populates ActionBinding with the original
element_id so the runtime can resolve
execute(actionId) → backend.execute({ element_id })
- fixture-backend.ts — in-memory DesktopCaptureBackend returning
pre-baked Excel + NowCerts trees. Lets the
MCP server work end-to-end on machines without
the real OS bridge (e.g. testing on a Mac
before the AXAPI bridge exists; CI runners
with no GUI). Round-trippable — execute() type
mutates a value map that the next capture()
reflects.
- index.ts — module exports
Wired into the package root src/index.ts under the existing v0.* section
headers, matching how PDF/audio/video are exported.
Tests: test/desktop/desktop-converter.test.ts — 8 tests covering:
- Valid v0.4 desktop snapshot from Excel fixture (validates clean
against schema + cross-field invariants)
- NowCerts fixture with editable inputs rendered as [INPUT:] tags
- Round-trip: type → re-capture → observe the typed text
- Synthesised desktop:// URL when none provided
- Explicit URL override respected
- Backend name reported in desktop_meta.a11y_backend
- Check_box aria.checked state preserved
- Static-only snapshot drops the `actions` field
Full suite: 283 passed, 10 skipped, 0 failed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires the v0.4 desktop kind through the existing MCP server so any
MCP-compatible AI tool (Claude Code, Cursor, Codex, Windsurf, Continue,
Cody, Agent Zero, etc.) can drive native applications via AgentMark by
adding one stanza to its mcp config:
{
"mcpServers": {
"agentmark": {
"command": "npx",
"args": ["-y", "@thinkfleet/agentmark", "mcp"]
}
}
}
Once connected the AI gets:
- agentmark_desktop_open — attach to a capture backend (default: fixture)
- agentmark_desktop_snapshot — capture the focused window (or specified
target) as v0.4 markdown
- agentmark_desktop_execute — drive an action by ID against the most
recent snapshot. Server resolves action_id → backend element_id via
the ActionBinding cached at snapshot time
- agentmark_desktop_close — release the backend
Backend selection at open-time: `fixture` ships today (in-memory Excel +
NowCerts trees, works on any OS). `windows_uia` and `macos_axapi` return
a friendly "not yet bundled" error until the real bridges ship in
follow-up PRs.
Action-type → ExecuteDesktopAction mapping happens server-side so the
client only needs to know the action_id from the snapshot, not the
underlying UIA/AXAPI pattern.
Files touched:
- src/mcp/types.ts — DesktopSession type + 'dt' prefix
- src/mcp/tool-defs.ts — 4 new tools added to ALL_TOOLS
- src/mcp/dispatcher.ts — handlers, switch cases, listSessions,
disposeAll all extended
- test/mcp/desktop-dispatcher.test.ts — 12 new tests covering open,
snapshot, execute round-trip, unknown action handling, close,
list_sessions integration
Full suite: 295 passed, 10 skipped, 0 failed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the Windows UIA bridge as a .NET 8 console project at
apps/agent-runner/bridges/windows/. Stdio JSON-RPC 2.0 server; one
message per line; stdout strictly framed JSON, stderr for diagnostics.
This first commit ships the protocol substrate and two no-UIA methods
so we can validate the cross-language pipeline before adding the heavy
COM-STA capture/execute logic in Phase 0e2/0e3:
- ping -> { pong, version, arch, processId }
- capabilities -> supported methods + UIA provider info
Built + smoke-tested over SSH from the Mac against a Parallels-shared
checkout. ARM64 native build (no x64 emulation overhead). Round-trip
verified: pipe two requests -> close stdin -> get two responses ->
bridge exits 0.
Implementation notes:
- Sync stdin reads (Console.In.ReadLine). Async ReadLineAsync has a
known issue on Windows pipes where it does not return null on EOF,
hanging the process even after the parent closes the pipe. Sync
reads handle EOF correctly. UIA is COM-STA anyway, so async would
not buy us anything for capture/execute.
- UTF8Encoding(false) on both pipes to avoid BOM corruption of the
JSON-RPC framing.
- All diagnostic output to stderr, never stdout.
- Smoke test uses .NET Process API rather than PowerShell pipes —
PowerShell's pipe-to-native-exe does not propagate EOF, which would
hang the bridge on smoke-test runs.
.gitignore extended for bin/, obj/, *.user, .vs/, .idea/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to PR #11 (which merged the v0.4 spec only). This PR ships
the Node-side producer + fixture backend + MCP tool surface + Windows
bridge IPC substrate — everything required for the spec-v0.4
desktop kind to work end-to-end with a real AI tool today, plus the
beginning of the Windows bridge that will replace the fixture backend
with live UIA in subsequent PRs.
End-to-end validated locally: Claude Code (configured to use
`agentmark` over MCP) successfully drove a full round-trip against
the FixtureBackend — open session, snapshot the NowCerts preset, type
into the Company Name field, re-snapshot, observe the typed value
reflected in the markdown.
What's in this PR
Node-side desktop module (`src/desktop/`)
`DesktopCapture`, `DesktopElement`, normalised `DesktopRole`
vocabulary, `ExecuteDesktopAction` (click | type | select |
check | expand | focus | scroll_to | key)
Headings for windows/panes/dialogs; lists for menus & trees; native
markdown tables for grids; `[ACTION:]`/`[INPUT:]` tags for
interactive elements; `[WINDOW:]`/`[ELEMENT:]` structural markers
Populates the `ActionBinding` with each action's underlying
`element_id` so the runtime resolves `execute(actionId)` →
`backend.execute({ element_id })`
presets. Round-trippable for tests
MCP surface (`src/mcp/`)
`agentmark_desktop_open` / `_close` / `_snapshot` / `_execute`
resolves action IDs without the client knowing native element IDs
`macos_axapi` — the OS ones return "not yet bundled" until the
bridges ship
Windows UIA bridge scaffold (`apps/agent-runner/bridges/windows/`)
JSON, stderr = diagnostics)
`ping`, `capabilities` — Phase 0e2 (`list_windows`, `capture`)
and 0e3 (`execute`) land in follow-up PRs
PowerShell pipe EOF bug)
checkout — ARM64 native (no x64 emulation), round-trip works, exit 0
`.gitignore` additions
Tests
`npm test` — 295 passed, 10 skipped (pre-existing), 0 failed.
Backwards compat
Pure additive. No existing exports changed; no existing tests touched
beyond bumping three hardcoded `'0.3'` version strings to `'0.4'` to
match the new `AGENTMARK_VERSION` constant.
Next PRs
🤖 Generated with Claude Code