feat: v0.4 — desktop kind (spec + Node producer + FixtureBackend + MCP tools)

[rrader26]

Summary

Ships kind: 'desktop' end-to-end: spec, Node-side producer, fixture
backend, and MCP tool surface so any MCP-compatible AI tool can use
AgentMark Desktop today.

OS-specific bridges (Windows UIA via FlaUI, macOS AXAPI via Swift) ship
in follow-up PRs as separate sidecar processes implementing the
DesktopCaptureBackend interface.

How to try it locally (works on any OS)

// ~/.config/claude/mcp.json  (or Cursor / Windsurf / Codex equivalent)
{
  "mcpServers": {
    "agentmark": {
      "command": "npx",
      "args": ["-y", "@thinkfleet/agentmark", "mcp"]
    }
  }
}

Then from any AI tool:

• "Open a desktop session"
• "Take a desktop snapshot"
• "Take a desktop snapshot of the nowcerts customer fixture"
• "Type 'Beta Industries' into the Company Name field"

The fixture backend serves pre-baked Excel + NowCerts trees so this
works without a real OS bridge installed.

What changed

Spec foundation (v0.4)

• AGENTMARK_VERSION → '0.4', SUPPORTED_SPEC_VERSIONS extended
• SnapshotKind extended with 'desktop'
• New DesktopMeta interface (mirrors MediaMeta pattern)
• desktop_meta?: DesktopMeta on Snapshot
• BodyTagKind extended with WINDOW (boundary marker) and ELEMENT
  (non-interactive accessibility element reference)
• New schema/agentmark-v0.4.json with desktop_meta definition
• Validator updates: v0.4 routing, structural-tag recognition

Node producer (src/desktop/)

• types.ts — DesktopCaptureBackend interface, DesktopCapture,
  DesktopElement, normalised DesktopRole vocabulary,
  ExecuteDesktopAction (click | type | select | check | expand |
  focus | scroll_to | key)
• body-builder.ts — walks the tree, emits markdown with the right
  mix of [ACTION:] / [INPUT:] / [WINDOW:] / [ELEMENT:] tags.
  Headings for windows/panes/dialogs, lists for menus and tree views,
  native markdown tables for grids
• desktop-converter.ts — convertDesktop() entry point. Populates
  the ActionBinding with each action's underlying element_id so
  the runtime resolves execute(actionId) → backend.execute({ element_id })
• fixture-backend.ts — in-memory backend with Excel + NowCerts
  presets. Round-trippable — execute(type) mutates an element-values
  map that subsequent capture() calls reflect
• index.ts — module exports, wired into package root

MCP surface (src/mcp/)

• 4 new tools registered in ALL_TOOLS:
  agentmark_desktop_open, _close, _snapshot, _execute
• Dispatcher caches binding + action types per session so
  _execute can resolve action IDs without the client knowing
  underlying element IDs
• Backend selection at open-time: fixture (default), windows_uia,
  macos_axapi — the OS ones return "not yet bundled" until bridges
  ship
• agentmark_list_sessions extended to report active desktop sessions

Tests

• test/spec-v0.4.test.ts (7 tests) — minimal Windows + macOS
  snapshots validate; bad platform / extra fields rejected;
  [WINDOW] / [ELEMENT] body tags accepted without action lookup;
  YAML round-trip preserves desktop_meta; v0.3 backwards compat
• test/desktop/desktop-converter.test.ts (8 tests) — fixture
  snapshots, synthesised + explicit URLs, backend name reporting,
  aria state, static-only snapshots, full type → execute → re-capture
  round-trip
• test/mcp/desktop-dispatcher.test.ts (12 tests) — tool registration,
  open backends, snapshot targeting, full execute round-trip,
  unknown action handling, session lifecycle

npm test — 295 passed, 10 skipped (pre-existing), 0 failed.

Backwards compat

v0.3 snapshots continue to validate against v0.4 unchanged. No required
field added, no enum narrowed, no existing exports changed.

Next PRs

1. C# FlaUI bridge (Windows) — real Excel/NowCerts/SAP GUI demos
2. macOS Swift/AXAPI bridge
3. WebSocket transport for the bridges + a WindowsUiaBackend /
   MacosAxapiBackend in agentmark-desktop
4. Signed installer bundling Node runtime + bridge as one app

🤖 Generated with Claude Code