feat(desktop): add list_targets MCP tool — fixes 'AI can't enumerate windows' gap - #19
Merged
Merged
Conversation
Real product gap caught while testing live: the AI on the Windows VM
opened a desktop session and tried to "list windows", but there was no
MCP tool for that. The bridge has `list_windows` and the backend
internals know about windows, but the agent-facing surface only had
`open` / `close` / `snapshot` / `execute` — so the AI was forced to
guess process_names to feed `snapshot` blindly.
Fix: expose enumeration as a first-class operation.
## Changes
`src/desktop/types.ts`:
- Add `listTargets(): Promise<DesktopTargetSummary[]>` to the
`DesktopCaptureBackend` interface.
- New `DesktopTargetSummary` shape — lightweight per-window record
with window_id, process_name, process_id, window_title,
window_class, has_focus. No element tree (cheap call).
`src/desktop/fixture-backend.ts`:
- Implements `listTargets()` by enumerating the preset map.
`src/desktop/windows-uia-backend.ts`:
- Implements `listTargets()` by calling the bridge's existing
`list_windows` JSON-RPC method and mapping camelCase wire fields
to the snake_case interface.
`src/mcp/tool-defs.ts`:
- New tool `agentmark_desktop_list_targets`. Single param:
`desktop_id`. Returns `{ windows: [...] }`.
`src/mcp/dispatcher.ts`:
- Routes `agentmark_desktop_list_targets` -> backend.listTargets().
## Tests (307 pass, 0 fail)
`test/desktop/windows-uia-backend.test.ts`:
- Asserts listTargets maps the bridge `windows` array to
snake_case DesktopTargetSummary correctly.
`test/mcp/desktop-dispatcher.test.ts`:
- Updates tool-count assertion (4 -> 5).
- New test: dispatcher returns fixture preset windows.
- New test: errors on unknown desktop_id.
## After this lands
The AI's exchange that prompted this fix becomes:
agentmark_desktop_open(backend=windows_uia) -> desktop_id
agentmark_desktop_list_targets(desktop_id) -> all real windows
agentmark_desktop_snapshot(desktop_id, windowId=...) -> chosen window
agentmark_desktop_execute(...) -> drive it
Instead of:
agentmark_desktop_open(backend=windows_uia) -> desktop_id
agentmark_desktop_snapshot(desktop_id) -> focused window only
[agent has no way to discover other windows]
## Note on MacosAxapiBackend
PR #18 (Node-side macOS backend) will need the same `listTargets()`
implementation. Will update that PR as a follow-up commit on the
feat/desktop-macos-axapi-backend branch once it merges, rather than
stacking deeper dependencies here.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Caught live while testing AgentMark MCP on the Windows VM with Claude Code. The AI opened a desktop session, captured the focused window (Claude itself), and then had no way to discover other open windows to capture. The bridge supports it (`list_windows` JSON-RPC method), the backend knows about it internally, but it was never exposed on the MCP surface.
This PR closes the gap with a new `agentmark_desktop_list_targets` tool. Independent of PR #16-#18 — branches off main, lands the missing piece for the already-live Windows path.
The AI's actual failure mode (before this fix)
```
Used agentmark: agentmark desktop open
Session opened. Let me try a snapshot...
Used agentmark: agentmark desktop snapshot
Desktop session is open (dt_5150d7c157fa, backend windows_uia), but I
have to flag a limitation: the agentmark MCP doesn't expose an
"enumerate all windows" call. agentmark_desktop_snapshot captures one
window at a time — either the focused one or a specific target...
The only window I can see right now is the focused one:
Claude (process claude, PID 11900...)
Or if you'd prefer a true window list, that needs to come from a
different tool (e.g. PowerShell Get-Process | ...)
```
After this PR, the AI just calls `agentmark_desktop_list_targets(desktop_id)` and gets the full list.
What changed
`src/desktop/types.ts`
`src/desktop/fixture-backend.ts`
`src/desktop/windows-uia-backend.ts`
`src/mcp/tool-defs.ts`
`src/mcp/dispatcher.ts`
Tests
307 pass, 0 fail, 10 skipped (pre-existing).
After this lands (Windows-side)
The user's AI exchange becomes the natural flow:
```
You: list my open windows
AI: [calls agentmark_desktop_list_targets] → I see Notepad, Excel, Chrome, ...
You: capture Notepad
AI: [calls agentmark_desktop_snapshot with that window_id] → here's the tree
You: type "Hello" into the editor
AI: [calls agentmark_desktop_execute] → done
```
Note on macOS
PR #18 (Node-side macOS backend) will need the same `listTargets()` implementation. I'll add it as a follow-up commit on the `feat/desktop-macos-axapi-backend` branch after #18 merges, rather than stacking deeper dependencies here.
🤖 Generated with Claude Code