Skip to content

feat(plugins): Vision Pack v0 — full-screen screenshot for AI fallback - #28

Merged
rrader26 merged 1 commit into
mainfrom
feat/vision-screenshot
May 12, 2026
Merged

feat(plugins): Vision Pack v0 — full-screen screenshot for AI fallback#28
rrader26 merged 1 commit into
mainfrom
feat/vision-screenshot

Conversation

@rrader26

Copy link
Copy Markdown
Contributor

Summary

  • New `createVisionPlugin()` adds `agentmark_screenshot` for full-screen capture.
  • Layer 2 of the desktop driver stack: when accessibility is too sparse to drive an app, the multimodal AI parses the screenshot directly.
  • Cross-platform pure-Node (shells out to OS-native screenshot CLIs).

Why

Closes the accessibility-coverage gap from the strategic discussion: 99% of well-designed apps expose a usable a11y tree, but custom-drawn UIs, weak-a11y Electron apps, and games don't. Screenshot + multimodal vision = universal fallback. No per-app engineering needed — Claude's own vision handles parsing.

What's in v0

Surface What
Capture Full-screen PNG, inline base64 OR write to disk
Platforms macOS `screencapture`, Windows PowerShell + .NET, Linux `gnome-screenshot`/`scrot`
Output `{ image_base64?, path?, bytes, captured_by, platform }`

What's NOT in v0 (deliberate)

  • Per-window screenshots — needs bridge work (UIA `GraphicsCaptureItem` / macOS `CGWindowListCreateImage`). Follow-up PR after window-management lands.
  • OmniParser-style structured GUI parsing — the AI does its own parsing for now; structured parsing inside the plugin can come later as a model integration.
  • Auto-fallback hook on `agentmark_desktop_snapshot` when the tree is sparse — needs a heuristic ("element_count < N, role-coverage shallow") + a clean control-flow path in the desktop plugin.

Usage

```ts
import { createMcpServer, createDesktopPlugin, createVisionPlugin } from '@thinkfleet/agentmark'

const vision = createVisionPlugin()
createMcpServer({ plugins: [desktop, vision, meta] })
```

When the agent gets an empty / sparse snapshot, it can fall back:
```
agentmark_screenshot { output_path: "/tmp/fallback.png" }
→ multimodal AI vision-parses the image with its native vision model
```

Test plan

  • `pnpm build` clean
  • `pnpm test` — 345 pass / 10 skip (3 new: plugin registration)
  • Manual on macOS: `agentmark_screenshot` returns a valid PNG
  • Manual on Windows: same; verify PowerShell .NET drawing path works on Win 11 ARM64

🤖 Generated with Claude Code

@rrader26
rrader26 force-pushed the feat/vision-screenshot branch from 89457aa to e7b8f22 Compare May 12, 2026 12:34
Layer 2 of the desktop driver stack: when the accessibility tree is too
sparse to drive an app reliably (custom-drawn UIs, weak-a11y Electron
apps, games), the agent grabs a screenshot and the multimodal AI parses
the image directly.

Tools shipped (1):
  agentmark_screenshot   full-screen PNG; inline base64 or write-to-path

Capture utilities per platform (no external Node deps):
  macOS:   screencapture -x (always installed)
  Windows: PowerShell + .NET System.Drawing (built-in to .NET 8)
  Linux:   gnome-screenshot OR scrot (install one)

Returned shape:
  { image_base64?, path?, bytes, captured_by, platform }

Inline base64 by default; pass `output_path` to write to disk and get
back just { path, bytes } — useful for large captures where the agent
doesn't need the raw bytes in-band.

Out of scope (planned follow-up):
  - Per-window capture (needs bridge work: UIA / AXAPI window handle
    → PNG buffer)
  - OmniParser-style structured GUI parsing inside the plugin
  - Auto-fallback on agentmark_desktop_snapshot when the tree is sparse

Tests (3 new, 345 total on this branch): plugin registration. The
screenshot itself needs a real GUI session, so it's smoke-tested via
the live demo paths rather than unit tests.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@rrader26
rrader26 force-pushed the feat/vision-screenshot branch from e7b8f22 to 32bbe90 Compare May 12, 2026 12:35
@rrader26
rrader26 merged commit e7af503 into main May 12, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants