Skip to content

feat(desktop): snapshot diff — return only what changed - #26

Merged
rrader26 merged 1 commit into
mainfrom
feat/desktop-snapshot-diff
May 12, 2026
Merged

feat(desktop): snapshot diff — return only what changed#26
rrader26 merged 1 commit into
mainfrom
feat/desktop-snapshot-diff

Conversation

@rrader26

Copy link
Copy Markdown
Contributor

Summary

  • New tool `agentmark_desktop_diff`: take a fresh capture, return the structured delta vs the last cached snapshot.
  • New library function `diffDesktopCaptures(before, after)`: pure, exportable, reusable.
  • `convertDesktop()` now returns the raw `DesktopCapture` via the new `DesktopConversionResult` type so the session can cache it for the next diff.

Why

Smaller token payload + faster reasoning for two extremely common agent questions:

  • "Did my action take effect?" → check `elements_changed` for the field that should have moved.
  • "Did a new dialog appear?" → check `elements_added` for a `dialog`-role node.

Today an agent answering either of those has to re-snapshot the full tree and reason over thousands of tokens. With diff, "no_changes" is one boolean; even a busy diff is typically <500 tokens.

Also: prerequisite for Recipes (the record/replay feature next on the roadmap). Recipes use diff as the verification primitive after each replayed step.

Diff shape

```json
{
"no_changes": false,
"summary": { "added": 1, "removed": 0, "changed": 2 },
"window_title_changed": null,
"focus_changed": { "from": "btn_save", "to": "txt_name" },
"elements_added": [{ "id": "...", "role": "...", "name": "...", "value": "..." }],
"elements_removed": [{ "id": "...", "role": "...", "name": "...", "value": "..." }],
"elements_changed": [
{ "id": "...", "role": "...", "name": "...", "changes": {
"value": { "from": "old", "to": "new" },
"aria.checked": { "from": false, "to": true }
}}
]
}
```

Semantics

  • Matching is by stable accessibility id (`DesktopElement.id`). Roles + names are auxiliary, not used for matching.
  • Sub-pixel bounds jitter (≤1px) is suppressed — compositors produce that constantly and it's never actionable.
  • After a successful diff, the session's cached capture moves forward. The NEXT diff is against this new baseline, not the original snapshot.
  • The action-id binding is not updated. Call `agentmark_desktop_snapshot` when you need fresh action_ids after a structural change. Diff is read-only on actionable state.

Test plan

  • `pnpm build` clean
  • `pnpm test` — 354 pass / 10 skip (7 new: pure-function coverage + tool integration)
  • Manual: type into a NowCerts field, diff, confirm only that field appears in elements_changed

Roadmap dependency chain

This unblocks the Recipes PR, which uses diff to verify each replayed step ("did the expected change happen? if not, fail this recipe step instead of plowing through stale state").

🤖 Generated with Claude Code

New tool `agentmark_desktop_diff` takes a fresh capture and compares it
against the most recent cached one on the session. Returns a structured
diff (added/removed/changed elements, window title shifts, focus
changes) instead of the whole tree. Way smaller payload for agents
verifying "did my action take effect?" or watching for new dialogs.

Library piece: `diffDesktopCaptures(before, after)` — pure function,
exported from `@thinkfleet/agentmark`. Matches elements by stable
accessibility id; reports primitive field changes (value, enabled,
selected, read_only, expanded, aria.*, bounds). Ignores ≤1px bounds
jitter (sub-pixel compositor noise isn't actionable).

`convertDesktop()` now returns the raw `DesktopCapture` alongside the
agentmark string + binding, via the new `DesktopConversionResult`
type. The desktop plugin caches it on the session so subsequent diffs
have a baseline; each diff updates the baseline forward so consecutive
diffs return only what changed since the previous diff.

The action-id binding is NOT updated by a diff — call
`agentmark_desktop_snapshot` when you need fresh action_ids after a
structural change. Diff is read-only on actionable state.

Tests (7 new):
- Diff pure function: no-op equality, value change, enabled flip,
  aria.checked flip, sub-pixel bounds suppression, real bounds shift,
  add/remove elements, window title + focus changes.
- Tool integration via FixtureBackend: no_changes after snapshot,
  value change after execute, errors before any snapshot.

Total 354 pass / 10 skip. Build clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@rrader26
rrader26 merged commit a4ec7d7 into main May 12, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants