feat(desktop): snapshot diff — return only what changed - #26
Merged
Conversation
New tool `agentmark_desktop_diff` takes a fresh capture and compares it against the most recent cached one on the session. Returns a structured diff (added/removed/changed elements, window title shifts, focus changes) instead of the whole tree. Way smaller payload for agents verifying "did my action take effect?" or watching for new dialogs. Library piece: `diffDesktopCaptures(before, after)` — pure function, exported from `@thinkfleet/agentmark`. Matches elements by stable accessibility id; reports primitive field changes (value, enabled, selected, read_only, expanded, aria.*, bounds). Ignores ≤1px bounds jitter (sub-pixel compositor noise isn't actionable). `convertDesktop()` now returns the raw `DesktopCapture` alongside the agentmark string + binding, via the new `DesktopConversionResult` type. The desktop plugin caches it on the session so subsequent diffs have a baseline; each diff updates the baseline forward so consecutive diffs return only what changed since the previous diff. The action-id binding is NOT updated by a diff — call `agentmark_desktop_snapshot` when you need fresh action_ids after a structural change. Diff is read-only on actionable state. Tests (7 new): - Diff pure function: no-op equality, value change, enabled flip, aria.checked flip, sub-pixel bounds suppression, real bounds shift, add/remove elements, window title + focus changes. - Tool integration via FixtureBackend: no_changes after snapshot, value change after execute, errors before any snapshot. Total 354 pass / 10 skip. Build clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
Smaller token payload + faster reasoning for two extremely common agent questions:
Today an agent answering either of those has to re-snapshot the full tree and reason over thousands of tokens. With diff, "no_changes" is one boolean; even a busy diff is typically <500 tokens.
Also: prerequisite for Recipes (the record/replay feature next on the roadmap). Recipes use diff as the verification primitive after each replayed step.
Diff shape
```json
{
"no_changes": false,
"summary": { "added": 1, "removed": 0, "changed": 2 },
"window_title_changed": null,
"focus_changed": { "from": "btn_save", "to": "txt_name" },
"elements_added": [{ "id": "...", "role": "...", "name": "...", "value": "..." }],
"elements_removed": [{ "id": "...", "role": "...", "name": "...", "value": "..." }],
"elements_changed": [
{ "id": "...", "role": "...", "name": "...", "changes": {
"value": { "from": "old", "to": "new" },
"aria.checked": { "from": false, "to": true }
}}
]
}
```
Semantics
Test plan
Roadmap dependency chain
This unblocks the Recipes PR, which uses diff to verify each replayed step ("did the expected change happen? if not, fail this recipe step instead of plowing through stale state").
🤖 Generated with Claude Code