feat(desktop): self-healing element resolution via structural fingerprints - #32
Merged
Conversation
…rints The problem Recipes (PR #31) exposes: a step references an element by backend element_id (resolved from action_id via the snapshot binding). Those IDs are sometimes stable, sometimes regenerated. An agent that saved "click element_id=btn_save_4711" yesterday may find that exact ID missing today even though the Save button is right there. The fix: compute a structural signature (role + name + parent context + adjacent siblings) for any element. Store it with your Recipe. When replay finds the original element_id missing, search the current snapshot for the best fingerprint match. Library piece — `src/desktop/fingerprint.ts`: - computeFingerprint(capture, elementId) - findByFingerprint(capture, fingerprint, { minScore? }) - scoreFingerprintMatch(target, candidate) All exported from `@thinkfleet/agentmark`. New MCP tools (2): agentmark_desktop_fingerprint compute fingerprint for an action_id agentmark_desktop_find_by_fingerprint resolve a fingerprint to a current element Scoring: role match is required (score 0 otherwise). Exact name match adds 60. Partial name (Jaccard over bigrams ≥0.8) adds 40. Parent role+name adds 15. Each matching sibling adds 5. Placeholder match adds 5. Default acceptance threshold: 60/100. Caller can lower for permissive matching. v0 scope: the primitives only. The agent calls agentmark_desktop_fingerprint at recipe-save time and embeds the result in the recipe's step args; at replay time it calls agentmark_desktop_find_by_fingerprint when an action_id resolution fails. Automatic transparent healing inside agentmark_desktop_execute is a v1 follow-up — keeps the agent in the loop and the recovery path visible in the reasoning chain. Tests (16 new, 421 total): - computeFingerprint shape + null for unknown ids - scoreFingerprintMatch: role gate, exact match, partial-name credit, parent+sibling weights pushing toward 100 - findByFingerprint: identical-structure ID-rotation match, no-match returns null, sibling context disambiguates duplicates, custom min_score honoured - Tool integration: fingerprint resolves through the snapshot binding, find_by_fingerprint round-trips, missing args error path, unknown fingerprint returns isError + found=false Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The problem
Recipe steps reference elements by `action_id`, which the snapshot binding resolves to a backend `element_id`. Those IDs are sometimes stable (UIA AutomationId), sometimes regenerated each launch, sometimes drift between app versions. An agent that saved "click `element_id=btn_save_4711`" yesterday may find that exact ID missing today even though the Save button is right there.
Today: recipe step silently fails. Today + this PR: agent computes a fingerprint at recipe-save time, falls back to fingerprint match at replay time.
How fingerprints work
A compact structural signature per element. Hand-tuned for tolerance:
Scoring is 0–100. Default acceptance threshold: 60 (exact role+name match clears it; sibling+parent context pushes scores toward 100).
v0 scope
The primitives only. The agent calls these explicitly:
```
agentmark_desktop_fingerprint { desktop_id, action_id: "act_btn_save" }
→ stores the fingerprint in the recipe step args
agentmark_desktop_find_by_fingerprint { desktop_id, fingerprint: {...} }
→ returns the best-matching current element_id + confidence score
```
Automatic transparent healing inside `agentmark_desktop_execute` is a v1 follow-up — keeps the agent in the loop and makes the recovery path visible in the reasoning chain (which has been the consistent design principle of every plugin in this series).
Test plan
🤖 Generated with Claude Code