click/drag: support window-client-relative pixel endpoints when window target is set#335
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8550bc58e6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@copilot see and address cr feedback test-first. comment on each with your resoultion |
Addressed the CR item in commit Resolution:
|
|
@copilot resolve the merge conflicts in this pull request |
Resolved by merging |
|
@copilot resolve the merge conflicts in this pull request |
Resolved in commit |
Description
Window-targeted captures return window-relative coordinates, but
click/dragpixel endpoints were interpreted as screen-absolute, forcing fragile monitor-offset math (especially multi-monitor/high-DPI).This change makes pixel endpoints window-client-relative whenever a window target is provided, while preserving absolute-pixel behavior when no window target is supplied.
click at:{x,y}withhandle/window/process/foregroundnow resolves asclientOrigin + (x,y).drag from/to {x,y}with a window target now resolves endpoints the same way.ClientToScreeninterop and shared client-origin helpers onWindowTargetingAgentCommand.ClickCommand/DragCommandwindow-target requirements and endpoint resolution paths.ToolCatalogto explicitly document window-relative vs absolute behavior.src/Agent/AgentInstructions.mdanddocs/agent_control.mdfor multi-monitor/per-monitor-DPI interpretation.{ "name": "click", "arguments": { "handle": 12345, "at": { "x": 412, "y": 88 } } }Type of Change
Testing
Checklist
Agent evidence (actuation/observation changes)
artifacts/<customer>/<run>.zipor a link)src/Agent/AgentInstructions.md, served asAgentServer.Instructions)Related Issues