Skip to content

click/drag: support window-client-relative pixel endpoints when window target is set#335

Merged
tig merged 5 commits into
developfrom
copilot/click-window-relative-coordinates
Jul 8, 2026
Merged

click/drag: support window-client-relative pixel endpoints when window target is set#335
tig merged 5 commits into
developfrom
copilot/click-window-relative-coordinates

Conversation

Copilot AI commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Description

Window-targeted captures return window-relative coordinates, but click/drag pixel endpoints were interpreted as screen-absolute, forcing fragile monitor-offset math (especially multi-monitor/high-DPI).
This change makes pixel endpoints window-client-relative whenever a window target is provided, while preserving absolute-pixel behavior when no window target is supplied.

  • Behavior change (actuation semantics)
    • click at:{x,y} with handle/window/process/foreground now resolves as clientOrigin + (x,y).
    • drag from/to {x,y} with a window target now resolves endpoints the same way.
    • No window target: pixel endpoints remain absolute screen coordinates.
  • Implementation
    • Added Win32 ClientToScreen interop and shared client-origin helpers on WindowTargetingAgentCommand.
    • Updated ClickCommand/DragCommand window-target requirements and endpoint resolution paths.
    • Preserved existing element-based endpoint behavior and error taxonomy.
  • Agent contract/docs
    • Updated tool schema descriptions in ToolCatalog to explicitly document window-relative vs absolute behavior.
    • Updated src/Agent/AgentInstructions.md and docs/agent_control.md for multi-monitor/per-monitor-DPI interpretation.
  • Example
    {
      "name": "click",
      "arguments": {
        "handle": 12345,
        "at": { "x": 412, "y": 88 }
      }
    }

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Code refactoring
  • Performance improvement
  • Test updates

Testing

  • All existing unit tests pass
  • Added new unit tests for changes
  • Manually tested the changes

Checklist

  • My code follows the project's code style
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

Agent evidence (actuation/observation changes)

  • N/A (not an actuation/observation change)
  • Attached or cited an MCEC evidence bundle (artifacts/<customer>/<run>.zip or a link)
  • If a dogfood finding changed how agents should drive MCEC, updated the connect-time guidance (src/Agent/AgentInstructions.md, served as AgentServer.Instructions)

Related Issues

Copilot AI assigned Copilot and tig Jul 8, 2026
Copilot AI linked an issue Jul 8, 2026 that may be closed by this pull request
Copilot AI changed the title [WIP] Add window-relative coordinates support for click action click/drag: support window-client-relative pixel endpoints when window target is set Jul 8, 2026
Copilot finished work on behalf of tig July 8, 2026 01:42
Copilot AI requested a review from tig July 8, 2026 01:42
@tig tig marked this pull request as ready for review July 8, 2026 15:24

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8550bc58e6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/Commands/WindowTargetingAgentCommand.cs Outdated
@tig

tig commented Jul 8, 2026

Copy link
Copy Markdown
Owner

@copilot see and address cr feedback test-first. comment on each with your resoultion

Copilot AI commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

@copilot see and address cr feedback test-first. comment on each with your resoultion

Addressed the CR item in commit 0767e44.

Resolution:

  • Updated click/drag pixel endpoint resolution to use the target window’s top-left window origin (matching capture coordinates), not client origin.
  • Updated tests first (AgentServerTests wording assertions), then implementation (WindowTargetingAgentCommand, ClickCommand, DragCommand), and docs/tool descriptions (ToolCatalog, AgentInstructions.md, docs/agent_control.md).
  • Removed now-unused ClientToScreen interop declaration.

Copilot finished work on behalf of tig July 8, 2026 15:38
@tig

tig commented Jul 8, 2026

Copy link
Copy Markdown
Owner

@copilot resolve the merge conflicts in this pull request

Copilot AI commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

@copilot resolve the merge conflicts in this pull request

Resolved by merging origin/develop and fixing the src/Agent/AgentInstructions.md conflict, preserving both the newer get-text guidance and the click pixel-coordinate semantics. Included in commit 38c0c43.

@tig

tig commented Jul 8, 2026

Copy link
Copy Markdown
Owner

@copilot resolve the merge conflicts in this pull request

Copilot AI commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

@copilot resolve the merge conflicts in this pull request

Resolved in commit 88e048a by merging latest origin/develop and fixing the docs/agent_control.md conflict while preserving the click/drag window-relative pixel semantics.

Copilot finished work on behalf of tig July 8, 2026 16:07
@tig tig merged commit b66316d into develop Jul 8, 2026
7 checks passed
@tig tig deleted the copilot/click-window-relative-coordinates branch July 8, 2026 16:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

click: window-relative coordinates with window target

2 participants