Antigravity Computer Use (agy-computer-use)

Architecture specification and production-bounded M10/D3 nine-tool computer-use surface for Google Antigravity & Gemini 3.6 Flash on macOS.

---

Technical Overview

agy-computer-use defines an enterprise-grade Computer Use platform for macOS. It combines native macOS screen perception, display topology management, bounded input synthesis, and Unix domain socket IPC via a native host application and an MCP server bridge.

Building on the completed Milestone D2 & M9 foundations, Milestone M10 implements and physically dogfoods the nine-tool MCP surface (computer_use_status, computer_use_observe, computer_use_ax_tree, computer_use_click, computer_use_move, computer_use_type, computer_use_shortcut, computer_use_scroll, computer_use_drag) for macOS desktop interactions.

Local ad-hoc staged-app/TCC operation is proven via ComputerUseHost.app. Stable team code signing, distribution, and notarization remain human-gated future work.

Core Architecture

flowchart TD
    subgraph Antigravity ["Google Antigravity / Gemini 3.6 Flash"]
        Agent["Antigravity Agent / Gemini Model"]
        Skill["Computer Use Skill (.agents/skills/computer-use)"]
    end

    subgraph Server ["Computer Use MCP Server (Node/TypeScript)"]
        MCP["MCP Protocol Handler (Stdio)"]
        SocketClient["Unix Domain Socket IPC Client"]
    end

    subgraph Native ["Staged Background Host (ComputerUseHost)"]
        SocketServer["Unix Domain Socket Listener (0700)"]
        AXEngine["AXUIElement Inspector (Active M10)"]
        CapEngine["Screen Capture Engine (macOS 14+ SCScreenshotManager)"]
        InputEngine["Input Synthesis Engine (Active M10 Bounded)"]
        CoordMapper["Coordinate & Display Scaler"]
    end

    Agent <--> Skill
    Skill <--> MCP
    MCP <-->|Length-Prefixed JSON-RPC| SocketClient
    SocketClient <-->|Unix Domain Socket| SocketServer
    SocketServer <--> AXEngine
    SocketServer <--> CapEngine
    SocketServer <--> InputEngine
    CapEngine -->|Display Bounds & Scale| CoordMapper

Loading

---

Tool API Specifications (Milestone M10 Active Surface)

The MCP server exposes the following nine active tools to Gemini 3.6 Flash / Antigravity:

Tool Name	Required Parameters	Description
computer_use_status	None	Returns host connectivity, active display topology, TCC permission state, and mutation lockout state.
computer_use_observe	display_id?	Captures primary or target display screenshot, returning capture_id, topology_version (top-sha256-...), and JPEG image payload.
computer_use_ax_tree	app_id?, max_depth?	Inspects accessibility UI element hierarchy (AXUIElement tree) of specified running application or frontmost application. Enforces depth, node, string caps, and secure text redaction.
computer_use_click	x, y, intent, capture_id, topology_version, button?, click_count?	Dispatches single mouse click at normalized (0..999) coordinates on active display topology.
computer_use_move	x, y, intent, capture_id, topology_version	Dispatches single mouse movement to normalized (0..999) coordinates without clicking.
computer_use_type	text, intent, capture_id, topology_version, press_enter?	Synthesizes Unicode text entry into focused window/element.
computer_use_shortcut	keys, intent, capture_id, topology_version	Dispatches bounded keyboard shortcut sequence (e.g. ['cmd', 'tab']).
computer_use_scroll	x, y, delta_y, intent, capture_id, topology_version, delta_x?	Dispatches finite scroll wheel input at normalized coordinates.
computer_use_drag	start_x, start_y, end_x, end_y, intent, capture_id, topology_version, button?	Dispatches same-display drag from start to end coordinates with guaranteed button release.

---

Requirements & Setup

• OS: macOS 14.0 (Sonoma) or newer.
• Runtimes:
  • Node.js v22.23.1 (managed via mise).
  • pnpm 10.33.0.
  • Swift 5.9+ / Xcode Command Line Tools.

Running Tests & Readiness Checks

• Production Host Lifecycle Commands:

  • Start Native Host:

    ./bin/agy-computer-use host-start

  • Check Host Status (read-only):

    ./bin/agy-computer-use host-status

  • Stop Native Host:

    ./bin/agy-computer-use host-stop

• Process Authority & Lifecycle Policies:

  • Terminal Owner Correlation: Host lifecycle commands (host-start, host-status, host-stop) communicate with the supervisor owner process over control.sock. host-stop RPC awaits exact native child close and socket residue verification before returning terminal receipt (containing generation ID, daemon PID, native PID, and native_closed: true).
  • Non-Override Runtime Directory Policy: Public CLI host commands operate strictly on the canonical runtime directory (/tmp/agy-computer-use-<uid>), enforcing single-owner Unix domain socket permissions (0700) and inode identity validation to prevent socket hijacking or symlink attacks. Custom runtime directory overrides (COMPUTER_USE_RUNTIME_DIR) are restricted to isolated test harnesses and rejected or fail-closed in public production CLI operations.

Note

Milestones D2 and M9 established the native observation foundation and bounded click/type/shortcut loop. Milestone M10/D3 implements and physically dogfoods the complete nine-tool v0.1 surface, including bounded AX inspection, pointer movement, scrolling, and same-display left-button dragging driven live via Google Antigravity. Local ad-hoc staged-app/TCC operation is proven; team signing/notarization remains future work. A denied Screen Recording or Accessibility state remains a human TCC gate.

• Authoritative Native Swift Test Authority:

  ./bin/agy-computer-use test-native

• Focused Production Host Lifecycle Test Authority:

  node --test bin/host-lifecycle.test.mjs

• TypeScript MCP Server Tests & TypeScript Check:

  cd mcp/computer-use-mcp && pnpm check && pnpm test

• Stage Background App & Verify Principal Classification:

  ./bin/agy-computer-use stage-host-app
  ./bin/agy-computer-use host-principal
  node --test bin/host-app.test.mjs

• Offline Production MCP Readiness Check:

  ./bin/agy-computer-use production-ready