diff --git a/AGENTS.md b/AGENTS.md index 7b6d8be..9d86fcc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -6,7 +6,8 @@ # AGENTS.md: driving (and testing) MCEC with an agent MCEC 3.0 (the **Model Context Environment Controller**) gives an AI agent eyes, hands, and a safe -front door on Windows. This file is the **self-reinforcing guidance loop**: the canonical guidance an +front door on Windows (Windows **computer use**, in the sense Claude, Codex, and similar agents use the +term). This file is the **self-reinforcing guidance loop**: the canonical guidance an agent needs, plus the recipe to *dogfood* it (drive MCEC through its own MCP server) so the guidance stays honest. Each time MCEC changes, re-run the dogfood and refine the guidance below and in `AgentServer.Instructions`. @@ -182,6 +183,9 @@ controller-bootstrap follow-ups. holds contributor and maintainer material only (CI, releasing, code signing, agent-server architecture notes, evidence bundles, doc-image regeneration recipes); it is not part of the site. **Release instructions:** [`dev/RELEASING.md`](dev/RELEASING.md) (cut from `main`, never `develop`). +- **No em dashes (—) in prose, anywhere in this repo.** Use parentheses or semicolons for an aside; for a + bullet-point heading (`* [Link](url) — description`) use a plain hyphen (`-`) or a colon (`:`) instead. + This applies to Markdown docs and code comments alike. - **Agent-facing guidance is part of "Done": not optional, not "later."** Any change to how an agent observes/targets/acts (a new tool, arg, failure mode, warning/error category, or driving technique) MUST update the connect-time playbook in diff --git a/README.md b/README.md index c029a91..78a0e84 100644 --- a/README.md +++ b/README.md @@ -6,9 +6,11 @@ By [Tig Kindel](https://twitter.com/tigkindel) - Copyright © [Kindel](http://ww agents on Windows, and the same battle-tested TCP/serial remote control for integration systems it has always been. -For agents, it is a small native Windows daemon a computer-use model **mounts, sees through, and drives** -over the **Model Context Protocol (MCP)**: capture windows, read UI Automation trees, find controls, launch -apps, and actuate keyboard/mouse input. For control systems, it listens on TCP/IP or a serial port and +For agents, it is a small native Windows daemon that gives a **computer use** model (the same capability +Claude, Codex, and similar agents use to see a screen and drive a keyboard and mouse) something to +**mount, see through, and drive** over the **Model Context Protocol (MCP)**: capture windows, read UI +Automation trees, find controls, launch apps, and actuate keyboard/mouse input. For control systems, it +listens on TCP/IP or a serial port and translates remote commands into keystrokes, text, mouse moves, window messages, and app launches ([Control4](https://www.control4.com/), [Crestron](http://www.crestron.com/), [iRule](http://www.iruleathome.com/), and others). The 3.0 agent surface is **purely additive**; classic remote control is unchanged. @@ -37,16 +39,50 @@ MCP client setup, and teardown — is in [Agent Control → Quick start](https:/ Full guides live on the [docs site](https://tig.github.io/mcec/): -* [Install](https://tig.github.io/mcec/install.html) — winget, what gets installed where, side-by-side copies -* [Configuration](https://tig.github.io/mcec/configuration.html) — Settings, `mcec.settings`, commands, logging -* [Agent Control](https://tig.github.io/mcec/agent_control.html) — observation, targeting, actuation, MCP/HTTP -* [Agent Safety](https://tig.github.io/mcec/safety-emergency-stop-and-provisioning.html) — consent, provisioning, emergency stop, overlay -* [Remote Control](https://tig.github.io/mcec/remote_control.html) — TCP/serial commands and User Activity Monitor -* [Examples](https://tig.github.io/mcec/examples.html) — worked agent-driving recipes +* [Install](https://tig.github.io/mcec/install.html): winget, what gets installed where, side-by-side copies +* [Configuration](https://tig.github.io/mcec/configuration.html): Settings, `mcec.settings`, commands, logging +* [Agent Control](https://tig.github.io/mcec/agent_control.html) - the computer use surface: observation, targeting, actuation, MCP/HTTP +* [Agent Safety](https://tig.github.io/mcec/safety-emergency-stop-and-provisioning.html): consent, provisioning, emergency stop, overlay +* [Remote Control](https://tig.github.io/mcec/remote_control.html): TCP/serial commands and User Activity Monitor +* [Examples](https://tig.github.io/mcec/examples.html): worked agent-driving recipes Developers and agents: [AGENTS.md](AGENTS.md) (connect-time guidance and the MCEC-drives-MCEC dogfood test). Contributor docs (CI, signing, architecture, image regeneration) live in [`dev/`](dev/). +## Frequently Asked Questions (FAQ) + +**Q: Claude, Copilot, and other agent platforms already support computer use. Why would I use MCEC?** + +Those platforms aim for cross-platform reach, so their computer use is built primarily on computer +vision: a screenshot, interpreted by a vision model, on every step. That is slow (a network round trip per +observation) and token-expensive (a full-window PNG on every turn). MCEC is Windows-only by design, so it +can go deeper than pixels: `query` reads the real Windows UI Automation tree (control type, name, +automation ID, bounds, enabled state, value), letting an agent target and verify a specific control instead +of re-parsing an image every step. `capture` (a real screenshot) is still there for when vision is the +right tool. And because MCEC speaks the **Model Context Protocol (MCP)**, it works with any MCP-capable +agent, Claude, Codex, or a custom client, so a workflow or automated system built on it isn't locked to one +AI provider. + +**Q: Is it safe?** + +It depends on how you use it, but the defaults are safe. The agent surface is off until you opt in; +every command ships individually disabled; every agent action is audit-logged on screen (the overlay) and +in the log; and a global emergency-stop hotkey lets the operator halt a session instantly. See +[Agent Safety](https://tig.github.io/mcec/safety-emergency-stop-and-provisioning.html) for the full model. + +**Q: What's the history of this thing?** + +* **3.0 (2026)**: Rebranded to the **Model Context Environment Controller**. Agent automation over MCP: + observation (`capture`/`query`/`record`), targeting (`find`/`wait-for`), actuation + (`invoke`/`launch`/`drag`/`click`), emergency stop, and isolated session provisioning; all opt-in and + off by default. +* **2.x (2019)**: Major rework: robust client/server, User Activity Monitor (occupancy sensing), Commands + Window with built-in test mode, per-monitor DPI support, config in `%APPDATA%`. +* **1.x (2004–2017)**: Born as Media Center Edition Controller (MS later dropped the "Edition" from + "Windows Media Center") for Windows Media Center HTPCs. Grew keyboard/mouse/window-message + simulation, `chars:` with Unicode escapes, serial support, multi-client TCP, and the `.commands` + extension file. Moved from SourceForge to CodePlex to GitHub. + ## Integrations * [Control4 User Activity Driver](https://github.com/tig/User_Activity) diff --git a/SECURITY.md b/SECURITY.md index 6261606..1779b0b 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -24,7 +24,8 @@ Security fixes target the latest 3.0.x release. Please reproduce on the current ## What is most valuable -MCEC's agent surface is **off by default** and gated behind explicit, independent opt-ins (see +MCEC's agent surface (its **computer use** capability) is **off by default** and gated behind explicit, +independent opt-ins (see [Agent Safety](docs/safety-emergency-stop-and-provisioning.md)). Reports are especially valuable when they show a way to: diff --git a/dev/agent-server-architecture.md b/dev/agent-server-architecture.md index bdef4cc..27cc82a 100644 --- a/dev/agent-server-architecture.md +++ b/dev/agent-server-architecture.md @@ -6,7 +6,7 @@ # Agent Server: Architecture Notes Developer doc ([`dev/`](README.md)); not published to GitHub Pages. A short tour of the MCEC 3.0 agent -subsystem. It lives under +subsystem (the machinery behind MCEC's **computer use** surface). It lives under `src/Agent/` and is **additive**: it reuses the existing `Command` / `CommandInvoker` pattern and adds no breaking changes to any existing command, transport, or default. diff --git a/docs/agent_control.md b/docs/agent_control.md index ec75826..f5bfd11 100644 --- a/docs/agent_control.md +++ b/docs/agent_control.md @@ -3,15 +3,21 @@ // Published under the MIT License - Source on GitHub: https://github.com/tig/mcec --> -# Agent Control +# Agent Control (Computer Use) -MCEC 3.0 turns the MCE Controller daemon into a small, opt-in automation server for +MCEC 3.0 turns the MCE Controller daemon into a small, opt-in **computer use** server (the same +capability Claude, Codex, and similar agents use to see a screen and drive a keyboard and mouse) for AI agents and scripts running on a Windows PC. It gives an agent three things: -- **Eyes**: capture a screenshot of a window (or the foreground window) as a PNG. +- **Eyes**: far more than a screenshot. `capture` still takes a PNG of a window (or a screen + region), but `query` reads a window's live **UI Automation tree** (control type, name, + automation ID, bounds, enabled/offscreen state, value), `displays` reports per-monitor + geometry (bounds, DPI, scale), and `windows`/`find`/`wait-for` discover and wait on windows + and UI elements by name, class, or automation ID. An agent can see structured state, not + just pixels. - **Hands**: invoke any existing MCEC command (the actuation layer you already use). -- **A front door**: query/find windows and UI elements, wait for conditions, and - drive all of the above over **MCP** (Model Context Protocol) or a tiny **HTTP** floor. +- **A front door**: drive all of the above over **MCP** (Model Context Protocol) or a tiny + **HTTP** floor, gated off by default and audited when on. The agent surface is a set of new commands (`capture`, `query`, `displays`, `windows`, `window`, `find`, `wait-for`, `invoke`, `record`, `launch`, `drag`, `click`, and `focus`) exposed as **tools over MCP/HTTP** @@ -541,9 +547,9 @@ rest of the tree is returned. ## Using MCEC from a desktop agent app MCEC can run **headless** as an MCP **stdio** server (no main window, no tray icon; the -on-screen command overlay and the emergency-stop hotkey still work) so a desktop agent app -(such as a desktop AI assistant or custom MCP client) can spawn it on demand and talk to it over standard -input/output: +on-screen command overlay and the emergency-stop hotkey still work) so a desktop agent app with +computer use capability (such as a desktop AI assistant or custom MCP client) can spawn it on demand +and talk to it over standard input/output: ``` mcec.exe mcp # or the equivalent legacy spelling: mcec.exe --mcp diff --git a/docs/configuration.md b/docs/configuration.md index 1265b5a..f17ff0b 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -10,8 +10,8 @@ Everything MCEC does is configured in one of three places: the **File ▸ Settin running MCEC, every Settings tab, the agent gates in `mcec.settings`, enabling commands, and logging. For **installing** MCEC and where its files live, see [Install](install.md); for the **security model** -behind the agent gates, see [Agent Safety](safety-emergency-stop-and-provisioning.md); for the agent -tools themselves, see [Agent Control](agent_control.md). +behind the agent gates, see [Agent Safety](safety-emergency-stop-and-provisioning.md); for the computer +use tools themselves, see [Agent Control](agent_control.md). If you want to use MCEC from a desktop agent app, see [Agent Control](agent_control.md). In practice: enable diff --git a/docs/index.md b/docs/index.md index 97d28cf..99b3d93 100644 --- a/docs/index.md +++ b/docs/index.md @@ -6,36 +6,58 @@ agents on Windows, and the same battle-tested TCP/serial remote control for integration systems it has always been. -It is a small, self-contained native Windows daemon that a computer-use model can **mount, see through, -and drive**: capture a window as a PNG, read its UI Automation tree, find and wait for controls, launch -apps, and actuate keyboard/mouse/window input, over the **Model Context Protocol (MCP)**. The agent -surface is opt-in and off by default; the 3.0 agent features are purely additive over the classic -remote-control command surface (network and serial), which is unchanged. +It is a small, self-contained native Windows daemon that gives a **computer use** model (the same +capability Claude, Codex, and similar agents use to see a screen and drive a keyboard and mouse) +something to **mount, see through, and drive**: capture a window as a PNG, read its UI Automation tree, +find and wait for controls, launch apps, and actuate keyboard/mouse/window input, over the **Model Context +Protocol (MCP)**. The agent surface is opt-in and off by default; the 3.0 agent features are purely +additive over the classic remote-control command surface (network and serial), which is unchanged. **Install with winget:** `winget install Kindel.mcec` — see [Install](install.html). ## The chapters -* **[Install](install.html)** — install with winget or the signed installer; what gets installed where, +* **[Install](install.html)**: install with winget or the signed installer; what gets installed where, running side-by-side copies, and disposable provisioned instances for agents. -* **[Configuration](configuration.html)** — everything you can configure: the Settings dialog (every +* **[Configuration](configuration.html)** - everything you can configure: the Settings dialog (every tab), the `mcec.settings` file, the command table, and logging. -* **[Agent Control](agent_control.html)** — the agent surface: the observation, +* **[Agent Control](agent_control.html)** - the computer use surface: the observation, targeting, and actuation tools, the structured result envelope, and the MCP / localhost-HTTP transports. -* **[Agent Safety](safety-emergency-stop-and-provisioning.html)** — command-access consent, disposable +* **[Agent Safety](safety-emergency-stop-and-provisioning.html)**: command-access consent, disposable isolated session provisioning, the emergency-stop hotkey, and the on-screen command overlay. -* **[Remote Control](remote_control.html)** — the classic role: listen on TCP/IP or a +* **[Remote Control](remote_control.html)** - the classic role: listen on TCP/IP or a serial port and translate remote commands into keystrokes, text, mouse, window messages, and app launches ([Control4](https://www.control4.com/), [iRule](http://www.iruleathome.com/), [Crestron](http://www.crestron.com/), and others). The same chapter covers the **User Activity Monitor**, which runs the flow in reverse: it reports when someone is actively using the PC, turning the machine into an occupancy sensor that can drive lighting and scenes. -* **[Examples](examples.html)** — worked agent-driving recipes (hero GIFs, prompt demos) and how to add more. +* **[Examples](examples.html)**: worked agent-driving recipes (hero GIFs, prompt demos) and how to add more. See also [AGENTS.md](https://github.com/tig/mcec/blob/main/AGENTS.md) for connect-time agent guidance and the dogfood recipe (MCEC driving MCEC). -## Version history +## Frequently Asked Questions (FAQ) + +**Q: Claude, Copilot, and other agent platforms already support computer use. Why would I use MCEC?** + +Those platforms aim for cross-platform reach, so their computer use is built primarily on computer +vision: a screenshot, interpreted by a vision model, on every step. That is slow (a network round trip per +observation) and token-expensive (a full-window PNG on every turn). MCEC is Windows-only by design, so it +can go deeper than pixels: `query` reads the real Windows UI Automation tree (control type, name, +automation ID, bounds, enabled state, value), letting an agent target and verify a specific control instead +of re-parsing an image every step. `capture` (a real screenshot) is still there for when vision is the +right tool. And because MCEC speaks the **Model Context Protocol (MCP)**, it works with any MCP-capable +agent, Claude, Codex, or a custom client, so a workflow or automated system built on it isn't locked to one +AI provider. + +**Q: Is it safe?** + +It depends on how you use it, but the defaults are safe. The agent surface is off until you opt in; +every command ships individually disabled; every agent action is audit-logged on screen (the overlay) and +in the log; and a global emergency-stop hotkey lets the operator halt a session instantly. See +[Agent Safety](safety-emergency-stop-and-provisioning.html) for the full model. + +**Q: What's the history of this thing?** * **3.0 (2026)**: Rebranded to the **Model Context Environment Controller**. Agent automation over MCP: observation (`capture`/`query`/`record`), targeting (`find`/`wait-for`), actuation diff --git a/docs/install.md b/docs/install.md index 3a6ee7f..1bd2f74 100644 --- a/docs/install.md +++ b/docs/install.md @@ -42,9 +42,10 @@ This directory-per-instance isolation is exactly what provisioning (below) autom ## Provisioning: disposable instances for agents The installed copy under Program Files deliberately **refuses to serve agents**: running `mcec.exe --mcp` -or starting the MCP/HTTP endpoint from Program Files is refused, because enabling the agent gates in your -installed config would leave them enabled if a session crashed. Instead, an authorized agent asks MCEC for -a fresh, disposable copy to drive (there is no `--provision` flag; provisioning is an in-product feature): +or starting the MCP/HTTP endpoint from Program Files is refused, because enabling the computer use gates +in your installed config would leave them enabled if a session crashed. Instead, an authorized agent asks +MCEC for a fresh, disposable copy to drive (there is no `--provision` flag; provisioning is an in-product +feature): 1. Turn on **Allow agents to provision disposable instances** on the Settings dialog's **Agent** tab (`AllowSessionProvisioning`). This is the one opt-in an operator performs. diff --git a/docs/remote_control.md b/docs/remote_control.md index 6d543a2..8f78e7f 100644 --- a/docs/remote_control.md +++ b/docs/remote_control.md @@ -22,7 +22,8 @@ MCEC was originally built to integrate a Windows Media Center home-theater PC (H whole-house A/V system, but it is general enough for any control system that can send text to a TCP/IP port or serial port. -> **Looking for the agent/AI side?** MCEC 3.0 is also an opt-in automation server for AI agents over the +> **Looking for the agent/AI side?** MCEC 3.0 is also an opt-in **computer use** server (the same +> capability Claude, Codex, and similar agents use to see a screen and drive a keyboard and mouse) over the > **Model Context Protocol (MCP)**: see the [Agent Control](agent_control.md) and > [Configuration](configuration.md) chapters. This page covers the classic remote-control features, which are > unchanged and remain the default behavior. diff --git a/docs/safety-emergency-stop-and-provisioning.md b/docs/safety-emergency-stop-and-provisioning.md index ce67d5e..bf9c860 100644 --- a/docs/safety-emergency-stop-and-provisioning.md +++ b/docs/safety-emergency-stop-and-provisioning.md @@ -5,7 +5,9 @@ # Agent Safety -Four related safety features for MCEC's agent surface. When MCEC is agent-driving the desktop, the **target +Four related safety features for MCEC's agent surface (MCEC's **computer use** capability, in the same +sense Claude, Codex, and similar agents use the term for seeing a screen and driving a keyboard and mouse). +When MCEC is agent-driving the desktop, the **target app has focus, not MCEC**; so the operator needs (a) a controlled way to grant more capability mid-session, (b) assurance that a session cannot leave the installed copy in an unsafe state, (c) a way to instantly intervene when a run goes wrong, and (d) visible narration that MCEC is driving. Command-access consent,