Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@
# AGENTS.md: driving (and testing) MCEC with an agent

MCEC 3.0 (the **Model Context Environment Controller**) gives an AI agent eyes, hands, and a safe
front door on Windows. This file is the **self-reinforcing guidance loop**: the canonical guidance an
front door on Windows (Windows **computer use**, in the sense Claude, Codex, and similar agents use the
term). This file is the **self-reinforcing guidance loop**: the canonical guidance an
agent needs, plus the recipe to *dogfood* it (drive MCEC through its own MCP server) so the guidance
stays honest. Each time MCEC changes, re-run the dogfood and refine the guidance below and in
`AgentServer.Instructions`.
Expand Down Expand Up @@ -182,6 +183,9 @@ controller-bootstrap follow-ups.
holds contributor and maintainer material only (CI, releasing, code signing, agent-server architecture
notes, evidence bundles, doc-image regeneration recipes); it is not part of the site. **Release
instructions:** [`dev/RELEASING.md`](dev/RELEASING.md) (cut from `main`, never `develop`).
- **No em dashes (—) in prose, anywhere in this repo.** Use parentheses or semicolons for an aside; for a
bullet-point heading (`* [Link](url) — description`) use a plain hyphen (`-`) or a colon (`:`) instead.
This applies to Markdown docs and code comments alike.
- **Agent-facing guidance is part of "Done": not optional, not "later."** Any change to how an agent
observes/targets/acts (a new tool, arg, failure mode, warning/error category, or driving technique)
MUST update the connect-time playbook in
Expand Down
54 changes: 45 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,11 @@ By [Tig Kindel](https://twitter.com/tigkindel) - Copyright © [Kindel](http://ww
agents on Windows, and the same battle-tested TCP/serial remote control for integration systems it has
always been.

For agents, it is a small native Windows daemon a computer-use model **mounts, sees through, and drives**
over the **Model Context Protocol (MCP)**: capture windows, read UI Automation trees, find controls, launch
apps, and actuate keyboard/mouse input. For control systems, it listens on TCP/IP or a serial port and
For agents, it is a small native Windows daemon that gives a **computer use** model (the same capability
Claude, Codex, and similar agents use to see a screen and drive a keyboard and mouse) something to
**mount, see through, and drive** over the **Model Context Protocol (MCP)**: capture windows, read UI
Automation trees, find controls, launch apps, and actuate keyboard/mouse input. For control systems, it
listens on TCP/IP or a serial port and
translates remote commands into keystrokes, text, mouse moves, window messages, and app launches
([Control4](https://www.control4.com/), [Crestron](http://www.crestron.com/), [iRule](http://www.iruleathome.com/),
and others). The 3.0 agent surface is **purely additive**; classic remote control is unchanged.
Expand Down Expand Up @@ -37,16 +39,50 @@ MCP client setup, and teardown — is in [Agent Control → Quick start](https:/

Full guides live on the [docs site](https://tig.github.io/mcec/):

* [Install](https://tig.github.io/mcec/install.html) winget, what gets installed where, side-by-side copies
* [Configuration](https://tig.github.io/mcec/configuration.html) Settings, `mcec.settings`, commands, logging
* [Agent Control](https://tig.github.io/mcec/agent_control.html) observation, targeting, actuation, MCP/HTTP
* [Agent Safety](https://tig.github.io/mcec/safety-emergency-stop-and-provisioning.html) consent, provisioning, emergency stop, overlay
* [Remote Control](https://tig.github.io/mcec/remote_control.html) TCP/serial commands and User Activity Monitor
* [Examples](https://tig.github.io/mcec/examples.html) worked agent-driving recipes
* [Install](https://tig.github.io/mcec/install.html): winget, what gets installed where, side-by-side copies
* [Configuration](https://tig.github.io/mcec/configuration.html): Settings, `mcec.settings`, commands, logging
* [Agent Control](https://tig.github.io/mcec/agent_control.html) - the computer use surface: observation, targeting, actuation, MCP/HTTP
* [Agent Safety](https://tig.github.io/mcec/safety-emergency-stop-and-provisioning.html): consent, provisioning, emergency stop, overlay
* [Remote Control](https://tig.github.io/mcec/remote_control.html): TCP/serial commands and User Activity Monitor
* [Examples](https://tig.github.io/mcec/examples.html): worked agent-driving recipes

Developers and agents: [AGENTS.md](AGENTS.md) (connect-time guidance and the MCEC-drives-MCEC dogfood test).
Contributor docs (CI, signing, architecture, image regeneration) live in [`dev/`](dev/).

## Frequently Asked Questions (FAQ)

**Q: Claude, Copilot, and other agent platforms already support computer use. Why would I use MCEC?**

Those platforms aim for cross-platform reach, so their computer use is built primarily on computer
vision: a screenshot, interpreted by a vision model, on every step. That is slow (a network round trip per
observation) and token-expensive (a full-window PNG on every turn). MCEC is Windows-only by design, so it
can go deeper than pixels: `query` reads the real Windows UI Automation tree (control type, name,
automation ID, bounds, enabled state, value), letting an agent target and verify a specific control instead
of re-parsing an image every step. `capture` (a real screenshot) is still there for when vision is the
right tool. And because MCEC speaks the **Model Context Protocol (MCP)**, it works with any MCP-capable
agent, Claude, Codex, or a custom client, so a workflow or automated system built on it isn't locked to one
AI provider.

**Q: Is it safe?**

It depends on how you use it, but the defaults are safe. The agent surface is off until you opt in;
every command ships individually disabled; every agent action is audit-logged on screen (the overlay) and
in the log; and a global emergency-stop hotkey lets the operator halt a session instantly. See
[Agent Safety](https://tig.github.io/mcec/safety-emergency-stop-and-provisioning.html) for the full model.

**Q: What's the history of this thing?**

* **3.0 (2026)**: Rebranded to the **Model Context Environment Controller**. Agent automation over MCP:
observation (`capture`/`query`/`record`), targeting (`find`/`wait-for`), actuation
(`invoke`/`launch`/`drag`/`click`), emergency stop, and isolated session provisioning; all opt-in and
off by default.
* **2.x (2019)**: Major rework: robust client/server, User Activity Monitor (occupancy sensing), Commands
Window with built-in test mode, per-monitor DPI support, config in `%APPDATA%`.
* **1.x (2004–2017)**: Born as Media Center Edition Controller (MS later dropped the "Edition" from
"Windows Media Center") for Windows Media Center HTPCs. Grew keyboard/mouse/window-message
simulation, `chars:` with Unicode escapes, serial support, multi-client TCP, and the `.commands`
extension file. Moved from SourceForge to CodePlex to GitHub.

## Integrations

* [Control4 User Activity Driver](https://github.com/tig/User_Activity)
Expand Down
3 changes: 2 additions & 1 deletion SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@ Security fixes target the latest 3.0.x release. Please reproduce on the current

## What is most valuable

MCEC's agent surface is **off by default** and gated behind explicit, independent opt-ins (see
MCEC's agent surface (its **computer use** capability) is **off by default** and gated behind explicit,
independent opt-ins (see
[Agent Safety](docs/safety-emergency-stop-and-provisioning.md)). Reports are especially valuable when they
show a way to:

Expand Down
2 changes: 1 addition & 1 deletion dev/agent-server-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
# Agent Server: Architecture Notes

Developer doc ([`dev/`](README.md)); not published to GitHub Pages. A short tour of the MCEC 3.0 agent
subsystem. It lives under
subsystem (the machinery behind MCEC's **computer use** surface). It lives under
`src/Agent/` and is **additive**: it reuses the existing `Command` / `CommandInvoker`
pattern and adds no breaking changes to any existing command, transport, or default.

Expand Down
22 changes: 14 additions & 8 deletions docs/agent_control.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,21 @@
// Published under the MIT License - Source on GitHub: https://github.com/tig/mcec
-->

# Agent Control
# Agent Control (Computer Use)

MCEC 3.0 turns the MCE Controller daemon into a small, opt-in automation server for
MCEC 3.0 turns the MCE Controller daemon into a small, opt-in **computer use** server (the same
capability Claude, Codex, and similar agents use to see a screen and drive a keyboard and mouse) for
AI agents and scripts running on a Windows PC. It gives an agent three things:

- **Eyes**: capture a screenshot of a window (or the foreground window) as a PNG.
- **Eyes**: far more than a screenshot. `capture` still takes a PNG of a window (or a screen
region), but `query` reads a window's live **UI Automation tree** (control type, name,
automation ID, bounds, enabled/offscreen state, value), `displays` reports per-monitor
geometry (bounds, DPI, scale), and `windows`/`find`/`wait-for` discover and wait on windows
and UI elements by name, class, or automation ID. An agent can see structured state, not
just pixels.
- **Hands**: invoke any existing MCEC command (the actuation layer you already use).
- **A front door**: query/find windows and UI elements, wait for conditions, and
drive all of the above over **MCP** (Model Context Protocol) or a tiny **HTTP** floor.
- **A front door**: drive all of the above over **MCP** (Model Context Protocol) or a tiny
**HTTP** floor, gated off by default and audited when on.

The agent surface is a set of new commands (`capture`, `query`, `displays`, `windows`, `window`, `find`,
`wait-for`, `invoke`, `record`, `launch`, `drag`, `click`, and `focus`) exposed as **tools over MCP/HTTP**
Expand Down Expand Up @@ -541,9 +547,9 @@ rest of the tree is returned.
## Using MCEC from a desktop agent app

MCEC can run **headless** as an MCP **stdio** server (no main window, no tray icon; the
on-screen command overlay and the emergency-stop hotkey still work) so a desktop agent app
(such as a desktop AI assistant or custom MCP client) can spawn it on demand and talk to it over standard
input/output:
on-screen command overlay and the emergency-stop hotkey still work) so a desktop agent app with
computer use capability (such as a desktop AI assistant or custom MCP client) can spawn it on demand
and talk to it over standard input/output:

```
mcec.exe mcp # or the equivalent legacy spelling: mcec.exe --mcp
Expand Down
4 changes: 2 additions & 2 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ Everything MCEC does is configured in one of three places: the **File ▸ Settin
running MCEC, every Settings tab, the agent gates in `mcec.settings`, enabling commands, and logging.

For **installing** MCEC and where its files live, see [Install](install.md); for the **security model**
behind the agent gates, see [Agent Safety](safety-emergency-stop-and-provisioning.md); for the agent
tools themselves, see [Agent Control](agent_control.md).
behind the agent gates, see [Agent Safety](safety-emergency-stop-and-provisioning.md); for the computer
use tools themselves, see [Agent Control](agent_control.md).

If you want to use MCEC from a desktop agent app, see [Agent Control](agent_control.md).
In practice: enable
Expand Down
46 changes: 34 additions & 12 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,36 +6,58 @@
agents on Windows, and the same battle-tested TCP/serial remote control for integration systems it has
always been.

It is a small, self-contained native Windows daemon that a computer-use model can **mount, see through,
and drive**: capture a window as a PNG, read its UI Automation tree, find and wait for controls, launch
apps, and actuate keyboard/mouse/window input, over the **Model Context Protocol (MCP)**. The agent
surface is opt-in and off by default; the 3.0 agent features are purely additive over the classic
remote-control command surface (network and serial), which is unchanged.
It is a small, self-contained native Windows daemon that gives a **computer use** model (the same
capability Claude, Codex, and similar agents use to see a screen and drive a keyboard and mouse)
something to **mount, see through, and drive**: capture a window as a PNG, read its UI Automation tree,
find and wait for controls, launch apps, and actuate keyboard/mouse/window input, over the **Model Context
Protocol (MCP)**. The agent surface is opt-in and off by default; the 3.0 agent features are purely
additive over the classic remote-control command surface (network and serial), which is unchanged.

**Install with winget:** `winget install Kindel.mcec` — see [Install](install.html).

## The chapters

* **[Install](install.html)** install with winget or the signed installer; what gets installed where,
* **[Install](install.html)**: install with winget or the signed installer; what gets installed where,
running side-by-side copies, and disposable provisioned instances for agents.
* **[Configuration](configuration.html)** everything you can configure: the Settings dialog (every
* **[Configuration](configuration.html)** - everything you can configure: the Settings dialog (every
tab), the `mcec.settings` file, the command table, and logging.
* **[Agent Control](agent_control.html)** the agent surface: the observation,
* **[Agent Control](agent_control.html)** - the computer use surface: the observation,
targeting, and actuation tools, the structured result envelope, and the MCP / localhost-HTTP transports.
* **[Agent Safety](safety-emergency-stop-and-provisioning.html)** command-access consent, disposable
* **[Agent Safety](safety-emergency-stop-and-provisioning.html)**: command-access consent, disposable
isolated session provisioning, the emergency-stop hotkey, and the on-screen command overlay.
* **[Remote Control](remote_control.html)** the classic role: listen on TCP/IP or a
* **[Remote Control](remote_control.html)** - the classic role: listen on TCP/IP or a
serial port and translate remote commands into keystrokes, text, mouse, window messages, and app
launches ([Control4](https://www.control4.com/), [iRule](http://www.iruleathome.com/),
[Crestron](http://www.crestron.com/), and others). The same chapter covers the **User Activity Monitor**,
which runs the flow in reverse: it reports when someone is actively using the PC, turning the machine into
an occupancy sensor that can drive lighting and scenes.
* **[Examples](examples.html)** worked agent-driving recipes (hero GIFs, prompt demos) and how to add more.
* **[Examples](examples.html)**: worked agent-driving recipes (hero GIFs, prompt demos) and how to add more.

See also [AGENTS.md](https://github.com/tig/mcec/blob/main/AGENTS.md) for connect-time agent guidance and
the dogfood recipe (MCEC driving MCEC).

## Version history
## Frequently Asked Questions (FAQ)

**Q: Claude, Copilot, and other agent platforms already support computer use. Why would I use MCEC?**

Those platforms aim for cross-platform reach, so their computer use is built primarily on computer
vision: a screenshot, interpreted by a vision model, on every step. That is slow (a network round trip per
observation) and token-expensive (a full-window PNG on every turn). MCEC is Windows-only by design, so it
can go deeper than pixels: `query` reads the real Windows UI Automation tree (control type, name,
automation ID, bounds, enabled state, value), letting an agent target and verify a specific control instead
of re-parsing an image every step. `capture` (a real screenshot) is still there for when vision is the
right tool. And because MCEC speaks the **Model Context Protocol (MCP)**, it works with any MCP-capable
agent, Claude, Codex, or a custom client, so a workflow or automated system built on it isn't locked to one
AI provider.

**Q: Is it safe?**

It depends on how you use it, but the defaults are safe. The agent surface is off until you opt in;
every command ships individually disabled; every agent action is audit-logged on screen (the overlay) and
in the log; and a global emergency-stop hotkey lets the operator halt a session instantly. See
[Agent Safety](safety-emergency-stop-and-provisioning.html) for the full model.

**Q: What's the history of this thing?**

* **3.0 (2026)**: Rebranded to the **Model Context Environment Controller**. Agent automation over MCP:
observation (`capture`/`query`/`record`), targeting (`find`/`wait-for`), actuation
Expand Down
7 changes: 4 additions & 3 deletions docs/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,10 @@ This directory-per-instance isolation is exactly what provisioning (below) autom
## Provisioning: disposable instances for agents

The installed copy under Program Files deliberately **refuses to serve agents**: running `mcec.exe --mcp`
or starting the MCP/HTTP endpoint from Program Files is refused, because enabling the agent gates in your
installed config would leave them enabled if a session crashed. Instead, an authorized agent asks MCEC for
a fresh, disposable copy to drive (there is no `--provision` flag; provisioning is an in-product feature):
or starting the MCP/HTTP endpoint from Program Files is refused, because enabling the computer use gates
in your installed config would leave them enabled if a session crashed. Instead, an authorized agent asks
MCEC for a fresh, disposable copy to drive (there is no `--provision` flag; provisioning is an in-product
feature):

1. Turn on **Allow agents to provision disposable instances** on the Settings dialog's **Agent** tab
(`AllowSessionProvisioning`). This is the one opt-in an operator performs.
Expand Down
3 changes: 2 additions & 1 deletion docs/remote_control.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ MCEC was originally built to integrate a Windows Media Center home-theater PC (H
whole-house A/V system, but it is general enough for any control system that can send text to a TCP/IP
port or serial port.

> **Looking for the agent/AI side?** MCEC 3.0 is also an opt-in automation server for AI agents over the
> **Looking for the agent/AI side?** MCEC 3.0 is also an opt-in **computer use** server (the same
> capability Claude, Codex, and similar agents use to see a screen and drive a keyboard and mouse) over the
> **Model Context Protocol (MCP)**: see the [Agent Control](agent_control.md) and
> [Configuration](configuration.md) chapters. This page covers the classic remote-control features, which are
> unchanged and remain the default behavior.
Expand Down
4 changes: 3 additions & 1 deletion docs/safety-emergency-stop-and-provisioning.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@

# Agent Safety

Four related safety features for MCEC's agent surface. When MCEC is agent-driving the desktop, the **target
Four related safety features for MCEC's agent surface (MCEC's **computer use** capability, in the same
sense Claude, Codex, and similar agents use the term for seeing a screen and driving a keyboard and mouse).
When MCEC is agent-driving the desktop, the **target
app has focus, not MCEC**; so the operator needs (a) a controlled way to grant more capability mid-session,
(b) assurance that a session cannot leave the installed copy in an unsafe state, (c) a way to instantly
intervene when a run goes wrong, and (d) visible narration that MCEC is driving. Command-access consent,
Expand Down
Loading