- What Is This?
- Features
- How It Works
- Demo (Planned)
- Installation
- What Works Today
- Quick Start
- Supported AI Providers
- Configuration
- CLI Reference
- Architecture
- Development
- Roadmap
- FAQ
- Contributing
- License
Project Keyboard turns any vision-capable LLM into a computer-use agent. Point it at a goal and it will:
- Watch your screen at configurable FPS
- Understand what it sees (buttons, text, layouts)
- Decide what action to take next
- Execute clicks, keystrokes, scrolling, and typing
Think of it as a universal macro engine where you describe what you want instead of scripting how to do it.
keyboard start --goal "open Chrome, go to gmail.com, compose an email to alice@example.com saying 'hello'"The agent sees the screen, finds Chrome, clicks it, types the URL, waits for Gmail to load, finds the compose button, and types the message — all autonomously.
👩💻 Developer? Jump to What Works Today for current status, Development for the dev loop, or
docs/plan.mdfor the implementation plan.
- 🖥️ High-FPS Screen Capture — DXGI Desktop Duplication via
dxcam, sub-10ms latency at 30 FPS - 🧠 Multi-Provider LLM — OpenAI GPT-4o, Anthropic Claude, Ollama (LLaVA, Gemma 3, etc.)
- 🔧 Rich Tool Suite —
click,type,keypress,scroll,move,wait,drag - 🛑 Emergency Stop — Press
Escanytime, anywhere. Hardware-level hotkey (WindowsRegisterHotKey) - 💬 Interactive Chat — Send new goals to a running agent from another terminal
- 🏠 Fully Local Mode — Ollama + local models = zero cloud dependency
- 📊 Change Detection — Skips redundant frames, only sends new info to the LLM
- 📝 Action Logging — Append-only JSONL log for debugging and replay
- 🎛️ CLI-First — Every feature accessible from the terminal. No GUI required.
flowchart TD
A[🖥️ Screen Capture\nDXGI / dxcam] --> B[🖼️ Preprocessing\nresize / change-detect / encode]
B --> C{Changed?}
C -->|No| A
C -->|Yes| D[🤖 Vision LLM\nGPT-4o / Claude / Ollama]
D --> E{Tool Call?}
E -->|Yes| F[⚡ Execute Action\nclick / type / scroll / wait]
E -->|No| G[💬 Text Response]
F --> H[📝 Log Action]
G --> H
H --> I{Done / Stopped?}
I -->|No| A
I -->|Yes| J[🛑 Shutdown]
| Stage | Module | What Happens |
|---|---|---|
| Capture | capture.py |
Grabs raw BGR frame from monitor via DXGI. DPI-aware. |
| Preprocess | preproc.py |
Resizes to max 1024px, detects pixel-diff vs previous frame, encodes to base64 JPEG |
| Think | llm.py |
Sends frame + goal + history to vision LLM with tool definitions |
| Act | actions.py |
Parses tool call, executes viapyautogui/pynput, returns result |
| Log | safety.py |
Appends timestamped action to JSONL log for replay/debug |
| Repeat | agent.py |
Loops at configured FPS until goal achieved or stopped |
| OpenAI GPT-4o | Anthropic Claude | Ollama (Local) | |
|---|---|---|---|
| Latency | ~1-3s | ~2-5s | ~0.5-2s (GPU) |
| Cost | ~$0.01/image | ~$0.02/image | Free |
| Vision Quality | Excellent | Excellent | Good (model-dependent) |
| Tool Calling | Native JSON | Native + computer-use | JSON mode |
| Internet Required | Yes | Yes | No |
| Setup | API key | API key | ollama pull llava |
When the full pipeline is complete, here's what a session will look like:
┌─ Terminal 1 (Agent) ────────────────────────────────────────────┐
│ $ keyboard start --goal "search for cats on Google" │
│ │
│ 🚀 Project Keyboard v0.1.0 | Provider: openai | Model: gpt-4o │
│ │
│ [12:00:01] 👁️ Observing... (1920×1080 → 1024×576, 45KB) │
│ [12:00:02] 🤔 LLM: I see the Windows desktop with a Chrome icon. │
│ I'll click the Chrome icon on the taskbar. │
│ [12:00:02] ⚡ click(x=150, y=1056, button="left") │
│ [12:00:03] 👁️ Observing... (frame skipped — no change) │
│ [12:00:04] 👁️ Observing... (1920×1080 → 1024×576, 52KB) │
│ [12:00:06] 🤔 LLM: Chrome is open. I see the address bar. │
│ I'll click it and type the search URL. │
│ [12:00:06] ⚡ click(x=400, y=60, button="left") │
│ [12:00:07] ⚡ type("google.com") │
│ [12:00:07] ⚡ keypress("enter") │
│ [12:00:08] 👁️ Observing... (1920×1080 → 1024×576, 78KB) │
│ [12:00:10] 🤔 LLM: Google homepage loaded. I see the search box. │
│ Typing the query... │
│ [12:00:10] ⚡ type("cats") │
│ [12:00:10] ⚡ keypress("enter") │
│ [12:00:13] ✅ Goal completed: search results for "cats" shown. │
│ │
│ ── Summary ────────────────────────────────────────────────── │
│ Frames captured: 14 | Sent to LLM: 4 | Actions taken: 6 │
│ Duration: 12.4s | Avg LLM latency: 2.1s | Avg FPS: 30.1 │
│ Log saved: ~/.keyboard/actions.log │
└──────────────────────────────────────────────────────────────────┘
┌─ Terminal 2 (Chat) ─────────────────────────────────────────────┐
│ $ keyboard chat "click on the first image result" │
│ → Sent to agent. Response: "Clicked the first cat image." │
└──────────────────────────────────────────────────────────────────┘
This is the target experience. See Roadmap for current progress.
- Windows 10/11 (DXGI Desktop Duplication requires Windows)
- Python 3.10+ (download)
- Git (optional, for cloning)
# Clone the repo
git clone https://github.com/your-username/project-keyboard.git
cd project-keyboard
# Create a virtual environment (recommended)
python -m venv .venv
.venv\Scripts\activate # Windows
# Install with dev dependencies
pip install -e ".[dev]"keyboard --help
# Should show all available commandsProject Keyboard is in active development. Here's what's functional right now:
| Command | Status | What It Does |
|---|---|---|
keyboard setup |
✅ | Interactive wizard — pick provider, set keys |
keyboard config show |
✅ | Display current configuration |
keyboard config set <k> <v> |
✅ | Update a setting (fps, model, provider…) |
keyboard capture --fps N |
✅ | Save raw screen frames to disk for debugging |
keyboard start |
🟡 | Stub — agent loop coming in Phase 2 |
keyboard chat / stop |
⬜ | IPC commands coming in Phase 3 |
| Module | Status | Responsibility |
|---|---|---|
capture.py |
✅ | DXGI screen capture via dxcam |
config.py |
✅ | .env + YAML config loading |
preproc.py |
✅ | Resize, change detection, base64 encoding |
llm.py |
✅ | Provider abstraction (OpenAI/Anthropic/Ollama) |
actions.py |
✅ | Tool schema + action execution |
safety.py |
✅ | Emergency stop (Esc hotkey) |
agent.py |
⬜ | Main observe→think→act loop (Phase 2) |
server.py |
⬜ | HTTP IPC for multi-terminal control (Phase 3) |
# 1. Set up your AI provider (interactive wizard)
keyboard setup
# 2. View your configuration
keyboard config show
# 3. Test screen capture — save 5 seconds of frames at 30 FPS
keyboard capture --fps 30 --duration 5
# 4. Run the test suite
pytest| Provider | Model | Type | Setup |
|---|---|---|---|
| OpenAI | GPT-4o, GPT-4o-mini | Cloud | OPENAI_API_KEY in .env |
| Anthropic | Claude 3.5 Sonnet, Claude 3 Opus | Cloud | ANTHROPIC_API_KEY in .env |
| Ollama | LLaVA 13B, Gemma 3 12B, Llama 3.2 Vision | Local | ollama pull <model> |
keyboard config set provider anthropic
keyboard config set model claude-3-5-sonnet-20241022# Install Ollama (https://ollama.com)
ollama pull llava:13b
# Configure
keyboard config set provider ollama
keyboard config set model llava:13b
# No API keys. No internet. Fully private.
keyboard start --goal "organize my desktop icons"Project Keyboard uses two configuration layers:
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
OLLAMA_HOST=http://localhost:11434provider: openai # openai | anthropic | ollama
model: gpt-4o # Model name
fps: 5 # Frames per second to capture
max_dim: 1024 # Max pixel dimension (resize larger dim to this)
change_threshold: 0.05 # Skip frame if < 5% pixels changed
quality: 85 # JPEG quality (1-100)
action_delay: 0.1 # Seconds between actionskeyboard config show # Print current config
keyboard config set fps 10 # Change capture rate
keyboard config set model gpt-4o-mini # Switch to cheaper model| Command | Description | Example |
|---|---|---|
keyboard setup |
Interactive setup wizard (provider, API keys) | keyboard setup |
keyboard start [--goal "..."] |
Launch agent loop | keyboard start --goal "open VS Code" |
keyboard chat "<message>" |
Send message to running agent | keyboard chat "click the save button" |
keyboard stop |
Gracefully stop the agent | keyboard stop |
keyboard status |
Show agent status & stats | keyboard status |
keyboard config show |
Display current configuration | keyboard config show |
keyboard config set <k> <v> |
Update a config key | keyboard config set fps 10 |
keyboard capture [--fps N] [--duration S] |
Debug: save raw frames to disk | keyboard capture --fps 30 --duration 5 |
--help, -h Show help for any command
--version, -V Show version number
project-keyboard/
├── .github/ # AI development infrastructure
│ ├── copilot-instructions.md # Always-on rules for AI assistants
│ ├── skills/ # Reusable on-demand workflows
│ │ ├── python-dev/ # Python conventions
│ │ ├── cli-designer/ # CLI design patterns
│ │ ├── screen-capture/ # dxcam/DXGI patterns
│ │ ├── llm-integration/ # LLM provider patterns
│ │ └── project-keyboard/ # Master project overview
│ └── agents/ # Specialist subagents
│ ├── python-architect.agent.md
│ ├── cli-designer.agent.md
│ └── project-keyboard-dev.agent.md
├── src/keyboard/ # Main Python package
│ ├── __init__.py # Package marker + version
│ ├── __main__.py # CLI entry point (click)
│ ├── capture.py # ScreenCapture (dxcam/DXGI)
│ ├── preproc.py # Resize, change-detect, encode
│ ├── llm.py # LLMProvider ABC + 3 implementations
│ ├── actions.py # Tool schema + ActionExecutor
│ ├── agent.py # Main observe→think→act loop
│ ├── safety.py # Emergency stop (Esc hotkey)
│ ├── config.py # .env + config.yaml loader
│ └── server.py # HTTP IPC (CLI ↔ running agent)
├── tests/ # pytest test suite
├── docs/
│ └── plan.md # Detailed implementation plan
├── pyproject.toml # Project config + dependencies
├── .env.example # Template for API keys
├── .gitignore
└── README.md # ← You are here
graph LR
A[__main__.py\nCLI Entry] --> B[agent.py\nMain Loop]
B --> C[capture.py\ndxcam]
B --> D[preproc.py\nPIL/Numpy]
B --> E[llm.py\nOpenAI/Anthropic/Ollama]
B --> F[actions.py\npyautogui/pynput]
B --> G[safety.py\npynput/ctypes]
H[config.py\ndotenv/PyYAML] --> B
H --> E
I[server.py\nhttpx] --> B
A --> I
A --> H
style A fill:#4A90D9,color:#fff
style B fill:#D94A4A,color:#fff
style E fill:#4AD97A,color:#fff
| # | Rule | Why |
|---|---|---|
| 1 | Pure Python v1 | No C++, Rust, or other languages. Python only. |
| 2 | Flat package | One concern per file. No deep nesting. |
| 3 | LLM does vision | No OpenCV, no ONNX. The LLM sees and understands. |
| 4 | Continuous loop | Watch → Think → Act → Repeat until goal or stop. |
| 5 | Emergency stop only | Esc hotkey +keyboard stop. No confirmation mode in v1. |
| 6 | Simple over clever | If a pattern adds complexity without clear benefit, skip it. |
pip install -e ".[dev]"ruff check src/ # Lint + format checking
ruff format --check src/ # Format verification
mypy src/ # Strict type checking
pytest # Run all tests
pytest --cov=keyboard --cov-report=html # Coverage reportpytest -m "not slow" # Skip slow tests
pytest -m "not integration" # Skip tests needing real APIs
pytest tests/test_capture.py -v # Run specific test file| Rule | Standard |
|---|---|
| Docstrings | Every public function/class has"""Why this exists.""" |
| Type Hints | On ALL public APIs |
| Linter | ruff (pycodestyle + pyflakes + isort + bugbear + pyupgrade) |
| Line Length | 100 characters |
| Imports | stdlib → third-party → local (isort-enforced) |
| Naming | snake_case functions, PascalCase classes, UPPER_CASE constants |
The typical workflow when hacking on Project Keyboard:
# 1. Make your changes
vim src/keyboard/capture.py
# 2. Lint + type-check
ruff check src/ && ruff format --check src/ && mypy src/
# 3. Run tests (skip slow/integration if needed)
pytest -m "not slow and not integration"
# 4. Run a specific test file with coverage
pytest tests/test_capture.py -v --cov=keyboard.capture --cov-report=termThis project was built with and for GitHub Copilot. The .github/ directory contains:
| Tool | Location | Purpose |
|---|---|---|
| Instructions | .github/copilot-instructions.md |
Always-on rules (auto-loaded) |
| Skills | .github/skills/ |
On-demand workflows (/skill-name) |
| Agents | .github/agents/ |
Specialist subagents (agent picker) |
Create new ones with /create-skill or /create-agent.
| File | Purpose |
|---|---|
docs/plan.md |
Detailed phase-by-phase implementation plan |
.github/copilot-instructions.md |
Always-on AI assistant rules (auto-loaded by Copilot) |
.github/skills/ |
Reusable on-demand workflows for Python, CLI, screen capture, LLMs |
.github/agents/ |
Specialist subagents for code review, CLI design, and feature work |
pyproject.toml |
All dependencies, ruff/mypy/pytest config |
-
ScreenCaptureclass (dxcam/DXGI) - Config system (
.env+config.yaml) -
keyboard setup+keyboard captureCLI - Initial test suite
- Frame preprocessing (resize, change-detect, encode)
- LLM provider abstraction (OpenAI, Anthropic, Ollama)
- Tool definitions + action executor
- Main agent loop (observe → think → act)
- Emergency stop (Esc hotkey)
- HTTP IPC server (
localhost:9876) -
keyboard chat,stop,statuscommands - Config show/set commands
- Windows
RegisterHotKey(cannot be blocked) - Pipeline hooks (on_observe, on_think, on_act)
- Append-only JSONL action log
- Graceful shutdown + atexit cleanup
- Recording & replay from action log
- Voice input via Whisper STT
- Optional confirmation mode
- Cross-platform support (Linux, macOS)
- GUI frontend (Electron/React)
Does this work without an internet connection?
Yes! Use Ollama with a local vision model like llava:13b or gemma3:12b. No API keys, no cloud — everything runs on your machine.
Is this safe? Can it do destructive things?
The agent only has the tools you give it (click, type, scroll, etc.). It can only interact with your desktop the same way you can. Press Esc at any time for an instant emergency stop.
We recommend:
- Running with visible feedback (watch the screen while it works)
- Starting with simple, reversible goals
- Not running as administrator
How much does it cost to run?
| Provider | Approximate Cost |
|---|---|
| OpenAI GPT-4o | ~$0.50–$2.00 / hour of agent runtime |
| Anthropic Claude | ~$1.00–$4.00 / hour |
| Ollama (local) | $0 (your electricity) |
Use gpt-4o-mini for significant cost savings at slightly reduced accuracy.
What can I use it for?
- Automating repetitive UI tasks
- Testing GUI applications
- Accessibility (control PC via natural language)
- Data entry automation
- "Computer, do X" — general-purpose desktop automation
Why Python? Why not Rust/C++?
v1 optimizes for development speed and iteration. The LLM is the bottleneck (1-3 seconds per inference), not capture or execution. If profiling shows Python is the bottleneck, we can optimize hot paths later.
Contributions are welcome! Here's how:
- Fork the repository
- Create a feature branch (
git checkout -b feat/amazing-idea) - Write code following the code style guide
- Add tests for new functionality
- Run quality gates:
ruff check src/ && mypy src/ && pytest - Commit (
git commit -m "feat: add amazing idea") - Push (
git push origin feat/amazing-idea) - Open a pull request
We use Conventional Commits:
feat:— new featurefix:— bug fixdocs:— documentationtest:— testsrefactor:— code restructuringchore:— build/tooling
MIT — see LICENSE for details.