🧠 Smart MCP Proxy — Hot-Swap MCP Aggregation + AI Concierge

One endpoint. Every MCP server. Zero restarts. Zero API keys.
Aggregate all your MCP servers into a single StreamableHTTP endpoint — add or remove servers at runtime,
share subprocess pools across agents, and let the optional AI concierge route plain-language queries.
Your enterprise MCP backbone — hot-swappable, shared, and agentic by design.

by Ringo / MilkyWay008 · github.com/MilkyWay008/Smart-MCP-Proxy · v1.0.0

📑 Table of Contents

💡 Why Smart MCP Proxy?
✨ Key Features
🧩 Problem 1 & 2 — Restarts & Resource Waste
🤖 Problem 3 — Dumb Tools Drain Agent Context
🏢 Beyond Tools — Enterprise IT Infrastructure
⚙️ Architecture Overview
🔧 Build 1 — Hot-Swap MCP Proxy
🧠 Build 2 — Smart MCP Layer (Concierge)
📚 Skill Templates
🚀 Quick Start
🖥️ CLI & Scripts
🔌 Connecting from Hermes
⚙️ Configuration
🧪 Testing
📁 Project Structure
🗺️ Future Work
📦 Dependencies
🔍 Search Keywords

💡 Why Smart MCP Proxy?

Your agent is smart. Your MCP servers are dumb tools — and dumb tools burn context, waste memory, and demand restarts every time you touch a config.

Smart MCP Proxy fixes the three things that break agentic systems:

Restarts on every MCP config change → gone. Edit a YAML file; the proxy hot-swaps servers live.
Resource waste — 3 agents × 7 servers = 21 subprocesses → 7 shared pools, spawned on demand, cleaned up when idle.
Context drain — agents juggling raw tools, params, and multi-step chains → the MCP Concierge does it for them. The agent just talks.

🧠 Think of it as an MCP concierge — tell it what you want done (search, desktop automation, file ops, app integration) and it handles the rest, returning only the final result.

✨ Key Features

🔄 Hot-Swap at Runtime

Add or remove MCP servers without restarting the proxy or your agent
File watcher (2s debounce) diffs config → connects/disconnects pools live
Perfect for managing MCP servers remotely — even over Telegram, away from your desk

🏗️ Shared, Efficient Subprocess Pools

One pool per server, shared across all connected agents — 3 agents + 7 servers = 7 pools, not 21
Overflow spawning: busy pool → spawn an extra subprocess on demand (spawn_timeout)
Idle cleanup: overflow subprocesses killed after inactivity (idle_timeout)
Crash recovery: 3 retries with 1s/2s/4s exponential backoff

🧠 Smart MCP Layer (MCP Concierge) — Optional

Two ways to use MCP: direct tools (proxy_windows-mcp__Screenshot) or plain language (mcp_proxy_ask("..."))
Routes queries via MCP Sampling — borrows the agent's own LLM (no API key embedded)
Falls back to keyword/skill matching when Sampling is unavailable
Multi-step orchestration: auto-chains follow-ups (compare → debate → synthesize), returns only the final result
Zero API keys — clone, run, works offline, portable across all agentic systems

🛡️ Production-Ready

Concurrent access via configurable semaphore pools
Per-call timeouts, full tool schemas preserved, images/binary supported
Rotating logs (5MB × 3), separate smart-layer logger
Zero infrastructure: no database, no web UI, no Docker — single Python process
Cross-platform: Windows, Linux, macOS with startup scripts for all

🧩 Problem 1 & 2 — Restarts & Resource Waste

The Restart Problem

Every time you add, remove, or modify an MCP server in Hermes, you must edit config.yaml, restart the gateway, and possibly restart the desktop app. Unreasonable when you're setting up an MCP server over Telegram while away from your desk.

The Resource Problem

Multi-agent MCP pairing makes it worse. If 3 agents each connect directly to 7 MCP servers, that's 21 subprocesses — each eating memory. Heavy servers like Playwright or windows-mcp make this unsustainable.

✅ The Solution — Build 1: A Shared MCP Proxy

Hot-swap: Edit a YAML config; file watcher diffs, connects/disconnects. No agent restart.
Resource sharing: One subprocess pool per server, shared across all agents. Overflow spawned on demand, cleaned up after idle.
Zero infrastructure: No database, no web UI, no Docker — single Python process.
Cross-platform: Windows, Linux, macOS with startup scripts.

The proxy supports concurrent access via configurable semaphore pools, enforces per-call timeouts, auto-recovers from subprocess crashes, and preserves full tool schemas and content types (including images).

🤖 Problem 3 — Dumb Tools Drain Agent Context

As powerful as MCP is, the more powerful the servers become, the more they consume the agent's context window. Every tool call — discovery, parameter juggling, result parsing — burns tokens and reasoning cycles, and accelerates hallucination. The agent almost always needs multiple round-trips just to get a simple result from a complex MCP server.

Savvy users write custom skills to reduce tool-call overhead, but:

Even with skills, the agent's context still carries intermediate results, error handling, and multi-step orchestration
Skills must be copied and debugged for each agentic system (Hermes, Cursor, Claude Desktop, etc.)
Non-savvy users cannot write skills at all — locked out of effective MCP usage
The burden is on the user to know which tool to call, with which parameters, in what order

In short: we have smart agents, but we have dumb tools. Dumb tools add burdens on agents and users.

✅ The Solution — Build 2: Smart MCP Layer (MCP Concierge)

An optional smart layer that wraps the proxy. When enabled, the agent gets a second way to interact with MCP servers — by simply talking:

Mode	How	For
A — Direct Tools	Existing MCP tools (`proxy_windows-mcp__Screenshot`, …)	Skill-guided or power users
B — Smart A2A	`mcp_proxy_ask(query)` — plain language	Everyone

The smart layer handles the heavy lifting:

Identifies the target MCP server from your query
(Optional) Loads matching skill templates from skills/<server-name>/<action>.md
Scores and selects the best skill by keyword matching
Infers parameters from the query (providers, formats, rounds, etc.)
Executes the tool call
Optionally chains follow-up steps (e.g., compare → debate → synthesize)
Returns only the final result — the agent's context is spared all intermediate noise

Routing: MCP Sampling is the primary method — the proxy asks the connected agent's own LLM via session.create_message(). No API key embedded. If the client doesn't support Sampling (or times out after 15s), it falls back to keyword matching.

Every response includes which server was used and the confidence level.

✅ Zero API keys embedded in the proxy
✅ Anyone can clone and run without configuring an LLM provider
✅ Works offline
✅ Portable across all agentic systems (Hermes, Cursor, Claude Desktop, etc.)

Skill templates are just markdown files in skills/<server-name>/. Create, edit, or delete them without touching any Python code. See SMART_LAYER_SKILLS_GUIDE.md for a complete walkthrough using mcp-rubber-duck as an example.

🏢 Beyond Tools — Enterprise IT Infrastructure

Imagine every machine in your organization — every PC, every server, every remote workstation — running one of these proxies, forming a cascading mcp-proxy hierarchy:

Your agent sees all 100,000 machines. A department head's agent sees only their 100.
Each machine has its own computer-use tools (screenshots, commands, files) and its own local reasoning.
Machine-to-machine communication over MCP, authenticated by token, organized by org structure.
Agentic remote brain. Local hands.

This isn't a tool. It's an operating system for enterprise agentic IT operations. See mcp-proxy-true-potential.md for the full vision.

⚙️ Architecture Overview

Agent (Hermes, Cursor, etc.)
         │
         │ StreamableHTTP :9876
         ▼
┌─────────────────────────────────────┐
│        Smart MCP Proxy              │
│                                     │
│  StreamableHTTP endpoint            │
│  ┌─────────────────────────────┐    │
│  │  proxy_* tools (direct)      │    │
│  │  mcp_proxy_ask (concierge)   │    │
│  │  proxy_add_server (manage)   │    │
│  │  proxy_remove_server (manage)│    │
│  │  proxy_reload_config (manage)│    │
│  │  proxy_list_pools (status)   │    │
│  └──────────┬──────────────────┘    │
│             │                       │
│  ┌──────────▼──────────────────┐    │
│  │ Connection Manager          │    │
│  │ ┌──────────┐ ┌──────────┐   │    │
│  │ │ rubber-  │ │ windows- │   │    │
│  │ │ duck     │ │ mcp      │   │    │
│  │ │ (stdio)  │ │ (stdio)  │   │    │
│  │ └──────────┘ └──────────┘   │    │
│  │ ┌──────────┐ ┌──────────┐   │    │
│  │ │ composio │ │ tinyfish │   │    │
│  │ │ (http)   │ │ (stdio)  │   │    │
│  │ └──────────┘ └──────────┘   │    │
│  └─────────────────────────────┘    │
│                                     │
│  File Watcher (watchdog)            │
│  ┌─────────────────────────────┐    │
│  │ proxy-config.yaml changes   │    │
│  │ → diff → add/remove pools  │    │
│  └─────────────────────────────┘    │
│                                     │
│  Smart Layer (optional)             │
│  ┌─────────────────────────────┐    │
│  │ SkillRouter → executor      │    │
│  │ skills/rubber-duck/*.md     │    │
│  │ No API key needed           │    │
│  └─────────────────────────────┘    │
└─────────────────────────────────────┘

🔧 Build 1 — Hot-Swap MCP Proxy

Core Components

Pool (src/pool.py)

Manages multiple downstream sessions per server with configurable pool
Overflow spawning: Each server starts with pool.min subprocesses (default: 1). When all are busy, the proxy waits spawn_timeout (default: 10s), then spawns another. Multiple agents hit the same server simultaneously — each gets their own subprocess.
Idle cleanup: Overflow subprocesses (beyond pool.min) killed after idle_timeout seconds (default: 60)
Resource sharing: One pool per server, shared across all agents — 3 agents + 7 servers = 7 pools, not 21
Discovers tools, resources, and prompts on connection
Timeout enforcement via asyncio.wait_for()
Crash recovery: 3 retries with 1s/2s/4s exponential backoff

PoolManager (src/manager.py)

Creates and manages Pool instances from config
Parallel start/close using asyncio.gather
Tool aggregation across all pools
Server status reporting

Server (src/server.py)

FastMCP StreamableHTTP endpoint on configurable host:port
Dynamic tool registration using app.add_tool() with proper parameter schemas
File watcher hot-reload (watchdog-based, 2s debounce)
Optional smart layer registration
Persistent file logging with rotation (5MB, 3 backups)
Instructions injected into agent's system prompt

Config (src/config.py)

YAML config file with Pydantic validation
Tilde expansion for portable paths (~/.mcp_servers/...)
Supports both dict and list server formats

🔄 Hot-Swap Mechanism

watchdog.Observer monitors proxy-config.yaml
On file change (2s debounce): diff old vs new server list
Removed servers → close pool, unregister tools
Added servers → create pool, start, register tools
No restart needed on either proxy or Hermes side

🛠️ Tool Registration

Each downstream tool is registered with:

Prefixed name: proxy_{server}___{tool}
Full parameter schema preserved (not generic arguments: object)
Dynamic inspect.Signature built from downstream tool's inputSchema
Images and binary content serialized as JSON

👁️ File Watcher

proxy-config.yaml changes ──[2s debounce]──▶ diff config
                                                  │
                          ┌───────────────────────┤
                          ▼                       ▼
                   Remove servers            Add servers
                   close pool                create pool
                   unregister tools          start pool
                                             register tools

🧠 Build 2 — Smart MCP Layer

Overview

Build 2 is an optional add-on that adds a natural-language query tool (mcp_proxy_ask) to the proxy — acting as an MCP concierge for agents. Enabled with --enable-smart.

Smart Router (`src/smart/router.py`)

Scans skills/ directory at startup
Matches queries to server-specific skill templates
Returns skill content for the best match

Executor (`src/smart/executor.py`)

Entry point: handle_ask(query, manager, router)

Flow:

Server identification — match query against server names, tool names, skill content
Skill matching — score skill templates by keyword overlap with query
Tool selection — find the best tool based on skill guidance
Argument inference — extract parameters from natural language
Execution — call pool.call_tool() with inferred arguments
Result — return serialized content as JSON

🔑 No API Key Embedded

The smart layer uses MCP Sampling as the primary router, with keyword matching as fallback:

session.create_message() (Sampling) — borrows the agent's own LLM to route queries
_find_server() — matches against server names, tool names, and skill content
_match_skill() — scores skills by keyword overlap with the query (fallback)
_build_arguments() — infers parameters using type-aware extraction
_infer_argument() — handles common patterns (providers, formats, rounds, booleans)

Zero API keys embedded. Anyone can clone and run without configuring an LLM provider.

🔗 Multi-Step Orchestration

The smart layer supports fire-and-forget research chains. When a skill template includes a Follow-up: line, the executor automatically chains tool calls:

Primary: compare_ducks(providers=[...], prompt="research topic")
  → Follow-up: ask_duck(provider="grok", prompt="Synthesize: {previous_result}")
  → Returns only the synthesized report — agent never sees intermediate results

Safety limits:

MAX_CHAIN_DEPTH = 4 — prevents infinite follow-up loops
MAX_PREVIOUS_RESULT_CHARS = 50000 — truncates oversized primary results
Image content (base64) stripped from follow-up context automatically
If follow-up fails, primary result is returned instead

📝 Separate Logger

Smart layer logs to logs/smart-layer.log with its own rotating handler (5MB, 3 backups).

📚 Skill Templates

Skills are plain markdown files in skills/<server-name>/<action>.md that guide the smart layer on how to use each tool.

skills/
└── rubber-duck/
    ├── ask.md           # duck_ask — single question to one provider
    ├── debate.md        # duck_debate — structured multi-round debate
    ├── compare.md       # compare_ducks — side-by-side responses
    ├── council.md       # duck_council — all providers (deprecated)
    ├── judge.md         # duck_judge — rank provider responses
    ├── vote.md          # duck_vote — multi-option voting
    ├── list_ducks.md    # list_ducks — provider health check
    ├── iterate.md       # duck_iterate — iterative refinement
    └── hybrid.md        # compare → debate → king duck verdict

Each skill file contains:

Tool name (markdown heading H1)
Best-for scenarios
Usage example with parameters
Parameter descriptions

When the smart layer receives a query, it checks skills/<server-name>/ for matching files, scores them against the query, and uses the best match to guide tool execution.

Add skills by dropping .md files — no code changes needed.

🚀 Quick Start

Step 1 — Install dependencies

pip install mcp fastmcp pydantic pyyaml watchdog click uvicorn httpx

Step 2 — Start the proxy

cd ~/.mcp_servers/smart-mcp-proxy

# Start with smart layer (recommended)
python -m src --enable-smart

# Or via scripts:
bin/smart-mcp-proxy.cmd        # Windows
bin/smart-mcp-proxy.sh         # Linux/macOS

Step 3 — Connect from Hermes

Add to any Hermes profile's config.yaml:

mcp_servers:
  smart-mcp-proxy:
    url: "http://localhost:9876/mcp"

No restart needed if gateway is already running — tools appear after /reload-mcp.

First Run

On first run, the proxy creates logs/smart-mcp-proxy.log (proxy logs) and logs/smart-layer.log (smart layer logs, if enabled). Both rotate at 5MB with 3 backups.

🖥️ CLI & Scripts

`bin/smart-mcp-proxy.cmd` (Windows)

bin\smart-mcp-proxy.cmd            start (default)
bin\smart-mcp-proxy.cmd stop       stop
bin\smart-mcp-proxy.cmd restart    restart
bin\smart-mcp-proxy.cmd status     check if running

`bin/smart-mcp-proxy.sh` (Linux/macOS)

bin/smart-mcp-proxy.sh              start (default)
bin/smart-mcp-proxy.sh stop         stop
bin/smart-mcp-proxy.sh restart      restart
bin/smart-mcp-proxy.sh status       check if running

Both scripts:

Use PID file in project root
Fall back to port detection if PID file is stale
Auto-create Python venv on first run (Linux/macOS)
Resolve proxy root from bin/ up one level
Poll for port up to 25s on start (handles slow downstream server connections)
Use a separate startup.log to avoid file lock conflicts with Python's RotatingFileHandler
Send all output to stderr for compatibility across cmd, PowerShell, and MSYS2

Once bin/ is in PATH, scripts run from anywhere:

smart-mcp-proxy start      # instead of bin\smart-mcp-proxy.cmd start
smart-mcp-proxy status     # works from any directory

Manual Start

python -m src                              # Build 1 only
python -m src --enable-smart               # Build 1 + Build 2
python -m src --host 127.0.0.1 --port 9876 --log-level info

🔌 Connecting from Hermes

Add to any Hermes profile's config.yaml:

mcp_servers:
  smart-mcp-proxy:
    url: "http://localhost:9876/mcp"

No restart needed if gateway is already running — tools appear after /reload-mcp.

⚙️ Configuration

`proxy-config.yaml`

proxy:
  host: "127.0.0.1"
  port: 9876
  log_level: "info"

servers:
  my-server:
    type: stdio                          # stdio, streamable_http
    command: "~/.mcp_servers/xxx/cmd"    # tilde-expanded automatically
    args: ["--flag"]                     # optional
    env:                                 # optional
      API_KEY: "sk-xxx"
    timeout: 120                         # call timeout in seconds

Both dict and list formats are supported for the servers key.

🧪 Testing

python -m unittest discover -s tests

📁 Project Structure

~/.mcp_servers/smart-mcp-proxy/
├── bin/
│   ├── smart-mcp-proxy.cmd       # Windows CLI
│   └── smart-mcp-proxy.sh        # Linux/macOS CLI
├── logs/
│   ├── smart-mcp-proxy.log       # Proxy logs (rotating)
│   ├── smart-layer.log           # Smart layer logs (rotating)
│   └── startup.log               # Startup output (rotating)
├── skills/
│   └── rubber-duck/              # Skill templates
│       ├── ask.md
│       ├── debate.md
│       ├── compare.md
│       ├── council.md
│       ├── judge.md
│       ├── vote.md
│       ├── list_ducks.md
│       ├── iterate.md
│       └── hybrid.md
├── src/
│   ├── __init__.py
│   ├── __main__.py               # Entry: python -m src
│   ├── cli.py                    # CLI argument parsing
│   ├── config.py                 # YAML config with tilde expansion
│   ├── models.py                 # Pydantic data models
│   ├── server.py                 # FastMCP endpoint + tool registration
│   ├── manager.py                # Pool lifecycle management
│   ├── pool.py                   # Per-server subprocess pool
│   ├── watcher.py                # File watcher for hot-reload
│   └── smart/                    # Build 2: Smart layer
│       ├── __init__.py
│       ├── router.py             # Skill loading and query routing
│       ├── executor.py           # Tool selection and execution
│       └── log.py                # Separate logger
├── tests/
│   ├── test_config.py
│   ├── test_pool.py
│   ├── test_server.py
│   ├── test_watcher.py
│   ├── test_manager.py
│   ├── test_router.py
│   ├── test_executor.py
│   ├── test_executor_bugs.py
│   └── test_multi_step.py
├── proxy-config.example.yaml      # Template (no secrets)
├── build-3-big_picture-infinite_standby_mcp.md  # Build 3 design doc
├── build-4-big_picture-auth_addon.md             # Build 4 design doc
├── build-5-big_picture-ssh-mcp.md                # Build 5 design doc
├── mcp-proxy-true-potential.md                   # Enterprise vision doc
├── SMART_LAYER_SKILLS_GUIDE.md
├── QUICKSTART.md
└── README.md (this file)

🗺️ Future Work

Feature	Status	Notes
Build 3 — Infinite Standby MCP	📋 Planned	On-demand MCP servers: load only when accessed, cache tool definitions, cooldown after idle. See `build-3-big_picture-infinite_standby_mcp.md`
Build 4 — Auth & Access Control	📋 Planned	HTTPS, token or username/password auth, multi-user support, rate limiting, audit logging. See `build-4-big_picture-auth_addon.md`
Build 5 — SSH-MCP (standalone)	📋 Planned	SSH + tmux wrapped as a FastMCP server for remote command execution, file transfer, persistent sessions. Standalone project designed to pair with the proxy + auth. See `build-5-big_picture-ssh-mcp.md`
Crash recovery hardening	📋 Planned	Metrics, circuit breakers, exponential backoff tuning
Hermes cron/kanban integration	📋 Planned	Automated proxy lifecycle management

📦 Dependencies

All runtime dependencies are pure Python and install via pip:

Package	Version	GitHub	Purpose
mcp (Python SDK)	≥1.0	https://github.com/modelcontextprotocol/python-sdk	Core MCP client/server protocol (stdio, SSE, StreamableHTTP transports)
FastMCP	≥3.4	https://github.com/PrefectHQ/fastmcp	MCP server framework — tool registration, StreamableHTTP endpoint
pydantic	≥2.0	https://github.com/pydantic/pydantic	Config validation via declarative data models
PyYAML	≥6.0	https://github.com/yaml/pyyaml	Parse `proxy-config.yaml`
watchdog	≥4.0	https://github.com/gorakhargosh/watchdog	File system watcher for hot-reload
click	≥8.0	https://github.com/pallets/click	CLI entry point
uvicorn	≥0.30	https://github.com/encode/uvicorn	ASGI server for StreamableHTTP endpoint
httpx	≥0.27	https://github.com/encode/httpx	Async HTTP client (used by MCP SDK internally)

pip install mcp fastmcp pydantic pyyaml watchdog click uvicorn httpx

🔍 Search Keywords

Smart MCP Proxy · MCP proxy · MCP aggregator · hot-swap MCP · MCP server management · MCP concierge · AI agent MCP · StreamableHTTP MCP · MCP tool aggregation · shared MCP pools · zero API key MCP · MCP sampling router · natural language MCP · agentic IT operations · MCP for Hermes · MCP for Cursor · MCP for Claude Desktop · enterprise MCP proxy · MCP resource sharing · smart layer MCP

📝 License

MIT — free to use, modify, and distribute.

Smart MCP Proxy — your enterprise MCP backbone: hot-swappable, shared, and agentic by design.
_{One endpoint. Every MCP server. Zero restarts. Zero API keys. 🧠}

Name		Name	Last commit message	Last commit date
Latest commit History 18 Commits
bin		bin
skills/rubber-duck		skills/rubber-duck
src		src
tests		tests
.gitignore		.gitignore
QUICKSTART.md		QUICKSTART.md
README.md		README.md
SMART_LAYER_SKILLS_GUIDE.md		SMART_LAYER_SKILLS_GUIDE.md
build-3-big_picture-infinite_standby_mcp.md		build-3-big_picture-infinite_standby_mcp.md
build-4-big_picture-auth_addon.md		build-4-big_picture-auth_addon.md
build-5-big_picture-ssh-mcp.md		build-5-big_picture-ssh-mcp.md
mcp-proxy-true-potential.md		mcp-proxy-true-potential.md
proxy-config.example.yaml		proxy-config.example.yaml

Folders and files

Latest commit

History

Repository files navigation