agent-harness

A framework-free, per-task agent orchestrator. It watches a queue directory for markdown task notes, runs each one as a headless Claude Code agent (subscription auth — no API key, no cloud SDK), gives the agent its tools through a real MCP server, and writes the deliverable back as a markdown done-note with cost, turn and termination accounting. Around the core sits an evaluation harness that scores the orchestrator's real output — deterministic checks fused with an evidence-grounded LLM judge — and a small CI pipeline.

queue/*.md ──► orchestrator.py ──► claude -p  (drives its own agent loop)
                    │                  │
                    │                  └── MCP tools (orch_tools_mcp.py):
                    │                      notes search/read · web search · grounded QA
                    │                      fleet probe · sandboxed python · workspace exec
                    │
                    └──► done/<note>.md  + transcript + evidence log
                              │
                              └──► eval/  (checks + judge + live fact-check)

Why framework-free

This started life as an AutoGen (autogen-agentchat 0.7.5) multi-agent team: planner, researcher, writer, router — 1,253 lines of SelectorGroupChat machinery. It was deleted after measurement, not on a whim:

• AutoGen went abandoned upstream (last release 2025-09-30, zero commits for months).
• At ~5 tasks/day, a hierarchical team was machinery that a single claude -p call subsumes: Claude Code already runs a tool loop, retries, and reports cost, turns and termination class natively.
• The only sanctioned way to drive subscription-auth Claude headless is the CLI — every framework's model layer is API-key-only. claude_sub_client.py (the retired AutoGen bridge, kept as a record of the migration) shows how far you have to bend a framework to avoid that; the answer was to stop.
• Prompt-injected "virtual" tools were declined by the model as unavailable; the same tools exposed as a real MCP server are called reliably (measured 2026-08-07). So the tools moved into orch_tools_mcp.py and the hand-rolled turn loop was deleted.

The code comments carry the measurements behind each of these calls — they are the documentation of record.

Safety model (mechanical, not prose)

• Tool allowlist is the gate: --allowedTools is scoped to mcp__orch__* and built-ins are disallowed. Measured: a skill instructing the agent to run Bash and write a file was denied at the tool layer.
• No host shell, by measurement: PreToolUse hooks (including any secret-guard) do not fire in headless claude -p. A host Bash tool would therefore be unguarded by construction — so execution is containerised. run_python and the execute-lane workspace run in disposable Docker containers with no network, tight resource caps, and (for the workspace) exactly one bind-mounted directory.
• One outward-facing tool: gh_create_issue, gated on a configured repo allowlist that is empty by default.
• Untrusted content is framed: everything from the web arrives wrapped in <untrusted_external_content> delimiters, and deliverables are sanitised for exfil vectors (image beacons, payload-bearing links, active HTML) before they leave the model boundary.
• Cost gates: explicit --model, a per-task --max-budget-usd, and the CLI's real total_cost_usd recorded on every note.

test_exec_lane.py is the adversarial test suite for these gates (path traversal, container isolation, no-network, orphan reaping, allowlist injection).

The eval harness

eval/ scores real done-notes with two fused signals — deterministic checks (convergence, stub detection, citations-when-asked, cost anomalies, tool-use sanity) and an LLM-as-judge scoring a 5-dimension rubric — plus golden and adversarial calibration cases, per-skill trigger-routing evals, baseline regression gating, and a judge↔human calibration loop.

The judge is deliberately not reference-free. Two inputs make its factual_grounding score a measurement instead of a memory test:

• Recorded evidence — the MCP server logs every tool call and result per run, so the judge scores attributability to what the agent actually retrieved.
• A live fact-check (eval/factcheck.py) — checkable claims are extracted from the deliverable and actually searched, and the results go into the judge prompt. This exists because a reference-free judge flagged six real releases as "fabricated" purely because they post-dated its training cutoff — an LLM judge scoring factual grounding from memory is largely scoring its own cutoff. Full write-up: The LLM-judge cutoff problem.

eval_orchestrator.py is a separate 13-case end-to-end suite that runs real tasks through the live agent loop and asserts mechanically (no judge): does each tool actually get used, do the safety boundaries hold under pressure (secret reads, outward sends, prompt injection via a planted note), does the sandbox leak containers, does an ops task stay read-only.

Honest cost note

The end-to-end suite and the LLM judge drive a live, logged-in claude CLI and cost real subscription usage (~US$0.30 for a full run of the 13-case suite; a judged sweep is similar). They are not free-runnable in CI. CI runs only what is deterministic and free: compilation plus the golden suite's regex checks (python -m eval.run_eval --no-judge golden).

Repo layout

orchestrator.py        the per-task runner (entry point)
orch_tools.py          tool implementations — pure functions, no framework, no LLM
orch_tools_mcp.py      FastMCP server exposing the tools to Claude Code
claude_sub_client.py   RETIRED AutoGen-era subscription bridge (kept as migration record)
eval_orchestrator.py   13-case live end-to-end suite (tools / safety / robustness)
test_exec_lane.py      adversarial gates for the containerised execute lane
sandbox/Dockerfile     the execute-lane workspace image (no network, minimal tools)
run.sh                 systemd path/timer entrypoint (flock + hard timeout + auth preflight)
eval/                  evaluation harness — see eval/README.md
docs/                  ADR + spec-driven development workflow
.github/               CI (compile + deterministic golden suite) + issue/PR templates

Configuration

Everything deployment-specific is an environment variable with a sane default:

Variable	Meaning	Default
AGENT_HARNESS_ROOT	repo/deploy root	the code's own directory
CLAUDE_BIN	Claude Code CLI binary	claude on PATH
NOTES_DIR	the notes store (an Obsidian-style vault) the tools read	<root>/notes
QUEUE_DIR	watched task-queue directory	<NOTES_DIR>/inbox/task-queue
SEARXNG_URL	local SearXNG search endpoint	http://127.0.0.1:8888/search
QDRANT_URL	Qdrant (notes semantic index)	http://127.0.0.1:6333
FLEET_PROBE_CMD	read-only host/fleet status script	unset → tool reports unconfigured
AGENT_WORKSPACE_DIR	execute-lane workspace (bind-mounted)	<root>/workspace
GH_ISSUE_REPO_ALLOWLIST	comma-separated repos gh_create_issue may touch	empty → refuses all
SLACK_BOT_TOKEN / SLACK_CHANNEL	optional done/failed notifications	unset → no-op
AGE_KEY_FILE / AGE_SECRETS_FILE	age-encrypted secrets (optional API keys)	~/.age-key / ~/.secrets/secrets.env.age
CLAUDE_CLI_DIR	dir containing the claude_cli wrapper the eval judge uses	unset → judge degrades
STAFF_PLUGINS_DIR	per-task-type skill plugin bundles	<root>/plugins

Running it

python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
docker build -t orch-workspace:1 sandbox/          # execute-lane image

# one task, directly
.venv/bin/python orchestrator.py --task my-task.md --dry-run

# queue mode (what the systemd path/timer unit calls)
./run.sh

# evaluate recent deliverables
.venv/bin/python -m eval.run_eval sweep --n 10

Task notes are markdown with optional key: value frontmatter (type: research|ops|code, model: haiku|sonnet|opus); the body is the task.

License

MIT — see LICENSE.