Self-Improving Harness Experiment

Background, motivation, and learnings: blog post.

A self-improvement experiment in two layers:

• Inner — the harness (src/harness/core.py): an LLM shell agent that solves tasks behind a common HarnessEnv interface. The checked-in default uses SWE-bench Verified; Harbor/Terminal-Bench remains available via config.
• Outer — the supervisor (uv run auto): a coding agent (Codex or Claude Code) that proposes one focused change to the inner harness per cycle, measures it against the current baseline, and keeps or discards it on statistical evidence.

Architecture: TECH_DESIGN.md. Operating policy the outer agent follows: program.md.

Setup

Requires Python 3.13+, Docker, uv, and a source checkout.

uv sync
source .venv/bin/activate

Configure the harness in config/harness_config.json; config/harness_config.template.json is the commented reference. uv run exp and uv run auto always read config/harness_config.json; switch substrates by editing its env_backend.

Backend task names:

• env_backend: "swe": task_names are SWE-bench Verified instance_ids from the test split at https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified.
• env_backend: "harbor": task_names are Terminal-Bench 2 task IDs from https://www.tbench.ai/benchmarks/terminal-bench-2, with local task_overrides/<task_id>/ taking precedence.

Credentials, per layer:

• Task-solving LLM (required for everything — every task step calls a model):
  • openrouter: set OPENROUTER_API_KEY in your shell or a repo-local .env. If the checked-in provider routing is unavailable to your account, adjust llm_provider_config.provider_kwargs.provider.
  • chatgpt_codex: run codex login once; auth is read from CODEX_HOME/auth.json or ~/.codex/auth.json.
  • Local or other models: implement src/llm/base.py.
• Outer agent (only for uv run auto): an authenticated codex (default) or claude CLI.

Run a one-off experiment

uv run exp runs the configured task panel against whatever code is checked out and writes one raw record to experiments/<id>/experiment.json — no baseline, no gating.

It uses paid/quota-limited LLM calls, needs Docker running, and requires a clean worktree (EXP_ALLOW_DIRTY_WORKTREE=1 bypasses, for throwaway runs only).

uv run exp

Progress streams to the terminal:

[####--------------------] 10/49 tasks (20%) | trials 28/105 | solved 9/12 | errors 0 | active 8 | 1m12s elapsed, ~4m40s left

Run the self-improvement loop

uv run auto                 # codex agent (default)
uv run auto --agent claude

Each cycle: the outer agent edits src/harness/core.py in a sparse worktree → the candidate runs in a throwaway sibling worktree → a statistical gate keeps (fast-forwards onto the primary) or discards → the agent writes a diagnosis.

Use a dedicated checkout. uv run auto:

• has no budget cap — it runs until Ctrl+C or an error, spending on both the outer agent and the task-solving LLM
• runs agent CLIs without permission prompts
• requires a clean, committed HEAD; working dirs (worktrees, an isolated codex-home symlinked to your ~/.codex auth) live under ../harness-experiment_supervisor/

It streams the agent's thinking and tool calls, then the same per-run progress bar as exp:

[supervisor] loop_iteration_started
[claude] I'll read the authoritative files to understand the current state...
  [toolcall] read .../workspace/program.md
  [toolcall] read .../workspace/src/harness/core.py

Results

• experiments/learning.md — the cumulative agent-written diagnosis memo. Start here.
• experiments/<id>/experiment.json — a run's raw outcome; loop.json next to it — the keep/discard decision with per-task evidence.
• Full artifact tree (per-trial logs, verifier output, supervisor dirs): TECH_DESIGN.md → Artifact layout.

Optional

Local package caches

For multi-trial runs, scripts/setup-apt-cache.sh reduces repeated apt bootstrap stalls. For ML-heavy tasks, scripts/setup-pypi-cache.sh reduces repeated verifier dependency downloads.

Task overrides

For env_backend: "harbor", drop a valid Harbor task layout under task_overrides/<task_id>/ to shadow a Terminal-Bench task or run a local task. Configure or disable overrides in config/harbor_config.toml.