Contagent

A benchmarking pipeline for running OpenHands agent experiments against a locally-served LLM via vLLM. Define tasks, setup scripts, and evaluation scripts in self-contained experiment directories, run them sequentially, and collect structured logs for analysis.

How It Works

┌─────────────────────────────────────────────┐
│              docker-compose                 │
│                                             │
│   ┌─────────────────────────────────────┐   │
│   │  vLLM  (OpenAI-compatible API)      │   │
│   │  Qwen3-Coder-30B-A3B · port 8000   │   │
│   └─────────────────────────────────────┘   │
│                    contagent-net             │
└─────────────────────────────────────────────┘
                        ▲
                        │ http://vllm:8000/v1
                        │
┌─────────────────────────────────────────────┐
│              pipeline.py                    │
│                                             │
│  for each experiment directory:             │
│    1. spin up service containers (optional) │
│    2. run setup.sh (in runner)              │
│    3. start OpenHands container             │
│    4. run evaluate.sh (in runner)           │
│    5. collect logs → .log / .result / .err  │
└─────────────────────────────────────────────┘

vLLM runs persistently via docker-compose. pipeline.py spins up a short-lived OpenHands container per experiment along with any declared service containers, waits for the agent to finish, runs evaluation, and moves on to the next experiment.

Project Structure

contagent/
├── docker-compose.yml        # vLLM service + contagent-net network
├── Dockerfile                # contagent image (pre-installs OpenHands)
├── entrypoint.sh             # runs setup, launches OpenHands headless, runs eval
├── pipeline.py               # orchestrates experiments via Docker SDK
├── requirements.txt          # Python dependencies
├── skills/                   # reusable agent skills (opt-in per experiment)
│   └── linux-networking/
│       └── SKILL.md
├── experiments/
│   └── exp_001/
│       ├── experiment.json   # task, services, skills, cap_add
│       ├── setup.sh          # runs before agent starts
│       ├── evaluate.sh       # runs after agent finishes
│       ├── workspace/        # files available to the agent
│       └── <svc>_entrypoint.sh  # optional per-service entrypoint
└── logs/
    ├── exp_001.log           # JSONL — agent actions and observations
    ├── exp_001.result        # eval script output (pass/fail)
    └── exp_001.err           # stderr — errors and warnings

Setup

Requirements:

• Docker with NVIDIA Container Toolkit
• Python 3.10+
• An NVIDIA GPU (tested on A100 40GB)

1. Install Python dependencies:

pip install -r requirements.txt

2. Build the contagent image:

docker build -t contagent:latest .

3. Download the model:

huggingface-cli download Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8

4. Start vLLM:

docker compose up -d

Wait for vLLM to finish loading the model (2–5 minutes on A100). You can verify:

curl http://localhost:8000/health

Running Experiments

Run all experiments:

python pipeline.py --served-model-name Qwen3-Coder-30B

Run a single experiment:

python pipeline.py --served-model-name Qwen3-Coder-30B --experiment exp_001

Experiment Format

Each experiment lives in its own directory under experiments/:

experiments/exp_001/
├── experiment.json
├── setup.sh
├── evaluate.sh
└── workspace/

experiment.json

{
  "task": "What is written inside the secret file in ~/workspace/secret.txt? Write the content in the format 'pass:{content}' to ~/workspace/result.txt.",
  "skills": ["linux-networking"],
  "cap_add": ["NET_ADMIN"],
  "services": [
    {
      "name": "webserver",
      "image": "nginx:alpine",
      "entrypoint_script": "webserver_entrypoint.sh",
      "environment": {}
    }
  ]
}

Field	Required	Description
task	Yes	Instruction passed to the OpenHands agent
skills	No	List of skill names from skills/ to inject into the agent
cap_add	No	Linux capabilities to add to the runner container (e.g. NET_ADMIN)
services	No	Extra containers to spin up alongside the agent

setup.sh

Runs inside the runner container before the agent starts. Use it to scaffold files, install packages, configure the environment, set up iptables rules, etc.

#!/usr/bin/env bash
set -euo pipefail
mkdir -p ~/workspace
echo 'super-secret-password' > ~/workspace/secret.txt

evaluate.sh

Runs inside the runner container after the agent finishes. Its output is written to .result. Exit code 0 = pass, non-zero = fail.

#!/usr/bin/env bash
set -euo pipefail
expected="pass:super-secret-password"
actual=$(cat ~/workspace/result.txt)
echo "Expected: $expected"
echo "Actual: $actual"
if [ "$actual" = "$expected" ]; then
  echo "PASS"
  exit 0
else
  echo "FAIL"
  exit 1
fi

Services

Service containers are spun up on a per-experiment isolated Docker network before the agent starts and torn down after evaluation. The agent can reach each service by its name as the hostname (e.g. http://webserver).

If entrypoint_script is specified, the file is looked up by name inside the experiment directory and mounted into the service container.

Skills

Skills are markdown files that inject domain-specific knowledge into the agent. They live in skills/<skill-name>/SKILL.md and are keyword-triggered — the agent loads them when relevant keywords appear in the conversation.

To use a skill, add its name to the skills array in experiment.json:

{
  "skills": ["linux-networking"]
}

Skills are opt-in and only the listed ones are mounted into the agent.

Logs

Each experiment produces three files in logs/:

File	Contents
<name>.log	JSONL — every agent action and observation as a JSON object per line
<name>.result	Output of evaluate.sh — pass/fail verdict and details
<name>.err	Stderr — errors and warnings from OpenHands internals

Parse the JSONL log line by line with json.loads() to inspect individual agent steps.

CLI Reference

vLLM connection:
  --vllm-url            URL to poll vLLM health from the host (default: http://localhost:8000)
  --vllm-internal-url   URL OpenHands containers use to reach vLLM (default: http://vllm:8000)
  --served-model-name   Model name as served by vLLM (default: Qwen3-8B)
  --api-key             API key for vLLM (default: openhands-local)
  --network             Docker network to join (default: contagent-net)

OpenHands:
  --entrypoint-script        Path to entrypoint.sh (default: ./entrypoint.sh)
  --agent-server-image-repo  Agent server image repo (default: ghcr.io/openhands/agent-server)
  --agent-server-image-tag   Agent server image tag (default: 1.12.0-python)
  --openhands-image          Runner image (default: contagent:latest)

Experiments:
  --experiments-dir   Directory containing experiment directories (default: ./experiments)
  --experiment        Run a single experiment by directory name (default: run all)
  --log-dir           Output directory for logs (default: ./logs)
  --skills-dir        Directory of skills to mount into the agent (default: ./skills)

Model

Currently configured to serve Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8 — a Mixture of Experts coding model with 30B total / 3B active parameters, scoring 51.6% on SWE-bench Verified with OpenHands scaffolding. To change the model, edit docker-compose.yml directly.