router-evals

Eval harnesses for benchmarking OpenRouter Auto and our internal router on agentic coding tasks, with Claude Code as the fixed harness. Every leg drives the same agent (real Claude Code, headless) over the same tasks; only the LLM endpoint changes — bare Anthropic/Bedrock, OpenRouter Auto (via a translating proxy), or our router gateway — so the resolution rate, cost, and model-distribution numbers are directly comparable. The headline question: does a router match top-model quality at materially lower cost?

Datasets / eval suites

Eval	What it measures	Tasks	Folder
Terminal-Bench	Agentic terminal/coding ability — each task runs in its own Docker container; graded pass/fail by the task's own tests	terminal-bench-core==0.1.1 = 10 tasks; the broader original-tasks set = 45	terminal-bench/
SWE-bench Multilingual	Real-GitHub-issue resolution across 9 languages (C, C++, Go, Java, JS, PHP, Ruby, Rust, TS) — resolution rate + per-language cost	language-stratified subset (default 10; full set 300)	swe-bench-multilingual/

(A Python-only SWE-bench Verified 3-way also exists in the source repo; this export carries the multilingual variant, which shares the same agent runner / grader / report code.)

Architecture

Claude Code harness (claude -p, headless)   ← the FIXED agent, identical across legs
        │  speaks the Anthropic Messages API
        ▼
   one of:
   ├─ Bedrock / Anthropic direct        (bare Sonnet / Opus baselines)
   ├─ openrouter_proxy.py  ──►  OpenRouter Auto   (Anthropic → OpenAI translate → openrouter/auto)
   └─ our router gateway   ──►  cheap/mid/Claude tiers   (router picks the model per turn)

openrouter_proxy.py is a small Anthropic-compatible HTTP proxy: Claude Code thinks it's talking to Anthropic, but each call is translated to OpenAI format and routed to openrouter/auto, which picks a model per request. The proxy tags every call with a per-task key and logs OpenRouter's own reported cost, so per-task spend is recoverable post-hoc. The translation layer lives in adapters/ (pure stdlib).

Prerequisites

• Docker + the Compose v2 plugin (docker compose) — terminal-bench runs one container per task; the SWE-bench grader builds/runs per-instance test images.
• Python 3.12 (recommended via uv).
• Python deps: pip install -r requirements.txt (or uv pip install -r requirements.txt) — installs terminal-bench (the tb CLI), requests, datasets, and swebench. The adapters/ translation layer and its tests are pure stdlib and need none of these.
• The claude CLI on PATH (the agent harness; headless claude -p) — npm install -g @anthropic-ai/claude-code.
• OPENROUTER_API_KEY (sk-or-...) for the OpenRouter Auto legs.
• The baseline models are not hardcoded — set BASELINE_SONNET_MODEL / BASELINE_OPUS_MODEL (and the harness --model flags) to whatever provider/models you have access to. The Sonnet baseline defaults to a public id; supply the Opus (ceiling) baseline yourself. Only the model id changes; the harness is identical across providers. Note the id format is provider-specific: the defaults use AWS Bedrock inference-profile ids (us.anthropic.claude-sonnet-4-6); on Anthropic-direct it's claude-sonnet-4-6, on OpenRouter anthropic/claude-sonnet-4-6, etc. — write the model the way your provider expects.
• For router legs only: a running router gateway + its auth token (ROUTER_EVAL_TOKEN), and (to switch tiers) a ROUTER_SET_TIER_CMD. These legs are not self-contained — see the per-folder READMEs.

Secrets

No keys, tokens, or org identifiers are committed. Everything sensitive is read from the environment at run time:

Env var	Used by	Meaning
OPENROUTER_API_KEY	terminal-bench OpenRouter legs	your OpenRouter key (sk-or-...)
ROUTER_EVAL_TOKEN	router legs	router gateway auth token
ROUTER_GATEWAY_URL	router legs	router gateway base URL
ROUTER_SET_TIER_CMD	router smoke/tier legs	command to switch tier, with a {tier} placeholder
BASELINE_SONNET_MODEL	baseline legs	Sonnet baseline model id (defaults to a public id)
BASELINE_OPUS_MODEL	baseline legs	Opus (ceiling) baseline model id — supply your own

Do not add .env* files to the repo (they're gitignored). Put local secrets in a gitignored env file and set -a; source <file>; set +a, or just export them in your shell.

Layout & PYTHONPATH

router-evals/
├── adapters/                # Anthropic↔OpenAI translation (pure stdlib) + its tests
├── terminal-bench/          # the main agentic eval (Docker-per-task)
└── swe-bench-multilingual/  # SWE-bench Multilingual eval

adapters/ lives at the repo root so both eval folders can from adapters import openai_translate. Add the repo root to PYTHONPATH when running anything that needs it (the terminal-bench proxy does — its runner scripts set this for you). E.g.:

export PYTHONPATH="$(pwd)"   # from the repo root