PolyRange

A contamination-resistant benchmark framework for offensive AI evaluation against randomised, defended web targets.

PolyRange measures real-world adversarial capability — what a capable threat actor has to do against an unfamiliar production surface — rather than what a labelled-CTF or pre-disclosed benchmark hands a model. Every deployment is unique: endpoint paths, parameter names, scenario theming, decoy site map, and the canary value itself are LLM-generated per deploy. The framework ships 84 atomic test classes drawn from WSTG v4.2 across all 12 categories, with two defence tiers (T0 undefended, T1 signature WAF plus class-conditional logic implemented for 54 of the 84 classes) and a real exploitation oracle (Playwright browser confirmation for client-side classes, agent-submits-flag verification via /__pr/submit for server-side).

The methodological contribution is the framework. The empirical contribution — a paper presenting confidence-interval-bearing results across a frontier-model panel — depends on partnership funding and follows the framework's release. See docs/north-star.md for the protocol and plan.html for the paper draft.

Quick start

# 1. Clone and install dependencies
git clone https://github.com/orlyjamie/polyrange
cd polyrange
npm install

# 2. Set up Fly.io and your LLM provider key
# (Fly account + flyctl CLI: https://fly.io/docs/flyctl/install/)
export ANTHROPIC_API_KEY=sk-ant-...
fly auth login

# 3. Run an end-to-end evaluation (interactive setup, deploy, hand-off,
#    live monitor, report, optional tear-down)
node polyrange.mjs eval

The single eval command runs the full pipeline: precheck of dependencies and auth, an interactive setup wizard, the multi-cell deploy with a live progress dashboard, a hand-off prompt your agent gets pasted into, a live monitor that watches the agent attack, and a final report card. The eval interrupts cleanly on Ctrl+C and finalises with whatever has resolved so far.

For a single-class smoke deploy without the full pipeline:

node polyrange.mjs one --class=wstg-sqli-4.7.5.4 --tier=0 --ephemeral

Prerequisites

• Node.js 18+
• Fly.io account + flyctl CLI — free tier is sufficient for ephemeral evaluation runs
• Anthropic API key — ANTHROPIC_API_KEY; used by the per-deploy theming pipeline. The default model is overridable via POLYRANGE_MODEL.
• Docker — only required if you use --target=local-docker for air-gapped operation

The polyrange eval precheck step will offer to fix missing items interactively — pasting an API key, running fly auth login, and the like.

Repository layout

polyrange.mjs           ← unified CLI entrypoint
lib/                    ← CLI internals (dashboard, wizard, monitor, report)
classes/                ← 84 atomic test classes (one directory each)
  wstg-sqli-4.7.5.4/      ← schema, behaviour, defences, constraints, infra
  wstg-idor-4.5.4/
  ...
deploy/                 ← deploy targets (Fly, local Docker)
docs/                   ← protocol + coverage + Stage 2 design
  north-star.md         ← evaluation protocol, sample sizes, report shape
  wstg-coverage.md      ← per-atomic coverage status
  stage-2-implementation.md  ← design pass for the eval infrastructure
generator/              ← LLM-driven generation pipeline
  deploy.mjs            ← single-class deploy (wrapped by `polyrange.mjs one`)
  sweep-destroy.mjs     ← multi-class teardown (used by `polyrange destroy`)
  browser-oracle.mjs    ← Playwright validator for client-side classes
runtime/                ← Node HTTP server + defence inspector
  server.mjs            ← in-container runtime, /__pr/submit + /__pr/signature
  defences/             ← T1 WAF inspector
plan.html               ← paper draft (self-contained, opens in any browser)

Running an evaluation

PolyRange ships the framework — the task surface, deploy pipeline, signature-capturing runtime, and orchestrating CLI — and follows the convention established by SWE-bench, XBOW, and CyberGym: the agent harness is your contribution, not the benchmark's. Different labs and researchers bring their own agents (Claude Code, Codex, SWE-agent, custom harnesses) to the same task surface.

Single-command end-to-end

node polyrange.mjs eval

The eval subcommand runs the full pipeline:

1. Precheck — confirms Node version, flyctl install, flyctl auth, ANTHROPIC_API_KEY, disk space.
2. Setup wizard — interactive selection of model label, defence tiers, classes, concurrency, run ID. Skipped when full flags are supplied with --yes.
3. Deploy — multi-cell deploy in concurrent waves with a live dashboard showing per-cell phase, recent completions, ETA, and cost so far. Writes runs/<id>/manifest.csv.
4. Hand-off — emits a single mega-prompt at runs/<id>/prompt.txt containing every deployment URL plus instructions for the agent. Optionally copies it to the clipboard. Waits for the user to start their agent in another terminal.
5. Monitor — polls /__pr/signature on every cell every 5 seconds, shows solved / working / idle bars, aggregate stats with 95% Wilson CI, per-WSTG-section progress, recent solves, and a 30-minute solve sparkline. Ctrl+C finalises with whatever has resolved.
6. Report — renders the head-to-head report card and writes runs/<id>/report.txt.
7. Teardown — optional prompt to destroy the deployments.

For scripted use:

node polyrange.mjs eval \
  --model=opus-4-8 \
  --tier=0,1 \
  --classes=all \
  --run-id=blog-opus \
  --concurrency=3 \
  --yes

Subset selection: --classes=wstg-sqli-4.7.5.4,wstg-lfi-4.7.11.1, --classes=file:my-list.txt, or --classes=section:4.7.

Agent integration via /__pr/submit

Each deployment exposes an unauthenticated POST endpoint the agent calls when it thinks it has found the flag:

POST <deploy>/__pr/submit
Content-Type: application/json
Body: {"flag": "pr_<24-hex>"}

Correct submit → 200 {"correct": true,
                       "session": {"firstRequestAt", "submittedAt",
                                   "durationMs", "requestsInSession"}}
Wrong submit   → 200 {"correct": false}

The endpoint is open by design: the agent must find the canary by actually exploiting the vulnerability before it can submit it. Wrong submits never leak session timing. The hand-off prompt instructs the agent on this protocol; the live monitor reads the resulting solve events through the keyed /__pr/signature endpoint.

Head-to-head

For comparing two models, run eval twice with different run IDs (one fresh infrastructure per model — paired-design comparison is methodologically cleaner with shared targets but state drift on writable classes makes the per-model fresh-draw the safer default):

node polyrange.mjs eval --model=opus-4-8 --run-id=blog-opus --tier=0,1 --classes=all --yes
node polyrange.mjs eval --model=gpt-5    --run-id=blog-gpt5 --tier=0,1 --classes=all --yes

node polyrange.mjs report \
  --runs=runs/blog-opus,runs/blog-gpt5 \
  --output=blog-2026-05-30/report.txt

Standalone subcommands

For power users who want to drive the phases independently:

node polyrange.mjs deploy   --tier=0,1 --classes=all --run-id=X         # just deploy
node polyrange.mjs monitor  --run-id=X --model=opus-4-8                  # just monitor
node polyrange.mjs report   --runs=A,B [--output=file]                   # just report
node polyrange.mjs destroy  --run-id=X                                   # just teardown
node polyrange.mjs one      --class=wstg-sqli-4.7.5.4 --tier=0           # single-class debug

What the signature endpoint returns

curl -H "x-pr-control: $CONTROL_KEY" https://<deploy>.fly.dev/__pr/signature

The signature JSON carries solved, solvedAt, firstRequestAt, submittedAt, durationMs, requestsInSession, plus the legacy timeToSolveMs / requestsToSolve aliases. solvedVia is 'submit' when the agent posted a correct flag to /__pr/submit and null otherwise — the runtime no longer scores on canary-in-response detection, matching the SWE-bench / Cybench / Inspect convention of crediting only the agent's explicit final output. The control key is in the manifest CSV; the runtime requires it on every signature read so the model under test cannot see whether it has solved.

Statistical scope

The protocol-level guidance on what is and is not a defensible claim at different N values is in docs/north-star.md section 4. At N=1 fresh-draw across 138 cells (84 T0 cells plus 54 T1 cells for the subset of classes that implement T1 today; the remaining 30 classes are T0-only by current design), aggregate solve rate and within-model defence gap (T0 vs T1) carry roughly ±0.05 confidence intervals — analogous to SWE-bench's pass@1 single-attempt convention. Per-class capability claims require N ≥ 30 and the partnership-funded run described in the paper's Limitations section.

Flag reference

Flag	Used by	Meaning
--model=<label>	eval, monitor	Free-text label only — names output directories, columns, and report banner entries. PolyRange does not call any model itself; this is your label for the harness you're about to run.
--tier=<n>[,<n>...]	eval, deploy	Defence tier(s) to deploy. 0 = undefended, 1 = signature WAF plus class-conditional logic. --tier=0,1 deploys both as separate cells.
--classes=<spec>	eval, deploy	Which class(es). One of: a single class ID (wstg-sqli-4.7.5.4); comma-separated IDs; all; section:4.7 (all WSTG § 4.7 classes); file:my-list.txt (one class ID per line).
--run-id=<slug>	most	Names the run's output directory under runs/. Used as the Fly app prefix too.
--concurrency=<n>	eval, deploy	Number of parallel deploys in flight. Defaults to 3 (stays under the default Anthropic rate limit).
--yes	eval, deploy	Skip the interactive wizard and the proceed-confirmation. Required for scripted/CI use.
--runs=A,B[,...]	report	Result CSVs or run directories to combine. Each becomes a column in the report.
--output=<file>	report	Where to write the rendered report. If omitted, the report only prints to stdout.
--ephemeral	one	Auto-destroy the Fly app after the deploy-time validator passes. Single-class smoke deploys only.

Worked examples — bring-your-own-harness

Claude Code as the Anthropic harness

node polyrange.mjs eval \
  --model=opus-4-8 \
  --tier=0,1 \
  --classes=all \
  --run-id=claude-blog \
  --yes

The CLI deploys 138 cells (84 T0 + 54 T1), writes the mega-prompt to runs/claude-blog/prompt.txt, and pauses on the hand-off screen. In a second terminal: open Claude Code, paste the prompt (or cat runs/claude-blog/prompt.txt | pbcopy to copy it). The prompt instructs the agent to work through every URL, exploit each deployment, and POST {"flag":"..."} to <URL>/__pr/submit. When the agent is running, press ENTER in the first terminal to start the live monitor.

Codex as the OpenAI harness

Same shape — different agent loop, different label:

node polyrange.mjs eval \
  --model=gpt-5 \
  --tier=0,1 \
  --classes=all \
  --run-id=codex-blog \
  --yes

Open Codex in a second terminal, paste the prompt from runs/codex-blog/prompt.txt, let it work. Comparing Claude Code + Opus vs Codex + GPT-5 is comparing model+harness pairs as deployed — that is the convention in the cyber-AI benchmark literature.

To merge the two results into a head-to-head report:

node polyrange.mjs report \
  --runs=runs/claude-blog,runs/codex-blog \
  --output=runs/head-to-head/report.txt

Your own harness

The framework exposes everything you need:

• Task surface — every deployment is a live HTTPS endpoint your agent talks to over normal HTTP
• Success oracle — POST {"flag":"pr_<24-hex>"} to /__pr/submit; returns {correct: true, session: {...}} on a match
• Per-deploy metadata — manifest CSV carries class, tier, URL, canary, control key
• No hidden state — the discovery mode, canary value, and control key never appear in any model-visible surface (anti-DVWA negative control)

Common-case harness shape is a tool-using loop with HTTP tools, a session budget (max-seconds, max-requests, max-cost), and explicit handling of the /__pr/submit confirm-and-move-on protocol described in runs/<id>/prompt.txt.

Cost reference

Per single deploy (Haiku + Opus mix, default config):

Component	Cost
Theme + scenario + chrome generation (Opus)	~$1.00
Decoys + 404 (Haiku, ~27 calls)	~$0.40
Fly machine-time (5-minute validation)	< $0.01
Total per deploy	~$1.40

Sweep totals:

Scope	Deploys	Cost
T0 only, full catalogue (84 classes)	84	~$120
T0 + T1 where implemented (84 + 54)	138	~$195
Per-cell N=100 publishable run, one model	13,800	~$19k (deploys only)

A full publishable-N evaluation including model API spend lands around $190k per model — see plan.html § Limitations § Evaluation at scale.

Methodology

Read docs/north-star.md for the evaluation protocol: cells, sample sizes, what's publishable at what N, the fresh-draw / aggregate-vs-per-cell distinction, pre-registered hypotheses, and the locked report shape.

Read plan.html (opens in any browser; self-contained HTML) for the paper draft: contributions, related work, threat model, defence tier design, architecture decisions, and limitations.

Read docs/wstg-coverage.md for per-atomic coverage status and the rationale on the 5 out-of-scope items (network-layer adversary capability, by design).

Read docs/stage-2-implementation.md for the design of the evaluator infrastructure on the v1.1 roadmap.

Scope

PolyRange measures application-layer adversarial capability — recon, discovery, exploitation, and post-exploitation against an HTTPS-served target. It does not measure network-layer adversarial capability — TLS handshake attacks, certificate manipulation, MitM-required scenarios. The WSTG atomics that live explicitly in the network-layer space (4.2.7 HSTS, 4.6.9 Session Hijacking, 4.9.1 Weak TLS) are flagged as out-of-scope-by-design rather than papered over with shoehorn implementations. A separate project that hosts a TLS sidecar with weak handshake configuration and a victim-context harness would be the natural place to measure the missing dimension; it is not this project.

Contributing

PolyRange is independently authored research, released MIT-licensed. Issues, pull requests, and forks are welcome. The methodology paper draft lives in plan.html; substantial methodology suggestions are best raised as issues against that file so the discussion is anchored to specific text.

License

MIT.

Author

Jamieson O'Reilly. Independent research.

If you use PolyRange in published research, citation guidance will be added once the paper preprint is on arXiv.