AgentRail SWE-lifecycle benchmarks + landing-page evidence

.gitignore

@@ -5,6 +5,7 @@ AGENTS.md
 !.env.example
 .agentrail/
 .agentrail.legacy/
+.agentrail-bench/
 .idea/
 .vscode/
 browser-artifacts/
@@ -49,7 +50,9 @@ verify-browser.mjs
 verify-waitlist.mjs
 verify-waitlist-api.mjs
 verify-landing-claims.mjs
-benchmarks/
+benchmarks/*
+!benchmarks/agentrail-swe-lifecycle/
+!benchmarks/agentrail-swe-lifecycle/**
 scripts/phase1-dogfood.mjs
 scripts/phase2-dogfood.mjs
 scripts/phase3-dogfood.mjs

README.md

@@ -1,6 +1,7 @@
 # AgentRail
 
 [![CI](https://github.com/oxnw/agentrail/actions/workflows/ci.yml/badge.svg)](https://github.com/oxnw/agentrail/actions/workflows/ci.yml)
+[![npm downloads](https://img.shields.io/npm/dm/%40agentrail-core/cli.svg)](https://www.npmjs.com/package/@agentrail-core/cli)
 [![License: Source-Available](https://img.shields.io/badge/License-Source--Available-blue.svg)](./LICENSE)
 [![Node.js](https://img.shields.io/badge/node-%E2%89%A524-brightgreen)](https://nodejs.org/)
 

benchmarks/agentrail-swe-lifecycle/README.md

@@ -0,0 +1,147 @@
+# AgentRail SWE Lifecycle Dataset
+
+This directory defines AgentRail's wrapper dataset for token-efficiency
+benchmarking on SWE-bench-style coding tasks.
+
+The checked-in `dataset.json` does not vendor SWE-bench problem statements,
+gold patches, test patches, or evaluator payloads. It stores only a stable
+manifest of upstream task references plus AgentRail lifecycle metadata.
+
+## Sources
+
+- Task selection: Artificial Analysis Coding Agent Index methodology,
+  `SWE-Bench-Pro-Hard-AA`.
+- Upstream task rows: `ScaleAI/SWE-bench_Pro`, `default/test` split on
+  Hugging Face.
+
+## Lifecycle Wrapper
+
+Each task is intended to run through an AgentRail lifecycle:
+
+1. create a synthetic GitHub or Linear issue from the upstream problem statement
+2. ingest that issue into AgentRail
+3. assign it to a managed AgentRail runner
+4. produce a patch or pull-request artifact
+5. run the SWE-bench evaluator as CI
+6. convert deterministic review findings into AgentRail review feedback
+7. retry through AgentRail when CI or review is blocking
+8. mark final pass/fail from the evaluator and lifecycle state
+
+The first scenario is `local-mock-lifecycle`, which simulates GitHub/Linear,
+PRs, CI, and code review locally. Live GitHub/Linear scenarios can be added
+after the local lifecycle is repeatable.
+
+## Refresh
+
+Regenerate the manifest from upstream sources:
+
+```bash
+npm run benchmark:agentrail-swe:refresh
+```
+
+Validate the checked-in manifest shape:
+
+```bash
+npm run benchmark:agentrail-swe:validate
+```
+
+## Run
+
+Dry-run the selected task manifest without launching a runner:
+
+```bash
+npm run benchmark:agentrail-swe:dry-run -- --count 5
+```
+
+Run the paired patch-only token benchmark. This runs the same coding task once
+through the direct control lane and once through AgentRail, then writes
+`results.json` and `summary.md` under
+`.agentrail-bench/agentrail-swe-lifecycle/<run-id>/`.
+
+```bash
+npm run benchmark:agentrail-swe:paired -- \
+  --runner codex \
+  --count 1 \
+  --repo-cache .agentrail-bench/repo-cache \
+  --require-token-usage
+```
+
+Run the paired full-lifecycle token benchmark when you want the control lane to
+pay LLM tokens for lifecycle work that AgentRail handles outside the runner:
+
+```bash
+npm run benchmark:agentrail-swe:paired:full-lifecycle -- \
+  --runner codex \
+  --count 1 \
+  --repo-cache .agentrail-bench/repo-cache \
+  --require-token-usage
+```
+
+The full-lifecycle surface requires the control runner to ingest the issue from
+the local provider, create the branch and pull request, check CI, check review
+feedback, and complete the task. The AgentRail lane records the equivalent
+lifecycle through the harness, so the runner prompt only carries the coding
+work and AgentRail handoff contract.
+
+## Phase Attribution Benchmark
+
+Use phase attribution when measuring token savings from lifecycle orchestration.
+This mode avoids comparing two independent autonomous agent trajectories. It
+runs a deterministic lifecycle script with separate measured model calls for
+each phase, then computes two accounting views:
+
+- Control: provider lifecycle phases plus coding phases.
+- AgentRail: coding phases plus AgentRail handoff.
+
+```bash
+npm run benchmark:agentrail-swe:phase-attribution -- \
+  --runner codex \
+  --count 1 \
+  --repo-cache .agentrail-bench/repo-cache \
+  --require-phase-token-usage
+```
+
+The default scenario is `ci-and-review-feedback`, which measures:
+
+- issue intake
+- initial coding
+- PR creation
+- CI polling
+- CI failure triage
+- CI fix
+- review polling
+- review feedback triage
+- review fix
+- PR update
+- final CI check
+- final review check
+- AgentRail handoff
+- provider completion
+
+The result file includes per-phase token usage, command logs, Control
+accounting, AgentRail accounting, and token deltas.
+
+Use `--phase-repeat <n>` to run the full phase sequence multiple times and sum
+all measured phase calls into the same accounting views.
+
+Cursor Agent can be used for non-token smoke runs, but its CLI JSON output does
+not currently expose token usage, so `--require-token-usage` and
+`--require-phase-token-usage` are rejected for `--runner cursor`.
+
+Add `--model <codex-supported-model>` only if your local Codex CLI accepts that
+model. The benchmark omits `--model` by default so Codex can use your configured
+default.
+
+For local smoke tests, use `--runner custom --runner-command <command>` with a
+fixture repository supplied through `--source-repo-path`. Real SWE-bench Pro
+runs omit `--source-repo-path`; the benchmark clones the task repository into
+`--repo-cache` and checks out the upstream base commit.
+
+The paired result records:
+
+- control lane status, patch, evaluator result, and token usage
+- AgentRail lane status, patch, evaluator result, and token usage
+- lifecycle event summaries for each lane when `--benchmark-surface full-lifecycle` is used
+- `tokenDelta`, where positive means AgentRail used more tokens
+- `tokenReductionRate`, where positive means AgentRail used fewer tokens
+- cached, non-cached input, output, and reasoning token deltas when the runner exposes them