Targeted: 0.5.0. Baseline: release/0.3.0.
Current evidence (dogfood, Aider harness) mostly submits pre-baked patches — it proves the mechanical pipeline, not editing capability; the SWE-bench harness is still preflight-only. Nothing today can falsify "the model spends rounds failing to land an edit".
- 30–50 real tasks, same model + same prompt, native tool loop vs this MCP server.
- Metrics: final test pass rate, first-patch success rate, rounds to green, regressions introduced, wall time.
- Output lands in
reports/; reuse the benchmarks/real_workloads.py skeleton where possible.
- This is the data source for the 0.5.0 acceptance thresholds (first-patch success > 80%, overall failure rate < 3%) and for the structured-only default decision.
Targeted: 0.5.0. Baseline:
release/0.3.0.Current evidence (dogfood, Aider harness) mostly submits pre-baked patches — it proves the mechanical pipeline, not editing capability; the SWE-bench harness is still preflight-only. Nothing today can falsify "the model spends rounds failing to land an edit".
reports/; reuse thebenchmarks/real_workloads.pyskeleton where possible.