completed_instances: 0 for ALL submissions — evaluation containers failing server-side (swe-bench_lite test)

[suhail-akhtar]

Bug: All submitted predictions return completed_instances: 0

Every submission we make to SWE-bench Lite (test) returns completed_instances: 0 and failed_instances: N — regardless of prediction quality, patch format, or run ID. This has been consistent across 4 separate submissions.

Evidence — All 4 Runs Return Identical Failure

Run ID	Submitted	Completed	Failed	Resolved
automotive-300-v1	299	0	299	0
automotive_v3	300	0	300	0
automotive_v3_final	300	0	300	0
automotive_v3_final2	300	0	300	0

Report Sample (automotive_v3_final2)

{
  "total_instances": 300,
  "submitted_instances": 300,
  "completed_instances": 0,
  "failed_instances": 300,
  "resolved_instances": 0,
  "schema_version": 2
}

Predictions Format — Verified Correct

• Valid JSON array, 300 entries
• Each entry has: instance_id, model_patch (standard unified diff diff --git format), model_name_or_path
• LF line endings confirmed (no CRLF)
• Only 1 empty patch out of 300

Local Harness Confirms Patches Are Valid

Running the same predictions.json through the local SWE-bench harness (swebench 4.1.0, WSL):

14/299 evaluated → ✓=14, ✖=0, error=0

14 out of 14 evaluated instances were resolved — confirming the patches are valid and apply correctly. The issue is server-side, not with our predictions.

What We Believe Is Happening

Based on reading the get_report.py source:

• failed_instances = evaluation Docker containers that crashed/failed to start on the server
• NOT test failures (those would appear as unresolved_instances)
• completed_instances: 0 = the evaluation infrastructure couldn't run a single job for our account

The "Evaluating predictions... 100%" message in the sb-cli output completes in ~4 seconds for 300 instances — impossibly fast, suggesting the server moves jobs to a failed state immediately rather than running them.

Request

1. Can you investigate why evaluation jobs fail immediately for run ID automotive_v3_final2?
2. If the predictions are valid (which local harness confirms they are), can you re-trigger evaluation server-side for our existing submission?
3. Is there a known issue with evaluation infrastructure for certain accounts?

Thank you!

[tamirdresher]

Same issue here. Submitted 300 predictions to swe-bench_lite/test (run IDs: squad_v1, squad_v2) — all 300 show as failed, 0 completed. Predictions verified locally with SWE-bench harness (198/300 resolved). Format is correct JSON list with instance_id, model_patch, model_name_or_path fields. sb-cli 0.1.5.

Is there an ETA on when the evaluation server will be restored? Is there an alternative submission path (e.g. via the experiments repo PR) that works in the meantime?

cc @carlosejimenez @ofirpress

[ofirpress]

Hi- we don't accept submissions anymore to any of our leaderboards, not via the experiments repo and not via sb-cli. We do wanna make sb-cli work in general though but we don't have a lot of bandwidth right now so not sure when/if we'll get to this issue, sorry