All submitted runs failed

[AswarS]

I encountered an issue where all submitted runs failed during evaluation using sb-cli.

Command used:
sb-cli submit swe-bench_verified test
--predictions_path ./swe_verify_test.json
--run_id test

Output summary:
Submitting predictions for test_v7_f - (swe-bench_verified test)
Submitting predictions... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:00:08
✓ Submitting predictions complete! Warning: 100 predictions already submitted. These will not be re-evaluated
Processing submission... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:00:02
✓ Processing submission complete!
Evaluating predictions... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:00:54
✓ Evaluating predictions complete!
Resolved (total): 0.00% (0 / 500)
Resolved (submitted): 0.00% (0 / 100)
Submitted: 20.00% (100)
Errors: 0
Pending: 0
Successful runs: 0
Failed runs: 100

submission:
[
{
"instance_id": "astropy__astropy-12907",
"model_patch": "diff --git a/astropy/modeling/separable.py b/astropy/modeling/separable.py\nindex a308e27297..45bea36085 100644\n--- a/astropy/modeling/separable.py\n+++ b/astropy/modeling/separable.py\n@@ -242,7 +242,7 @@ def _cstack(left, right):\n cright = _coord_matrix(right, 'right', noutp)\n else:\n cright = np.zeros((noutp, right.shape[1]))\n- cright[-right.shape[0]:, -right.shape[1]:] = 1\n+ cright[-right.shape[0]:, -right.shape[1]:] = right\n \n return np.hstack([cleft, cright])\n \n",
"model_name_or_path": "mco-4"
}...
}

Questions:

1. What are the common causes for all runs failing ?
2. Is there a way to access detailed evaluation logs ?

[Brvcket]

I've run into the same problem recently. It's definitely not the submissions themselves, but a service error. What used to return 12/102 on the swe bench mm dev split, now returns 0/102 consistently (using the same prediction file).

[sugarshaw95]

+1 all failed,0 even for runs submitted before

[ICELING626]

Can confirm, I'm experiencing the same behavior. swe-bench-m test

[WangChen100]

+1 the same problem, all failed, does anyone solving it?

[SwithinTan]

same issue here

[real-ljs]

same

[jarvis-llm-codec]

Additional datapoint — even the official gold patch fails (swe-bench_verified / test).

Hit the same wall today (2026-07-04) with a fresh account/API key, and ran a controlled isolation:

1. Submitted 2 predictions (astropy__astropy-12907, astropy__astropy-13033) → Failed runs: 2, completed_instances: 0, error_instances: 0.
2. Ruled out our side: base-commit blob verified against GitHub (a308e27297… at d16bfe05), clean LF endings, git apply --check passes on a fresh checkout. Our 12907 patch is byte-identical to the gold patch except the index line.
3. Re-submitted with a sanitized model_name_or_path (no + or parentheses) → same result.
4. Control experiment: submitted the official gold patch from princeton-nlp/SWE-bench_Verified itself (run_id jlc-gold-control) → Failed runs: 1. A gold patch cannot legitimately fail, so the failure has to be server-side.
5. Same predictions through the official local harness (swebench==4.1.0, --namespace swebench): both instances complete normally (1 resolved / 1 unresolved, 0 errors).

Two more observations that may help:

• The evaluation phase takes exactly ~1:27 whether we submit 1 or 2 instances — jobs appear to die instantly while the CLI polls to a fixed state (consistent with the ~4s-for-300 observation in completed_instances: 0 for ALL submissions — evaluation containers failing server-side (swe-bench_lite test) #27).
• get-quotas on our account lists only swe-bench-m and swe_bench_lite; swe-bench_verified never appears, although submit/list-runs accept it. If verified needs separate authorization, runs failing silently is a confusing way to surface that; if not, the eval containers are failing server-side.

Run IDs available for debugging: jlc-smoke-probe, jlc-smoke-probe-2, jlc-gold-control. Happy to re-run anything that helps.