[Bug] Evaluation results dropped to 0% despite successful submission and valid predictions

[ICELING626]

Describe the bug

[Bug] Evaluation results dropped to 0% despite successful submission and valid predictions

Description

I am encountering an issue where the evaluation results for my submissions have suddenly dropped to 0%, whereas the same data and model configuration yielded normal results (approx. 43% resolved) just a few days ago.

Context

• Dataset: swe-bench-m test
• Status: The submission process completes successfully with no errors reported by the CLI.
• Change: No changes were made to the prediction generation logic or the input data. The only difference is the time of submission.

Evidence

1. Previous Successful Run (A few days ago)

The evaluation completed normally, showing a resolved rate of ~43%.

Submitting predictions for qwen3-5_with_custom - (swe-bench-m test)
  Submitting predictions... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:00:48
✓ Submitting predictions complete!
  Warning: 504 predictions already submitted. These will not be re-evaluated
  6 new predictions uploaded - these cannot be changed
  Processing submission... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:00:04
✓ Processing submission complete!
  Evaluating predictions... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:20:36
✓ Evaluating predictions complete!
Resolved (total): 43.33% (224 / 517)
Resolved (submitted): 43.92% (224 / 510)
Submitted: 98.65% (510)
Errors: 0
Pending: 0
Successful runs: 496
Failed runs: 14
Saved full report to sb-cli-reports/swe-bench-m__test__qwen3-5_with_custom.json!

Steps/Code to Reproduce

sb-cli submit swe-bench-m test --predictions_path <prediction_file> --run_id <run_id>

Expected Results

Submitting predictions for qwen3-5_with_custom - (swe-bench-m test)
  Submitting predictions... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:00:48
✓ Submitting predictions complete!
  Warning: 504 predictions already submitted. These will not be re-evaluated
  6 new predictions uploaded - these cannot be changed
  Processing submission... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:00:04
✓ Processing submission complete!
  Evaluating predictions... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:20:36
✓ Evaluating predictions complete!
Resolved (total): 43.33% (224 / 517)
Resolved (submitted): 43.92% (224 / 510)
Submitted: 98.65% (510)
Errors: 0
Pending: 0
Successful runs: 496
Failed runs: 14

Actual Results

  Submitting predictions for qwen3-5-plus_with_custom-1 - (swe-bench-m test)
  Submitting predictions... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:00:39
✓ Submitting predictions complete!
  Warning: 510 predictions already submitted. These will not be re-evaluated
  Processing submission... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:00:05
✓ Processing submission complete!
  Evaluating predictions... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:00:04
✓ Evaluating predictions complete!
Resolved (total): 0.00% (0 / 517)
Resolved (submitted): 0.00% (0 / 510)
Submitted: 98.65% (510)
Errors: 0
Pending: 0
Successful runs: 0
Failed runs: 510

System Information

MacOS

[stringing]

same here with the verified subset

[dengyx21]

the same problem

[ICELING626]

Update: Issue Persists

The issue has not been resolved. I attempted a new submission on the swe-bench-m dev split, but the result is still 0% resolved with all runs failing, despite the submission process completing without errors.

Latest Submission Log (dev_gold)

Submitting predictions for dev_gold - (swe-bench-m dev)
  Submitting predictions... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:00:19
✓ Submitting predictions complete!
  102 new predictions uploaded - these cannot be changed
  Processing submission... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:00:23
✓ Processing submission complete!
  Evaluating predictions... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0% 0:01:49
✓ Evaluating predictions complete!
Resolved (total): 0.00% (0 / 102)
Resolved (submitted): 0.00% (0 / 102)
Submitted: 100.00% (102)
Errors: 0
Pending: 0
Successful runs: 0
Failed runs: 102

[stringing]

Hi @klieret, thought you might have some insights, would you mind taking a look when you have time? Thanks!!!

[DivineYan]

same

[ICELING626]

Hi @carlosejimenez , Could you please take a look at this when you have a moment? Would be greatly appreciated. Thanks～

[Danpythonman]

Confirming the same failure on swe-bench-m test. Predictions that previously evaluated successfully (my saved reports show a non-zero resolved rate) now return 0% resolved, with every submitted instance landing in failed_ids. The "Evaluating predictions" step completes in seconds, matching the immediate-failure pattern noted in #27.

To rule out my predictions as the cause, I tested the evaluation infrastructure directly with gold patches. The test split's patch field is stripped, so I used the public swe-bench-m dev split, where gold patches are available.

Results:

• 20/20 gold patches for chartjs/Chart.js (dev) -> all failed
• 5/5 gold patches, one per repo (Automattic/wp-calypso, chartjs/Chart.js, diegomura/react-pdf, markedjs/marked, processing/p5.js) -> all failed

In every case: completed_instances: 0, error_instances: 0, no resolved/unresolved. The same signature is reported on swe-bench_verified test (#25) and swe-bench_lite test (#27), so it appears to span datasets.

The sb-cli reports all look like this, with failed_instances being the number of instances submitted:

{
  "submitted_instances": 5,
  "completed_instances": 0,
  "failed_instances": 5,
  "resolved_instances": 0,
  "unresolved_instances": 0,
  "error_instances": 0,
  "schema_version": 2
}

Environment:

• sb-cli version: 0.1.5
• Approx. date the previously-working reports were generated: March 20, 2026