eval: grader API errors are scored 0.00 instead of surfacing as errors

[Savid]

Problem

When the grading model is unavailable, the eval harness scores every affected run 0.00 and marks it failed, instead of surfacing a grading error. A grader outage is therefore indistinguishable from "the product answered every question wrong".

Observed in Eval Smoke run 28565884088 (PR #274) and in the master run 28556183486 — both report:

0 passed (0%) / 8 failed (100%) / 0 errors (0%)   pass-rate 0%  mean-score 0.000

while every case in reports/eval.json carries the same grader_reason:

API error: 503 Service Unavailable {"error":{"message":"Error from provider (Console Go):
Inference is temporarily unavailable", "code":"failover_exhausted"}}

The agent answers in the artifact are correct (e.g. forky_node_coverage returned all 4 sepolia nodes with the right consensus clients) — the product was fine, only the grade call failed. The 0% pass-rate, the red check, and the JUnit output all misattribute a grader-infrastructure outage as product failure.

Proposed fix

In the eval scripts, detect grade results whose reason indicates a provider/API error (e.g. grader_reason starting with API error:) and:

• retry the grade call with backoff, and if still failing,
• report the case as errored (excluded from pass-rate/mean-score) rather than failed-with-0.00, and
• exit with a distinct message ("grading unavailable") so the check failure reads as infra, not regression.

Impact

Low frequency, high confusion: any grader outage turns every open PR's smoke check red with a "0% pass" that looks like a total regression.

🤖 Generated with Claude Code