classify_error() has no branch for HTTP 402 / insufficient balance — model failover depends on two implicit fallthroughs

[epicdm]

Summary

classify_error() in backend/app/services/llm/failover.py classifies provider errors for model failover. It has explicit keyword branches for auth, validation, content policy, rate limiting, 5xx, network and transient errors — but no branch for HTTP 402 / insufficient balance / payment required / quota / credit.

Grepping backend/app/services/llm/ for 402, insufficient, balance, payment required, quota or credit returns nothing on main.

A 402 still reaches the fallback model today, but only through two implicit paths, neither of which names it:

1. the [llm error] / [llm call error] / [error] prefix branch returns RETRYABLE; and
2. the function's final default returns UNKNOWN, and is_retryable_error() in caller.py tests classify_error(...) != FailoverErrorType.NON_RETRYABLE — so UNKNOWN is treated as retryable.

Note also that "402" is not a substring of any entry in the non-retryable status list ["401", "403", "400", "422"], so it does not get misclassified — it simply falls through.

Why this is worth an explicit branch

Both paths are incidental, and either can be removed by an ordinary refactor with no test failure:

• Tightening is_retryable_error() from != NON_RETRYABLE to == RETRYABLE is a one-token change that silently ends 402 failover for every error classified UNKNOWN.
• Narrowing or reordering the [llm error] prefix branch has the same effect for prefixed errors.

The failure mode is quiet and expensive: when a provider returns 402 for an exhausted balance and failover does not fire, call_llm() returns the provider's error string, which callers surface as assistant output. A billing condition on one provider becomes a user-visible failure rather than a transparent switch to the fallback model — which is exactly what a configured fallback_model_id exists to prevent.

402 is also a case where failover is unambiguously the right action: retrying the same model and credential cannot succeed, but a fallback model on a different provider can.

Suggested change

An explicit branch, placed with the other retryable categories:

    # Retryable: billing/quota exhaustion on this provider. Retrying the same
    # model+credential cannot succeed, but a fallback model on a different
    # provider can — this is precisely what failover is for.
    if any(kw in error_msg for kw in [
        "402", "insufficient balance", "insufficient_quota",
        "payment required", "quota exceeded", "billing", "out of credit",
    ]):
        return FailoverErrorType.RETRYABLE

Placed after the auth/validation/content-policy checks so an "invalid api key" style message is still non-retryable, and before the generic status-code checks.

Happy to open a PR with this plus a test if the approach looks right.

Environment

Observed on v1.11.0; classify_error() on current main is unchanged in this respect.