Verify bare model-family resolution against the live API (blocked: no Anthropic credit)

[jeremymanning]

Split out of the live-provider work so it is not silently forgotten.

What is verified

The live CI job confirmed against the real API:

test	result
generate() with claude-haiku-4-5-20251001	PASSED
max_tokens actually applied	PASSED
models.list() / discover_models()	PASSED
health_check()	PASSED

That validates the main fix: explicit model ids are no longer rewritten. Previously _normalize_model_name substring-matched the family and rewrote any id containing haiku/opus/sonnet to a hard-coded 2024 model, so every current Claude model was unreachable.

What is NOT verified

test_every_family_alias_resolves_to_a_real_model — whether a bare name ("haiku", "opus", "sonnet") resolves through client.models.list() to a servable id.

It could not run:

400 invalid_request_error: Your credit balance is too low to access the
Anthropic API.

This is an account precondition, not a known defect. The test now skips with that explicit reason rather than reporting a misleading failure, and still fails hard under ORCHESTRATOR_REQUIRE_LIVE=1 so the live job cannot go green on an unusable account.

To close

1. Add credit to the Anthropic account.
2. Run gh workflow run live-tests.yml.
3. Confirm all five live tests pass. test_models_api_lists_servable_models prints the served model list, which is the primary source for which ids exist — two rounds of hard-coded ids have already been wrong here (retired 2024 dates, then invented -latest aliases).

Note on cost

The whole live suite is a handful of Haiku calls capped at max_tokens=32. The credit exhaustion was pre-existing, not caused by these tests.

[jeremymanning]

Live coverage is no longer blocked on credit

A DartmouthProvider has landed (a117b75). Dartmouth Chat is an OpenAI-compatible gateway serving 7 models at zero cost per token, so real-model acceptance coverage no longer depends on a funded Anthropic account:

DARTMOUTH_CHAT_API_KEY=... pytest -m live -k dartmouth

Verified against the live gateway: 7 passed, 2 skipped (skips were an upstream qwen3.5 outage, not defects).

This does not close this issue — the Anthropic family-alias resolution is still unverified and still needs credit. But it removes the urgency: the project now has real-model coverage that costs nothing, and the Anthropic path is one provider rather than the only one.

Two findings from building it, both relevant to any provider work here

1. Free model endpoints flap independently. qwen.qwen3.5-122b returned Cannot connect to host vllm-qwen35.ai-prod.svc.cluster.local:8000 minutes after working fine. It arrives as an HTTP 400 — indistinguishable from a bad request without reading the body. Live tests must skip on this, not fail.

2. Several free models are reasoning models. They spend tokens on reasoning_content before emitting any content, so a small max_tokens returns content: null. Observed: qwen3.5 at max_tokens=16 burned 307 completion tokens thinking and answered nothing. Returning "" there would be a silent wrong answer.

Both are handled by provider.generate_free(), which walks the free set and reports which model actually answered.

[jeremymanning]

Re-confirmed against CI (run 30681083746, 2026-08-01)

Still blocked, and the block is definitively environmental rather than a code
defect — the API says so itself:

Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error',
'message': 'Your credit balance is too low to access the Anthropic API.
Please go to Plans & Billing to upgrade or purchase credits.'}}

3 failed, 2 passed, 13 skipped, 3203 deselected

The two that pass are the ones that do not need generation. The three that
fail — including test_every_family_alias_resolves_to_a_real_model, the test
this issue exists for — fail with an explicit message stating that live
provider behaviour was NOT verified. That is the intended behaviour: they
refuse to report unverified behaviour as working.

What changed since the last comment: Dartmouth Chat now has a green live
job (live-dartmouth, 9 passed), so the project has real-model CI coverage
that does not depend on this account. The README and ADR now name Dartmouth
as the supported provider and state plainly that Anthropic is unverified.

This issue still needs credit on the Anthropic account to close. Nothing else
blocks it.