Gateway (LiteLLM) model-metadata API gaps block automated/runtime model-limit refresh

[srid]

Summary

While working on #82 (refreshing opencode model limits from the gateway) — and exploring the
broader idea of having the wrapper fetch models dynamically at runtime instead of hardcoding
them in Nix — it turns out the gateway (grid.ai.juspay.net, a LiteLLM proxy) does not
currently expose a clean, complete source of truth for per-model metadata. Several gaps make any
automated refresh or runtime-fetch approach unsafe today.

This issue documents those API shortcomings so they can be raised with the gateway owners.

How this was checked: plain curl with a JUSPAY_API_KEY read key (no VPN), against
/v1/models, /v1/model/info, and /model_group/info. Snapshot taken 2026-06-21.

Shortcomings

1. /v1/model/info doesn't cover all served models

/v1/models advertises 19 model ids, but /v1/model/info returns metadata for only 11 of
them. Eight served models have no model_info entry at all — including models we actively use
in config:

gemini-3-pro-preview        ← in our config (juspay.nix), no limits available
gemini-3-flash-preview      ← in our config, no limits available
minimaxai/minimax-m2        ← served as minimax-m2 in our config, no limits available
open-fast                   ← in our config (small_model + explore agent), no limits available
claude-sonnet-4-5-20250929
claude-haiku-4-5-20251001
claude-3-5-haiku-20241022
claude-3-5-haiku@20241022

A refresher/runtime-fetcher that reads /v1/model/info simply cannot get limits for these.

2. Duplicate deployments per model_name report conflicting limits

Some model_names appear multiple times in /v1/model/info (multiple backing deployments) with
different values, and nothing indicates which is authoritative:

model_name	entries	(max_input, max_output, max_tokens)
kimi-latest	2	(262144, 32000, 32000) and (262144, 32000, 262144) ⚠️
open-large	2	(262144, 32000, 32000) and (262144, 32000, 262144) ⚠️
claude-sonnet-4-6	4	all (1000000, 64000, 64000) (consistent)
gemini-embedding-001	2	all (2048, –, 2048) (consistent)

A consumer has to guess a merge policy (min? max? first?) for kimi-latest/open-large.

3. Placeholder caps: shared context window reported as the output cap

For some models all three of max_tokens / max_input_tokens / max_output_tokens collapse to a
single number, which is the shared context window, not a real output cap:

model_name	max_input	max_output	max_tokens
glm-latest	1000000	1000000	1000000
glm-flash-experimental	128000	128000	128000

Per #82, this is actively misleading: requesting max_tokens = 1000000 on glm-latest
400s with ContextWindowExceededError once a prompt is included, and 1000001 is rejected
with "supports at most 1000000 completion tokens". So a naive copy of max_output_tokens
produces a config the gateway then rejects. We need a real, separate max_output_tokens.

4. Surprising model_name → upstream aliasing

claude-opus-4-5 is backed by upstream vertex_ai/claude-opus-4-6 (litellm_params.model).
If intentional (alias), fine — but it means the served name doesn't describe what actually runs,
which is worth confirming/documenting.

5. No authorized endpoint for complete metadata with a read key

/model_group/info (which can return grouped/consolidated info) returns 403 with the read
key. So there's no single authenticated endpoint that yields complete, de-duplicated per-model
limits without elevated access.

Asks (what would unblock #82 / runtime fetch)

1. Parity: every id in /v1/models should have a /v1/model/info entry with limits.
2. Authoritative limits per model_name: either de-duplicate, or guarantee all deployments of
   a name agree, or expose a single consolidated value.
3. Real output caps: report max_output_tokens distinct from the context window; don't return
   the shared window as the output cap (the value should be one the gateway will actually accept).
4. A read-key-accessible metadata endpoint (e.g. fix /model_group/info perms, or document the
   intended one) so consumers can fetch complete metadata without VPN/admin scope.
5. (Nice to have) confirm/document intentional aliases like claude-opus-4-5 → claude-opus-4-6.

Notes for our side

Even if all of the above were fixed, fields opencode needs that are not in model_info —
friendly model selection, reasoningEffort tiers, which subset to expose — remain hand-curated.
So this is about making limits trustworthy, not regenerating the whole table. See #82 (refresh
limits) and #81 (reasoning-effort tiers) for the consumer-side work.

_{Filed from a spike investigating dynamic/runtime model fetching for the opencode wrapper.}