Skip to content

Refresh opencode model limits (context/output) from the LiteLLM gateway #82

Description

@srid

Problem

The model table in coding-agents/opencode/settings/juspay.nix hand-maintains each
model's context (max input) and output (max output) token limits. These drift
from what the gateway actually serves, and nothing catches the drift.

Concrete example (found while working on #81): glm-latest (GLM-5.2) is configured as
context = 202752; output = 32000, but the gateway advertises and enforces a much
larger window.

The gateway already exposes the source of truth

grid.ai.juspay.net is a LiteLLM proxy. Two authenticated endpoints return live
model metadata (the JUSPAY_API_KEY works for reads — VPN only needed to create a key):

  • GET /v1/models — the list of served model ids
  • GET /v1/model/info — per-model litellm_params.model (the upstream model, e.g.
    zai-org/GLM-5.2-dev) and model_info.{max_input_tokens, max_output_tokens, max_tokens, supports_reasoning, …}

Live values today:

key upstream gateway max_input gateway max_output config today
glm-latest zai-org/GLM-5.2-dev 1000000 1000000 202752 / 32000
glm-flash-experimental GLM-4.7-Flash 128000 128000 262144 / 32000

Both differ from the committed config.

Proposal

Add a way to regenerate the limits from the gateway, e.g.:

  • a just recipe / script (just refresh-models) that curls /v1/model/info with
    JUSPAY_API_KEY and updates the context/output values, and/or
  • a CI check (alongside the daily flake.lock bump) that flags drift.

Caveats to design around

  • Auth in CI/v1/model/info needs JUSPAY_API_KEY as a secret.
  • Placeholder valuesglm-latest reports 1000000 for all three of
    max_tokens/input/output: it's a single shared 1M window, not a 1M output cap.
    max_tokens = 1000000 actually 400s with ContextWindowExceededError once a
    prompt is included, and 1000001 is rejected ("supports at most 1000000 completion
    tokens"). So a naive copy is unsafe — the refresher needs a sane output cap
    (e.g. min(max_output, 128000)). Note OpenCode also caps the wire max_tokens at
    32000 for custom-provider models regardless of limit.output, so the refresher
    should keep output at 32000 — a larger value just shrinks the usable input budget.
  • Hand-curated fields — which models to expose, friendly names, the
    reasoningEffort tiers, and modalities are not in model_info. This should
    update limits for already-listed models, not blindly regenerate the whole table.

Status

#81 corrects the GLM-5.2 limits by hand (context = 1000000, output = 128000).
This issue tracks automating the same for the whole table so it never drifts again
— e.g. glm-flash-experimental is also off (config 262144 / 32000 vs gateway
128000 / 128000).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions