Skip to content

Remove per-model gateway images — one shared model gateway, models are runtime-only #294

Description

@elronbandel

Problem

The model axis currently has two image kinds: the generic gateway flavors (models/bifrost|litellm|portkey, model selected at runtime via EVAL_MODEL) and five pinned per-model images (models/gpt-5, models/gpt-5.4, models/gpt-4.1-mini, models/claude-opus-4, models/claude-sonnet-4, all FROM core/litellm).

The per-model images create constant confusion:

  • Two overlapping meanings of "model": an image name (EVAL_GATEWAY_IMAGE=gpt-5.4) vs a runtime handle (EVAL_MODEL=openai/gpt-5.4). The deploy/oc tooling conflates them (--set gatewayImage=$MODEL with --model bifrost), and its handle plumbing (evalModel) isn't even read by the chart.
  • A parallel base image (core/litellm) and version machinery (EVAL_LITELLM_VERSION, eval.model.litellm_version labels, rego gates) that exist only for these five images — the runtime override has had no implementer since Delete dead eval-entrypoint.sh #50.
  • Docs/templates must explain "two paths" when the generic gateway already covers every model with zero build (Models: any LiteLLM handle must work with zero build — finish the generic-proxy transition; rule 1 is stale #187 made it the default).

Proposal

Kill the per-model image concept entirely. One shared model gateway: EVAL_MODEL=<provider>/<model> is the only way to pick a model; EVAL_GATEWAY_IMAGE picks only the proxy flavor (bifrost default; litellm/portkey). models/replay stays as the sole sanctioned pinned exception (models rule 17).

  • Delete the five containers/models/<model> dirs and the now-orphaned containers/core/litellm base.
  • Doctrine: models rule 1 forbids per-model images; deprecate rules 12–13's litellm-version machinery; gateways rules 16–17 become "one combo per flavor, never per model"; drop EVAL_LITELLM_VERSION/--litellm-version from principle 9 and src rules 10/12.
  • Retarget the guards: static sweep + check.rs assert every models/* image is FROM gateways/<flavor> (replay excepted); remove the unreachable litellm-version rego rules.
  • Fix the deploy/oc scripts so --model is the handle and --gateway the flavor.
  • Collapse docs/PR/issue templates to the single-path story.

Per contributing rule 2, this lands as two PRs: a doctrine half and an implementation half (the #187 pattern).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions