Exgentic · elronbandel · Aug 9, 2026
diff --git a/.agents/RULES.md b/.agents/RULES.md
@@ -33,7 +33,7 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S
 
    - **Container version** (the Eval Containers-authored wiring) is selected by the **image tag**, set via `EVAL_BENCHMARK_TAG`, `EVAL_AGENT_TAG`, `EVAL_MODEL_TAG`. This is Docker's native versioning mechanism — different tag, different pull, different bits. A release tag MAY point at a digest produced by an earlier release when the image's build inputs are unchanged ([delivery/RULES.md](delivery/RULES.md) rules 11–15). The fleet-wide default is **one release version**: a SemVer set by the git tag (`latest` on `main`), applied to every image and to the per-benchmark `eval-<benchmark>` compose + `charts/eval` artifacts. `Cargo.toml` and the Helm `Chart.yaml` MUST carry that same version (guard: `tests/static/check.rs`). Bumps: **major** = breaking (a benchmark removed or renamed, the `EVAL_*` contract or `result.json`/output format changed); **minor** = additive (new benchmarks/agents/models, new backward-compatible flags); **patch** = rebuilds, base-image/CVE updates, fixes with no behavior change. The per-component `EVAL_*_TAG` overrides pull a single artifact at a different release version. The tag encodes *our* version, never the upstream software version (that is the Internal version below).
 
-   - **Internal version** (the upstream software baked or installed inside) is selected at runtime via `EVAL_BENCHMARK_VERSION`, `EVAL_AGENT_VERSION`, `EVAL_LITELLM_VERSION`. The framework launcher (`/usr/local/bin/run`) MUST read these env vars, install or activate the requested version, and write the resolved version to the run output directory so every run record is self-describing.
+   - **Internal version** (the upstream software baked or installed inside) is selected at runtime via `EVAL_BENCHMARK_VERSION`, `EVAL_AGENT_VERSION`. The framework launcher (`/usr/local/bin/run`) MUST read these env vars, install or activate the requested version, and write the resolved version to the run output directory so every run record is self-describing. Model images have no internal-version knob: the gateway proxy version is pinned at build time and recorded in the `gateway.<flavor>_version` label ([models](models/RULES.md) rule 12).
 
    Both axes are orthogonal: tag controls which container to pull, env var controls what runs inside it. Concrete implementation rules live in `.agents/benchmarks/RULES.md`, `.agents/agents/RULES.md`, and `.agents/models/RULES.md`.
 
@@ -160,3 +160,4 @@ not as issues. The issue tracker is for tracked work only.
 | 2026-06-14 | Principle 9: the fleet-version default now spans the per-benchmark `eval-<benchmark>` compose artifacts (one self-contained compose per benchmark, flattened at publish) rather than a single shared `evaluate` artifact. A published OCI compose can't carry a dynamic per-benchmark `include:` — `docker compose publish` flattens includes — so per-benchmark sidecars are baked in at publish and `run --mode compose` consumes one artifact with a single `-f`. See [delivery/RULES.md](delivery/RULES.md) rule 3. |
 | 2026-06-14 | Added the `contributing/` topic: the normative core of the root `CONTRIBUTING.md` moved into [`contributing/RULES.md`](contributing/RULES.md), and the References pointer now targets that doctrine. `CONTRIBUTING.md` becomes the human-facing guide; the issue taxonomy ("Issue vocabulary") stays here. No principle in this file changed. |
 | 2026-08-09 | Principle 9: a release tag MAY carry forward an earlier release's digest when the image's build inputs are unchanged. Refines — does not repeal — "different tag, different pull, different bits": that phrase constrains what a *different* tag means, not how many tags one digest may carry; carrying the byte-identical digest forward is stricter immutability than a non-reproducible rebuild. Mechanics in [delivery/RULES.md](delivery/RULES.md) rules 11–15. |
+| 2026-08-09 | Principle 9: dropped `EVAL_LITELLM_VERSION` from the internal-version axis — per-model images (and their `core/litellm` base, the only implementer) were removed; model images pin the gateway proxy version at build time (`gateway.<flavor>_version`, [models](models/RULES.md) rule 12). The internal-version knob now covers benchmarks and agents only. |
diff --git a/.agents/benchmarks/add-benchmark/assets/TEMPLATE.md b/.agents/benchmarks/add-benchmark/assets/TEMPLATE.md
@@ -146,13 +146,15 @@ Validate: `eval-containers oracle <name> --task-id <task> --local` — gold MUST
 
 ## Non-default canonical (different model or agent)
 
-If a benchmark's canonical isn't `bifrost` × `claude-code`, override:
+If a benchmark's canonical isn't `bifrost` × `claude-code`, override. The model
+is always runtime configuration (`EVAL_MODEL`); the gateway image only picks
+the proxy flavor (`bifrost`/`litellm`/`portkey`):
 
 ```yaml
-# compose.yaml — add gateway image + EVAL_MODEL overrides
+# compose.yaml — add gateway flavor + EVAL_MODEL overrides
 services:
   gateway:
-    image: ${EVAL_REGISTRY:-ghcr.io/exgentic}/models/<other-combo>:latest
+    image: ${EVAL_REGISTRY:-ghcr.io/exgentic}/models/<other-flavor>:latest
     environment:
       EVAL_MODEL: <other-provider/other-model>
   runner:
@@ -165,10 +167,9 @@ services:
 # k8s — pass the non-default axes as --set values (no manifest editing):
 helm template {name} benchmarks/_chart \
   --set benchmark={name} \
-  --set agent=<other-agent> \          # → runner image evals/{name}--<other-agent>
-  --set gatewayImage=<other-combo> \   # → gateway image models/<other-combo>
-  --set evalModel=<other-provider/other-model> \
-  --set model=<friendly-label>
+  --set agent=<other-agent> \           # → runner image evals/{name}--<other-agent>
+  --set gatewayImage=<other-flavor> \   # → gateway image models/<other-flavor>
+  --set model=<other-provider/other-model>
 ```
 
 ## Gotchas

diff --git a/.agents/gateways/RULES.md b/.agents/gateways/RULES.md
@@ -86,24 +86,9 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S
 
 15. **Gateway image coords.** Gateway implementation images live at `<registry>/gateways/<flavor>:<tag>`. The flavor is a single token (no `--`). Tag is the version per the project's pin-by-default convention.
 
-16. **Model+gateway combo images are OPTIONAL convenience wrappers.** Pre-built `(model, gateway)` combos MAY be published at `<registry>/models/<model>--<gateway>:<tag>`, using the project's double-dash separator convention. The combo image's Dockerfile MUST be `FROM <registry>/gateways/<gateway>:<tag>` and MUST add nothing more than the config template (per rule 4) — no baked credentials, no baked model name, no other layers. The combo image is a packaging convenience so users don't have to mount the template themselves; it MUST NOT change runtime behavior compared to running the bare `gateways/<flavor>` image with the template mounted at the same path. The two deployment styles are equivalent contracts:
+16. **One combo per flavor — never per model.** Each gateway flavor publishes exactly one model image at `<registry>/models/<flavor>:<tag>`. Its Dockerfile MUST be `FROM <registry>/gateways/<flavor>:<tag>` and MUST add nothing more than the config template (per rule 4) — no baked credentials, no baked model name, no other layers. It MUST NOT change runtime behavior compared to running the bare `gateways/<flavor>` image with the template mounted at the same path; both MUST yield byte-identical routing for the same `EVAL_MODEL`. The framework MUST NOT publish per-model combo images (`models/<model>--<gateway>`, or any image naming a model) — in the framework, the model is runtime configuration (`EVAL_MODEL`), never an image. Downstream users MAY bake their own template-only derivation of a published `gateways/<flavor>` in their own registry (the non-default path, [models](../models/RULES.md) rule 1a); it MUST stay template-only and MUST NOT publish under the framework's namespaces.
 
-    ```
-    # Style A — combo image (preset template baked in)
-    docker run -e EVAL_MODEL=<provider>/<model> \
-               -e <upstream-creds> \
-               <registry>/models/<model>--<gateway>:<tag>
-
-    # Style B — bare gateway image (mount your own template)
-    docker run -e EVAL_MODEL=<provider>/<model> \
-               -e <upstream-creds> \
-               -v ./config.json.template:/opt/gateway/data/config.json.template \
-               <registry>/gateways/<gateway>:<tag>
-    ```
-
-    Both MUST yield byte-identical routing behavior for the same `EVAL_MODEL`. Style A is recommended for canonical/shared setups (the template is auditable in the repo); style B is for one-off experimentation.
-
-17. **Source layout mirrors registry.** `gateways/<flavor>/` in the repo holds the gateway implementation. `models/<model>--<gateway>/` holds the combo Dockerfile + per-model config files referenced by it. No other directories may publish under `gateways/` or `models/`.
+17. **Source layout mirrors registry.** `gateways/<flavor>/` in the repo holds the gateway implementation. `models/<flavor>/` holds that flavor's combo Dockerfile + config template (plus the sanctioned `models/replay` exception, [models](../models/RULES.md) rule 17). No other directories may publish under `gateways/` or `models/`.
 
 ### Independence and Composition
 
@@ -122,7 +107,7 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S
 ## References
 
 - [Top-level Rules](../RULES.md)
-- [Models](../models/RULES.md) — model-image conventions for pre-built (model, gateway) combos
+- [Models](../models/RULES.md) — model-image conventions for the per-flavor combos
 - [Benchmarks](../benchmarks/RULES.md)
 - [Compose / Repository](../compose/RULES.md)
 
@@ -136,3 +121,4 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S
 | 2026-07-22 | Rules 2, 2b, 22 rewritten for the `EVAL_MODEL` (bare, opaque handle — never parsed) + optional `EVAL_MODEL_API` (wire protocol, `anthropic\|openai\|gemini`) model. |
 | 2026-07-29 | Rule 2b: three modes — `EVAL_MODEL` set ⇒ pin (default keeps the inbound wire so server tools survive), `EVAL_MODEL_API` overrides the wire, unset ⇒ passthrough. Rule 6 permits a header-injection shim (bifrost fronts Caddy to stamp the inbound wire — CEL can't see the path); rule 12 lists the shim config. |
 | 2026-07-30 | Rule 6: the shim binary+config MUST live under `/opt/gateway/` and be static (Caddy) — bifrost switched nginx→Caddy so the single-container `-standalone` bundle (one `COPY /opt/gateway`) actually boots the gateway (nginx at `/usr/sbin`, musl, silently broke every bundle). Guard: `tests/static/check.rs::gateway_shim_lives_under_opt_gateway`. |
+| 2026-08-09 | Rules 16–17 rewritten with the removal of per-model images: each flavor publishes exactly one combo at `models/<flavor>` (thin template wrapper over `gateways/<flavor>`); the framework no longer publishes per-model combos (`models/<model>--<gateway>`) — the model is runtime configuration. Downstream template-only derivations of a published `gateways/<flavor>` remain permitted (models rule 1a) as the explicit non-default path. |
diff --git a/.agents/models/RULES.md b/.agents/models/RULES.md
@@ -15,7 +15,9 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S
 
 ### Routing
 
-1. **Generic gateway, runtime model.** The model service MUST be a generic LLM proxy whose upstream `<provider>/<model>` is selected at **runtime** from `EVAL_MODEL` (a LiteLLM handle, e.g. `openai/gpt-5.4`, `anthropic/claude-sonnet-4-5`). Any LiteLLM-supported provider and model MUST work with no new image — no build, no publish. A model image MAY pin a fixed upstream only as a deliberate exception (e.g. the `replay` image, or a fixed corporate endpoint).
+1. **Generic gateway, runtime model — the framework ships no per-model images.** The model service MUST be a generic LLM proxy whose upstream `<provider>/<model>` is selected at **runtime** from `EVAL_MODEL` (e.g. `openai/gpt-5.4`, `anthropic/claude-sonnet-4-5`). Any provider and model MUST work with no new image — no build, no publish. The framework MUST NOT ship or publish images that bake a fixed upstream model; the sole in-tree exception is the `replay` image (rule 17), which serves recorded trajectories instead of any upstream.
+
+1a. **Custom model images are downstream artifacts (non-default).** Users MAY derive a custom model image in their own registry — `FROM` a published `gateways/<flavor>`, adding only a config template ([gateways](../gateways/RULES.md) rule 16) — and select it through the standard image seam (`EVAL_GATEWAY_IMAGE` / `gatewayImageRef`). Such images are user-owned: they MUST NOT live in the framework's tree or publish under its registry namespaces, and the framework owes them only the stability of the gateway image contract and the image seam.
 
 2. **Wildcard route.** The proxy MUST route every model name the agent requests to the `EVAL_MODEL` upstream — a wildcard (`*`) route, plus any explicit aliases a proxy backend requires where a provider-native passthrough path bypasses the wildcard.
 
@@ -47,19 +49,19 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S
 
 ### Versioning
 
-12. **Reproducible by default.** The LiteLLM version MUST be pinned at build time as a default (`ARG LITELLM_VERSION=<semver>` or via the `core/litellm` base image tag) and recorded in `eval.model.litellm_version`. The routing layer MUST be reproducible from the pinned image tag and version regardless of which upstream `EVAL_MODEL` selects; the resolved upstream model and version MUST be recorded in the run output.
+12. **Reproducible by default.** The proxy version MUST be pinned at build time via the `gateways/<flavor>` base image and recorded in its `gateway.<flavor>_version` label ([gateways](../gateways/RULES.md) rule 14). The routing layer MUST be reproducible from the pinned image tag regardless of which upstream `EVAL_MODEL` selects; the resolved upstream model MUST be recorded in the run output.
 
-13. **Runtime version override.** The entrypoint MUST read `EVAL_LITELLM_VERSION` and, when set, install or activate that LiteLLM version in place of the default before the proxy starts. The entrypoint MUST write the resolved version to `/output/model/version.json`. When unset, the build-time default applies. `EVAL_MODEL_TAG` selects which container version (image tag) to pull — that's Docker's job, not the entrypoint's.
+13. **Deprecated (2026-08-09).** The `EVAL_LITELLM_VERSION` runtime override was removed with the pinned per-model images: the proxy version is fixed at build time (rule 12). `EVAL_MODEL_TAG` still selects which container version (image tag) to pull — that's Docker's job.
 
 ### Image
 
 14. **Health endpoint.** The model service MUST expose a health check on port 4000. The eval container MUST wait for it before starting.
 
-15. **Labels.** Every model image MUST include labels: `eval.type`, `eval.model.name`, `eval.model.provider`, `eval.model.litellm_version`.
+15. **Labels.** Every model image MUST include `eval.type` and its gateway's `gateway.kind` ([gateways](../gateways/RULES.md) rule 14); the `replay` exception instead carries `eval.model.name` and `eval.model.provider`.
 
 ### Budget
 
-16. **Hard budget cap.** The proxy MUST enforce a per-run hard cap on spend via `EVAL_MODEL_MAX_BUDGET` (USD). When crossed, the proxy MUST reject further requests with `BudgetExceededError` so the agent's next call fails fast. Default cap is `$1`. Configurable via `.env` or `eval-containers run --max-budget <N>`; no model-specific value MAY be hardcoded in image config (per [.agents/compose/RULES.md](../compose/RULES.md) rule 10). The enforcement entrypoint lives in `containers/core/litellm/eval-litellm-entrypoint.sh` and rewrites `/app/config.yaml`'s `max_budget` at container start from the env var.
+16. **Hard budget cap.** The proxy MUST enforce a per-run hard cap on spend via `EVAL_MODEL_MAX_BUDGET` (USD). When crossed, the proxy MUST reject further requests with `BudgetExceededError` so the agent's next call fails fast. Default cap is `$1`. Configurable via `.env` or `eval-containers run --max-budget <N>`; no model-specific value MAY be hardcoded in image config (per [.agents/compose/RULES.md](../compose/RULES.md) rule 10). Enforcement is each gateway flavor's job at container start.
 
 ### Replay
 
@@ -79,3 +81,4 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S
 | 2026-06-14 | Added rule 17 (Replay): the replay model serves recorded trajectories with no API keys, indistinguishable from a live service. Absorbed from the retired `tests/containers/RULES.md` (rules 5–6) during the test-governance heal. |
 | 2026-04-15 | Added rule 16: `EVAL_MODEL_MAX_BUDGET` hard-cap (default $1) enforced by the shared core/litellm entrypoint wrapper. |
 | 2026-06-17 | Rewrote rule 1 (Routing): the model service is a **generic gateway** that selects `<provider>/<model>` at runtime from `EVAL_MODEL`, so any LiteLLM-supported model works with no new image (no build, no publish) — matching the already-generic `bifrost`/`litellm` gateway images and resolving the contradiction with rule 9 (any provider *without modifying Eval Containers*). Updated the abstract (generic proxy), rule 2 (the wildcard routes the `EVAL_MODEL` upstream), and rule 12 (reproducibility from the pinned image tag + version + recorded resolved upstream, not a baked model). Pinned per-model images are now a deliberate exception (e.g. `replay`). Doctrine half of #187. |
+| 2026-08-09 | Per-model images removed from the framework: rule 1 now forbids the framework shipping images that bake an upstream model — `replay` (rule 17) is the sole in-tree exception. New rule 1a keeps custom model images possible as **downstream** artifacts (user-owned, `FROM gateways/<flavor>` + template only, selected via the standard image seam) — explicitly non-default. The five pinned images (`gpt-5`, `gpt-5.4`, `gpt-4.1-mini`, `claude-opus-4`, `claude-sonnet-4`) and their `core/litellm` base are deleted; only the gateway-flavor combos (`models/bifrost\|litellm\|portkey`) and `models/replay` remain. Rule 12 retargeted to the gateway version pin (`gateway.<flavor>_version`); rule 13 (`EVAL_LITELLM_VERSION` runtime override) deprecated in place; rule 15 labels retargeted to `gateway.kind`; rule 16's enforcement pointer (`core/litellm/eval-litellm-entrypoint.sh`) dropped with the base image. |