From 0050146aca7f4233a0af81a878e7de342235ceb9 Mon Sep 17 00:00:00 2001 From: Elron Bandel Date: Sun, 9 Aug 2026 12:19:06 +0300 Subject: [PATCH] =?UTF-8?q?rules(models,gateways):=20remove=20the=20per-mo?= =?UTF-8?q?del=20image=20concept=20=E2=80=94=20one=20shared=20gateway,=20m?= =?UTF-8?q?odels=20are=20runtime-only=20(#294)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Doctrine half of #294 (the #187 pattern). The model axis keeps exactly one image kind: the shared generic gateway, with the model always selected at runtime via EVAL_MODEL. - models rule 1: images that bake an upstream model are forbidden; replay (rule 17) is the sole sanctioned exception. - models rule 12 retargeted to the gateway build-time version pin (gateway._version); rule 13 (EVAL_LITELLM_VERSION runtime override — unimplemented since #50) deprecated in place; rule 15 labels retargeted to gateway.kind; rule 16's enforcement pointer to the deleted core/litellm entrypoint dropped. - gateways rules 16-17: one combo per flavor (models/ FROM gateways/), per-model combos (models/--) forbidden. - RULES.md principle 9 + src rules 10/12: EVAL_LITELLM_VERSION / --litellm-version dropped from the internal-version axis. - add-benchmark template + audit-rules-drift example updated to the single-path story. Checked against: meta/rules 9-10 (changelogs, deprecate-in-place), contributing 1-3. Signed-off-by: Elron Bandel --- .agents/RULES.md | 3 ++- .../add-benchmark/assets/TEMPLATE.md | 15 +++++++------ .agents/gateways/RULES.md | 22 ++++--------------- .agents/models/RULES.md | 13 ++++++----- .agents/src/RULES.md | 5 +++-- .../verification/audit-rules-drift/SKILL.md | 2 +- 6 files changed, 26 insertions(+), 34 deletions(-) diff --git a/.agents/RULES.md b/.agents/RULES.md index df520eb2..cbfd20d1 100644 --- a/.agents/RULES.md +++ b/.agents/RULES.md @@ -33,7 +33,7 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S - **Container version** (the Eval Containers-authored wiring) is selected by the **image tag**, set via `EVAL_BENCHMARK_TAG`, `EVAL_AGENT_TAG`, `EVAL_MODEL_TAG`. This is Docker's native versioning mechanism — different tag, different pull, different bits. A release tag MAY point at a digest produced by an earlier release when the image's build inputs are unchanged ([delivery/RULES.md](delivery/RULES.md) rules 11–15). The fleet-wide default is **one release version**: a SemVer set by the git tag (`latest` on `main`), applied to every image and to the per-benchmark `eval-` compose + `charts/eval` artifacts. `Cargo.toml` and the Helm `Chart.yaml` MUST carry that same version (guard: `tests/static/check.rs`). Bumps: **major** = breaking (a benchmark removed or renamed, the `EVAL_*` contract or `result.json`/output format changed); **minor** = additive (new benchmarks/agents/models, new backward-compatible flags); **patch** = rebuilds, base-image/CVE updates, fixes with no behavior change. The per-component `EVAL_*_TAG` overrides pull a single artifact at a different release version. The tag encodes *our* version, never the upstream software version (that is the Internal version below). - - **Internal version** (the upstream software baked or installed inside) is selected at runtime via `EVAL_BENCHMARK_VERSION`, `EVAL_AGENT_VERSION`, `EVAL_LITELLM_VERSION`. The framework launcher (`/usr/local/bin/run`) MUST read these env vars, install or activate the requested version, and write the resolved version to the run output directory so every run record is self-describing. + - **Internal version** (the upstream software baked or installed inside) is selected at runtime via `EVAL_BENCHMARK_VERSION`, `EVAL_AGENT_VERSION`. The framework launcher (`/usr/local/bin/run`) MUST read these env vars, install or activate the requested version, and write the resolved version to the run output directory so every run record is self-describing. Model images have no internal-version knob: the gateway proxy version is pinned at build time and recorded in the `gateway._version` label ([models](models/RULES.md) rule 12). Both axes are orthogonal: tag controls which container to pull, env var controls what runs inside it. Concrete implementation rules live in `.agents/benchmarks/RULES.md`, `.agents/agents/RULES.md`, and `.agents/models/RULES.md`. @@ -160,3 +160,4 @@ not as issues. The issue tracker is for tracked work only. | 2026-06-14 | Principle 9: the fleet-version default now spans the per-benchmark `eval-` compose artifacts (one self-contained compose per benchmark, flattened at publish) rather than a single shared `evaluate` artifact. A published OCI compose can't carry a dynamic per-benchmark `include:` — `docker compose publish` flattens includes — so per-benchmark sidecars are baked in at publish and `run --mode compose` consumes one artifact with a single `-f`. See [delivery/RULES.md](delivery/RULES.md) rule 3. | | 2026-06-14 | Added the `contributing/` topic: the normative core of the root `CONTRIBUTING.md` moved into [`contributing/RULES.md`](contributing/RULES.md), and the References pointer now targets that doctrine. `CONTRIBUTING.md` becomes the human-facing guide; the issue taxonomy ("Issue vocabulary") stays here. No principle in this file changed. | | 2026-08-09 | Principle 9: a release tag MAY carry forward an earlier release's digest when the image's build inputs are unchanged. Refines — does not repeal — "different tag, different pull, different bits": that phrase constrains what a *different* tag means, not how many tags one digest may carry; carrying the byte-identical digest forward is stricter immutability than a non-reproducible rebuild. Mechanics in [delivery/RULES.md](delivery/RULES.md) rules 11–15. | +| 2026-08-09 | Principle 9: dropped `EVAL_LITELLM_VERSION` from the internal-version axis — per-model images (and their `core/litellm` base, the only implementer) were removed; model images pin the gateway proxy version at build time (`gateway._version`, [models](models/RULES.md) rule 12). The internal-version knob now covers benchmarks and agents only. | diff --git a/.agents/benchmarks/add-benchmark/assets/TEMPLATE.md b/.agents/benchmarks/add-benchmark/assets/TEMPLATE.md index f39fb551..90405298 100644 --- a/.agents/benchmarks/add-benchmark/assets/TEMPLATE.md +++ b/.agents/benchmarks/add-benchmark/assets/TEMPLATE.md @@ -146,13 +146,15 @@ Validate: `eval-containers oracle --task-id --local` — gold MUST ## Non-default canonical (different model or agent) -If a benchmark's canonical isn't `bifrost` × `claude-code`, override: +If a benchmark's canonical isn't `bifrost` × `claude-code`, override. The model +is always runtime configuration (`EVAL_MODEL`); the gateway image only picks +the proxy flavor (`bifrost`/`litellm`/`portkey`): ```yaml -# compose.yaml — add gateway image + EVAL_MODEL overrides +# compose.yaml — add gateway flavor + EVAL_MODEL overrides services: gateway: - image: ${EVAL_REGISTRY:-ghcr.io/exgentic}/models/:latest + image: ${EVAL_REGISTRY:-ghcr.io/exgentic}/models/:latest environment: EVAL_MODEL: runner: @@ -165,10 +167,9 @@ services: # k8s — pass the non-default axes as --set values (no manifest editing): helm template {name} benchmarks/_chart \ --set benchmark={name} \ - --set agent= \ # → runner image evals/{name}-- - --set gatewayImage= \ # → gateway image models/ - --set evalModel= \ - --set model= + --set agent= \ # → runner image evals/{name}-- + --set gatewayImage= \ # → gateway image models/ + --set model= ``` ## Gotchas diff --git a/.agents/gateways/RULES.md b/.agents/gateways/RULES.md index fc1eef3d..fb5789ca 100644 --- a/.agents/gateways/RULES.md +++ b/.agents/gateways/RULES.md @@ -86,24 +86,9 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S 15. **Gateway image coords.** Gateway implementation images live at `/gateways/:`. The flavor is a single token (no `--`). Tag is the version per the project's pin-by-default convention. -16. **Model+gateway combo images are OPTIONAL convenience wrappers.** Pre-built `(model, gateway)` combos MAY be published at `/models/--:`, using the project's double-dash separator convention. The combo image's Dockerfile MUST be `FROM /gateways/:` and MUST add nothing more than the config template (per rule 4) — no baked credentials, no baked model name, no other layers. The combo image is a packaging convenience so users don't have to mount the template themselves; it MUST NOT change runtime behavior compared to running the bare `gateways/` image with the template mounted at the same path. The two deployment styles are equivalent contracts: +16. **One combo per flavor — never per model.** Each gateway flavor publishes exactly one model image at `/models/:`. Its Dockerfile MUST be `FROM /gateways/:` and MUST add nothing more than the config template (per rule 4) — no baked credentials, no baked model name, no other layers. It MUST NOT change runtime behavior compared to running the bare `gateways/` image with the template mounted at the same path; both MUST yield byte-identical routing for the same `EVAL_MODEL`. The framework MUST NOT publish per-model combo images (`models/--`, or any image naming a model) — in the framework, the model is runtime configuration (`EVAL_MODEL`), never an image. Downstream users MAY bake their own template-only derivation of a published `gateways/` in their own registry (the non-default path, [models](../models/RULES.md) rule 1a); it MUST stay template-only and MUST NOT publish under the framework's namespaces. - ``` - # Style A — combo image (preset template baked in) - docker run -e EVAL_MODEL=/ \ - -e \ - /models/--: - - # Style B — bare gateway image (mount your own template) - docker run -e EVAL_MODEL=/ \ - -e \ - -v ./config.json.template:/opt/gateway/data/config.json.template \ - /gateways/: - ``` - - Both MUST yield byte-identical routing behavior for the same `EVAL_MODEL`. Style A is recommended for canonical/shared setups (the template is auditable in the repo); style B is for one-off experimentation. - -17. **Source layout mirrors registry.** `gateways//` in the repo holds the gateway implementation. `models/--/` holds the combo Dockerfile + per-model config files referenced by it. No other directories may publish under `gateways/` or `models/`. +17. **Source layout mirrors registry.** `gateways//` in the repo holds the gateway implementation. `models//` holds that flavor's combo Dockerfile + config template (plus the sanctioned `models/replay` exception, [models](../models/RULES.md) rule 17). No other directories may publish under `gateways/` or `models/`. ### Independence and Composition @@ -122,7 +107,7 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S ## References - [Top-level Rules](../RULES.md) -- [Models](../models/RULES.md) — model-image conventions for pre-built (model, gateway) combos +- [Models](../models/RULES.md) — model-image conventions for the per-flavor combos - [Benchmarks](../benchmarks/RULES.md) - [Compose / Repository](../compose/RULES.md) @@ -136,3 +121,4 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S | 2026-07-22 | Rules 2, 2b, 22 rewritten for the `EVAL_MODEL` (bare, opaque handle — never parsed) + optional `EVAL_MODEL_API` (wire protocol, `anthropic\|openai\|gemini`) model. | | 2026-07-29 | Rule 2b: three modes — `EVAL_MODEL` set ⇒ pin (default keeps the inbound wire so server tools survive), `EVAL_MODEL_API` overrides the wire, unset ⇒ passthrough. Rule 6 permits a header-injection shim (bifrost fronts Caddy to stamp the inbound wire — CEL can't see the path); rule 12 lists the shim config. | | 2026-07-30 | Rule 6: the shim binary+config MUST live under `/opt/gateway/` and be static (Caddy) — bifrost switched nginx→Caddy so the single-container `-standalone` bundle (one `COPY /opt/gateway`) actually boots the gateway (nginx at `/usr/sbin`, musl, silently broke every bundle). Guard: `tests/static/check.rs::gateway_shim_lives_under_opt_gateway`. | +| 2026-08-09 | Rules 16–17 rewritten with the removal of per-model images: each flavor publishes exactly one combo at `models/` (thin template wrapper over `gateways/`); the framework no longer publishes per-model combos (`models/--`) — the model is runtime configuration. Downstream template-only derivations of a published `gateways/` remain permitted (models rule 1a) as the explicit non-default path. | diff --git a/.agents/models/RULES.md b/.agents/models/RULES.md index 628d0b85..23312e6b 100644 --- a/.agents/models/RULES.md +++ b/.agents/models/RULES.md @@ -15,7 +15,9 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S ### Routing -1. **Generic gateway, runtime model.** The model service MUST be a generic LLM proxy whose upstream `/` is selected at **runtime** from `EVAL_MODEL` (a LiteLLM handle, e.g. `openai/gpt-5.4`, `anthropic/claude-sonnet-4-5`). Any LiteLLM-supported provider and model MUST work with no new image — no build, no publish. A model image MAY pin a fixed upstream only as a deliberate exception (e.g. the `replay` image, or a fixed corporate endpoint). +1. **Generic gateway, runtime model — the framework ships no per-model images.** The model service MUST be a generic LLM proxy whose upstream `/` is selected at **runtime** from `EVAL_MODEL` (e.g. `openai/gpt-5.4`, `anthropic/claude-sonnet-4-5`). Any provider and model MUST work with no new image — no build, no publish. The framework MUST NOT ship or publish images that bake a fixed upstream model; the sole in-tree exception is the `replay` image (rule 17), which serves recorded trajectories instead of any upstream. + +1a. **Custom model images are downstream artifacts (non-default).** Users MAY derive a custom model image in their own registry — `FROM` a published `gateways/`, adding only a config template ([gateways](../gateways/RULES.md) rule 16) — and select it through the standard image seam (`EVAL_GATEWAY_IMAGE` / `gatewayImageRef`). Such images are user-owned: they MUST NOT live in the framework's tree or publish under its registry namespaces, and the framework owes them only the stability of the gateway image contract and the image seam. 2. **Wildcard route.** The proxy MUST route every model name the agent requests to the `EVAL_MODEL` upstream — a wildcard (`*`) route, plus any explicit aliases a proxy backend requires where a provider-native passthrough path bypasses the wildcard. @@ -47,19 +49,19 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S ### Versioning -12. **Reproducible by default.** The LiteLLM version MUST be pinned at build time as a default (`ARG LITELLM_VERSION=` or via the `core/litellm` base image tag) and recorded in `eval.model.litellm_version`. The routing layer MUST be reproducible from the pinned image tag and version regardless of which upstream `EVAL_MODEL` selects; the resolved upstream model and version MUST be recorded in the run output. +12. **Reproducible by default.** The proxy version MUST be pinned at build time via the `gateways/` base image and recorded in its `gateway._version` label ([gateways](../gateways/RULES.md) rule 14). The routing layer MUST be reproducible from the pinned image tag regardless of which upstream `EVAL_MODEL` selects; the resolved upstream model MUST be recorded in the run output. -13. **Runtime version override.** The entrypoint MUST read `EVAL_LITELLM_VERSION` and, when set, install or activate that LiteLLM version in place of the default before the proxy starts. The entrypoint MUST write the resolved version to `/output/model/version.json`. When unset, the build-time default applies. `EVAL_MODEL_TAG` selects which container version (image tag) to pull — that's Docker's job, not the entrypoint's. +13. **Deprecated (2026-08-09).** The `EVAL_LITELLM_VERSION` runtime override was removed with the pinned per-model images: the proxy version is fixed at build time (rule 12). `EVAL_MODEL_TAG` still selects which container version (image tag) to pull — that's Docker's job. ### Image 14. **Health endpoint.** The model service MUST expose a health check on port 4000. The eval container MUST wait for it before starting. -15. **Labels.** Every model image MUST include labels: `eval.type`, `eval.model.name`, `eval.model.provider`, `eval.model.litellm_version`. +15. **Labels.** Every model image MUST include `eval.type` and its gateway's `gateway.kind` ([gateways](../gateways/RULES.md) rule 14); the `replay` exception instead carries `eval.model.name` and `eval.model.provider`. ### Budget -16. **Hard budget cap.** The proxy MUST enforce a per-run hard cap on spend via `EVAL_MODEL_MAX_BUDGET` (USD). When crossed, the proxy MUST reject further requests with `BudgetExceededError` so the agent's next call fails fast. Default cap is `$1`. Configurable via `.env` or `eval-containers run --max-budget `; no model-specific value MAY be hardcoded in image config (per [.agents/compose/RULES.md](../compose/RULES.md) rule 10). The enforcement entrypoint lives in `containers/core/litellm/eval-litellm-entrypoint.sh` and rewrites `/app/config.yaml`'s `max_budget` at container start from the env var. +16. **Hard budget cap.** The proxy MUST enforce a per-run hard cap on spend via `EVAL_MODEL_MAX_BUDGET` (USD). When crossed, the proxy MUST reject further requests with `BudgetExceededError` so the agent's next call fails fast. Default cap is `$1`. Configurable via `.env` or `eval-containers run --max-budget `; no model-specific value MAY be hardcoded in image config (per [.agents/compose/RULES.md](../compose/RULES.md) rule 10). Enforcement is each gateway flavor's job at container start. ### Replay @@ -79,3 +81,4 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S | 2026-06-14 | Added rule 17 (Replay): the replay model serves recorded trajectories with no API keys, indistinguishable from a live service. Absorbed from the retired `tests/containers/RULES.md` (rules 5–6) during the test-governance heal. | | 2026-04-15 | Added rule 16: `EVAL_MODEL_MAX_BUDGET` hard-cap (default $1) enforced by the shared core/litellm entrypoint wrapper. | | 2026-06-17 | Rewrote rule 1 (Routing): the model service is a **generic gateway** that selects `/` at runtime from `EVAL_MODEL`, so any LiteLLM-supported model works with no new image (no build, no publish) — matching the already-generic `bifrost`/`litellm` gateway images and resolving the contradiction with rule 9 (any provider *without modifying Eval Containers*). Updated the abstract (generic proxy), rule 2 (the wildcard routes the `EVAL_MODEL` upstream), and rule 12 (reproducibility from the pinned image tag + version + recorded resolved upstream, not a baked model). Pinned per-model images are now a deliberate exception (e.g. `replay`). Doctrine half of #187. | +| 2026-08-09 | Per-model images removed from the framework: rule 1 now forbids the framework shipping images that bake an upstream model — `replay` (rule 17) is the sole in-tree exception. New rule 1a keeps custom model images possible as **downstream** artifacts (user-owned, `FROM gateways/` + template only, selected via the standard image seam) — explicitly non-default. The five pinned images (`gpt-5`, `gpt-5.4`, `gpt-4.1-mini`, `claude-opus-4`, `claude-sonnet-4`) and their `core/litellm` base are deleted; only the gateway-flavor combos (`models/bifrost\|litellm\|portkey`) and `models/replay` remain. Rule 12 retargeted to the gateway version pin (`gateway._version`); rule 13 (`EVAL_LITELLM_VERSION` runtime override) deprecated in place; rule 15 labels retargeted to `gateway.kind`; rule 16's enforcement pointer (`core/litellm/eval-litellm-entrypoint.sh`) dropped with the base image. | diff --git a/.agents/src/RULES.md b/.agents/src/RULES.md index cce9bb3e..74ac5266 100644 --- a/.agents/src/RULES.md +++ b/.agents/src/RULES.md @@ -53,13 +53,13 @@ Each tool MUST be a standard release the user can install and invoke themselves 9. **Local-first.** The CLI SHOULD prefer locally cached images. It MUST support `--local` for development against local compose files. -10. **Env var ↔ CLI flag parity.** Every `EVAL_*` environment variable documented in the README or used by any `oci://` compose artifact MUST have a matching `--kebab-case` CLI flag derived by stripping `EVAL_` and lowercasing: `EVAL_BENCHMARK` → `--benchmark`, `EVAL_AGENT_VERSION` → `--agent-version`, `EVAL_TASK_ID` → `--task-id`, `EVAL_TIMEOUT` → `--timeout`, `EVAL_LITELLM_VERSION` → `--litellm-version`, and so on. No exceptions: if it's an env var the user can set, it MUST have a flag form. Positional shortcuts (e.g. `eval-containers run aime` accepting `aime` as the benchmark) are allowed but MUST NOT replace the corresponding `--flag`; both forms MUST work. When both a CLI flag and an env var are set, the CLI flag MUST override the env var. The CLI's sole job in `eval-containers run` is to translate every flag into its corresponding `EVAL_*` env var and shell out to the standard command for the selected `--mode` — the exact `docker compose … up`, `docker run …`, or `helm template … \| kubectl apply -f -` the README documents. +10. **Env var ↔ CLI flag parity.** Every `EVAL_*` environment variable documented in the README or used by any `oci://` compose artifact MUST have a matching `--kebab-case` CLI flag derived by stripping `EVAL_` and lowercasing: `EVAL_BENCHMARK` → `--benchmark`, `EVAL_AGENT_VERSION` → `--agent-version`, `EVAL_TASK_ID` → `--task-id`, `EVAL_TIMEOUT` → `--timeout`, and so on. No exceptions: if it's an env var the user can set, it MUST have a flag form. Positional shortcuts (e.g. `eval-containers run aime` accepting `aime` as the benchmark) are allowed but MUST NOT replace the corresponding `--flag`; both forms MUST work. When both a CLI flag and an env var are set, the CLI flag MUST override the env var. The CLI's sole job in `eval-containers run` is to translate every flag into its corresponding `EVAL_*` env var and shell out to the standard command for the selected `--mode` — the exact `docker compose … up`, `docker run …`, or `helm template … \| kubectl apply -f -` the README documents. ### Commands 11. **Build.** `eval-containers build agent|bench|model|eval` — each MUST map to a single `docker buildx bake ` invocation, which executes the artifact's build graph declared in its `docker-bake.hcl` (top-level RULES.md principle 15). Per-task variants (`--task-id`), which sit outside the static bake graph, fall through to a single `docker build`. The combined eval image has **two** bake targets: `eval` (the lean base) and `eval-standalone` (the single-container standalone bundle, `FROM` the lean base via `core/standalone.Dockerfile`); `build eval --standalone` selects the bundle, which builds `eval` first as a wired dependency so the bundle has a real build-graph node (principle 3 — no ad-hoc `docker build` for a published artifact). The variant is a name suffix (`evals/---standalone`), never a tag — the `:tag` is the release version (principle 9). The bundle's `--local` build (`run --mode container --local`) is likewise a single `docker build -f core/standalone.Dockerfile --build-context eval-base=docker-image://`, outside the bake graph. Building in a cluster is **not** a separate code path: it is the same `docker buildx bake` pointed at an in-cluster builder (`docker buildx create --driver kubernetes`). The CLI exposes this as `build --builder ` — a passthrough of buildx's own `--builder` that implies `--push` (a remote builder can't load into local Docker); a missing builder fails with the one-time `docker buildx create` command to run. The reserved value `build --builder oc` selects the **OpenShift `BuildConfig` backend** instead of buildx: it builds a single artifact in-cluster with `oc start-build` (buildah under the platform's `builder` SCC) — the no-admin path where baseline PodSecurity blocks in-cluster BuildKit. It MUST read the artifact's resolved build spec — `context`, `dockerfile`, and the eval combination's base-image args — from `docker buildx bake --print ` (the bake file stays the only build-graph artifact, principle 3); it adds only the OpenShift-specific translation: the single-segment imagestream naming (`core/otel` → `core-otel`, `benchmarks/aime` → `aime`) and the `REGISTRY`/`REGISTRY_SUFFIX` build args that resolve the parameterized `${REGISTRY}/...${REGISTRY_SUFFIX}` FROMs to the internal registry (binary builds ignore `oc start-build --build-arg`, so they live in the BuildConfig spec). Either backend builds **one** artifact: the CLI MUST NOT re-derive or order the build graph for any platform; dependency-ordered cold-graph builds are a thin loop over `build` that lives outside the CLI (e.g. `deploy/examples/openshift/`). Every build also stamps the OCI provenance label `org.opencontainers.image.source` (the pointer GitHub uses to link a package to its repo — on an Actions push or a one-time UI "Connect repository"; a manual push doesn't auto-link) — fleet-wide via `--set "*.labels…"`, and explicit on the per-task `docker build`; never stored in the per-artifact bake files (top-level principle 15.f). -12. **Run.** `eval-containers run {benchmark} --agent {name} --task-id {id}` — maps to the standard command for the chosen `--mode`: `docker compose up` (compose, the default), `docker run` (container), or `helm template -f --set … \| kubectl apply -f -` (job). `--mode container` runs the **standalone bundle** (`evals/---standalone`, the lean base + the in-process gateway/otelcol/process-compose); `--mode compose`/`job` run the lean base with the gateway + otelcol as sidecars. MUST accept both the container-tag axis (`--benchmark-tag`, `--agent-tag`, `--model-tag`) and the internal-version axis (`--benchmark-version`, `--agent-version`, `--litellm-version`), plus `--model`, `--timeout`, `--local`. Cluster- and platform-specific settings for `job` mode (e.g. the service account an OpenShift cluster requires) MUST be supplied as a composable Helm values file via `--overlay ` — not encoded per-platform inside the CLI (principle 3); the reference OpenShift overlay is `deploy/values-openshift.yaml`. +12. **Run.** `eval-containers run {benchmark} --agent {name} --task-id {id}` — maps to the standard command for the chosen `--mode`: `docker compose up` (compose, the default), `docker run` (container), or `helm template -f --set … \| kubectl apply -f -` (job). `--mode container` runs the **standalone bundle** (`evals/---standalone`, the lean base + the in-process gateway/otelcol/process-compose); `--mode compose`/`job` run the lean base with the gateway + otelcol as sidecars. MUST accept both the container-tag axis (`--benchmark-tag`, `--agent-tag`, `--model-tag`) and the internal-version axis (`--benchmark-version`, `--agent-version`), plus `--model`, `--timeout`, `--local`. Cluster- and platform-specific settings for `job` mode (e.g. the service account an OpenShift cluster requires) MUST be supplied as a composable Helm values file via `--overlay ` — not encoded per-platform inside the CLI (principle 3); the reference OpenShift overlay is `deploy/values-openshift.yaml`. 13. **Report.** `eval-containers report ./output/` — MUST walk the output directory, read `result.json` files, and aggregate. MUST support `--format csv|json`. @@ -90,3 +90,4 @@ Each tool MUST be a standard release the user can install and invoke themselves | 2026-06-03 | Principle 11 (Build) + tools table: added the reserved `build --builder oc` OpenShift `BuildConfig` backend — reads the build spec (context, dockerfile, eval base-image args) from `docker buildx bake --print` (bake stays the source of truth) and translates it to `oc start-build` (buildah), the no-admin in-cluster path where baseline PodSecurity blocks BuildKit. One artifact, no graph ordering in the CLI; the ordered loop lives in `deploy/examples/openshift/`. | | 2026-06-11 | Principle 11 (Build): every build stamps the OCI provenance label `org.opencontainers.image.source` (the pointer GitHub uses to link a package to its repo, via an Actions push or UI connect — not a manual push) — fleet-wide via `--set "*.labels…"`, explicit on per-task `docker build`; not in the per-artifact bake files (15.f). | | 2026-06-14 | Lean base + single-container standalone bundle (#142). Principle 11 (Build): the combined eval image gains a second bake target — `eval` (lean base) + `eval-standalone` (the standalone bundle, `FROM` the lean base via `core/standalone.Dockerfile`); `build eval --standalone` selects it. The variant is a name suffix (`evals/---standalone`), not a tag (the tag is the version). Principle 12 (Run): `--mode container` runs the standalone bundle; `--mode compose`/`job` run the lean base with the gateway + otelcol as sidecars. Tools table: `docker build` also builds the local `--mode container` bundle via `core/standalone.Dockerfile --build-context eval-base=docker-image://…`. | +| 2026-08-09 | Dropped `--litellm-version` / `EVAL_LITELLM_VERSION` from principles 10 and 12 — the knob's only implementer (the pinned per-model images and their `core/litellm` base) was removed; the gateway proxy version is a build-time pin ([models](../models/RULES.md) rule 12). | diff --git a/.agents/verification/audit-rules-drift/SKILL.md b/.agents/verification/audit-rules-drift/SKILL.md index f970d45f..cc29de93 100644 --- a/.agents/verification/audit-rules-drift/SKILL.md +++ b/.agents/verification/audit-rules-drift/SKILL.md @@ -80,7 +80,7 @@ report (what was flagged last time). 5. **For every mechanizable drift, propose a new catalog rule.** State the rule ID, severity, and the predicate (e.g. "assert `core/runner/run` references `EVAL_BENCHMARK_VERSION` and writes `/output/task/version.json`"; - "assert every `models/*/Dockerfile` contains `LABEL eval.model.litellm_version=`"; + "assert every `models/*/Dockerfile` is `FROM` a `gateways/` image"; "assert no benchmark `compose.yaml` uses `EVAL_*_VERSION` as a Docker image tag"; "assert every file under `tests/run/replay/fixtures/` ends in `.traces.jsonl`"). WHY: this is the mechanical > procedural escalation