refactor(chart): bundle benchmark presets in the chart (self-contained, OCI-ready)

CHANGELOG.md

@@ -42,6 +42,14 @@ addition; the patch on a bug fix that doesn't change the rule surface.
 
 ### Changed
 
+- **The k8s `job` mode is now a self-contained Helm chart.** A benchmark
+  is selected with `--set benchmark=<x>` instead of
+  `-f benchmarks/<x>/values.yaml`; the 4 benchmarks with bespoke topology
+  (`osworld`, `tau-bench`, `visualwebarena`, `webarena`) moved into the
+  chart as `benchmarks/_chart/presets/<x>.yaml` (loaded via `.Files.Get`),
+  and the 98 one-line `values.yaml` files were deleted. The chart now
+  renders with no external file, so it can be packaged and published to an
+  OCI registry. Renders byte-identical to the prior `-f values.yaml` form.
 - **Agent Dockerfiles: 1957 → 585 lines (70% reduction)** across all
   20 agents via the Rule 11 refactor onto shared bases.
 - **91 of 100 benchmarks** refactored to extend `core/benchmark-base-*`.

README.md

@@ -77,22 +77,22 @@ eval-containers run aime --task-id 0 --agent codex --mode job
 |---|---|---|
 | `compose` *(default)* | `docker compose -f benchmarks/<x>/compose.yaml up` | Local laptop, full stack with gateway + OTel sidecars, fastest iteration. |
 | `container` | `docker run -e EVAL_MODEL=... <eval-image>` | CI smoke tests, one-shot runs against an existing model proxy, minimal footprint. |
-| `job` | `helm template benchmarks/_chart -f benchmarks/<x>/values.yaml \| kubectl apply -f -` | Kubernetes clusters. Production-scale regressions (1000s of tasks in parallel). |
+| `job` | `helm template benchmarks/_chart --set benchmark=<x> \| kubectl apply -f -` | Kubernetes clusters. Production-scale regressions (1000s of tasks in parallel). |
 
 ### Kubernetes (`--mode job`)
 
-Every benchmark is a small [Helm](https://helm.sh/) `values.yaml` over one shared chart (`benchmarks/_chart`) — render it and apply, no CLI needed:
+Every benchmark renders from one shared [Helm](https://helm.sh/) chart (`benchmarks/_chart`) — select it with `--set benchmark=<x>` and apply, no CLI needed. A benchmark with bespoke topology (extra Deployments/sidecars) adds a `presets/<x>.yaml` inside the chart; standard ones need nothing:
 
 ```bash
-helm template aime benchmarks/_chart -f benchmarks/aime/values.yaml \
+helm template aime benchmarks/_chart --set benchmark=aime \
   --set agent=claude-code,task=0 | kubectl apply -f -
 ```
 
 The CLI does exactly that, mapping every axis to a `--set`:
 
 ```bash
 eval-containers run aime --agent codex --task-id 42 --mode job
-# → helm template aime-codex-task-42 benchmarks/_chart -f benchmarks/aime/values.yaml \
+# → helm template aime-codex-task-42 benchmarks/_chart --set benchmark=aime \
 #       --set registry=…,agent=codex,task=42 | kubectl apply -f -
 ```
 
@@ -102,7 +102,7 @@ Platform specifics (corp registry, NodeAffinity, NetworkPolicies, a different se
 eval-containers run aime --agent codex --mode job \
   --overlay deploy/values-openshift.yaml \
   --registry image-registry.openshift-image-registry.svc:5000/<namespace>
-# → helm template … -f benchmarks/aime/values.yaml -f deploy/values-openshift.yaml … | kubectl apply -f -
+# → helm template … --set benchmark=aime -f deploy/values-openshift.yaml … | kubectl apply -f -
 ```
 
 On OpenShift, create the service account once and use `oc` in place of `kubectl`:

benchmarks/osworld/values.yaml → benchmarks/_chart/presets/osworld.yaml

@@ -1,4 +1,3 @@
-benchmark: osworld
 activeDeadlineSeconds: 2100
 timeout: '1800'
 resources:

benchmarks/tau-bench/values.yaml → benchmarks/_chart/presets/tau-bench.yaml

@@ -1,4 +1,3 @@
-benchmark: tau-bench
 activeDeadlineSeconds: 2100
 timeout: '1800'
 runnerArgs: python3 /app/agent.py; rc=$?; /usr/local/bin/reap-sidecars; exit $rc

benchmarks/visualwebarena/values.yaml → benchmarks/_chart/presets/visualwebarena.yaml

@@ -1,4 +1,3 @@
-benchmark: visualwebarena
 activeDeadlineSeconds: 2100
 timeout: '1800'
 resources:

benchmarks/webarena/values.yaml → benchmarks/_chart/presets/webarena.yaml

@@ -1,4 +1,3 @@
-benchmark: webarena
 activeDeadlineSeconds: 2100
 timeout: '1800'
 resources:

benchmarks/_chart/templates/_helpers.tpl

@@ -1,6 +1,22 @@
+{{/*
+Effective values = chart defaults / --set overrides (.Values) with the selected
+benchmark's preset overlaid. The benchmark is named via `--set benchmark=<x>`;
+its bespoke topology (sidecars, resources, extra manifests) lives in
+`presets/<x>.yaml` inside the chart, loaded here so a `helm template` of the
+packaged chart needs no external file. Standard benchmarks have no preset —
+`.Files.Get` returns "" → empty overlay → the chart defaults apply unchanged.
+Presets only set structural keys; the per-run axes (agent/task/model/…) come
+from --set and are never in a preset, so preset-wins is safe.
+*/}}
+{{- define "eval.values" -}}
+{{- $name := required "benchmark is required (--set benchmark=<x>)" .Values.benchmark -}}
+{{- $preset := .Files.Get (printf "presets/%s.yaml" $name) | fromYaml | default dict -}}
+{{- mergeOverwrite (deepCopy .Values) $preset | toYaml -}}
+{{- end -}}
+
 {{/* Shared labels: benchmark/agent/task, plus sweep-id only when set. */}}
 {{- define "eval.labels" -}}
-benchmark: {{ required "benchmark is required (set it in benchmarks/<x>/values.yaml)" .Values.benchmark }}
+benchmark: {{ required "benchmark is required (--set benchmark=<x>)" .Values.benchmark }}
 agent: {{ .Values.agent }}
 task: {{ .Values.task | quote }}
 {{- with .Values.sweepId }}

benchmarks/_chart/templates/job.yaml

@@ -1,30 +1,31 @@
+{{- $v := include "eval.values" . | fromYaml -}}
 apiVersion: batch/v1
 kind: Job
 metadata:
-  name: {{ .Values.benchmark }}-{{ .Values.agent }}-task-{{ .Values.task }}
+  name: {{ $v.benchmark }}-{{ $v.agent }}-task-{{ $v.task }}
   labels:
     {{- include "eval.labels" . | nindent 4 }}
 spec:
   backoffLimit: 0
   ttlSecondsAfterFinished: 3600
-  activeDeadlineSeconds: {{ .Values.activeDeadlineSeconds }}
+  activeDeadlineSeconds: {{ $v.activeDeadlineSeconds }}
   template:
     metadata:
       labels:
         {{- include "eval.labels" . | nindent 8 }}
     spec:
       restartPolicy: Never
       shareProcessNamespace: true
-      {{- with .Values.serviceAccountName }}
+      {{- with $v.serviceAccountName }}
       serviceAccountName: {{ . }}
       {{- end }}
-      {{- with .Values.initContainers }}
+      {{- with $v.initContainers }}
       initContainers:
         {{- toYaml . | nindent 8 }}
       {{- end }}
       containers:
         - name: otelcol
-          image: {{ .Values.registry }}/core/otel:latest
+          image: {{ $v.registry }}/core/otel:latest
           imagePullPolicy: IfNotPresent
           ports:
             - { name: otlp-grpc, containerPort: 4317 }
@@ -34,7 +35,7 @@ spec:
             - { name: output, mountPath: /output }
 
         - name: gateway
-          image: {{ .Values.registry }}/models/{{ .Values.gatewayImage }}:{{ .Values.gatewayTag }}
+          image: {{ $v.registry }}/models/{{ $v.gatewayImage }}:{{ $v.gatewayTag }}
           imagePullPolicy: IfNotPresent
           ports:
             - { name: gateway, containerPort: 4000 }
@@ -45,78 +46,78 @@ spec:
             - name: OPENAI_API_BASE
               valueFrom:
                 secretKeyRef: { name: eval-secrets, key: OPENAI_API_BASE }
-            - { name: EVAL_MODEL,                  value: {{ .Values.evalModel | quote }} }
+            - { name: EVAL_MODEL,                  value: {{ $v.evalModel | quote }} }
             - { name: HOST,                        value: "127.0.0.1" }
             # shareProcessNamespace + sidecar pod model puts otelcol on localhost.
             - { name: OTEL_EXPORTER_OTLP_ENDPOINT, value: "http://localhost:4318" }
-            {{- with .Values.maxBudget }}
+            {{- with $v.maxBudget }}
             - { name: EVAL_MODEL_MAX_BUDGET, value: {{ . | quote }} }
             {{- end }}
-            {{- with .Values.litellmVersion }}
+            {{- with $v.litellmVersion }}
             - { name: EVAL_LITELLM_VERSION, value: {{ . | quote }} }
             {{- end }}
-            {{- with .Values.gatewayExtraEnv }}
+            {{- with $v.gatewayExtraEnv }}
             {{- toYaml . | nindent 12 }}
             {{- end }}
 
         - name: runner
-          image: {{ .Values.registry }}/evals/{{ .Values.benchmark }}--{{ .Values.agent }}:{{ .Values.runnerTag }}
+          image: {{ $v.registry }}/evals/{{ $v.benchmark }}--{{ $v.agent }}:{{ $v.runnerTag }}
           imagePullPolicy: IfNotPresent
           command: ["/bin/bash", "-c"]
           # $? / $rc are the container's shell — Helm leaves them untouched
           # (they're not {{`{{ }}`}} actions). A few benchmarks override the
           # command (e.g. tau-bench runs python3 /app/agent.py).
-          args: [{{ .Values.runnerArgs | quote }}]
+          args: [{{ $v.runnerArgs | quote }}]
           env:
             # Framework axes in BOTH namespaces, mirroring compose/services.yaml:
             # EVAL_* is canonical (read by /eval-entrypoint.sh + per-benchmark
             # /entrypoint.sh — incl. the task materialization that sets $TASK);
             # the bare names are the process-compose legacy path. Set both so the
             # run works regardless of which entrypoint the eval image uses.
-            - { name: EVAL_BENCHMARK,              value: {{ .Values.benchmark | quote }} }
-            - { name: EVAL_AGENT,                  value: {{ .Values.agent | quote }} }
-            - { name: EVAL_MODEL,                  value: {{ .Values.model | quote }} }
-            - { name: EVAL_TASK_ID,                value: {{ .Values.task | quote }} }
-            - { name: EVAL_TIMEOUT,                value: {{ .Values.timeout | quote }} }
-            - { name: BENCHMARK,                   value: {{ .Values.benchmark | quote }} }
-            - { name: AGENT,                       value: {{ .Values.agent | quote }} }
-            - { name: MODEL,                       value: {{ .Values.model | quote }} }
-            - { name: TASK_ID,                     value: {{ .Values.task | quote }} }
-            - { name: TIMEOUT,                     value: {{ .Values.timeout | quote }} }
+            - { name: EVAL_BENCHMARK,              value: {{ $v.benchmark | quote }} }
+            - { name: EVAL_AGENT,                  value: {{ $v.agent | quote }} }
+            - { name: EVAL_MODEL,                  value: {{ $v.model | quote }} }
+            - { name: EVAL_TASK_ID,                value: {{ $v.task | quote }} }
+            - { name: EVAL_TIMEOUT,                value: {{ $v.timeout | quote }} }
+            - { name: BENCHMARK,                   value: {{ $v.benchmark | quote }} }
+            - { name: AGENT,                       value: {{ $v.agent | quote }} }
+            - { name: MODEL,                       value: {{ $v.model | quote }} }
+            - { name: TASK_ID,                     value: {{ $v.task | quote }} }
+            - { name: TIMEOUT,                     value: {{ $v.timeout | quote }} }
             - { name: ANTHROPIC_BASE_URL,          value: "http://localhost:4000/anthropic" }
             - { name: OPENAI_BASE_URL,             value: "http://localhost:4000/openai/v1" }
             - { name: GOOGLE_GEMINI_BASE_URL,      value: "http://localhost:4000/genai" }
             - { name: ANTHROPIC_API_KEY,           value: "sk-proxy" }
             - { name: OPENAI_API_KEY,              value: "sk-proxy" }
             - { name: GEMINI_API_KEY,              value: "sk-proxy" }
             - { name: OTEL_EXPORTER_OTLP_ENDPOINT, value: "http://localhost:4318" }
-            {{- with .Values.benchmarkVersion }}
+            {{- with $v.benchmarkVersion }}
             - { name: EVAL_BENCHMARK_VERSION, value: {{ . | quote }} }
             {{- end }}
-            {{- with .Values.agentVersion }}
+            {{- with $v.agentVersion }}
             - { name: EVAL_AGENT_VERSION, value: {{ . | quote }} }
             {{- end }}
-            {{- with .Values.runnerExtraEnv }}
+            {{- with $v.runnerExtraEnv }}
             {{- toYaml . | nindent 12 }}
             {{- end }}
           resources:
-            {{- toYaml .Values.resources | nindent 12 }}
+            {{- toYaml $v.resources | nindent 12 }}
           volumeMounts:
             - { name: output, mountPath: /output }
             - { name: tmp,    mountPath: /tmp }
             - { name: logs,   mountPath: /logs }
-            {{- with .Values.runnerExtraVolumeMounts }}
+            {{- with $v.runnerExtraVolumeMounts }}
             {{- toYaml . | nindent 12 }}
             {{- end }}
 
       volumes:
         - { name: output, emptyDir: {} }
         - { name: tmp,    emptyDir: { medium: Memory, sizeLimit: 1Gi } }
         - { name: logs,   emptyDir: { medium: Memory, sizeLimit: 100Mi } }
-        {{- with .Values.extraVolumes }}
+        {{- with $v.extraVolumes }}
         {{- toYaml . | nindent 8 }}
         {{- end }}
-{{- range .Values.extraManifests }}
+{{- range $v.extraManifests }}
 ---
 {{ toYaml . }}
 {{- end }}

benchmarks/_chart/values.yaml

@@ -1,7 +1,8 @@
-# Defaults for the shared eval Job. A benchmark's values.yaml sets `benchmark`
-# (required) and overrides only what differs; the per-run axes (agent, task,
-# model, …) come from `--set` (or the eval-containers CLI). envsubst is gone:
-# Helm fills these and gets the types right (task stays a quoted string).
+# Defaults for the shared eval Job. The benchmark is selected with
+# `--set benchmark=<x>` (required); a benchmark with bespoke topology overrides
+# what differs via an optional `presets/<x>.yaml` in this chart. The per-run axes
+# (agent, task, model, …) come from `--set` (or the eval-containers CLI). envsubst
+# is gone: Helm fills these and gets the types right (task stays a quoted string).
 
 # ── Per-run axes (overridden at deploy via --set / the eval-containers CLI) ──
 agent: claude-code

benchmarks/aime/README.md

@@ -33,7 +33,7 @@ Uses the shared `core/test-exact-match` scorer: the agent's stdout is compared a
 - `Dockerfile` — builds the benchmark base image (tasks data + verifier).
 - `container.Dockerfile` — single-mode deployment artifact (1-line registry pin).
 - `compose.yaml` — compose-mode deployment artifact (`include:` shared base + aime overrides).
-- `values.yaml` — k8s-mode deployment artifact (Helm values over the shared `benchmarks/_chart`).
+- k8s — the shared chart `benchmarks/_chart`, selected with `--set benchmark=aime` (no per-benchmark file; aime has no bespoke topology).
 - `README.md` — this file.
 
 ## Running — three deployment surfaces
@@ -42,7 +42,7 @@ Uses the shared `core/test-exact-match` scorer: the agent's stdout is compared a
 |------|------|------------|
 | **single** | `container.Dockerfile` | `docker run -e OPENAI_API_KEY=… -e OPENAI_API_BASE=… <image>` |
 | **compose** | `compose.yaml` | `docker compose -f benchmarks/aime/compose.yaml up` |
-| **k8s** | `values.yaml` | `helm template aime benchmarks/_chart -f benchmarks/aime/values.yaml \| kubectl apply -f -` (needs `eval-secrets`) |
+| **k8s** | shared chart | `helm template aime benchmarks/_chart --set benchmark=aime \| kubectl apply -f -` (needs `eval-secrets`) |
 
 ```bash
 # Single mode — just docker run
@@ -60,7 +60,7 @@ OPENAI_API_KEY=… OPENAI_API_BASE=… \
 kubectl create secret generic eval-secrets \
   --from-literal=OPENAI_API_KEY="$OPENAI_API_KEY" \
   --from-literal=OPENAI_API_BASE="$OPENAI_API_BASE"
-helm template aime benchmarks/_chart -f benchmarks/aime/values.yaml | kubectl apply -f -
+helm template aime benchmarks/_chart --set benchmark=aime | kubectl apply -f -
 ```
 
 ## Different task
@@ -74,7 +74,7 @@ TASK_ID=42 docker compose -f benchmarks/aime/compose.yaml up
 For k8s, the task is a Helm value:
 
 ```bash
-helm template aime benchmarks/_chart -f benchmarks/aime/values.yaml --set task=42 | kubectl apply -f -
+helm template aime benchmarks/_chart --set benchmark=aime --set task=42 | kubectl apply -f -
 ```
 
 ## Build args