Observed
Per-task benchmark images carry eval.benchmark.upstream_base, but its value is
that one baked task's upstream image, not the benchmark's:
$ oc image info --filter-by-os=linux/amd64 \
ghcr.io/exgentic/evals/swe-bench-astropy__astropy-12907--claude-code:latest -o json
eval.benchmark.name = swe-bench
eval.benchmark.upstream_base = ghcr.io/epoch-research/swe-bench.eval.x86_64.astropy__astropy-12907:latest
$ oc image info --filter-by-os=linux/amd64 \
ghcr.io/exgentic/evals/swe-bench-sympy__sympy-24661--claude-code:latest -o json
eval.benchmark.name = swe-bench
eval.benchmark.upstream_base = ghcr.io/epoch-research/swe-bench.eval.x86_64.sympy__sympy-24661:latest
Every other eval.benchmark.* key is identical across the two (name,
description, env, tasks, paper, url, data_revision, harbor_impl,
inspect_impl, internet). upstream_base is the only one that varies, i.e.
the only one that is not a property of the benchmark.
The sibling per-task keys on the same image are already correctly namespaced
outside eval.benchmark.* — swe-bench.instance_id, swe-bench.repo,
swe-bench.env_image.
Expected
A key under eval.benchmark.* describes the benchmark, so any consumer grouping
by that prefix can present the group as benchmark-level facts. upstream_base
should move to a per-task prefix (eval.task.upstream_base, or the existing
<family>. namespace alongside instance_id).
Impact
Exgentic/dashboard renders a run's eval.benchmark.* labels as the benchmark's
provenance panel. For a 500-task swe-bench run there is no family-level image,
so it reads a task image, and upstream_base would show one arbitrary task's
base as the whole benchmark's. Exgentic/dashboard#53 works around it by dropping
the key by name; that workaround is deleted once this lands.
The same shape will recur for any consumer that trusts the prefix, and for any
per-task key added under eval.benchmark.* later.
Environment
ghcr.io/exgentic/evals/* at :latest, observed 2026-08-05. Affects the three
per-task families: swe-bench (500 tasks), terminal-bench (89), skills-bench (87).
Shared-env families are unaffected — they have one image, so nothing varies.
Observed
Per-task benchmark images carry
eval.benchmark.upstream_base, but its value isthat one baked task's upstream image, not the benchmark's:
Every other
eval.benchmark.*key is identical across the two (name,description,env,tasks,paper,url,data_revision,harbor_impl,inspect_impl,internet).upstream_baseis the only one that varies, i.e.the only one that is not a property of the benchmark.
The sibling per-task keys on the same image are already correctly namespaced
outside
eval.benchmark.*—swe-bench.instance_id,swe-bench.repo,swe-bench.env_image.Expected
A key under
eval.benchmark.*describes the benchmark, so any consumer groupingby that prefix can present the group as benchmark-level facts.
upstream_baseshould move to a per-task prefix (
eval.task.upstream_base, or the existing<family>.namespace alongsideinstance_id).Impact
Exgentic/dashboard renders a run's
eval.benchmark.*labels as the benchmark'sprovenance panel. For a 500-task swe-bench run there is no family-level image,
so it reads a task image, and
upstream_basewould show one arbitrary task'sbase as the whole benchmark's. Exgentic/dashboard#53 works around it by dropping
the key by name; that workaround is deleted once this lands.
The same shape will recur for any consumer that trusts the prefix, and for any
per-task key added under
eval.benchmark.*later.Environment
ghcr.io/exgentic/evals/*at:latest, observed 2026-08-05. Affects the threeper-task families: swe-bench (500 tasks), terminal-bench (89), skills-bench (87).
Shared-env families are unaffected — they have one image, so nothing varies.