What happens
eval-containers build model bifrost --builder oc fails, reproducibly, twice in a row, ~90s in:
catalog warm failed
error: build error: building at STEP "RUN set -eu; /opt/gateway/main -app-dir /opt/gateway/data -port 4000 ...
for _ in $(seq 1 80); do curl -sf -m1 http://127.0.0.1:4000/api/health && { ok=1; break; }; sleep 0.5; done;
[ "$ok" = 1 ] || { echo "catalog warm failed"; exit 1; }": while running runtime: exit status 1
The build step starts the gateway and polls /api/health for 40s (80 × 0.5s). It never comes up.
Suspicion
main's HEAD at the time was 0a4a9e4 fix(gateways): bump bifrost v1.5.15 -> v1.6.7, so the version bump is the obvious suspect — either v1.6.7 is slower to warm its catalog than 40s allows on a shared node, or it changed the health endpoint/startup contract. Not bisected.
Impact
Blocks build model bifrost on the cluster. It did not block the eval-image path in the end — the combination image takes only BENCHMARK_IMAGE and AGENT_IMAGE, and the gateway is a runtime sidecar — so a cluster run can use an already-published model imagestream. But deploy/examples/openshift/build-eval.sh builds the model in its dependency loop, so the documented fleet path is broken.
Next step
Bisect against v1.5.15 to separate "too slow" from "actually broken". If it is only slow, the retry budget in that RUN step needs raising; the current 40s has no margin on a loaded node.
What happens
eval-containers build model bifrost --builder ocfails, reproducibly, twice in a row, ~90s in:The build step starts the gateway and polls
/api/healthfor 40s (80 × 0.5s). It never comes up.Suspicion
main's HEAD at the time was0a4a9e4 fix(gateways): bump bifrost v1.5.15 -> v1.6.7, so the version bump is the obvious suspect — either v1.6.7 is slower to warm its catalog than 40s allows on a shared node, or it changed the health endpoint/startup contract. Not bisected.Impact
Blocks
build model bifroston the cluster. It did not block the eval-image path in the end — the combination image takes onlyBENCHMARK_IMAGEandAGENT_IMAGE, and the gateway is a runtime sidecar — so a cluster run can use an already-published model imagestream. Butdeploy/examples/openshift/build-eval.shbuilds the model in its dependency loop, so the documented fleet path is broken.Next step
Bisect against v1.5.15 to separate "too slow" from "actually broken". If it is only slow, the retry budget in that RUN step needs raising; the current 40s has no margin on a loaded node.