GAIA image hardcodes web_search="live", which cannot be disabled and fails closed in air-gapped deployments

[natan-ergas]

Summary

/run.sh in the GAIA codex image hardcodes live web search on. There is no
env var, flag, or config path that turns it off, because run-agent launches
the agent under a scrubbed env -i allow-list that web_search is not on (and
it is not env-driven in codex anyway). In a cluster with no egress and no search
backend, codex offers the model a tool that always errors — burning turns and
degrading scores for reasons unrelated to the model under test.

Since whether search is available is a property of the deployment, not of the
benchmark, this seems like it should be configurable rather than fixed.

Where it is

/run.sh:

#!/bin/bash
# web_search must be the top-level key (codex ignores [tools].web_search).
export OPENAI_API_KEY="${OPENAI_API_KEY:-sk-proxy}"
exec codex exec \
  --model "${CODEX_MODEL:-default}" \
  ...
  -c 'web_search="live"' \
  -c 'tools.view_image=true' \
  ...

codex --help (codex-cli 0.120.0) documents search as opt-in:

      --search
          Enable live web search. When enabled, the native Responses `web_search` tool is available

So codex's own default is off, and the image is deliberately turning it on.

Why it cannot be worked around

run-agent scrubs the environment:

timeout "${TIMEOUT:-300}" gosu agent env -i \
  PATH=... HOME=/home/agent USER=agent LANG=C.UTF-8 \
  TASK="${TASK:-}" MODEL="${MODEL:-}" TIMEOUT="${TIMEOUT:-300}" \
  ...
  /run.sh > /output/agent/stdout.log 2> /output/agent/stderr.log || rc=$?

web_search is not on the allow-list, so nothing set outside the container
reaches it. The only lever is editing /run.sh — which is what we do, at
runtime, with a guarded match-or-skip patch. That works, but it means every
air-gapped consumer of this image has to patch a file inside it.

Suggested fix

Make it env-gated, defaulting to the current behaviour so nothing changes for
existing users:

SEARCH_ARGS=()
case "${EVAL_WEB_SEARCH:-live}" in
  off|false|0) ;;                                   # omit the flag entirely
  *)           SEARCH_ARGS+=(-c 'web_search="live"') ;;
esac
exec codex exec --model "${CODEX_MODEL:-default}" ... "${SEARCH_ARGS[@]}" ...

with EVAL_WEB_SEARCH added to the run-agent allow-list.

We suggest omitting the flag rather than setting web_search="off" or
false. We could not verify a disable enum: codex exec -c web_search=bogus --help also exits 0 because --help short-circuits before config validation,
so that probe proves nothing and a guessed value risks being silently ignored.
Omission lands on codex's documented default, which is the behaviour we want.

Relationship to EVAL_GAIA_SUBSET

EVAL_GAIA_SUBSET=no-search is a different lever — it is task selection
(keep the 54 tasks that don't need search, skip the 111 that do). It does not
stop codex from offering a broken tool within the tasks that remain. Both are
useful; neither substitutes for the other.

Environment

• Image: ghcr.io/exgentic/evals/gaia--codex-standalone:latest
  (digest sha256:d5c4ef4100bcf391056b511a6e676af4722cc1fd4a2580df583c2944d3baf7df,
  built 2026-08-10)
• codex-cli 0.120.0; re-verified 2026-08-12 against that image: /run.sh
  still carries -c 'web_search="live"' unconditionally, and run-agent's
  env -i allow-list is PATH/HOME/USER/LANG/TASK/MODEL/TIMEOUT/
  EVAL_AGENT_REASONING_EFFORT plus the three provider base-URL/key pairs --
  no web_search, no EVAL_WEB_SEARCH. Both claims below still hold.
• Deployment: llm-d on OpenShift, no egress, model served by vLLM.