Verify and repair the shell tool for native-default Codex runs

[molebox]

TLDR is that new versions of codex cli seem to have had a bug in our harness. If we didnt pass an explicit model to it - running as native default, then it couldnt access tools, and even made up output. It could still respond though, and so our research runs have been semi-broken since July 9th.

Problem

Codex CLI >= 0.144.0 (published 2026-07-09; bisected — 0.143.0 is the last good version) exposes no shell/exec tool to the model when config.toml uses a custom model_provider (e.g. the AI Gateway) and omits the model key — exactly the shape our native-default runs write. The model still answers, so runs "complete", but it cannot run commands, read files, or use installed skills. Worse, it was observed fabricating command output for predictable commands instead of reporting the missing tool (claimed echo output it never ran; a random-nonce test proved no tool existed).

Since agent CLIs are installed unpinned at run time, every native-default gateway Codex task since ~2026-07-09 has run toolless. Downstream impact observed in a0-local production data: codex usedWebResearch is 0/180 in the two most recent research-baseline runs (gateway codex researches via shell/curl), and plugin-lane codex shards cannot have read their installed skills.

The single deciding variable, verified with an isolated CODEX_HOME: adding an explicit model = "<the same model the CLI resolves natively>" to the profile config fully restores the tool.

Fix (run.mjs, native-default runs only)

1. Shell canary before the real task: ask codex to echo <random nonce>. Proof requires a command_execution item carrying the nonce — an agent_message merely echoing it does NOT count (fabrication-proof).
2. Repair on failure: read the model the CLI natively resolved from the canary's own session file and re-state it as an explicit top-level model key in ~/.codex/default.config.toml (prepended — a line appended after a [table] header would join that table). Same semantics: the CLI chose the model; we only re-state it.
3. Re-verify, fail loudly: if the canary still fails, the run errors instead of executing a toolless agent and publishing it as normal behavior.
4. The repair is surfaced as an optional modelRepair field on RunnerResult (evidence, additive).

Explicit-model runs already write model into the config and are unaffected (verified).

Verification

• 6 new unit tests (canary prompt, fabrication rejection, failed-exec rejection, repair TOML shape/prefixing); full suite green (277 passed).
• End-to-end against the AI Gateway with codex-cli 0.145.0 and an isolated HOME: canary fails → repair writes model = "openai/gpt-5.6-sol" → canary passes → real task runs with a genuine command_execution reading a random nonce file.
• Version bisect: 0.140.0 / 0.142.5 / 0.143.0 OK; 0.144.0 / 0.144.4 / 0.144.6 / 0.145.0 toolless.

Cost note

Native-default runs pay one tiny canary exec per task (two on the repair path). Given the fabrication behavior, post-hoc transcript checks cannot substitute for pre-verification.

[molebox]

Addressed three findings from a multi-agent review pass (f820e33):

1. modelRepair now actually propagates — RunnerResult → AgentRunResult → EvalRunResult, mirroring observedModel, so persisted results record which runs needed the repair. Repairs dropping to zero across runs is the removal signal for this workaround once the upstream CLI is fixed.
2. Canary fail-loud path preserves evidence — the captured canary CLI output/transcript is returned on the error result (previously discarded), matching the login and real-exec failure paths, so a still-toolless failure is triageable.
3. Canary memoized per sandbox — verified outcome written to ~/.codex/agent-eval-canary.json; judge assertions (toSatisfyCriterion/toScoreAtLeast re-invoke run.mjs once per assertion under validation: 'vitest') no longer pay a canary exec each. The marker re-reports the original repairedModel so every invocation carries consistent evidence.

Live verification against the gateway: first invocation repairs + records modelRepair: gpt-5.6-sol (14.6s incl. real task); second invocation in the same HOME skips the canary via the marker (1.7s) and still reports the repair. 279 unit tests green (2 new for the marker parser).

Known residuals (from the review, below reporting threshold): the pre-existing lexicographic session-file fallback could attribute a canary session to a real exec that dies before thread.started — narrow trigger, worth a follow-up excluding canary thread ids; and the 180s canary timeout is generous for an echo round-trip.

[molebox]

Second review pass addressed (dd84f36):

• modelRepair plumbing now tested at the same layers as observedModel (its precedent): agentResultToEvalRunData mapping + omit-when-absent shape, persisted result.json, and the runner-level path. A refactor can no longer silently drop the repair evidence that serves as the workaround's removal signal.
• Fallback channel carries modelRepair: the __AGENT_RESULT__ status line and readRunnerResult's reconstruction now preserve it, matching observedModel (previously lost whenever the result file couldn't be read back).
• Canary timeout 180s → 60s — an echo round-trip never needs more; worst-case pre-task overhead drops from 360s to 120s of the 600s default sandbox budget.
• Cross-reference comments at the orchestrator's sandbox-creation step and eval-helper's header document the canary marker's dependence on the shared per-run sandbox lifetime, so a future sandbox-per-invocation change surfaces the coupling.

280 tests green, build clean.