Re-verify codex-core shell_snapshot::tests::try_new_* on current main: 2026-07-25 baselines were certified by the broken testbox gate (#421)

[andrei-hasna]

Summary

codex-core shell_snapshot::tests::try_new_creates_and_deletes_snapshot_file and
shell_snapshot::tests::try_new_uses_distinct_generation_paths were genuinely failing on
main on 2026-07-25 (both panicking with snapshot should be created: "validation_failed").

Several PRs excluded those two tests from their testbox verification as "pre-existing and
environmental", partly on the strength of baseline probe runs whose reported exit status was
fabricated by the testbox gate bug fixed in #421 (merged as 9fd3620).

We need to re-establish, against today's main and with the fixed gate, (a) whether these
tests still fail, and (b) whether the exclusions were justified.

Why the baseline evidence is not trustworthy

Before #421, .github/workflows/blacksmith-testbox.yml interpolated the dispatch input directly
into the step script (bash -lc "${{ inputs.build_command }}"), so the outer shell expanded
$?, ${PIPESTATUS[0]}, $(...) and $((...)) before the inner shell ever parsed the command.

The baseline probe in question, run
30147337015 (main @ 23a664c),
submitted:

cd codex-rs && cargo nextest run --locked -p codex-core --no-fail-fast \
  -E 'test(shell_snapshot::tests::try_new_)'; echo "BASELINE_MAIN_EXIT=$?"; exit 0

The workflow log shows what the shell actually received:

##[group]cd codex-rs && cargo nextest run --locked -p codex-core --no-fail-fast -E 'test(shell_snapshot::tests::try_new_)'; echo BASELINE_MAIN_EXIT=0; exit 0

$? had already been replaced with the literal 0. BASELINE_MAIN_EXIT=0 was a hard-coded
constant, not the test exit status, and the trailing exit 0 forced the run to
conclusion=success regardless of the outcome.

What the logs actually show

To be precise about what is and is not in doubt:

• The body of run 30147337015's log does contain a real nextest verdict:
  Summary [0.193s] 2 tests run: 0 passed, 2 failed. So the underlying observation ("these two
  fail on main") is corroborated by that run's own output — it is the headline
  BASELINE_MAIN_EXIT=0 token and the green run conclusion that are worthless.
• Independently, run 30146410545
  (main, full cargo nextest run -p codex-core --no-fail-fast) also failed both
  try_new_* tests at core/src/shell_snapshot_tests.rs:301 and :332.

So the 2026-07-25 "pre-existing on main" conclusion looks substantively right, but it was
reported on the back of a probe that could not have detected the opposite answer. That is the
problem: the method could not fail, so it certifies nothing, and any other exclusion approved
by the same pattern is equally uncertified.

Affected / to audit

• PR fix(goal): harden blocked-goal hold semantics #379 (fix(goal): harden blocked-goal hold semantics, merged 2026-07-25) — explicitly
  excluded test(shell_snapshot::tests::try_new_) citing run 30147337015 as the main baseline.
  Related runs: 30146755675 (head,
  2 failures), 30147333548 (head with
  the two tests excluded → success).
• Any other PR merged on or before 2026-07-25 that excluded tests as "pre-existing" on the basis of
  a testbox baseline probe using $?, ${PIPESTATUS[0]}, $(...) or $((...)).

Ask

1. Re-run the two shell_snapshot::tests::try_new_* tests against today's main via
   blacksmith-testbox.yml with the ci(blacksmith): stop the testbox gate from reporting failures as success #421 fix in place (no ; exit 0, no $? echo — the gate
   now reports the real status by itself). Record the run id.
2. If they still fail: determine whether it is a genuine defect or a Blacksmith-runner environment
   limitation (the PR fix(goal): harden blocked-goal hold semantics #379 rationale was "shell_snapshot needs a real login shell"), and either
   fix the tests, fix the runner env, or mark them #[ignore]/env-gated so they stop being excluded
   ad hoc by every PR.
3. If they now pass: identify what changed and confirm nothing was merged behind an unjustified
   exclusion.
4. Sweep the pre-2026-07-26 testbox baseline probes for the same fabricated-status pattern and
   re-verify any exclusion that rested on one.

Context

• Fix: ci(blacksmith): stop the testbox gate from reporting failures as success #421 — ci(blacksmith): stop the testbox gate from reporting failures as success (9fd3620).
• Other false verdict found in the same audit: run
  30190166288 — clippy error[E0004]
  reported as conclusion=success. 2 of 38 audited runs produced false verdicts; neither reached
  main.