Standardize article-suite timeout fairness

[bingran-you]

Summary

• audit all 180 selected model-task-trial trajectories and their retry histories
• standardize every model to a 3,600-second agent-idle timeout and 3,600-second Daytona PTY-read timeout
• identify 10 cells influenced by the prior model-specific safeguards:
  • 8 cells with a 900-second GPT PTY timeout in retry history
  • 2 selected GPT cells terminated by the prior 600-second idle watchdog
• rerun only those 10 affected cells on Daytona under protocol v2.2
• regenerate all leaderboard, provenance, website, and plot artifacts
• publish a machine-readable fairness audit and detailed methodology note

Audit result

• selected cells audited: 180/180
• historically affected cells rerun: 10
• currently selected idle/PTY-influenced cells: 0
• shared wall-clock timeout cells: 2 Claude Ant trials at the common 5,400-second task cap
• task digest, agent wall-clock, verifier, and build limits remain identical per task across models

Leaderboard impact

Model	Previous IQM	Fair IQM	Change
Claude Opus 4.8	62.03	62.03	0.00
GPT-5.6 Sol	36.65	40.68	+4.03
GPT-5.5	10.66	13.87	+3.21
GPT-5.4 Mini	-17.41	-17.41	0.00

Ranking order is unchanged.

Validation

• uv run python scripts/audit_article_suite_timeout_fairness.py (status: ok)
• uv run python scripts/validate_tasks.py
• uv run bench tasks check tasks/simulation_heuristics_ant_v1 --level publication-grade
• uv run python scripts/audit_simulation_heuristics_ant_v1_leaderboard.py
• uv run ruff check .
• uv run pytest -q (108 passed, 18 skipped)
• uv build
• CI-equivalent Docker oracle smoke (1/1, 100%)
• localhost HTML verification: 9 task cards, fair final scores, timeout-audit link

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
genesis-bench	Ready	Preview, Comment	Jul 15, 2026 7:18pm

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2189ce0095

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Check transport errors for PTY timeouts

When a selected result records the Daytona watchdog in transport_error_info (which this change now copies into metadata), this audit can still publish status: "ok" because the PTY check only searches the terminal_error string. In that scenario a cell with transport_error_info containing a PTY readline timeout would remain out of selected_timeout_influenced_cells, so the fairness audit could falsely pass; include transport_error_info/the error category in this check as well.

Useful? React with 👍 / 👎.