Standardize article-suite timeout fairness

README.md

@@ -147,6 +147,8 @@ All published article-suite rows run in Daytona sandboxes.
 
 See [`docs/article-suite-scoring.md`](docs/article-suite-scoring.md) for the
 formula, research precedent, and statistical limitations.
+The selected-trajectory timeout audit and repaired cells are documented in
+[`docs/article-suite-timeout-fairness.md`](docs/article-suite-timeout-fairness.md).
 
 ## Contribute a Task
 

docs/article-suite-scoring.md

@@ -2,6 +2,11 @@
 
 GenesisBench separates per-task normalization from cross-task aggregation.
 
+All compared models use identical task wall-clock, verifier, build,
+60-minute agent-idle, and 60-minute Daytona PTY-read limits. Infrastructure
+timeouts are repaired at the affected model-task-trial cell rather than scored
+as model performance.
+
 ## Per-task normalized score
 
 Every task evaluates a candidate on its hidden suite and compares the resulting

docs/article-suite-timeout-fairness.md

@@ -0,0 +1,52 @@
+# Article-suite timeout fairness audit
+
+The published five-trial leaderboard was audited cell by cell against the
+selected source trajectories for all `4 × 9 × 5 = 180` model-task-trial cells.
+
+## Fairness policy
+
+- Task wall-clock, verifier, build, hidden-suite, and task-digest settings are
+  identical between models within each task.
+- Protocol v2.2 sets both the agent-idle and Daytona PTY-read safeguards to
+  `3,600` seconds for every model.
+- Infrastructure timeout attempts are not scored as model performance. Only
+  affected model-task-trial cells are rerun.
+
+## Audit and repairs
+
+The historical audit found 10 cells whose result or retry path was affected by
+the earlier model-specific safeguards:
+
+- 8 cells had a `900`-second GPT Daytona PTY timeout in their retry history.
+- 2 selected GPT cells ended at the earlier `600`-second idle watchdog.
+
+All 10 cells were rerun with protocol v2.2. The final audit reports:
+
+```text
+selected cells: 180
+selected idle/PTY-influenced cells: 0
+historically affected cells rerun: 10
+```
+
+Two Claude Ant cells reached the shared task wall-clock limit of `5,400`
+seconds. They remain valid because the same primary limit applied to every
+model; they were not caused by the earlier idle/PTY difference.
+
+## Leaderboard impact
+
+| Model | Previous IQM | Fair IQM | Change |
+| --- | ---: | ---: | ---: |
+| GPT-5.5 | `10.66` | `13.87` | `+3.21` |
+| GPT-5.6 Sol | `36.65` | `40.68` | `+4.03` |
+| Claude Opus 4.8 | `62.03` | `62.03` | `0.00` |
+| GPT-5.4 Mini | `-17.41` | `-17.41` | `0.00` |
+
+The ranking order did not change.
+
+The machine-readable audit is
+[`leaderboard/article_suite_timeout_fairness_audit.json`](../leaderboard/article_suite_timeout_fairness_audit.json).
+Reproduce it with:
+
+```bash
+uv run python scripts/audit_article_suite_timeout_fairness.py
+```

experiments/article_suite/README.md

@@ -27,6 +27,8 @@ The current leaderboard protocol is defined in `protocol.toml`:
   normalized trial-task scores;
 - variability shown as the sample standard deviation of the five per-trial
   nine-task IQMs.
+- identical 60-minute agent-idle and Daytona PTY-read safeguards for every
+  model, in addition to the task-specific wall-clock limits.
 
 The resulting matrix is:
 
@@ -126,6 +128,11 @@ The leaderboard builder selects the latest complete per-model batch matching
 `protocol.toml`; it never mixes older single-run results or partial batches into
 the five-trial leaderboard.
 
+The selected 180 cells and timeout safeguards are audited by
+`scripts/audit_article_suite_timeout_fairness.py`. See
+`docs/article-suite-timeout-fairness.md` for the 10 repaired cells and the
+leaderboard impact.
+
 GPT-5.6 Sol is routed directly through OpenCode's Azure Responses-API provider
 with reasoning effort `max`. BenchFlow still owns sandboxing, task staging,
 ACP trajectory capture, and verifier execution. Trusted LiteLLM usage tracking

experiments/article_suite/models.toml

@@ -6,13 +6,17 @@ display_name = "GPT-5.6 Sol"
 model = "azure/gpt-5.6-sol"
 provider = "azure"
 provider_reasoning_effort = "max"
+agent_idle_timeout_sec = 3600
+daytona_pty_readline_timeout_sec = 3600
 
 [[models]]
 id = "gpt-5.5"
 display_name = "GPT-5.5"
 model = "azure/gpt-5.5"
 provider = "azure"
 provider_reasoning_effort = "xhigh"
+agent_idle_timeout_sec = 3600
+daytona_pty_readline_timeout_sec = 3600
 
 [[models]]
 id = "claude-opus-4.8"
@@ -29,3 +33,5 @@ display_name = "GPT-5.4 Mini"
 model = "azure/gpt-5.4-mini"
 provider = "azure"
 provider_reasoning_effort = "xhigh"
+agent_idle_timeout_sec = 3600
+daytona_pty_readline_timeout_sec = 3600

experiments/article_suite/protocol.toml

@@ -1,7 +1,11 @@
-version = "2.1"
+version = "2.2"
 trials = 5
 agent_timeout_multiplier = 3
 
+[fairness]
+agent_idle_timeout_sec = 3600
+daytona_pty_readline_timeout_sec = 3600
+
 [baseline_agent_timeout_sec]
 simulation_heuristics_ant_v1 = 1800
 simulation_heuristics_pong_ram_v1 = 1800

leaderboard/ARTICLE_SUITE.md

@@ -15,3 +15,5 @@ The final score is RLiable-style IQM over the complete 5 × 9 trial-task score m
 ![Final GenesisBench article-suite leaderboard](article_suite_final_leaderboard.png)
 
 Machine-readable rankings and score-detail paths are available in [`article_suite.json`](article_suite.json). The scoring rationale is documented in [`docs/article-suite-scoring.md`](../docs/article-suite-scoring.md).
+
+The selected trajectory timeout audit is available in [`article_suite_timeout_fairness_audit.json`](article_suite_timeout_fairness_audit.json) and [`docs/article-suite-timeout-fairness.md`](../docs/article-suite-timeout-fairness.md).