Publish five-trial Daytona article-suite leaderboard

README.md

@@ -47,7 +47,8 @@ Requirements:
 
 - Python 3.12+
 - [`uv`](https://docs.astral.sh/uv/)
-- Docker for isolated agent experiments
+- Daytona credentials for authoritative leaderboard experiments
+- Docker only for optional local task development
 
 Install and validate:
 
@@ -98,13 +99,15 @@ Run one model across all nine tasks:
 
 ```bash
 uv run python scripts/run_article_suite.py \
+  --env-file /path/to/credentials.env \
   --model gpt-5.6-sol
 ```
 
 Run all four canonical models and rebuild the 10 leaderboard artifacts:
 
 ```bash
 uv run python scripts/run_article_suite.py \
+  --env-file /path/to/credentials.env \
   --all-models
 uv run python scripts/build_article_suite_leaderboard.py
 ```
@@ -115,7 +118,7 @@ mapping is documented in `docs/learning-beyond-gradients-suite.md`.
 
 ## Article-Suite Leaderboard
 
-The first OpenCode sweep across all nine article-derived tasks:
+The five-trial OpenCode sweep across all nine article-derived tasks:
 
 ![GenesisBench final normalized leaderboard](leaderboard/article_suite_final_leaderboard.png)
 
@@ -128,19 +131,22 @@ Scores are unbounded normalized values: `0` matches the public starter and
 `100` matches the trusted article-level reference. Negative scores are genuine
 regressions; scores above `100` exceed the reference.
 
-The final ranking uses the interquartile mean (IQM) of the nine task scores:
-sort them, remove the lowest two and highest two, then average the middle five.
-The chart uses a plot-only positive index equal to `IQM + 100`; the raw IQM
-remains the ranking metric and is retained in the JSON.
+Each model runs five independent trials. The final ranking uses RLiable-style
+interquartile mean (IQM) over all 45 normalized trial-task scores: remove the
+lowest 11 and highest 11, then average the middle 23. The displayed `±` value
+is the sample standard deviation of the five per-trial nine-task IQMs. The chart
+uses a plot-only positive index equal to `IQM + 100`; raw IQM remains in the
+JSON.
 
 Inference settings are provider-specific: GPT-5.6 Sol and Claude Opus 4.8 use
 `max`; GPT-5.5 and GPT-5.4 Mini use `xhigh`. These labels come from different
 provider interfaces and are categorical settings, not a shared numeric compute
 scale. Exact routes are documented in
 [`docs/learning-beyond-gradients-suite.md`](docs/learning-beyond-gradients-suite.md).
+All published article-suite rows run in Daytona sandboxes.
 
 See [`docs/article-suite-scoring.md`](docs/article-suite-scoring.md) for the
-formula, research precedent, and current single-run limitation.
+formula, research precedent, and statistical limitations.
 
 ## Contribute a Task
 

docs/article-suite-scoring.md

@@ -23,25 +23,46 @@ This follows the same family of baseline normalization used by benchmarks such
 as [D4RL](https://arxiv.org/abs/2004.07219), which normalizes returns between
 task-specific lower and upper reference scores.
 
+## Five-trial estimator
+
+Each model runs the complete nine-task suite five independent times. For task
+`t`, the published task score is the arithmetic mean of its five normalized
+trial scores. The task leaderboard also publishes their sample standard
+deviation.
+
+Task plots use native raw environment scores. If a task's verifier intentionally
+fails closed with `normalized_score = 0` and no observed raw score, the plot uses
+the frozen starter anchor as the raw-score equivalent. The trial artifact keeps
+`observed_raw_score = null` and records
+`raw_score_source = starter_anchor_equivalent_for_fail_closed_timeout`; this
+case is never presented as an observed environment return.
+
 ## Final normalized score
 
-The primary final metric is the interquartile mean (IQM), implemented as the
-25% trimmed mean used by
-[RLiable](https://github.com/google-research/rliable):
+The primary cross-task metric is the interquartile mean (IQM), implemented as
+the 25% trimmed mean used by
+[RLiable's `aggregate_iqm`](https://github.com/google-research/rliable/blob/master/rliable/metrics.py#L61-L70).
+RLiable computes IQM across the complete `num_runs × num_tasks` score matrix:
 
 ```text
-scores = sort(the nine normalized task scores)
-trim_count = floor(0.25 * 9) = 2
-final_normalized_score = mean(scores[2:7])
+scores = sort(flatten(the 5 × 9 normalized score matrix))
+trim_count = floor(0.25 * 45) = 11
+final_normalized_score = mean(scores[11:34])
 ```
 
-In words: remove the lowest two and highest two task scores and average the
-middle five.
+This retains the middle `23` of the `45` trial-task scores. It matches
+RLiable's `aggregate_iqm`, which applies a 25% trimmed mean with `axis=None`.
+
+For a readable run-to-run variability diagnostic, the leaderboard separately
+computes one nine-task IQM per trial and publishes the sample standard
+deviation of those five trial-level IQMs. That displayed `±` value is not a
+bootstrap confidence interval around the pooled IQM.
 
 The JSON also publishes:
 
 - `arithmetic_mean_normalized_score`;
 - `median_normalized_score`;
+- `mean_trial_iqm_normalized_score`;
 - the original `average_normalized_score` as a backward-compatible alias for
   the arithmetic mean.
 
@@ -88,13 +109,10 @@ endpoints and would rewrite every displayed score whenever the comparison set
 changes. Clipping negative IQM values to zero would erase meaningful
 differences.
 
-## Current statistical limitation
-
-The published sweep currently has one independent agent run per model/task
-pair. That is sufficient to recompute deterministic aggregate metrics, but not
-to estimate statistically meaningful bootstrap confidence intervals or
-probability of improvement between models.
+## Statistical limitation
 
-A future multi-run release should retain IQM and additionally report
-stratified-bootstrap confidence intervals, performance profiles, and pairwise
-probability of improvement following the RLiable protocol.
+Five trials expose run-to-run variability and are materially stronger than a
+single run, but they remain a small sample. The leaderboard reports sample
+standard deviation rather than claiming narrow confidence intervals. A larger
+future release can add stratified-bootstrap confidence intervals, performance
+profiles, and pairwise probability of improvement following RLiable.

docs/learning-beyond-gradients-suite.md

@@ -49,22 +49,25 @@ zero and are not presented as article reproductions.
 All new leaderboard runs use the BenchFlow `opencode` ACP harness. The canonical
 matrix is:
 
-| Model | Exact route | Harness | Provider reasoning setting |
-| --- | --- | --- | --- |
-| GPT-5.6 Sol | Azure `azure/gpt-5.6-sol` | OpenCode | `max` |
-| GPT-5.5 | Azure `azure/gpt-5.5` | OpenCode | `xhigh` |
-| Claude Opus 4.8 | Claude OAuth `anthropic/claude-opus-4-8` through the pinned OpenCode plugin | OpenCode | `max` |
-| GPT-5.4 Mini | Azure `azure/gpt-5.4-mini` | OpenCode | `xhigh` |
+| Model | Exact route | Harness | Provider reasoning setting | Sandbox |
+| --- | --- | --- | --- | --- |
+| GPT-5.6 Sol | Azure `azure/gpt-5.6-sol` | OpenCode | `max` | Daytona |
+| GPT-5.5 | Azure `azure/gpt-5.5` | OpenCode | `xhigh` | Daytona |
+| Claude Opus 4.8 | Claude OAuth `anthropic/claude-opus-4-8` through the pinned OpenCode plugin | OpenCode | `max` | Daytona |
+| GPT-5.4 Mini | Azure `azure/gpt-5.4-mini` | OpenCode | `xhigh` | Daytona |
 
 `max` and `xhigh` are provider-specific categorical labels. They indicate the
 configured reasoning setting for that route; they are not interchangeable
 units and should not be read as a shared numeric inference-compute scale.
 
 OpenCode talks directly to the provider because BenchFlow 0.6.5's
 chat-completions gateway cannot faithfully transform Azure GPT-5.6 Sol tool
-calls. BenchFlow continues to own task staging, Daytona/Docker isolation, ACP
+calls. BenchFlow continues to own task staging, Daytona isolation, ACP
 trajectory capture, timing, and verifier execution.
 
+The published matrix uses Daytona for every task. Docker is retained only as an
+optional local development backend.
+
 ## Leaderboard outputs
 
 The offline article-suite report contains 10 independent leaderboards in a
@@ -79,10 +82,12 @@ Each task first receives an unbounded anchor-normalized score:
 task score = 100 * (candidate - starter) / (reference - starter)
 ```
 
-The primary final score is the 25% trimmed interquartile mean (IQM). With nine
-tasks, the two lowest and two highest normalized scores are removed and the
-middle five are averaged. Arithmetic mean and median remain secondary
-diagnostics.
+Each model runs five independent trials. Task leaderboards report mean ± sample
+standard deviation across those trials. The primary cross-task score is the
+RLiable-style 25% trimmed interquartile mean (IQM) over the complete `5 × 9`
+score matrix: flatten all 45 normalized scores, remove the lowest 11 and
+highest 11, and average the middle 23. The displayed `±` value is the sample
+standard deviation of the five per-trial nine-task IQMs.
 
 The repository README intentionally shows only the final leaderboard image.
 The nine task panels use each environment's native raw score. The final chart

docs/simulation-heuristics-ant-v1-task-design.md

@@ -176,6 +176,9 @@ The first published sweep uses:
 - isolated Docker workspaces;
 - each model's highest supported reasoning level.
 
+The current article-suite protocol triples the Ant agent timeout to 90 minutes
+and reports mean ± sample standard deviation across five independent trials.
+
 Simulation Heuristics Ant v1 currently enforces wall-clock time but does not
 centrally meter every simulator step. Future hosted versions should add an
 authoritative interaction meter before making sample-efficiency claims across

experiments/article_suite/README.md

@@ -16,14 +16,42 @@ the nine task packages derived from the article:
 Every run uses BenchFlow's registered `opencode` ACP harness. OpenHands is not
 part of this suite.
 
+## Experiment protocol
+
+The current leaderboard protocol is defined in `protocol.toml`:
+
+- five independent full-suite trials per model;
+- three times the original agent wall-clock timeout for every task;
+- task scores reported as mean ± sample standard deviation across trials;
+- final score computed as RLiable-style IQM over all `5 × 9 = 45`
+  normalized trial-task scores;
+- variability shown as the sample standard deviation of the five per-trial
+  nine-task IQMs.
+
+The resulting matrix is:
+
+```text
+4 models × 5 trials × 9 tasks = 180 model-task runs
+```
+
+| Tasks | Previous cap | Current cap |
+| --- | ---: | ---: |
+| Ant, Pong, both Breakouts, Montezuma, VizDoom D1/D3 | 30 min | 90 min |
+| HalfCheetah | 60 min | 180 min |
+| Atari57 aggregate search | 7 days | 21 days |
+
+Atari57's cap is exceptional because one task contains 342 accounted search
+slots. It can be scheduled separately when operating capacity is limited; the
+cap is a maximum, not an expected runtime.
+
 ## Inference settings
 
-| Model | Exact route | Provider-specific reasoning setting |
-| --- | --- | --- |
-| GPT-5.6 Sol | `azure/gpt-5.6-sol` | `max` |
-| GPT-5.5 | `azure/gpt-5.5` | `xhigh` |
-| Claude Opus 4.8 | `anthropic/claude-opus-4-8` via Claude OAuth and the pinned OpenCode plugin | `max` |
-| GPT-5.4 Mini | `azure/gpt-5.4-mini` | `xhigh` |
+| Model | Exact route | Provider-specific reasoning setting | Sandbox |
+| --- | --- | --- | --- |
+| GPT-5.6 Sol | `azure/gpt-5.6-sol` | `max` | Daytona |
+| GPT-5.5 | `azure/gpt-5.5` | `xhigh` | Daytona |
+| Claude Opus 4.8 | `anthropic/claude-opus-4-8` via Claude OAuth and the pinned OpenCode plugin | `max` | Daytona |
+| GPT-5.4 Mini | `azure/gpt-5.4-mini` | `xhigh` | Daytona |
 
 The setting names are categorical labels exposed by each provider integration,
 not a common numeric measure of inference compute. All four routes use the
@@ -50,39 +78,53 @@ CLAUDE_CODE_OAUTH_TOKEN
 
 ## Run
 
-Docker must be running because the authoritative suite uses isolated BenchFlow
-task environments:
+The authoritative suite uses isolated Daytona sandboxes in the `us` target:
 
 ```bash
 uv run python scripts/run_article_suite.py \
   --env-file /path/to/credentials.env \
   --model gpt-5.6-sol
 ```
 
-Atari57 requests more CPU, memory, and storage than the current Daytona account
-allows. The all-nine commands therefore use the default local Docker sandbox.
-Use Daytona only for selected non-Atari tasks.
+The July 14, 2026 Atari57 capacity canary confirmed that Daytona runs all nine
+tasks. BenchFlow currently clamps Atari57 from 14 requested CPUs to 4 and from
+102,400 MB requested storage to 10,240 MB; these are execution limits, not score
+transformations, and the remote verifier still completes normally.
+
+Local Docker remains available for development with `--sandbox docker`. Such
+runs default to calibrated `linux/amd64` images; they are not selected for the
+published article-suite leaderboard.
 
 Run all four models sequentially:
 
 ```bash
 uv run python scripts/run_article_suite.py \
   --env-file /path/to/credentials.env \
-  --all-models
+  --all-models \
+  --trials 5 \
+  --batch-id article-suite-v2-daytona
 ```
 
-Long runs are resumable at task granularity. For example:
+Runs are resumable at model-trial granularity. Reusing `--batch-id` skips
+completed trials. Models may also be scheduled into separate batch directories
+for parallel execution. The leaderboard builder chooses one complete
+five-trial batch per model and never mixes trials from different batches within
+the same model.
+
+Use `--trial` to schedule or repair selected trials:
 
 ```bash
 uv run python scripts/run_article_suite.py \
   --env-file /path/to/credentials.env \
   --model gpt-5.6-sol \
   --task simulation_heuristics_halfcheetah_v1 \
-  --sandbox daytona
+  --trial 3 \
+  --batch-id article-suite-v2-daytona
 ```
 
-The leaderboard builder selects the latest successful result for every
-model/task pair across all run batches.
+The leaderboard builder selects the latest complete per-model batch matching
+`protocol.toml`; it never mixes older single-run results or partial batches into
+the five-trial leaderboard.
 
 GPT-5.6 Sol is routed directly through OpenCode's Azure Responses-API provider
 with reasoning effort `max`. BenchFlow still owns sandboxing, task staging,
@@ -109,8 +151,14 @@ The builder writes:
   ranking displayed as `IQM + 100` so every current plotted value is positive.
 
 Each task maps its starter policy to `0` and its trusted article-level reference
-to `100`. The final primary aggregate is the interquartile mean: remove the two
-lowest and two highest of the nine normalized scores, then average the middle
-five. Arithmetic mean and median remain secondary diagnostics. The top-level
-repository README shows only the final image. The additive display offset does
-not affect ranking or score gaps; raw IQM remains available in the JSON.
+to `100`. The final leaderboard flattens the five-by-nine normalized score
+matrix, removes the lowest 11 and highest 11 values, and averages the middle
+23. Per-task sample standard deviations and the sample standard deviation of
+the five trial-level IQMs remain visible diagnostics. The additive display
+offset does not affect ranking or score gaps; raw IQM remains available in the
+JSON.
+
+For task-defined fail-closed verifier timeouts where no native raw score exists,
+the raw-score plot uses the frozen starter anchor as the equivalent of
+normalized `0`. Trial metadata preserves the missing observation and labels the
+display source explicitly.

experiments/article_suite/models.toml

@@ -20,7 +20,7 @@ display_name = "Claude Opus 4.8"
 model = "anthropic/claude-opus-4-8"
 provider = "claude_oauth"
 provider_reasoning_effort = "max"
-agent_idle_timeout_sec = 0
+agent_idle_timeout_sec = 3600
 daytona_pty_readline_timeout_sec = 3600
 
 [[models]]

experiments/article_suite/protocol.toml

@@ -0,0 +1,21 @@
+version = "2.1"
+trials = 5
+agent_timeout_multiplier = 3
+
+[baseline_agent_timeout_sec]
+simulation_heuristics_ant_v1 = 1800
+simulation_heuristics_pong_ram_v1 = 1800
+simulation_heuristics_breakout_ram_v1 = 1800
+simulation_heuristics_breakout_rgb_v1 = 1800
+simulation_heuristics_halfcheetah_v1 = 3600
+simulation_heuristics_vizdoom_d1_v1 = 1800
+simulation_heuristics_vizdoom_d3_v1 = 1800
+simulation_heuristics_atari57_v1 = 604800
+simulation_heuristics_montezuma_v1 = 1800
+
+[aggregation]
+task_score = "arithmetic_mean_across_trials"
+task_variability = "sample_standard_deviation"
+trial_score = "interquartile_mean_across_nine_tasks"
+final_score = "interquartile_mean_across_all_trial_task_scores"
+final_variability = "sample_standard_deviation_of_trial_level_iqms"

leaderboard/ARTICLE_SUITE.md

@@ -2,15 +2,15 @@
 
 The first image contains the nine independently ranked task leaderboards. The second image contains the final cross-task ranking.
 
-The nine task panels use each environment's native raw score. The final scientific metric remains unbounded IQM.
+The nine task panels report five-trial mean ± sample standard deviation for each environment's native raw score.
 
-## Nine task leaderboards
+## Leaderboards
 
 ![Nine task-specific GenesisBench leaderboards](article_suite_task_leaderboards.png)
 
 ## Final normalized score
 
-The primary score is the interquartile mean (IQM): sort the nine task scores, remove the lowest two and highest two, then average the middle five. The image uses a plot-only positive display index equal to `IQM + 100`; raw IQM, arithmetic mean, and median remain in the JSON.
+The final score is RLiable-style IQM over the complete 5 × 9 trial-task score matrix: flatten all 45 normalized scores, trim the lowest 11 and highest 11, and average the middle 23. The displayed ± value is the sample standard deviation of the five per-trial IQMs. The image uses a plot-only positive display index equal to `IQM + 100`; raw metrics remain in the JSON.
 
 ![Final GenesisBench article-suite leaderboard](article_suite_final_leaderboard.png)