diff --git a/README.md b/README.md index a033528..824a1a3 100644 --- a/README.md +++ b/README.md @@ -117,15 +117,10 @@ mapping is documented in `docs/learning-beyond-gradients-suite.md`. The first OpenCode sweep across all nine article-derived tasks: -| Rank | Model | Nine-task average | -| ---: | --- | ---: | -| 1 | GPT-5.5 | 43.19 | -| 2 | Claude Opus 4.8 | 39.82 | -| 3 | GPT-5.6 Sol | 39.38 | -| 4 | GPT-5.4 Mini | -29.72 | +![GenesisBench final normalized leaderboard](leaderboard/article_suite_final_leaderboard.png) See [`leaderboard/ARTICLE_SUITE.md`](leaderboard/ARTICLE_SUITE.md) for the nine -task-specific leaderboards followed by the final average leaderboard. +task-specific leaderboards followed by the final cross-task leaderboard. `leaderboard/article_suite.json` contains the same 10-board structure in machine-readable form. @@ -133,6 +128,20 @@ Scores are unbounded normalized values: `0` matches the public starter and `100` matches the trusted article-level reference. Negative scores are genuine regressions; scores above `100` exceed the reference. +The final ranking uses the interquartile mean (IQM) of the nine task scores: +sort them, remove the lowest two and highest two, then average the middle five. +The chart uses a plot-only positive index equal to `IQM + 100`; the raw IQM +remains the ranking metric and is retained in the JSON. + +Inference settings are provider-specific: GPT-5.6 Sol and Claude Opus 4.8 use +`max`; GPT-5.5 and GPT-5.4 Mini use `xhigh`. These labels come from different +provider interfaces and are categorical settings, not a shared numeric compute +scale. Exact routes are documented in +[`docs/learning-beyond-gradients-suite.md`](docs/learning-beyond-gradients-suite.md). + +See [`docs/article-suite-scoring.md`](docs/article-suite-scoring.md) for the +formula, research precedent, and current single-run limitation. + ## Contribute a Task Create a scaffold: diff --git a/docs/article-suite-scoring.md b/docs/article-suite-scoring.md new file mode 100644 index 0000000..9ad74f4 --- /dev/null +++ b/docs/article-suite-scoring.md @@ -0,0 +1,100 @@ +# Article-suite scoring methodology + +GenesisBench separates per-task normalization from cross-task aggregation. + +## Per-task normalized score + +Every task evaluates a candidate on its hidden suite and compares the resulting +raw score with two frozen anchors: + +```text +normalized_task_score = + 100 * (candidate_score - starter_score) + / (reference_score - starter_score) +``` + +The public starter maps to `0`; the trusted article-level reference maps to +`100`. Scores are intentionally unbounded: + +- negative scores perform below the starter; +- scores above `100` outperform the reference. + +This follows the same family of baseline normalization used by benchmarks such +as [D4RL](https://arxiv.org/abs/2004.07219), which normalizes returns between +task-specific lower and upper reference scores. + +## Final normalized score + +The primary final metric is the interquartile mean (IQM), implemented as the +25% trimmed mean used by +[RLiable](https://github.com/google-research/rliable): + +```text +scores = sort(the nine normalized task scores) +trim_count = floor(0.25 * 9) = 2 +final_normalized_score = mean(scores[2:7]) +``` + +In words: remove the lowest two and highest two task scores and average the +middle five. + +The JSON also publishes: + +- `arithmetic_mean_normalized_score`; +- `median_normalized_score`; +- the original `average_normalized_score` as a backward-compatible alias for + the arithmetic mean. + +## Why IQM is primary + +The original GenesisBench aggregate was the unweighted arithmetic mean of nine +normalized task scores. That is simple and remains useful, but a single very +large positive or negative task can move the final rank substantially. + +The NeurIPS paper +[*Deep Reinforcement Learning at the Edge of the Statistical +Precipice*](https://proceedings.neurips.cc/paper_files/paper/2021/hash/f514cec81cb148559cf475e7426eed5e-Abstract.html) +recommends robust aggregate metrics such as IQM, together with performance +profiles and uncertainty estimates. Atari research also commonly reports both +median and mean human-normalized scores; the +[Agent57 paper](https://arxiv.org/abs/2003.13350) explicitly notes that a high +average can hide weak performance on many individual games. + +GenesisBench therefore uses: + +1. task-specific native raw-score leaderboards for visibility; +2. IQM as the primary cross-task rank; +3. arithmetic mean and median as secondary diagnostics. + +## Positive display index + +The final image uses a fixed additive transform: + +```text +positive_display_score = final_normalized_score + 100 +``` + +This is presentation-only. Raw IQM remains the official ranking field. + +A fixed offset is preferred over cohort min-max scaling because it: + +- preserves every model-to-model difference exactly; +- does not change when another model is added; +- gives a stable interpretation: an aggregate starter-level IQM of `0` + displays as `100`. + +Min-max scaling would force the current best and worst models to arbitrary +endpoints and would rewrite every displayed score whenever the comparison set +changes. Clipping negative IQM values to zero would erase meaningful +differences. + +## Current statistical limitation + +The published sweep currently has one independent agent run per model/task +pair. That is sufficient to recompute deterministic aggregate metrics, but not +to estimate statistically meaningful bootstrap confidence intervals or +probability of improvement between models. + +A future multi-run release should retain IQM and additionally report +stratified-bootstrap confidence intervals, performance profiles, and pairwise +probability of improvement following the RLiable protocol. diff --git a/docs/learning-beyond-gradients-suite.md b/docs/learning-beyond-gradients-suite.md index 4261da0..17e9e30 100644 --- a/docs/learning-beyond-gradients-suite.md +++ b/docs/learning-beyond-gradients-suite.md @@ -49,12 +49,16 @@ zero and are not presented as article reproductions. All new leaderboard runs use the BenchFlow `opencode` ACP harness. The canonical matrix is: -| Model | Provider | Effort | -| --- | --- | --- | -| GPT-5.6 Sol | Azure | `max` | -| GPT-5.5 | Azure | `xhigh` | -| Claude Opus 4.8 | Claude OAuth through pinned OpenCode plugin | `max` | -| GPT-5.4 Mini | Azure | `xhigh` | +| Model | Exact route | Harness | Provider reasoning setting | +| --- | --- | --- | --- | +| GPT-5.6 Sol | Azure `azure/gpt-5.6-sol` | OpenCode | `max` | +| GPT-5.5 | Azure `azure/gpt-5.5` | OpenCode | `xhigh` | +| Claude Opus 4.8 | Claude OAuth `anthropic/claude-opus-4-8` through the pinned OpenCode plugin | OpenCode | `max` | +| GPT-5.4 Mini | Azure `azure/gpt-5.4-mini` | OpenCode | `xhigh` | + +`max` and `xhigh` are provider-specific categorical labels. They indicate the +configured reasoning setting for that route; they are not interchangeable +units and should not be read as a shared numeric inference-compute scale. OpenCode talks directly to the provider because BenchFlow 0.6.5's chat-completions gateway cannot faithfully transform Azure GPT-5.6 Sol tool @@ -67,18 +71,26 @@ The offline article-suite report contains 10 independent leaderboards in a fixed order: 1. one leaderboard for each of the nine article-derived tasks; -2. the final averaged leaderboard. +2. the final cross-task leaderboard. -The final score is the unweighted arithmetic mean: +Each task first receives an unbounded anchor-normalized score: ```text -average = sum(nine normalized task scores) / 9 +task score = 100 * (candidate - starter) / (reference - starter) ``` -The repository README intentionally shows only the final averaged leaderboard. -Detailed task rankings and score-artifact links live in -`leaderboard/ARTICLE_SUITE.md`; the matching machine-readable structure lives -in `leaderboard/article_suite.json`. +The primary final score is the 25% trimmed interquartile mean (IQM). With nine +tasks, the two lowest and two highest normalized scores are removed and the +middle five are averaged. Arithmetic mean and median remain secondary +diagnostics. + +The repository README intentionally shows only the final leaderboard image. +The nine task panels use each environment's native raw score. The final chart +uses a positive plot index equal to `IQM + 100`, while raw IQM remains the +official ranking metric. Both images live in `leaderboard/ARTICLE_SUITE.md`; +the matching machine-readable structure lives in +`leaderboard/article_suite.json`. See `docs/article-suite-scoring.md` for the +research rationale and statistical limitations. The runner and resumable leaderboard builder live in: diff --git a/experiments/article_suite/README.md b/experiments/article_suite/README.md index 0867c37..cdc3125 100644 --- a/experiments/article_suite/README.md +++ b/experiments/article_suite/README.md @@ -16,6 +16,20 @@ the nine task packages derived from the article: Every run uses BenchFlow's registered `opencode` ACP harness. OpenHands is not part of this suite. +## Inference settings + +| Model | Exact route | Provider-specific reasoning setting | +| --- | --- | --- | +| GPT-5.6 Sol | `azure/gpt-5.6-sol` | `max` | +| GPT-5.5 | `azure/gpt-5.5` | `xhigh` | +| Claude Opus 4.8 | `anthropic/claude-opus-4-8` via Claude OAuth and the pinned OpenCode plugin | `max` | +| GPT-5.4 Mini | `azure/gpt-5.4-mini` | `xhigh` | + +The setting names are categorical labels exposed by each provider integration, +not a common numeric measure of inference compute. All four routes use the +OpenCode harness; the route and reasoning setting are stored with every +published result. + ## Credentials Credential values are read from the process environment or an env file. They @@ -85,11 +99,18 @@ uv run python scripts/build_article_suite_leaderboard.py The builder writes: -- `leaderboard/ARTICLE_SUITE.md`: nine task-specific leaderboards followed by - the final average leaderboard; +- `leaderboard/ARTICLE_SUITE.md`: one nine-panel task image followed by the + final IQM leaderboard image; - `leaderboard/article_suite.json`: the same 10 leaderboards plus the model-centric score records used for reproducibility. +- `leaderboard/article_suite_task_leaderboards.png`: nine task-specific + leaderboard panels using native raw environment scores; +- `leaderboard/article_suite_final_leaderboard.png`: the final cross-task + ranking displayed as `IQM + 100` so every current plotted value is positive. -The final aggregate is the arithmetic mean of the nine normalized task scores. Each task maps its starter policy to `0` and its trusted article-level reference -to `100`. The top-level repository README shows only this final aggregate. +to `100`. The final primary aggregate is the interquartile mean: remove the two +lowest and two highest of the nine normalized scores, then average the middle +five. Arithmetic mean and median remain secondary diagnostics. The top-level +repository README shows only the final image. The additive display offset does +not affect ranking or score gaps; raw IQM remains available in the JSON. diff --git a/leaderboard/ARTICLE_SUITE.md b/leaderboard/ARTICLE_SUITE.md index acb3749..780343d 100644 --- a/leaderboard/ARTICLE_SUITE.md +++ b/leaderboard/ARTICLE_SUITE.md @@ -1,115 +1,17 @@ # GenesisBench Learning Beyond Gradients Article Suite -This offline report contains 10 independent leaderboards: one for each of the nine article-derived tasks, followed by the final nine-task average. +The first image contains the nine independently ranked task leaderboards. The second image contains the final cross-task ranking. -Task scores are unbounded normalized values. The public starter maps to 0 and the trusted article-level reference maps to 100. +The nine task panels use each environment's native raw score. The final scientific metric remains unbounded IQM. -## 1. Ant +## Nine task leaderboards -Task: `simulation_heuristics_ant_v1` +![Nine task-specific GenesisBench leaderboards](article_suite_task_leaderboards.png) -| Rank | Model | Harness | Effort | Normalized score | Score details | -| ---: | --- | --- | --- | ---: | --- | -| 1 | GPT-5.6 Sol | opencode | max | 39.20 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/score.json) | -| 2 | Claude Opus 4.8 | opencode | max | 14.05 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/score.json) | -| 3 | GPT-5.5 | opencode | xhigh | -16.89 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/score.json) | -| 4 | GPT-5.4 Mini | opencode | xhigh | -31.36 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/score.json) | +## Final normalized score -## 2. Pong +The primary score is the interquartile mean (IQM): sort the nine task scores, remove the lowest two and highest two, then average the middle five. The image uses a plot-only positive display index equal to `IQM + 100`; raw IQM, arithmetic mean, and median remain in the JSON. -Task: `simulation_heuristics_pong_ram_v1` +![Final GenesisBench article-suite leaderboard](article_suite_final_leaderboard.png) -| Rank | Model | Harness | Effort | Normalized score | Score details | -| ---: | --- | --- | --- | ---: | --- | -| 1 | Claude Opus 4.8 | opencode | max | 100.00 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/score.json) | -| 2 | GPT-5.6 Sol | opencode | max | 52.92 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/score.json) | -| 3 | GPT-5.5 | opencode | xhigh | 45.83 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/score.json) | -| 4 | GPT-5.4 Mini | opencode | xhigh | -75.00 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/score.json) | - -## 3. Breakout RAM - -Task: `simulation_heuristics_breakout_ram_v1` - -| Rank | Model | Harness | Effort | Normalized score | Score details | -| ---: | --- | --- | --- | ---: | --- | -| 1 | GPT-5.6 Sol | opencode | max | 100.00 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/score.json) | -| 2 | GPT-5.5 | opencode | xhigh | 95.60 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/score.json) | -| 3 | GPT-5.4 Mini | opencode | xhigh | 2.52 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/score.json) | -| 4 | Claude Opus 4.8 | opencode | max | 0.00 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/score.json) | - -## 4. Breakout RGB - -Task: `simulation_heuristics_breakout_rgb_v1` - -| Rank | Model | Harness | Effort | Normalized score | Score details | -| ---: | --- | --- | --- | ---: | --- | -| 1 | GPT-5.5 | opencode | xhigh | 70.76 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/score.json) | -| 1 | GPT-5.6 Sol | opencode | max | 70.76 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/score.json) | -| 3 | GPT-5.4 Mini | opencode | xhigh | 15.52 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/score.json) | -| 4 | Claude Opus 4.8 | opencode | max | 0.00 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/score.json) | - -## 5. HalfCheetah - -Task: `simulation_heuristics_halfcheetah_v1` - -| Rank | Model | Harness | Effort | Normalized score | Score details | -| ---: | --- | --- | --- | ---: | --- | -| 1 | Claude Opus 4.8 | opencode | max | 26.99 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/score.json) | -| 2 | GPT-5.6 Sol | opencode | max | 15.87 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/score.json) | -| 3 | GPT-5.5 | opencode | xhigh | 2.74 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/score.json) | -| 4 | GPT-5.4 Mini | opencode | xhigh | -8.17 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/score.json) | - -## 6. VizDoom D1 - -Task: `simulation_heuristics_vizdoom_d1_v1` - -| Rank | Model | Harness | Effort | Normalized score | Score details | -| ---: | --- | --- | --- | ---: | --- | -| 1 | Claude Opus 4.8 | opencode | max | 95.06 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/score.json) | -| 2 | GPT-5.5 | opencode | xhigh | 85.20 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/score.json) | -| 3 | GPT-5.6 Sol | opencode | max | 49.41 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/score.json) | -| 4 | GPT-5.4 Mini | opencode | xhigh | -161.12 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/score.json) | - -## 7. VizDoom D3 - -Task: `simulation_heuristics_vizdoom_d3_v1` - -| Rank | Model | Harness | Effort | Normalized score | Score details | -| ---: | --- | --- | --- | ---: | --- | -| 1 | Claude Opus 4.8 | opencode | max | 122.26 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/score.json) | -| 2 | GPT-5.6 Sol | opencode | max | 26.24 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/score.json) | -| 3 | GPT-5.5 | opencode | xhigh | 5.48 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/score.json) | -| 4 | GPT-5.4 Mini | opencode | xhigh | -9.86 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/score.json) | - -## 8. Atari57 - -Task: `simulation_heuristics_atari57_v1` - -| Rank | Model | Harness | Effort | Normalized score | Score details | -| ---: | --- | --- | --- | ---: | --- | -| 1 | Claude Opus 4.8 | opencode | max | 0.00 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/score.json) | -| 1 | GPT-5.4 Mini | opencode | xhigh | 0.00 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/score.json) | -| 1 | GPT-5.5 | opencode | xhigh | 0.00 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/score.json) | -| 1 | GPT-5.6 Sol | opencode | max | 0.00 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/score.json) | - -## 9. Montezuma's Revenge - -Task: `simulation_heuristics_montezuma_v1` - -| Rank | Model | Harness | Effort | Normalized score | Score details | -| ---: | --- | --- | --- | ---: | --- | -| 1 | GPT-5.5 | opencode | xhigh | 100.00 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/score.json) | -| 2 | Claude Opus 4.8 | opencode | max | 0.00 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/score.json) | -| 2 | GPT-5.4 Mini | opencode | xhigh | 0.00 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/score.json) | -| 2 | GPT-5.6 Sol | opencode | max | 0.00 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/score.json) | - -## 10. Nine-task average - -Arithmetic mean of the nine normalized task scores. - -| Rank | Model | Harness | Effort | Nine-task average | -| ---: | --- | --- | --- | ---: | -| 1 | GPT-5.5 | opencode | xhigh | 43.19 | -| 2 | Claude Opus 4.8 | opencode | max | 39.82 | -| 3 | GPT-5.6 Sol | opencode | max | 39.38 | -| 4 | GPT-5.4 Mini | opencode | xhigh | -29.72 | +Machine-readable rankings and score-detail paths are available in [`article_suite.json`](article_suite.json). The scoring rationale is documented in [`docs/article-suite-scoring.md`](../docs/article-suite-scoring.md). diff --git a/leaderboard/README.md b/leaderboard/README.md index b085e94..968603f 100644 --- a/leaderboard/README.md +++ b/leaderboard/README.md @@ -2,15 +2,10 @@ ## Learning Beyond Gradients article suite -| Rank | Model | Nine-task average | -| ---: | --- | ---: | -| 1 | GPT-5.5 | 43.19 | -| 2 | Claude Opus 4.8 | 39.82 | -| 3 | GPT-5.6 Sol | 39.38 | -| 4 | GPT-5.4 Mini | -29.72 | +![GenesisBench final normalized leaderboard](article_suite_final_leaderboard.png) See [`ARTICLE_SUITE.md`](ARTICLE_SUITE.md) for 10 independent leaderboards: -nine task-specific rankings followed by the final average ranking. +nine task-specific rankings followed by the final IQM ranking. ## Legacy Simulation Heuristics Ant v1 diff --git a/leaderboard/article_suite.json b/leaderboard/article_suite.json index 4916a6a..b490741 100644 --- a/leaderboard/article_suite.json +++ b/leaderboard/article_suite.json @@ -1,52 +1,142 @@ { - "aggregation": "arithmetic_mean_of_normalized_task_scores", + "aggregation": { + "display_transform": { + "formula": "positive_display_score = final_normalized_score + 100", + "offset": 100.0, + "purpose": "plot_only", + "ranking_field": "final_normalized_score", + "type": "additive_offset" + }, + "primary_field": "final_normalized_score", + "primary_metric": "interquartile_mean", + "retained_score_count": 5, + "score_bounds": "unbounded", + "secondary_fields": [ + "arithmetic_mean_normalized_score", + "median_normalized_score" + ], + "trim_fraction_per_tail": 0.25, + "trimmed_score_count_per_tail": 2, + "uncertainty": "not_estimated_single_run_per_model_task" + }, "benchmark": "learning_beyond_gradients_article_suite", "generated_at": "2026-07-13T15:34:22.260689+00:00", + "inference_settings": { + "cross_provider_comparability": "labels_are_not_a_shared_numeric_compute_scale", + "field": "provider_reasoning_effort", + "interpretation": "provider_specific_categorical_setting", + "models": [ + { + "harness": "opencode", + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh" + }, + { + "harness": "opencode", + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "max" + }, + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max" + }, + { + "harness": "opencode", + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh" + } + ] + }, "leaderboard_count": 10, "leaderboards": [ { "id": "simulation_heuristics_ant_v1", "label": "Ant", - "metric": "normalized_score", + "metric": "raw_score", + "raw_metric": { + "label": "Weighted hidden return", + "unit": "return" + }, "rows": [ { "harness": "opencode", "model": "GPT-5.6 Sol", "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", "normalized_score": 39.203600385, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "max", "rank": 1, + "raw_score": 3815.360622145591, + "reference_score": 5942.333244755838, "source_run": "leaderboard/runs/article_suite/20260713T093815Z/gpt-5.6-sol", + "starter_score": 2443.815838992282, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/score.json" }, { "harness": "opencode", "model": "Claude Opus 4.8", "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", "normalized_score": 14.050353128, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 2, + "raw_score": 2935.3698887625987, + "reference_score": 5942.333244755838, "source_run": "leaderboard/runs/article_suite/20260713T142919Z/claude-opus-4.8", + "starter_score": 2443.815838992282, "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/score.json" }, { "harness": "opencode", "model": "GPT-5.5", "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", "normalized_score": -16.891796576, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 3, + "raw_score": 1852.8533956222025, + "reference_score": 5942.333244755838, "source_run": "leaderboard/runs/article_suite/20260713T095807Z/gpt-5.5", + "starter_score": 2443.815838992282, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/score.json" }, { "harness": "opencode", "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", "normalized_score": -31.35954984, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 4, + "raw_score": 1346.6965294546999, + "reference_score": 5942.333244755838, "source_run": "leaderboard/runs/article_suite/20260713T104315Z/gpt-5.4-mini", + "starter_score": 2443.815838992282, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/score.json" } ] @@ -54,46 +144,74 @@ { "id": "simulation_heuristics_pong_ram_v1", "label": "Pong", - "metric": "normalized_score", + "metric": "raw_score", + "raw_metric": { + "label": "Native Pong score", + "unit": "points" + }, "rows": [ { "harness": "opencode", "model": "Claude Opus 4.8", "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", "normalized_score": 100.0, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 1, + "raw_score": 21.0, + "reference_score": 21.0, "source_run": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "starter_score": -2.9999999999999996, "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/score.json" }, { "harness": "opencode", "model": "GPT-5.6 Sol", "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", "normalized_score": 52.916666667, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "max", "rank": 2, + "raw_score": 9.7, + "reference_score": 21.0, "source_run": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "starter_score": -2.9999999999999996, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/score.json" }, { "harness": "opencode", "model": "GPT-5.5", "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", "normalized_score": 45.833333333, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 3, + "raw_score": 8.0, + "reference_score": 21.0, "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "starter_score": -2.9999999999999996, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/score.json" }, { "harness": "opencode", "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", "normalized_score": -75.0, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 4, + "raw_score": -21.0, + "reference_score": 21.0, "source_run": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "starter_score": -2.9999999999999996, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/score.json" } ] @@ -101,46 +219,74 @@ { "id": "simulation_heuristics_breakout_ram_v1", "label": "Breakout RAM", - "metric": "normalized_score", + "metric": "raw_score", + "raw_metric": { + "label": "Native Breakout return", + "unit": "points" + }, "rows": [ { "harness": "opencode", "model": "GPT-5.6 Sol", "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", "normalized_score": 100.0, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "max", "rank": 1, + "raw_score": 864.0, + "reference_score": 864.0, "source_run": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "starter_score": 387.0, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/score.json" }, { "harness": "opencode", "model": "GPT-5.5", "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", "normalized_score": 95.597484277, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 2, + "raw_score": 843.0, + "reference_score": 864.0, "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "starter_score": 387.0, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/score.json" }, { "harness": "opencode", "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", "normalized_score": 2.51572327, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 3, + "raw_score": 399.0, + "reference_score": 864.0, "source_run": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "starter_score": 387.0, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/score.json" }, { "harness": "opencode", "model": "Claude Opus 4.8", "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", "normalized_score": 0.0, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 4, + "raw_score": 387.0, + "reference_score": 864.0, "source_run": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "starter_score": 387.0, "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/score.json" } ] @@ -148,46 +294,74 @@ { "id": "simulation_heuristics_breakout_rgb_v1", "label": "Breakout RGB", - "metric": "normalized_score", + "metric": "raw_score", + "raw_metric": { + "label": "Native Breakout return", + "unit": "points" + }, "rows": [ { "harness": "opencode", "model": "GPT-5.5", "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", "normalized_score": 70.758122744, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 1, + "raw_score": 702.0, + "reference_score": 864.0, "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "starter_score": 310.0, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/score.json" }, { "harness": "opencode", "model": "GPT-5.6 Sol", "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", "normalized_score": 70.758122744, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "max", "rank": 1, + "raw_score": 702.0, + "reference_score": 864.0, "source_run": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "starter_score": 310.0, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/score.json" }, { "harness": "opencode", "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", "normalized_score": 15.523465704, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 3, + "raw_score": 396.0, + "reference_score": 864.0, "source_run": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "starter_score": 310.0, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/score.json" }, { "harness": "opencode", "model": "Claude Opus 4.8", "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", "normalized_score": 0.0, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 4, + "raw_score": 310.0, + "reference_score": 864.0, "source_run": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "starter_score": 310.0, "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/score.json" } ] @@ -195,46 +369,74 @@ { "id": "simulation_heuristics_halfcheetah_v1", "label": "HalfCheetah", - "metric": "normalized_score", + "metric": "raw_score", + "raw_metric": { + "label": "Weighted hidden return", + "unit": "return" + }, "rows": [ { "harness": "opencode", "model": "Claude Opus 4.8", "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", "normalized_score": 26.986707946, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 1, + "raw_score": 6021.504892591717, + "reference_score": 11487.256451471967, "source_run": "leaderboard/runs/article_suite/20260713T111714Z/claude-opus-4.8", + "starter_score": 4001.2886158917113, "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/score.json" }, { "harness": "opencode", "model": "GPT-5.6 Sol", "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", "normalized_score": 15.873072595, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "max", "rank": 2, + "raw_score": 5189.541724894487, + "reference_score": 11487.256451471967, "source_run": "leaderboard/runs/article_suite/20260713T084340Z/gpt-5.6-sol", + "starter_score": 4001.2886158917113, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/score.json" }, { "harness": "opencode", "model": "GPT-5.5", "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", "normalized_score": 2.73755349, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 3, + "raw_score": 4206.220989602626, + "reference_score": 11487.256451471967, "source_run": "leaderboard/runs/article_suite/20260713T103810Z/gpt-5.5", + "starter_score": 4001.2886158917113, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/score.json" }, { "harness": "opencode", "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", "normalized_score": -8.167582379, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 4, + "raw_score": 3389.8660260183765, + "reference_score": 11487.256451471967, "source_run": "leaderboard/runs/article_suite/20260713T093314Z/gpt-5.4-mini", + "starter_score": 4001.2886158917113, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/score.json" } ] @@ -242,46 +444,74 @@ { "id": "simulation_heuristics_vizdoom_d1_v1", "label": "VizDoom D1", - "metric": "normalized_score", + "metric": "raw_score", + "raw_metric": { + "label": "Native D1 mean reward", + "unit": "reward" + }, "rows": [ { "harness": "opencode", "model": "Claude Opus 4.8", "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", "normalized_score": 95.055812742, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 1, + "raw_score": 0.7992199431359768, + "reference_score": 0.8178199748694897, "source_run": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "starter_score": 0.44161998964846133, "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/score.json" }, { "harness": "opencode", "model": "GPT-5.5", "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", "normalized_score": 85.199359782, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 2, + "raw_score": 0.7621399685554205, + "reference_score": 0.8178199748694897, "source_run": "leaderboard/runs/article_suite/20260713T102218Z/gpt-5.5", + "starter_score": 0.44161998964846133, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/score.json" }, { "harness": "opencode", "model": "GPT-5.6 Sol", "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", "normalized_score": 49.409889366, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "max", "rank": 3, + "raw_score": 0.62749998614192, + "reference_score": 0.8178199748694897, "source_run": "leaderboard/runs/article_suite/20260713T101636Z/gpt-5.6-sol", + "starter_score": 0.44161998964846133, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/score.json" }, { "harness": "opencode", "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", "normalized_score": -161.116433137, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 4, + "raw_score": -0.16450000800192355, + "reference_score": 0.8178199748694897, "source_run": "leaderboard/runs/article_suite/20260713T102707Z/gpt-5.4-mini", + "starter_score": 0.44161998964846133, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/score.json" } ] @@ -289,46 +519,74 @@ { "id": "simulation_heuristics_vizdoom_d3_v1", "label": "VizDoom D3", - "metric": "normalized_score", + "metric": "raw_score", + "raw_metric": { + "label": "Native D3 mean reward", + "unit": "reward" + }, "rows": [ { "harness": "opencode", "model": "Claude Opus 4.8", "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", "normalized_score": 122.259358289, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 1, + "raw_score": 397.7, + "reference_score": 331.1, "source_run": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "starter_score": 31.9, "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/score.json" }, { "harness": "opencode", "model": "GPT-5.6 Sol", "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", "normalized_score": 26.236631016, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "max", "rank": 2, + "raw_score": 110.39999999999998, + "reference_score": 331.1, "source_run": "leaderboard/runs/article_suite/20260713T083940Z/gpt-5.6-sol", + "starter_score": 31.9, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/score.json" }, { "harness": "opencode", "model": "GPT-5.5", "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", "normalized_score": 5.481283422, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 3, + "raw_score": 48.3, + "reference_score": 331.1, "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "starter_score": 31.9, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/score.json" }, { "harness": "opencode", "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", "normalized_score": -9.859625668, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 4, + "raw_score": 2.4, + "reference_score": 331.1, "source_run": "leaderboard/runs/article_suite/20260713T104034Z/gpt-5.4-mini", + "starter_score": 31.9, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/score.json" } ] @@ -336,46 +594,74 @@ { "id": "simulation_heuristics_atari57_v1", "label": "Atari57", - "metric": "normalized_score", + "metric": "raw_score", + "raw_metric": { + "label": "Median best-mode HNS", + "unit": "HNS" + }, "rows": [ { "harness": "opencode", "model": "Claude Opus 4.8", "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", "normalized_score": 0.0, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 1, + "raw_score": 0.0, + "reference_score": 0.8283015254994576, "source_run": "leaderboard/runs/article_suite/20260713T123222Z/claude-opus-4.8", + "starter_score": 0.0, "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/score.json" }, { "harness": "opencode", "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", "normalized_score": 0.0, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 1, + "raw_score": 0.0, + "reference_score": 0.8283015254994576, "source_run": "leaderboard/runs/article_suite/20260713T111944Z/gpt-5.4-mini", + "starter_score": 0.0, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/score.json" }, { "harness": "opencode", "model": "GPT-5.5", "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", "normalized_score": 0.0, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 1, + "raw_score": 0.0, + "reference_score": 0.8283015254994576, "source_run": "leaderboard/runs/article_suite/20260713T111604Z/gpt-5.5", + "starter_score": 0.0, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/score.json" }, { "harness": "opencode", "model": "GPT-5.6 Sol", "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", "normalized_score": 0.0, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "max", "rank": 1, + "raw_score": 0.0, + "reference_score": 0.8283015254994576, "source_run": "leaderboard/runs/article_suite/20260713T111156Z/gpt-5.6-sol", + "starter_score": 0.0, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/score.json" } ] @@ -383,84 +669,147 @@ { "id": "simulation_heuristics_montezuma_v1", "label": "Montezuma's Revenge", - "metric": "normalized_score", + "metric": "raw_score", + "raw_metric": { + "label": "Capped native return", + "unit": "points" + }, "rows": [ { "harness": "opencode", "model": "GPT-5.5", "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", "normalized_score": 100.0, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 1, + "raw_score": 400.0, + "reference_score": 400.0, "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "starter_score": 0.0, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/score.json" }, { "harness": "opencode", "model": "Claude Opus 4.8", "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", "normalized_score": 0.0, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 2, + "raw_score": 0.0, + "reference_score": 400.0, "source_run": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "starter_score": 0.0, "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/score.json" }, { "harness": "opencode", "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", "normalized_score": 0.0, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 2, + "raw_score": 0.0, + "reference_score": 400.0, "source_run": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "starter_score": 0.0, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/score.json" }, { "harness": "opencode", "model": "GPT-5.6 Sol", "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", "normalized_score": 0.0, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "max", "rank": 2, + "raw_score": 0.0, + "reference_score": 400.0, "source_run": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "starter_score": 0.0, "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/score.json" } ] }, { - "id": "average", - "label": "Nine-task average", - "metric": "average_normalized_score", + "display_metric": "positive_display_score", + "display_transform": { + "formula": "positive_display_score = final_normalized_score + 100", + "offset": 100.0, + "purpose": "plot_only", + "type": "additive_offset" + }, + "id": "final", + "label": "Final normalized score", + "metric": "final_normalized_score", "rows": [ { + "arithmetic_mean_normalized_score": 43.19059338577778, "average_normalized_score": 43.19059338577778, + "final_normalized_score": 42.0019305542, "harness": "opencode", + "median_normalized_score": 45.833333333, "model": "GPT-5.5", "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "positive_display_score": 142.0019305542, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 1 }, { - "average_normalized_score": 39.81691467833333, + "arithmetic_mean_normalized_score": 39.37755364144445, + "average_normalized_score": 39.37755364144445, + "final_normalized_score": 36.7279720058, "harness": "opencode", - "model": "Claude Opus 4.8", - "model_id": "claude-opus-4.8", + "median_normalized_score": 39.203600385, + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", + "positive_display_score": 136.72797200579998, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "max", "rank": 2 }, { - "average_normalized_score": 39.37755364144445, + "arithmetic_mean_normalized_score": 39.81691467833333, + "average_normalized_score": 39.81691467833333, + "final_normalized_score": 27.218574763200003, "harness": "opencode", - "model": "GPT-5.6 Sol", - "model_id": "gpt-5.6-sol", + "median_normalized_score": 14.050353128, + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "positive_display_score": 127.2185747632, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 3 }, { + "arithmetic_mean_normalized_score": -29.71822245, "average_normalized_score": -29.71822245, + "final_normalized_score": -9.8773515774, "harness": "opencode", + "median_normalized_score": -8.167582379, "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", + "positive_display_score": 90.1226484226, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 4 } @@ -469,12 +818,29 @@ ], "rows": [ { + "arithmetic_mean_normalized_score": 43.19059338577778, "average_normalized_score": 43.19059338577778, + "final_normalized_score": 42.0019305542, "harness": "opencode", + "median_normalized_score": 45.833333333, "model": "GPT-5.5", "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 1, + "raw_task_scores": { + "simulation_heuristics_ant_v1": 1852.8533956222025, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 843.0, + "simulation_heuristics_breakout_rgb_v1": 702.0, + "simulation_heuristics_halfcheetah_v1": 4206.220989602626, + "simulation_heuristics_montezuma_v1": 400.0, + "simulation_heuristics_pong_ram_v1": 8.0, + "simulation_heuristics_vizdoom_d1_v1": 0.7621399685554205, + "simulation_heuristics_vizdoom_d3_v1": 48.3 + }, "source_runs": { "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T095807Z/gpt-5.5", "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T111604Z/gpt-5.5", @@ -497,6 +863,44 @@ "simulation_heuristics_vizdoom_d1_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/score.json", "simulation_heuristics_vizdoom_d3_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/score.json" }, + "task_anchors": { + "simulation_heuristics_ant_v1": { + "reference_score": 5942.333244755838, + "starter_score": 2443.815838992282 + }, + "simulation_heuristics_atari57_v1": { + "reference_score": 0.8283015254994576, + "starter_score": 0.0 + }, + "simulation_heuristics_breakout_ram_v1": { + "reference_score": 864.0, + "starter_score": 387.0 + }, + "simulation_heuristics_breakout_rgb_v1": { + "reference_score": 864.0, + "starter_score": 310.0 + }, + "simulation_heuristics_halfcheetah_v1": { + "reference_score": 11487.256451471967, + "starter_score": 4001.2886158917113 + }, + "simulation_heuristics_montezuma_v1": { + "reference_score": 400.0, + "starter_score": 0.0 + }, + "simulation_heuristics_pong_ram_v1": { + "reference_score": 21.0, + "starter_score": -2.9999999999999996 + }, + "simulation_heuristics_vizdoom_d1_v1": { + "reference_score": 0.8178199748694897, + "starter_score": 0.44161998964846133 + }, + "simulation_heuristics_vizdoom_d3_v1": { + "reference_score": 331.1, + "starter_score": 31.9 + } + }, "task_scores": { "simulation_heuristics_ant_v1": -16.891796576, "simulation_heuristics_atari57_v1": 0.0, @@ -510,53 +914,29 @@ } }, { - "average_normalized_score": 39.81691467833333, - "harness": "opencode", - "model": "Claude Opus 4.8", - "model_id": "claude-opus-4.8", - "provider_reasoning_effort": "max", - "rank": 2, - "source_runs": { - "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T142919Z/claude-opus-4.8", - "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T123222Z/claude-opus-4.8", - "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", - "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", - "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T111714Z/claude-opus-4.8", - "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", - "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", - "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", - "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8" - }, - "submission_details": { - "simulation_heuristics_ant_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/score.json", - "simulation_heuristics_atari57_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/score.json", - "simulation_heuristics_breakout_ram_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/score.json", - "simulation_heuristics_breakout_rgb_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/score.json", - "simulation_heuristics_halfcheetah_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/score.json", - "simulation_heuristics_montezuma_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/score.json", - "simulation_heuristics_pong_ram_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/score.json", - "simulation_heuristics_vizdoom_d1_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/score.json", - "simulation_heuristics_vizdoom_d3_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/score.json" - }, - "task_scores": { - "simulation_heuristics_ant_v1": 14.050353128, - "simulation_heuristics_atari57_v1": 0.0, - "simulation_heuristics_breakout_ram_v1": 0.0, - "simulation_heuristics_breakout_rgb_v1": 0.0, - "simulation_heuristics_halfcheetah_v1": 26.986707946, - "simulation_heuristics_montezuma_v1": 0.0, - "simulation_heuristics_pong_ram_v1": 100.0, - "simulation_heuristics_vizdoom_d1_v1": 95.055812742, - "simulation_heuristics_vizdoom_d3_v1": 122.259358289 - } - }, - { + "arithmetic_mean_normalized_score": 39.37755364144445, "average_normalized_score": 39.37755364144445, + "final_normalized_score": 36.7279720058, "harness": "opencode", + "median_normalized_score": 39.203600385, "model": "GPT-5.6 Sol", "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "max", - "rank": 3, + "rank": 2, + "raw_task_scores": { + "simulation_heuristics_ant_v1": 3815.360622145591, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 864.0, + "simulation_heuristics_breakout_rgb_v1": 702.0, + "simulation_heuristics_halfcheetah_v1": 5189.541724894487, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 9.7, + "simulation_heuristics_vizdoom_d1_v1": 0.62749998614192, + "simulation_heuristics_vizdoom_d3_v1": 110.39999999999998 + }, "source_runs": { "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T093815Z/gpt-5.6-sol", "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T111156Z/gpt-5.6-sol", @@ -579,6 +959,44 @@ "simulation_heuristics_vizdoom_d1_v1": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/score.json", "simulation_heuristics_vizdoom_d3_v1": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/score.json" }, + "task_anchors": { + "simulation_heuristics_ant_v1": { + "reference_score": 5942.333244755838, + "starter_score": 2443.815838992282 + }, + "simulation_heuristics_atari57_v1": { + "reference_score": 0.8283015254994576, + "starter_score": 0.0 + }, + "simulation_heuristics_breakout_ram_v1": { + "reference_score": 864.0, + "starter_score": 387.0 + }, + "simulation_heuristics_breakout_rgb_v1": { + "reference_score": 864.0, + "starter_score": 310.0 + }, + "simulation_heuristics_halfcheetah_v1": { + "reference_score": 11487.256451471967, + "starter_score": 4001.2886158917113 + }, + "simulation_heuristics_montezuma_v1": { + "reference_score": 400.0, + "starter_score": 0.0 + }, + "simulation_heuristics_pong_ram_v1": { + "reference_score": 21.0, + "starter_score": -2.9999999999999996 + }, + "simulation_heuristics_vizdoom_d1_v1": { + "reference_score": 0.8178199748694897, + "starter_score": 0.44161998964846133 + }, + "simulation_heuristics_vizdoom_d3_v1": { + "reference_score": 331.1, + "starter_score": 31.9 + } + }, "task_scores": { "simulation_heuristics_ant_v1": 39.203600385, "simulation_heuristics_atari57_v1": 0.0, @@ -592,12 +1010,125 @@ } }, { + "arithmetic_mean_normalized_score": 39.81691467833333, + "average_normalized_score": 39.81691467833333, + "final_normalized_score": 27.218574763200003, + "harness": "opencode", + "median_normalized_score": 14.050353128, + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max", + "rank": 3, + "raw_task_scores": { + "simulation_heuristics_ant_v1": 2935.3698887625987, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 387.0, + "simulation_heuristics_breakout_rgb_v1": 310.0, + "simulation_heuristics_halfcheetah_v1": 6021.504892591717, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 21.0, + "simulation_heuristics_vizdoom_d1_v1": 0.7992199431359768, + "simulation_heuristics_vizdoom_d3_v1": 397.7 + }, + "source_runs": { + "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T142919Z/claude-opus-4.8", + "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T123222Z/claude-opus-4.8", + "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T111714Z/claude-opus-4.8", + "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8" + }, + "submission_details": { + "simulation_heuristics_ant_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/score.json", + "simulation_heuristics_atari57_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/score.json", + "simulation_heuristics_breakout_ram_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/score.json", + "simulation_heuristics_breakout_rgb_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/score.json", + "simulation_heuristics_halfcheetah_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/score.json", + "simulation_heuristics_montezuma_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/score.json", + "simulation_heuristics_pong_ram_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/score.json", + "simulation_heuristics_vizdoom_d1_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/score.json", + "simulation_heuristics_vizdoom_d3_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/score.json" + }, + "task_anchors": { + "simulation_heuristics_ant_v1": { + "reference_score": 5942.333244755838, + "starter_score": 2443.815838992282 + }, + "simulation_heuristics_atari57_v1": { + "reference_score": 0.8283015254994576, + "starter_score": 0.0 + }, + "simulation_heuristics_breakout_ram_v1": { + "reference_score": 864.0, + "starter_score": 387.0 + }, + "simulation_heuristics_breakout_rgb_v1": { + "reference_score": 864.0, + "starter_score": 310.0 + }, + "simulation_heuristics_halfcheetah_v1": { + "reference_score": 11487.256451471967, + "starter_score": 4001.2886158917113 + }, + "simulation_heuristics_montezuma_v1": { + "reference_score": 400.0, + "starter_score": 0.0 + }, + "simulation_heuristics_pong_ram_v1": { + "reference_score": 21.0, + "starter_score": -2.9999999999999996 + }, + "simulation_heuristics_vizdoom_d1_v1": { + "reference_score": 0.8178199748694897, + "starter_score": 0.44161998964846133 + }, + "simulation_heuristics_vizdoom_d3_v1": { + "reference_score": 331.1, + "starter_score": 31.9 + } + }, + "task_scores": { + "simulation_heuristics_ant_v1": 14.050353128, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 0.0, + "simulation_heuristics_breakout_rgb_v1": 0.0, + "simulation_heuristics_halfcheetah_v1": 26.986707946, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 100.0, + "simulation_heuristics_vizdoom_d1_v1": 95.055812742, + "simulation_heuristics_vizdoom_d3_v1": 122.259358289 + } + }, + { + "arithmetic_mean_normalized_score": -29.71822245, "average_normalized_score": -29.71822245, + "final_normalized_score": -9.8773515774, "harness": "opencode", + "median_normalized_score": -8.167582379, "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 4, + "raw_task_scores": { + "simulation_heuristics_ant_v1": 1346.6965294546999, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 399.0, + "simulation_heuristics_breakout_rgb_v1": 396.0, + "simulation_heuristics_halfcheetah_v1": 3389.8660260183765, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": -21.0, + "simulation_heuristics_vizdoom_d1_v1": -0.16450000800192355, + "simulation_heuristics_vizdoom_d3_v1": 2.4 + }, "source_runs": { "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T104315Z/gpt-5.4-mini", "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T111944Z/gpt-5.4-mini", @@ -620,6 +1151,44 @@ "simulation_heuristics_vizdoom_d1_v1": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/score.json", "simulation_heuristics_vizdoom_d3_v1": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/score.json" }, + "task_anchors": { + "simulation_heuristics_ant_v1": { + "reference_score": 5942.333244755838, + "starter_score": 2443.815838992282 + }, + "simulation_heuristics_atari57_v1": { + "reference_score": 0.8283015254994576, + "starter_score": 0.0 + }, + "simulation_heuristics_breakout_ram_v1": { + "reference_score": 864.0, + "starter_score": 387.0 + }, + "simulation_heuristics_breakout_rgb_v1": { + "reference_score": 864.0, + "starter_score": 310.0 + }, + "simulation_heuristics_halfcheetah_v1": { + "reference_score": 11487.256451471967, + "starter_score": 4001.2886158917113 + }, + "simulation_heuristics_montezuma_v1": { + "reference_score": 400.0, + "starter_score": 0.0 + }, + "simulation_heuristics_pong_ram_v1": { + "reference_score": 21.0, + "starter_score": -2.9999999999999996 + }, + "simulation_heuristics_vizdoom_d1_v1": { + "reference_score": 0.8178199748694897, + "starter_score": 0.44161998964846133 + }, + "simulation_heuristics_vizdoom_d3_v1": { + "reference_score": 331.1, + "starter_score": 31.9 + } + }, "task_scores": { "simulation_heuristics_ant_v1": -31.35954984, "simulation_heuristics_atari57_v1": 0.0, diff --git a/leaderboard/article_suite_final_leaderboard.png b/leaderboard/article_suite_final_leaderboard.png new file mode 100644 index 0000000..3b61646 Binary files /dev/null and b/leaderboard/article_suite_final_leaderboard.png differ diff --git a/leaderboard/article_suite_task_leaderboards.png b/leaderboard/article_suite_task_leaderboards.png new file mode 100644 index 0000000..dd44da9 Binary files /dev/null and b/leaderboard/article_suite_task_leaderboards.png differ diff --git a/scripts/build_article_suite_leaderboard.py b/scripts/build_article_suite_leaderboard.py index c63f629..16ff80e 100644 --- a/scripts/build_article_suite_leaderboard.py +++ b/scripts/build_article_suite_leaderboard.py @@ -4,6 +4,8 @@ import argparse import json import math +import shutil +import statistics import tomllib from datetime import UTC, datetime from pathlib import Path @@ -36,7 +38,51 @@ "simulation_heuristics_atari57_v1": "Atari57", "simulation_heuristics_montezuma_v1": "Montezuma's Revenge", } -AVERAGE_LEADERBOARD_ID = "average" +TASK_RAW_METRICS = { + "simulation_heuristics_ant_v1": { + "label": "Weighted hidden return", + "unit": "return", + }, + "simulation_heuristics_pong_ram_v1": { + "label": "Native Pong score", + "unit": "points", + }, + "simulation_heuristics_breakout_ram_v1": { + "label": "Native Breakout return", + "unit": "points", + }, + "simulation_heuristics_breakout_rgb_v1": { + "label": "Native Breakout return", + "unit": "points", + }, + "simulation_heuristics_halfcheetah_v1": { + "label": "Weighted hidden return", + "unit": "return", + }, + "simulation_heuristics_vizdoom_d1_v1": { + "label": "Native D1 mean reward", + "unit": "reward", + }, + "simulation_heuristics_vizdoom_d3_v1": { + "label": "Native D3 mean reward", + "unit": "reward", + }, + "simulation_heuristics_atari57_v1": { + "label": "Median best-mode HNS", + "unit": "HNS", + }, + "simulation_heuristics_montezuma_v1": { + "label": "Capped native return", + "unit": "points", + }, +} +PROVIDER_LABELS = { + "azure": "Azure direct", + "claude_oauth": "Claude OAuth via pinned OpenCode plugin", +} +FINAL_LEADERBOARD_ID = "final" +IQM_TRIM_FRACTION = 0.25 +FINAL_DISPLAY_OFFSET = 100.0 TASK_DIGEST_COMPATIBILITY = { "simulation_heuristics_ant_v1": { "sha256:bbb533da0cb86459f4d49dee667e6c73ac54c0188bc40e54e911d50ef3c3bc38": ( @@ -76,7 +122,7 @@ def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser( - description="Build nine task leaderboards and their final average." + description="Build nine task leaderboards and their final IQM score." ) parser.add_argument( "--runs-root", @@ -228,6 +274,26 @@ def _rank_rows( return ranked +def _interquartile_mean(scores: list[float]) -> float: + if not scores: + raise ValueError("IQM requires at least one score") + ordered = sorted(float(score) for score in scores) + trim_count = int(len(ordered) * IQM_TRIM_FRACTION) + retained = ordered[trim_count : len(ordered) - trim_count] + if not retained: + raise ValueError("IQM trimming removed every score") + return math.fsum(retained) / len(retained) + + +def _aggregate_task_scores(task_scores: dict[str, float]) -> dict[str, float]: + scores = [float(task_scores[task]) for task in TASKS] + return { + "final_normalized_score": _interquartile_mean(scores), + "arithmetic_mean_normalized_score": math.fsum(scores) / len(scores), + "median_normalized_score": float(statistics.median(scores)), + } + + def _build_leaderboards( rows: list[dict[str, Any]], ) -> list[dict[str, Any]]: @@ -237,11 +303,21 @@ def _build_leaderboards( { "model_id": row["model_id"], "model": row["model"], + "model_route": row["model_route"], + "provider": row["provider"], + "provider_label": row["provider_label"], "harness": row["harness"], "provider_reasoning_effort": row[ "provider_reasoning_effort" ], "normalized_score": row["task_scores"][task], + "raw_score": row["raw_task_scores"][task], + "starter_score": row["task_anchors"][task][ + "starter_score" + ], + "reference_score": row["task_anchors"][task][ + "reference_score" + ], "submission_detail": row["submission_details"][task], "source_run": row["source_runs"][task], } @@ -251,34 +327,52 @@ def _build_leaderboards( { "id": task, "label": TASK_LABELS[task], - "metric": "normalized_score", + "metric": "raw_score", + "raw_metric": TASK_RAW_METRICS[task], "rows": _rank_rows( task_rows, - score_key="normalized_score", + score_key="raw_score", ), } ) - average_rows = [ + final_rows = [ { "model_id": row["model_id"], "model": row["model"], + "model_route": row["model_route"], + "provider": row["provider"], + "provider_label": row["provider_label"], "harness": row["harness"], "provider_reasoning_effort": row[ "provider_reasoning_effort" ], + "final_normalized_score": row["final_normalized_score"], + "arithmetic_mean_normalized_score": row[ + "arithmetic_mean_normalized_score" + ], + "median_normalized_score": row["median_normalized_score"], + "positive_display_score": row["final_normalized_score"] + + FINAL_DISPLAY_OFFSET, "average_normalized_score": row["average_normalized_score"], } for row in rows ] leaderboards.append( { - "id": AVERAGE_LEADERBOARD_ID, - "label": "Nine-task average", - "metric": "average_normalized_score", + "id": FINAL_LEADERBOARD_ID, + "label": "Final normalized score", + "metric": "final_normalized_score", + "display_metric": "positive_display_score", + "display_transform": { + "type": "additive_offset", + "offset": FINAL_DISPLAY_OFFSET, + "formula": "positive_display_score = final_normalized_score + 100", + "purpose": "plot_only", + }, "rows": _rank_rows( - average_rows, - score_key="average_normalized_score", + final_rows, + score_key="final_normalized_score", ), } ) @@ -297,71 +391,41 @@ def _leaderboard_relative_path(path: str) -> str: def _render_article_suite_markdown( leaderboards: list[dict[str, Any]], ) -> str: - markdown = [ + final_board = leaderboards[-1] + if final_board["id"] != FINAL_LEADERBOARD_ID: + raise ValueError("Final leaderboard must be last") + return "\n".join( + [ "# GenesisBench Learning Beyond Gradients Article Suite", "", - "This offline report contains 10 independent leaderboards: one for " - "each of the nine article-derived tasks, followed by the final " - "nine-task average.", + "The first image contains the nine independently ranked task " + "leaderboards. The second image contains the final cross-task ranking.", "", - "Task scores are unbounded normalized values. The public starter maps " - "to 0 and the trusted article-level reference maps to 100.", - ] - for index, board in enumerate(leaderboards, start=1): - markdown.extend(["", f"## {index}. {board['label']}", ""]) - if board["id"] == AVERAGE_LEADERBOARD_ID: - markdown.extend( - [ - "Arithmetic mean of the nine normalized task scores.", - "", - "| Rank | Model | Harness | Effort | Nine-task average |", - "| ---: | --- | --- | --- | ---: |", - ] - ) - for row in board["rows"]: - markdown.append( - "| " - + " | ".join( - [ - str(row["rank"]), - row["model"], - row["harness"], - row["provider_reasoning_effort"], - f"{row['average_normalized_score']:.2f}", - ] - ) - + " |" - ) - continue - - markdown.extend( - [ - f"Task: `{board['id']}`", - "", - "| Rank | Model | Harness | Effort | Normalized score | " - "Score details |", - "| ---: | --- | --- | --- | ---: | --- |", - ] - ) - for row in board["rows"]: - detail_path = _leaderboard_relative_path( - row["submission_detail"] - ) - markdown.append( - "| " - + " | ".join( - [ - str(row["rank"]), - row["model"], - row["harness"], - row["provider_reasoning_effort"], - f"{row['normalized_score']:.2f}", - f"[JSON]({detail_path})", - ] - ) - + " |" - ) - return "\n".join(markdown) + "\n" + "The nine task panels use each environment's native raw score. " + "The final scientific metric remains unbounded IQM.", + "", + "## Nine task leaderboards", + "", + "![Nine task-specific GenesisBench leaderboards]" + "(article_suite_task_leaderboards.png)", + "", + "## Final normalized score", + "", + "The primary score is the interquartile mean (IQM): sort the nine task " + "scores, remove the lowest two and highest two, then average the middle " + "five. The image uses a plot-only positive display index equal to " + "`IQM + 100`; raw IQM, arithmetic mean, and median remain in the JSON.", + "", + "![Final GenesisBench article-suite leaderboard]" + "(article_suite_final_leaderboard.png)", + "", + "Machine-readable rankings and score-detail paths are available in " + "[`article_suite.json`](article_suite.json). The scoring rationale is " + "documented in " + "[`docs/article-suite-scoring.md`](../docs/article-suite-scoring.md).", + "", + ] + ) def _latest_model_runs(runs_root: Path) -> dict[str, Path]: @@ -493,6 +557,8 @@ def main() -> None: task: _normalized_task_score(model_results[task][1]) for task in TASKS } + raw_task_scores: dict[str, float] = {} + task_anchors: dict[str, dict[str, float]] = {} submission_details: dict[str, str] = {} for task in TASKS: _, result_row, model_root, digest = model_results[task] @@ -516,6 +582,27 @@ def main() -> None: sanitized_score = _sanitize_score_paths( json.loads(source_score.read_text()) ) + raw_score = sanitized_score.get("score") + starter_score = sanitized_score.get("starter_score") + reference_score = sanitized_score.get("reference_score") + for name, value in ( + ("score", raw_score), + ("starter_score", starter_score), + ("reference_score", reference_score), + ): + if ( + not isinstance(value, int | float) + or isinstance(value, bool) + or not math.isfinite(float(value)) + ): + raise RuntimeError( + f"{model_id}/{task} has invalid {name} {value!r}" + ) + raw_task_scores[task] = float(raw_score) + task_anchors[task] = { + "starter_score": float(starter_score), + "reference_score": float(reference_score), + } (destination / "score.json").write_text( json.dumps(sanitized_score, indent=2, sort_keys=True) + "\n" ) @@ -546,17 +633,30 @@ def main() -> None: submission_details[task] = str( (destination / "score.json").relative_to(REPO_ROOT) ) - average = sum(task_scores.values()) / len(TASKS) + aggregate_scores = _aggregate_task_scores(task_scores) ranked.append( { "model_id": metadata["model"]["id"], "model": metadata["model"]["display_name"], + "model_route": metadata["model"]["model"], + "provider": metadata["model"]["provider"], + "provider_label": PROVIDER_LABELS.get( + metadata["model"]["provider"], + metadata["model"]["provider"], + ), "harness": metadata["harness"], "provider_reasoning_effort": metadata[ "provider_reasoning_effort" ], - "average_normalized_score": average, + **aggregate_scores, + # Backward-compatible alias for consumers of the first + # published schema. This is not the primary ranking metric. + "average_normalized_score": aggregate_scores[ + "arithmetic_mean_normalized_score" + ], "task_scores": task_scores, + "raw_task_scores": raw_task_scores, + "task_anchors": task_anchors, "submission_details": submission_details, "source_runs": { task: str( @@ -568,7 +668,7 @@ def main() -> None: ) ranked = _rank_rows( ranked, - score_key="average_normalized_score", + score_key="final_normalized_score", ) leaderboards = _build_leaderboards(ranked) @@ -577,8 +677,51 @@ def main() -> None: "task_count": len(TASKS), "leaderboard_count": len(leaderboards), "tasks": list(TASKS), - "aggregation": "arithmetic_mean_of_normalized_task_scores", + "aggregation": { + "primary_metric": "interquartile_mean", + "primary_field": "final_normalized_score", + "trim_fraction_per_tail": IQM_TRIM_FRACTION, + "trimmed_score_count_per_tail": int( + len(TASKS) * IQM_TRIM_FRACTION + ), + "retained_score_count": len(TASKS) + - 2 * int(len(TASKS) * IQM_TRIM_FRACTION), + "score_bounds": "unbounded", + "secondary_fields": [ + "arithmetic_mean_normalized_score", + "median_normalized_score", + ], + "uncertainty": "not_estimated_single_run_per_model_task", + "display_transform": { + "type": "additive_offset", + "offset": FINAL_DISPLAY_OFFSET, + "formula": "positive_display_score = final_normalized_score + 100", + "purpose": "plot_only", + "ranking_field": "final_normalized_score", + }, + }, "leaderboards": leaderboards, + "inference_settings": { + "field": "provider_reasoning_effort", + "interpretation": "provider_specific_categorical_setting", + "cross_provider_comparability": ( + "labels_are_not_a_shared_numeric_compute_scale" + ), + "models": [ + { + "model_id": row["model_id"], + "model": row["model"], + "model_route": row["model_route"], + "provider": row["provider"], + "provider_label": row["provider_label"], + "harness": row["harness"], + "provider_reasoning_effort": row[ + "provider_reasoning_effort" + ], + } + for row in ranked + ], + }, "task_digest_compatibility": TASK_DIGEST_COMPATIBILITY, "source_runs": { model_id: { @@ -599,10 +742,26 @@ def main() -> None: } args.output.parent.mkdir(parents=True, exist_ok=True) args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n") + website_data = REPO_ROOT / "website" / "assets" / "article_suite.json" + website_data.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(args.output, website_data) (args.output.parent / "ARTICLE_SUITE.md").write_text( _render_article_suite_markdown(leaderboards) ) + try: + from scripts.plot_article_suite_leaderboards import ( + render_article_suite_leaderboards, + ) + except ModuleNotFoundError: + from plot_article_suite_leaderboards import ( + render_article_suite_leaderboards, + ) + + render_article_suite_leaderboards( + payload, + leaderboard_dir=args.output.parent, + ) print(args.output) diff --git a/scripts/plot_article_suite_leaderboards.py b/scripts/plot_article_suite_leaderboards.py new file mode 100644 index 0000000..06bcc1e --- /dev/null +++ b/scripts/plot_article_suite_leaderboards.py @@ -0,0 +1,308 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import json +import math +from pathlib import Path +from typing import Any + +import matplotlib + +matplotlib.use("Agg") + +import matplotlib.pyplot as plt +from matplotlib.patches import Patch + + +REPO_ROOT = Path(__file__).resolve().parents[1] +TASK_IMAGE_NAME = "article_suite_task_leaderboards.png" +FINAL_IMAGE_NAME = "article_suite_final_leaderboard.png" +MODEL_COLORS = { + "gpt-5.5": "#111111", + "gpt-5.6-sol": "#444444", + "claude-opus-4.8": "#777777", + "gpt-5.4-mini": "#b5b5b5", +} + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Plot article-suite task and final leaderboards." + ) + parser.add_argument( + "--leaderboard", + type=Path, + default=REPO_ROOT / "leaderboard" / "article_suite.json", + ) + parser.add_argument( + "--leaderboard-dir", + type=Path, + default=REPO_ROOT / "leaderboard", + ) + return parser.parse_args() + + +def _style() -> None: + plt.rcParams.update( + { + "font.family": "DejaVu Sans", + "axes.edgecolor": "#d1d1d1", + "axes.labelcolor": "#333333", + "xtick.color": "#666666", + "ytick.color": "#171717", + "text.color": "#171717", + } + ) + + +def _score_limits(task_boards: list[dict[str, Any]]) -> tuple[float, float]: + scores = [ + float(row["raw_score"]) + for board in task_boards + for row in board["rows"] + ] + lower = min(-25.0, 25.0 * math.floor((min(scores) - 10.0) / 25.0)) + upper = max(125.0, 25.0 * math.ceil((max(scores) + 10.0) / 25.0)) + return lower, upper + + +def _bar_label_x(score: float, span: float) -> tuple[float, str]: + offset = span * 0.012 + if score >= 0: + return score + offset, "left" + return score - offset, "right" + + +def _plot_task_leaderboards( + task_boards: list[dict[str, Any]], + output: Path, +) -> None: + _style() + figure, axes = plt.subplots(3, 3, figsize=(18, 14), dpi=160) + figure.patch.set_facecolor("#fafafa") + + for axis, board in zip(axes.flat, task_boards, strict=True): + axis.set_facecolor("#fafafa") + rows = board["rows"] + names = [row["model"] for row in rows] + scores = [float(row["raw_score"]) for row in rows] + starter_score = float(rows[0]["starter_score"]) + reference_score = float(rows[0]["reference_score"]) + all_values = [*scores, starter_score, reference_score] + value_min = min(all_values) + value_max = max(all_values) + padding = max((value_max - value_min) * 0.16, abs(value_max) * 0.05, 1.0) + lower = min(0.0, value_min - padding) + upper = max(0.0, value_max + padding) + span = upper - lower + colors = [ + MODEL_COLORS.get(row["model_id"], "#666666") for row in rows + ] + bars = axis.barh(names, scores, color=colors, height=0.62) + axis.invert_yaxis() + axis.axvline( + starter_score, + color="#777777", + linewidth=1.0, + linestyle=(0, (1, 3)), + ) + axis.axvline( + reference_score, + color="#8a8a8a", + linewidth=1.0, + linestyle=(0, (4, 4)), + ) + axis.set_xlim(lower, upper) + axis.set_title( + board["label"], + loc="left", + fontsize=15, + fontweight="bold", + pad=10, + ) + axis.text( + 0, + 1.01, + board["raw_metric"]["label"], + transform=axis.transAxes, + fontsize=8, + color="#777777", + ) + axis.grid(axis="x", color="#e3e3e3", linewidth=0.8) + axis.set_axisbelow(True) + axis.spines[["top", "right", "left"]].set_visible(False) + axis.tick_params(axis="y", length=0, labelsize=9.5) + axis.tick_params(axis="x", labelsize=8.5) + for bar, score in zip(bars, scores, strict=True): + if score < lower + span * 0.08: + x = score + span * 0.012 + alignment = "left" + label_color = "#ffffff" + else: + x, alignment = _bar_label_x(score, span) + label_color = "#171717" + axis.text( + x, + bar.get_y() + bar.get_height() / 2, + f"{score:.1f}", + va="center", + ha=alignment, + fontsize=9, + fontweight="bold", + color=label_color, + ) + + figure.suptitle( + "GenesisBench — Nine Task Leaderboards", + x=0.055, + y=0.985, + ha="left", + fontsize=24, + fontweight="bold", + ) + figure.text( + 0.055, + 0.955, + "Native raw score for each environment · task-specific axes", + fontsize=11, + color="#666666", + ) + legend_handles = [ + Patch(facecolor=color, label=model_id) + for model_id, color in MODEL_COLORS.items() + ] + figure.legend( + handles=legend_handles, + loc="lower center", + ncol=4, + frameon=False, + fontsize=10, + bbox_to_anchor=(0.5, 0.01), + ) + figure.text( + 0.945, + 0.02, + "Dotted: starter · dashed: article reference", + ha="right", + fontsize=9, + color="#777777", + ) + figure.tight_layout(rect=(0.035, 0.055, 0.98, 0.935)) + output.parent.mkdir(parents=True, exist_ok=True) + figure.savefig( + output, + bbox_inches="tight", + facecolor=figure.get_facecolor(), + metadata={"Software": "GenesisBench"}, + ) + plt.close(figure) + + +def _plot_final_leaderboard( + final_board: dict[str, Any], + output: Path, +) -> None: + _style() + rows = final_board["rows"] + names = [f"{row['rank']} {row['model']}" for row in rows] + scores = [float(row["positive_display_score"]) for row in rows] + colors = [ + MODEL_COLORS.get(row["model_id"], "#666666") for row in rows + ] + + figure, axis = plt.subplots(figsize=(12, 7.2), dpi=160) + figure.patch.set_facecolor("#fafafa") + axis.set_facecolor("#fafafa") + bars = axis.barh(names, scores, color=colors, height=0.62) + axis.invert_yaxis() + lower = 0.0 + upper = max(160.0, math.ceil((max(scores) + 10.0) / 10.0) * 10.0) + span = upper - lower + axis.set_xlim(lower, upper) + axis.axvline( + 100, + color="#777777", + linewidth=1.0, + linestyle=(0, (1, 3)), + ) + axis.grid(axis="x", color="#e3e3e3", linewidth=0.8) + axis.set_axisbelow(True) + axis.spines[["top", "right", "left"]].set_visible(False) + axis.tick_params(axis="y", length=0, labelsize=12) + axis.tick_params(axis="x", labelsize=10) + axis.set_xlabel("Positive display index (IQM + 100)", fontsize=11) + + for bar, row, score in zip(bars, rows, scores, strict=True): + label_color = ( + "#171717" if row["model_id"] == "gpt-5.4-mini" else "#ffffff" + ) + axis.text( + score - span * 0.015, + bar.get_y() + bar.get_height() / 2, + f"{score:.2f} (IQM {row['final_normalized_score']:.2f})", + va="center", + ha="right", + fontsize=10.5, + fontweight="bold", + color=label_color, + ) + + axis.set_title( + "GenesisBench — Final Article-Suite Leaderboard", + loc="left", + fontsize=22, + fontweight="bold", + pad=22, + ) + axis.text( + 0, + 1.025, + "Plot-only positive index = IQM + 100 · ranking and gaps unchanged · " + "dotted line = starter-level aggregate", + transform=axis.transAxes, + fontsize=10.5, + color="#666666", + ) + figure.subplots_adjust(left=0.25, right=0.97, top=0.82, bottom=0.15) + output.parent.mkdir(parents=True, exist_ok=True) + figure.savefig( + output, + bbox_inches="tight", + facecolor=figure.get_facecolor(), + metadata={"Software": "GenesisBench"}, + ) + plt.close(figure) + + +def render_article_suite_leaderboards( + payload: dict[str, Any], + *, + leaderboard_dir: Path, +) -> tuple[Path, Path]: + boards = payload["leaderboards"] + task_boards = boards[:-1] + final_board = boards[-1] + if len(task_boards) != 9 or final_board["id"] != "final": + raise ValueError("Expected nine task boards followed by final board") + + task_output = leaderboard_dir / TASK_IMAGE_NAME + final_output = leaderboard_dir / FINAL_IMAGE_NAME + _plot_task_leaderboards(task_boards, task_output) + _plot_final_leaderboard(final_board, final_output) + return task_output, final_output + + +def main() -> None: + args = parse_args() + payload = json.loads(args.leaderboard.read_text()) + outputs = render_article_suite_leaderboards( + payload, + leaderboard_dir=args.leaderboard_dir, + ) + for output in outputs: + print(output) + + +if __name__ == "__main__": + main() diff --git a/tests/test_article_suite_tooling.py b/tests/test_article_suite_tooling.py index 7afb4a9..b26b828 100644 --- a/tests/test_article_suite_tooling.py +++ b/tests/test_article_suite_tooling.py @@ -420,21 +420,49 @@ def test_latest_task_results_merges_partial_runs(tmp_path: Path) -> None: assert selected[model_id][leaderboard.TASKS[1]][3] == "sha256:digest-2" -def test_offline_report_builds_nine_task_boards_then_average() -> None: +def test_iqm_matches_rliable_25_percent_trimmed_mean() -> None: + assert leaderboard._interquartile_mean(list(range(9))) == 4.0 + assert leaderboard._interquartile_mean( + [-100.0, -50.0, 1.0, 2.0, 3.0, 4.0, 5.0, 100.0, 500.0] + ) == 3.0 + + +def test_offline_report_builds_nine_task_boards_then_final() -> None: rows = [] - for model_id, model, average in ( + for model_id, model, baseline in ( ("model-a", "Model A", 5.0), ("model-b", "Model B", 10.0), ): - task_scores = {task: 0.0 for task in leaderboard.TASKS} + task_scores = {task: baseline for task in leaderboard.TASKS} + if model_id == "model-a": + task_scores[leaderboard.TASKS[0]] = 10.0 + task_scores[leaderboard.TASKS[1]] = 30.0 + else: + task_scores[leaderboard.TASKS[0]] = 20.0 + task_scores[leaderboard.TASKS[1]] = 5.0 + aggregates = leaderboard._aggregate_task_scores(task_scores) rows.append( - { - "model_id": model_id, - "model": model, - "harness": "opencode", + { + "model_id": model_id, + "model": model, + "model_route": f"test/{model_id}", + "provider": "test", + "provider_label": "Test provider", + "harness": "opencode", "provider_reasoning_effort": "max", - "average_normalized_score": average, + **aggregates, + "average_normalized_score": aggregates[ + "arithmetic_mean_normalized_score" + ], "task_scores": task_scores, + "raw_task_scores": task_scores, + "task_anchors": { + task: { + "starter_score": 0.0, + "reference_score": 100.0, + } + for task in leaderboard.TASKS + }, "submission_details": { task: f"leaderboard/submissions/{model_id}/{task}.json" for task in leaderboard.TASKS @@ -445,17 +473,18 @@ def test_offline_report_builds_nine_task_boards_then_average() -> None: }, } ) - rows[0]["task_scores"][leaderboard.TASKS[0]] = 10.0 - rows[1]["task_scores"][leaderboard.TASKS[0]] = 20.0 - rows[0]["task_scores"][leaderboard.TASKS[1]] = 30.0 - rows[1]["task_scores"][leaderboard.TASKS[1]] = 5.0 boards = leaderboard._build_leaderboards(rows) markdown = leaderboard._render_article_suite_markdown(boards) assert len(boards) == 10 assert [board["id"] for board in boards[:-1]] == list(leaderboard.TASKS) - assert boards[-1]["id"] == leaderboard.AVERAGE_LEADERBOARD_ID + assert boards[-1]["id"] == leaderboard.FINAL_LEADERBOARD_ID + assert boards[0]["metric"] == "raw_score" + assert boards[-1]["display_transform"]["offset"] == 100.0 + assert all( + row["positive_display_score"] > 0 for row in boards[-1]["rows"] + ) assert [row["model_id"] for row in boards[0]["rows"]] == [ "model-b", "model-a", @@ -464,12 +493,11 @@ def test_offline_report_builds_nine_task_boards_then_average() -> None: "model-a", "model-b", ] - assert [row["rank"] for row in boards[2]["rows"]] == [1, 1] + assert [row["rank"] for row in boards[2]["rows"]] == [1, 2] assert [row["model_id"] for row in boards[-1]["rows"]] == [ "model-b", "model-a", ] - assert markdown.count("| Rank | Model | Harness | Effort |") == 10 - assert markdown.rstrip().split("## ")[-1].startswith( - "10. Nine-task average" - ) + assert "article_suite_task_leaderboards.png" in markdown + assert "article_suite_final_leaderboard.png" in markdown + assert "| Rank |" not in markdown diff --git a/tests/test_leaderboard_artifacts.py b/tests/test_leaderboard_artifacts.py index de1c3a0..b82e97f 100644 --- a/tests/test_leaderboard_artifacts.py +++ b/tests/test_leaderboard_artifacts.py @@ -3,6 +3,7 @@ import json import math from pathlib import Path +import struct REPO_ROOT = Path(__file__).resolve().parents[1] @@ -10,6 +11,16 @@ ARTICLE_SUITE = REPO_ROOT / "leaderboard" / "article_suite.json" ARTICLE_SUITE_MARKDOWN = REPO_ROOT / "leaderboard" / "ARTICLE_SUITE.md" ROOT_README = REPO_ROOT / "README.md" +WEBSITE = REPO_ROOT / "website" / "index.html" +WEBSITE_DATA = REPO_ROOT / "website" / "assets" / "article_suite.json" +TASK_IMAGE = REPO_ROOT / "leaderboard" / "article_suite_task_leaderboards.png" +FINAL_IMAGE = REPO_ROOT / "leaderboard" / "article_suite_final_leaderboard.png" + + +def _png_dimensions(path: Path) -> tuple[int, int]: + header = path.read_bytes()[:24] + assert header[:8] == b"\x89PNG\r\n\x1a\n" + return struct.unpack(">II", header[16:24]) def test_packaged_leaderboard_is_self_contained() -> None: @@ -47,21 +58,54 @@ def test_article_suite_leaderboard_is_complete_and_self_contained() -> None: assert [board["id"] for board in payload["leaderboards"][:-1]] == payload[ "tasks" ] - assert payload["leaderboards"][-1]["id"] == "average" - - averages = [ - row["average_normalized_score"] for row in payload["rows"] + assert payload["leaderboards"][-1]["id"] == "final" + assert payload["aggregation"]["primary_metric"] == "interquartile_mean" + assert payload["aggregation"]["trim_fraction_per_tail"] == 0.25 + assert payload["aggregation"]["trimmed_score_count_per_tail"] == 2 + assert payload["aggregation"]["retained_score_count"] == 5 + assert ( + payload["inference_settings"]["cross_provider_comparability"] + == "labels_are_not_a_shared_numeric_compute_scale" + ) + inference_by_model = { + row["model_id"]: row for row in payload["inference_settings"]["models"] + } + assert inference_by_model["gpt-5.6-sol"]["provider_reasoning_effort"] == "max" + assert inference_by_model["gpt-5.5"]["provider_reasoning_effort"] == "xhigh" + assert ( + inference_by_model["claude-opus-4.8"]["provider_reasoning_effort"] + == "max" + ) + assert ( + inference_by_model["gpt-5.4-mini"]["provider_reasoning_effort"] + == "xhigh" + ) + + final_scores = [ + row["final_normalized_score"] for row in payload["rows"] ] - assert averages == sorted(averages, reverse=True) + assert final_scores == sorted(final_scores, reverse=True) for row in payload["rows"]: assert row["harness"] == "opencode" assert set(row["task_scores"]) == set(payload["tasks"]) + assert set(row["raw_task_scores"]) == set(payload["tasks"]) + assert set(row["task_anchors"]) == set(payload["tasks"]) assert set(row["submission_details"]) == set(payload["tasks"]) + sorted_scores = sorted(row["task_scores"].values()) + expected_iqm = sum(sorted_scores[2:7]) / 5 assert math.isclose( - row["average_normalized_score"], + row["final_normalized_score"], + expected_iqm, + ) + assert math.isclose( + row["arithmetic_mean_normalized_score"], sum(row["task_scores"].values()) / payload["task_count"], ) + assert ( + row["average_normalized_score"] + == row["arithmetic_mean_normalized_score"] + ) for task, relative_score_path in row["submission_details"].items(): score_path = REPO_ROOT / relative_score_path metadata_path = score_path.with_name("metadata.json") @@ -90,27 +134,68 @@ def assert_no_absolute_artifact_paths(value: object) -> None: model_rows = {row["model_id"]: row for row in payload["rows"]} for board in payload["leaderboards"][:-1]: - scores = [row["normalized_score"] for row in board["rows"]] + scores = [row["raw_score"] for row in board["rows"]] assert scores == sorted(scores, reverse=True) assert len(board["rows"]) == len(payload["rows"]) for row in board["rows"]: - assert row["normalized_score"] == model_rows[row["model_id"]][ - "task_scores" - ][board["id"]] - - average_board = payload["leaderboards"][-1] + model_row = model_rows[row["model_id"]] + assert row["normalized_score"] == model_row["task_scores"][ + board["id"] + ] + assert row["raw_score"] == model_row["raw_task_scores"][ + board["id"] + ] + assert row["starter_score"] == model_row["task_anchors"][ + board["id"] + ]["starter_score"] + assert row["reference_score"] == model_row["task_anchors"][ + board["id"] + ]["reference_score"] + + final_board = payload["leaderboards"][-1] assert [ - row["average_normalized_score"] for row in average_board["rows"] - ] == averages + row["final_normalized_score"] for row in final_board["rows"] + ] == final_scores + assert final_board["display_transform"]["offset"] == 100.0 + assert all( + math.isclose( + row["positive_display_score"], + row["final_normalized_score"] + 100.0, + ) + for row in final_board["rows"] + ) + assert [row["model"] for row in payload["rows"]] == [ + "GPT-5.5", + "GPT-5.6 Sol", + "Claude Opus 4.8", + "GPT-5.4 Mini", + ] markdown = ARTICLE_SUITE_MARKDOWN.read_text() - assert markdown.count("| Rank | Model | Harness | Effort |") == 10 - headings = [ - line for line in markdown.splitlines() if line.startswith("## ") - ] - assert len(headings) == 10 - assert headings[-1] == "## 10. Nine-task average" + assert "article_suite_task_leaderboards.png" in markdown + assert "article_suite_final_leaderboard.png" in markdown + assert "| Rank |" not in markdown root_readme = ROOT_README.read_text() - assert "| Rank | Model | Nine-task average |" in root_readme + assert "leaderboard/article_suite_final_leaderboard.png" in root_readme + assert "leaderboard/article_suite_task_leaderboards.png" not in root_readme + assert "| Rank | Model | Nine-task average |" not in root_readme assert "## Legacy Ant-Only Leaderboard" not in root_readme + + website = WEBSITE.read_text() + assert "assets/article_suite_task_leaderboards.png" not in website + assert "assets/article_suite_final_leaderboard.png" not in website + assert '

Leaderboards

' in website + assert "Task-level view" not in website + assert 'id="taskLeaderboards"' in website + assert 'id="finalLeaderboard"' in website + assert 'id="inferenceSettings"' in website + assert "not a shared numeric compute scale" in website + assert 'fetch("assets/article_suite.json")' in website + assert json.loads(WEBSITE_DATA.read_text()) == payload + + for source in (TASK_IMAGE, FINAL_IMAGE): + assert source.is_file() + width, height = _png_dimensions(source) + assert width >= 1500 + assert height >= 800 diff --git a/website/README.md b/website/README.md index 0fc9cfc..22f26a2 100644 --- a/website/README.md +++ b/website/README.md @@ -18,9 +18,23 @@ a full humanoid. Composed from the project's own six pencil illustrations with A Page content follows the repo README: an interactive six-stage pipeline (heuristic policies → VLA training, simulation → real world → more hardware → 2.0) whose first stage embeds an interactive improvement-loop diagram on the MuJoCo Ant (assets/ant.png), an environments diagram -(simulation frameworks: Genesis, Isaac Sim, RoboCasa, MuJoCo · real-world operation), and the -research background. GenesisBench 1.0 — language intelligence → physical intelligence; -2.0 — world intelligence. +(simulation frameworks: Genesis, Isaac Sim, RoboCasa, MuJoCo · real-world operation), article-suite +results with task and final leaderboard images, and the research background. GenesisBench 1.0 — +language intelligence → physical intelligence; 2.0 — world intelligence. + +The website copies `leaderboard/article_suite.json` to +`website/assets/article_suite.json` and renders both plots as responsive native +HTML/CSS. No raster leaderboard image is embedded on the page. + +Offline README/report images are generated separately by: + +```bash +uv run python scripts/plot_article_suite_leaderboards.py +``` + +The HTML task panels use native raw environment scores. The final HTML plot +uses a fixed positive display index (`IQM + 100`) while retaining raw IQM as +the official ranking metric. ## Deploy — genesisbench.benchflow.ai (Vercel, same as clawsbench) diff --git a/website/assets/article_suite.json b/website/assets/article_suite.json new file mode 100644 index 0000000..b490741 --- /dev/null +++ b/website/assets/article_suite.json @@ -0,0 +1,1278 @@ +{ + "aggregation": { + "display_transform": { + "formula": "positive_display_score = final_normalized_score + 100", + "offset": 100.0, + "purpose": "plot_only", + "ranking_field": "final_normalized_score", + "type": "additive_offset" + }, + "primary_field": "final_normalized_score", + "primary_metric": "interquartile_mean", + "retained_score_count": 5, + "score_bounds": "unbounded", + "secondary_fields": [ + "arithmetic_mean_normalized_score", + "median_normalized_score" + ], + "trim_fraction_per_tail": 0.25, + "trimmed_score_count_per_tail": 2, + "uncertainty": "not_estimated_single_run_per_model_task" + }, + "benchmark": "learning_beyond_gradients_article_suite", + "generated_at": "2026-07-13T15:34:22.260689+00:00", + "inference_settings": { + "cross_provider_comparability": "labels_are_not_a_shared_numeric_compute_scale", + "field": "provider_reasoning_effort", + "interpretation": "provider_specific_categorical_setting", + "models": [ + { + "harness": "opencode", + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh" + }, + { + "harness": "opencode", + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "max" + }, + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max" + }, + { + "harness": "opencode", + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh" + } + ] + }, + "leaderboard_count": 10, + "leaderboards": [ + { + "id": "simulation_heuristics_ant_v1", + "label": "Ant", + "metric": "raw_score", + "raw_metric": { + "label": "Weighted hidden return", + "unit": "return" + }, + "rows": [ + { + "harness": "opencode", + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", + "normalized_score": 39.203600385, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "max", + "rank": 1, + "raw_score": 3815.360622145591, + "reference_score": 5942.333244755838, + "source_run": "leaderboard/runs/article_suite/20260713T093815Z/gpt-5.6-sol", + "starter_score": 2443.815838992282, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/score.json" + }, + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "normalized_score": 14.050353128, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max", + "rank": 2, + "raw_score": 2935.3698887625987, + "reference_score": 5942.333244755838, + "source_run": "leaderboard/runs/article_suite/20260713T142919Z/claude-opus-4.8", + "starter_score": 2443.815838992282, + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "normalized_score": -16.891796576, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 3, + "raw_score": 1852.8533956222025, + "reference_score": 5942.333244755838, + "source_run": "leaderboard/runs/article_suite/20260713T095807Z/gpt-5.5", + "starter_score": 2443.815838992282, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", + "normalized_score": -31.35954984, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 4, + "raw_score": 1346.6965294546999, + "reference_score": 5942.333244755838, + "source_run": "leaderboard/runs/article_suite/20260713T104315Z/gpt-5.4-mini", + "starter_score": 2443.815838992282, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/score.json" + } + ] + }, + { + "id": "simulation_heuristics_pong_ram_v1", + "label": "Pong", + "metric": "raw_score", + "raw_metric": { + "label": "Native Pong score", + "unit": "points" + }, + "rows": [ + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "normalized_score": 100.0, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max", + "rank": 1, + "raw_score": 21.0, + "reference_score": 21.0, + "source_run": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "starter_score": -2.9999999999999996, + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", + "normalized_score": 52.916666667, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "max", + "rank": 2, + "raw_score": 9.7, + "reference_score": 21.0, + "source_run": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "starter_score": -2.9999999999999996, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "normalized_score": 45.833333333, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 3, + "raw_score": 8.0, + "reference_score": 21.0, + "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "starter_score": -2.9999999999999996, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", + "normalized_score": -75.0, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 4, + "raw_score": -21.0, + "reference_score": 21.0, + "source_run": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "starter_score": -2.9999999999999996, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/score.json" + } + ] + }, + { + "id": "simulation_heuristics_breakout_ram_v1", + "label": "Breakout RAM", + "metric": "raw_score", + "raw_metric": { + "label": "Native Breakout return", + "unit": "points" + }, + "rows": [ + { + "harness": "opencode", + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", + "normalized_score": 100.0, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "max", + "rank": 1, + "raw_score": 864.0, + "reference_score": 864.0, + "source_run": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "starter_score": 387.0, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "normalized_score": 95.597484277, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 2, + "raw_score": 843.0, + "reference_score": 864.0, + "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "starter_score": 387.0, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", + "normalized_score": 2.51572327, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 3, + "raw_score": 399.0, + "reference_score": 864.0, + "source_run": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "starter_score": 387.0, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/score.json" + }, + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "normalized_score": 0.0, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max", + "rank": 4, + "raw_score": 387.0, + "reference_score": 864.0, + "source_run": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "starter_score": 387.0, + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/score.json" + } + ] + }, + { + "id": "simulation_heuristics_breakout_rgb_v1", + "label": "Breakout RGB", + "metric": "raw_score", + "raw_metric": { + "label": "Native Breakout return", + "unit": "points" + }, + "rows": [ + { + "harness": "opencode", + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "normalized_score": 70.758122744, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 1, + "raw_score": 702.0, + "reference_score": 864.0, + "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "starter_score": 310.0, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", + "normalized_score": 70.758122744, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "max", + "rank": 1, + "raw_score": 702.0, + "reference_score": 864.0, + "source_run": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "starter_score": 310.0, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", + "normalized_score": 15.523465704, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 3, + "raw_score": 396.0, + "reference_score": 864.0, + "source_run": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "starter_score": 310.0, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/score.json" + }, + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "normalized_score": 0.0, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max", + "rank": 4, + "raw_score": 310.0, + "reference_score": 864.0, + "source_run": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "starter_score": 310.0, + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/score.json" + } + ] + }, + { + "id": "simulation_heuristics_halfcheetah_v1", + "label": "HalfCheetah", + "metric": "raw_score", + "raw_metric": { + "label": "Weighted hidden return", + "unit": "return" + }, + "rows": [ + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "normalized_score": 26.986707946, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max", + "rank": 1, + "raw_score": 6021.504892591717, + "reference_score": 11487.256451471967, + "source_run": "leaderboard/runs/article_suite/20260713T111714Z/claude-opus-4.8", + "starter_score": 4001.2886158917113, + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", + "normalized_score": 15.873072595, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "max", + "rank": 2, + "raw_score": 5189.541724894487, + "reference_score": 11487.256451471967, + "source_run": "leaderboard/runs/article_suite/20260713T084340Z/gpt-5.6-sol", + "starter_score": 4001.2886158917113, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "normalized_score": 2.73755349, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 3, + "raw_score": 4206.220989602626, + "reference_score": 11487.256451471967, + "source_run": "leaderboard/runs/article_suite/20260713T103810Z/gpt-5.5", + "starter_score": 4001.2886158917113, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", + "normalized_score": -8.167582379, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 4, + "raw_score": 3389.8660260183765, + "reference_score": 11487.256451471967, + "source_run": "leaderboard/runs/article_suite/20260713T093314Z/gpt-5.4-mini", + "starter_score": 4001.2886158917113, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/score.json" + } + ] + }, + { + "id": "simulation_heuristics_vizdoom_d1_v1", + "label": "VizDoom D1", + "metric": "raw_score", + "raw_metric": { + "label": "Native D1 mean reward", + "unit": "reward" + }, + "rows": [ + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "normalized_score": 95.055812742, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max", + "rank": 1, + "raw_score": 0.7992199431359768, + "reference_score": 0.8178199748694897, + "source_run": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "starter_score": 0.44161998964846133, + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "normalized_score": 85.199359782, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 2, + "raw_score": 0.7621399685554205, + "reference_score": 0.8178199748694897, + "source_run": "leaderboard/runs/article_suite/20260713T102218Z/gpt-5.5", + "starter_score": 0.44161998964846133, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", + "normalized_score": 49.409889366, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "max", + "rank": 3, + "raw_score": 0.62749998614192, + "reference_score": 0.8178199748694897, + "source_run": "leaderboard/runs/article_suite/20260713T101636Z/gpt-5.6-sol", + "starter_score": 0.44161998964846133, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", + "normalized_score": -161.116433137, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 4, + "raw_score": -0.16450000800192355, + "reference_score": 0.8178199748694897, + "source_run": "leaderboard/runs/article_suite/20260713T102707Z/gpt-5.4-mini", + "starter_score": 0.44161998964846133, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/score.json" + } + ] + }, + { + "id": "simulation_heuristics_vizdoom_d3_v1", + "label": "VizDoom D3", + "metric": "raw_score", + "raw_metric": { + "label": "Native D3 mean reward", + "unit": "reward" + }, + "rows": [ + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "normalized_score": 122.259358289, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max", + "rank": 1, + "raw_score": 397.7, + "reference_score": 331.1, + "source_run": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "starter_score": 31.9, + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", + "normalized_score": 26.236631016, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "max", + "rank": 2, + "raw_score": 110.39999999999998, + "reference_score": 331.1, + "source_run": "leaderboard/runs/article_suite/20260713T083940Z/gpt-5.6-sol", + "starter_score": 31.9, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "normalized_score": 5.481283422, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 3, + "raw_score": 48.3, + "reference_score": 331.1, + "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "starter_score": 31.9, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", + "normalized_score": -9.859625668, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 4, + "raw_score": 2.4, + "reference_score": 331.1, + "source_run": "leaderboard/runs/article_suite/20260713T104034Z/gpt-5.4-mini", + "starter_score": 31.9, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/score.json" + } + ] + }, + { + "id": "simulation_heuristics_atari57_v1", + "label": "Atari57", + "metric": "raw_score", + "raw_metric": { + "label": "Median best-mode HNS", + "unit": "HNS" + }, + "rows": [ + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "normalized_score": 0.0, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max", + "rank": 1, + "raw_score": 0.0, + "reference_score": 0.8283015254994576, + "source_run": "leaderboard/runs/article_suite/20260713T123222Z/claude-opus-4.8", + "starter_score": 0.0, + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", + "normalized_score": 0.0, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 1, + "raw_score": 0.0, + "reference_score": 0.8283015254994576, + "source_run": "leaderboard/runs/article_suite/20260713T111944Z/gpt-5.4-mini", + "starter_score": 0.0, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "normalized_score": 0.0, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 1, + "raw_score": 0.0, + "reference_score": 0.8283015254994576, + "source_run": "leaderboard/runs/article_suite/20260713T111604Z/gpt-5.5", + "starter_score": 0.0, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", + "normalized_score": 0.0, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "max", + "rank": 1, + "raw_score": 0.0, + "reference_score": 0.8283015254994576, + "source_run": "leaderboard/runs/article_suite/20260713T111156Z/gpt-5.6-sol", + "starter_score": 0.0, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/score.json" + } + ] + }, + { + "id": "simulation_heuristics_montezuma_v1", + "label": "Montezuma's Revenge", + "metric": "raw_score", + "raw_metric": { + "label": "Capped native return", + "unit": "points" + }, + "rows": [ + { + "harness": "opencode", + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "normalized_score": 100.0, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 1, + "raw_score": 400.0, + "reference_score": 400.0, + "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "starter_score": 0.0, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/score.json" + }, + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "normalized_score": 0.0, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max", + "rank": 2, + "raw_score": 0.0, + "reference_score": 400.0, + "source_run": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "starter_score": 0.0, + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", + "normalized_score": 0.0, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 2, + "raw_score": 0.0, + "reference_score": 400.0, + "source_run": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "starter_score": 0.0, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", + "normalized_score": 0.0, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "max", + "rank": 2, + "raw_score": 0.0, + "reference_score": 400.0, + "source_run": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "starter_score": 0.0, + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/score.json" + } + ] + }, + { + "display_metric": "positive_display_score", + "display_transform": { + "formula": "positive_display_score = final_normalized_score + 100", + "offset": 100.0, + "purpose": "plot_only", + "type": "additive_offset" + }, + "id": "final", + "label": "Final normalized score", + "metric": "final_normalized_score", + "rows": [ + { + "arithmetic_mean_normalized_score": 43.19059338577778, + "average_normalized_score": 43.19059338577778, + "final_normalized_score": 42.0019305542, + "harness": "opencode", + "median_normalized_score": 45.833333333, + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "positive_display_score": 142.0019305542, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 1 + }, + { + "arithmetic_mean_normalized_score": 39.37755364144445, + "average_normalized_score": 39.37755364144445, + "final_normalized_score": 36.7279720058, + "harness": "opencode", + "median_normalized_score": 39.203600385, + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", + "positive_display_score": 136.72797200579998, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "max", + "rank": 2 + }, + { + "arithmetic_mean_normalized_score": 39.81691467833333, + "average_normalized_score": 39.81691467833333, + "final_normalized_score": 27.218574763200003, + "harness": "opencode", + "median_normalized_score": 14.050353128, + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "positive_display_score": 127.2185747632, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max", + "rank": 3 + }, + { + "arithmetic_mean_normalized_score": -29.71822245, + "average_normalized_score": -29.71822245, + "final_normalized_score": -9.8773515774, + "harness": "opencode", + "median_normalized_score": -8.167582379, + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", + "positive_display_score": 90.1226484226, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 4 + } + ] + } + ], + "rows": [ + { + "arithmetic_mean_normalized_score": 43.19059338577778, + "average_normalized_score": 43.19059338577778, + "final_normalized_score": 42.0019305542, + "harness": "opencode", + "median_normalized_score": 45.833333333, + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 1, + "raw_task_scores": { + "simulation_heuristics_ant_v1": 1852.8533956222025, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 843.0, + "simulation_heuristics_breakout_rgb_v1": 702.0, + "simulation_heuristics_halfcheetah_v1": 4206.220989602626, + "simulation_heuristics_montezuma_v1": 400.0, + "simulation_heuristics_pong_ram_v1": 8.0, + "simulation_heuristics_vizdoom_d1_v1": 0.7621399685554205, + "simulation_heuristics_vizdoom_d3_v1": 48.3 + }, + "source_runs": { + "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T095807Z/gpt-5.5", + "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T111604Z/gpt-5.5", + "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T103810Z/gpt-5.5", + "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T102218Z/gpt-5.5", + "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5" + }, + "submission_details": { + "simulation_heuristics_ant_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/score.json", + "simulation_heuristics_atari57_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/score.json", + "simulation_heuristics_breakout_ram_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/score.json", + "simulation_heuristics_breakout_rgb_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/score.json", + "simulation_heuristics_halfcheetah_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/score.json", + "simulation_heuristics_montezuma_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/score.json", + "simulation_heuristics_pong_ram_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/score.json", + "simulation_heuristics_vizdoom_d1_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/score.json", + "simulation_heuristics_vizdoom_d3_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/score.json" + }, + "task_anchors": { + "simulation_heuristics_ant_v1": { + "reference_score": 5942.333244755838, + "starter_score": 2443.815838992282 + }, + "simulation_heuristics_atari57_v1": { + "reference_score": 0.8283015254994576, + "starter_score": 0.0 + }, + "simulation_heuristics_breakout_ram_v1": { + "reference_score": 864.0, + "starter_score": 387.0 + }, + "simulation_heuristics_breakout_rgb_v1": { + "reference_score": 864.0, + "starter_score": 310.0 + }, + "simulation_heuristics_halfcheetah_v1": { + "reference_score": 11487.256451471967, + "starter_score": 4001.2886158917113 + }, + "simulation_heuristics_montezuma_v1": { + "reference_score": 400.0, + "starter_score": 0.0 + }, + "simulation_heuristics_pong_ram_v1": { + "reference_score": 21.0, + "starter_score": -2.9999999999999996 + }, + "simulation_heuristics_vizdoom_d1_v1": { + "reference_score": 0.8178199748694897, + "starter_score": 0.44161998964846133 + }, + "simulation_heuristics_vizdoom_d3_v1": { + "reference_score": 331.1, + "starter_score": 31.9 + } + }, + "task_scores": { + "simulation_heuristics_ant_v1": -16.891796576, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 95.597484277, + "simulation_heuristics_breakout_rgb_v1": 70.758122744, + "simulation_heuristics_halfcheetah_v1": 2.73755349, + "simulation_heuristics_montezuma_v1": 100.0, + "simulation_heuristics_pong_ram_v1": 45.833333333, + "simulation_heuristics_vizdoom_d1_v1": 85.199359782, + "simulation_heuristics_vizdoom_d3_v1": 5.481283422 + } + }, + { + "arithmetic_mean_normalized_score": 39.37755364144445, + "average_normalized_score": 39.37755364144445, + "final_normalized_score": 36.7279720058, + "harness": "opencode", + "median_normalized_score": 39.203600385, + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "max", + "rank": 2, + "raw_task_scores": { + "simulation_heuristics_ant_v1": 3815.360622145591, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 864.0, + "simulation_heuristics_breakout_rgb_v1": 702.0, + "simulation_heuristics_halfcheetah_v1": 5189.541724894487, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 9.7, + "simulation_heuristics_vizdoom_d1_v1": 0.62749998614192, + "simulation_heuristics_vizdoom_d3_v1": 110.39999999999998 + }, + "source_runs": { + "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T093815Z/gpt-5.6-sol", + "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T111156Z/gpt-5.6-sol", + "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T084340Z/gpt-5.6-sol", + "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T101636Z/gpt-5.6-sol", + "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T083940Z/gpt-5.6-sol" + }, + "submission_details": { + "simulation_heuristics_ant_v1": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/score.json", + "simulation_heuristics_atari57_v1": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/score.json", + "simulation_heuristics_breakout_ram_v1": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/score.json", + "simulation_heuristics_breakout_rgb_v1": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/score.json", + "simulation_heuristics_halfcheetah_v1": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/score.json", + "simulation_heuristics_montezuma_v1": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/score.json", + "simulation_heuristics_pong_ram_v1": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/score.json", + "simulation_heuristics_vizdoom_d1_v1": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/score.json", + "simulation_heuristics_vizdoom_d3_v1": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/score.json" + }, + "task_anchors": { + "simulation_heuristics_ant_v1": { + "reference_score": 5942.333244755838, + "starter_score": 2443.815838992282 + }, + "simulation_heuristics_atari57_v1": { + "reference_score": 0.8283015254994576, + "starter_score": 0.0 + }, + "simulation_heuristics_breakout_ram_v1": { + "reference_score": 864.0, + "starter_score": 387.0 + }, + "simulation_heuristics_breakout_rgb_v1": { + "reference_score": 864.0, + "starter_score": 310.0 + }, + "simulation_heuristics_halfcheetah_v1": { + "reference_score": 11487.256451471967, + "starter_score": 4001.2886158917113 + }, + "simulation_heuristics_montezuma_v1": { + "reference_score": 400.0, + "starter_score": 0.0 + }, + "simulation_heuristics_pong_ram_v1": { + "reference_score": 21.0, + "starter_score": -2.9999999999999996 + }, + "simulation_heuristics_vizdoom_d1_v1": { + "reference_score": 0.8178199748694897, + "starter_score": 0.44161998964846133 + }, + "simulation_heuristics_vizdoom_d3_v1": { + "reference_score": 331.1, + "starter_score": 31.9 + } + }, + "task_scores": { + "simulation_heuristics_ant_v1": 39.203600385, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 100.0, + "simulation_heuristics_breakout_rgb_v1": 70.758122744, + "simulation_heuristics_halfcheetah_v1": 15.873072595, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 52.916666667, + "simulation_heuristics_vizdoom_d1_v1": 49.409889366, + "simulation_heuristics_vizdoom_d3_v1": 26.236631016 + } + }, + { + "arithmetic_mean_normalized_score": 39.81691467833333, + "average_normalized_score": 39.81691467833333, + "final_normalized_score": 27.218574763200003, + "harness": "opencode", + "median_normalized_score": 14.050353128, + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max", + "rank": 3, + "raw_task_scores": { + "simulation_heuristics_ant_v1": 2935.3698887625987, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 387.0, + "simulation_heuristics_breakout_rgb_v1": 310.0, + "simulation_heuristics_halfcheetah_v1": 6021.504892591717, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 21.0, + "simulation_heuristics_vizdoom_d1_v1": 0.7992199431359768, + "simulation_heuristics_vizdoom_d3_v1": 397.7 + }, + "source_runs": { + "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T142919Z/claude-opus-4.8", + "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T123222Z/claude-opus-4.8", + "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T111714Z/claude-opus-4.8", + "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8" + }, + "submission_details": { + "simulation_heuristics_ant_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/score.json", + "simulation_heuristics_atari57_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/score.json", + "simulation_heuristics_breakout_ram_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/score.json", + "simulation_heuristics_breakout_rgb_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/score.json", + "simulation_heuristics_halfcheetah_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/score.json", + "simulation_heuristics_montezuma_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/score.json", + "simulation_heuristics_pong_ram_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/score.json", + "simulation_heuristics_vizdoom_d1_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/score.json", + "simulation_heuristics_vizdoom_d3_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/score.json" + }, + "task_anchors": { + "simulation_heuristics_ant_v1": { + "reference_score": 5942.333244755838, + "starter_score": 2443.815838992282 + }, + "simulation_heuristics_atari57_v1": { + "reference_score": 0.8283015254994576, + "starter_score": 0.0 + }, + "simulation_heuristics_breakout_ram_v1": { + "reference_score": 864.0, + "starter_score": 387.0 + }, + "simulation_heuristics_breakout_rgb_v1": { + "reference_score": 864.0, + "starter_score": 310.0 + }, + "simulation_heuristics_halfcheetah_v1": { + "reference_score": 11487.256451471967, + "starter_score": 4001.2886158917113 + }, + "simulation_heuristics_montezuma_v1": { + "reference_score": 400.0, + "starter_score": 0.0 + }, + "simulation_heuristics_pong_ram_v1": { + "reference_score": 21.0, + "starter_score": -2.9999999999999996 + }, + "simulation_heuristics_vizdoom_d1_v1": { + "reference_score": 0.8178199748694897, + "starter_score": 0.44161998964846133 + }, + "simulation_heuristics_vizdoom_d3_v1": { + "reference_score": 331.1, + "starter_score": 31.9 + } + }, + "task_scores": { + "simulation_heuristics_ant_v1": 14.050353128, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 0.0, + "simulation_heuristics_breakout_rgb_v1": 0.0, + "simulation_heuristics_halfcheetah_v1": 26.986707946, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 100.0, + "simulation_heuristics_vizdoom_d1_v1": 95.055812742, + "simulation_heuristics_vizdoom_d3_v1": 122.259358289 + } + }, + { + "arithmetic_mean_normalized_score": -29.71822245, + "average_normalized_score": -29.71822245, + "final_normalized_score": -9.8773515774, + "harness": "opencode", + "median_normalized_score": -8.167582379, + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 4, + "raw_task_scores": { + "simulation_heuristics_ant_v1": 1346.6965294546999, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 399.0, + "simulation_heuristics_breakout_rgb_v1": 396.0, + "simulation_heuristics_halfcheetah_v1": 3389.8660260183765, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": -21.0, + "simulation_heuristics_vizdoom_d1_v1": -0.16450000800192355, + "simulation_heuristics_vizdoom_d3_v1": 2.4 + }, + "source_runs": { + "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T104315Z/gpt-5.4-mini", + "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T111944Z/gpt-5.4-mini", + "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T093314Z/gpt-5.4-mini", + "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T102707Z/gpt-5.4-mini", + "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T104034Z/gpt-5.4-mini" + }, + "submission_details": { + "simulation_heuristics_ant_v1": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/score.json", + "simulation_heuristics_atari57_v1": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/score.json", + "simulation_heuristics_breakout_ram_v1": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/score.json", + "simulation_heuristics_breakout_rgb_v1": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/score.json", + "simulation_heuristics_halfcheetah_v1": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/score.json", + "simulation_heuristics_montezuma_v1": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/score.json", + "simulation_heuristics_pong_ram_v1": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/score.json", + "simulation_heuristics_vizdoom_d1_v1": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/score.json", + "simulation_heuristics_vizdoom_d3_v1": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/score.json" + }, + "task_anchors": { + "simulation_heuristics_ant_v1": { + "reference_score": 5942.333244755838, + "starter_score": 2443.815838992282 + }, + "simulation_heuristics_atari57_v1": { + "reference_score": 0.8283015254994576, + "starter_score": 0.0 + }, + "simulation_heuristics_breakout_ram_v1": { + "reference_score": 864.0, + "starter_score": 387.0 + }, + "simulation_heuristics_breakout_rgb_v1": { + "reference_score": 864.0, + "starter_score": 310.0 + }, + "simulation_heuristics_halfcheetah_v1": { + "reference_score": 11487.256451471967, + "starter_score": 4001.2886158917113 + }, + "simulation_heuristics_montezuma_v1": { + "reference_score": 400.0, + "starter_score": 0.0 + }, + "simulation_heuristics_pong_ram_v1": { + "reference_score": 21.0, + "starter_score": -2.9999999999999996 + }, + "simulation_heuristics_vizdoom_d1_v1": { + "reference_score": 0.8178199748694897, + "starter_score": 0.44161998964846133 + }, + "simulation_heuristics_vizdoom_d3_v1": { + "reference_score": 331.1, + "starter_score": 31.9 + } + }, + "task_scores": { + "simulation_heuristics_ant_v1": -31.35954984, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 2.51572327, + "simulation_heuristics_breakout_rgb_v1": 15.523465704, + "simulation_heuristics_halfcheetah_v1": -8.167582379, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": -75.0, + "simulation_heuristics_vizdoom_d1_v1": -161.116433137, + "simulation_heuristics_vizdoom_d3_v1": -9.859625668 + } + } + ], + "source_runs": { + "claude-opus-4.8": { + "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T142919Z/claude-opus-4.8", + "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T123222Z/claude-opus-4.8", + "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T111714Z/claude-opus-4.8", + "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8" + }, + "gpt-5.4-mini": { + "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T104315Z/gpt-5.4-mini", + "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T111944Z/gpt-5.4-mini", + "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T093314Z/gpt-5.4-mini", + "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T102707Z/gpt-5.4-mini", + "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T104034Z/gpt-5.4-mini" + }, + "gpt-5.5": { + "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T095807Z/gpt-5.5", + "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T111604Z/gpt-5.5", + "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T103810Z/gpt-5.5", + "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T102218Z/gpt-5.5", + "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5" + }, + "gpt-5.6-sol": { + "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T093815Z/gpt-5.6-sol", + "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T111156Z/gpt-5.6-sol", + "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T084340Z/gpt-5.6-sol", + "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T101636Z/gpt-5.6-sol", + "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T083940Z/gpt-5.6-sol" + } + }, + "task_count": 9, + "task_digest_compatibility": { + "simulation_heuristics_ant_v1": { + "sha256:9da0e00147cf66804e6c2fc17869606bea8f260850c5447989a8880eef940d45": "Score-equivalent to the current task. The later change adds only a CI smoke config and documentation; the publication scoring config is unchanged.", + "sha256:bbb533da0cb86459f4d49dee667e6c73ac54c0188bc40e54e911d50ef3c3bc38": "Score-equivalent to the current task. Later changes add a fail-closed verifier timeout plus a CI-only smoke config and documentation; the publication scoring config is unchanged." + }, + "simulation_heuristics_halfcheetah_v1": { + "sha256:80c439f53e4ab964f9d7443cd7fb8f25cf6645a0bc288b0496b871c1800ebe78": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged." + }, + "simulation_heuristics_vizdoom_d1_v1": { + "sha256:b3431b238bec4e66af6189d30bcd15d5cd227144dfe8bf0dd1011aa5416c1436": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged." + }, + "simulation_heuristics_vizdoom_d3_v1": { + "sha256:fcc6ed05673b7981aa17d60ca4e0d355fbcb57a6cbca293c0fa201ab97cdd081": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged." + } + }, + "tasks": [ + "simulation_heuristics_ant_v1", + "simulation_heuristics_pong_ram_v1", + "simulation_heuristics_breakout_ram_v1", + "simulation_heuristics_breakout_rgb_v1", + "simulation_heuristics_halfcheetah_v1", + "simulation_heuristics_vizdoom_d1_v1", + "simulation_heuristics_vizdoom_d3_v1", + "simulation_heuristics_atari57_v1", + "simulation_heuristics_montezuma_v1" + ] +} diff --git a/website/index.html b/website/index.html index 0aac83b..60f799b 100644 --- a/website/index.html +++ b/website/index.html @@ -225,7 +225,72 @@ .env-root{margin-bottom:1rem} .env-grid{grid-template-columns:1fr;margin-top:0;gap:2.4rem} .env-mid{position:static;transform:none;justify-self:center;order:2;margin:-1.6rem 0} - .env-card:first-child{order:1}.env-card:last-child{order:3} +.env-card:first-child{order:1}.env-card:last-child{order:3} +} + +/* leaderboard results */ +.results-stack{display:grid;gap:clamp(1.5rem,3vw,2.4rem);margin-top:clamp(1.8rem,3vw,2.5rem)} +.result-panel{background:var(--card);border:1px solid var(--line);border-radius:8px; + padding:clamp(1rem,2.2vw,1.5rem);overflow:hidden} +.result-panel>h3{font-size:clamp(1.15rem,2vw,1.45rem);margin-bottom:.35rem} +.result-note{font-size:.875rem;line-height:1.5;color:var(--ink-soft);margin-bottom:1.2rem} +.inference-panel{margin-top:1.4rem;padding:1rem 1.1rem;border:1px solid var(--line); + border-radius:8px;background:var(--card)} +.inference-head{display:flex;flex-wrap:wrap;align-items:baseline;justify-content:space-between;gap:.4rem 1rem} +.inference-head h3{font-size:1rem;font-weight:600} +.inference-head p{font-size:.75rem;color:var(--ink-soft)} +.inference-grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(11rem,1fr));gap:.7rem;margin-top:.8rem} +.inference-card{padding:.72rem .8rem;border:1px solid var(--line);border-radius:5px;background:var(--paper)} +.inference-model{display:block;font-size:.82rem;font-weight:600;color:var(--forest-ink)} +.inference-route{display:block;margin-top:.12rem;font-size:.64rem;line-height:1.35;color:var(--ink-soft); + overflow-wrap:anywhere} +.inference-effort{display:inline-flex;margin-top:.45rem;padding:.12rem .42rem;border:1px solid var(--line-strong); + border-radius:999px;font-size:.65rem;font-weight:600;color:var(--forest-ink)} +.leaderboard-grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(17.5rem,1fr));gap:1rem} +.chart-card{min-width:0;border:1px solid var(--line);border-radius:6px;padding:.9rem; + background:var(--paper);overflow:hidden} +.chart-card h4{font-size:.9875rem;font-weight:600;letter-spacing:-.01em} +.chart-metric{display:block;margin-top:.12rem;font-size:.7rem;color:var(--ink-soft);line-height:1.35} +.chart-body{display:grid;grid-template-columns:5.2rem minmax(0,1fr) 3.7rem; + gap:.35rem;align-items:stretch;min-width:0;margin-top:.8rem} +.chart-labels,.chart-values,.chart-bars{display:grid;grid-template-rows:repeat(4,16px);gap:.48rem} +.chart-model{font-size:.69rem;line-height:16px;color:var(--ink);overflow-wrap:anywhere} +.chart-value{font-size:.72rem;line-height:16px;font-variant-numeric:tabular-nums;white-space:nowrap;text-align:right} +.chart-plot{position:relative;min-width:0;background: + repeating-linear-gradient(90deg,transparent 0,transparent calc(25% - 1px),var(--line) 25%)} +.chart-zero,.chart-anchor{position:absolute;top:0;bottom:0;width:1px;pointer-events:none} +.chart-zero{background:var(--forest-ink);opacity:.55} +.chart-anchor{border-left:1px dashed var(--gold)} +.chart-anchor.is-starter{border-left-style:dotted} +.chart-bars{position:relative;z-index:1} +.chart-bar-row{position:relative;height:16px} +.chart-bar{position:absolute;top:2px;height:12px;min-width:2px;border-radius:1px;background:#555} +.model-gpt-5-5{background:#111}.model-gpt-5-6-sol{background:#4b4b4b} +.model-claude-opus-4-8{background:#858585}.model-gpt-5-4-mini{background:#b7b7b7} +.chart-anchors{display:flex;justify-content:space-between;gap:.6rem;margin-top:.7rem; + font-size:.64rem;line-height:1.25;color:var(--ink-soft);font-variant-numeric:tabular-nums} +.final-board{display:grid;gap:.75rem;margin-top:1rem} +.final-row{display:grid;grid-template-columns:minmax(8.8rem,1fr) minmax(12rem,3fr) minmax(8rem,auto); + gap:.8rem;align-items:center} +.final-model{font-size:.9rem;font-weight:500} +.final-plot{position:relative;height:30px;background: + repeating-linear-gradient(90deg,transparent 0,transparent calc(20% - 1px),var(--line) 20%)} +.final-plot .chart-anchor{top:-4px;bottom:-4px} +.final-bar{position:absolute;left:0;top:4px;height:22px;border-radius:1px;background:#555} +.final-value{font-size:.88rem;font-weight:600;font-variant-numeric:tabular-nums;white-space:nowrap;text-align:right} +.final-value small{display:block;font-size:.68rem;font-weight:400;color:var(--ink-soft)} +.chart-error{padding:1rem;border:1px dashed var(--line-strong);border-radius:6px;color:var(--ink-soft)} +.result-actions{display:flex;flex-wrap:wrap;gap:.7rem 1.2rem;margin-top:1.2rem;font-size:.9rem} +.result-actions a{color:var(--forest);text-decoration:underline;text-underline-offset:3px; + text-decoration-color:color-mix(in srgb,var(--forest) 35%,transparent)} +.result-actions a:hover{text-decoration-color:var(--forest)} +@media(max-width:680px){ + .leaderboard-grid{grid-template-columns:1fr} + .chart-body{grid-template-columns:5.2rem minmax(0,1fr) 3.7rem} + .final-row{grid-template-columns:1fr} + .final-plot{height:26px}.final-bar{height:18px} + .final-value{text-align:left;display:flex;align-items:baseline;gap:.6rem} + .final-value small{display:inline} } /* research list */ @@ -275,6 +340,7 @@
@@ -410,6 +476,46 @@

Real-world operation

+ +
+
+

Results

+
+

Four coding models, nine article-derived tasks, one robust cross-task score.

+

The task panels show each environment's native raw score. Cross-task ranking first normalizes each task so the public starter scores 0 and the trusted article-level reference scores 100, then uses the interquartile mean: remove the lowest two and highest two task scores and average the middle five.

+ +
+
+

Leaderboards

+

Native raw score for each environment. Dotted markers show the starter; dashed markers show the article reference.

+
+

Loading task leaderboards…

+
+
+
+

Final normalized score

+

Positive display index = IQM + 100. Ranking and score gaps are unchanged; the dotted marker is the starter-level aggregate.

+
+

Loading final leaderboard…

+
+
+
+
+ Detailed leaderboard artifacts ↗ + Scoring methodology ↗ +
+
+
+
+
@@ -643,5 +749,126 @@

Research background

})(); + +