diff --git a/README.md b/README.md index 4aed901..a033528 100644 --- a/README.md +++ b/README.md @@ -101,7 +101,7 @@ uv run python scripts/run_article_suite.py \ --model gpt-5.6-sol ``` -Run all four canonical models and rebuild the aggregate leaderboard: +Run all four canonical models and rebuild the 10 leaderboard artifacts: ```bash uv run python scripts/run_article_suite.py \ @@ -124,32 +124,15 @@ The first OpenCode sweep across all nine article-derived tasks: | 3 | GPT-5.6 Sol | 39.38 | | 4 | GPT-5.4 Mini | -29.72 | -See [`leaderboard/ARTICLE_SUITE.md`](leaderboard/ARTICLE_SUITE.md) for every -per-task score and `leaderboard/article_suite.json` for the machine-readable -leaderboard. +See [`leaderboard/ARTICLE_SUITE.md`](leaderboard/ARTICLE_SUITE.md) for the nine +task-specific leaderboards followed by the final average leaderboard. +`leaderboard/article_suite.json` contains the same 10-board structure in +machine-readable form. Scores are unbounded normalized values: `0` matches the public starter and `100` matches the trusted article-level reference. Negative scores are genuine regressions; scores above `100` exceed the reference. -## Legacy Ant-Only Leaderboard - -![GenesisBench Simulation Heuristics Ant v1 leaderboard](leaderboard/simulation_heuristics_ant_v1_leaderboard.png) - -The table below is the historical Ant-only OpenHands sweep. It remains for -provenance; new GenesisBench leaderboard runs use OpenCode and the nine-task -article suite. - -| Rank | Model | Hidden-suite score | -| ---: | --- | ---: | -| 1 | GPT-5.6 Sol | 3417.86 | -| 2 | GPT-5.5 | 2382.23 | -| 3 | GPT-5.4 Mini | 2369.61 | -| 4 | Claude Opus 4.8 | 2235.71 | - -These are single-run research results, not multi-trial estimates of model -quality. See `leaderboard/REPORT.md` for setup details and limitations. - ## Contribute a Task Create a scaffold: diff --git a/docs/learning-beyond-gradients-suite.md b/docs/learning-beyond-gradients-suite.md index c0e818f..4261da0 100644 --- a/docs/learning-beyond-gradients-suite.md +++ b/docs/learning-beyond-gradients-suite.md @@ -61,15 +61,25 @@ chat-completions gateway cannot faithfully transform Azure GPT-5.6 Sol tool calls. BenchFlow continues to own task staging, Daytona/Docker isolation, ACP trajectory capture, timing, and verifier execution. -## Aggregate score +## Leaderboard outputs -The article-suite leaderboard reports every normalized task score and their -unweighted arithmetic mean: +The offline article-suite report contains 10 independent leaderboards in a +fixed order: + +1. one leaderboard for each of the nine article-derived tasks; +2. the final averaged leaderboard. + +The final score is the unweighted arithmetic mean: ```text average = sum(nine normalized task scores) / 9 ``` +The repository README intentionally shows only the final averaged leaderboard. +Detailed task rankings and score-artifact links live in +`leaderboard/ARTICLE_SUITE.md`; the matching machine-readable structure lives +in `leaderboard/article_suite.json`. + The runner and resumable leaderboard builder live in: - `scripts/run_article_suite.py` diff --git a/experiments/article_suite/README.md b/experiments/article_suite/README.md index 8a908d2..0867c37 100644 --- a/experiments/article_suite/README.md +++ b/experiments/article_suite/README.md @@ -77,12 +77,19 @@ is disabled for this suite because BenchFlow 0.6.5's chat-completions gateway cannot faithfully transform GPT-5.6 Sol tool calls; the run metadata records this limitation explicitly. -## Build the aggregate leaderboard +## Build all 10 leaderboards ```bash uv run python scripts/build_article_suite_leaderboard.py ``` -The aggregate score is the arithmetic mean of the nine normalized task scores. +The builder writes: + +- `leaderboard/ARTICLE_SUITE.md`: nine task-specific leaderboards followed by + the final average leaderboard; +- `leaderboard/article_suite.json`: the same 10 leaderboards plus the + model-centric score records used for reproducibility. + +The final aggregate is the arithmetic mean of the nine normalized task scores. Each task maps its starter policy to `0` and its trusted article-level reference -to `100`. +to `100`. The top-level repository README shows only this final aggregate. diff --git a/leaderboard/ARTICLE_SUITE.md b/leaderboard/ARTICLE_SUITE.md index 08187da..acb3749 100644 --- a/leaderboard/ARTICLE_SUITE.md +++ b/leaderboard/ARTICLE_SUITE.md @@ -1,10 +1,115 @@ # GenesisBench Learning Beyond Gradients Article Suite -The aggregate score is the arithmetic mean of nine normalized task scores. Starter policies map to 0 and trusted article-level references map to 100. - -| Rank | Model | Harness | Effort | Average | Ant | Pong | Breakout RAM | Breakout RGB | HalfCheetah | Doom D1 | Doom D3 | Atari57 | Montezuma | -| ---: | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| 1 | GPT-5.5 | opencode | xhigh | 43.19 | -16.89 | 45.83 | 95.60 | 70.76 | 2.74 | 85.20 | 5.48 | 0.00 | 100.00 | -| 2 | Claude Opus 4.8 | opencode | max | 39.82 | 14.05 | 100.00 | 0.00 | 0.00 | 26.99 | 95.06 | 122.26 | 0.00 | 0.00 | -| 3 | GPT-5.6 Sol | opencode | max | 39.38 | 39.20 | 52.92 | 100.00 | 70.76 | 15.87 | 49.41 | 26.24 | 0.00 | 0.00 | -| 4 | GPT-5.4 Mini | opencode | xhigh | -29.72 | -31.36 | -75.00 | 2.52 | 15.52 | -8.17 | -161.12 | -9.86 | 0.00 | 0.00 | +This offline report contains 10 independent leaderboards: one for each of the nine article-derived tasks, followed by the final nine-task average. + +Task scores are unbounded normalized values. The public starter maps to 0 and the trusted article-level reference maps to 100. + +## 1. Ant + +Task: `simulation_heuristics_ant_v1` + +| Rank | Model | Harness | Effort | Normalized score | Score details | +| ---: | --- | --- | --- | ---: | --- | +| 1 | GPT-5.6 Sol | opencode | max | 39.20 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/score.json) | +| 2 | Claude Opus 4.8 | opencode | max | 14.05 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/score.json) | +| 3 | GPT-5.5 | opencode | xhigh | -16.89 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/score.json) | +| 4 | GPT-5.4 Mini | opencode | xhigh | -31.36 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/score.json) | + +## 2. Pong + +Task: `simulation_heuristics_pong_ram_v1` + +| Rank | Model | Harness | Effort | Normalized score | Score details | +| ---: | --- | --- | --- | ---: | --- | +| 1 | Claude Opus 4.8 | opencode | max | 100.00 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/score.json) | +| 2 | GPT-5.6 Sol | opencode | max | 52.92 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/score.json) | +| 3 | GPT-5.5 | opencode | xhigh | 45.83 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/score.json) | +| 4 | GPT-5.4 Mini | opencode | xhigh | -75.00 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/score.json) | + +## 3. Breakout RAM + +Task: `simulation_heuristics_breakout_ram_v1` + +| Rank | Model | Harness | Effort | Normalized score | Score details | +| ---: | --- | --- | --- | ---: | --- | +| 1 | GPT-5.6 Sol | opencode | max | 100.00 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/score.json) | +| 2 | GPT-5.5 | opencode | xhigh | 95.60 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/score.json) | +| 3 | GPT-5.4 Mini | opencode | xhigh | 2.52 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/score.json) | +| 4 | Claude Opus 4.8 | opencode | max | 0.00 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/score.json) | + +## 4. Breakout RGB + +Task: `simulation_heuristics_breakout_rgb_v1` + +| Rank | Model | Harness | Effort | Normalized score | Score details | +| ---: | --- | --- | --- | ---: | --- | +| 1 | GPT-5.5 | opencode | xhigh | 70.76 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/score.json) | +| 1 | GPT-5.6 Sol | opencode | max | 70.76 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/score.json) | +| 3 | GPT-5.4 Mini | opencode | xhigh | 15.52 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/score.json) | +| 4 | Claude Opus 4.8 | opencode | max | 0.00 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/score.json) | + +## 5. HalfCheetah + +Task: `simulation_heuristics_halfcheetah_v1` + +| Rank | Model | Harness | Effort | Normalized score | Score details | +| ---: | --- | --- | --- | ---: | --- | +| 1 | Claude Opus 4.8 | opencode | max | 26.99 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/score.json) | +| 2 | GPT-5.6 Sol | opencode | max | 15.87 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/score.json) | +| 3 | GPT-5.5 | opencode | xhigh | 2.74 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/score.json) | +| 4 | GPT-5.4 Mini | opencode | xhigh | -8.17 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/score.json) | + +## 6. VizDoom D1 + +Task: `simulation_heuristics_vizdoom_d1_v1` + +| Rank | Model | Harness | Effort | Normalized score | Score details | +| ---: | --- | --- | --- | ---: | --- | +| 1 | Claude Opus 4.8 | opencode | max | 95.06 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/score.json) | +| 2 | GPT-5.5 | opencode | xhigh | 85.20 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/score.json) | +| 3 | GPT-5.6 Sol | opencode | max | 49.41 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/score.json) | +| 4 | GPT-5.4 Mini | opencode | xhigh | -161.12 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/score.json) | + +## 7. VizDoom D3 + +Task: `simulation_heuristics_vizdoom_d3_v1` + +| Rank | Model | Harness | Effort | Normalized score | Score details | +| ---: | --- | --- | --- | ---: | --- | +| 1 | Claude Opus 4.8 | opencode | max | 122.26 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/score.json) | +| 2 | GPT-5.6 Sol | opencode | max | 26.24 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/score.json) | +| 3 | GPT-5.5 | opencode | xhigh | 5.48 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/score.json) | +| 4 | GPT-5.4 Mini | opencode | xhigh | -9.86 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/score.json) | + +## 8. Atari57 + +Task: `simulation_heuristics_atari57_v1` + +| Rank | Model | Harness | Effort | Normalized score | Score details | +| ---: | --- | --- | --- | ---: | --- | +| 1 | Claude Opus 4.8 | opencode | max | 0.00 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/score.json) | +| 1 | GPT-5.4 Mini | opencode | xhigh | 0.00 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/score.json) | +| 1 | GPT-5.5 | opencode | xhigh | 0.00 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/score.json) | +| 1 | GPT-5.6 Sol | opencode | max | 0.00 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/score.json) | + +## 9. Montezuma's Revenge + +Task: `simulation_heuristics_montezuma_v1` + +| Rank | Model | Harness | Effort | Normalized score | Score details | +| ---: | --- | --- | --- | ---: | --- | +| 1 | GPT-5.5 | opencode | xhigh | 100.00 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/score.json) | +| 2 | Claude Opus 4.8 | opencode | max | 0.00 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/score.json) | +| 2 | GPT-5.4 Mini | opencode | xhigh | 0.00 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/score.json) | +| 2 | GPT-5.6 Sol | opencode | max | 0.00 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/score.json) | + +## 10. Nine-task average + +Arithmetic mean of the nine normalized task scores. + +| Rank | Model | Harness | Effort | Nine-task average | +| ---: | --- | --- | --- | ---: | +| 1 | GPT-5.5 | opencode | xhigh | 43.19 | +| 2 | Claude Opus 4.8 | opencode | max | 39.82 | +| 3 | GPT-5.6 Sol | opencode | max | 39.38 | +| 4 | GPT-5.4 Mini | opencode | xhigh | -29.72 | diff --git a/leaderboard/README.md b/leaderboard/README.md index 00da15b..b085e94 100644 --- a/leaderboard/README.md +++ b/leaderboard/README.md @@ -9,7 +9,8 @@ | 3 | GPT-5.6 Sol | 39.38 | | 4 | GPT-5.4 Mini | -29.72 | -See [`ARTICLE_SUITE.md`](ARTICLE_SUITE.md) for all nine task columns. +See [`ARTICLE_SUITE.md`](ARTICLE_SUITE.md) for 10 independent leaderboards: +nine task-specific rankings followed by the final average ranking. ## Legacy Simulation Heuristics Ant v1 diff --git a/leaderboard/article_suite.json b/leaderboard/article_suite.json index 0148ab5..4916a6a 100644 --- a/leaderboard/article_suite.json +++ b/leaderboard/article_suite.json @@ -2,6 +2,471 @@ "aggregation": "arithmetic_mean_of_normalized_task_scores", "benchmark": "learning_beyond_gradients_article_suite", "generated_at": "2026-07-13T15:34:22.260689+00:00", + "leaderboard_count": 10, + "leaderboards": [ + { + "id": "simulation_heuristics_ant_v1", + "label": "Ant", + "metric": "normalized_score", + "rows": [ + { + "harness": "opencode", + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "normalized_score": 39.203600385, + "provider_reasoning_effort": "max", + "rank": 1, + "source_run": "leaderboard/runs/article_suite/20260713T093815Z/gpt-5.6-sol", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/score.json" + }, + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "normalized_score": 14.050353128, + "provider_reasoning_effort": "max", + "rank": 2, + "source_run": "leaderboard/runs/article_suite/20260713T142919Z/claude-opus-4.8", + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "normalized_score": -16.891796576, + "provider_reasoning_effort": "xhigh", + "rank": 3, + "source_run": "leaderboard/runs/article_suite/20260713T095807Z/gpt-5.5", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "normalized_score": -31.35954984, + "provider_reasoning_effort": "xhigh", + "rank": 4, + "source_run": "leaderboard/runs/article_suite/20260713T104315Z/gpt-5.4-mini", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/score.json" + } + ] + }, + { + "id": "simulation_heuristics_pong_ram_v1", + "label": "Pong", + "metric": "normalized_score", + "rows": [ + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "normalized_score": 100.0, + "provider_reasoning_effort": "max", + "rank": 1, + "source_run": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "normalized_score": 52.916666667, + "provider_reasoning_effort": "max", + "rank": 2, + "source_run": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "normalized_score": 45.833333333, + "provider_reasoning_effort": "xhigh", + "rank": 3, + "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "normalized_score": -75.0, + "provider_reasoning_effort": "xhigh", + "rank": 4, + "source_run": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/score.json" + } + ] + }, + { + "id": "simulation_heuristics_breakout_ram_v1", + "label": "Breakout RAM", + "metric": "normalized_score", + "rows": [ + { + "harness": "opencode", + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "normalized_score": 100.0, + "provider_reasoning_effort": "max", + "rank": 1, + "source_run": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "normalized_score": 95.597484277, + "provider_reasoning_effort": "xhigh", + "rank": 2, + "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "normalized_score": 2.51572327, + "provider_reasoning_effort": "xhigh", + "rank": 3, + "source_run": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/score.json" + }, + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "normalized_score": 0.0, + "provider_reasoning_effort": "max", + "rank": 4, + "source_run": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/score.json" + } + ] + }, + { + "id": "simulation_heuristics_breakout_rgb_v1", + "label": "Breakout RGB", + "metric": "normalized_score", + "rows": [ + { + "harness": "opencode", + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "normalized_score": 70.758122744, + "provider_reasoning_effort": "xhigh", + "rank": 1, + "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "normalized_score": 70.758122744, + "provider_reasoning_effort": "max", + "rank": 1, + "source_run": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "normalized_score": 15.523465704, + "provider_reasoning_effort": "xhigh", + "rank": 3, + "source_run": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/score.json" + }, + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "normalized_score": 0.0, + "provider_reasoning_effort": "max", + "rank": 4, + "source_run": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/score.json" + } + ] + }, + { + "id": "simulation_heuristics_halfcheetah_v1", + "label": "HalfCheetah", + "metric": "normalized_score", + "rows": [ + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "normalized_score": 26.986707946, + "provider_reasoning_effort": "max", + "rank": 1, + "source_run": "leaderboard/runs/article_suite/20260713T111714Z/claude-opus-4.8", + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "normalized_score": 15.873072595, + "provider_reasoning_effort": "max", + "rank": 2, + "source_run": "leaderboard/runs/article_suite/20260713T084340Z/gpt-5.6-sol", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "normalized_score": 2.73755349, + "provider_reasoning_effort": "xhigh", + "rank": 3, + "source_run": "leaderboard/runs/article_suite/20260713T103810Z/gpt-5.5", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "normalized_score": -8.167582379, + "provider_reasoning_effort": "xhigh", + "rank": 4, + "source_run": "leaderboard/runs/article_suite/20260713T093314Z/gpt-5.4-mini", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/score.json" + } + ] + }, + { + "id": "simulation_heuristics_vizdoom_d1_v1", + "label": "VizDoom D1", + "metric": "normalized_score", + "rows": [ + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "normalized_score": 95.055812742, + "provider_reasoning_effort": "max", + "rank": 1, + "source_run": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "normalized_score": 85.199359782, + "provider_reasoning_effort": "xhigh", + "rank": 2, + "source_run": "leaderboard/runs/article_suite/20260713T102218Z/gpt-5.5", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "normalized_score": 49.409889366, + "provider_reasoning_effort": "max", + "rank": 3, + "source_run": "leaderboard/runs/article_suite/20260713T101636Z/gpt-5.6-sol", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "normalized_score": -161.116433137, + "provider_reasoning_effort": "xhigh", + "rank": 4, + "source_run": "leaderboard/runs/article_suite/20260713T102707Z/gpt-5.4-mini", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/score.json" + } + ] + }, + { + "id": "simulation_heuristics_vizdoom_d3_v1", + "label": "VizDoom D3", + "metric": "normalized_score", + "rows": [ + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "normalized_score": 122.259358289, + "provider_reasoning_effort": "max", + "rank": 1, + "source_run": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "normalized_score": 26.236631016, + "provider_reasoning_effort": "max", + "rank": 2, + "source_run": "leaderboard/runs/article_suite/20260713T083940Z/gpt-5.6-sol", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "normalized_score": 5.481283422, + "provider_reasoning_effort": "xhigh", + "rank": 3, + "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "normalized_score": -9.859625668, + "provider_reasoning_effort": "xhigh", + "rank": 4, + "source_run": "leaderboard/runs/article_suite/20260713T104034Z/gpt-5.4-mini", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/score.json" + } + ] + }, + { + "id": "simulation_heuristics_atari57_v1", + "label": "Atari57", + "metric": "normalized_score", + "rows": [ + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "normalized_score": 0.0, + "provider_reasoning_effort": "max", + "rank": 1, + "source_run": "leaderboard/runs/article_suite/20260713T123222Z/claude-opus-4.8", + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "normalized_score": 0.0, + "provider_reasoning_effort": "xhigh", + "rank": 1, + "source_run": "leaderboard/runs/article_suite/20260713T111944Z/gpt-5.4-mini", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "normalized_score": 0.0, + "provider_reasoning_effort": "xhigh", + "rank": 1, + "source_run": "leaderboard/runs/article_suite/20260713T111604Z/gpt-5.5", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "normalized_score": 0.0, + "provider_reasoning_effort": "max", + "rank": 1, + "source_run": "leaderboard/runs/article_suite/20260713T111156Z/gpt-5.6-sol", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/score.json" + } + ] + }, + { + "id": "simulation_heuristics_montezuma_v1", + "label": "Montezuma's Revenge", + "metric": "normalized_score", + "rows": [ + { + "harness": "opencode", + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "normalized_score": 100.0, + "provider_reasoning_effort": "xhigh", + "rank": 1, + "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/score.json" + }, + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "normalized_score": 0.0, + "provider_reasoning_effort": "max", + "rank": 2, + "source_run": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "normalized_score": 0.0, + "provider_reasoning_effort": "xhigh", + "rank": 2, + "source_run": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/score.json" + }, + { + "harness": "opencode", + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "normalized_score": 0.0, + "provider_reasoning_effort": "max", + "rank": 2, + "source_run": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/score.json" + } + ] + }, + { + "id": "average", + "label": "Nine-task average", + "metric": "average_normalized_score", + "rows": [ + { + "average_normalized_score": 43.19059338577778, + "harness": "opencode", + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "provider_reasoning_effort": "xhigh", + "rank": 1 + }, + { + "average_normalized_score": 39.81691467833333, + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "provider_reasoning_effort": "max", + "rank": 2 + }, + { + "average_normalized_score": 39.37755364144445, + "harness": "opencode", + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "provider_reasoning_effort": "max", + "rank": 3 + }, + { + "average_normalized_score": -29.71822245, + "harness": "opencode", + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "provider_reasoning_effort": "xhigh", + "rank": 4 + } + ] + } + ], "rows": [ { "average_normalized_score": 43.19059338577778, @@ -217,7 +682,17 @@ "task_count": 9, "task_digest_compatibility": { "simulation_heuristics_ant_v1": { - "sha256:bbb533da0cb86459f4d49dee667e6c73ac54c0188bc40e54e911d50ef3c3bc38": "Score-equivalent to the current task. The only later change adds a fail-closed internal timeout for candidates whose verifier would otherwise exceed BenchFlow's deadline." + "sha256:9da0e00147cf66804e6c2fc17869606bea8f260850c5447989a8880eef940d45": "Score-equivalent to the current task. The later change adds only a CI smoke config and documentation; the publication scoring config is unchanged.", + "sha256:bbb533da0cb86459f4d49dee667e6c73ac54c0188bc40e54e911d50ef3c3bc38": "Score-equivalent to the current task. Later changes add a fail-closed verifier timeout plus a CI-only smoke config and documentation; the publication scoring config is unchanged." + }, + "simulation_heuristics_halfcheetah_v1": { + "sha256:80c439f53e4ab964f9d7443cd7fb8f25cf6645a0bc288b0496b871c1800ebe78": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged." + }, + "simulation_heuristics_vizdoom_d1_v1": { + "sha256:b3431b238bec4e66af6189d30bcd15d5cd227144dfe8bf0dd1011aa5416c1436": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged." + }, + "simulation_heuristics_vizdoom_d3_v1": { + "sha256:fcc6ed05673b7981aa17d60ca4e0d355fbcb57a6cbca293c0fa201ab97cdd081": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged." } }, "tasks": [ diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/metadata.json index a4719be..e020be1 100644 --- a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/metadata.json +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/metadata.json @@ -1,7 +1,7 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:9da0e00147cf66804e6c2fc17869606bea8f260850c5447989a8880eef940d45", - "digest_compatibility_note": null, + "current_task_digest": "sha256:5cf3805d59b03cb9d0bebb6f154eab38beb655918b8321500d692f5010a00cb2", + "digest_compatibility_note": "Score-equivalent to the current task. The later change adds only a CI smoke config and documentation; the publication scoring config is unchanged.", "finished_at": 1783956862.2606888, "harness": "opencode", "model": { diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/metadata.json index 0a6c45a..2857988 100644 --- a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/metadata.json +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/metadata.json @@ -1,7 +1,7 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:80c439f53e4ab964f9d7443cd7fb8f25cf6645a0bc288b0496b871c1800ebe78", - "digest_compatibility_note": null, + "current_task_digest": "sha256:c7e3b5e466431f4908c1e96cc7de80191f7a7c5ab5a63b3eea4d708f7f447fbf", + "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", "finished_at": 1783947693.1667001, "harness": "opencode", "model": { diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/metadata.json index 601ec25..09b40a6 100644 --- a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/metadata.json +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/metadata.json @@ -1,7 +1,7 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:b3431b238bec4e66af6189d30bcd15d5cd227144dfe8bf0dd1011aa5416c1436", - "digest_compatibility_note": null, + "current_task_digest": "sha256:316e198d8fa71e705c8b7fa54f41faf64e1f3c0ba5c4c92b8116b4bf78c5712b", + "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", "finished_at": 1783945917.227471, "harness": "opencode", "model": { diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/metadata.json index 7c52bcb..69b8b27 100644 --- a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/metadata.json +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/metadata.json @@ -1,7 +1,7 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:fcc6ed05673b7981aa17d60ca4e0d355fbcb57a6cbca293c0fa201ab97cdd081", - "digest_compatibility_note": null, + "current_task_digest": "sha256:29434bdcdce19a6e0cb24d77c800d172d49a9529875ffd4201cf928e242f4e62", + "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", "finished_at": 1783945917.227471, "harness": "opencode", "model": { diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/metadata.json index 4b26367..bdcc009 100644 --- a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/metadata.json @@ -1,7 +1,7 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:9da0e00147cf66804e6c2fc17869606bea8f260850c5447989a8880eef940d45", - "digest_compatibility_note": "Score-equivalent to the current task. The only later change adds a fail-closed internal timeout for candidates whose verifier would otherwise exceed BenchFlow's deadline.", + "current_task_digest": "sha256:5cf3805d59b03cb9d0bebb6f154eab38beb655918b8321500d692f5010a00cb2", + "digest_compatibility_note": "Score-equivalent to the current task. Later changes add a fail-closed verifier timeout plus a CI-only smoke config and documentation; the publication scoring config is unchanged.", "finished_at": 1783941566.485236, "harness": "opencode", "model": { diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/metadata.json index 50495ed..f000def 100644 --- a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/metadata.json @@ -1,7 +1,7 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:80c439f53e4ab964f9d7443cd7fb8f25cf6645a0bc288b0496b871c1800ebe78", - "digest_compatibility_note": null, + "current_task_digest": "sha256:c7e3b5e466431f4908c1e96cc7de80191f7a7c5ab5a63b3eea4d708f7f447fbf", + "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", "finished_at": 1783937429.008938, "harness": "opencode", "model": { diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/metadata.json index a34d1f2..d00892d 100644 --- a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/metadata.json @@ -1,7 +1,7 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:b3431b238bec4e66af6189d30bcd15d5cd227144dfe8bf0dd1011aa5416c1436", - "digest_compatibility_note": null, + "current_task_digest": "sha256:316e198d8fa71e705c8b7fa54f41faf64e1f3c0ba5c4c92b8116b4bf78c5712b", + "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", "finished_at": 1783938535.150049, "harness": "opencode", "model": { diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/metadata.json index 5cb7fd2..cf37cb7 100644 --- a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/metadata.json @@ -1,7 +1,7 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:fcc6ed05673b7981aa17d60ca4e0d355fbcb57a6cbca293c0fa201ab97cdd081", - "digest_compatibility_note": null, + "current_task_digest": "sha256:29434bdcdce19a6e0cb24d77c800d172d49a9529875ffd4201cf928e242f4e62", + "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", "finished_at": 1783939368.238364, "harness": "opencode", "model": { diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/metadata.json index d79bc52..5a0d184 100644 --- a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/metadata.json @@ -1,7 +1,7 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:9da0e00147cf66804e6c2fc17869606bea8f260850c5447989a8880eef940d45", - "digest_compatibility_note": "Score-equivalent to the current task. The only later change adds a fail-closed internal timeout for candidates whose verifier would otherwise exceed BenchFlow's deadline.", + "current_task_digest": "sha256:5cf3805d59b03cb9d0bebb6f154eab38beb655918b8321500d692f5010a00cb2", + "digest_compatibility_note": "Score-equivalent to the current task. Later changes add a fail-closed verifier timeout plus a CI-only smoke config and documentation; the publication scoring config is unchanged.", "finished_at": 1783939074.870018, "harness": "opencode", "model": { diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/metadata.json index 632f109..5affbbc 100644 --- a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/metadata.json @@ -1,7 +1,7 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:80c439f53e4ab964f9d7443cd7fb8f25cf6645a0bc288b0496b871c1800ebe78", - "digest_compatibility_note": null, + "current_task_digest": "sha256:c7e3b5e466431f4908c1e96cc7de80191f7a7c5ab5a63b3eea4d708f7f447fbf", + "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", "finished_at": 1783941357.9386308, "harness": "opencode", "model": { diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/metadata.json index d436810..7015a52 100644 --- a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/metadata.json @@ -1,7 +1,7 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:b3431b238bec4e66af6189d30bcd15d5cd227144dfe8bf0dd1011aa5416c1436", - "digest_compatibility_note": null, + "current_task_digest": "sha256:316e198d8fa71e705c8b7fa54f41faf64e1f3c0ba5c4c92b8116b4bf78c5712b", + "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", "finished_at": 1783938341.5167592, "harness": "opencode", "model": { diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/metadata.json index ff9a0de..f1e4a55 100644 --- a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/metadata.json @@ -1,7 +1,7 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:fcc6ed05673b7981aa17d60ca4e0d355fbcb57a6cbca293c0fa201ab97cdd081", - "digest_compatibility_note": null, + "current_task_digest": "sha256:29434bdcdce19a6e0cb24d77c800d172d49a9529875ffd4201cf928e242f4e62", + "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", "finished_at": 1783933771.613736, "harness": "opencode", "model": { diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/metadata.json index aaab68a..660774b 100644 --- a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/metadata.json @@ -1,7 +1,7 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:9da0e00147cf66804e6c2fc17869606bea8f260850c5447989a8880eef940d45", - "digest_compatibility_note": "Score-equivalent to the current task. The only later change adds a fail-closed internal timeout for candidates whose verifier would otherwise exceed BenchFlow's deadline.", + "current_task_digest": "sha256:5cf3805d59b03cb9d0bebb6f154eab38beb655918b8321500d692f5010a00cb2", + "digest_compatibility_note": "Score-equivalent to the current task. Later changes add a fail-closed verifier timeout plus a CI-only smoke config and documentation; the publication scoring config is unchanged.", "finished_at": 1783939823.3358562, "harness": "opencode", "model": { diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/metadata.json index ee6e268..71b273c 100644 --- a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/metadata.json @@ -1,7 +1,7 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:80c439f53e4ab964f9d7443cd7fb8f25cf6645a0bc288b0496b871c1800ebe78", - "digest_compatibility_note": null, + "current_task_digest": "sha256:c7e3b5e466431f4908c1e96cc7de80191f7a7c5ab5a63b3eea4d708f7f447fbf", + "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", "finished_at": 1783935158.328714, "harness": "opencode", "model": { diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/metadata.json index cacb6ff..acad2e8 100644 --- a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/metadata.json @@ -1,7 +1,7 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:b3431b238bec4e66af6189d30bcd15d5cd227144dfe8bf0dd1011aa5416c1436", - "digest_compatibility_note": null, + "current_task_digest": "sha256:316e198d8fa71e705c8b7fa54f41faf64e1f3c0ba5c4c92b8116b4bf78c5712b", + "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", "finished_at": 1783938003.487764, "harness": "opencode", "model": { diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/metadata.json index 15f1fe9..d2ae053 100644 --- a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/metadata.json @@ -1,7 +1,7 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:fcc6ed05673b7981aa17d60ca4e0d355fbcb57a6cbca293c0fa201ab97cdd081", - "digest_compatibility_note": null, + "current_task_digest": "sha256:29434bdcdce19a6e0cb24d77c800d172d49a9529875ffd4201cf928e242f4e62", + "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", "finished_at": 1783932419.4689522, "harness": "opencode", "model": { diff --git a/scripts/build_article_suite_leaderboard.py b/scripts/build_article_suite_leaderboard.py index 5265b68..c63f629 100644 --- a/scripts/build_article_suite_leaderboard.py +++ b/scripts/build_article_suite_leaderboard.py @@ -25,20 +25,58 @@ "simulation_heuristics_atari57_v1", "simulation_heuristics_montezuma_v1", ) +TASK_LABELS = { + "simulation_heuristics_ant_v1": "Ant", + "simulation_heuristics_pong_ram_v1": "Pong", + "simulation_heuristics_breakout_ram_v1": "Breakout RAM", + "simulation_heuristics_breakout_rgb_v1": "Breakout RGB", + "simulation_heuristics_halfcheetah_v1": "HalfCheetah", + "simulation_heuristics_vizdoom_d1_v1": "VizDoom D1", + "simulation_heuristics_vizdoom_d3_v1": "VizDoom D3", + "simulation_heuristics_atari57_v1": "Atari57", + "simulation_heuristics_montezuma_v1": "Montezuma's Revenge", +} +AVERAGE_LEADERBOARD_ID = "average" TASK_DIGEST_COMPATIBILITY = { "simulation_heuristics_ant_v1": { "sha256:bbb533da0cb86459f4d49dee667e6c73ac54c0188bc40e54e911d50ef3c3bc38": ( - "Score-equivalent to the current task. The only later change adds " - "a fail-closed internal timeout for candidates whose verifier " - "would otherwise exceed BenchFlow's deadline." + "Score-equivalent to the current task. Later changes add a " + "fail-closed verifier timeout plus a CI-only smoke config and " + "documentation; the publication scoring config is unchanged." + ), + "sha256:9da0e00147cf66804e6c2fc17869606bea8f260850c5447989a8880eef940d45": ( + "Score-equivalent to the current task. The later change adds only " + "a CI smoke config and documentation; the publication scoring " + "config is unchanged." + ), + }, + "simulation_heuristics_halfcheetah_v1": { + "sha256:80c439f53e4ab964f9d7443cd7fb8f25cf6645a0bc288b0496b871c1800ebe78": ( + "Score-equivalent to the current task. The later loader fix " + "registers submitted modules before execution so postponed " + "dataclass annotations import correctly; scoring is unchanged." ) - } + }, + "simulation_heuristics_vizdoom_d1_v1": { + "sha256:b3431b238bec4e66af6189d30bcd15d5cd227144dfe8bf0dd1011aa5416c1436": ( + "Score-equivalent to the current task. The later loader fix " + "registers submitted modules before execution so postponed " + "dataclass annotations import correctly; scoring is unchanged." + ) + }, + "simulation_heuristics_vizdoom_d3_v1": { + "sha256:fcc6ed05673b7981aa17d60ca4e0d355fbcb57a6cbca293c0fa201ab97cdd081": ( + "Score-equivalent to the current task. The later loader fix " + "registers submitted modules before execution so postponed " + "dataclass annotations import correctly; scoring is unchanged." + ) + }, } def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser( - description="Build the aggregate nine-task article-suite leaderboard." + description="Build nine task leaderboards and their final average." ) parser.add_argument( "--runs-root", @@ -167,6 +205,165 @@ def _expected_models() -> dict[str, dict[str, Any]]: return {model["id"]: model for model in payload["models"]} +def _rank_rows( + rows: list[dict[str, Any]], + *, + score_key: str, +) -> list[dict[str, Any]]: + ranked = sorted( + (dict(row) for row in rows), + key=lambda row: ( + -float(row[score_key]), + str(row["model_id"]), + ), + ) + previous_score: float | None = None + current_rank = 0 + for position, row in enumerate(ranked, start=1): + score = float(row[score_key]) + if previous_score is None or score != previous_score: + current_rank = position + row["rank"] = current_rank + previous_score = score + return ranked + + +def _build_leaderboards( + rows: list[dict[str, Any]], +) -> list[dict[str, Any]]: + leaderboards: list[dict[str, Any]] = [] + for task in TASKS: + task_rows = [ + { + "model_id": row["model_id"], + "model": row["model"], + "harness": row["harness"], + "provider_reasoning_effort": row[ + "provider_reasoning_effort" + ], + "normalized_score": row["task_scores"][task], + "submission_detail": row["submission_details"][task], + "source_run": row["source_runs"][task], + } + for row in rows + ] + leaderboards.append( + { + "id": task, + "label": TASK_LABELS[task], + "metric": "normalized_score", + "rows": _rank_rows( + task_rows, + score_key="normalized_score", + ), + } + ) + + average_rows = [ + { + "model_id": row["model_id"], + "model": row["model"], + "harness": row["harness"], + "provider_reasoning_effort": row[ + "provider_reasoning_effort" + ], + "average_normalized_score": row["average_normalized_score"], + } + for row in rows + ] + leaderboards.append( + { + "id": AVERAGE_LEADERBOARD_ID, + "label": "Nine-task average", + "metric": "average_normalized_score", + "rows": _rank_rows( + average_rows, + score_key="average_normalized_score", + ), + } + ) + return leaderboards + + +def _leaderboard_relative_path(path: str) -> str: + detail_path = Path(path) + try: + detail_path = detail_path.relative_to("leaderboard") + except ValueError: + pass + return detail_path.as_posix() + + +def _render_article_suite_markdown( + leaderboards: list[dict[str, Any]], +) -> str: + markdown = [ + "# GenesisBench Learning Beyond Gradients Article Suite", + "", + "This offline report contains 10 independent leaderboards: one for " + "each of the nine article-derived tasks, followed by the final " + "nine-task average.", + "", + "Task scores are unbounded normalized values. The public starter maps " + "to 0 and the trusted article-level reference maps to 100.", + ] + for index, board in enumerate(leaderboards, start=1): + markdown.extend(["", f"## {index}. {board['label']}", ""]) + if board["id"] == AVERAGE_LEADERBOARD_ID: + markdown.extend( + [ + "Arithmetic mean of the nine normalized task scores.", + "", + "| Rank | Model | Harness | Effort | Nine-task average |", + "| ---: | --- | --- | --- | ---: |", + ] + ) + for row in board["rows"]: + markdown.append( + "| " + + " | ".join( + [ + str(row["rank"]), + row["model"], + row["harness"], + row["provider_reasoning_effort"], + f"{row['average_normalized_score']:.2f}", + ] + ) + + " |" + ) + continue + + markdown.extend( + [ + f"Task: `{board['id']}`", + "", + "| Rank | Model | Harness | Effort | Normalized score | " + "Score details |", + "| ---: | --- | --- | --- | ---: | --- |", + ] + ) + for row in board["rows"]: + detail_path = _leaderboard_relative_path( + row["submission_detail"] + ) + markdown.append( + "| " + + " | ".join( + [ + str(row["rank"]), + row["model"], + row["harness"], + row["provider_reasoning_effort"], + f"{row['normalized_score']:.2f}", + f"[JSON]({detail_path})", + ] + ) + + " |" + ) + return "\n".join(markdown) + "\n" + + def _latest_model_runs(runs_root: Path) -> dict[str, Path]: selected: dict[str, tuple[float, Path]] = {} for metadata_path in runs_root.glob("*/*/run_metadata.json"): @@ -369,18 +566,19 @@ def main() -> None: }, } ) - ranked.sort( - key=lambda row: row["average_normalized_score"], - reverse=True, + ranked = _rank_rows( + ranked, + score_key="average_normalized_score", ) - for rank, row in enumerate(ranked, start=1): - row["rank"] = rank + leaderboards = _build_leaderboards(ranked) payload = { "benchmark": "learning_beyond_gradients_article_suite", "task_count": len(TASKS), + "leaderboard_count": len(leaderboards), "tasks": list(TASKS), "aggregation": "arithmetic_mean_of_normalized_task_scores", + "leaderboards": leaderboards, "task_digest_compatibility": TASK_DIGEST_COMPATIBILITY, "source_runs": { model_id: { @@ -402,48 +600,8 @@ def main() -> None: args.output.parent.mkdir(parents=True, exist_ok=True) args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n") - labels = { - "simulation_heuristics_ant_v1": "Ant", - "simulation_heuristics_pong_ram_v1": "Pong", - "simulation_heuristics_breakout_ram_v1": "Breakout RAM", - "simulation_heuristics_breakout_rgb_v1": "Breakout RGB", - "simulation_heuristics_halfcheetah_v1": "HalfCheetah", - "simulation_heuristics_vizdoom_d1_v1": "Doom D1", - "simulation_heuristics_vizdoom_d3_v1": "Doom D3", - "simulation_heuristics_atari57_v1": "Atari57", - "simulation_heuristics_montezuma_v1": "Montezuma", - } - header = [ - "Rank", - "Model", - "Harness", - "Effort", - "Average", - *(labels[task] for task in TASKS), - ] - alignment = ["---:", "---", "---", "---", "---:", *(["---:"] * len(TASKS))] - markdown = [ - "# GenesisBench Learning Beyond Gradients Article Suite", - "", - "The aggregate score is the arithmetic mean of nine normalized task " - "scores. Starter policies map to 0 and trusted article-level references " - "map to 100.", - "", - "| " + " | ".join(header) + " |", - "| " + " | ".join(alignment) + " |", - ] - for row in ranked: - values = [ - str(row["rank"]), - row["model"], - row["harness"], - row["provider_reasoning_effort"], - f"{row['average_normalized_score']:.2f}", - *(f"{row['task_scores'][task]:.2f}" for task in TASKS), - ] - markdown.append("| " + " | ".join(values) + " |") (args.output.parent / "ARTICLE_SUITE.md").write_text( - "\n".join(markdown) + "\n" + _render_article_suite_markdown(leaderboards) ) print(args.output) diff --git a/tests/test_article_suite_tooling.py b/tests/test_article_suite_tooling.py index 2b76d64..7afb4a9 100644 --- a/tests/test_article_suite_tooling.py +++ b/tests/test_article_suite_tooling.py @@ -291,23 +291,22 @@ def test_article_score_packaging_sanitizes_absolute_artifact_paths() -> None: assert sanitized["episodes"][0]["policy_path"] == "submitted_artifact" -def test_ant_timeout_only_digest_change_is_score_compatible() -> None: - old_digest = next( - iter( - leaderboard.TASK_DIGEST_COMPATIBILITY[ - "simulation_heuristics_ant_v1" - ] - ) - ) - - note = leaderboard._digest_compatibility_note( +def test_non_scoring_digest_changes_are_explicitly_compatible() -> None: + for task in ( "simulation_heuristics_ant_v1", - old_digest, - "sha256:new", - ) + "simulation_heuristics_halfcheetah_v1", + "simulation_heuristics_vizdoom_d1_v1", + "simulation_heuristics_vizdoom_d3_v1", + ): + for old_digest in leaderboard.TASK_DIGEST_COMPATIBILITY[task]: + note = leaderboard._digest_compatibility_note( + task, + old_digest, + "sha256:new", + ) - assert note is not None - assert "fail-closed internal timeout" in note + assert note is not None + assert "Score-equivalent" in note def test_scored_agent_timeout_is_a_valid_leaderboard_result( @@ -419,3 +418,58 @@ def test_latest_task_results_merges_partial_runs(tmp_path: Path) -> None: assert selected[model_id][leaderboard.TASKS[0]][1]["reward"] == 0.1 assert selected[model_id][leaderboard.TASKS[1]][1]["reward"] == 0.2 assert selected[model_id][leaderboard.TASKS[1]][3] == "sha256:digest-2" + + +def test_offline_report_builds_nine_task_boards_then_average() -> None: + rows = [] + for model_id, model, average in ( + ("model-a", "Model A", 5.0), + ("model-b", "Model B", 10.0), + ): + task_scores = {task: 0.0 for task in leaderboard.TASKS} + rows.append( + { + "model_id": model_id, + "model": model, + "harness": "opencode", + "provider_reasoning_effort": "max", + "average_normalized_score": average, + "task_scores": task_scores, + "submission_details": { + task: f"leaderboard/submissions/{model_id}/{task}.json" + for task in leaderboard.TASKS + }, + "source_runs": { + task: f"leaderboard/runs/{model_id}/{task}" + for task in leaderboard.TASKS + }, + } + ) + rows[0]["task_scores"][leaderboard.TASKS[0]] = 10.0 + rows[1]["task_scores"][leaderboard.TASKS[0]] = 20.0 + rows[0]["task_scores"][leaderboard.TASKS[1]] = 30.0 + rows[1]["task_scores"][leaderboard.TASKS[1]] = 5.0 + + boards = leaderboard._build_leaderboards(rows) + markdown = leaderboard._render_article_suite_markdown(boards) + + assert len(boards) == 10 + assert [board["id"] for board in boards[:-1]] == list(leaderboard.TASKS) + assert boards[-1]["id"] == leaderboard.AVERAGE_LEADERBOARD_ID + assert [row["model_id"] for row in boards[0]["rows"]] == [ + "model-b", + "model-a", + ] + assert [row["model_id"] for row in boards[1]["rows"]] == [ + "model-a", + "model-b", + ] + assert [row["rank"] for row in boards[2]["rows"]] == [1, 1] + assert [row["model_id"] for row in boards[-1]["rows"]] == [ + "model-b", + "model-a", + ] + assert markdown.count("| Rank | Model | Harness | Effort |") == 10 + assert markdown.rstrip().split("## ")[-1].startswith( + "10. Nine-task average" + ) diff --git a/tests/test_leaderboard_artifacts.py b/tests/test_leaderboard_artifacts.py index a0a6737..de1c3a0 100644 --- a/tests/test_leaderboard_artifacts.py +++ b/tests/test_leaderboard_artifacts.py @@ -8,6 +8,8 @@ REPO_ROOT = Path(__file__).resolve().parents[1] LEADERBOARD = REPO_ROOT / "leaderboard" / "simulation_heuristics_ant_v1.json" ARTICLE_SUITE = REPO_ROOT / "leaderboard" / "article_suite.json" +ARTICLE_SUITE_MARKDOWN = REPO_ROOT / "leaderboard" / "ARTICLE_SUITE.md" +ROOT_README = REPO_ROOT / "README.md" def test_packaged_leaderboard_is_self_contained() -> None: @@ -38,9 +40,14 @@ def test_article_suite_leaderboard_is_complete_and_self_contained() -> None: assert payload["benchmark"] == "learning_beyond_gradients_article_suite" assert payload["task_count"] == 9 + assert payload["leaderboard_count"] == 10 assert len(payload["tasks"]) == 9 assert len(payload["rows"]) == 4 assert [row["rank"] for row in payload["rows"]] == [1, 2, 3, 4] + assert [board["id"] for board in payload["leaderboards"][:-1]] == payload[ + "tasks" + ] + assert payload["leaderboards"][-1]["id"] == "average" averages = [ row["average_normalized_score"] for row in payload["rows"] @@ -80,3 +87,30 @@ def assert_no_absolute_artifact_paths(value: object) -> None: assert_no_absolute_artifact_paths(item) assert_no_absolute_artifact_paths(score) + + model_rows = {row["model_id"]: row for row in payload["rows"]} + for board in payload["leaderboards"][:-1]: + scores = [row["normalized_score"] for row in board["rows"]] + assert scores == sorted(scores, reverse=True) + assert len(board["rows"]) == len(payload["rows"]) + for row in board["rows"]: + assert row["normalized_score"] == model_rows[row["model_id"]][ + "task_scores" + ][board["id"]] + + average_board = payload["leaderboards"][-1] + assert [ + row["average_normalized_score"] for row in average_board["rows"] + ] == averages + + markdown = ARTICLE_SUITE_MARKDOWN.read_text() + assert markdown.count("| Rank | Model | Harness | Effort |") == 10 + headings = [ + line for line in markdown.splitlines() if line.startswith("## ") + ] + assert len(headings) == 10 + assert headings[-1] == "## 10. Nine-task average" + + root_readme = ROOT_README.read_text() + assert "| Rank | Model | Nine-task average |" in root_readme + assert "## Legacy Ant-Only Leaderboard" not in root_readme