Skip to content

Revise article scoring and publish HTML leaderboards #19

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bingran-you merged 1 commit into from
Jul 13, 2026
Merged

Revise article scoring and publish HTML leaderboards #19

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 16 additions & 7 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,22 +117,31 @@ mapping is documented in `docs/learning-beyond-gradients-suite.md`.

The first OpenCode sweep across all nine article-derived tasks:

| Rank | Model | Nine-task average |
| ---: | --- | ---: |
| 1 | GPT-5.5 | 43.19 |
| 2 | Claude Opus 4.8 | 39.82 |
| 3 | GPT-5.6 Sol | 39.38 |
| 4 | GPT-5.4 Mini | -29.72 |
![GenesisBench final normalized leaderboard](leaderboard/article_suite_final_leaderboard.png)

See [`leaderboard/ARTICLE_SUITE.md`](leaderboard/ARTICLE_SUITE.md) for the nine
task-specific leaderboards followed by the final average leaderboard.
task-specific leaderboards followed by the final cross-task leaderboard.
`leaderboard/article_suite.json` contains the same 10-board structure in
machine-readable form.

Scores are unbounded normalized values: `0` matches the public starter and
`100` matches the trusted article-level reference. Negative scores are genuine
regressions; scores above `100` exceed the reference.

The final ranking uses the interquartile mean (IQM) of the nine task scores:
sort them, remove the lowest two and highest two, then average the middle five.
The chart uses a plot-only positive index equal to `IQM + 100`; the raw IQM
remains the ranking metric and is retained in the JSON.

Inference settings are provider-specific: GPT-5.6 Sol and Claude Opus 4.8 use
`max`; GPT-5.5 and GPT-5.4 Mini use `xhigh`. These labels come from different
provider interfaces and are categorical settings, not a shared numeric compute
scale. Exact routes are documented in
[`docs/learning-beyond-gradients-suite.md`](docs/learning-beyond-gradients-suite.md).

See [`docs/article-suite-scoring.md`](docs/article-suite-scoring.md) for the
formula, research precedent, and current single-run limitation.

## Contribute a Task

Create a scaffold:
Expand Down
100 changes: 100 additions & 0 deletions docs/article-suite-scoring.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# Article-suite scoring methodology

GenesisBench separates per-task normalization from cross-task aggregation.

## Per-task normalized score

Every task evaluates a candidate on its hidden suite and compares the resulting
raw score with two frozen anchors:

```text
normalized_task_score =
100 * (candidate_score - starter_score)
/ (reference_score - starter_score)
```

The public starter maps to `0`; the trusted article-level reference maps to
`100`. Scores are intentionally unbounded:

- negative scores perform below the starter;
- scores above `100` outperform the reference.

This follows the same family of baseline normalization used by benchmarks such
as [D4RL](https://arxiv.org/abs/2004.07219), which normalizes returns between
task-specific lower and upper reference scores.

## Final normalized score

The primary final metric is the interquartile mean (IQM), implemented as the
25% trimmed mean used by
[RLiable](https://github.com/google-research/rliable):

```text
scores = sort(the nine normalized task scores)
trim_count = floor(0.25 * 9) = 2
final_normalized_score = mean(scores[2:7])
```

In words: remove the lowest two and highest two task scores and average the
middle five.

The JSON also publishes:

- `arithmetic_mean_normalized_score`;
- `median_normalized_score`;
- the original `average_normalized_score` as a backward-compatible alias for
the arithmetic mean.

## Why IQM is primary

The original GenesisBench aggregate was the unweighted arithmetic mean of nine
normalized task scores. That is simple and remains useful, but a single very
large positive or negative task can move the final rank substantially.

The NeurIPS paper
[*Deep Reinforcement Learning at the Edge of the Statistical
Precipice*](https://proceedings.neurips.cc/paper_files/paper/2021/hash/f514cec81cb148559cf475e7426eed5e-Abstract.html)
recommends robust aggregate metrics such as IQM, together with performance
profiles and uncertainty estimates. Atari research also commonly reports both
median and mean human-normalized scores; the
[Agent57 paper](https://arxiv.org/abs/2003.13350) explicitly notes that a high
average can hide weak performance on many individual games.

GenesisBench therefore uses:

1. task-specific native raw-score leaderboards for visibility;
2. IQM as the primary cross-task rank;
3. arithmetic mean and median as secondary diagnostics.

## Positive display index

The final image uses a fixed additive transform:

```text
positive_display_score = final_normalized_score + 100
```

This is presentation-only. Raw IQM remains the official ranking field.

A fixed offset is preferred over cohort min-max scaling because it:

- preserves every model-to-model difference exactly;
- does not change when another model is added;
- gives a stable interpretation: an aggregate starter-level IQM of `0`
displays as `100`.

Min-max scaling would force the current best and worst models to arbitrary
endpoints and would rewrite every displayed score whenever the comparison set
changes. Clipping negative IQM values to zero would erase meaningful
differences.

## Current statistical limitation

The published sweep currently has one independent agent run per model/task
pair. That is sufficient to recompute deterministic aggregate metrics, but not
to estimate statistically meaningful bootstrap confidence intervals or
probability of improvement between models.

A future multi-run release should retain IQM and additionally report
stratified-bootstrap confidence intervals, performance profiles, and pairwise
probability of improvement following the RLiable protocol.
38 changes: 25 additions & 13 deletions docs/learning-beyond-gradients-suite.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,12 +49,16 @@ zero and are not presented as article reproductions.
All new leaderboard runs use the BenchFlow `opencode` ACP harness. The canonical
matrix is:

| Model | Provider | Effort |
| --- | --- | --- |
| GPT-5.6 Sol | Azure | `max` |
| GPT-5.5 | Azure | `xhigh` |
| Claude Opus 4.8 | Claude OAuth through pinned OpenCode plugin | `max` |
| GPT-5.4 Mini | Azure | `xhigh` |
| Model | Exact route | Harness | Provider reasoning setting |
| --- | --- | --- | --- |
| GPT-5.6 Sol | Azure `azure/gpt-5.6-sol` | OpenCode | `max` |
| GPT-5.5 | Azure `azure/gpt-5.5` | OpenCode | `xhigh` |
| Claude Opus 4.8 | Claude OAuth `anthropic/claude-opus-4-8` through the pinned OpenCode plugin | OpenCode | `max` |
| GPT-5.4 Mini | Azure `azure/gpt-5.4-mini` | OpenCode | `xhigh` |

`max` and `xhigh` are provider-specific categorical labels. They indicate the
configured reasoning setting for that route; they are not interchangeable
units and should not be read as a shared numeric inference-compute scale.

OpenCode talks directly to the provider because BenchFlow 0.6.5's
chat-completions gateway cannot faithfully transform Azure GPT-5.6 Sol tool
Expand All @@ -67,18 +71,26 @@ The offline article-suite report contains 10 independent leaderboards in a
fixed order:

1. one leaderboard for each of the nine article-derived tasks;
2. the final averaged leaderboard.
2. the final cross-task leaderboard.

The final score is the unweighted arithmetic mean:
Each task first receives an unbounded anchor-normalized score:

```text
average = sum(nine normalized task scores) / 9
task score = 100 * (candidate - starter) / (reference - starter)
```

The repository README intentionally shows only the final averaged leaderboard.
Detailed task rankings and score-artifact links live in
`leaderboard/ARTICLE_SUITE.md`; the matching machine-readable structure lives
in `leaderboard/article_suite.json`.
The primary final score is the 25% trimmed interquartile mean (IQM). With nine
tasks, the two lowest and two highest normalized scores are removed and the
middle five are averaged. Arithmetic mean and median remain secondary
diagnostics.

The repository README intentionally shows only the final leaderboard image.
The nine task panels use each environment's native raw score. The final chart
uses a positive plot index equal to `IQM + 100`, while raw IQM remains the
official ranking metric. Both images live in `leaderboard/ARTICLE_SUITE.md`;
the matching machine-readable structure lives in
`leaderboard/article_suite.json`. See `docs/article-suite-scoring.md` for the
research rationale and statistical limitations.

The runner and resumable leaderboard builder live in:

Expand Down
29 changes: 25 additions & 4 deletions experiments/article_suite/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,20 @@ the nine task packages derived from the article:
Every run uses BenchFlow's registered `opencode` ACP harness. OpenHands is not
part of this suite.

## Inference settings

| Model | Exact route | Provider-specific reasoning setting |
| --- | --- | --- |
| GPT-5.6 Sol | `azure/gpt-5.6-sol` | `max` |
| GPT-5.5 | `azure/gpt-5.5` | `xhigh` |
| Claude Opus 4.8 | `anthropic/claude-opus-4-8` via Claude OAuth and the pinned OpenCode plugin | `max` |
| GPT-5.4 Mini | `azure/gpt-5.4-mini` | `xhigh` |

The setting names are categorical labels exposed by each provider integration,
not a common numeric measure of inference compute. All four routes use the
OpenCode harness; the route and reasoning setting are stored with every
published result.

## Credentials

Credential values are read from the process environment or an env file. They
Expand Down Expand Up @@ -85,11 +99,18 @@ uv run python scripts/build_article_suite_leaderboard.py

The builder writes:

- `leaderboard/ARTICLE_SUITE.md`: nine task-specific leaderboards followed by
the final average leaderboard;
- `leaderboard/ARTICLE_SUITE.md`: one nine-panel task image followed by the
final IQM leaderboard image;
- `leaderboard/article_suite.json`: the same 10 leaderboards plus the
model-centric score records used for reproducibility.
- `leaderboard/article_suite_task_leaderboards.png`: nine task-specific
leaderboard panels using native raw environment scores;
- `leaderboard/article_suite_final_leaderboard.png`: the final cross-task
ranking displayed as `IQM + 100` so every current plotted value is positive.

The final aggregate is the arithmetic mean of the nine normalized task scores.
Each task maps its starter policy to `0` and its trusted article-level reference
to `100`. The top-level repository README shows only this final aggregate.
to `100`. The final primary aggregate is the interquartile mean: remove the two
lowest and two highest of the nine normalized scores, then average the middle
five. Arithmetic mean and median remain secondary diagnostics. The top-level
repository README shows only the final image. The additive display offset does
not affect ranking or score gaps; raw IQM remains available in the JSON.
114 changes: 8 additions & 106 deletions leaderboard/ARTICLE_SUITE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,115 +1,17 @@
# GenesisBench Learning Beyond Gradients Article Suite

This offline report contains 10 independent leaderboards: one for each of the nine article-derived tasks, followed by the final nine-task average.
The first image contains the nine independently ranked task leaderboards. The second image contains the final cross-task ranking.

Task scores are unbounded normalized values. The public starter maps to 0 and the trusted article-level reference maps to 100.
The nine task panels use each environment's native raw score. The final scientific metric remains unbounded IQM.

## 1. Ant
## Nine task leaderboards

Task: `simulation_heuristics_ant_v1`
![Nine task-specific GenesisBench leaderboards](article_suite_task_leaderboards.png)

| Rank | Model | Harness | Effort | Normalized score | Score details |
| ---: | --- | --- | --- | ---: | --- |
| 1 | GPT-5.6 Sol | opencode | max | 39.20 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/score.json) |
| 2 | Claude Opus 4.8 | opencode | max | 14.05 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/score.json) |
| 3 | GPT-5.5 | opencode | xhigh | -16.89 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/score.json) |
| 4 | GPT-5.4 Mini | opencode | xhigh | -31.36 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/score.json) |
## Final normalized score

## 2. Pong
The primary score is the interquartile mean (IQM): sort the nine task scores, remove the lowest two and highest two, then average the middle five. The image uses a plot-only positive display index equal to `IQM + 100`; raw IQM, arithmetic mean, and median remain in the JSON.

Task: `simulation_heuristics_pong_ram_v1`
![Final GenesisBench article-suite leaderboard](article_suite_final_leaderboard.png)

| Rank | Model | Harness | Effort | Normalized score | Score details |
| ---: | --- | --- | --- | ---: | --- |
| 1 | Claude Opus 4.8 | opencode | max | 100.00 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/score.json) |
| 2 | GPT-5.6 Sol | opencode | max | 52.92 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/score.json) |
| 3 | GPT-5.5 | opencode | xhigh | 45.83 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/score.json) |
| 4 | GPT-5.4 Mini | opencode | xhigh | -75.00 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/score.json) |

## 3. Breakout RAM

Task: `simulation_heuristics_breakout_ram_v1`

| Rank | Model | Harness | Effort | Normalized score | Score details |
| ---: | --- | --- | --- | ---: | --- |
| 1 | GPT-5.6 Sol | opencode | max | 100.00 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/score.json) |
| 2 | GPT-5.5 | opencode | xhigh | 95.60 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/score.json) |
| 3 | GPT-5.4 Mini | opencode | xhigh | 2.52 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/score.json) |
| 4 | Claude Opus 4.8 | opencode | max | 0.00 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/score.json) |

## 4. Breakout RGB

Task: `simulation_heuristics_breakout_rgb_v1`

| Rank | Model | Harness | Effort | Normalized score | Score details |
| ---: | --- | --- | --- | ---: | --- |
| 1 | GPT-5.5 | opencode | xhigh | 70.76 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/score.json) |
| 1 | GPT-5.6 Sol | opencode | max | 70.76 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/score.json) |
| 3 | GPT-5.4 Mini | opencode | xhigh | 15.52 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/score.json) |
| 4 | Claude Opus 4.8 | opencode | max | 0.00 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/score.json) |

## 5. HalfCheetah

Task: `simulation_heuristics_halfcheetah_v1`

| Rank | Model | Harness | Effort | Normalized score | Score details |
| ---: | --- | --- | --- | ---: | --- |
| 1 | Claude Opus 4.8 | opencode | max | 26.99 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/score.json) |
| 2 | GPT-5.6 Sol | opencode | max | 15.87 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/score.json) |
| 3 | GPT-5.5 | opencode | xhigh | 2.74 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/score.json) |
| 4 | GPT-5.4 Mini | opencode | xhigh | -8.17 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/score.json) |

## 6. VizDoom D1

Task: `simulation_heuristics_vizdoom_d1_v1`

| Rank | Model | Harness | Effort | Normalized score | Score details |
| ---: | --- | --- | --- | ---: | --- |
| 1 | Claude Opus 4.8 | opencode | max | 95.06 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/score.json) |
| 2 | GPT-5.5 | opencode | xhigh | 85.20 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/score.json) |
| 3 | GPT-5.6 Sol | opencode | max | 49.41 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/score.json) |
| 4 | GPT-5.4 Mini | opencode | xhigh | -161.12 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/score.json) |

## 7. VizDoom D3

Task: `simulation_heuristics_vizdoom_d3_v1`

| Rank | Model | Harness | Effort | Normalized score | Score details |
| ---: | --- | --- | --- | ---: | --- |
| 1 | Claude Opus 4.8 | opencode | max | 122.26 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/score.json) |
| 2 | GPT-5.6 Sol | opencode | max | 26.24 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/score.json) |
| 3 | GPT-5.5 | opencode | xhigh | 5.48 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/score.json) |
| 4 | GPT-5.4 Mini | opencode | xhigh | -9.86 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/score.json) |

## 8. Atari57

Task: `simulation_heuristics_atari57_v1`

| Rank | Model | Harness | Effort | Normalized score | Score details |
| ---: | --- | --- | --- | ---: | --- |
| 1 | Claude Opus 4.8 | opencode | max | 0.00 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/score.json) |
| 1 | GPT-5.4 Mini | opencode | xhigh | 0.00 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/score.json) |
| 1 | GPT-5.5 | opencode | xhigh | 0.00 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/score.json) |
| 1 | GPT-5.6 Sol | opencode | max | 0.00 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/score.json) |

## 9. Montezuma's Revenge

Task: `simulation_heuristics_montezuma_v1`

| Rank | Model | Harness | Effort | Normalized score | Score details |
| ---: | --- | --- | --- | ---: | --- |
| 1 | GPT-5.5 | opencode | xhigh | 100.00 | [JSON](article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/score.json) |
| 2 | Claude Opus 4.8 | opencode | max | 0.00 | [JSON](article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/score.json) |
| 2 | GPT-5.4 Mini | opencode | xhigh | 0.00 | [JSON](article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/score.json) |
| 2 | GPT-5.6 Sol | opencode | max | 0.00 | [JSON](article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/score.json) |

## 10. Nine-task average

Arithmetic mean of the nine normalized task scores.

| Rank | Model | Harness | Effort | Nine-task average |
| ---: | --- | --- | --- | ---: |
| 1 | GPT-5.5 | opencode | xhigh | 43.19 |
| 2 | Claude Opus 4.8 | opencode | max | 39.82 |
| 3 | GPT-5.6 Sol | opencode | max | 39.38 |
| 4 | GPT-5.4 Mini | opencode | xhigh | -29.72 |
Machine-readable rankings and score-detail paths are available in [`article_suite.json`](article_suite.json). The scoring rationale is documented in [`docs/article-suite-scoring.md`](../docs/article-suite-scoring.md).
9 changes: 2 additions & 7 deletions leaderboard/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,10 @@

## Learning Beyond Gradients article suite

| Rank | Model | Nine-task average |
| ---: | --- | ---: |
| 1 | GPT-5.5 | 43.19 |
| 2 | Claude Opus 4.8 | 39.82 |
| 3 | GPT-5.6 Sol | 39.38 |
| 4 | GPT-5.4 Mini | -29.72 |
![GenesisBench final normalized leaderboard](article_suite_final_leaderboard.png)

See [`ARTICLE_SUITE.md`](ARTICLE_SUITE.md) for 10 independent leaderboards:
nine task-specific rankings followed by the final average ranking.
nine task-specific rankings followed by the final IQM ranking.

## Legacy Simulation Heuristics Ant v1

Expand Down
Loading