diff --git a/README.md b/README.md index 824a1a3..c91c22e 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,8 @@ Requirements: - Python 3.12+ - [`uv`](https://docs.astral.sh/uv/) -- Docker for isolated agent experiments +- Daytona credentials for authoritative leaderboard experiments +- Docker only for optional local task development Install and validate: @@ -98,6 +99,7 @@ Run one model across all nine tasks: ```bash uv run python scripts/run_article_suite.py \ + --env-file /path/to/credentials.env \ --model gpt-5.6-sol ``` @@ -105,6 +107,7 @@ Run all four canonical models and rebuild the 10 leaderboard artifacts: ```bash uv run python scripts/run_article_suite.py \ + --env-file /path/to/credentials.env \ --all-models uv run python scripts/build_article_suite_leaderboard.py ``` @@ -115,7 +118,7 @@ mapping is documented in `docs/learning-beyond-gradients-suite.md`. ## Article-Suite Leaderboard -The first OpenCode sweep across all nine article-derived tasks: +The five-trial OpenCode sweep across all nine article-derived tasks: ![GenesisBench final normalized leaderboard](leaderboard/article_suite_final_leaderboard.png) @@ -128,19 +131,22 @@ Scores are unbounded normalized values: `0` matches the public starter and `100` matches the trusted article-level reference. Negative scores are genuine regressions; scores above `100` exceed the reference. -The final ranking uses the interquartile mean (IQM) of the nine task scores: -sort them, remove the lowest two and highest two, then average the middle five. -The chart uses a plot-only positive index equal to `IQM + 100`; the raw IQM -remains the ranking metric and is retained in the JSON. +Each model runs five independent trials. The final ranking uses RLiable-style +interquartile mean (IQM) over all 45 normalized trial-task scores: remove the +lowest 11 and highest 11, then average the middle 23. The displayed `±` value +is the sample standard deviation of the five per-trial nine-task IQMs. The chart +uses a plot-only positive index equal to `IQM + 100`; raw IQM remains in the +JSON. Inference settings are provider-specific: GPT-5.6 Sol and Claude Opus 4.8 use `max`; GPT-5.5 and GPT-5.4 Mini use `xhigh`. These labels come from different provider interfaces and are categorical settings, not a shared numeric compute scale. Exact routes are documented in [`docs/learning-beyond-gradients-suite.md`](docs/learning-beyond-gradients-suite.md). +All published article-suite rows run in Daytona sandboxes. See [`docs/article-suite-scoring.md`](docs/article-suite-scoring.md) for the -formula, research precedent, and current single-run limitation. +formula, research precedent, and statistical limitations. ## Contribute a Task diff --git a/docs/article-suite-scoring.md b/docs/article-suite-scoring.md index 9ad74f4..a5054fb 100644 --- a/docs/article-suite-scoring.md +++ b/docs/article-suite-scoring.md @@ -23,25 +23,46 @@ This follows the same family of baseline normalization used by benchmarks such as [D4RL](https://arxiv.org/abs/2004.07219), which normalizes returns between task-specific lower and upper reference scores. +## Five-trial estimator + +Each model runs the complete nine-task suite five independent times. For task +`t`, the published task score is the arithmetic mean of its five normalized +trial scores. The task leaderboard also publishes their sample standard +deviation. + +Task plots use native raw environment scores. If a task's verifier intentionally +fails closed with `normalized_score = 0` and no observed raw score, the plot uses +the frozen starter anchor as the raw-score equivalent. The trial artifact keeps +`observed_raw_score = null` and records +`raw_score_source = starter_anchor_equivalent_for_fail_closed_timeout`; this +case is never presented as an observed environment return. + ## Final normalized score -The primary final metric is the interquartile mean (IQM), implemented as the -25% trimmed mean used by -[RLiable](https://github.com/google-research/rliable): +The primary cross-task metric is the interquartile mean (IQM), implemented as +the 25% trimmed mean used by +[RLiable's `aggregate_iqm`](https://github.com/google-research/rliable/blob/master/rliable/metrics.py#L61-L70). +RLiable computes IQM across the complete `num_runs × num_tasks` score matrix: ```text -scores = sort(the nine normalized task scores) -trim_count = floor(0.25 * 9) = 2 -final_normalized_score = mean(scores[2:7]) +scores = sort(flatten(the 5 × 9 normalized score matrix)) +trim_count = floor(0.25 * 45) = 11 +final_normalized_score = mean(scores[11:34]) ``` -In words: remove the lowest two and highest two task scores and average the -middle five. +This retains the middle `23` of the `45` trial-task scores. It matches +RLiable's `aggregate_iqm`, which applies a 25% trimmed mean with `axis=None`. + +For a readable run-to-run variability diagnostic, the leaderboard separately +computes one nine-task IQM per trial and publishes the sample standard +deviation of those five trial-level IQMs. That displayed `±` value is not a +bootstrap confidence interval around the pooled IQM. The JSON also publishes: - `arithmetic_mean_normalized_score`; - `median_normalized_score`; +- `mean_trial_iqm_normalized_score`; - the original `average_normalized_score` as a backward-compatible alias for the arithmetic mean. @@ -88,13 +109,10 @@ endpoints and would rewrite every displayed score whenever the comparison set changes. Clipping negative IQM values to zero would erase meaningful differences. -## Current statistical limitation - -The published sweep currently has one independent agent run per model/task -pair. That is sufficient to recompute deterministic aggregate metrics, but not -to estimate statistically meaningful bootstrap confidence intervals or -probability of improvement between models. +## Statistical limitation -A future multi-run release should retain IQM and additionally report -stratified-bootstrap confidence intervals, performance profiles, and pairwise -probability of improvement following the RLiable protocol. +Five trials expose run-to-run variability and are materially stronger than a +single run, but they remain a small sample. The leaderboard reports sample +standard deviation rather than claiming narrow confidence intervals. A larger +future release can add stratified-bootstrap confidence intervals, performance +profiles, and pairwise probability of improvement following RLiable. diff --git a/docs/learning-beyond-gradients-suite.md b/docs/learning-beyond-gradients-suite.md index 17e9e30..d8b87ed 100644 --- a/docs/learning-beyond-gradients-suite.md +++ b/docs/learning-beyond-gradients-suite.md @@ -49,12 +49,12 @@ zero and are not presented as article reproductions. All new leaderboard runs use the BenchFlow `opencode` ACP harness. The canonical matrix is: -| Model | Exact route | Harness | Provider reasoning setting | -| --- | --- | --- | --- | -| GPT-5.6 Sol | Azure `azure/gpt-5.6-sol` | OpenCode | `max` | -| GPT-5.5 | Azure `azure/gpt-5.5` | OpenCode | `xhigh` | -| Claude Opus 4.8 | Claude OAuth `anthropic/claude-opus-4-8` through the pinned OpenCode plugin | OpenCode | `max` | -| GPT-5.4 Mini | Azure `azure/gpt-5.4-mini` | OpenCode | `xhigh` | +| Model | Exact route | Harness | Provider reasoning setting | Sandbox | +| --- | --- | --- | --- | --- | +| GPT-5.6 Sol | Azure `azure/gpt-5.6-sol` | OpenCode | `max` | Daytona | +| GPT-5.5 | Azure `azure/gpt-5.5` | OpenCode | `xhigh` | Daytona | +| Claude Opus 4.8 | Claude OAuth `anthropic/claude-opus-4-8` through the pinned OpenCode plugin | OpenCode | `max` | Daytona | +| GPT-5.4 Mini | Azure `azure/gpt-5.4-mini` | OpenCode | `xhigh` | Daytona | `max` and `xhigh` are provider-specific categorical labels. They indicate the configured reasoning setting for that route; they are not interchangeable @@ -62,9 +62,12 @@ units and should not be read as a shared numeric inference-compute scale. OpenCode talks directly to the provider because BenchFlow 0.6.5's chat-completions gateway cannot faithfully transform Azure GPT-5.6 Sol tool -calls. BenchFlow continues to own task staging, Daytona/Docker isolation, ACP +calls. BenchFlow continues to own task staging, Daytona isolation, ACP trajectory capture, timing, and verifier execution. +The published matrix uses Daytona for every task. Docker is retained only as an +optional local development backend. + ## Leaderboard outputs The offline article-suite report contains 10 independent leaderboards in a @@ -79,10 +82,12 @@ Each task first receives an unbounded anchor-normalized score: task score = 100 * (candidate - starter) / (reference - starter) ``` -The primary final score is the 25% trimmed interquartile mean (IQM). With nine -tasks, the two lowest and two highest normalized scores are removed and the -middle five are averaged. Arithmetic mean and median remain secondary -diagnostics. +Each model runs five independent trials. Task leaderboards report mean ± sample +standard deviation across those trials. The primary cross-task score is the +RLiable-style 25% trimmed interquartile mean (IQM) over the complete `5 × 9` +score matrix: flatten all 45 normalized scores, remove the lowest 11 and +highest 11, and average the middle 23. The displayed `±` value is the sample +standard deviation of the five per-trial nine-task IQMs. The repository README intentionally shows only the final leaderboard image. The nine task panels use each environment's native raw score. The final chart diff --git a/docs/simulation-heuristics-ant-v1-task-design.md b/docs/simulation-heuristics-ant-v1-task-design.md index 4516a87..4c9a1b3 100644 --- a/docs/simulation-heuristics-ant-v1-task-design.md +++ b/docs/simulation-heuristics-ant-v1-task-design.md @@ -176,6 +176,9 @@ The first published sweep uses: - isolated Docker workspaces; - each model's highest supported reasoning level. +The current article-suite protocol triples the Ant agent timeout to 90 minutes +and reports mean ± sample standard deviation across five independent trials. + Simulation Heuristics Ant v1 currently enforces wall-clock time but does not centrally meter every simulator step. Future hosted versions should add an authoritative interaction meter before making sample-efficiency claims across diff --git a/experiments/article_suite/README.md b/experiments/article_suite/README.md index cdc3125..b1d6ee5 100644 --- a/experiments/article_suite/README.md +++ b/experiments/article_suite/README.md @@ -16,14 +16,42 @@ the nine task packages derived from the article: Every run uses BenchFlow's registered `opencode` ACP harness. OpenHands is not part of this suite. +## Experiment protocol + +The current leaderboard protocol is defined in `protocol.toml`: + +- five independent full-suite trials per model; +- three times the original agent wall-clock timeout for every task; +- task scores reported as mean ± sample standard deviation across trials; +- final score computed as RLiable-style IQM over all `5 × 9 = 45` + normalized trial-task scores; +- variability shown as the sample standard deviation of the five per-trial + nine-task IQMs. + +The resulting matrix is: + +```text +4 models × 5 trials × 9 tasks = 180 model-task runs +``` + +| Tasks | Previous cap | Current cap | +| --- | ---: | ---: | +| Ant, Pong, both Breakouts, Montezuma, VizDoom D1/D3 | 30 min | 90 min | +| HalfCheetah | 60 min | 180 min | +| Atari57 aggregate search | 7 days | 21 days | + +Atari57's cap is exceptional because one task contains 342 accounted search +slots. It can be scheduled separately when operating capacity is limited; the +cap is a maximum, not an expected runtime. + ## Inference settings -| Model | Exact route | Provider-specific reasoning setting | -| --- | --- | --- | -| GPT-5.6 Sol | `azure/gpt-5.6-sol` | `max` | -| GPT-5.5 | `azure/gpt-5.5` | `xhigh` | -| Claude Opus 4.8 | `anthropic/claude-opus-4-8` via Claude OAuth and the pinned OpenCode plugin | `max` | -| GPT-5.4 Mini | `azure/gpt-5.4-mini` | `xhigh` | +| Model | Exact route | Provider-specific reasoning setting | Sandbox | +| --- | --- | --- | --- | +| GPT-5.6 Sol | `azure/gpt-5.6-sol` | `max` | Daytona | +| GPT-5.5 | `azure/gpt-5.5` | `xhigh` | Daytona | +| Claude Opus 4.8 | `anthropic/claude-opus-4-8` via Claude OAuth and the pinned OpenCode plugin | `max` | Daytona | +| GPT-5.4 Mini | `azure/gpt-5.4-mini` | `xhigh` | Daytona | The setting names are categorical labels exposed by each provider integration, not a common numeric measure of inference compute. All four routes use the @@ -50,8 +78,7 @@ CLAUDE_CODE_OAUTH_TOKEN ## Run -Docker must be running because the authoritative suite uses isolated BenchFlow -task environments: +The authoritative suite uses isolated Daytona sandboxes in the `us` target: ```bash uv run python scripts/run_article_suite.py \ @@ -59,30 +86,45 @@ uv run python scripts/run_article_suite.py \ --model gpt-5.6-sol ``` -Atari57 requests more CPU, memory, and storage than the current Daytona account -allows. The all-nine commands therefore use the default local Docker sandbox. -Use Daytona only for selected non-Atari tasks. +The July 14, 2026 Atari57 capacity canary confirmed that Daytona runs all nine +tasks. BenchFlow currently clamps Atari57 from 14 requested CPUs to 4 and from +102,400 MB requested storage to 10,240 MB; these are execution limits, not score +transformations, and the remote verifier still completes normally. + +Local Docker remains available for development with `--sandbox docker`. Such +runs default to calibrated `linux/amd64` images; they are not selected for the +published article-suite leaderboard. Run all four models sequentially: ```bash uv run python scripts/run_article_suite.py \ --env-file /path/to/credentials.env \ - --all-models + --all-models \ + --trials 5 \ + --batch-id article-suite-v2-daytona ``` -Long runs are resumable at task granularity. For example: +Runs are resumable at model-trial granularity. Reusing `--batch-id` skips +completed trials. Models may also be scheduled into separate batch directories +for parallel execution. The leaderboard builder chooses one complete +five-trial batch per model and never mixes trials from different batches within +the same model. + +Use `--trial` to schedule or repair selected trials: ```bash uv run python scripts/run_article_suite.py \ --env-file /path/to/credentials.env \ --model gpt-5.6-sol \ --task simulation_heuristics_halfcheetah_v1 \ - --sandbox daytona + --trial 3 \ + --batch-id article-suite-v2-daytona ``` -The leaderboard builder selects the latest successful result for every -model/task pair across all run batches. +The leaderboard builder selects the latest complete per-model batch matching +`protocol.toml`; it never mixes older single-run results or partial batches into +the five-trial leaderboard. GPT-5.6 Sol is routed directly through OpenCode's Azure Responses-API provider with reasoning effort `max`. BenchFlow still owns sandboxing, task staging, @@ -109,8 +151,14 @@ The builder writes: ranking displayed as `IQM + 100` so every current plotted value is positive. Each task maps its starter policy to `0` and its trusted article-level reference -to `100`. The final primary aggregate is the interquartile mean: remove the two -lowest and two highest of the nine normalized scores, then average the middle -five. Arithmetic mean and median remain secondary diagnostics. The top-level -repository README shows only the final image. The additive display offset does -not affect ranking or score gaps; raw IQM remains available in the JSON. +to `100`. The final leaderboard flattens the five-by-nine normalized score +matrix, removes the lowest 11 and highest 11 values, and averages the middle +23. Per-task sample standard deviations and the sample standard deviation of +the five trial-level IQMs remain visible diagnostics. The additive display +offset does not affect ranking or score gaps; raw IQM remains available in the +JSON. + +For task-defined fail-closed verifier timeouts where no native raw score exists, +the raw-score plot uses the frozen starter anchor as the equivalent of +normalized `0`. Trial metadata preserves the missing observation and labels the +display source explicitly. diff --git a/experiments/article_suite/models.toml b/experiments/article_suite/models.toml index a09287e..6b3a7c5 100644 --- a/experiments/article_suite/models.toml +++ b/experiments/article_suite/models.toml @@ -20,7 +20,7 @@ display_name = "Claude Opus 4.8" model = "anthropic/claude-opus-4-8" provider = "claude_oauth" provider_reasoning_effort = "max" -agent_idle_timeout_sec = 0 +agent_idle_timeout_sec = 3600 daytona_pty_readline_timeout_sec = 3600 [[models]] diff --git a/experiments/article_suite/protocol.toml b/experiments/article_suite/protocol.toml new file mode 100644 index 0000000..cbb5a6f --- /dev/null +++ b/experiments/article_suite/protocol.toml @@ -0,0 +1,21 @@ +version = "2.1" +trials = 5 +agent_timeout_multiplier = 3 + +[baseline_agent_timeout_sec] +simulation_heuristics_ant_v1 = 1800 +simulation_heuristics_pong_ram_v1 = 1800 +simulation_heuristics_breakout_ram_v1 = 1800 +simulation_heuristics_breakout_rgb_v1 = 1800 +simulation_heuristics_halfcheetah_v1 = 3600 +simulation_heuristics_vizdoom_d1_v1 = 1800 +simulation_heuristics_vizdoom_d3_v1 = 1800 +simulation_heuristics_atari57_v1 = 604800 +simulation_heuristics_montezuma_v1 = 1800 + +[aggregation] +task_score = "arithmetic_mean_across_trials" +task_variability = "sample_standard_deviation" +trial_score = "interquartile_mean_across_nine_tasks" +final_score = "interquartile_mean_across_all_trial_task_scores" +final_variability = "sample_standard_deviation_of_trial_level_iqms" diff --git a/leaderboard/ARTICLE_SUITE.md b/leaderboard/ARTICLE_SUITE.md index 780343d..ac334bb 100644 --- a/leaderboard/ARTICLE_SUITE.md +++ b/leaderboard/ARTICLE_SUITE.md @@ -2,15 +2,15 @@ The first image contains the nine independently ranked task leaderboards. The second image contains the final cross-task ranking. -The nine task panels use each environment's native raw score. The final scientific metric remains unbounded IQM. +The nine task panels report five-trial mean ± sample standard deviation for each environment's native raw score. -## Nine task leaderboards +## Leaderboards ![Nine task-specific GenesisBench leaderboards](article_suite_task_leaderboards.png) ## Final normalized score -The primary score is the interquartile mean (IQM): sort the nine task scores, remove the lowest two and highest two, then average the middle five. The image uses a plot-only positive display index equal to `IQM + 100`; raw IQM, arithmetic mean, and median remain in the JSON. +The final score is RLiable-style IQM over the complete 5 × 9 trial-task score matrix: flatten all 45 normalized scores, trim the lowest 11 and highest 11, and average the middle 23. The displayed ± value is the sample standard deviation of the five per-trial IQMs. The image uses a plot-only positive display index equal to `IQM + 100`; raw metrics remain in the JSON. ![Final GenesisBench article-suite leaderboard](article_suite_final_leaderboard.png) diff --git a/leaderboard/article_suite.json b/leaderboard/article_suite.json index b490741..37b8aa1 100644 --- a/leaderboard/article_suite.json +++ b/leaderboard/article_suite.json @@ -7,20 +7,39 @@ "ranking_field": "final_normalized_score", "type": "additive_offset" }, + "final_estimator": "iqm_across_all_45_trial_task_normalized_scores", + "final_variability": "sample_standard_deviation_of_five_trial_iqms", + "pooled_score_count": 45, "primary_field": "final_normalized_score", "primary_metric": "interquartile_mean", - "retained_score_count": 5, + "retained_score_count": 23, "score_bounds": "unbounded", "secondary_fields": [ "arithmetic_mean_normalized_score", - "median_normalized_score" + "median_normalized_score", + "mean_trial_iqm_normalized_score" ], + "task_estimator": "mean_across_five_trials", + "task_variability": "sample_standard_deviation", + "trial_estimator": "iqm_across_nine_tasks", "trim_fraction_per_tail": 0.25, - "trimmed_score_count_per_tail": 2, - "uncertainty": "not_estimated_single_run_per_model_task" + "trimmed_score_count_per_tail": 11 + }, + "batch_id": "per-model-batches", + "batch_ids": { + "claude-opus-4.8": "article-suite-v2-daytona-20260714-claude", + "gpt-5.4-mini": "article-suite-v2-daytona-20260714-gpt54mini", + "gpt-5.5": "article-suite-v2-daytona-20260714-gpt55", + "gpt-5.6-sol": "article-suite-v2-daytona-20260714-gpt56" }, "benchmark": "learning_beyond_gradients_article_suite", - "generated_at": "2026-07-13T15:34:22.260689+00:00", + "execution": { + "sandbox": "daytona", + "sandboxes": [ + "daytona" + ] + }, + "generated_at": "2026-07-15T12:56:18.690710+00:00", "inference_settings": { "cross_provider_comparability": "labels_are_not_a_shared_numeric_compute_scale", "field": "provider_reasoning_effort", @@ -28,12 +47,13 @@ "models": [ { "harness": "opencode", - "model": "GPT-5.5", - "model_id": "gpt-5.5", - "model_route": "azure/gpt-5.5", - "provider": "azure", - "provider_label": "Azure direct", - "provider_reasoning_effort": "xhigh" + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max", + "sandbox": "daytona" }, { "harness": "opencode", @@ -42,16 +62,18 @@ "model_route": "azure/gpt-5.6-sol", "provider": "azure", "provider_label": "Azure direct", - "provider_reasoning_effort": "max" + "provider_reasoning_effort": "max", + "sandbox": "daytona" }, { "harness": "opencode", - "model": "Claude Opus 4.8", - "model_id": "claude-opus-4.8", - "model_route": "anthropic/claude-opus-4-8", - "provider": "claude_oauth", - "provider_label": "Claude OAuth via pinned OpenCode plugin", - "provider_reasoning_effort": "max" + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "sandbox": "daytona" }, { "harness": "opencode", @@ -60,7 +82,8 @@ "model_route": "azure/gpt-5.4-mini", "provider": "azure", "provider_label": "Azure direct", - "provider_reasoning_effort": "xhigh" + "provider_reasoning_effort": "xhigh", + "sandbox": "daytona" } ] }, @@ -77,67 +100,111 @@ "rows": [ { "harness": "opencode", - "model": "GPT-5.6 Sol", - "model_id": "gpt-5.6-sol", - "model_route": "azure/gpt-5.6-sol", - "normalized_score": 39.203600385, - "provider": "azure", - "provider_label": "Azure direct", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "normalized_score": 1.820348169, + "normalized_score_stddev": 4.070422248601275, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 1, - "raw_score": 3815.360622145591, + "raw_score": 2507.501036532837, + "raw_score_imputation_count": 4, + "raw_score_stddev": 142.40443086118393, "reference_score": 5942.333244755838, - "source_run": "leaderboard/runs/article_suite/20260713T093815Z/gpt-5.6-sol", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], "starter_score": 2443.815838992282, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", - "model": "Claude Opus 4.8", - "model_id": "claude-opus-4.8", - "model_route": "anthropic/claude-opus-4-8", - "normalized_score": 14.050353128, - "provider": "claude_oauth", - "provider_label": "Claude OAuth via pinned OpenCode plugin", + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", + "normalized_score": -4.9009087386, + "normalized_score_stddev": 11.991465810903719, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "max", "rank": 2, - "raw_score": 2935.3698887625987, + "raw_score": 2272.356693724495, + "raw_score_imputation_count": 0, + "raw_score_stddev": 419.52351859557746, "reference_score": 5942.333244755838, - "source_run": "leaderboard/runs/article_suite/20260713T142919Z/claude-opus-4.8", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "starter_score": 2443.815838992282, - "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.5", "model_id": "gpt-5.5", "model_route": "azure/gpt-5.5", - "normalized_score": -16.891796576, + "normalized_score": -12.5116242462, + "normalized_score_stddev": 8.756962566015595, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 3, - "raw_score": 1852.8533956222025, + "raw_score": 2006.0944869926293, + "raw_score_imputation_count": 0, + "raw_score_stddev": 306.36385957656836, "reference_score": 5942.333244755838, - "source_run": "leaderboard/runs/article_suite/20260713T095807Z/gpt-5.5", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05/tasks-81ec8d98a4" + ], "starter_score": 2443.815838992282, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", "model_route": "azure/gpt-5.4-mini", - "normalized_score": -31.35954984, + "normalized_score": -47.232499900200004, + "normalized_score_stddev": 10.40778807635678, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 4, - "raw_score": 1346.6965294546999, + "raw_score": 791.3786088082715, + "raw_score_imputation_count": 0, + "raw_score_stddev": 364.1182774109385, "reference_score": 5942.333244755838, - "source_run": "leaderboard/runs/article_suite/20260713T104315Z/gpt-5.4-mini", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "starter_score": 2443.815838992282, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/score.json", + "trial_count": 5 } ] }, @@ -155,64 +222,108 @@ "model": "Claude Opus 4.8", "model_id": "claude-opus-4.8", "model_route": "anthropic/claude-opus-4-8", - "normalized_score": 100.0, + "normalized_score": 99.6666666666, + "normalized_score_stddev": 0.7453559926490017, "provider": "claude_oauth", "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 1, - "raw_score": 21.0, + "raw_score": 20.919999999999998, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.17888543819998254, "reference_score": 21.0, - "source_run": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], "starter_score": -2.9999999999999996, - "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.6 Sol", "model_id": "gpt-5.6-sol", "model_route": "azure/gpt-5.6-sol", - "normalized_score": 52.916666667, + "normalized_score": 91.6666666666, + "normalized_score_stddev": 9.771699499561077, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "max", "rank": 2, - "raw_score": 9.7, + "raw_score": 19.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 2.345207879911715, "reference_score": 21.0, - "source_run": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "starter_score": -2.9999999999999996, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.5", "model_id": "gpt-5.5", "model_route": "azure/gpt-5.5", - "normalized_score": 45.833333333, + "normalized_score": 49.1666666668, + "normalized_score_stddev": 40.44028519521613, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 3, - "raw_score": 8.0, + "raw_score": 8.8, + "raw_score_imputation_count": 0, + "raw_score_stddev": 9.705668446840743, "reference_score": 21.0, - "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-885696cfa4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], "starter_score": -2.9999999999999996, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", "model_route": "azure/gpt-5.4-mini", - "normalized_score": -75.0, + "normalized_score": -40.8333333332, + "normalized_score_stddev": 67.46912874308083, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 4, - "raw_score": -21.0, + "raw_score": -12.8, + "raw_score_imputation_count": 0, + "raw_score_stddev": 16.192590898309017, "reference_score": 21.0, - "source_run": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "starter_score": -2.9999999999999996, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/score.json", + "trial_count": 5 } ] }, @@ -225,69 +336,113 @@ "unit": "points" }, "rows": [ + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "normalized_score": 94.004192872, + "normalized_score_stddev": 7.012257406411853, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max", + "rank": 1, + "raw_score": 835.4, + "raw_score_imputation_count": 0, + "raw_score_stddev": 33.448467827390836, + "reference_score": 864.0, + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "starter_score": 387.0, + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/score.json", + "trial_count": 5 + }, { "harness": "opencode", "model": "GPT-5.6 Sol", "model_id": "gpt-5.6-sol", "model_route": "azure/gpt-5.6-sol", - "normalized_score": 100.0, + "normalized_score": 43.899371069400004, + "normalized_score_stddev": 47.34804435501101, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "max", - "rank": 1, - "raw_score": 864.0, + "rank": 2, + "raw_score": 596.4, + "raw_score_imputation_count": 0, + "raw_score_stddev": 225.8501715739884, "reference_score": 864.0, - "source_run": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "starter_score": 387.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.5", "model_id": "gpt-5.5", "model_route": "azure/gpt-5.5", - "normalized_score": 95.597484277, + "normalized_score": 20.9224318658, + "normalized_score_stddev": 27.49546024414613, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", - "rank": 2, - "raw_score": 843.0, + "rank": 3, + "raw_score": 486.8, + "raw_score_imputation_count": 0, + "raw_score_stddev": 131.153345363357, "reference_score": 864.0, - "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-e6fa3546aa", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], "starter_score": 387.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", "model_route": "azure/gpt-5.4-mini", - "normalized_score": 2.51572327, + "normalized_score": -1.3417190774, + "normalized_score_stddev": 9.228605826577622, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", - "rank": 3, - "raw_score": 399.0, - "reference_score": 864.0, - "source_run": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", - "starter_score": 387.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/score.json" - }, - { - "harness": "opencode", - "model": "Claude Opus 4.8", - "model_id": "claude-opus-4.8", - "model_route": "anthropic/claude-opus-4-8", - "normalized_score": 0.0, - "provider": "claude_oauth", - "provider_label": "Claude OAuth via pinned OpenCode plugin", - "provider_reasoning_effort": "max", "rank": 4, - "raw_score": 387.0, + "raw_score": 380.6, + "raw_score_imputation_count": 0, + "raw_score_stddev": 44.0204497932495, "reference_score": 864.0, - "source_run": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02/simulation_heuristics_breakout_ram_v1", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "starter_score": 387.0, - "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/score.json", + "trial_count": 5 } ] }, @@ -300,69 +455,113 @@ "unit": "points" }, "rows": [ + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "normalized_score": 89.2779783394, + "normalized_score_stddev": 13.287454681897577, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max", + "rank": 1, + "raw_score": 804.6, + "raw_score_imputation_count": 0, + "raw_score_stddev": 73.61249893869926, + "reference_score": 864.0, + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "starter_score": 310.0, + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/score.json", + "trial_count": 5 + }, { "harness": "opencode", "model": "GPT-5.5", "model_id": "gpt-5.5", "model_route": "azure/gpt-5.5", - "normalized_score": 70.758122744, + "normalized_score": 60.86642599300001, + "normalized_score_stddev": 22.118506348049813, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", - "rank": 1, - "raw_score": 702.0, + "rank": 2, + "raw_score": 647.2, + "raw_score_imputation_count": 0, + "raw_score_stddev": 122.53652516698847, "reference_score": 864.0, - "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-885696cfa4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], "starter_score": 310.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.6 Sol", "model_id": "gpt-5.6-sol", "model_route": "azure/gpt-5.6-sol", - "normalized_score": 70.758122744, + "normalized_score": 60.86642599300001, + "normalized_score_stddev": 22.118506348049813, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "max", - "rank": 1, - "raw_score": 702.0, + "rank": 2, + "raw_score": 647.2, + "raw_score_imputation_count": 0, + "raw_score_stddev": 122.53652516698847, "reference_score": 864.0, - "source_run": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "starter_score": 310.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", "model_route": "azure/gpt-5.4-mini", - "normalized_score": 15.523465704, + "normalized_score": 14.332129963999998, + "normalized_score_stddev": 51.9448986087478, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", - "rank": 3, - "raw_score": 396.0, - "reference_score": 864.0, - "source_run": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", - "starter_score": 310.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/score.json" - }, - { - "harness": "opencode", - "model": "Claude Opus 4.8", - "model_id": "claude-opus-4.8", - "model_route": "anthropic/claude-opus-4-8", - "normalized_score": 0.0, - "provider": "claude_oauth", - "provider_label": "Claude OAuth via pinned OpenCode plugin", - "provider_reasoning_effort": "max", "rank": 4, - "raw_score": 310.0, + "raw_score": 389.4, + "raw_score_imputation_count": 0, + "raw_score_stddev": 287.77473829368694, "reference_score": 864.0, - "source_run": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "starter_score": 310.0, - "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/score.json", + "trial_count": 5 } ] }, @@ -380,64 +579,108 @@ "model": "Claude Opus 4.8", "model_id": "claude-opus-4.8", "model_route": "anthropic/claude-opus-4-8", - "normalized_score": 26.986707946, + "normalized_score": 40.934675074999994, + "normalized_score_stddev": 24.40596858830019, "provider": "claude_oauth", "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 1, - "raw_score": 6021.504892591717, + "raw_score": 7065.645225601496, + "raw_score_imputation_count": 0, + "raw_score_stddev": 1827.0229584820154, "reference_score": 11487.256451471967, - "source_run": "leaderboard/runs/article_suite/20260713T111714Z/claude-opus-4.8", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], "starter_score": 4001.2886158917113, - "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.6 Sol", "model_id": "gpt-5.6-sol", "model_route": "azure/gpt-5.6-sol", - "normalized_score": 15.873072595, + "normalized_score": 31.8442100886, + "normalized_score_stddev": 3.8873875677992533, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "max", "rank": 2, - "raw_score": 5189.541724894487, + "raw_score": 6385.135940620764, + "raw_score_imputation_count": 0, + "raw_score_stddev": 291.0085829715705, "reference_score": 11487.256451471967, - "source_run": "leaderboard/runs/article_suite/20260713T084340Z/gpt-5.6-sol", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "starter_score": 4001.2886158917113, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.5", "model_id": "gpt-5.5", "model_route": "azure/gpt-5.5", - "normalized_score": 2.73755349, + "normalized_score": 9.5689381094, + "normalized_score_stddev": 9.330884765209642, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 3, - "raw_score": 4206.220989602626, + "raw_score": 4717.6162449605235, + "raw_score_imputation_count": 0, + "raw_score_stddev": 698.5070322948952, "reference_score": 11487.256451471967, - "source_run": "leaderboard/runs/article_suite/20260713T103810Z/gpt-5.5", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01/simulation_heuristics_halfcheetah_v1", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-e6fa3546aa", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05/tasks-81ec8d98a4" + ], "starter_score": 4001.2886158917113, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", "model_route": "azure/gpt-5.4-mini", - "normalized_score": -8.167582379, + "normalized_score": -39.8837988972, + "normalized_score_stddev": 19.209845675650264, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 4, - "raw_score": 3389.8660260183765, + "raw_score": 1015.6002588284422, + "raw_score_imputation_count": 0, + "raw_score_stddev": 1438.0428685581014, "reference_score": 11487.256451471967, - "source_run": "leaderboard/runs/article_suite/20260713T093314Z/gpt-5.4-mini", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "starter_score": 4001.2886158917113, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/score.json", + "trial_count": 5 } ] }, @@ -455,64 +698,108 @@ "model": "Claude Opus 4.8", "model_id": "claude-opus-4.8", "model_route": "anthropic/claude-opus-4-8", - "normalized_score": 95.055812742, + "normalized_score": 107.1451288772, + "normalized_score_stddev": 14.745599637347764, "provider": "claude_oauth", "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 1, - "raw_score": 0.7992199431359768, - "reference_score": 0.8178199748694897, - "source_run": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "raw_score": 0.8446999486498534, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.05547294365604739, + "reference_score": 0.8178199748694898, + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], "starter_score": 0.44161998964846133, - "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", - "model": "GPT-5.5", - "model_id": "gpt-5.5", - "model_route": "azure/gpt-5.5", - "normalized_score": 85.199359782, + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", + "normalized_score": 56.834663982799995, + "normalized_score_stddev": 33.111283305681766, "provider": "azure", "provider_label": "Azure direct", - "provider_reasoning_effort": "xhigh", + "provider_reasoning_effort": "max", "rank": 2, - "raw_score": 0.7621399685554205, - "reference_score": 0.8178199748694897, - "source_run": "leaderboard/runs/article_suite/20260713T102218Z/gpt-5.5", + "raw_score": 0.6554319871515035, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.12456464290218099, + "reference_score": 0.8178199748694898, + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "starter_score": 0.44161998964846133, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", - "model": "GPT-5.6 Sol", - "model_id": "gpt-5.6-sol", - "model_route": "azure/gpt-5.6-sol", - "normalized_score": 49.409889366, + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "normalized_score": 47.33864911959999, + "normalized_score_stddev": 95.13699952993868, "provider": "azure", "provider_label": "Azure direct", - "provider_reasoning_effort": "max", + "provider_reasoning_effort": "xhigh", "rank": 3, - "raw_score": 0.62749998614192, - "reference_score": 0.8178199748694897, - "source_run": "leaderboard/runs/article_suite/20260713T101636Z/gpt-5.6-sol", + "raw_score": 0.6197079806402326, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.3579053781713541, + "reference_score": 0.8178199748694898, + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], "starter_score": 0.44161998964846133, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", "model_route": "azure/gpt-5.4-mini", - "normalized_score": -161.116433137, + "normalized_score": -186.0627378054, + "normalized_score_stddev": 3.359451433761567, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 4, - "raw_score": -0.16450000800192355, - "reference_score": 0.8178199748694897, - "source_run": "leaderboard/runs/article_suite/20260713T102707Z/gpt-5.4-mini", + "raw_score": -0.25834800247848033, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.012638255798059739, + "reference_score": 0.8178199748694898, + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "starter_score": 0.44161998964846133, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/score.json", + "trial_count": 5 } ] }, @@ -530,64 +817,108 @@ "model": "Claude Opus 4.8", "model_id": "claude-opus-4.8", "model_route": "anthropic/claude-opus-4-8", - "normalized_score": 122.259358289, + "normalized_score": 96.4371657756, + "normalized_score_stddev": 42.45851366513837, "provider": "claude_oauth", "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 1, - "raw_score": 397.7, + "raw_score": 320.44, + "raw_score_imputation_count": 0, + "raw_score_stddev": 127.03587288636228, "reference_score": 331.1, - "source_run": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], "starter_score": 31.9, - "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", - "model": "GPT-5.6 Sol", - "model_id": "gpt-5.6-sol", - "model_route": "azure/gpt-5.6-sol", - "normalized_score": 26.236631016, + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "normalized_score": 23.2620320856, + "normalized_score_stddev": 41.04765230004701, "provider": "azure", "provider_label": "Azure direct", - "provider_reasoning_effort": "max", + "provider_reasoning_effort": "xhigh", "rank": 2, - "raw_score": 110.39999999999998, + "raw_score": 101.49999999999999, + "raw_score_imputation_count": 0, + "raw_score_stddev": 122.81457568220475, "reference_score": 331.1, - "source_run": "leaderboard/runs/article_suite/20260713T083940Z/gpt-5.6-sol", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], "starter_score": 31.9, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", - "model": "GPT-5.5", - "model_id": "gpt-5.5", - "model_route": "azure/gpt-5.5", - "normalized_score": 5.481283422, + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", + "normalized_score": 16.5307486632, + "normalized_score_stddev": 45.27484310055022, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 3, - "raw_score": 48.3, + "raw_score": 81.36, + "raw_score_imputation_count": 0, + "raw_score_stddev": 135.46233055724386, "reference_score": 331.1, - "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01/simulation_heuristics_vizdoom_d3_v1", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "starter_score": 31.9, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", - "model": "GPT-5.4 Mini", - "model_id": "gpt-5.4-mini", - "model_route": "azure/gpt-5.4-mini", - "normalized_score": -9.859625668, + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", + "normalized_score": 16.3034759358, + "normalized_score_stddev": 23.995771403261536, "provider": "azure", "provider_label": "Azure direct", - "provider_reasoning_effort": "xhigh", + "provider_reasoning_effort": "max", "rank": 4, - "raw_score": 2.4, + "raw_score": 80.67999999999999, + "raw_score_imputation_count": 0, + "raw_score_stddev": 71.79534803871347, "reference_score": 331.1, - "source_run": "leaderboard/runs/article_suite/20260713T104034Z/gpt-5.4-mini", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "starter_score": 31.9, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/score.json", + "trial_count": 5 } ] }, @@ -606,15 +937,26 @@ "model_id": "claude-opus-4.8", "model_route": "anthropic/claude-opus-4-8", "normalized_score": 0.0, + "normalized_score_stddev": 0.0, "provider": "claude_oauth", "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 1, "raw_score": 0.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.0, "reference_score": 0.8283015254994576, - "source_run": "leaderboard/runs/article_suite/20260713T123222Z/claude-opus-4.8", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], "starter_score": 0.0, - "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", @@ -622,15 +964,26 @@ "model_id": "gpt-5.4-mini", "model_route": "azure/gpt-5.4-mini", "normalized_score": 0.0, + "normalized_score_stddev": 0.0, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 1, "raw_score": 0.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.0, "reference_score": 0.8283015254994576, - "source_run": "leaderboard/runs/article_suite/20260713T111944Z/gpt-5.4-mini", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "starter_score": 0.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", @@ -638,15 +991,26 @@ "model_id": "gpt-5.5", "model_route": "azure/gpt-5.5", "normalized_score": 0.0, + "normalized_score_stddev": 0.0, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 1, "raw_score": 0.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.0, "reference_score": 0.8283015254994576, - "source_run": "leaderboard/runs/article_suite/20260713T111604Z/gpt-5.5", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], "starter_score": 0.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", @@ -654,15 +1018,26 @@ "model_id": "gpt-5.6-sol", "model_route": "azure/gpt-5.6-sol", "normalized_score": 0.0, + "normalized_score_stddev": 0.0, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "max", "rank": 1, "raw_score": 0.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.0, "reference_score": 0.8283015254994576, - "source_run": "leaderboard/runs/article_suite/20260713T111156Z/gpt-5.6-sol", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "starter_score": 0.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/score.json", + "trial_count": 5 } ] }, @@ -677,19 +1052,30 @@ "rows": [ { "harness": "opencode", - "model": "GPT-5.5", - "model_id": "gpt-5.5", - "model_route": "azure/gpt-5.5", - "normalized_score": 100.0, + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", + "normalized_score": 74.5, + "normalized_score_stddev": 43.31570154112709, "provider": "azure", "provider_label": "Azure direct", - "provider_reasoning_effort": "xhigh", + "provider_reasoning_effort": "max", "rank": 1, - "raw_score": 400.0, + "raw_score": 298.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 173.26280616450836, "reference_score": 400.0, - "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "starter_score": 0.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", @@ -697,15 +1083,26 @@ "model_id": "claude-opus-4.8", "model_route": "anthropic/claude-opus-4-8", "normalized_score": 0.0, + "normalized_score_stddev": 0.0, "provider": "claude_oauth", "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 2, "raw_score": 0.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.0, "reference_score": 400.0, - "source_run": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], "starter_score": 0.0, - "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", @@ -713,31 +1110,53 @@ "model_id": "gpt-5.4-mini", "model_route": "azure/gpt-5.4-mini", "normalized_score": 0.0, + "normalized_score_stddev": 0.0, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 2, "raw_score": 0.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.0, "reference_score": 400.0, - "source_run": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "starter_score": 0.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", - "model": "GPT-5.6 Sol", - "model_id": "gpt-5.6-sol", - "model_route": "azure/gpt-5.6-sol", + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", "normalized_score": 0.0, + "normalized_score_stddev": 0.0, "provider": "azure", "provider_label": "Azure direct", - "provider_reasoning_effort": "max", + "provider_reasoning_effort": "xhigh", "rank": 2, "raw_score": 0.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.0, "reference_score": 400.0, - "source_run": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], "starter_score": 0.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/score.json", + "trial_count": 5 } ] }, @@ -754,114 +1173,232 @@ "metric": "final_normalized_score", "rows": [ { - "arithmetic_mean_normalized_score": 43.19059338577778, - "average_normalized_score": 43.19059338577778, - "final_normalized_score": 42.0019305542, + "arithmetic_mean_normalized_score": 58.80957286386666, + "average_normalized_score": 58.80957286386666, + "final_normalized_score": 62.03374323269565, + "final_normalized_score_stddev": 10.38022421100604, "harness": "opencode", - "median_normalized_score": 45.833333333, - "model": "GPT-5.5", - "model_id": "gpt-5.5", - "model_route": "azure/gpt-5.5", - "positive_display_score": 142.0019305542, - "provider": "azure", - "provider_label": "Azure direct", - "provider_reasoning_effort": "xhigh", - "rank": 1 + "mean_trial_iqm_normalized_score": 61.137710440759996, + "median_normalized_score": 79.783393502, + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "positive_display_score": 162.03374323269566, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max", + "rank": 1, + "sandbox": "daytona", + "trial_count": 5 }, { - "arithmetic_mean_normalized_score": 39.37755364144445, - "average_normalized_score": 39.37755364144445, - "final_normalized_score": 36.7279720058, + "arithmetic_mean_normalized_score": 41.223767221955555, + "average_normalized_score": 41.223767221955555, + "final_normalized_score": 36.651764607913044, + "final_normalized_score_stddev": 5.524217450335815, "harness": "opencode", - "median_normalized_score": 39.203600385, + "mean_trial_iqm_normalized_score": 37.73164397808, + "median_normalized_score": 32.792734613, "model": "GPT-5.6 Sol", "model_id": "gpt-5.6-sol", "model_route": "azure/gpt-5.6-sol", - "positive_display_score": 136.72797200579998, + "positive_display_score": 136.65176460791304, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "max", - "rank": 2 + "rank": 2, + "sandbox": "daytona", + "trial_count": 5 }, { - "arithmetic_mean_normalized_score": 39.81691467833333, - "average_normalized_score": 39.81691467833333, - "final_normalized_score": 27.218574763200003, + "arithmetic_mean_normalized_score": 22.06816884377778, + "average_normalized_score": 22.06816884377778, + "final_normalized_score": 10.662970787652174, + "final_normalized_score_stddev": 18.67984640949238, "harness": "opencode", - "median_normalized_score": 14.050353128, - "model": "Claude Opus 4.8", - "model_id": "claude-opus-4.8", - "model_route": "anthropic/claude-opus-4-8", - "positive_display_score": 127.2185747632, - "provider": "claude_oauth", - "provider_label": "Claude OAuth via pinned OpenCode plugin", - "provider_reasoning_effort": "max", - "rank": 3 + "mean_trial_iqm_normalized_score": 16.7775760408, + "median_normalized_score": 5.870020964, + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "positive_display_score": 110.66297078765217, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 3, + "sandbox": "daytona", + "trial_count": 5 }, { - "arithmetic_mean_normalized_score": -29.71822245, - "average_normalized_score": -29.71822245, - "final_normalized_score": -9.8773515774, + "arithmetic_mean_normalized_score": -31.610134487355555, + "average_normalized_score": -31.610134487355555, + "final_normalized_score": -17.411123288956524, + "final_normalized_score_stddev": 5.902995817222085, "harness": "opencode", - "median_normalized_score": -8.167582379, + "mean_trial_iqm_normalized_score": -18.58872338544, + "median_normalized_score": -3.609625668, "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", "model_route": "azure/gpt-5.4-mini", - "positive_display_score": 90.1226484226, + "positive_display_score": 82.58887671104347, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", - "rank": 4 + "rank": 4, + "sandbox": "daytona", + "trial_count": 5 } ] } ], + "protocol": { + "agent_timeout_multiplier": 3, + "aggregation": { + "final_score": "interquartile_mean_across_all_trial_task_scores", + "final_variability": "sample_standard_deviation_of_trial_level_iqms", + "task_score": "arithmetic_mean_across_trials", + "task_variability": "sample_standard_deviation", + "trial_score": "interquartile_mean_across_nine_tasks" + }, + "baseline_agent_timeout_sec": { + "simulation_heuristics_ant_v1": 1800, + "simulation_heuristics_atari57_v1": 604800, + "simulation_heuristics_breakout_ram_v1": 1800, + "simulation_heuristics_breakout_rgb_v1": 1800, + "simulation_heuristics_halfcheetah_v1": 3600, + "simulation_heuristics_montezuma_v1": 1800, + "simulation_heuristics_pong_ram_v1": 1800, + "simulation_heuristics_vizdoom_d1_v1": 1800, + "simulation_heuristics_vizdoom_d3_v1": 1800 + }, + "trials": 5, + "version": "2.1" + }, "rows": [ { - "arithmetic_mean_normalized_score": 43.19059338577778, - "average_normalized_score": 43.19059338577778, - "final_normalized_score": 42.0019305542, + "arithmetic_mean_normalized_score": 58.80957286386666, + "average_normalized_score": 58.80957286386666, + "final_normalized_score": 62.03374323269565, + "final_normalized_score_stddev": 10.38022421100604, "harness": "opencode", - "median_normalized_score": 45.833333333, - "model": "GPT-5.5", - "model_id": "gpt-5.5", - "model_route": "azure/gpt-5.5", - "provider": "azure", - "provider_label": "Azure direct", - "provider_reasoning_effort": "xhigh", + "mean_trial_iqm_normalized_score": 61.137710440759996, + "median_normalized_score": 79.783393502, + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max", "rank": 1, + "raw_task_score_imputation_counts": { + "simulation_heuristics_ant_v1": 4, + "simulation_heuristics_atari57_v1": 0, + "simulation_heuristics_breakout_ram_v1": 0, + "simulation_heuristics_breakout_rgb_v1": 0, + "simulation_heuristics_halfcheetah_v1": 0, + "simulation_heuristics_montezuma_v1": 0, + "simulation_heuristics_pong_ram_v1": 0, + "simulation_heuristics_vizdoom_d1_v1": 0, + "simulation_heuristics_vizdoom_d3_v1": 0 + }, + "raw_task_score_stddevs": { + "simulation_heuristics_ant_v1": 142.40443086118393, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 33.448467827390836, + "simulation_heuristics_breakout_rgb_v1": 73.61249893869926, + "simulation_heuristics_halfcheetah_v1": 1827.0229584820154, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 0.17888543819998254, + "simulation_heuristics_vizdoom_d1_v1": 0.05547294365604739, + "simulation_heuristics_vizdoom_d3_v1": 127.03587288636228 + }, "raw_task_scores": { - "simulation_heuristics_ant_v1": 1852.8533956222025, + "simulation_heuristics_ant_v1": 2507.501036532837, "simulation_heuristics_atari57_v1": 0.0, - "simulation_heuristics_breakout_ram_v1": 843.0, - "simulation_heuristics_breakout_rgb_v1": 702.0, - "simulation_heuristics_halfcheetah_v1": 4206.220989602626, - "simulation_heuristics_montezuma_v1": 400.0, - "simulation_heuristics_pong_ram_v1": 8.0, - "simulation_heuristics_vizdoom_d1_v1": 0.7621399685554205, - "simulation_heuristics_vizdoom_d3_v1": 48.3 + "simulation_heuristics_breakout_ram_v1": 835.4, + "simulation_heuristics_breakout_rgb_v1": 804.6, + "simulation_heuristics_halfcheetah_v1": 7065.645225601496, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 20.919999999999998, + "simulation_heuristics_vizdoom_d1_v1": 0.8446999486498534, + "simulation_heuristics_vizdoom_d3_v1": 320.44 }, + "sandbox": "daytona", "source_runs": { - "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T095807Z/gpt-5.5", - "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T111604Z/gpt-5.5", - "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", - "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", - "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T103810Z/gpt-5.5", - "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", - "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", - "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T102218Z/gpt-5.5", - "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5" + "simulation_heuristics_ant_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_atari57_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_breakout_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_breakout_rgb_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_halfcheetah_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_montezuma_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_pong_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_vizdoom_d1_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_vizdoom_d3_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ] }, "submission_details": { - "simulation_heuristics_ant_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/score.json", - "simulation_heuristics_atari57_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/score.json", - "simulation_heuristics_breakout_ram_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/score.json", - "simulation_heuristics_breakout_rgb_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/score.json", - "simulation_heuristics_halfcheetah_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/score.json", - "simulation_heuristics_montezuma_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/score.json", - "simulation_heuristics_pong_ram_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/score.json", - "simulation_heuristics_vizdoom_d1_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/score.json", - "simulation_heuristics_vizdoom_d3_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/score.json" + "simulation_heuristics_ant_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/score.json", + "simulation_heuristics_atari57_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/score.json", + "simulation_heuristics_breakout_ram_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/score.json", + "simulation_heuristics_breakout_rgb_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/score.json", + "simulation_heuristics_halfcheetah_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/score.json", + "simulation_heuristics_montezuma_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/score.json", + "simulation_heuristics_pong_ram_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/score.json", + "simulation_heuristics_vizdoom_d1_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/score.json", + "simulation_heuristics_vizdoom_d3_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/score.json" }, "task_anchors": { "simulation_heuristics_ant_v1": { @@ -893,7 +1430,7 @@ "starter_score": -2.9999999999999996 }, "simulation_heuristics_vizdoom_d1_v1": { - "reference_score": 0.8178199748694897, + "reference_score": 0.8178199748694898, "starter_score": 0.44161998964846133 }, "simulation_heuristics_vizdoom_d3_v1": { @@ -901,24 +1438,102 @@ "starter_score": 31.9 } }, + "task_score_stddevs": { + "simulation_heuristics_ant_v1": 4.070422248601275, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 7.012257406411853, + "simulation_heuristics_breakout_rgb_v1": 13.287454681897577, + "simulation_heuristics_halfcheetah_v1": 24.40596858830019, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 0.7453559926490017, + "simulation_heuristics_vizdoom_d1_v1": 14.745599637347764, + "simulation_heuristics_vizdoom_d3_v1": 42.45851366513837 + }, "task_scores": { - "simulation_heuristics_ant_v1": -16.891796576, + "simulation_heuristics_ant_v1": 1.820348169, "simulation_heuristics_atari57_v1": 0.0, - "simulation_heuristics_breakout_ram_v1": 95.597484277, - "simulation_heuristics_breakout_rgb_v1": 70.758122744, - "simulation_heuristics_halfcheetah_v1": 2.73755349, - "simulation_heuristics_montezuma_v1": 100.0, - "simulation_heuristics_pong_ram_v1": 45.833333333, - "simulation_heuristics_vizdoom_d1_v1": 85.199359782, - "simulation_heuristics_vizdoom_d3_v1": 5.481283422 + "simulation_heuristics_breakout_ram_v1": 94.004192872, + "simulation_heuristics_breakout_rgb_v1": 89.2779783394, + "simulation_heuristics_halfcheetah_v1": 40.934675074999994, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 99.6666666666, + "simulation_heuristics_vizdoom_d1_v1": 107.1451288772, + "simulation_heuristics_vizdoom_d3_v1": 96.4371657756 + }, + "trial_count": 5, + "trial_final_normalized_scores": { + "1": 59.1255877926, + "2": 66.2620495092, + "3": 44.389717685200004, + "4": 71.6451968772, + "5": 64.2660003396 + }, + "trial_task_normalized_scores": { + "1": { + "simulation_heuristics_ant_v1": 0.0, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 98.742138365, + "simulation_heuristics_breakout_rgb_v1": 70.758122744, + "simulation_heuristics_halfcheetah_v1": 41.669121704, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 100.0, + "simulation_heuristics_vizdoom_d1_v1": 124.779364311, + "simulation_heuristics_vizdoom_d3_v1": 84.45855615 + }, + "2": { + "simulation_heuristics_ant_v1": 9.101740845, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 100.0, + "simulation_heuristics_breakout_rgb_v1": 79.783393502, + "simulation_heuristics_halfcheetah_v1": 42.425113199, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 100.0, + "simulation_heuristics_vizdoom_d1_v1": 100.2179605, + "simulation_heuristics_vizdoom_d3_v1": 114.104278075 + }, + "3": { + "simulation_heuristics_ant_v1": 0.0, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 98.322851153, + "simulation_heuristics_breakout_rgb_v1": 100.0, + "simulation_heuristics_halfcheetah_v1": 9.827504681, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 98.333333333, + "simulation_heuristics_vizdoom_d1_v1": 85.890478581, + "simulation_heuristics_vizdoom_d3_v1": 27.907754011 + }, + "4": { + "simulation_heuristics_ant_v1": 0.0, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 84.696016771, + "simulation_heuristics_breakout_rgb_v1": 95.848375451, + "simulation_heuristics_halfcheetah_v1": 77.681592164, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 100.0, + "simulation_heuristics_vizdoom_d1_v1": 113.721417769, + "simulation_heuristics_vizdoom_d3_v1": 134.35828877 + }, + "5": { + "simulation_heuristics_ant_v1": 0.0, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 88.259958071, + "simulation_heuristics_breakout_rgb_v1": 100.0, + "simulation_heuristics_halfcheetah_v1": 33.070043627, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 100.0, + "simulation_heuristics_vizdoom_d1_v1": 111.116423225, + "simulation_heuristics_vizdoom_d3_v1": 121.356951872 + } } }, { - "arithmetic_mean_normalized_score": 39.37755364144445, - "average_normalized_score": 39.37755364144445, - "final_normalized_score": 36.7279720058, + "arithmetic_mean_normalized_score": 41.223767221955555, + "average_normalized_score": 41.223767221955555, + "final_normalized_score": 36.651764607913044, + "final_normalized_score_stddev": 5.524217450335815, "harness": "opencode", - "median_normalized_score": 39.203600385, + "mean_trial_iqm_normalized_score": 37.73164397808, + "median_normalized_score": 32.792734613, "model": "GPT-5.6 Sol", "model_id": "gpt-5.6-sol", "model_route": "azure/gpt-5.6-sol", @@ -926,27 +1541,104 @@ "provider_label": "Azure direct", "provider_reasoning_effort": "max", "rank": 2, + "raw_task_score_imputation_counts": { + "simulation_heuristics_ant_v1": 0, + "simulation_heuristics_atari57_v1": 0, + "simulation_heuristics_breakout_ram_v1": 0, + "simulation_heuristics_breakout_rgb_v1": 0, + "simulation_heuristics_halfcheetah_v1": 0, + "simulation_heuristics_montezuma_v1": 0, + "simulation_heuristics_pong_ram_v1": 0, + "simulation_heuristics_vizdoom_d1_v1": 0, + "simulation_heuristics_vizdoom_d3_v1": 0 + }, + "raw_task_score_stddevs": { + "simulation_heuristics_ant_v1": 419.52351859557746, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 225.8501715739884, + "simulation_heuristics_breakout_rgb_v1": 122.53652516698847, + "simulation_heuristics_halfcheetah_v1": 291.0085829715705, + "simulation_heuristics_montezuma_v1": 173.26280616450836, + "simulation_heuristics_pong_ram_v1": 2.345207879911715, + "simulation_heuristics_vizdoom_d1_v1": 0.12456464290218099, + "simulation_heuristics_vizdoom_d3_v1": 71.79534803871347 + }, "raw_task_scores": { - "simulation_heuristics_ant_v1": 3815.360622145591, + "simulation_heuristics_ant_v1": 2272.356693724495, "simulation_heuristics_atari57_v1": 0.0, - "simulation_heuristics_breakout_ram_v1": 864.0, - "simulation_heuristics_breakout_rgb_v1": 702.0, - "simulation_heuristics_halfcheetah_v1": 5189.541724894487, - "simulation_heuristics_montezuma_v1": 0.0, - "simulation_heuristics_pong_ram_v1": 9.7, - "simulation_heuristics_vizdoom_d1_v1": 0.62749998614192, - "simulation_heuristics_vizdoom_d3_v1": 110.39999999999998 + "simulation_heuristics_breakout_ram_v1": 596.4, + "simulation_heuristics_breakout_rgb_v1": 647.2, + "simulation_heuristics_halfcheetah_v1": 6385.135940620764, + "simulation_heuristics_montezuma_v1": 298.0, + "simulation_heuristics_pong_ram_v1": 19.0, + "simulation_heuristics_vizdoom_d1_v1": 0.6554319871515035, + "simulation_heuristics_vizdoom_d3_v1": 80.67999999999999 }, + "sandbox": "daytona", "source_runs": { - "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T093815Z/gpt-5.6-sol", - "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T111156Z/gpt-5.6-sol", - "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", - "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", - "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T084340Z/gpt-5.6-sol", - "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", - "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", - "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T101636Z/gpt-5.6-sol", - "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T083940Z/gpt-5.6-sol" + "simulation_heuristics_ant_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_atari57_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_breakout_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_breakout_rgb_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_halfcheetah_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_montezuma_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_pong_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_vizdoom_d1_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_vizdoom_d3_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ] }, "submission_details": { "simulation_heuristics_ant_v1": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/score.json", @@ -989,7 +1681,7 @@ "starter_score": -2.9999999999999996 }, "simulation_heuristics_vizdoom_d1_v1": { - "reference_score": 0.8178199748694897, + "reference_score": 0.8178199748694898, "starter_score": 0.44161998964846133 }, "simulation_heuristics_vizdoom_d3_v1": { @@ -997,63 +1689,218 @@ "starter_score": 31.9 } }, + "task_score_stddevs": { + "simulation_heuristics_ant_v1": 11.991465810903719, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 47.34804435501101, + "simulation_heuristics_breakout_rgb_v1": 22.118506348049813, + "simulation_heuristics_halfcheetah_v1": 3.8873875677992533, + "simulation_heuristics_montezuma_v1": 43.31570154112709, + "simulation_heuristics_pong_ram_v1": 9.771699499561077, + "simulation_heuristics_vizdoom_d1_v1": 33.111283305681766, + "simulation_heuristics_vizdoom_d3_v1": 23.995771403261536 + }, "task_scores": { - "simulation_heuristics_ant_v1": 39.203600385, + "simulation_heuristics_ant_v1": -4.9009087386, "simulation_heuristics_atari57_v1": 0.0, - "simulation_heuristics_breakout_ram_v1": 100.0, - "simulation_heuristics_breakout_rgb_v1": 70.758122744, - "simulation_heuristics_halfcheetah_v1": 15.873072595, - "simulation_heuristics_montezuma_v1": 0.0, - "simulation_heuristics_pong_ram_v1": 52.916666667, - "simulation_heuristics_vizdoom_d1_v1": 49.409889366, - "simulation_heuristics_vizdoom_d3_v1": 26.236631016 + "simulation_heuristics_breakout_ram_v1": 43.899371069400004, + "simulation_heuristics_breakout_rgb_v1": 60.86642599300001, + "simulation_heuristics_halfcheetah_v1": 31.8442100886, + "simulation_heuristics_montezuma_v1": 74.5, + "simulation_heuristics_pong_ram_v1": 91.6666666666, + "simulation_heuristics_vizdoom_d1_v1": 56.834663982799995, + "simulation_heuristics_vizdoom_d3_v1": 16.3034759358 + }, + "trial_count": 5, + "trial_final_normalized_scores": { + "1": 37.797729594399996, + "2": 41.203895079, + "3": 28.388586371000002, + "4": 42.346736302800004, + "5": 38.921272543200004 + }, + "trial_task_normalized_scores": { + "1": { + "simulation_heuristics_ant_v1": 9.173888471, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 100.0, + "simulation_heuristics_breakout_rgb_v1": 70.758122744, + "simulation_heuristics_halfcheetah_v1": 26.12140084, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 79.166666667, + "simulation_heuristics_vizdoom_d1_v1": 42.828284045, + "simulation_heuristics_vizdoom_d3_v1": 40.106951872 + }, + "2": { + "simulation_heuristics_ant_v1": -7.591479, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 6.918238994, + "simulation_heuristics_breakout_rgb_v1": 70.758122744, + "simulation_heuristics_halfcheetah_v1": 33.28729024, + "simulation_heuristics_montezuma_v1": 100.0, + "simulation_heuristics_pong_ram_v1": 100.0, + "simulation_heuristics_vizdoom_d1_v1": 95.055823417, + "simulation_heuristics_vizdoom_d3_v1": -3.442513369 + }, + "3": { + "simulation_heuristics_ant_v1": -11.713338376, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 3.5639413, + "simulation_heuristics_breakout_rgb_v1": 70.758122744, + "simulation_heuristics_halfcheetah_v1": 36.605449762, + "simulation_heuristics_montezuma_v1": 100.0, + "simulation_heuristics_pong_ram_v1": 83.333333333, + "simulation_heuristics_vizdoom_d1_v1": 31.015418049, + "simulation_heuristics_vizdoom_d3_v1": 0.0 + }, + "4": { + "simulation_heuristics_ant_v1": -19.696444908, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 18.448637317, + "simulation_heuristics_breakout_rgb_v1": 70.758122744, + "simulation_heuristics_halfcheetah_v1": 32.792734613, + "simulation_heuristics_montezuma_v1": 100.0, + "simulation_heuristics_pong_ram_v1": 95.833333333, + "simulation_heuristics_vizdoom_d1_v1": 89.73418684, + "simulation_heuristics_vizdoom_d3_v1": 0.0 + }, + "5": { + "simulation_heuristics_ant_v1": 5.32283012, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 90.566037736, + "simulation_heuristics_breakout_rgb_v1": 21.299638989, + "simulation_heuristics_halfcheetah_v1": 30.414174988, + "simulation_heuristics_montezuma_v1": 72.5, + "simulation_heuristics_pong_ram_v1": 100.0, + "simulation_heuristics_vizdoom_d1_v1": 25.539607563, + "simulation_heuristics_vizdoom_d3_v1": 44.852941176 + } } }, { - "arithmetic_mean_normalized_score": 39.81691467833333, - "average_normalized_score": 39.81691467833333, - "final_normalized_score": 27.218574763200003, + "arithmetic_mean_normalized_score": 22.06816884377778, + "average_normalized_score": 22.06816884377778, + "final_normalized_score": 10.662970787652174, + "final_normalized_score_stddev": 18.67984640949238, "harness": "opencode", - "median_normalized_score": 14.050353128, - "model": "Claude Opus 4.8", - "model_id": "claude-opus-4.8", - "model_route": "anthropic/claude-opus-4-8", - "provider": "claude_oauth", - "provider_label": "Claude OAuth via pinned OpenCode plugin", - "provider_reasoning_effort": "max", + "mean_trial_iqm_normalized_score": 16.7775760408, + "median_normalized_score": 5.870020964, + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", "rank": 3, + "raw_task_score_imputation_counts": { + "simulation_heuristics_ant_v1": 0, + "simulation_heuristics_atari57_v1": 0, + "simulation_heuristics_breakout_ram_v1": 0, + "simulation_heuristics_breakout_rgb_v1": 0, + "simulation_heuristics_halfcheetah_v1": 0, + "simulation_heuristics_montezuma_v1": 0, + "simulation_heuristics_pong_ram_v1": 0, + "simulation_heuristics_vizdoom_d1_v1": 0, + "simulation_heuristics_vizdoom_d3_v1": 0 + }, + "raw_task_score_stddevs": { + "simulation_heuristics_ant_v1": 306.36385957656836, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 131.153345363357, + "simulation_heuristics_breakout_rgb_v1": 122.53652516698847, + "simulation_heuristics_halfcheetah_v1": 698.5070322948952, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 9.705668446840743, + "simulation_heuristics_vizdoom_d1_v1": 0.3579053781713541, + "simulation_heuristics_vizdoom_d3_v1": 122.81457568220475 + }, "raw_task_scores": { - "simulation_heuristics_ant_v1": 2935.3698887625987, + "simulation_heuristics_ant_v1": 2006.0944869926293, "simulation_heuristics_atari57_v1": 0.0, - "simulation_heuristics_breakout_ram_v1": 387.0, - "simulation_heuristics_breakout_rgb_v1": 310.0, - "simulation_heuristics_halfcheetah_v1": 6021.504892591717, + "simulation_heuristics_breakout_ram_v1": 486.8, + "simulation_heuristics_breakout_rgb_v1": 647.2, + "simulation_heuristics_halfcheetah_v1": 4717.6162449605235, "simulation_heuristics_montezuma_v1": 0.0, - "simulation_heuristics_pong_ram_v1": 21.0, - "simulation_heuristics_vizdoom_d1_v1": 0.7992199431359768, - "simulation_heuristics_vizdoom_d3_v1": 397.7 + "simulation_heuristics_pong_ram_v1": 8.8, + "simulation_heuristics_vizdoom_d1_v1": 0.6197079806402326, + "simulation_heuristics_vizdoom_d3_v1": 101.49999999999999 }, + "sandbox": "daytona", "source_runs": { - "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T142919Z/claude-opus-4.8", - "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T123222Z/claude-opus-4.8", - "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", - "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", - "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T111714Z/claude-opus-4.8", - "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", - "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", - "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", - "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8" + "simulation_heuristics_ant_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05/tasks-81ec8d98a4" + ], + "simulation_heuristics_atari57_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_breakout_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-e6fa3546aa", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_breakout_rgb_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-885696cfa4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_halfcheetah_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01/simulation_heuristics_halfcheetah_v1", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-e6fa3546aa", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05/tasks-81ec8d98a4" + ], + "simulation_heuristics_montezuma_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_pong_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-885696cfa4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_vizdoom_d1_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_vizdoom_d3_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ] }, "submission_details": { - "simulation_heuristics_ant_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/score.json", - "simulation_heuristics_atari57_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/score.json", - "simulation_heuristics_breakout_ram_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/score.json", - "simulation_heuristics_breakout_rgb_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/score.json", - "simulation_heuristics_halfcheetah_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/score.json", - "simulation_heuristics_montezuma_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/score.json", - "simulation_heuristics_pong_ram_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/score.json", - "simulation_heuristics_vizdoom_d1_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/score.json", - "simulation_heuristics_vizdoom_d3_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/score.json" + "simulation_heuristics_ant_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/score.json", + "simulation_heuristics_atari57_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/score.json", + "simulation_heuristics_breakout_ram_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/score.json", + "simulation_heuristics_breakout_rgb_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/score.json", + "simulation_heuristics_halfcheetah_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/score.json", + "simulation_heuristics_montezuma_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/score.json", + "simulation_heuristics_pong_ram_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/score.json", + "simulation_heuristics_vizdoom_d1_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/score.json", + "simulation_heuristics_vizdoom_d3_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/score.json" }, "task_anchors": { "simulation_heuristics_ant_v1": { @@ -1085,7 +1932,7 @@ "starter_score": -2.9999999999999996 }, "simulation_heuristics_vizdoom_d1_v1": { - "reference_score": 0.8178199748694897, + "reference_score": 0.8178199748694898, "starter_score": 0.44161998964846133 }, "simulation_heuristics_vizdoom_d3_v1": { @@ -1093,24 +1940,102 @@ "starter_score": 31.9 } }, + "task_score_stddevs": { + "simulation_heuristics_ant_v1": 8.756962566015595, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 27.49546024414613, + "simulation_heuristics_breakout_rgb_v1": 22.118506348049813, + "simulation_heuristics_halfcheetah_v1": 9.330884765209642, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 40.44028519521613, + "simulation_heuristics_vizdoom_d1_v1": 95.13699952993868, + "simulation_heuristics_vizdoom_d3_v1": 41.04765230004701 + }, "task_scores": { - "simulation_heuristics_ant_v1": 14.050353128, + "simulation_heuristics_ant_v1": -12.5116242462, "simulation_heuristics_atari57_v1": 0.0, - "simulation_heuristics_breakout_ram_v1": 0.0, - "simulation_heuristics_breakout_rgb_v1": 0.0, - "simulation_heuristics_halfcheetah_v1": 26.986707946, + "simulation_heuristics_breakout_ram_v1": 20.9224318658, + "simulation_heuristics_breakout_rgb_v1": 60.86642599300001, + "simulation_heuristics_halfcheetah_v1": 9.5689381094, "simulation_heuristics_montezuma_v1": 0.0, - "simulation_heuristics_pong_ram_v1": 100.0, - "simulation_heuristics_vizdoom_d1_v1": 95.055812742, - "simulation_heuristics_vizdoom_d3_v1": 122.259358289 + "simulation_heuristics_pong_ram_v1": 49.1666666668, + "simulation_heuristics_vizdoom_d1_v1": 47.33864911959999, + "simulation_heuristics_vizdoom_d3_v1": 23.2620320856 + }, + "trial_count": 5, + "trial_final_normalized_scores": { + "1": 5.4339319906, + "2": 48.2436373978, + "3": 18.718893057000002, + "4": 9.604625305799999, + "5": 1.8867924528 + }, + "trial_task_normalized_scores": { + "1": { + "simulation_heuristics_ant_v1": -6.720960755, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 5.870020964, + "simulation_heuristics_breakout_rgb_v1": 21.299638989, + "simulation_heuristics_halfcheetah_v1": 0.0, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 41.666666667, + "simulation_heuristics_vizdoom_d1_v1": -117.814998349, + "simulation_heuristics_vizdoom_d3_v1": 22.326203209 + }, + "2": { + "simulation_heuristics_ant_v1": -24.004147668, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 70.020964361, + "simulation_heuristics_breakout_rgb_v1": 70.758122744, + "simulation_heuristics_halfcheetah_v1": 21.272433217, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 79.166666667, + "simulation_heuristics_vizdoom_d1_v1": 85.486445859, + "simulation_heuristics_vizdoom_d3_v1": 94.585561497 + }, + "3": { + "simulation_heuristics_ant_v1": -13.99811598, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 10.062893082, + "simulation_heuristics_breakout_rgb_v1": 70.758122744, + "simulation_heuristics_halfcheetah_v1": 12.773449459, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 100.0, + "simulation_heuristics_vizdoom_d1_v1": 104.662416785, + "simulation_heuristics_vizdoom_d3_v1": -0.601604278 + }, + "4": { + "simulation_heuristics_ant_v1": -1.398078834, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 9.224318658, + "simulation_heuristics_breakout_rgb_v1": 70.758122744, + "simulation_heuristics_halfcheetah_v1": 13.798807871, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 25.0, + "simulation_heuristics_vizdoom_d1_v1": 111.839447567, + "simulation_heuristics_vizdoom_d3_v1": 0.0 + }, + "5": { + "simulation_heuristics_ant_v1": -16.436817994, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 9.433962264, + "simulation_heuristics_breakout_rgb_v1": 70.758122744, + "simulation_heuristics_halfcheetah_v1": 0.0, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 0.0, + "simulation_heuristics_vizdoom_d1_v1": 52.519933736, + "simulation_heuristics_vizdoom_d3_v1": 0.0 + } } }, { - "arithmetic_mean_normalized_score": -29.71822245, - "average_normalized_score": -29.71822245, - "final_normalized_score": -9.8773515774, + "arithmetic_mean_normalized_score": -31.610134487355555, + "average_normalized_score": -31.610134487355555, + "final_normalized_score": -17.411123288956524, + "final_normalized_score_stddev": 5.902995817222085, "harness": "opencode", - "median_normalized_score": -8.167582379, + "mean_trial_iqm_normalized_score": -18.58872338544, + "median_normalized_score": -3.609625668, "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", "model_route": "azure/gpt-5.4-mini", @@ -1118,27 +2043,104 @@ "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 4, + "raw_task_score_imputation_counts": { + "simulation_heuristics_ant_v1": 0, + "simulation_heuristics_atari57_v1": 0, + "simulation_heuristics_breakout_ram_v1": 0, + "simulation_heuristics_breakout_rgb_v1": 0, + "simulation_heuristics_halfcheetah_v1": 0, + "simulation_heuristics_montezuma_v1": 0, + "simulation_heuristics_pong_ram_v1": 0, + "simulation_heuristics_vizdoom_d1_v1": 0, + "simulation_heuristics_vizdoom_d3_v1": 0 + }, + "raw_task_score_stddevs": { + "simulation_heuristics_ant_v1": 364.1182774109385, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 44.0204497932495, + "simulation_heuristics_breakout_rgb_v1": 287.77473829368694, + "simulation_heuristics_halfcheetah_v1": 1438.0428685581014, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 16.192590898309017, + "simulation_heuristics_vizdoom_d1_v1": 0.012638255798059739, + "simulation_heuristics_vizdoom_d3_v1": 135.46233055724386 + }, "raw_task_scores": { - "simulation_heuristics_ant_v1": 1346.6965294546999, + "simulation_heuristics_ant_v1": 791.3786088082715, "simulation_heuristics_atari57_v1": 0.0, - "simulation_heuristics_breakout_ram_v1": 399.0, - "simulation_heuristics_breakout_rgb_v1": 396.0, - "simulation_heuristics_halfcheetah_v1": 3389.8660260183765, + "simulation_heuristics_breakout_ram_v1": 380.6, + "simulation_heuristics_breakout_rgb_v1": 389.4, + "simulation_heuristics_halfcheetah_v1": 1015.6002588284422, "simulation_heuristics_montezuma_v1": 0.0, - "simulation_heuristics_pong_ram_v1": -21.0, - "simulation_heuristics_vizdoom_d1_v1": -0.16450000800192355, - "simulation_heuristics_vizdoom_d3_v1": 2.4 + "simulation_heuristics_pong_ram_v1": -12.8, + "simulation_heuristics_vizdoom_d1_v1": -0.25834800247848033, + "simulation_heuristics_vizdoom_d3_v1": 81.36 }, + "sandbox": "daytona", "source_runs": { - "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T104315Z/gpt-5.4-mini", - "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T111944Z/gpt-5.4-mini", - "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", - "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", - "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T093314Z/gpt-5.4-mini", - "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", - "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", - "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T102707Z/gpt-5.4-mini", - "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T104034Z/gpt-5.4-mini" + "simulation_heuristics_ant_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_atari57_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_breakout_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02/simulation_heuristics_breakout_ram_v1", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_breakout_rgb_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_halfcheetah_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_montezuma_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_pong_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_vizdoom_d1_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_vizdoom_d3_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01/simulation_heuristics_vizdoom_d3_v1", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ] }, "submission_details": { "simulation_heuristics_ant_v1": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/score.json", @@ -1181,7 +2183,7 @@ "starter_score": -2.9999999999999996 }, "simulation_heuristics_vizdoom_d1_v1": { - "reference_score": 0.8178199748694897, + "reference_score": 0.8178199748694898, "starter_score": 0.44161998964846133 }, "simulation_heuristics_vizdoom_d3_v1": { @@ -1189,63 +2191,355 @@ "starter_score": 31.9 } }, + "task_score_stddevs": { + "simulation_heuristics_ant_v1": 10.40778807635678, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 9.228605826577622, + "simulation_heuristics_breakout_rgb_v1": 51.9448986087478, + "simulation_heuristics_halfcheetah_v1": 19.209845675650264, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 67.46912874308083, + "simulation_heuristics_vizdoom_d1_v1": 3.359451433761567, + "simulation_heuristics_vizdoom_d3_v1": 45.27484310055022 + }, "task_scores": { - "simulation_heuristics_ant_v1": -31.35954984, + "simulation_heuristics_ant_v1": -47.232499900200004, "simulation_heuristics_atari57_v1": 0.0, - "simulation_heuristics_breakout_ram_v1": 2.51572327, - "simulation_heuristics_breakout_rgb_v1": 15.523465704, - "simulation_heuristics_halfcheetah_v1": -8.167582379, + "simulation_heuristics_breakout_ram_v1": -1.3417190774, + "simulation_heuristics_breakout_rgb_v1": 14.332129963999998, + "simulation_heuristics_halfcheetah_v1": -39.8837988972, "simulation_heuristics_montezuma_v1": 0.0, - "simulation_heuristics_pong_ram_v1": -75.0, - "simulation_heuristics_vizdoom_d1_v1": -161.116433137, - "simulation_heuristics_vizdoom_d3_v1": -9.859625668 + "simulation_heuristics_pong_ram_v1": -40.8333333332, + "simulation_heuristics_vizdoom_d1_v1": -186.0627378054, + "simulation_heuristics_vizdoom_d3_v1": 16.5307486632 + }, + "trial_count": 5, + "trial_final_normalized_scores": { + "1": -17.9990639644, + "2": -14.669657405, + "3": -18.885235435600002, + "4": -28.268315785200002, + "5": -13.121344337 + }, + "trial_task_normalized_scores": { + "1": { + "simulation_heuristics_ant_v1": -36.632193477, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 0.209643606, + "simulation_heuristics_breakout_rgb_v1": 8.664259928, + "simulation_heuristics_halfcheetah_v1": -53.572769951, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": -75.0, + "simulation_heuristics_vizdoom_d1_v1": -180.053168387, + "simulation_heuristics_vizdoom_d3_v1": 97.159090909 + }, + "2": { + "simulation_heuristics_ant_v1": -50.421772091, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": -0.628930818, + "simulation_heuristics_breakout_rgb_v1": 95.848375451, + "simulation_heuristics_halfcheetah_v1": -12.137156309, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": -75.0, + "simulation_heuristics_vizdoom_d1_v1": -187.56513016, + "simulation_heuristics_vizdoom_d3_v1": -10.160427807 + }, + "3": { + "simulation_heuristics_ant_v1": -40.65567106, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 3.983228512, + "simulation_heuristics_breakout_rgb_v1": 0.0, + "simulation_heuristics_halfcheetah_v1": -53.770506118, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": -58.333333333, + "simulation_heuristics_vizdoom_d1_v1": -187.56513016, + "simulation_heuristics_vizdoom_d3_v1": 1.771390374 + }, + "4": { + "simulation_heuristics_ant_v1": -63.436898931, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 6.708595388, + "simulation_heuristics_breakout_rgb_v1": -48.194945848, + "simulation_heuristics_halfcheetah_v1": -27.203049655, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": -75.0, + "simulation_heuristics_vizdoom_d1_v1": -187.56513016, + "simulation_heuristics_vizdoom_d3_v1": -2.506684492 + }, + "5": { + "simulation_heuristics_ant_v1": -45.015963942, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": -16.981132075, + "simulation_heuristics_breakout_rgb_v1": 15.342960289, + "simulation_heuristics_halfcheetah_v1": -52.735512453, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 79.166666667, + "simulation_heuristics_vizdoom_d1_v1": -187.56513016, + "simulation_heuristics_vizdoom_d3_v1": -3.609625668 + } } } ], "source_runs": { "claude-opus-4.8": { - "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T142919Z/claude-opus-4.8", - "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T123222Z/claude-opus-4.8", - "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", - "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", - "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T111714Z/claude-opus-4.8", - "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", - "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", - "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", - "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8" + "simulation_heuristics_ant_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_atari57_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_breakout_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_breakout_rgb_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_halfcheetah_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_montezuma_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_pong_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_vizdoom_d1_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_vizdoom_d3_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ] }, "gpt-5.4-mini": { - "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T104315Z/gpt-5.4-mini", - "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T111944Z/gpt-5.4-mini", - "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", - "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", - "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T093314Z/gpt-5.4-mini", - "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", - "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", - "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T102707Z/gpt-5.4-mini", - "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T104034Z/gpt-5.4-mini" + "simulation_heuristics_ant_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_atari57_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_breakout_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02/simulation_heuristics_breakout_ram_v1", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_breakout_rgb_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_halfcheetah_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_montezuma_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_pong_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_vizdoom_d1_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_vizdoom_d3_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01/simulation_heuristics_vizdoom_d3_v1", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ] }, "gpt-5.5": { - "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T095807Z/gpt-5.5", - "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T111604Z/gpt-5.5", - "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", - "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", - "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T103810Z/gpt-5.5", - "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", - "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", - "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T102218Z/gpt-5.5", - "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5" + "simulation_heuristics_ant_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05/tasks-81ec8d98a4" + ], + "simulation_heuristics_atari57_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_breakout_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-e6fa3546aa", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_breakout_rgb_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-885696cfa4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_halfcheetah_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01/simulation_heuristics_halfcheetah_v1", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-e6fa3546aa", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05/tasks-81ec8d98a4" + ], + "simulation_heuristics_montezuma_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_pong_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-885696cfa4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_vizdoom_d1_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_vizdoom_d3_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ] }, "gpt-5.6-sol": { - "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T093815Z/gpt-5.6-sol", - "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T111156Z/gpt-5.6-sol", - "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", - "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", - "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T084340Z/gpt-5.6-sol", - "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", - "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", - "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T101636Z/gpt-5.6-sol", - "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T083940Z/gpt-5.6-sol" + "simulation_heuristics_ant_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_atari57_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_breakout_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_breakout_rgb_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_halfcheetah_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_montezuma_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_pong_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_vizdoom_d1_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_vizdoom_d3_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ] } }, "task_count": 9, diff --git a/leaderboard/article_suite_final_leaderboard.png b/leaderboard/article_suite_final_leaderboard.png index 3b61646..cb471e3 100644 Binary files a/leaderboard/article_suite_final_leaderboard.png and b/leaderboard/article_suite_final_leaderboard.png differ diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/metadata.json index e020be1..2169219 100644 --- a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/metadata.json +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/metadata.json @@ -1,11 +1,9 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:5cf3805d59b03cb9d0bebb6f154eab38beb655918b8321500d692f5010a00cb2", - "digest_compatibility_note": "Score-equivalent to the current task. The later change adds only a CI smoke config and documentation; the publication scoring config is unchanged.", - "finished_at": 1783956862.2606888, + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", "harness": "opencode", "model": { - "agent_idle_timeout_sec": 0, + "agent_idle_timeout_sec": 3600, "daytona_pty_readline_timeout_sec": 3600, "display_name": "Claude Opus 4.8", "id": "claude-opus-4.8", @@ -13,17 +11,19 @@ "provider": "claude_oauth", "provider_reasoning_effort": "max" }, - "normalized_score": 14.050353128, + "normalized_score": 1.820348169, + "normalized_score_stddev": 4.070422248601275, "provider_reasoning_effort": "max", - "source_run_id": "leaderboard/runs/article_suite/20260713T142919Z/claude-opus-4.8", + "raw_score": 2507.501036532837, + "raw_score_imputation_count": 4, + "raw_score_stddev": 142.40443086118393, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], "task": "simulation_heuristics_ant_v1", - "task_digest": "sha256:9da0e00147cf66804e6c2fc17869606bea8f260850c5447989a8880eef940d45", - "token_usage": { - "final_input_tokens": 0.0, - "final_output_tokens": 0.0, - "input_tokens": 0.0, - "output_tokens": 0.0, - "total_tokens": 0.0 - }, - "tool_calls": 48 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/score.json index ac313ff..32c21d4 100644 --- a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/score.json +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/score.json @@ -1,141 +1,59 @@ { - "hidden_nominal": { - "episodes": [ - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.025934887526091188, - "policy_error": null, - "return": 2947.879693227018, - "seed": 101, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": 102.77571899817438, - "x_velocity": 1.8673987220154231 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.02599245577584952, - "policy_error": null, - "return": 3000.163186740716, - "seed": 211, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": 105.33522389819196, - "x_velocity": 2.1574081280110136 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.02603654988342896, - "policy_error": null, - "return": 2772.4088822782287, - "seed": 307, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": 93.7636515447122, - "x_velocity": 2.4267700657179603 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.02577067888341844, - "policy_error": null, - "return": 2990.8335078285263, - "seed": 401, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": 104.90485217547173, - "x_velocity": 2.078351224632513 - } - ], - "fall_rate": 0.0, - "invalid_episode_rate": 0.0, - "max_return": 3000.163186740716, - "max_steps": 1000, - "mean_action_latency_ms": 0.025933643017197028, - "mean_return": 2927.821317518622, - "mean_x_position": 101.69486165413757, - "min_return": 2772.4088822782287, - "policy_path": "submitted_artifact", - "score": 2927.821317518622 - }, - "hidden_nominal_weight": 0.7, - "hidden_robustness": { - "episodes": [ - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.025812971696723253, - "policy_error": null, - "return": 3166.4558227561265, - "seed": 503, - "terminated": false, - "truncated": true, - "variant": "light_low_friction", - "x_position": 113.74752702527248, - "x_velocity": 3.0189141499053562 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.02593717718264088, - "policy_error": null, - "return": 3177.1212044211834, - "seed": 601, - "terminated": false, - "truncated": true, - "variant": "light_low_friction", - "x_position": 114.37538391255823, - "x_velocity": 2.4397722759977114 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.026038738433271646, - "policy_error": null, - "return": 2839.9126867592117, - "seed": 503, - "terminated": false, - "truncated": true, - "variant": "heavy_weak_actuators", - "x_position": 97.73666401480078, - "x_velocity": 0.6070665236407535 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.026594983239192516, - "policy_error": null, - "return": 2628.443172724322, - "seed": 601, - "terminated": false, - "truncated": true, - "variant": "heavy_weak_actuators", - "x_position": 87.07088591700881, - "x_velocity": 1.804739145616736 - } - ], - "fall_rate": 0.0, - "invalid_episode_rate": 0.0, - "max_return": 3177.1212044211834, - "max_steps": 1000, - "mean_action_latency_ms": 0.026095967637957074, - "mean_return": 2952.983221665211, - "mean_x_position": 103.23261521741007, - "min_return": 2628.443172724322, - "policy_path": "submitted_artifact", - "score": 2952.983221665211 - }, - "hidden_robustness_weight": 0.3, - "normalized_score": 14.050353128, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": 1.820348169, + "normalized_score_stddev": 4.070422248601275, + "raw_score_imputation_count": 4, "reference_score": 5942.333244755838, - "score": 2935.3698887625987, + "score": 2507.501036532837, + "score_stddev": 142.40443086118393, "starter_score": 2443.815838992282, - "unique_policy_evaluations": 3 + "task": "simulation_heuristics_ant_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 0.0, + "observed_raw_score": null, + "raw_score": 2443.815838992282, + "raw_score_source": "starter_anchor_equivalent_for_fail_closed_timeout", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01/tasks-81ec8d98a4", + "trial": 1 + }, + { + "normalized_score": 9.101740845, + "observed_raw_score": 2762.241826695058, + "raw_score": 2762.241826695058, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "trial": 2 + }, + { + "normalized_score": 0.0, + "observed_raw_score": null, + "raw_score": 2443.815838992282, + "raw_score_source": "starter_anchor_equivalent_for_fail_closed_timeout", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "trial": 3 + }, + { + "normalized_score": 0.0, + "observed_raw_score": null, + "raw_score": 2443.815838992282, + "raw_score_source": "starter_anchor_equivalent_for_fail_closed_timeout", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "trial": 4 + }, + { + "normalized_score": 0.0, + "observed_raw_score": null, + "raw_score": 2443.815838992282, + "raw_score_source": "starter_anchor_equivalent_for_fail_closed_timeout", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05", + "trial": 5 + } + ] } diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-01/metadata.json new file mode 100644 index 0000000..f442b81 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-01/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "digest_compatibility_note": null, + "finished_at": 1784105005.326783, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 0.0, + "observed_raw_score": null, + "provider_reasoning_effort": "max", + "raw_score": 2443.815838992282, + "raw_score_source": "starter_anchor_equivalent_for_fail_closed_timeout", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01/tasks-81ec8d98a4", + "task": "simulation_heuristics_ant_v1", + "task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "token_usage": { + "final_input_tokens": 0.0, + "final_output_tokens": 0.0, + "input_tokens": 0.0, + "output_tokens": 0.0, + "total_tokens": 0.0 + }, + "tool_calls": 52, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-01/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-01/score.json new file mode 100644 index 0000000..c5bece8 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-01/score.json @@ -0,0 +1,6 @@ +{ + "normalized_score": 0.0, + "score": null, + "timeout_seconds": 3900, + "verifier_timeout": true +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-02/metadata.json new file mode 100644 index 0000000..18a11ef --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-02/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "digest_compatibility_note": null, + "finished_at": 1784103481.173677, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 9.101740845, + "observed_raw_score": 2762.241826695058, + "provider_reasoning_effort": "max", + "raw_score": 2762.241826695058, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "task": "simulation_heuristics_ant_v1", + "task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "token_usage": { + "final_input_tokens": 0.0, + "final_output_tokens": 0.0, + "input_tokens": 0.0, + "output_tokens": 0.0, + "total_tokens": 0.0 + }, + "tool_calls": 75, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-02/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-02/score.json new file mode 100644 index 0000000..2248f64 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-02/score.json @@ -0,0 +1,141 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 53.30563308298588, + "policy_error": null, + "return": 2462.6958737105706, + "seed": 101, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 82.17225037670244, + "x_velocity": 1.6847347109145971 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 55.477168429642916, + "policy_error": null, + "return": 2864.7157423617828, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 102.17853506143068, + "x_velocity": 1.9541062168079293 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 55.207165353000164, + "policy_error": null, + "return": 2486.0698110071953, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 83.28267351663796, + "x_velocity": 0.4153314073650449 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 56.19777377322316, + "policy_error": null, + "return": 2985.27186137575, + "seed": 401, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 108.24431210651534, + "x_velocity": 1.64362469988788 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 2985.27186137575, + "max_steps": 1000, + "mean_action_latency_ms": 55.04693515971303, + "mean_return": 2699.6883221138246, + "mean_x_position": 93.9694427653216, + "min_return": 2462.6958737105706, + "policy_path": "submitted_artifact", + "score": 2699.6883221138246 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 56.27190098166466, + "policy_error": null, + "return": 3138.8040933119382, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 115.86236716561513, + "x_velocity": 2.2301605735646035 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 54.710381008684635, + "policy_error": null, + "return": 3073.575124572617, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 112.69542014820925, + "x_velocity": 2.3028379066158777 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 55.404857095330954, + "policy_error": null, + "return": 2715.961883054363, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 95.26212960331155, + "x_velocity": 1.694659112316117 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 56.29627299308777, + "policy_error": null, + "return": 2704.4589152661615, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 94.78120825640332, + "x_velocity": 2.1434038102083264 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 3138.8040933119382, + "max_steps": 1000, + "mean_action_latency_ms": 55.670853019692004, + "mean_return": 2908.20000405127, + "mean_x_position": 104.65028129338481, + "min_return": 2704.4589152661615, + "policy_path": "submitted_artifact", + "score": 2908.20000405127 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": 9.101740845, + "reference_score": 5942.333244755838, + "score": 2762.241826695058, + "starter_score": 2443.815838992282, + "unique_policy_evaluations": 3 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-03/metadata.json new file mode 100644 index 0000000..411ac21 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-03/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "digest_compatibility_note": null, + "finished_at": 1784113784.201615, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 0.0, + "observed_raw_score": null, + "provider_reasoning_effort": "max", + "raw_score": 2443.815838992282, + "raw_score_source": "starter_anchor_equivalent_for_fail_closed_timeout", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "task": "simulation_heuristics_ant_v1", + "task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 639.0, + "input_tokens": 2.0, + "output_tokens": 639.0, + "total_tokens": 47729.0 + }, + "tool_calls": 39, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-03/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-03/score.json new file mode 100644 index 0000000..c5bece8 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-03/score.json @@ -0,0 +1,6 @@ +{ + "normalized_score": 0.0, + "score": null, + "timeout_seconds": 3900, + "verifier_timeout": true +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-04/metadata.json new file mode 100644 index 0000000..92f84b1 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-04/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "digest_compatibility_note": null, + "finished_at": 1784120178.69071, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 0.0, + "observed_raw_score": null, + "provider_reasoning_effort": "max", + "raw_score": 2443.815838992282, + "raw_score_source": "starter_anchor_equivalent_for_fail_closed_timeout", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "task": "simulation_heuristics_ant_v1", + "task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 742.0, + "input_tokens": 2.0, + "output_tokens": 742.0, + "total_tokens": 40033.0 + }, + "tool_calls": 26, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-04/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-04/score.json new file mode 100644 index 0000000..c5bece8 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-04/score.json @@ -0,0 +1,6 @@ +{ + "normalized_score": 0.0, + "score": null, + "timeout_seconds": 3900, + "verifier_timeout": true +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-05/metadata.json new file mode 100644 index 0000000..7d597aa --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-05/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "digest_compatibility_note": null, + "finished_at": 1784110074.558851, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 0.0, + "observed_raw_score": null, + "provider_reasoning_effort": "max", + "raw_score": 2443.815838992282, + "raw_score_source": "starter_anchor_equivalent_for_fail_closed_timeout", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05", + "task": "simulation_heuristics_ant_v1", + "task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 565.0, + "input_tokens": 2.0, + "output_tokens": 565.0, + "total_tokens": 45246.0 + }, + "tool_calls": 33, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-05/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-05/score.json new file mode 100644 index 0000000..c5bece8 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/trial-05/score.json @@ -0,0 +1,6 @@ +{ + "normalized_score": 0.0, + "score": null, + "timeout_seconds": 3900, + "verifier_timeout": true +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/metadata.json index 4b1b2ef..9ab0d85 100644 --- a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/metadata.json +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/metadata.json @@ -1,11 +1,10 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:a3ec10ab61a65c03ca17e6460147a3178e222bd1c48964727ea9daae54717ebe", - "digest_compatibility_note": null, - "finished_at": 1783951813.6763892, + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", "harness": "opencode", "model": { - "agent_idle_timeout_sec": 0, + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, "display_name": "Claude Opus 4.8", "id": "claude-opus-4.8", "model": "anthropic/claude-opus-4-8", @@ -13,16 +12,18 @@ "provider_reasoning_effort": "max" }, "normalized_score": 0.0, + "normalized_score_stddev": 0.0, "provider_reasoning_effort": "max", - "source_run_id": "leaderboard/runs/article_suite/20260713T123222Z/claude-opus-4.8", + "raw_score": 0.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.0, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], "task": "simulation_heuristics_atari57_v1", - "task_digest": "sha256:a3ec10ab61a65c03ca17e6460147a3178e222bd1c48964727ea9daae54717ebe", - "token_usage": { - "final_input_tokens": 2.0, - "final_output_tokens": 789.0, - "input_tokens": 2.0, - "output_tokens": 789.0, - "total_tokens": 74672.0 - }, - "tool_calls": 65 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/score.json index 7788108..6498808 100644 --- a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/score.json +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/score.json @@ -1,15 +1,59 @@ { - "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", - "evaluation": null, - "interaction_budget": { - "completed_trajectories": 0, - "counted_env_steps": 0, - "planned_trajectories": 342, - "target_env_steps": 6840000000 - }, + "benchmark": "learning_beyond_gradients_article_suite", "normalized_score": 0.0, - "protocol_complete": false, + "normalized_score_stddev": 0.0, + "raw_score_imputation_count": 0, "reference_score": 0.8283015254994576, "score": 0.0, - "starter_score": 0.0 + "score_stddev": 0.0, + "starter_score": 0.0, + "task": "simulation_heuristics_atari57_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "trial": 1 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "trial": 2 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "trial": 3 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "trial": 4 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05", + "trial": 5 + } + ] } diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-01/metadata.json new file mode 100644 index 0000000..ea092b4 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-01/metadata.json @@ -0,0 +1,27 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "digest_compatibility_note": null, + "finished_at": 1784091892.627182, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 0, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "max", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "task": "simulation_heuristics_atari57_v1", + "task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "token_usage": null, + "tool_calls": 33, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-01/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-01/score.json new file mode 100644 index 0000000..7788108 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-01/score.json @@ -0,0 +1,15 @@ +{ + "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", + "evaluation": null, + "interaction_budget": { + "completed_trajectories": 0, + "counted_env_steps": 0, + "planned_trajectories": 342, + "target_env_steps": 6840000000 + }, + "normalized_score": 0.0, + "protocol_complete": false, + "reference_score": 0.8283015254994576, + "score": 0.0, + "starter_score": 0.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-02/metadata.json new file mode 100644 index 0000000..3868622 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-02/metadata.json @@ -0,0 +1,27 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "digest_compatibility_note": null, + "finished_at": 1784095422.450964, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "max", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "task": "simulation_heuristics_atari57_v1", + "task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "token_usage": null, + "tool_calls": 39, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-02/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-02/score.json new file mode 100644 index 0000000..7788108 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-02/score.json @@ -0,0 +1,15 @@ +{ + "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", + "evaluation": null, + "interaction_budget": { + "completed_trajectories": 0, + "counted_env_steps": 0, + "planned_trajectories": 342, + "target_env_steps": 6840000000 + }, + "normalized_score": 0.0, + "protocol_complete": false, + "reference_score": 0.8283015254994576, + "score": 0.0, + "starter_score": 0.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-03/metadata.json new file mode 100644 index 0000000..4e03187 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-03/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "digest_compatibility_note": null, + "finished_at": 1784113784.201615, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "max", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "task": "simulation_heuristics_atari57_v1", + "task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 664.0, + "input_tokens": 2.0, + "output_tokens": 664.0, + "total_tokens": 50513.0 + }, + "tool_calls": 29, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-03/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-03/score.json new file mode 100644 index 0000000..7788108 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-03/score.json @@ -0,0 +1,15 @@ +{ + "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", + "evaluation": null, + "interaction_budget": { + "completed_trajectories": 0, + "counted_env_steps": 0, + "planned_trajectories": 342, + "target_env_steps": 6840000000 + }, + "normalized_score": 0.0, + "protocol_complete": false, + "reference_score": 0.8283015254994576, + "score": 0.0, + "starter_score": 0.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-04/metadata.json new file mode 100644 index 0000000..c2a9574 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-04/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "digest_compatibility_note": null, + "finished_at": 1784120178.69071, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "max", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "task": "simulation_heuristics_atari57_v1", + "task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 879.0, + "input_tokens": 2.0, + "output_tokens": 879.0, + "total_tokens": 51441.0 + }, + "tool_calls": 34, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-04/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-04/score.json new file mode 100644 index 0000000..7788108 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-04/score.json @@ -0,0 +1,15 @@ +{ + "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", + "evaluation": null, + "interaction_budget": { + "completed_trajectories": 0, + "counted_env_steps": 0, + "planned_trajectories": 342, + "target_env_steps": 6840000000 + }, + "normalized_score": 0.0, + "protocol_complete": false, + "reference_score": 0.8283015254994576, + "score": 0.0, + "starter_score": 0.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-05/metadata.json new file mode 100644 index 0000000..623771e --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-05/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "digest_compatibility_note": null, + "finished_at": 1784110074.558851, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "max", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05", + "task": "simulation_heuristics_atari57_v1", + "task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 678.0, + "input_tokens": 2.0, + "output_tokens": 678.0, + "total_tokens": 59275.0 + }, + "tool_calls": 33, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-05/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-05/score.json new file mode 100644 index 0000000..7788108 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/trial-05/score.json @@ -0,0 +1,15 @@ +{ + "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", + "evaluation": null, + "interaction_budget": { + "completed_trajectories": 0, + "counted_env_steps": 0, + "planned_trajectories": 342, + "target_env_steps": 6840000000 + }, + "normalized_score": 0.0, + "protocol_complete": false, + "reference_score": 0.8283015254994576, + "score": 0.0, + "starter_score": 0.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/metadata.json index 517c316..8c1da72 100644 --- a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/metadata.json +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/metadata.json @@ -1,27 +1,29 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:77215337f89f950f83b8f7436f799959c2ddccd4102552c6302b13d30cc57199", - "digest_compatibility_note": null, - "finished_at": 1783940397.5345018, + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", "harness": "opencode", "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, "display_name": "Claude Opus 4.8", "id": "claude-opus-4.8", "model": "anthropic/claude-opus-4-8", "provider": "claude_oauth", "provider_reasoning_effort": "max" }, - "normalized_score": 0.0, + "normalized_score": 94.004192872, + "normalized_score_stddev": 7.012257406411853, "provider_reasoning_effort": "max", - "source_run_id": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "raw_score": 835.4, + "raw_score_imputation_count": 0, + "raw_score_stddev": 33.448467827390836, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], "task": "simulation_heuristics_breakout_ram_v1", - "task_digest": "sha256:77215337f89f950f83b8f7436f799959c2ddccd4102552c6302b13d30cc57199", - "token_usage": { - "final_input_tokens": 0.0, - "final_output_tokens": 0.0, - "input_tokens": 0.0, - "output_tokens": 0.0, - "total_tokens": 0.0 - }, - "tool_calls": 53 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/score.json index 66044e3..eb22ae8 100644 --- a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/score.json +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/score.json @@ -1,80 +1,59 @@ { - "normalized_score": 0.0, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": 94.004192872, + "normalized_score_stddev": 7.012257406411853, + "raw_score_imputation_count": 0, "reference_score": 864.0, - "score": 387.0, + "score": 835.4, + "score_stddev": 33.448467827390836, "starter_score": 387.0, - "suites": { - "hidden_nominal": { - "completion_rate": 0.0, - "episodes": [ - { - "final_lives": 4, - "invalid_action": false, - "length": 108000, - "mean_action_latency_ms": 0.004508449714329025, - "observation_mode": "ram", - "policy_error": null, - "return": 387.0, - "seed": 101, - "terminated": false, - "truncated": true, - "variant": "nominal", - "warmup_steps": 0 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 387.0, - "max_steps": 108000, - "mean_action_latency_ms": 0.004508449714329025, - "mean_return": 387.0, - "min_return": 387.0, - "observation_mode": "ram", - "policy_path": "submitted_artifact", - "score": 387.0 + "task": "simulation_heuristics_breakout_ram_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 98.742138365, + "observed_raw_score": 858.0, + "raw_score": 858.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "trial": 1 }, - "hidden_shifted_start": { - "completion_rate": 0.0, - "episodes": [ - { - "final_lives": 4, - "invalid_action": false, - "length": 108000, - "mean_action_latency_ms": 0.004490307122641416, - "observation_mode": "ram", - "policy_error": null, - "return": 387.0, - "seed": 211, - "terminated": false, - "truncated": true, - "variant": "left_6", - "warmup_steps": 6 - }, - { - "final_lives": 4, - "invalid_action": false, - "length": 108000, - "mean_action_latency_ms": 0.00453408551294598, - "observation_mode": "ram", - "policy_error": null, - "return": 387.0, - "seed": 211, - "terminated": false, - "truncated": true, - "variant": "right_6", - "warmup_steps": 6 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 387.0, - "max_steps": 108000, - "mean_action_latency_ms": 0.004512196317793698, - "mean_return": 387.0, - "min_return": 387.0, - "observation_mode": "ram", - "policy_path": "submitted_artifact", - "score": 387.0 + { + "normalized_score": 100.0, + "observed_raw_score": 864.0, + "raw_score": 864.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "trial": 2 + }, + { + "normalized_score": 98.322851153, + "observed_raw_score": 856.0, + "raw_score": 856.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "trial": 3 + }, + { + "normalized_score": 84.696016771, + "observed_raw_score": 791.0, + "raw_score": 791.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "trial": 4 + }, + { + "normalized_score": 88.259958071, + "observed_raw_score": 808.0, + "raw_score": 808.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05", + "trial": 5 } - }, - "target_gap": -477.0, - "target_score": 864.0 + ] } diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-01/metadata.json new file mode 100644 index 0000000..750732f --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-01/metadata.json @@ -0,0 +1,27 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "digest_compatibility_note": null, + "finished_at": 1784091892.627182, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 0, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 98.742138365, + "observed_raw_score": 858.0, + "provider_reasoning_effort": "max", + "raw_score": 858.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "task": "simulation_heuristics_breakout_ram_v1", + "task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "token_usage": null, + "tool_calls": 77, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-01/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-01/score.json new file mode 100644 index 0000000..1bbb4a3 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-01/score.json @@ -0,0 +1,80 @@ +{ + "normalized_score": 98.742138365, + "reference_score": 864.0, + "score": 858.0, + "starter_score": 387.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 24908, + "mean_action_latency_ms": 0.010744514943164675, + "observation_mode": "ram", + "policy_error": null, + "return": 858.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 858.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.010744514943164675, + "mean_return": 858.0, + "min_return": 858.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 858.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 24902, + "mean_action_latency_ms": 0.013409103591148561, + "observation_mode": "ram", + "policy_error": null, + "return": 858.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "left_6", + "warmup_steps": 6 + }, + { + "final_lives": 0, + "invalid_action": false, + "length": 24902, + "mean_action_latency_ms": 0.0102101662041011, + "observation_mode": "ram", + "policy_error": null, + "return": 858.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 858.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.011809634897624832, + "mean_return": 858.0, + "min_return": 858.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 858.0 + } + }, + "target_gap": -6.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-02/metadata.json new file mode 100644 index 0000000..bf63b6a --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-02/metadata.json @@ -0,0 +1,27 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "digest_compatibility_note": null, + "finished_at": 1784095422.450964, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 100.0, + "observed_raw_score": 864.0, + "provider_reasoning_effort": "max", + "raw_score": 864.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "task": "simulation_heuristics_breakout_ram_v1", + "task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "token_usage": null, + "tool_calls": 64, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-02/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-02/score.json new file mode 100644 index 0000000..52ec671 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-02/score.json @@ -0,0 +1,80 @@ +{ + "normalized_score": 100.0, + "reference_score": 864.0, + "score": 864.0, + "starter_score": 387.0, + "suites": { + "hidden_nominal": { + "completion_rate": 1.0, + "episodes": [ + { + "final_lives": 1, + "invalid_action": false, + "length": 108000, + "mean_action_latency_ms": 0.00618563983934345, + "observation_mode": "ram", + "policy_error": null, + "return": 864.0, + "seed": 101, + "terminated": false, + "truncated": true, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 864.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.00618563983934345, + "mean_return": 864.0, + "min_return": 864.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 864.0 + }, + "hidden_shifted_start": { + "completion_rate": 1.0, + "episodes": [ + { + "final_lives": 1, + "invalid_action": false, + "length": 108000, + "mean_action_latency_ms": 0.004654731135815382, + "observation_mode": "ram", + "policy_error": null, + "return": 864.0, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "left_6", + "warmup_steps": 6 + }, + { + "final_lives": 1, + "invalid_action": false, + "length": 108000, + "mean_action_latency_ms": 0.005017738013217847, + "observation_mode": "ram", + "policy_error": null, + "return": 864.0, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 864.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.004836234574516615, + "mean_return": 864.0, + "min_return": 864.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 864.0 + } + }, + "target_gap": 0.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-03/metadata.json new file mode 100644 index 0000000..95899f5 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-03/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "digest_compatibility_note": null, + "finished_at": 1784113784.201615, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 98.322851153, + "observed_raw_score": 856.0, + "provider_reasoning_effort": "max", + "raw_score": 856.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "task": "simulation_heuristics_breakout_ram_v1", + "task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 874.0, + "input_tokens": 2.0, + "output_tokens": 874.0, + "total_tokens": 70599.0 + }, + "tool_calls": 52, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-03/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-03/score.json new file mode 100644 index 0000000..d1cc97d --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-03/score.json @@ -0,0 +1,80 @@ +{ + "normalized_score": 98.322851153, + "reference_score": 864.0, + "score": 856.0, + "starter_score": 387.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 1, + "invalid_action": false, + "length": 108000, + "mean_action_latency_ms": 0.005891924245386488, + "observation_mode": "ram", + "policy_error": null, + "return": 856.0, + "seed": 101, + "terminated": false, + "truncated": true, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 856.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.005891924245386488, + "mean_return": 856.0, + "min_return": 856.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 856.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 1, + "invalid_action": false, + "length": 108000, + "mean_action_latency_ms": 0.006003775383794197, + "observation_mode": "ram", + "policy_error": null, + "return": 856.0, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "left_6", + "warmup_steps": 6 + }, + { + "final_lives": 1, + "invalid_action": false, + "length": 108000, + "mean_action_latency_ms": 0.006029885624431902, + "observation_mode": "ram", + "policy_error": null, + "return": 856.0, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 856.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.006016830504113049, + "mean_return": 856.0, + "min_return": 856.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 856.0 + } + }, + "target_gap": -8.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-04/metadata.json new file mode 100644 index 0000000..a9af6dc --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-04/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "digest_compatibility_note": null, + "finished_at": 1784120178.69071, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 84.696016771, + "observed_raw_score": 791.0, + "provider_reasoning_effort": "max", + "raw_score": 791.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "task": "simulation_heuristics_breakout_ram_v1", + "task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 805.0, + "input_tokens": 2.0, + "output_tokens": 805.0, + "total_tokens": 91245.0 + }, + "tool_calls": 77, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-04/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-04/score.json new file mode 100644 index 0000000..0e83e2b --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-04/score.json @@ -0,0 +1,80 @@ +{ + "normalized_score": 84.696016771, + "reference_score": 864.0, + "score": 791.0, + "starter_score": 387.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18976, + "mean_action_latency_ms": 0.022319979131540588, + "observation_mode": "ram", + "policy_error": null, + "return": 791.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 791.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.022319979131540588, + "mean_return": 791.0, + "min_return": 791.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 791.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18970, + "mean_action_latency_ms": 0.02042731011466064, + "observation_mode": "ram", + "policy_error": null, + "return": 791.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "left_6", + "warmup_steps": 6 + }, + { + "final_lives": 0, + "invalid_action": false, + "length": 18970, + "mean_action_latency_ms": 0.021491505713268338, + "observation_mode": "ram", + "policy_error": null, + "return": 791.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 791.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.020959407913964487, + "mean_return": 791.0, + "min_return": 791.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 791.0 + } + }, + "target_gap": -73.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-05/metadata.json new file mode 100644 index 0000000..b4164e6 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-05/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "digest_compatibility_note": null, + "finished_at": 1784110074.558851, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 88.259958071, + "observed_raw_score": 808.0, + "provider_reasoning_effort": "max", + "raw_score": 808.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05", + "task": "simulation_heuristics_breakout_ram_v1", + "task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 1032.0, + "input_tokens": 2.0, + "output_tokens": 1032.0, + "total_tokens": 76194.0 + }, + "tool_calls": 64, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-05/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-05/score.json new file mode 100644 index 0000000..c03445f --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/trial-05/score.json @@ -0,0 +1,80 @@ +{ + "normalized_score": 88.259958071, + "reference_score": 864.0, + "score": 808.0, + "starter_score": 387.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18257, + "mean_action_latency_ms": 0.013532146544999606, + "observation_mode": "ram", + "policy_error": null, + "return": 808.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 808.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.013532146544999606, + "mean_return": 808.0, + "min_return": 808.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 808.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18251, + "mean_action_latency_ms": 0.014314604377633584, + "observation_mode": "ram", + "policy_error": null, + "return": 808.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "left_6", + "warmup_steps": 6 + }, + { + "final_lives": 0, + "invalid_action": false, + "length": 18251, + "mean_action_latency_ms": 0.013893069338634578, + "observation_mode": "ram", + "policy_error": null, + "return": 808.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 808.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.014103836858134081, + "mean_return": 808.0, + "min_return": 808.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 808.0 + } + }, + "target_gap": -56.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/metadata.json index a796e83..2ab5dd4 100644 --- a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/metadata.json +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/metadata.json @@ -1,27 +1,29 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:7e126380af5345a4bfa0c9d29e45e68b8f1aa20d5d655d12187744100b24441a", - "digest_compatibility_note": null, - "finished_at": 1783940397.5345018, + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", "harness": "opencode", "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, "display_name": "Claude Opus 4.8", "id": "claude-opus-4.8", "model": "anthropic/claude-opus-4-8", "provider": "claude_oauth", "provider_reasoning_effort": "max" }, - "normalized_score": 0.0, + "normalized_score": 89.2779783394, + "normalized_score_stddev": 13.287454681897577, "provider_reasoning_effort": "max", - "source_run_id": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "raw_score": 804.6, + "raw_score_imputation_count": 0, + "raw_score_stddev": 73.61249893869926, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], "task": "simulation_heuristics_breakout_rgb_v1", - "task_digest": "sha256:7e126380af5345a4bfa0c9d29e45e68b8f1aa20d5d655d12187744100b24441a", - "token_usage": { - "final_input_tokens": 0.0, - "final_output_tokens": 0.0, - "input_tokens": 0.0, - "output_tokens": 0.0, - "total_tokens": 0.0 - }, - "tool_calls": 46 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/score.json index 6f0c5d2..3f28fef 100644 --- a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/score.json +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/score.json @@ -1,66 +1,59 @@ { - "normalized_score": 0.0, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": 89.2779783394, + "normalized_score_stddev": 13.287454681897577, + "raw_score_imputation_count": 0, "reference_score": 864.0, - "score": 310.0, + "score": 804.6, + "score_stddev": 73.61249893869926, "starter_score": 310.0, - "suites": { - "hidden_nominal": { - "completion_rate": 0.0, - "episodes": [ - { - "final_lives": 0, - "invalid_action": false, - "length": 11358, - "mean_action_latency_ms": 0.4728028780663805, - "observation_mode": "rgb", - "policy_error": null, - "return": 310.0, - "seed": 101, - "terminated": true, - "truncated": false, - "variant": "nominal", - "warmup_steps": 0 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 310.0, - "max_steps": 30000, - "mean_action_latency_ms": 0.4728028780663805, - "mean_return": 310.0, - "min_return": 310.0, - "observation_mode": "rgb", - "policy_path": "submitted_artifact", - "score": 310.0 + "task": "simulation_heuristics_breakout_rgb_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 70.758122744, + "observed_raw_score": 702.0, + "raw_score": 702.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "trial": 1 }, - "hidden_shifted_start": { - "completion_rate": 0.0, - "episodes": [ - { - "final_lives": 0, - "invalid_action": false, - "length": 11352, - "mean_action_latency_ms": 0.48375160852496374, - "observation_mode": "rgb", - "policy_error": null, - "return": 310.0, - "seed": 211, - "terminated": true, - "truncated": false, - "variant": "right_6", - "warmup_steps": 6 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 310.0, - "max_steps": 30000, - "mean_action_latency_ms": 0.48375160852496374, - "mean_return": 310.0, - "min_return": 310.0, - "observation_mode": "rgb", - "policy_path": "submitted_artifact", - "score": 310.0 + { + "normalized_score": 79.783393502, + "observed_raw_score": 752.0, + "raw_score": 752.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "trial": 2 + }, + { + "normalized_score": 100.0, + "observed_raw_score": 864.0, + "raw_score": 864.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "trial": 3 + }, + { + "normalized_score": 95.848375451, + "observed_raw_score": 841.0, + "raw_score": 841.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "trial": 4 + }, + { + "normalized_score": 100.0, + "observed_raw_score": 864.0, + "raw_score": 864.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05", + "trial": 5 } - }, - "target_gap": -554.0, - "target_score": 864.0 + ] } diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-01/metadata.json new file mode 100644 index 0000000..55861a2 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-01/metadata.json @@ -0,0 +1,27 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "digest_compatibility_note": null, + "finished_at": 1784091892.627182, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 0, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 70.758122744, + "observed_raw_score": 702.0, + "provider_reasoning_effort": "max", + "raw_score": 702.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "task": "simulation_heuristics_breakout_rgb_v1", + "task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "token_usage": null, + "tool_calls": 27, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-01/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-01/score.json new file mode 100644 index 0000000..531554f --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-01/score.json @@ -0,0 +1,66 @@ +{ + "normalized_score": 70.758122744, + "reference_score": 864.0, + "score": 702.0, + "starter_score": 310.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18811, + "mean_action_latency_ms": 0.4689395815858075, + "observation_mode": "rgb", + "policy_error": null, + "return": 702.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 702.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.4689395815858075, + "mean_return": 702.0, + "min_return": 702.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 702.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18805, + "mean_action_latency_ms": 0.4954714461768824, + "observation_mode": "rgb", + "policy_error": null, + "return": 702.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 702.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.4954714461768824, + "mean_return": 702.0, + "min_return": 702.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 702.0 + } + }, + "target_gap": -162.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-02/metadata.json new file mode 100644 index 0000000..02c4fe0 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-02/metadata.json @@ -0,0 +1,27 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "digest_compatibility_note": null, + "finished_at": 1784095422.450964, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 79.783393502, + "observed_raw_score": 752.0, + "provider_reasoning_effort": "max", + "raw_score": 752.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "task": "simulation_heuristics_breakout_rgb_v1", + "task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "token_usage": null, + "tool_calls": 29, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-02/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-02/score.json new file mode 100644 index 0000000..e26cc15 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-02/score.json @@ -0,0 +1,66 @@ +{ + "normalized_score": 79.783393502, + "reference_score": 864.0, + "score": 752.0, + "starter_score": 310.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 15015, + "mean_action_latency_ms": 0.4848904382031817, + "observation_mode": "rgb", + "policy_error": null, + "return": 752.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 752.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.4848904382031817, + "mean_return": 752.0, + "min_return": 752.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 752.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 15009, + "mean_action_latency_ms": 0.48138529735958185, + "observation_mode": "rgb", + "policy_error": null, + "return": 752.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 752.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.48138529735958185, + "mean_return": 752.0, + "min_return": 752.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 752.0 + } + }, + "target_gap": -112.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-03/metadata.json new file mode 100644 index 0000000..73c7249 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-03/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "digest_compatibility_note": null, + "finished_at": 1784113784.201615, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 100.0, + "observed_raw_score": 864.0, + "provider_reasoning_effort": "max", + "raw_score": 864.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "task": "simulation_heuristics_breakout_rgb_v1", + "task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 722.0, + "input_tokens": 2.0, + "output_tokens": 722.0, + "total_tokens": 66867.0 + }, + "tool_calls": 51, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-03/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-03/score.json new file mode 100644 index 0000000..3a98942 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-03/score.json @@ -0,0 +1,66 @@ +{ + "normalized_score": 100.0, + "reference_score": 864.0, + "score": 864.0, + "starter_score": 310.0, + "suites": { + "hidden_nominal": { + "completion_rate": 1.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 20302, + "mean_action_latency_ms": 0.5165470093474472, + "observation_mode": "rgb", + "policy_error": null, + "return": 864.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 864.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.5165470093474472, + "mean_return": 864.0, + "min_return": 864.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 864.0 + }, + "hidden_shifted_start": { + "completion_rate": 1.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 20296, + "mean_action_latency_ms": 0.46674083973466285, + "observation_mode": "rgb", + "policy_error": null, + "return": 864.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 864.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.46674083973466285, + "mean_return": 864.0, + "min_return": 864.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 864.0 + } + }, + "target_gap": 0.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-04/metadata.json new file mode 100644 index 0000000..40d16ef --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-04/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "digest_compatibility_note": null, + "finished_at": 1784120178.69071, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 95.848375451, + "observed_raw_score": 841.0, + "provider_reasoning_effort": "max", + "raw_score": 841.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "task": "simulation_heuristics_breakout_rgb_v1", + "task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 653.0, + "input_tokens": 2.0, + "output_tokens": 653.0, + "total_tokens": 47410.0 + }, + "tool_calls": 35, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-04/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-04/score.json new file mode 100644 index 0000000..1b4a50b --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-04/score.json @@ -0,0 +1,66 @@ +{ + "normalized_score": 95.848375451, + "reference_score": 864.0, + "score": 841.0, + "starter_score": 310.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18935, + "mean_action_latency_ms": 0.46257201945180465, + "observation_mode": "rgb", + "policy_error": null, + "return": 841.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 841.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.46257201945180465, + "mean_return": 841.0, + "min_return": 841.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 841.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18929, + "mean_action_latency_ms": 0.4842979390184187, + "observation_mode": "rgb", + "policy_error": null, + "return": 841.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 841.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.4842979390184187, + "mean_return": 841.0, + "min_return": 841.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 841.0 + } + }, + "target_gap": -23.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-05/metadata.json new file mode 100644 index 0000000..7d688f4 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-05/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "digest_compatibility_note": null, + "finished_at": 1784110074.558851, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 100.0, + "observed_raw_score": 864.0, + "provider_reasoning_effort": "max", + "raw_score": 864.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05", + "task": "simulation_heuristics_breakout_rgb_v1", + "task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 541.0, + "input_tokens": 2.0, + "output_tokens": 541.0, + "total_tokens": 42018.0 + }, + "tool_calls": 26, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-05/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-05/score.json new file mode 100644 index 0000000..4eb428c --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/trial-05/score.json @@ -0,0 +1,66 @@ +{ + "normalized_score": 100.0, + "reference_score": 864.0, + "score": 864.0, + "starter_score": 310.0, + "suites": { + "hidden_nominal": { + "completion_rate": 1.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18415, + "mean_action_latency_ms": 0.5421389741941927, + "observation_mode": "rgb", + "policy_error": null, + "return": 864.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 864.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.5421389741941927, + "mean_return": 864.0, + "min_return": 864.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 864.0 + }, + "hidden_shifted_start": { + "completion_rate": 1.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18409, + "mean_action_latency_ms": 0.47512363141202363, + "observation_mode": "rgb", + "policy_error": null, + "return": 864.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 864.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.47512363141202363, + "mean_return": 864.0, + "min_return": 864.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 864.0 + } + }, + "target_gap": 0.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/metadata.json index 2857988..1442526 100644 --- a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/metadata.json +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/metadata.json @@ -1,28 +1,29 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:c7e3b5e466431f4908c1e96cc7de80191f7a7c5ab5a63b3eea4d708f7f447fbf", - "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", - "finished_at": 1783947693.1667001, + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", "harness": "opencode", "model": { - "agent_idle_timeout_sec": 0, + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, "display_name": "Claude Opus 4.8", "id": "claude-opus-4.8", "model": "anthropic/claude-opus-4-8", "provider": "claude_oauth", "provider_reasoning_effort": "max" }, - "normalized_score": 26.986707946, + "normalized_score": 40.934675074999994, + "normalized_score_stddev": 24.40596858830019, "provider_reasoning_effort": "max", - "source_run_id": "leaderboard/runs/article_suite/20260713T111714Z/claude-opus-4.8", + "raw_score": 7065.645225601496, + "raw_score_imputation_count": 0, + "raw_score_stddev": 1827.0229584820154, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], "task": "simulation_heuristics_halfcheetah_v1", - "task_digest": "sha256:80c439f53e4ab964f9d7443cd7fb8f25cf6645a0bc288b0496b871c1800ebe78", - "token_usage": { - "final_input_tokens": 0.0, - "final_output_tokens": 0.0, - "input_tokens": 0.0, - "output_tokens": 0.0, - "total_tokens": 0.0 - }, - "tool_calls": 45 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/score.json index 62f5166..6a2bfba 100644 --- a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/score.json +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/score.json @@ -1,98 +1,59 @@ { - "hidden_nominal": { - "episodes": [ - { - "control_cost": 317.1566189676523, - "forward_reward": 6636.299989167559, - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 122.95847277203575, - "policy_error": null, - "return": 6319.143370199905, - "seed": 211, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": 331.8043416293857, - "x_velocity": 8.216671429596408 - }, - { - "control_cost": 314.24980780109763, - "forward_reward": 6272.707764268625, - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 123.8214553501457, - "policy_error": null, - "return": 5958.45795646753, - "seed": 307, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": 313.6619722793685, - "x_velocity": 2.1795015376377602 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 6319.143370199905, - "max_steps": 1000, - "mean_action_latency_ms": 123.38996406109072, - "mean_control_cost": 315.703213384375, - "mean_forward_reward": 6454.503876718092, - "mean_return": 6138.800663333717, - "mean_x_position": 322.73315695437714, - "min_return": 5958.45795646753, - "policy_path": "submitted_artifact", - "score": 6138.800663333717 - }, - "hidden_nominal_weight": 0.7, - "hidden_robustness": { - "episodes": [ - { - "control_cost": 312.9004553593695, - "forward_reward": 7637.3131141575, - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 119.81285913009197, - "policy_error": null, - "return": 7324.412658798133, - "seed": 503, - "terminated": false, - "truncated": true, - "variant": "light_low_friction", - "x_position": 381.8479773304825, - "x_velocity": 8.203061195650385 - }, - { - "control_cost": 321.7466217055917, - "forward_reward": 4492.963484628226, - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 125.38341190014035, - "policy_error": null, - "return": 4171.216862922633, - "seed": 503, - "terminated": false, - "truncated": true, - "variant": "heavy_weak_actuators", - "x_position": 224.63049585401836, - "x_velocity": 2.2801165132939816 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 7324.412658798133, - "max_steps": 1000, - "mean_action_latency_ms": 122.59813551511616, - "mean_control_cost": 317.3235385324806, - "mean_forward_reward": 6065.138299392864, - "mean_return": 5747.814760860383, - "mean_x_position": 303.23923659225045, - "min_return": 4171.216862922633, - "policy_path": "submitted_artifact", - "score": 5747.814760860383 - }, - "hidden_robustness_weight": 0.3, - "normalized_score": 26.986707946, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": 40.934675074999994, + "normalized_score_stddev": 24.40596858830019, + "raw_score_imputation_count": 0, "reference_score": 11487.256451471967, - "score": 6021.504892591717, - "starter_score": 4001.2886158917113 + "score": 7065.645225601496, + "score_stddev": 1827.0229584820154, + "starter_score": 4001.2886158917113, + "task": "simulation_heuristics_halfcheetah_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 41.669121704, + "observed_raw_score": 7120.625663985211, + "raw_score": 7120.625663985211, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01/tasks-81ec8d98a4", + "trial": 1 + }, + { + "normalized_score": 42.425113199, + "observed_raw_score": 7177.218944151435, + "raw_score": 7177.218944151435, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "trial": 2 + }, + { + "normalized_score": 9.827504681, + "observed_raw_score": 4736.972455374246, + "raw_score": 4736.972455374246, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "trial": 3 + }, + { + "normalized_score": 77.681592164, + "observed_raw_score": 9816.507619476308, + "raw_score": 9816.507619476308, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "trial": 4 + }, + { + "normalized_score": 33.070043627, + "observed_raw_score": 6476.901445020279, + "raw_score": 6476.901445020279, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05", + "trial": 5 + } + ] } diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-01/metadata.json new file mode 100644 index 0000000..bdff75f --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-01/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "digest_compatibility_note": null, + "finished_at": 1784105005.326783, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 41.669121704, + "observed_raw_score": 7120.625663985211, + "provider_reasoning_effort": "max", + "raw_score": 7120.625663985211, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01/tasks-81ec8d98a4", + "task": "simulation_heuristics_halfcheetah_v1", + "task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 879.0, + "input_tokens": 2.0, + "output_tokens": 879.0, + "total_tokens": 66620.0 + }, + "tool_calls": 42, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-01/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-01/score.json new file mode 100644 index 0000000..34e48e5 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-01/score.json @@ -0,0 +1,98 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "control_cost": 216.9793297406286, + "forward_reward": 7295.607231443617, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 107.2283208090812, + "policy_error": null, + "return": 7078.627901702992, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 364.7697037431885, + "x_velocity": 7.8573870029917 + }, + { + "control_cost": 216.39393178746104, + "forward_reward": 7492.55225557065, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 100.47347120195627, + "policy_error": null, + "return": 7276.158323783191, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 374.6541968444695, + "x_velocity": 8.689615893155452 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 7276.158323783191, + "max_steps": 1000, + "mean_action_latency_ms": 103.85089600551873, + "mean_control_cost": 216.68663076404482, + "mean_forward_reward": 7394.079743507134, + "mean_return": 7177.3931127430915, + "mean_x_position": 369.711950293829, + "min_return": 7078.627901702992, + "policy_path": "submitted_artifact", + "score": 7177.3931127430915 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "control_cost": 211.68124675936997, + "forward_reward": 8162.0617044361115, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 99.07558528333902, + "policy_error": null, + "return": 7950.380457676742, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 408.0854068444129, + "x_velocity": 8.569682279233056 + }, + { + "control_cost": 227.81793746631593, + "forward_reward": 6253.7740468898955, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 102.60634107515216, + "policy_error": null, + "return": 6025.956109423578, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 312.67102396710186, + "x_velocity": 6.482589896866079 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 7950.380457676742, + "max_steps": 1000, + "mean_action_latency_ms": 100.84096317924559, + "mean_control_cost": 219.74959211284295, + "mean_forward_reward": 7207.9178756630035, + "mean_return": 6988.16828355016, + "mean_x_position": 360.3782154057574, + "min_return": 6025.956109423578, + "policy_path": "submitted_artifact", + "score": 6988.16828355016 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": 41.669121704, + "reference_score": 11487.256451471967, + "score": 7120.625663985211, + "starter_score": 4001.2886158917113 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-02/metadata.json new file mode 100644 index 0000000..8cbd51e --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-02/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "digest_compatibility_note": null, + "finished_at": 1784103481.173677, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 42.425113199, + "observed_raw_score": 7177.218944151435, + "provider_reasoning_effort": "max", + "raw_score": 7177.218944151435, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "task": "simulation_heuristics_halfcheetah_v1", + "task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 713.0, + "input_tokens": 2.0, + "output_tokens": 713.0, + "total_tokens": 63371.0 + }, + "tool_calls": 54, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-02/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-02/score.json new file mode 100644 index 0000000..b897c55 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-02/score.json @@ -0,0 +1,98 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "control_cost": 482.71121522784233, + "forward_reward": 7724.365205335394, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 93.49850322777638, + "policy_error": null, + "return": 7241.653990107552, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 386.207602437777, + "x_velocity": 7.696556120473804 + }, + { + "control_cost": 481.5849095992744, + "forward_reward": 7749.66527786909, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 91.38956539268838, + "policy_error": null, + "return": 7268.080368269815, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 387.5098479593916, + "x_velocity": 7.579619895990968 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 7268.080368269815, + "max_steps": 1000, + "mean_action_latency_ms": 92.44403431023238, + "mean_control_cost": 482.14806241355836, + "mean_forward_reward": 7737.015241602242, + "mean_return": 7254.867179188684, + "mean_x_position": 386.8587251985843, + "min_return": 7241.653990107552, + "policy_path": "submitted_artifact", + "score": 7254.867179188684 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "control_cost": 474.207764826715, + "forward_reward": 8659.786268333257, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 94.15423411270604, + "policy_error": null, + "return": 8185.578503506544, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 432.97163503927004, + "x_velocity": 8.50221577903123 + }, + { + "control_cost": 488.1766548678279, + "forward_reward": 6294.6776094903225, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 91.42772945581237, + "policy_error": null, + "return": 5806.5009546224965, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 314.71620209712324, + "x_velocity": 7.274202832299448 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 8185.578503506544, + "max_steps": 1000, + "mean_action_latency_ms": 92.7909817842592, + "mean_control_cost": 481.19220984727144, + "mean_forward_reward": 7477.231938911789, + "mean_return": 6996.03972906452, + "mean_x_position": 373.8439185681966, + "min_return": 5806.5009546224965, + "policy_path": "submitted_artifact", + "score": 6996.03972906452 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": 42.425113199, + "reference_score": 11487.256451471967, + "score": 7177.218944151435, + "starter_score": 4001.2886158917113 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-03/metadata.json new file mode 100644 index 0000000..680e091 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-03/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "digest_compatibility_note": null, + "finished_at": 1784113784.201615, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 9.827504681, + "observed_raw_score": 4736.972455374246, + "provider_reasoning_effort": "max", + "raw_score": 4736.972455374246, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "task": "simulation_heuristics_halfcheetah_v1", + "task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 715.0, + "input_tokens": 2.0, + "output_tokens": 715.0, + "total_tokens": 77110.0 + }, + "tool_calls": 43, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-03/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-03/score.json new file mode 100644 index 0000000..7d14281 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-03/score.json @@ -0,0 +1,98 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "control_cost": 182.5611578244716, + "forward_reward": 5023.671149400799, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 66.28844551439397, + "policy_error": null, + "return": 4841.109991576325, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 251.1728996410475, + "x_velocity": 4.118891776198561 + }, + { + "control_cost": 181.0709079708904, + "forward_reward": 4744.209112650303, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 66.24573376518674, + "policy_error": null, + "return": 4563.138204679409, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 237.237039698452, + "x_velocity": 6.551172084272852 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 4841.109991576325, + "max_steps": 1000, + "mean_action_latency_ms": 66.26708963979036, + "mean_control_cost": 181.816032897681, + "mean_forward_reward": 4883.940131025551, + "mean_return": 4702.124098127867, + "mean_x_position": 244.20496966974974, + "min_return": 4563.138204679409, + "policy_path": "submitted_artifact", + "score": 4702.124098127867 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "control_cost": 176.80347473919392, + "forward_reward": 5286.2748889876, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 66.09967904584482, + "policy_error": null, + "return": 5109.4714142484045, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 264.2960660719875, + "x_velocity": 7.130679153278417 + }, + { + "control_cost": 190.2128691636026, + "forward_reward": 4717.312032813463, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 67.95282475976273, + "policy_error": null, + "return": 4527.09916364986, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 235.8479232632804, + "x_velocity": 4.574510613925662 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 5109.4714142484045, + "max_steps": 1000, + "mean_action_latency_ms": 67.02625190280378, + "mean_control_cost": 183.50817195139825, + "mean_forward_reward": 5001.7934609005315, + "mean_return": 4818.285288949132, + "mean_x_position": 250.07199466763396, + "min_return": 4527.09916364986, + "policy_path": "submitted_artifact", + "score": 4818.285288949132 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": 9.827504681, + "reference_score": 11487.256451471967, + "score": 4736.972455374246, + "starter_score": 4001.2886158917113 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-04/metadata.json new file mode 100644 index 0000000..d9f008e --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-04/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "digest_compatibility_note": null, + "finished_at": 1784120178.69071, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 77.681592164, + "observed_raw_score": 9816.507619476308, + "provider_reasoning_effort": "max", + "raw_score": 9816.507619476308, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "task": "simulation_heuristics_halfcheetah_v1", + "task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 685.0, + "input_tokens": 2.0, + "output_tokens": 685.0, + "total_tokens": 60238.0 + }, + "tool_calls": 43, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-04/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-04/score.json new file mode 100644 index 0000000..ed6305b --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-04/score.json @@ -0,0 +1,98 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "control_cost": 313.84867538511753, + "forward_reward": 11530.070047297319, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 268.4568623137893, + "policy_error": null, + "return": 11216.221371912197, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 576.4928445358736, + "x_velocity": 15.71878857193862 + }, + { + "control_cost": 315.44752060621977, + "forward_reward": 10465.72143509192, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 273.3478279002011, + "policy_error": null, + "return": 10150.2739144857, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 523.3126558205325, + "x_velocity": 12.38688854705487 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 11216.221371912197, + "max_steps": 1000, + "mean_action_latency_ms": 270.9023451069952, + "mean_control_cost": 314.64809799566865, + "mean_forward_reward": 10997.89574119462, + "mean_return": 10683.247643198949, + "mean_x_position": 549.9027501782031, + "min_return": 10150.2739144857, + "policy_path": "submitted_artifact", + "score": 10683.247643198949 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "control_cost": 316.3446640148759, + "forward_reward": 12398.42478686888, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 272.70919528073864, + "policy_error": null, + "return": 12082.080122854004, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 619.9035609660523, + "x_velocity": 17.535914067353914 + }, + { + "control_cost": 310.6027032248676, + "forward_reward": 3816.7510419511573, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 310.3828021203517, + "policy_error": null, + "return": 3506.1483387262915, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 190.81987372016516, + "x_velocity": 3.972590016083312 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 12082.080122854004, + "max_steps": 1000, + "mean_action_latency_ms": 291.54599870054517, + "mean_control_cost": 313.47368361987174, + "mean_forward_reward": 8107.587914410018, + "mean_return": 7794.114230790148, + "mean_x_position": 405.3617173431087, + "min_return": 3506.1483387262915, + "policy_path": "submitted_artifact", + "score": 7794.114230790148 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": 77.681592164, + "reference_score": 11487.256451471967, + "score": 9816.507619476308, + "starter_score": 4001.2886158917113 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-05/metadata.json new file mode 100644 index 0000000..843e40b --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-05/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "digest_compatibility_note": null, + "finished_at": 1784110074.558851, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 33.070043627, + "observed_raw_score": 6476.901445020279, + "provider_reasoning_effort": "max", + "raw_score": 6476.901445020279, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05", + "task": "simulation_heuristics_halfcheetah_v1", + "task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 755.0, + "input_tokens": 2.0, + "output_tokens": 755.0, + "total_tokens": 81343.0 + }, + "tool_calls": 78, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-05/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-05/score.json new file mode 100644 index 0000000..c6517e9 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/trial-05/score.json @@ -0,0 +1,98 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "control_cost": 537.0098397731781, + "forward_reward": 7286.725229616571, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 78.96866185136605, + "policy_error": null, + "return": 6749.715389843393, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 364.32560365183633, + "x_velocity": 6.803107815885114 + }, + { + "control_cost": 543.8137017786503, + "forward_reward": 7505.228732072868, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 78.23577671992825, + "policy_error": null, + "return": 6961.415030294214, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 375.28802066957945, + "x_velocity": 8.009307505704985 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 6961.415030294214, + "max_steps": 1000, + "mean_action_latency_ms": 78.60221928564715, + "mean_control_cost": 540.4117707759142, + "mean_forward_reward": 7395.97698084472, + "mean_return": 6855.565210068804, + "mean_x_position": 369.80681216070786, + "min_return": 6749.715389843393, + "policy_path": "submitted_artifact", + "score": 6855.565210068804 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "control_cost": 546.6592897474766, + "forward_reward": 6061.660344240933, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 80.89556574082235, + "policy_error": null, + "return": 5515.001054493453, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 303.0653388346534, + "x_velocity": 0.9667981553032734 + }, + { + "control_cost": 542.7115781009197, + "forward_reward": 6214.41584342158, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 76.65571035514586, + "policy_error": null, + "return": 5671.704265320658, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 310.70311379368593, + "x_velocity": 7.646479966903701 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 5671.704265320658, + "max_steps": 1000, + "mean_action_latency_ms": 78.7756380479841, + "mean_control_cost": 544.6854339241982, + "mean_forward_reward": 6138.038093831256, + "mean_return": 5593.352659907056, + "mean_x_position": 306.88422631416967, + "min_return": 5515.001054493453, + "policy_path": "submitted_artifact", + "score": 5593.352659907056 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": 33.070043627, + "reference_score": 11487.256451471967, + "score": 6476.901445020279, + "starter_score": 4001.2886158917113 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/metadata.json index e58d81a..d20068b 100644 --- a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/metadata.json +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/metadata.json @@ -1,11 +1,10 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:33650c821735e31eea00f2ba46ae09b5d19134820c2ec005e8e96d92cb0b66d0", - "digest_compatibility_note": null, - "finished_at": 1783945917.227471, + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", "harness": "opencode", "model": { - "agent_idle_timeout_sec": 0, + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, "display_name": "Claude Opus 4.8", "id": "claude-opus-4.8", "model": "anthropic/claude-opus-4-8", @@ -13,16 +12,18 @@ "provider_reasoning_effort": "max" }, "normalized_score": 0.0, + "normalized_score_stddev": 0.0, "provider_reasoning_effort": "max", - "source_run_id": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "raw_score": 0.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.0, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], "task": "simulation_heuristics_montezuma_v1", - "task_digest": "sha256:33650c821735e31eea00f2ba46ae09b5d19134820c2ec005e8e96d92cb0b66d0", - "token_usage": { - "final_input_tokens": 0.0, - "final_output_tokens": 0.0, - "input_tokens": 0.0, - "output_tokens": 0.0, - "total_tokens": 0.0 - }, - "tool_calls": 109 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/score.json index 57093ba..4c46778 100644 --- a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/score.json +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/score.json @@ -1,216 +1,59 @@ { + "benchmark": "learning_beyond_gradients_article_suite", "normalized_score": 0.0, + "normalized_score_stddev": 0.0, + "raw_score_imputation_count": 0, "reference_score": 400.0, "score": 0.0, + "score_stddev": 0.0, "starter_score": 0.0, - "suites": { - "hidden_seeds": { - "evaluation": { - "capped_mean_score": 0.0, - "episodes": [ - { - "bootstrap_steps": 0, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.09138794615864754, - "policy_error": null, - "policy_steps": 2600, - "pre_policy_noops": 0, - "return": 0.0, - "seed": 10007, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "canonical" - }, - { - "bootstrap_steps": 0, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.15000148533055416, - "policy_error": null, - "policy_steps": 2600, - "pre_policy_noops": 0, - "return": 0.0, - "seed": 10103, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "canonical" - } - ], - "invalid_episode_rate": 0.0, - "max_steps": 2600, - "mean_action_latency_ms": 0.12069471574460085, - "mean_return": 0.0, - "policy_path": "submitted_artifact", - "recovery_success_rate": 0.0, - "score": 0.0, - "target_score": 400.0, - "target_success_rate": 0.0 - }, - "score": 0.0, - "weight": 0.1 + "task": "simulation_heuristics_montezuma_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "trial": 1 }, - "noop_variants": { - "evaluation": { - "capped_mean_score": 0.0, - "episodes": [ - { - "bootstrap_steps": 0, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.120189058112786, - "policy_error": null, - "policy_steps": 2597, - "pre_policy_noops": 3, - "return": 0.0, - "seed": 10001, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "startup_noop_3" - }, - { - "bootstrap_steps": 0, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.10412243624359288, - "policy_error": null, - "policy_steps": 2587, - "pre_policy_noops": 13, - "return": 0.0, - "seed": 10001, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "startup_noop_13" - } - ], - "invalid_episode_rate": 0.0, - "max_steps": 2600, - "mean_action_latency_ms": 0.11215574717818944, - "mean_return": 0.0, - "policy_path": "submitted_artifact", - "recovery_success_rate": 0.0, - "score": 0.0, - "target_score": 400.0, - "target_success_rate": 0.0 - }, - "score": 0.0, - "weight": 0.1 + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "trial": 2 }, - "recovery": { - "evaluation": { - "capped_mean_score": 0.0, - "episodes": [ - { - "bootstrap_steps": 512, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.09218373670173012, - "policy_error": null, - "policy_steps": 2084, - "pre_policy_noops": 4, - "return": 0.0, - "seed": 10001, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "recover_512_noop_4" - }, - { - "bootstrap_steps": 512, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.1754581480501936, - "policy_error": null, - "policy_steps": 2080, - "pre_policy_noops": 8, - "return": 0.0, - "seed": 10001, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "recover_512_noop_8" - }, - { - "bootstrap_steps": 768, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.0963938997580054, - "policy_error": null, - "policy_steps": 1830, - "pre_policy_noops": 2, - "return": 0.0, - "seed": 10001, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "recover_768_noop_2" - }, - { - "bootstrap_steps": 900, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.09158327189438484, - "policy_error": null, - "policy_steps": 1700, - "pre_policy_noops": 0, - "return": 0.0, - "seed": 10001, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "recover_900" - } - ], - "invalid_episode_rate": 0.0, - "max_steps": 2600, - "mean_action_latency_ms": 0.1139047641010785, - "mean_return": 0.0, - "policy_path": "submitted_artifact", - "recovery_success_rate": 0.0, - "score": 0.0, - "target_score": 400.0, - "target_success_rate": 0.0 - }, - "score": 0.0, - "weight": 0.55 + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "trial": 3 }, - "reproduction": { - "evaluation": { - "capped_mean_score": 0.0, - "episodes": [ - { - "bootstrap_steps": 0, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.09677237544495326, - "policy_error": null, - "policy_steps": 2600, - "pre_policy_noops": 0, - "return": 0.0, - "seed": 10001, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "canonical" - } - ], - "invalid_episode_rate": 0.0, - "max_steps": 2600, - "mean_action_latency_ms": 0.09677237544495326, - "mean_return": 0.0, - "policy_path": "submitted_artifact", - "recovery_success_rate": 0.0, - "score": 0.0, - "target_score": 400.0, - "target_success_rate": 0.0 - }, - "score": 0.0, - "weight": 0.25 + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "trial": 4 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05", + "trial": 5 } - }, - "target_score": 400.0 + ] } diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-01/metadata.json new file mode 100644 index 0000000..fe3443f --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-01/metadata.json @@ -0,0 +1,27 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "digest_compatibility_note": null, + "finished_at": 1784091892.627182, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 0, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "max", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "task": "simulation_heuristics_montezuma_v1", + "task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "token_usage": null, + "tool_calls": 83, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-01/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-01/score.json new file mode 100644 index 0000000..38eb957 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-01/score.json @@ -0,0 +1,216 @@ +{ + "normalized_score": 0.0, + "reference_score": 400.0, + "score": 0.0, + "starter_score": 0.0, + "suites": { + "hidden_seeds": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.13363889884203672, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10007, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.11153222133333866, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10103, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.1225855600876877, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "noop_variants": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.10264223693587778, + "policy_error": null, + "policy_steps": 2597, + "pre_policy_noops": 3, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_3" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.18725695333666423, + "policy_error": null, + "policy_steps": 2587, + "pre_policy_noops": 13, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_13" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.144949595136271, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "recovery": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.17558119535506944, + "policy_error": null, + "policy_steps": 2084, + "pre_policy_noops": 4, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_4" + }, + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.20531030806103864, + "policy_error": null, + "policy_steps": 2080, + "pre_policy_noops": 8, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_8" + }, + { + "bootstrap_steps": 768, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.2152952117478994, + "policy_error": null, + "policy_steps": 1830, + "pre_policy_noops": 2, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_768_noop_2" + }, + { + "bootstrap_steps": 900, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.2336501219200299, + "policy_error": null, + "policy_steps": 1700, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_900" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.20745920927100933, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.55 + }, + "reproduction": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.19716624994404042, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.19716624994404042, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.25 + } + }, + "target_score": 400.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-02/metadata.json new file mode 100644 index 0000000..cadf07a --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-02/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "digest_compatibility_note": null, + "finished_at": 1784103481.173677, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "max", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "task": "simulation_heuristics_montezuma_v1", + "task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 1154.0, + "input_tokens": 2.0, + "output_tokens": 1154.0, + "total_tokens": 203477.0 + }, + "tool_calls": 176, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-02/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-02/score.json new file mode 100644 index 0000000..8501f42 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-02/score.json @@ -0,0 +1,216 @@ +{ + "normalized_score": 0.0, + "reference_score": 400.0, + "score": 0.0, + "starter_score": 0.0, + "suites": { + "hidden_seeds": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.15869200301284975, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10007, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.16896820295817003, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10103, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.16383010298550987, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "noop_variants": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.167863872038501, + "policy_error": null, + "policy_steps": 2597, + "pre_policy_noops": 3, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_3" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.1702725324437028, + "policy_error": null, + "policy_steps": 2587, + "pre_policy_noops": 13, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_13" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.16906820224110192, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "recovery": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.17015963617172414, + "policy_error": null, + "policy_steps": 2084, + "pre_policy_noops": 4, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_4" + }, + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.16862764437414277, + "policy_error": null, + "policy_steps": 2080, + "pre_policy_noops": 8, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_8" + }, + { + "bootstrap_steps": 768, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.16230010335929082, + "policy_error": null, + "policy_steps": 1830, + "pre_policy_noops": 2, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_768_noop_2" + }, + { + "bootstrap_steps": 900, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.16726607771809487, + "policy_error": null, + "policy_steps": 1700, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_900" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.16708836540581315, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.55 + }, + "reproduction": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.19757260049048525, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.19757260049048525, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.25 + } + }, + "target_score": 400.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-03/metadata.json new file mode 100644 index 0000000..012080e --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-03/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "digest_compatibility_note": null, + "finished_at": 1784113784.201615, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "max", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "task": "simulation_heuristics_montezuma_v1", + "task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 1116.0, + "input_tokens": 2.0, + "output_tokens": 1116.0, + "total_tokens": 179280.0 + }, + "tool_calls": 157, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-03/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-03/score.json new file mode 100644 index 0000000..7dd9311 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-03/score.json @@ -0,0 +1,216 @@ +{ + "normalized_score": 0.0, + "reference_score": 400.0, + "score": 0.0, + "starter_score": 0.0, + "suites": { + "hidden_seeds": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.24561254540458322, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10007, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.2738133262699613, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10103, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.2597129358372723, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "noop_variants": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.26810068951279187, + "policy_error": null, + "policy_steps": 2597, + "pre_policy_noops": 3, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_3" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.29698070221493084, + "policy_error": null, + "policy_steps": 2587, + "pre_policy_noops": 13, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_13" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.2825406958638613, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "recovery": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.2891079987660265, + "policy_error": null, + "policy_steps": 2084, + "pre_policy_noops": 4, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_4" + }, + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.2915435451611232, + "policy_error": null, + "policy_steps": 2080, + "pre_policy_noops": 8, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_8" + }, + { + "bootstrap_steps": 768, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.24200746016017075, + "policy_error": null, + "policy_steps": 1830, + "pre_policy_noops": 2, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_768_noop_2" + }, + { + "bootstrap_steps": 900, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.43065784978406396, + "policy_error": null, + "policy_steps": 1700, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_900" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.3133292134678461, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.55 + }, + "reproduction": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.29693001872286773, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.29693001872286773, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.25 + } + }, + "target_score": 400.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-04/metadata.json new file mode 100644 index 0000000..e9cf50e --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-04/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "digest_compatibility_note": null, + "finished_at": 1784120178.69071, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "max", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "task": "simulation_heuristics_montezuma_v1", + "task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 957.0, + "input_tokens": 2.0, + "output_tokens": 957.0, + "total_tokens": 138378.0 + }, + "tool_calls": 108, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-04/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-04/score.json new file mode 100644 index 0000000..d3c302d --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-04/score.json @@ -0,0 +1,216 @@ +{ + "normalized_score": 0.0, + "reference_score": 400.0, + "score": 0.0, + "starter_score": 0.0, + "suites": { + "hidden_seeds": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.19070517047881508, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10007, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.20787954814015672, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10103, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.1992923593094859, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "noop_variants": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.18376840348794776, + "policy_error": null, + "policy_steps": 2597, + "pre_policy_noops": 3, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_3" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.18674363970762292, + "policy_error": null, + "policy_steps": 2587, + "pre_policy_noops": 13, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_13" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.18525602159778534, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "recovery": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.16994389444821997, + "policy_error": null, + "policy_steps": 2084, + "pre_policy_noops": 4, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_4" + }, + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.18020036234072626, + "policy_error": null, + "policy_steps": 2080, + "pre_policy_noops": 8, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_8" + }, + { + "bootstrap_steps": 768, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.1754253388947395, + "policy_error": null, + "policy_steps": 1830, + "pre_policy_noops": 2, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_768_noop_2" + }, + { + "bootstrap_steps": 900, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.1691843068008037, + "policy_error": null, + "policy_steps": 1700, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_900" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.17368847562112233, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.55 + }, + "reproduction": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.19144873045241603, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.19144873045241603, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.25 + } + }, + "target_score": 400.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-05/metadata.json new file mode 100644 index 0000000..02b1eab --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-05/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "digest_compatibility_note": null, + "finished_at": 1784110074.558851, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "max", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05", + "task": "simulation_heuristics_montezuma_v1", + "task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 829.0, + "input_tokens": 2.0, + "output_tokens": 829.0, + "total_tokens": 109401.0 + }, + "tool_calls": 82, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-05/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-05/score.json new file mode 100644 index 0000000..58549fc --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/trial-05/score.json @@ -0,0 +1,216 @@ +{ + "normalized_score": 0.0, + "reference_score": 400.0, + "score": 0.0, + "starter_score": 0.0, + "suites": { + "hidden_seeds": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.23152703750663653, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10007, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.2300503070000559, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10103, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.2307886722533462, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "noop_variants": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.28467528003099035, + "policy_error": null, + "policy_steps": 2597, + "pre_policy_noops": 3, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_3" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.22972625472187236, + "policy_error": null, + "policy_steps": 2587, + "pre_policy_noops": 13, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_13" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.2572007673764314, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "recovery": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.21834217488762142, + "policy_error": null, + "policy_steps": 2084, + "pre_policy_noops": 4, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_4" + }, + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.22418423311766952, + "policy_error": null, + "policy_steps": 2080, + "pre_policy_noops": 8, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_8" + }, + { + "bootstrap_steps": 768, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.20750347865857388, + "policy_error": null, + "policy_steps": 1830, + "pre_policy_noops": 2, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_768_noop_2" + }, + { + "bootstrap_steps": 900, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.1860837300088914, + "policy_error": null, + "policy_steps": 1700, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_900" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.20902840416818907, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.55 + }, + "reproduction": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.3194810065906495, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.3194810065906495, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.25 + } + }, + "target_score": 400.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/metadata.json index c925b50..5c0687b 100644 --- a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/metadata.json +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/metadata.json @@ -1,27 +1,29 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:f0b6f90f1d8072e75f0e10f90f4b51e51e31b3590bb2c2736778ba155b23389f", - "digest_compatibility_note": null, - "finished_at": 1783940397.5345018, + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", "harness": "opencode", "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, "display_name": "Claude Opus 4.8", "id": "claude-opus-4.8", "model": "anthropic/claude-opus-4-8", "provider": "claude_oauth", "provider_reasoning_effort": "max" }, - "normalized_score": 100.0, + "normalized_score": 99.6666666666, + "normalized_score_stddev": 0.7453559926490017, "provider_reasoning_effort": "max", - "source_run_id": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "raw_score": 20.919999999999998, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.17888543819998254, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], "task": "simulation_heuristics_pong_ram_v1", - "task_digest": "sha256:f0b6f90f1d8072e75f0e10f90f4b51e51e31b3590bb2c2736778ba155b23389f", - "token_usage": { - "final_input_tokens": 2.0, - "final_output_tokens": 654.0, - "input_tokens": 2.0, - "output_tokens": 654.0, - "total_tokens": 67139.0 - }, - "tool_calls": 41 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/score.json index 26916ec..abad243 100644 --- a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/score.json +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/score.json @@ -1,133 +1,59 @@ { - "normalized_score": 100.0, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": 99.6666666666, + "normalized_score_stddev": 0.7453559926490017, + "raw_score_imputation_count": 0, "reference_score": 21.0, - "score": 21.0, + "score": 20.919999999999998, + "score_stddev": 0.17888543819998254, "starter_score": -2.9999999999999996, - "suite_weights": { - "hidden_nominal": 0.7, - "hidden_reset_robustness": 0.3 - }, - "suites": { - "hidden_nominal": { - "episodes": [ - { - "invalid_action": false, - "length": 22904, - "mean_action_latency_ms": 0.0027309710952318966, - "points_against": 0, - "points_for": 21, - "policy_error": null, - "score": 21.0, - "seed": 101, - "target_reached": true, - "terminated": true, - "truncated": false, - "variant": "article_nominal", - "won": true - }, - { - "invalid_action": false, - "length": 22904, - "mean_action_latency_ms": 0.0028047818248786377, - "points_against": 0, - "points_for": 21, - "policy_error": null, - "score": 21.0, - "seed": 211, - "target_reached": true, - "terminated": true, - "truncated": false, - "variant": "article_nominal", - "won": true - }, - { - "invalid_action": false, - "length": 22904, - "mean_action_latency_ms": 0.002708563199211491, - "points_against": 0, - "points_for": 21, - "policy_error": null, - "score": 21.0, - "seed": 307, - "target_reached": true, - "terminated": true, - "truncated": false, - "variant": "article_nominal", - "won": true - } - ], - "invalid_episode_rate": 0.0, - "max_score": 21.0, - "max_steps": 27000, - "mean_action_latency_ms": 0.002748105373107342, - "mean_score": 21.0, - "min_score": 21.0, - "policy_path": "submitted_artifact", - "score": 21.0, - "target_score": 21.0, - "target_score_rate": 1.0, - "win_rate": 1.0 + "task": "simulation_heuristics_pong_ram_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 100.0, + "observed_raw_score": 21.0, + "raw_score": 21.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "trial": 1 }, - "hidden_reset_robustness": { - "episodes": [ - { - "invalid_action": false, - "length": 22904, - "mean_action_latency_ms": 0.0026605053441459132, - "points_against": 0, - "points_for": 21, - "policy_error": null, - "score": 21.0, - "seed": 503, - "target_reached": true, - "terminated": true, - "truncated": false, - "variant": "randomized_reset", - "won": true - }, - { - "invalid_action": false, - "length": 22900, - "mean_action_latency_ms": 0.0028647481988481036, - "points_against": 0, - "points_for": 21, - "policy_error": null, - "score": 21.0, - "seed": 601, - "target_reached": true, - "terminated": true, - "truncated": false, - "variant": "randomized_reset", - "won": true - }, - { - "invalid_action": false, - "length": 22901, - "mean_action_latency_ms": 0.003009833623868847, - "points_against": 0, - "points_for": 21, - "policy_error": null, - "score": 21.0, - "seed": 701, - "target_reached": true, - "terminated": true, - "truncated": false, - "variant": "randomized_reset", - "won": true - } - ], - "invalid_episode_rate": 0.0, - "max_score": 21.0, - "max_steps": 27000, - "mean_action_latency_ms": 0.0028450290556209547, - "mean_score": 21.0, - "min_score": 21.0, - "policy_path": "submitted_artifact", - "score": 21.0, - "target_score": 21.0, - "target_score_rate": 1.0, - "win_rate": 1.0 + { + "normalized_score": 100.0, + "observed_raw_score": 21.0, + "raw_score": 21.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "trial": 2 + }, + { + "normalized_score": 98.333333333, + "observed_raw_score": 20.6, + "raw_score": 20.6, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "trial": 3 + }, + { + "normalized_score": 100.0, + "observed_raw_score": 21.0, + "raw_score": 21.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "trial": 4 + }, + { + "normalized_score": 100.0, + "observed_raw_score": 21.0, + "raw_score": 21.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05", + "trial": 5 } - }, - "target_score": 21.0 + ] } diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-01/metadata.json new file mode 100644 index 0000000..d6e56bc --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-01/metadata.json @@ -0,0 +1,27 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "digest_compatibility_note": null, + "finished_at": 1784091892.627182, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 0, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 100.0, + "observed_raw_score": 21.0, + "provider_reasoning_effort": "max", + "raw_score": 21.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "task": "simulation_heuristics_pong_ram_v1", + "task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "token_usage": null, + "tool_calls": 33, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-01/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-01/score.json new file mode 100644 index 0000000..750812f --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-01/score.json @@ -0,0 +1,133 @@ +{ + "normalized_score": 100.0, + "reference_score": 21.0, + "score": 21.0, + "starter_score": -2.9999999999999996, + "suite_weights": { + "hidden_nominal": 0.7, + "hidden_reset_robustness": 0.3 + }, + "suites": { + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 13022, + "mean_action_latency_ms": 0.003845989117674922, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 101, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 13022, + "mean_action_latency_ms": 0.00348335048218128, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 211, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 13022, + "mean_action_latency_ms": 0.00331374755483381, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 307, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 21.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.0035476957182300037, + "mean_score": 21.0, + "min_score": 21.0, + "policy_path": "submitted_artifact", + "score": 21.0, + "target_score": 21.0, + "target_score_rate": 1.0, + "win_rate": 1.0 + }, + "hidden_reset_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 13022, + "mean_action_latency_ms": 0.0034263546598897543, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 503, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 13018, + "mean_action_latency_ms": 0.003591433183840502, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 601, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 13019, + "mean_action_latency_ms": 0.0032570316770371317, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 701, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 21.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.003424939840255796, + "mean_score": 21.0, + "min_score": 21.0, + "policy_path": "submitted_artifact", + "score": 21.0, + "target_score": 21.0, + "target_score_rate": 1.0, + "win_rate": 1.0 + } + }, + "target_score": 21.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-02/metadata.json new file mode 100644 index 0000000..3419217 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-02/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "digest_compatibility_note": null, + "finished_at": 1784103481.173677, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 100.0, + "observed_raw_score": 21.0, + "provider_reasoning_effort": "max", + "raw_score": 21.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "task": "simulation_heuristics_pong_ram_v1", + "task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 718.0, + "input_tokens": 2.0, + "output_tokens": 718.0, + "total_tokens": 89133.0 + }, + "tool_calls": 91, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-02/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-02/score.json new file mode 100644 index 0000000..3502343 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-02/score.json @@ -0,0 +1,133 @@ +{ + "normalized_score": 100.0, + "reference_score": 21.0, + "score": 21.0, + "starter_score": -2.9999999999999996, + "suite_weights": { + "hidden_nominal": 0.7, + "hidden_reset_robustness": 0.3 + }, + "suites": { + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 22454, + "mean_action_latency_ms": 0.0028379423181053775, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 101, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 22454, + "mean_action_latency_ms": 0.0030083261738271063, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 211, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 22454, + "mean_action_latency_ms": 0.0031064807097996197, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 307, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 21.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.0029842497339107013, + "mean_score": 21.0, + "min_score": 21.0, + "policy_path": "submitted_artifact", + "score": 21.0, + "target_score": 21.0, + "target_score_rate": 1.0, + "win_rate": 1.0 + }, + "hidden_reset_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 22454, + "mean_action_latency_ms": 0.002851974543184374, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 503, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 22226, + "mean_action_latency_ms": 0.0029462226961921973, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 601, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 22227, + "mean_action_latency_ms": 0.0029047821996318816, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 701, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 21.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.002900993146336151, + "mean_score": 21.0, + "min_score": 21.0, + "policy_path": "submitted_artifact", + "score": 21.0, + "target_score": 21.0, + "target_score_rate": 1.0, + "win_rate": 1.0 + } + }, + "target_score": 21.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-03/metadata.json new file mode 100644 index 0000000..eaf7605 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-03/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "digest_compatibility_note": null, + "finished_at": 1784113784.201615, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 98.333333333, + "observed_raw_score": 20.6, + "provider_reasoning_effort": "max", + "raw_score": 20.6, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "task": "simulation_heuristics_pong_ram_v1", + "task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 636.0, + "input_tokens": 2.0, + "output_tokens": 636.0, + "total_tokens": 53285.0 + }, + "tool_calls": 39, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-03/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-03/score.json new file mode 100644 index 0000000..2f28595 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-03/score.json @@ -0,0 +1,133 @@ +{ + "normalized_score": 98.333333333, + "reference_score": 21.0, + "score": 20.6, + "starter_score": -2.9999999999999996, + "suite_weights": { + "hidden_nominal": 0.7, + "hidden_reset_robustness": 0.3 + }, + "suites": { + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 26488, + "mean_action_latency_ms": 0.0027088946167788037, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 101, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 26488, + "mean_action_latency_ms": 0.002499058073635765, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 211, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 26488, + "mean_action_latency_ms": 0.002477960235791405, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 307, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 21.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.0025619709754019916, + "mean_score": 21.0, + "min_score": 21.0, + "policy_path": "submitted_artifact", + "score": 21.0, + "target_score": 21.0, + "target_score_rate": 1.0, + "win_rate": 1.0 + }, + "hidden_reset_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 26488, + "mean_action_latency_ms": 0.002778544350627363, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 503, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 25288, + "mean_action_latency_ms": 0.0025734643523906303, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 601, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.002560478548898741, + "points_against": 0, + "points_for": 17, + "policy_error": null, + "score": 17.0, + "seed": 701, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "randomized_reset", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 21.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.002637495750638911, + "mean_score": 19.666666666666668, + "min_score": 17.0, + "policy_path": "submitted_artifact", + "score": 19.666666666666668, + "target_score": 21.0, + "target_score_rate": 0.6666666666666666, + "win_rate": 1.0 + } + }, + "target_score": 21.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-04/metadata.json new file mode 100644 index 0000000..29a4f07 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-04/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "digest_compatibility_note": null, + "finished_at": 1784120178.69071, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 100.0, + "observed_raw_score": 21.0, + "provider_reasoning_effort": "max", + "raw_score": 21.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "task": "simulation_heuristics_pong_ram_v1", + "task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 655.0, + "input_tokens": 2.0, + "output_tokens": 655.0, + "total_tokens": 44227.0 + }, + "tool_calls": 36, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-04/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-04/score.json new file mode 100644 index 0000000..9a2b88b --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-04/score.json @@ -0,0 +1,133 @@ +{ + "normalized_score": 100.0, + "reference_score": 21.0, + "score": 21.0, + "starter_score": -2.9999999999999996, + "suite_weights": { + "hidden_nominal": 0.7, + "hidden_reset_robustness": 0.3 + }, + "suites": { + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 15668, + "mean_action_latency_ms": 0.0024617414361943974, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 101, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 15668, + "mean_action_latency_ms": 0.0025949721740673344, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 211, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 15668, + "mean_action_latency_ms": 0.0026414834015463227, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 307, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 21.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.002566065670602685, + "mean_score": 21.0, + "min_score": 21.0, + "policy_path": "submitted_artifact", + "score": 21.0, + "target_score": 21.0, + "target_score_rate": 1.0, + "win_rate": 1.0 + }, + "hidden_reset_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 15668, + "mean_action_latency_ms": 0.0024331453364372885, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 503, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 15664, + "mean_action_latency_ms": 0.00271954305335143, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 601, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 15665, + "mean_action_latency_ms": 0.002665638302664216, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 701, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 21.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.0026061088974843115, + "mean_score": 21.0, + "min_score": 21.0, + "policy_path": "submitted_artifact", + "score": 21.0, + "target_score": 21.0, + "target_score_rate": 1.0, + "win_rate": 1.0 + } + }, + "target_score": 21.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-05/metadata.json new file mode 100644 index 0000000..a291875 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-05/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "digest_compatibility_note": null, + "finished_at": 1784110074.558851, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 100.0, + "observed_raw_score": 21.0, + "provider_reasoning_effort": "max", + "raw_score": 21.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05", + "task": "simulation_heuristics_pong_ram_v1", + "task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 526.0, + "input_tokens": 2.0, + "output_tokens": 526.0, + "total_tokens": 55373.0 + }, + "tool_calls": 46, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-05/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-05/score.json new file mode 100644 index 0000000..3db8105 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/trial-05/score.json @@ -0,0 +1,133 @@ +{ + "normalized_score": 100.0, + "reference_score": 21.0, + "score": 21.0, + "starter_score": -2.9999999999999996, + "suite_weights": { + "hidden_nominal": 0.7, + "hidden_reset_robustness": 0.3 + }, + "suites": { + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 13974, + "mean_action_latency_ms": 0.00260778553439996, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 101, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 13974, + "mean_action_latency_ms": 0.002719340187471978, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 211, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 13974, + "mean_action_latency_ms": 0.0028223785357417825, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 307, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 21.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.0027165014192045735, + "mean_score": 21.0, + "min_score": 21.0, + "policy_path": "submitted_artifact", + "score": 21.0, + "target_score": 21.0, + "target_score_rate": 1.0, + "win_rate": 1.0 + }, + "hidden_reset_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 13974, + "mean_action_latency_ms": 0.002692841905865915, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 503, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 17582, + "mean_action_latency_ms": 0.002550391572547822, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 601, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 17583, + "mean_action_latency_ms": 0.0027424702412885645, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 701, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 21.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.002661901239900767, + "mean_score": 21.0, + "min_score": 21.0, + "policy_path": "submitted_artifact", + "score": 21.0, + "target_score": 21.0, + "target_score_rate": 1.0, + "win_rate": 1.0 + } + }, + "target_score": 21.0 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/metadata.json index 09b40a6..18eb2c9 100644 --- a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/metadata.json +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/metadata.json @@ -1,28 +1,29 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:316e198d8fa71e705c8b7fa54f41faf64e1f3c0ba5c4c92b8116b4bf78c5712b", - "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", - "finished_at": 1783945917.227471, + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", "harness": "opencode", "model": { - "agent_idle_timeout_sec": 0, + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, "display_name": "Claude Opus 4.8", "id": "claude-opus-4.8", "model": "anthropic/claude-opus-4-8", "provider": "claude_oauth", "provider_reasoning_effort": "max" }, - "normalized_score": 95.055812742, + "normalized_score": 107.1451288772, + "normalized_score_stddev": 14.745599637347764, "provider_reasoning_effort": "max", - "source_run_id": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "raw_score": 0.8446999486498534, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.05547294365604739, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], "task": "simulation_heuristics_vizdoom_d1_v1", - "task_digest": "sha256:b3431b238bec4e66af6189d30bcd15d5cd227144dfe8bf0dd1011aa5416c1436", - "token_usage": { - "final_input_tokens": 2.0, - "final_output_tokens": 886.0, - "input_tokens": 2.0, - "output_tokens": 886.0, - "total_tokens": 76716.0 - }, - "tool_calls": 74 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/score.json index f0c8e14..813af40 100644 --- a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/score.json +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/score.json @@ -1,211 +1,59 @@ { - "anchor_calibration": { - "envpool_version": "1.1.1", - "mode": "numeric", - "platform": "linux-x86_64" - }, - "isolation": { - "mode": "subprocess_per_suite", - "suite_process_count": 2, - "unique_working_directories": true - }, - "normalized_score": 95.055812742, - "reference_score": 0.8178199748694897, - "resource_cleanup": { - "removed_bytes": 22517424, - "removed_resources": 27, - "runs": [ - { - "removed_bytes": 22517424, - "removed_resources": 27, - "terminated_processes": 0 - }, - { - "removed_bytes": 0, - "removed_resources": 0, - "terminated_processes": 0 - } - ], - "terminated_processes": 0 - }, - "score": 0.7992199431359768, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": 107.1451288772, + "normalized_score_stddev": 14.745599637347764, + "raw_score_imputation_count": 0, + "reference_score": 0.8178199748694898, + "score": 0.8446999486498534, + "score_stddev": 0.05547294365604739, "starter_score": 0.44161998964846133, - "suites": { - "hidden_nominal": { - "batch_seed": 101, - "envpool_version": "1.1.1", - "episodes": [ - { - "final_variables": { - "HEALTH": 60.0 - }, - "invalid_action": false, - "lane": 0, - "length": 2100, - "mean_action_latency_ms": 0.0805211794518289, - "policy_error": null, - "return": 0.7839999310672283, - "seed": 101, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "HEALTH": 92.0 - }, - "invalid_action": false, - "lane": 1, - "length": 2100, - "mean_action_latency_ms": 0.07343609506885211, - "policy_error": null, - "return": 0.8799999393522739, - "seed": 102, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "HEALTH": 92.0 - }, - "invalid_action": false, - "lane": 2, - "length": 2100, - "mean_action_latency_ms": 0.06823671626902762, - "policy_error": null, - "return": 0.9279999360442162, - "seed": 103, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "HEALTH": 85.0 - }, - "invalid_action": false, - "lane": 3, - "length": 2100, - "mean_action_latency_ms": 0.06539602542207354, - "policy_error": null, - "return": 0.7649999484419823, - "seed": 104, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "HEALTH": 92.0 - }, - "invalid_action": false, - "lane": 4, - "length": 2100, - "mean_action_latency_ms": 0.06430936090293385, - "policy_error": null, - "return": 0.8799999393522739, - "seed": 105, - "terminated": false, - "truncated": true - } - ], - "frame_skip": 1, - "invalid_episode_rate": 0.0, - "max_return": 0.9279999360442162, - "max_steps": 2100, - "mean_action_latency_ms": 0.07037987542294319, - "mean_return": 0.8473999388515949, - "min_return": 0.7649999484419823, - "policy_path": "submitted_artifact", - "render_height": 180, - "render_width": 240, - "scenario": "d1", - "score": 0.8473999388515949 + "task": "simulation_heuristics_vizdoom_d1_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 124.779364311, + "observed_raw_score": 0.9110399397462606, + "raw_score": 0.9110399397462606, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "trial": 1 }, - "hidden_secondary": { - "batch_seed": 607, - "envpool_version": "1.1.1", - "episodes": [ - { - "final_variables": { - "HEALTH": 100.0 - }, - "invalid_action": false, - "lane": 0, - "length": 2100, - "mean_action_latency_ms": 0.09118304100065004, - "policy_error": null, - "return": 0.9359999448060989, - "seed": 607, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "HEALTH": 100.0 - }, - "invalid_action": false, - "lane": 1, - "length": 2100, - "mean_action_latency_ms": 0.07564157158845947, - "policy_error": null, - "return": 0.8399999439716339, - "seed": 608, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "HEALTH": 6.0 - }, - "invalid_action": false, - "lane": 2, - "length": 768, - "mean_action_latency_ms": 0.05176020446621502, - "policy_error": null, - "return": -0.062000010162591934, - "seed": 609, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "HEALTH": 92.0 - }, - "invalid_action": false, - "lane": 3, - "length": 2100, - "mean_action_latency_ms": 0.06992494243951071, - "policy_error": null, - "return": 0.8159999437630177, - "seed": 610, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "HEALTH": 100.0 - }, - "invalid_action": false, - "lane": 4, - "length": 2100, - "mean_action_latency_ms": 0.07116406978595824, - "policy_error": null, - "return": 0.9039999432861805, - "seed": 611, - "terminated": false, - "truncated": true - } - ], - "frame_skip": 1, - "invalid_episode_rate": 0.0, - "max_return": 0.9359999448060989, - "max_steps": 2100, - "mean_action_latency_ms": 0.0719347658561587, - "mean_return": 0.6867999531328678, - "min_return": -0.062000010162591934, - "policy_path": "submitted_artifact", - "render_height": 180, - "render_width": 240, - "scenario": "d1", - "score": 0.6867999531328678 + { + "normalized_score": 100.2179605, + "observed_raw_score": 0.8186399422399699, + "raw_score": 0.8186399422399699, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "trial": 2 + }, + { + "normalized_score": 85.890478581, + "observed_raw_score": 0.7647399573773146, + "raw_score": 0.7647399573773146, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "trial": 3 + }, + { + "normalized_score": 113.721417769, + "observed_raw_score": 0.869439946487546, + "raw_score": 0.869439946487546, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "trial": 4 + }, + { + "normalized_score": 111.116423225, + "observed_raw_score": 0.8596399573981761, + "raw_score": 0.8596399573981761, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05", + "trial": 5 } - } + ] } diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-01/metadata.json new file mode 100644 index 0000000..45a1e7c --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-01/metadata.json @@ -0,0 +1,27 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "digest_compatibility_note": null, + "finished_at": 1784091892.627182, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 0, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 124.779364311, + "observed_raw_score": 0.9110399397462606, + "provider_reasoning_effort": "max", + "raw_score": 0.9110399397462606, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "task": "simulation_heuristics_vizdoom_d1_v1", + "task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "token_usage": null, + "tool_calls": 56, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-01/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-01/score.json new file mode 100644 index 0000000..554963c --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-01/score.json @@ -0,0 +1,211 @@ +{ + "anchor_calibration": { + "envpool_version": "1.1.1", + "mode": "numeric", + "platform": "linux-x86_64" + }, + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 124.779364311, + "reference_score": 0.8178199748694897, + "resource_cleanup": { + "removed_bytes": 22517424, + "removed_resources": 27, + "runs": [ + { + "removed_bytes": 22517424, + "removed_resources": 27, + "terminated_processes": 0 + }, + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + } + ], + "terminated_processes": 0 + }, + "score": 0.9110399397462606, + "starter_score": 0.44161998964846133, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 92.0 + }, + "invalid_action": false, + "lane": 0, + "length": 2100, + "mean_action_latency_ms": 0.24391628713125274, + "policy_error": null, + "return": 0.9279999397695065, + "seed": 101, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 92.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.21579154367957795, + "policy_error": null, + "return": 0.8959999457001686, + "seed": 102, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 2, + "length": 2100, + "mean_action_latency_ms": 0.2100713967922188, + "policy_error": null, + "return": 0.8079999461770058, + "seed": 103, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 3, + "length": 2100, + "mean_action_latency_ms": 0.2112453971945104, + "policy_error": null, + "return": 1.0159999318420887, + "seed": 104, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 92.0 + }, + "invalid_action": false, + "lane": 4, + "length": 2100, + "mean_action_latency_ms": 0.20802469658000128, + "policy_error": null, + "return": 0.8639999404549599, + "seed": 105, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 1.0159999318420887, + "max_steps": 2100, + "mean_action_latency_ms": 0.21780986427551224, + "mean_return": 0.9023999407887459, + "min_return": 0.8079999461770058, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.9023999407887459 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 92.0 + }, + "invalid_action": false, + "lane": 0, + "length": 2100, + "mean_action_latency_ms": 0.2690378620865799, + "policy_error": null, + "return": 0.9119999371469021, + "seed": 607, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 92.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.23362073515142714, + "policy_error": null, + "return": 0.8479999415576458, + "seed": 608, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 2, + "length": 2100, + "mean_action_latency_ms": 0.2250840284285091, + "policy_error": null, + "return": 1.0159999318420887, + "seed": 609, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 92.0 + }, + "invalid_action": false, + "lane": 3, + "length": 2100, + "mean_action_latency_ms": 0.22367460564488456, + "policy_error": null, + "return": 0.895999938249588, + "seed": 610, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 4, + "length": 2100, + "mean_action_latency_ms": 0.21831203252077103, + "policy_error": null, + "return": 0.9839999377727509, + "seed": 611, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 1.0159999318420887, + "max_steps": 2100, + "mean_action_latency_ms": 0.23394585276643437, + "mean_return": 0.9311999373137951, + "min_return": 0.8479999415576458, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.9311999373137951 + } + } +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-02/metadata.json new file mode 100644 index 0000000..bf9b224 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-02/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "digest_compatibility_note": null, + "finished_at": 1784103481.173677, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 100.2179605, + "observed_raw_score": 0.8186399422399699, + "provider_reasoning_effort": "max", + "raw_score": 0.8186399422399699, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "task": "simulation_heuristics_vizdoom_d1_v1", + "task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 497.0, + "input_tokens": 2.0, + "output_tokens": 497.0, + "total_tokens": 44166.0 + }, + "tool_calls": 45, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-02/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-02/score.json new file mode 100644 index 0000000..694c1d4 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-02/score.json @@ -0,0 +1,211 @@ +{ + "anchor_calibration": { + "envpool_version": "1.1.1", + "mode": "numeric", + "platform": "linux-x86_64" + }, + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 100.2179605, + "reference_score": 0.8178199748694897, + "resource_cleanup": { + "removed_bytes": 22517424, + "removed_resources": 27, + "runs": [ + { + "removed_bytes": 22517424, + "removed_resources": 27, + "terminated_processes": 10 + }, + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + } + ], + "terminated_processes": 10 + }, + "score": 0.8186399422399699, + "starter_score": 0.44161998964846133, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 92.0 + }, + "invalid_action": false, + "lane": 0, + "length": 2100, + "mean_action_latency_ms": 0.3118751056137539, + "policy_error": null, + "return": 0.9599999338388443, + "seed": 101, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 92.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.2870050250064759, + "policy_error": null, + "return": 0.9439999349415302, + "seed": 102, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 92.0 + }, + "invalid_action": false, + "lane": 2, + "length": 2100, + "mean_action_latency_ms": 0.27801345235535074, + "policy_error": null, + "return": 1.0399999283254147, + "seed": 103, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 84.0 + }, + "invalid_action": false, + "lane": 3, + "length": 2100, + "mean_action_latency_ms": 0.2784380777960732, + "policy_error": null, + "return": 0.9999999292194843, + "seed": 104, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 4, + "length": 2100, + "mean_action_latency_ms": 0.27746129337520825, + "policy_error": null, + "return": 1.0479999296367168, + "seed": 105, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 1.0479999296367168, + "max_steps": 2100, + "mean_action_latency_ms": 0.2865585908293724, + "mean_return": 0.9983999311923981, + "min_return": 0.9439999349415302, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.9983999311923981 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 8.0 + }, + "invalid_action": false, + "lane": 0, + "length": 896, + "mean_action_latency_ms": 0.263508569332771, + "policy_error": null, + "return": 0.011999990791082382, + "seed": 607, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 84.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.2698555127495811, + "policy_error": null, + "return": 0.9359999531880021, + "seed": 608, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 2, + "length": 1184, + "mean_action_latency_ms": 0.2511514671700629, + "policy_error": null, + "return": 0.13599997013807297, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 3, + "length": 2100, + "mean_action_latency_ms": 0.2544784714423475, + "policy_error": null, + "return": 1.0479999370872974, + "seed": 610, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 4, + "length": 640, + "mean_action_latency_ms": 0.22346189070958644, + "policy_error": null, + "return": -0.13600001111626625, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 1.0479999370872974, + "max_steps": 2100, + "mean_action_latency_ms": 0.2524911822808698, + "mean_return": 0.39919996801763774, + "min_return": -0.13600001111626625, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.39919996801763774 + } + } +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-03/metadata.json new file mode 100644 index 0000000..f2ef1d3 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-03/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "digest_compatibility_note": null, + "finished_at": 1784113784.201615, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 85.890478581, + "observed_raw_score": 0.7647399573773146, + "provider_reasoning_effort": "max", + "raw_score": 0.7647399573773146, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "task": "simulation_heuristics_vizdoom_d1_v1", + "task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 643.0, + "input_tokens": 2.0, + "output_tokens": 643.0, + "total_tokens": 80001.0 + }, + "tool_calls": 82, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-03/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-03/score.json new file mode 100644 index 0000000..7962df8 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-03/score.json @@ -0,0 +1,211 @@ +{ + "anchor_calibration": { + "envpool_version": "1.1.1", + "mode": "numeric", + "platform": "linux-x86_64" + }, + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 85.890478581, + "reference_score": 0.8178199748694897, + "resource_cleanup": { + "removed_bytes": 0, + "removed_resources": 0, + "runs": [ + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + }, + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + } + ], + "terminated_processes": 0 + }, + "score": 0.7647399573773146, + "starter_score": 0.44161998964846133, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 0, + "length": 2100, + "mean_action_latency_ms": 0.1697839761064166, + "policy_error": null, + "return": 0.8399999514222145, + "seed": 101, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.13330904323430287, + "policy_error": null, + "return": 1.0479999519884586, + "seed": 102, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 2, + "length": 768, + "mean_action_latency_ms": 0.10509851078192393, + "policy_error": null, + "return": -0.08800001442432404, + "seed": 103, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 3, + "length": 2100, + "mean_action_latency_ms": 0.12157683127692767, + "policy_error": null, + "return": 1.015999961644411, + "seed": 104, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 4, + "length": 2100, + "mean_action_latency_ms": 0.13299051760917618, + "policy_error": null, + "return": 0.9359999485313892, + "seed": 105, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 1.0479999519884586, + "max_steps": 2100, + "mean_action_latency_ms": 0.13255177580174946, + "mean_return": 0.7503999598324299, + "min_return": -0.08800001442432404, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.7503999598324299 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 55.0 + }, + "invalid_action": false, + "lane": 0, + "length": 2100, + "mean_action_latency_ms": 0.13740847774204754, + "policy_error": null, + "return": 0.7909999527037144, + "seed": 607, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.12030396433103652, + "policy_error": null, + "return": 0.8719999548047781, + "seed": 608, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 2, + "length": 1952, + "mean_action_latency_ms": 0.10989920606409184, + "policy_error": null, + "return": 0.4559999629855156, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 3, + "length": 2100, + "mean_action_latency_ms": 0.11926762493593353, + "policy_error": null, + "return": 0.9519999362528324, + "seed": 610, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 4, + "length": 2100, + "mean_action_latency_ms": 0.10791894669334094, + "policy_error": null, + "return": 0.9199999514967203, + "seed": 611, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 0.9519999362528324, + "max_steps": 2100, + "mean_action_latency_ms": 0.11895964395329009, + "mean_return": 0.7981999516487122, + "min_return": 0.4559999629855156, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.7981999516487122 + } + } +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-04/metadata.json new file mode 100644 index 0000000..1d499e4 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-04/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "digest_compatibility_note": null, + "finished_at": 1784120178.69071, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 113.721417769, + "observed_raw_score": 0.869439946487546, + "provider_reasoning_effort": "max", + "raw_score": 0.869439946487546, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "task": "simulation_heuristics_vizdoom_d1_v1", + "task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 690.0, + "input_tokens": 2.0, + "output_tokens": 690.0, + "total_tokens": 75108.0 + }, + "tool_calls": 77, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-04/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-04/score.json new file mode 100644 index 0000000..dd23ce4 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-04/score.json @@ -0,0 +1,211 @@ +{ + "anchor_calibration": { + "envpool_version": "1.1.1", + "mode": "numeric", + "platform": "linux-x86_64" + }, + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 113.721417769, + "reference_score": 0.8178199748694897, + "resource_cleanup": { + "removed_bytes": 2501936, + "removed_resources": 3, + "runs": [ + { + "removed_bytes": 2501936, + "removed_resources": 3, + "terminated_processes": 0 + }, + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + } + ], + "terminated_processes": 0 + }, + "score": 0.869439946487546, + "starter_score": 0.44161998964846133, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 0, + "length": 2100, + "mean_action_latency_ms": 1.1248829153676827, + "policy_error": null, + "return": 0.9999999403953552, + "seed": 101, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 1.1053740046918392, + "policy_error": null, + "return": 0.8879999537020922, + "seed": 102, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 92.0 + }, + "invalid_action": false, + "lane": 2, + "length": 2100, + "mean_action_latency_ms": 1.1104905365833215, + "policy_error": null, + "return": 0.9439999349415302, + "seed": 103, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 3, + "length": 2100, + "mean_action_latency_ms": 1.1052003697980017, + "policy_error": null, + "return": 0.9359999448060989, + "seed": 104, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 4, + "length": 2100, + "mean_action_latency_ms": 1.1021780071868783, + "policy_error": null, + "return": 0.871999941766262, + "seed": 105, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 0.9999999403953552, + "max_steps": 2100, + "mean_action_latency_ms": 1.1096251667255446, + "mean_return": 0.9279999431222677, + "min_return": 0.871999941766262, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.9279999431222677 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 0, + "length": 1280, + "mean_action_latency_ms": 1.1242613807553425, + "policy_error": null, + "return": 0.11999997682869434, + "seed": 607, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 1.1289286373981409, + "policy_error": null, + "return": 0.871999941766262, + "seed": 608, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 2, + "length": 2100, + "mean_action_latency_ms": 1.11133736780002, + "policy_error": null, + "return": 0.823999959975481, + "seed": 609, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 92.0 + }, + "invalid_action": false, + "lane": 3, + "length": 2100, + "mean_action_latency_ms": 1.1117433251014777, + "policy_error": null, + "return": 0.9279999546706676, + "seed": 610, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 4, + "length": 2100, + "mean_action_latency_ms": 1.1116926691361837, + "policy_error": null, + "return": 0.9199999384582043, + "seed": 611, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 0.9279999546706676, + "max_steps": 2100, + "mean_action_latency_ms": 1.1175926760382329, + "mean_return": 0.7327999543398619, + "min_return": 0.11999997682869434, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.7327999543398619 + } + } +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-05/metadata.json new file mode 100644 index 0000000..2ff2704 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-05/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "digest_compatibility_note": null, + "finished_at": 1784110074.558851, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 111.116423225, + "observed_raw_score": 0.8596399573981761, + "provider_reasoning_effort": "max", + "raw_score": 0.8596399573981761, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05", + "task": "simulation_heuristics_vizdoom_d1_v1", + "task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 666.0, + "input_tokens": 2.0, + "output_tokens": 666.0, + "total_tokens": 73959.0 + }, + "tool_calls": 58, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-05/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-05/score.json new file mode 100644 index 0000000..1de4be7 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/trial-05/score.json @@ -0,0 +1,211 @@ +{ + "anchor_calibration": { + "envpool_version": "1.1.1", + "mode": "numeric", + "platform": "linux-x86_64" + }, + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 111.116423225, + "reference_score": 0.8178199748694897, + "resource_cleanup": { + "removed_bytes": 0, + "removed_resources": 0, + "runs": [ + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + }, + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + } + ], + "terminated_processes": 0 + }, + "score": 0.8596399573981761, + "starter_score": 0.44161998964846133, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 0, + "length": 2100, + "mean_action_latency_ms": 0.24469348380253428, + "policy_error": null, + "return": 0.9519999492913485, + "seed": 101, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 84.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.2217625356501057, + "policy_error": null, + "return": 0.9039999675005674, + "seed": 102, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 28.0 + }, + "invalid_action": false, + "lane": 2, + "length": 2100, + "mean_action_latency_ms": 0.2203719646093391, + "policy_error": null, + "return": 0.607999961823225, + "seed": 103, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 3, + "length": 2100, + "mean_action_latency_ms": 0.2115432093186038, + "policy_error": null, + "return": 0.9199999645352364, + "seed": 104, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 4, + "length": 2100, + "mean_action_latency_ms": 0.1966855027491138, + "policy_error": null, + "return": 0.8559999465942383, + "seed": 105, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 0.9519999492913485, + "max_steps": 2100, + "mean_action_latency_ms": 0.21901133922593935, + "mean_return": 0.8479999579489231, + "min_return": 0.607999961823225, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.8479999579489231 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 0, + "length": 2100, + "mean_action_latency_ms": 0.25667997254502206, + "policy_error": null, + "return": 0.9679999556392431, + "seed": 607, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 78.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.23039182825457483, + "policy_error": null, + "return": 0.8899999614804983, + "seed": 608, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 2, + "length": 2100, + "mean_action_latency_ms": 0.21199343548644156, + "policy_error": null, + "return": 0.9199999570846558, + "seed": 609, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 76.0 + }, + "invalid_action": false, + "lane": 3, + "length": 2100, + "mean_action_latency_ms": 0.21185577004438355, + "policy_error": null, + "return": 0.8639999553561211, + "seed": 610, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 52.0 + }, + "invalid_action": false, + "lane": 4, + "length": 2100, + "mean_action_latency_ms": 0.21339360181064831, + "policy_error": null, + "return": 0.791999951004982, + "seed": 611, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 0.9679999556392431, + "max_steps": 2100, + "mean_action_latency_ms": 0.22486292162821409, + "mean_return": 0.8867999561131, + "min_return": 0.791999951004982, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.8867999561131 + } + } +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/metadata.json index 69b8b27..ed1021e 100644 --- a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/metadata.json +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/metadata.json @@ -1,28 +1,29 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:29434bdcdce19a6e0cb24d77c800d172d49a9529875ffd4201cf928e242f4e62", - "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", - "finished_at": 1783945917.227471, + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", "harness": "opencode", "model": { - "agent_idle_timeout_sec": 0, + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, "display_name": "Claude Opus 4.8", "id": "claude-opus-4.8", "model": "anthropic/claude-opus-4-8", "provider": "claude_oauth", "provider_reasoning_effort": "max" }, - "normalized_score": 122.259358289, + "normalized_score": 96.4371657756, + "normalized_score_stddev": 42.45851366513837, "provider_reasoning_effort": "max", - "source_run_id": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "raw_score": 320.44, + "raw_score_imputation_count": 0, + "raw_score_stddev": 127.03587288636228, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], "task": "simulation_heuristics_vizdoom_d3_v1", - "task_digest": "sha256:fcc6ed05673b7981aa17d60ca4e0d355fbcb57a6cbca293c0fa201ab97cdd081", - "token_usage": { - "final_input_tokens": 0.0, - "final_output_tokens": 0.0, - "input_tokens": 0.0, - "output_tokens": 0.0, - "total_tokens": 0.0 - }, - "tool_calls": 89 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/score.json index 19961f6..2906acb 100644 --- a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/score.json +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/score.json @@ -1,229 +1,59 @@ { - "isolation": { - "mode": "subprocess_per_suite", - "suite_process_count": 2, - "unique_working_directories": true - }, - "normalized_score": 122.259358289, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": 96.4371657756, + "normalized_score_stddev": 42.45851366513837, + "raw_score_imputation_count": 0, "reference_score": 331.1, - "score": 397.7, + "score": 320.44, + "score_stddev": 127.03587288636228, "starter_score": 31.9, - "suites": { - "hidden_nominal": { - "batch_seed": 101, - "envpool_version": "1.1.1", - "episodes": [ - { - "final_variables": { - "AMMO2": 0.0, - "DAMAGECOUNT": 255.0, - "HEALTH": 91.0, - "HITCOUNT": 26.0, - "KILLCOUNT": 23.0 - }, - "invalid_action": false, - "lane": 0, - "length": 1050, - "mean_action_latency_ms": 2.7141923617039407, - "policy_error": null, - "return": 485.0, - "seed": 101, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "AMMO2": 74.0, - "DAMAGECOUNT": 240.0, - "HEALTH": 91.0, - "HITCOUNT": 22.0, - "KILLCOUNT": 22.0 - }, - "invalid_action": false, - "lane": 1, - "length": 1050, - "mean_action_latency_ms": 2.699280280087675, - "policy_error": null, - "return": 460.0, - "seed": 102, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "AMMO2": 0.0, - "DAMAGECOUNT": 110.0, - "HEALTH": 9.0, - "HITCOUNT": 11.0, - "KILLCOUNT": 11.0 - }, - "invalid_action": false, - "lane": 2, - "length": 570, - "mean_action_latency_ms": 2.7295927664167, - "policy_error": null, - "return": 220.0, - "seed": 103, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 2.0, - "DAMAGECOUNT": 390.0, - "HEALTH": 100.0, - "HITCOUNT": 40.0, - "KILLCOUNT": 34.0 - }, - "invalid_action": false, - "lane": 3, - "length": 1050, - "mean_action_latency_ms": 2.7340504562570933, - "policy_error": null, - "return": 730.0, - "seed": 104, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "AMMO2": 0.0, - "DAMAGECOUNT": 165.0, - "HEALTH": 10.0, - "HITCOUNT": 19.0, - "KILLCOUNT": 17.0 - }, - "invalid_action": false, - "lane": 4, - "length": 780, - "mean_action_latency_ms": 2.7058095945857272, - "policy_error": null, - "return": 335.0, - "seed": 105, - "terminated": true, - "truncated": false - } - ], - "frame_skip": 2, - "invalid_episode_rate": 0.0, - "max_return": 730.0, - "max_steps": 1050, - "mean_action_latency_ms": 2.7165850918102272, - "mean_return": 446.0, - "min_return": 220.0, - "policy_path": "submitted_artifact", - "render_height": 480, - "render_width": 640, - "scenario": "d3", - "score": 446.0 + "task": "simulation_heuristics_vizdoom_d3_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 84.45855615, + "observed_raw_score": 284.6, + "raw_score": 284.6, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "trial": 1 }, - "hidden_secondary": { - "batch_seed": 607, - "envpool_version": "1.1.1", - "episodes": [ - { - "final_variables": { - "AMMO2": 0.0, - "DAMAGECOUNT": 130.0, - "HEALTH": 9.0, - "HITCOUNT": 10.0, - "KILLCOUNT": 10.0 - }, - "invalid_action": false, - "lane": 0, - "length": 650, - "mean_action_latency_ms": 2.729112282395363, - "policy_error": null, - "return": 230.0, - "seed": 607, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 0.0, - "DAMAGECOUNT": 25.0, - "HEALTH": 22.0, - "HITCOUNT": 2.0, - "KILLCOUNT": 2.0 - }, - "invalid_action": false, - "lane": 1, - "length": 432, - "mean_action_latency_ms": 2.695080444768623, - "policy_error": null, - "return": 45.0, - "seed": 608, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 15.0, - "DAMAGECOUNT": 145.0, - "HEALTH": 100.0, - "HITCOUNT": 14.0, - "KILLCOUNT": 14.0 - }, - "invalid_action": false, - "lane": 2, - "length": 1050, - "mean_action_latency_ms": 2.6943841115349816, - "policy_error": null, - "return": 285.0, - "seed": 609, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "AMMO2": 0.0, - "DAMAGECOUNT": 130.0, - "HEALTH": 24.0, - "HITCOUNT": 11.0, - "KILLCOUNT": 11.0 - }, - "invalid_action": false, - "lane": 3, - "length": 663, - "mean_action_latency_ms": 2.6851553528894128, - "policy_error": null, - "return": 240.0, - "seed": 610, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 71.0, - "DAMAGECOUNT": 335.0, - "HEALTH": 100.0, - "HITCOUNT": 35.0, - "KILLCOUNT": 29.0 - }, - "invalid_action": false, - "lane": 4, - "length": 1050, - "mean_action_latency_ms": 2.6883774224136556, - "policy_error": null, - "return": 625.0, - "seed": 611, - "terminated": false, - "truncated": true - } - ], - "frame_skip": 2, - "invalid_episode_rate": 0.0, - "max_return": 625.0, - "max_steps": 1050, - "mean_action_latency_ms": 2.6984219228004074, - "mean_return": 285.0, - "min_return": 45.0, - "policy_path": "submitted_artifact", - "render_height": 480, - "render_width": 640, - "scenario": "d3", - "score": 285.0 + { + "normalized_score": 114.104278075, + "observed_raw_score": 373.3, + "raw_score": 373.3, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "trial": 2 + }, + { + "normalized_score": 27.907754011, + "observed_raw_score": 115.4, + "raw_score": 115.4, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "trial": 3 + }, + { + "normalized_score": 134.35828877, + "observed_raw_score": 433.9, + "raw_score": 433.9, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "trial": 4 + }, + { + "normalized_score": 121.356951872, + "observed_raw_score": 394.99999999999994, + "raw_score": 394.99999999999994, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05", + "trial": 5 } - } + ] } diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-01/metadata.json new file mode 100644 index 0000000..2f90e0c --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-01/metadata.json @@ -0,0 +1,27 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "digest_compatibility_note": null, + "finished_at": 1784091892.627182, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 0, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 84.45855615, + "observed_raw_score": 284.6, + "provider_reasoning_effort": "max", + "raw_score": 284.6, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "task": "simulation_heuristics_vizdoom_d3_v1", + "task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "token_usage": null, + "tool_calls": 46, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-01/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-01/score.json new file mode 100644 index 0000000..cf67c5f --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-01/score.json @@ -0,0 +1,229 @@ +{ + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 84.45855615, + "reference_score": 331.1, + "score": 284.6, + "starter_score": 31.9, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 175.0, + "HEALTH": 51.0, + "HITCOUNT": 17.0, + "KILLCOUNT": 17.0 + }, + "invalid_action": false, + "lane": 0, + "length": 1050, + "mean_action_latency_ms": 15.284964506115232, + "policy_error": null, + "return": 345.0, + "seed": 101, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 38.0, + "DAMAGECOUNT": 200.0, + "HEALTH": 100.0, + "HITCOUNT": 19.0, + "KILLCOUNT": 19.0 + }, + "invalid_action": false, + "lane": 1, + "length": 1050, + "mean_action_latency_ms": 15.020975293148132, + "policy_error": null, + "return": 390.0, + "seed": 102, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 165.0, + "HEALTH": 66.0, + "HITCOUNT": 18.0, + "KILLCOUNT": 15.0 + }, + "invalid_action": false, + "lane": 2, + "length": 1050, + "mean_action_latency_ms": 15.18144062587193, + "policy_error": null, + "return": 315.0, + "seed": 103, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 17.0, + "DAMAGECOUNT": 95.0, + "HEALTH": 85.0, + "HITCOUNT": 9.0, + "KILLCOUNT": 9.0 + }, + "invalid_action": false, + "lane": 3, + "length": 1050, + "mean_action_latency_ms": 14.89276012139661, + "policy_error": null, + "return": 185.0, + "seed": 104, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 115.0, + "HEALTH": 1.0, + "HITCOUNT": 10.0, + "KILLCOUNT": 10.0 + }, + "invalid_action": false, + "lane": 4, + "length": 850, + "mean_action_latency_ms": 15.219280618955107, + "policy_error": null, + "return": 215.0, + "seed": 105, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 390.0, + "max_steps": 1050, + "mean_action_latency_ms": 15.119884233097403, + "mean_return": 290.0, + "min_return": 185.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 290.0 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 3.0, + "DAMAGECOUNT": 175.0, + "HEALTH": 100.0, + "HITCOUNT": 16.0, + "KILLCOUNT": 16.0 + }, + "invalid_action": false, + "lane": 0, + "length": 1050, + "mean_action_latency_ms": 15.051581827657563, + "policy_error": null, + "return": 335.0, + "seed": 607, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 9.0, + "DAMAGECOUNT": 135.0, + "HEALTH": 70.0, + "HITCOUNT": 12.0, + "KILLCOUNT": 11.0 + }, + "invalid_action": false, + "lane": 1, + "length": 1050, + "mean_action_latency_ms": 14.659090471409616, + "policy_error": null, + "return": 245.0, + "seed": 608, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 80.0, + "HEALTH": 98.0, + "HITCOUNT": 10.0, + "KILLCOUNT": 8.0 + }, + "invalid_action": false, + "lane": 2, + "length": 1050, + "mean_action_latency_ms": 14.936716308196386, + "policy_error": null, + "return": 160.0, + "seed": 609, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 70.0, + "HEALTH": 16.0, + "HITCOUNT": 6.0, + "KILLCOUNT": 6.0 + }, + "invalid_action": false, + "lane": 3, + "length": 841, + "mean_action_latency_ms": 15.088144242409152, + "policy_error": null, + "return": 130.0, + "seed": 610, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 8.0, + "DAMAGECOUNT": 260.0, + "HEALTH": 43.0, + "HITCOUNT": 26.0, + "KILLCOUNT": 23.0 + }, + "invalid_action": false, + "lane": 4, + "length": 760, + "mean_action_latency_ms": 14.820271451026201, + "policy_error": null, + "return": 490.0, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 490.0, + "max_steps": 1050, + "mean_action_latency_ms": 14.911160860139784, + "mean_return": 272.0, + "min_return": 130.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 272.0 + } + } +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-02/metadata.json new file mode 100644 index 0000000..33af5e0 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-02/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "digest_compatibility_note": null, + "finished_at": 1784103481.173677, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 114.104278075, + "observed_raw_score": 373.3, + "provider_reasoning_effort": "max", + "raw_score": 373.3, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "task": "simulation_heuristics_vizdoom_d3_v1", + "task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 781.0, + "input_tokens": 2.0, + "output_tokens": 781.0, + "total_tokens": 83729.0 + }, + "tool_calls": 81, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-02/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-02/score.json new file mode 100644 index 0000000..3a722fe --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-02/score.json @@ -0,0 +1,229 @@ +{ + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 114.104278075, + "reference_score": 331.1, + "score": 373.3, + "starter_score": 31.9, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 14.0, + "DAMAGECOUNT": 250.0, + "HEALTH": 64.0, + "HITCOUNT": 24.0, + "KILLCOUNT": 22.0 + }, + "invalid_action": false, + "lane": 0, + "length": 1050, + "mean_action_latency_ms": 2.6489505810397014, + "policy_error": null, + "return": 470.0, + "seed": 101, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 47.0, + "DAMAGECOUNT": 170.0, + "HEALTH": 100.0, + "HITCOUNT": 17.0, + "KILLCOUNT": 16.0 + }, + "invalid_action": false, + "lane": 1, + "length": 1050, + "mean_action_latency_ms": 2.6111182215667905, + "policy_error": null, + "return": 330.0, + "seed": 102, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 18.0, + "DAMAGECOUNT": 195.0, + "HEALTH": 100.0, + "HITCOUNT": 21.0, + "KILLCOUNT": 21.0 + }, + "invalid_action": false, + "lane": 2, + "length": 1050, + "mean_action_latency_ms": 2.607750023404757, + "policy_error": null, + "return": 405.0, + "seed": 103, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 52.0, + "DAMAGECOUNT": 155.0, + "HEALTH": 100.0, + "HITCOUNT": 17.0, + "KILLCOUNT": 16.0 + }, + "invalid_action": false, + "lane": 3, + "length": 1050, + "mean_action_latency_ms": 2.6270770494427, + "policy_error": null, + "return": 315.0, + "seed": 104, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 16.0, + "DAMAGECOUNT": 205.0, + "HEALTH": 64.0, + "HITCOUNT": 17.0, + "KILLCOUNT": 17.0 + }, + "invalid_action": false, + "lane": 4, + "length": 1050, + "mean_action_latency_ms": 2.5743907264300754, + "policy_error": null, + "return": 375.0, + "seed": 105, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 470.0, + "max_steps": 1050, + "mean_action_latency_ms": 2.6138573203768045, + "mean_return": 379.0, + "min_return": 315.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 379.0 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 25.0, + "DAMAGECOUNT": 275.0, + "HEALTH": 37.0, + "HITCOUNT": 25.0, + "KILLCOUNT": 23.0 + }, + "invalid_action": false, + "lane": 0, + "length": 1050, + "mean_action_latency_ms": 2.519550248980522, + "policy_error": null, + "return": 505.0, + "seed": 607, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 31.0, + "DAMAGECOUNT": 125.0, + "HEALTH": 22.0, + "HITCOUNT": 12.0, + "KILLCOUNT": 11.0 + }, + "invalid_action": false, + "lane": 1, + "length": 724, + "mean_action_latency_ms": 2.419648944137834, + "policy_error": null, + "return": 235.0, + "seed": 608, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 7.0, + "DAMAGECOUNT": 105.0, + "HEALTH": 76.0, + "HITCOUNT": 12.0, + "KILLCOUNT": 11.0 + }, + "invalid_action": false, + "lane": 2, + "length": 1050, + "mean_action_latency_ms": 2.4466749919312343, + "policy_error": null, + "return": 215.0, + "seed": 609, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 10.0, + "DAMAGECOUNT": 120.0, + "HEALTH": 100.0, + "HITCOUNT": 13.0, + "KILLCOUNT": 12.0 + }, + "invalid_action": false, + "lane": 3, + "length": 1050, + "mean_action_latency_ms": 2.473283779053461, + "policy_error": null, + "return": 240.0, + "seed": 610, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 14.0, + "DAMAGECOUNT": 315.0, + "HEALTH": 3.0, + "HITCOUNT": 33.0, + "KILLCOUNT": 29.0 + }, + "invalid_action": false, + "lane": 4, + "length": 1050, + "mean_action_latency_ms": 2.426553957519077, + "policy_error": null, + "return": 605.0, + "seed": 611, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 605.0, + "max_steps": 1050, + "mean_action_latency_ms": 2.4571423843244258, + "mean_return": 360.0, + "min_return": 215.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 360.0 + } + } +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-03/metadata.json new file mode 100644 index 0000000..3394278 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-03/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "digest_compatibility_note": null, + "finished_at": 1784113784.201615, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 27.907754011, + "observed_raw_score": 115.4, + "provider_reasoning_effort": "max", + "raw_score": 115.4, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "task": "simulation_heuristics_vizdoom_d3_v1", + "task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 766.0, + "input_tokens": 2.0, + "output_tokens": 766.0, + "total_tokens": 105494.0 + }, + "tool_calls": 111, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-03/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-03/score.json new file mode 100644 index 0000000..6b97193 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-03/score.json @@ -0,0 +1,229 @@ +{ + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 27.907754011, + "reference_score": 331.1, + "score": 115.4, + "starter_score": 31.9, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 115.0, + "HEALTH": 13.0, + "HITCOUNT": 12.0, + "KILLCOUNT": 13.0 + }, + "invalid_action": false, + "lane": 0, + "length": 871, + "mean_action_latency_ms": 5.191563379477141, + "policy_error": null, + "return": 245.0, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 5.0, + "DAMAGECOUNT": 60.0, + "HEALTH": 100.0, + "HITCOUNT": 6.0, + "KILLCOUNT": 6.0 + }, + "invalid_action": false, + "lane": 1, + "length": 1050, + "mean_action_latency_ms": 5.616477965599015, + "policy_error": null, + "return": 120.0, + "seed": 102, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 5.0, + "HEALTH": 13.0, + "HITCOUNT": 1.0, + "KILLCOUNT": 1.0 + }, + "invalid_action": false, + "lane": 2, + "length": 218, + "mean_action_latency_ms": 5.221411619984775, + "policy_error": null, + "return": 15.0, + "seed": 103, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 55.0, + "HEALTH": 6.0, + "HITCOUNT": 5.0, + "KILLCOUNT": 5.0 + }, + "invalid_action": false, + "lane": 3, + "length": 785, + "mean_action_latency_ms": 5.310116978777442, + "policy_error": null, + "return": 105.0, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 35.0, + "HEALTH": 24.0, + "HITCOUNT": 3.0, + "KILLCOUNT": 3.0 + }, + "invalid_action": false, + "lane": 4, + "length": 495, + "mean_action_latency_ms": 5.26673330201043, + "policy_error": null, + "return": 65.0, + "seed": 105, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 245.0, + "max_steps": 1050, + "mean_action_latency_ms": 5.32126064916976, + "mean_return": 110.0, + "min_return": 15.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 110.0 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 180.0, + "HEALTH": 85.0, + "HITCOUNT": 14.0, + "KILLCOUNT": 14.0 + }, + "invalid_action": false, + "lane": 0, + "length": 1050, + "mean_action_latency_ms": 4.973905859958558, + "policy_error": null, + "return": 320.0, + "seed": 607, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 155.0, + "HEALTH": 100.0, + "HITCOUNT": 15.0, + "KILLCOUNT": 14.0 + }, + "invalid_action": false, + "lane": 1, + "length": 1050, + "mean_action_latency_ms": 5.025606070246015, + "policy_error": null, + "return": 295.0, + "seed": 608, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 15.0, + "HEALTH": 13.0, + "HITCOUNT": 1.0, + "KILLCOUNT": 1.0 + }, + "invalid_action": false, + "lane": 2, + "length": 498, + "mean_action_latency_ms": 5.162659276979994, + "policy_error": null, + "return": 25.0, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 16.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 3, + "length": 420, + "mean_action_latency_ms": 5.050705515203022, + "policy_error": null, + "return": 0.0, + "seed": 610, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 6.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 13.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 4, + "length": 134, + "mean_action_latency_ms": 4.93826628176134, + "policy_error": null, + "return": 0.0, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 320.0, + "max_steps": 1050, + "mean_action_latency_ms": 5.030228600829786, + "mean_return": 128.0, + "min_return": 0.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 128.0 + } + } +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-04/metadata.json new file mode 100644 index 0000000..92e162e --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-04/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "digest_compatibility_note": null, + "finished_at": 1784120178.69071, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 134.35828877, + "observed_raw_score": 433.9, + "provider_reasoning_effort": "max", + "raw_score": 433.9, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "task": "simulation_heuristics_vizdoom_d3_v1", + "task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 693.0, + "input_tokens": 2.0, + "output_tokens": 693.0, + "total_tokens": 55411.0 + }, + "tool_calls": 53, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-04/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-04/score.json new file mode 100644 index 0000000..02aae4e --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-04/score.json @@ -0,0 +1,229 @@ +{ + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 134.35828877, + "reference_score": 331.1, + "score": 433.9, + "starter_score": 31.9, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 40.0, + "DAMAGECOUNT": 285.0, + "HEALTH": 19.0, + "HITCOUNT": 31.0, + "KILLCOUNT": 23.0 + }, + "invalid_action": false, + "lane": 0, + "length": 1038, + "mean_action_latency_ms": 2.639886598337822, + "policy_error": null, + "return": 515.0, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 78.0, + "DAMAGECOUNT": 220.0, + "HEALTH": 100.0, + "HITCOUNT": 21.0, + "KILLCOUNT": 20.0 + }, + "invalid_action": false, + "lane": 1, + "length": 1050, + "mean_action_latency_ms": 2.6140935044913065, + "policy_error": null, + "return": 420.0, + "seed": 102, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 37.0, + "DAMAGECOUNT": 225.0, + "HEALTH": 92.0, + "HITCOUNT": 21.0, + "KILLCOUNT": 22.0 + }, + "invalid_action": false, + "lane": 2, + "length": 1050, + "mean_action_latency_ms": 2.5884796253272464, + "policy_error": null, + "return": 445.0, + "seed": 103, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 37.0, + "DAMAGECOUNT": 225.0, + "HEALTH": 100.0, + "HITCOUNT": 21.0, + "KILLCOUNT": 21.0 + }, + "invalid_action": false, + "lane": 3, + "length": 1050, + "mean_action_latency_ms": 2.6081558281467077, + "policy_error": null, + "return": 435.0, + "seed": 104, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 50.0, + "DAMAGECOUNT": 240.0, + "HEALTH": 49.0, + "HITCOUNT": 21.0, + "KILLCOUNT": 20.0 + }, + "invalid_action": false, + "lane": 4, + "length": 1050, + "mean_action_latency_ms": 2.5989928948027745, + "policy_error": null, + "return": 440.0, + "seed": 105, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 515.0, + "max_steps": 1050, + "mean_action_latency_ms": 2.6099216902211717, + "mean_return": 451.0, + "min_return": 420.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 451.0 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 26.0, + "DAMAGECOUNT": 165.0, + "HEALTH": 100.0, + "HITCOUNT": 18.0, + "KILLCOUNT": 15.0 + }, + "invalid_action": false, + "lane": 0, + "length": 1050, + "mean_action_latency_ms": 2.8095236810899915, + "policy_error": null, + "return": 315.0, + "seed": 607, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 41.0, + "DAMAGECOUNT": 225.0, + "HEALTH": 91.0, + "HITCOUNT": 22.0, + "KILLCOUNT": 20.0 + }, + "invalid_action": false, + "lane": 1, + "length": 1050, + "mean_action_latency_ms": 2.7814781985112598, + "policy_error": null, + "return": 425.0, + "seed": 608, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 36.0, + "DAMAGECOUNT": 175.0, + "HEALTH": 97.0, + "HITCOUNT": 20.0, + "KILLCOUNT": 19.0 + }, + "invalid_action": false, + "lane": 2, + "length": 1050, + "mean_action_latency_ms": 2.7548863880691075, + "policy_error": null, + "return": 365.0, + "seed": 609, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 47.0, + "DAMAGECOUNT": 140.0, + "HEALTH": 96.0, + "HITCOUNT": 15.0, + "KILLCOUNT": 14.0 + }, + "invalid_action": false, + "lane": 3, + "length": 1050, + "mean_action_latency_ms": 2.8123922575087774, + "policy_error": null, + "return": 280.0, + "seed": 610, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 20.0, + "DAMAGECOUNT": 305.0, + "HEALTH": 73.0, + "HITCOUNT": 29.0, + "KILLCOUNT": 28.0 + }, + "invalid_action": false, + "lane": 4, + "length": 1050, + "mean_action_latency_ms": 2.7688996422858465, + "policy_error": null, + "return": 585.0, + "seed": 611, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 585.0, + "max_steps": 1050, + "mean_action_latency_ms": 2.7854360334929966, + "mean_return": 394.0, + "min_return": 280.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 394.0 + } + } +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-05/metadata.json new file mode 100644 index 0000000..af7fab6 --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-05/metadata.json @@ -0,0 +1,33 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "digest_compatibility_note": null, + "finished_at": 1784110074.558851, + "harness": "opencode", + "model": { + "agent_idle_timeout_sec": 3600, + "daytona_pty_readline_timeout_sec": 3600, + "display_name": "Claude Opus 4.8", + "id": "claude-opus-4.8", + "model": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_reasoning_effort": "max" + }, + "normalized_score": 121.356951872, + "observed_raw_score": 394.99999999999994, + "provider_reasoning_effort": "max", + "raw_score": 394.99999999999994, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05", + "task": "simulation_heuristics_vizdoom_d3_v1", + "task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "token_usage": { + "final_input_tokens": 2.0, + "final_output_tokens": 944.0, + "input_tokens": 2.0, + "output_tokens": 944.0, + "total_tokens": 63917.0 + }, + "tool_calls": 76, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-05/score.json b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-05/score.json new file mode 100644 index 0000000..1a894cd --- /dev/null +++ b/leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/trial-05/score.json @@ -0,0 +1,229 @@ +{ + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 121.356951872, + "reference_score": 331.1, + "score": 394.99999999999994, + "starter_score": 31.9, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 19.0, + "DAMAGECOUNT": 225.0, + "HEALTH": 72.0, + "HITCOUNT": 22.0, + "KILLCOUNT": 21.0 + }, + "invalid_action": false, + "lane": 0, + "length": 1050, + "mean_action_latency_ms": 2.2454084349530086, + "policy_error": null, + "return": 435.0, + "seed": 101, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 44.0, + "DAMAGECOUNT": 215.0, + "HEALTH": 100.0, + "HITCOUNT": 21.0, + "KILLCOUNT": 20.0 + }, + "invalid_action": false, + "lane": 1, + "length": 1050, + "mean_action_latency_ms": 2.194430966462408, + "policy_error": null, + "return": 415.0, + "seed": 102, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 145.0, + "HEALTH": 1.0, + "HITCOUNT": 17.0, + "KILLCOUNT": 16.0 + }, + "invalid_action": false, + "lane": 2, + "length": 886, + "mean_action_latency_ms": 2.1056848878785126, + "policy_error": null, + "return": 305.0, + "seed": 103, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 190.0, + "HEALTH": 4.0, + "HITCOUNT": 20.0, + "KILLCOUNT": 18.0 + }, + "invalid_action": false, + "lane": 3, + "length": 987, + "mean_action_latency_ms": 2.140365518335151, + "policy_error": null, + "return": 370.0, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 220.0, + "HEALTH": 18.0, + "HITCOUNT": 17.0, + "KILLCOUNT": 17.0 + }, + "invalid_action": false, + "lane": 4, + "length": 731, + "mean_action_latency_ms": 2.0704486505440585, + "policy_error": null, + "return": 390.0, + "seed": 105, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 435.0, + "max_steps": 1050, + "mean_action_latency_ms": 2.151267691634628, + "mean_return": 383.0, + "min_return": 305.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 383.0 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 150.0, + "HEALTH": 4.0, + "HITCOUNT": 12.0, + "KILLCOUNT": 12.0 + }, + "invalid_action": false, + "lane": 0, + "length": 953, + "mean_action_latency_ms": 2.245819913409064, + "policy_error": null, + "return": 270.0, + "seed": 607, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 43.0, + "DAMAGECOUNT": 205.0, + "HEALTH": 52.0, + "HITCOUNT": 18.0, + "KILLCOUNT": 16.0 + }, + "invalid_action": false, + "lane": 1, + "length": 1050, + "mean_action_latency_ms": 2.1853225962037133, + "policy_error": null, + "return": 365.0, + "seed": 608, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 15.0, + "DAMAGECOUNT": 225.0, + "HEALTH": 72.0, + "HITCOUNT": 23.0, + "KILLCOUNT": 21.0 + }, + "invalid_action": false, + "lane": 2, + "length": 1050, + "mean_action_latency_ms": 2.1648102076280686, + "policy_error": null, + "return": 435.0, + "seed": 609, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 54.0, + "DAMAGECOUNT": 205.0, + "HEALTH": 100.0, + "HITCOUNT": 21.0, + "KILLCOUNT": 19.0 + }, + "invalid_action": false, + "lane": 3, + "length": 1050, + "mean_action_latency_ms": 2.2007864784626734, + "policy_error": null, + "return": 395.0, + "seed": 610, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 22.0, + "DAMAGECOUNT": 350.0, + "HEALTH": 94.0, + "HITCOUNT": 34.0, + "KILLCOUNT": 30.0 + }, + "invalid_action": false, + "lane": 4, + "length": 1050, + "mean_action_latency_ms": 2.110321663674854, + "policy_error": null, + "return": 650.0, + "seed": 611, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 650.0, + "max_steps": 1050, + "mean_action_latency_ms": 2.181412171875675, + "mean_return": 423.0, + "min_return": 270.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 423.0 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/metadata.json index bdcc009..0969005 100644 --- a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:5cf3805d59b03cb9d0bebb6f154eab38beb655918b8321500d692f5010a00cb2", - "digest_compatibility_note": "Score-equivalent to the current task. Later changes add a fail-closed verifier timeout plus a CI-only smoke config and documentation; the publication scoring config is unchanged.", - "finished_at": 1783941566.485236, + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", "harness": "opencode", "model": { "display_name": "GPT-5.4 Mini", @@ -11,17 +9,19 @@ "provider": "azure", "provider_reasoning_effort": "xhigh" }, - "normalized_score": -31.35954984, + "normalized_score": -47.232499900200004, + "normalized_score_stddev": 10.40778807635678, "provider_reasoning_effort": "xhigh", - "source_run_id": "leaderboard/runs/article_suite/20260713T104315Z/gpt-5.4-mini", + "raw_score": 791.3786088082715, + "raw_score_imputation_count": 0, + "raw_score_stddev": 364.1182774109385, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "task": "simulation_heuristics_ant_v1", - "task_digest": "sha256:bbb533da0cb86459f4d49dee667e6c73ac54c0188bc40e54e911d50ef3c3bc38", - "token_usage": { - "final_input_tokens": 937.0, - "final_output_tokens": 225.0, - "input_tokens": 937.0, - "output_tokens": 225.0, - "total_tokens": 16010.0 - }, - "tool_calls": 16 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/score.json index d52943f..301faa2 100644 --- a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/score.json @@ -1,141 +1,59 @@ { - "hidden_nominal": { - "episodes": [ - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.0308447620482184, - "policy_error": null, - "return": 1280.8625113025073, - "seed": 101, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": 19.567406497399496, - "x_velocity": -0.9134368973606399 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.0313320025743451, - "policy_error": null, - "return": 1283.799292320564, - "seed": 211, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": 19.068321119541704, - "x_velocity": 0.46909212578562176 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.031062497466336936, - "policy_error": null, - "return": 1572.9322897606214, - "seed": 307, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": 33.97698637026477, - "x_velocity": 0.5331860974504821 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.031104908091947436, - "policy_error": null, - "return": 1434.9175295491777, - "seed": 401, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": 26.72069848487529, - "x_velocity": -0.37097842149982796 - } - ], - "fall_rate": 0.0, - "invalid_episode_rate": 0.0, - "max_return": 1572.9322897606214, - "max_steps": 1000, - "mean_action_latency_ms": 0.031086042545211967, - "mean_return": 1393.1279057332176, - "mean_x_position": 24.833353118020312, - "min_return": 1280.8625113025073, - "policy_path": "submitted_artifact", - "score": 1393.1279057332176 - }, - "hidden_nominal_weight": 0.7, - "hidden_robustness": { - "episodes": [ - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.030873946117935702, - "policy_error": null, - "return": 1236.6108861288565, - "seed": 503, - "terminated": false, - "truncated": true, - "variant": "light_low_friction", - "x_position": 17.199024467312558, - "x_velocity": -1.096540091826057 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.030729250283911824, - "policy_error": null, - "return": 1224.1827372398914, - "seed": 601, - "terminated": false, - "truncated": true, - "variant": "light_low_friction", - "x_position": 16.74834914717757, - "x_velocity": -0.43087566591353266 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.031124928907956928, - "policy_error": null, - "return": 1374.2109418292014, - "seed": 503, - "terminated": false, - "truncated": true, - "variant": "heavy_weak_actuators", - "x_position": 24.592110185851382, - "x_velocity": -0.4027636104083854 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.03148250802769326, - "policy_error": null, - "return": 1118.4220406880218, - "seed": 601, - "terminated": false, - "truncated": true, - "variant": "heavy_weak_actuators", - "x_position": 11.881812530236811, - "x_velocity": -0.21271365790770602 - } - ], - "fall_rate": 0.0, - "invalid_episode_rate": 0.0, - "max_return": 1374.2109418292014, - "max_steps": 1000, - "mean_action_latency_ms": 0.031052658334374428, - "mean_return": 1238.3566514714926, - "mean_x_position": 17.605324082644582, - "min_return": 1118.4220406880218, - "policy_path": "submitted_artifact", - "score": 1238.3566514714926 - }, - "hidden_robustness_weight": 0.3, - "normalized_score": -31.35954984, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": -47.232499900200004, + "normalized_score_stddev": 10.40778807635678, + "raw_score_imputation_count": 0, "reference_score": 5942.333244755838, - "score": 1346.6965294546999, + "score": 791.3786088082715, + "score_stddev": 364.1182774109385, "starter_score": 2443.815838992282, - "unique_policy_evaluations": 3 + "task": "simulation_heuristics_ant_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": -36.632193477, + "observed_raw_score": 1162.2321740972948, + "raw_score": 1162.2321740972948, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "trial": 1 + }, + { + "normalized_score": -50.421772091, + "observed_raw_score": 679.8013660996173, + "raw_score": 679.8013660996173, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "trial": 2 + }, + { + "normalized_score": -40.65567106, + "observed_raw_score": 1021.4701105134167, + "raw_score": 1021.4701105134167, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "trial": 3 + }, + { + "normalized_score": -63.436898931, + "observed_raw_score": 224.46488820508256, + "raw_score": 224.46488820508256, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "trial": 4 + }, + { + "normalized_score": -45.015963942, + "observed_raw_score": 868.924505125946, + "raw_score": 868.924505125946, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05", + "trial": 5 + } + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-01/metadata.json new file mode 100644 index 0000000..9195a3f --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "digest_compatibility_note": null, + "finished_at": 1784074742.653706, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -36.632193477, + "observed_raw_score": 1162.2321740972948, + "provider_reasoning_effort": "xhigh", + "raw_score": 1162.2321740972948, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "task": "simulation_heuristics_ant_v1", + "task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "token_usage": { + "final_input_tokens": 1068.0, + "final_output_tokens": 214.0, + "input_tokens": 1068.0, + "output_tokens": 214.0, + "total_tokens": 13570.0 + }, + "tool_calls": 13, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-01/score.json new file mode 100644 index 0000000..3ced24b --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-01/score.json @@ -0,0 +1,141 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.05079498700797558, + "policy_error": null, + "return": 1167.8677100600023, + "seed": 101, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 11.175843505943456, + "x_velocity": 0.20855238520613995 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.04772006720304489, + "policy_error": null, + "return": 1175.3293069577048, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 11.463909675999494, + "x_velocity": 0.4701366429275211 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.04787748120725155, + "policy_error": null, + "return": 1172.950251183951, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 11.40396057156987, + "x_velocity": -0.09404230641905542 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.04929140396416187, + "policy_error": null, + "return": 1174.8313998034919, + "seed": 401, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 11.511968058118002, + "x_velocity": 0.16373706834254875 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 1175.3293069577048, + "max_steps": 1000, + "mean_action_latency_ms": 0.04892098484560847, + "mean_return": 1172.7446670012876, + "mean_x_position": 11.388920452907705, + "min_return": 1167.8677100600023, + "policy_path": "submitted_artifact", + "score": 1172.7446670012876 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.04909919574856758, + "policy_error": null, + "return": 1173.6668963890054, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 11.404777607577957, + "x_velocity": 0.37076989967683716 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.0500356275588274, + "policy_error": null, + "return": 1169.2141465712723, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 11.28466429536818, + "x_velocity": 0.18657761748361423 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.0482600312680006, + "policy_error": null, + "return": 1104.822901410063, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 7.878434853357875, + "x_velocity": 0.3782598002854698 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.04925558716058731, + "policy_error": null, + "return": 1103.1081515815727, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 7.885126985240324, + "x_velocity": 0.43571343844256205 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 1173.6668963890054, + "max_steps": 1000, + "mean_action_latency_ms": 0.049162610433995724, + "mean_return": 1137.7030239879782, + "mean_x_position": 9.613250935386084, + "min_return": 1103.1081515815727, + "policy_path": "submitted_artifact", + "score": 1137.7030239879782 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": -36.632193477, + "reference_score": 5942.333244755838, + "score": 1162.2321740972948, + "starter_score": 2443.815838992282, + "unique_policy_evaluations": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-02/metadata.json new file mode 100644 index 0000000..cbaf302 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "digest_compatibility_note": null, + "finished_at": 1784074623.04269, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -50.421772091, + "observed_raw_score": 679.8013660996173, + "provider_reasoning_effort": "xhigh", + "raw_score": 679.8013660996173, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "task": "simulation_heuristics_ant_v1", + "task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "token_usage": { + "final_input_tokens": 1804.0, + "final_output_tokens": 227.0, + "input_tokens": 1804.0, + "output_tokens": 227.0, + "total_tokens": 14831.0 + }, + "tool_calls": 15, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-02/score.json new file mode 100644 index 0000000..b13caed --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-02/score.json @@ -0,0 +1,141 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.03470570407807827, + "policy_error": null, + "return": 661.1200549894642, + "seed": 101, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 3.4621582770031734, + "x_velocity": 0.2025644351883038 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.033866518177092075, + "policy_error": null, + "return": 664.0100186648003, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 3.559100932920981, + "x_velocity": -0.7384183769069175 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.034416694194078445, + "policy_error": null, + "return": 725.6258347266435, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 6.670450530919695, + "x_velocity": -0.9298584352459294 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.03547857282683253, + "policy_error": null, + "return": 696.0138896824699, + "seed": 401, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 5.226447659407584, + "x_velocity": -0.7061748742179041 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 725.6258347266435, + "max_steps": 1000, + "mean_action_latency_ms": 0.03461687231902033, + "mean_return": 686.6924495158444, + "mean_x_position": 4.729539350062858, + "min_return": 661.1200549894642, + "policy_path": "submitted_artifact", + "score": 686.6924495158444 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.036167888436466455, + "policy_error": null, + "return": 711.448754210964, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 5.932994803747869, + "x_velocity": -0.961583320736743 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.03447609907016158, + "policy_error": null, + "return": 661.9436724841174, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 3.617944828961479, + "x_velocity": -1.0404538157633425 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.03433279972523451, + "policy_error": null, + "return": 616.2518542379725, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 0.9972236887030465, + "x_velocity": 0.45277562220235046 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.03465622337535024, + "policy_error": null, + "return": 665.2444049139627, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 3.5082959468421335, + "x_velocity": -0.778853255686327 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 711.448754210964, + "max_steps": 1000, + "mean_action_latency_ms": 0.034908252651803195, + "mean_return": 663.7221714617541, + "mean_x_position": 3.514114817063632, + "min_return": 616.2518542379725, + "policy_path": "submitted_artifact", + "score": 663.7221714617541 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": -50.421772091, + "reference_score": 5942.333244755838, + "score": 679.8013660996173, + "starter_score": 2443.815838992282, + "unique_policy_evaluations": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-03/metadata.json new file mode 100644 index 0000000..b925bb9 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "digest_compatibility_note": null, + "finished_at": 1784076792.366179, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -40.65567106, + "observed_raw_score": 1021.4701105134167, + "provider_reasoning_effort": "xhigh", + "raw_score": 1021.4701105134167, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "task": "simulation_heuristics_ant_v1", + "task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "token_usage": { + "final_input_tokens": 955.0, + "final_output_tokens": 186.0, + "input_tokens": 955.0, + "output_tokens": 186.0, + "total_tokens": 12917.0 + }, + "tool_calls": 13, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-03/score.json new file mode 100644 index 0000000..cc24114 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-03/score.json @@ -0,0 +1,141 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.03117045760154724, + "policy_error": null, + "return": 1021.3786544166594, + "seed": 101, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 4.590898009343264, + "x_velocity": 0.49083065714123464 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.030989252030849457, + "policy_error": null, + "return": 956.765106287673, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 1.3265835335945975, + "x_velocity": 0.12928318629197744 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.03152957372367382, + "policy_error": null, + "return": 833.4434213187216, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": -3.8307353876758747, + "x_velocity": 0.701502144719317 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.03087330237030983, + "policy_error": null, + "return": 948.3381252553354, + "seed": 401, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 1.0706916465409477, + "x_velocity": 0.0961604485496137 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 1021.3786544166594, + "max_steps": 1000, + "mean_action_latency_ms": 0.031140646431595087, + "mean_return": 939.9813268195974, + "mean_x_position": 0.7893594504507335, + "min_return": 833.4434213187216, + "policy_path": "submitted_artifact", + "score": 939.9813268195974 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.03172572702169418, + "policy_error": null, + "return": 1074.2497463760844, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 6.828832722440472, + "x_velocity": -0.18292587321589338 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.03123953565955162, + "policy_error": null, + "return": 1403.417215834971, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 23.070547985686026, + "x_velocity": 1.4068501791643229 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.03332991525530815, + "policy_error": null, + "return": 1017.9484512177769, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 4.3610588251879046, + "x_velocity": -0.01681906340161632 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.03082619048655033, + "policy_error": null, + "return": 1350.8270097671482, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 20.67642938310105, + "x_velocity": 1.4784502581643721 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 1403.417215834971, + "max_steps": 1000, + "mean_action_latency_ms": 0.03178034210577607, + "mean_return": 1211.6106057989953, + "mean_x_position": 13.734217229103862, + "min_return": 1017.9484512177769, + "policy_path": "submitted_artifact", + "score": 1211.6106057989953 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": -40.65567106, + "reference_score": 5942.333244755838, + "score": 1021.4701105134167, + "starter_score": 2443.815838992282, + "unique_policy_evaluations": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-04/metadata.json new file mode 100644 index 0000000..65c1129 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-04/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "digest_compatibility_note": null, + "finished_at": 1784079105.716872, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -63.436898931, + "observed_raw_score": 224.46488820508256, + "provider_reasoning_effort": "xhigh", + "raw_score": 224.46488820508256, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "task": "simulation_heuristics_ant_v1", + "task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "token_usage": { + "final_input_tokens": 417.0, + "final_output_tokens": 195.0, + "input_tokens": 417.0, + "output_tokens": 195.0, + "total_tokens": 20580.0 + }, + "tool_calls": 14, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-04/score.json new file mode 100644 index 0000000..a19cb7b --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-04/score.json @@ -0,0 +1,141 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.2050457689911127, + "policy_error": null, + "return": 247.251059527061, + "seed": 101, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 1.1497861044999784, + "x_velocity": -0.19082433235750873 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.20786283863708377, + "policy_error": null, + "return": 233.2901425382695, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 0.9949116897613196, + "x_velocity": -0.09089488190535455 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.20552087109535933, + "policy_error": null, + "return": 270.25106097610524, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 2.309251849118103, + "x_velocity": 0.385347542769372 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.21530861128121614, + "policy_error": null, + "return": 74.01758729070941, + "seed": 401, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": -4.945776738490404, + "x_velocity": -0.4939369755770784 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 270.25106097610524, + "max_steps": 1000, + "mean_action_latency_ms": 0.208434522501193, + "mean_return": 206.2024625830363, + "mean_x_position": -0.12295677377775083, + "min_return": 74.01758729070941, + "policy_path": "submitted_artifact", + "score": 206.2024625830363 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.2277924157679081, + "policy_error": null, + "return": 291.0284780883622, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 3.4880726224594563, + "x_velocity": 0.13730382915408512 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.20450706966221333, + "policy_error": null, + "return": 253.69411522521386, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 2.0015998853567507, + "x_velocity": -0.1976269179872947 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.20643250178545713, + "policy_error": null, + "return": 260.4231180413944, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 1.3187373549474792, + "x_velocity": -0.5264940345873725 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.21114367432892323, + "policy_error": null, + "return": 263.1631472711247, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 1.9346372051645138, + "x_velocity": -0.1137730734536424 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 291.0284780883622, + "max_steps": 1000, + "mean_action_latency_ms": 0.21246891538612545, + "mean_return": 267.0772146565238, + "mean_x_position": 2.18576176698205, + "min_return": 253.69411522521386, + "policy_path": "submitted_artifact", + "score": 267.0772146565238 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": -63.436898931, + "reference_score": 5942.333244755838, + "score": 224.46488820508256, + "starter_score": 2443.815838992282, + "unique_policy_evaluations": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-05/metadata.json new file mode 100644 index 0000000..8e9bb83 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-05/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "digest_compatibility_note": null, + "finished_at": 1784081904.218542, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -45.015963942, + "observed_raw_score": 868.924505125946, + "provider_reasoning_effort": "xhigh", + "raw_score": 868.924505125946, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05", + "task": "simulation_heuristics_ant_v1", + "task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "token_usage": { + "final_input_tokens": 563.0, + "final_output_tokens": 155.0, + "input_tokens": 563.0, + "output_tokens": 155.0, + "total_tokens": 14030.0 + }, + "tool_calls": 17, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-05/score.json new file mode 100644 index 0000000..3fd5f38 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/trial-05/score.json @@ -0,0 +1,141 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.03174800565466285, + "policy_error": null, + "return": 863.2252173806411, + "seed": 101, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": -2.6752002309802005, + "x_velocity": -0.6151503958241022 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.0320410430431366, + "policy_error": null, + "return": 990.4861005760741, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 4.40708513246575, + "x_velocity": 1.4661311157660784 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.03198018344119191, + "policy_error": null, + "return": 884.118436362535, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": -1.1610970371657476, + "x_velocity": 0.21778487391873824 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.032590074464678764, + "policy_error": null, + "return": 667.3453967075939, + "seed": 401, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": -12.88451160345243, + "x_velocity": -0.8357494638282148 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 990.4861005760741, + "max_steps": 1000, + "mean_action_latency_ms": 0.03208982665091753, + "mean_return": 851.293787756711, + "mean_x_position": -3.0784309347831567, + "min_return": 667.3453967075939, + "policy_path": "submitted_artifact", + "score": 851.293787756711 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.031184049788862467, + "policy_error": null, + "return": 906.0229735415761, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": -0.4807497292716936, + "x_velocity": 0.9216958759163618 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.031641209963709116, + "policy_error": null, + "return": 893.4962563482845, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": -0.6920203503336937, + "x_velocity": -0.3571486396303847 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.03132007783278823, + "policy_error": null, + "return": 894.1716261252768, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": -0.3747660545904444, + "x_velocity": 0.32957554080385365 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.03161599161103368, + "policy_error": null, + "return": 946.5605266015077, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 2.9992113546628016, + "x_velocity": -0.4218426548825249 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 946.5605266015077, + "max_steps": 1000, + "mean_action_latency_ms": 0.03144033229909837, + "mean_return": 910.0628456541613, + "mean_x_position": 0.3629188051167425, + "min_return": 893.4962563482845, + "policy_path": "submitted_artifact", + "score": 910.0628456541613 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": -45.015963942, + "reference_score": 5942.333244755838, + "score": 868.924505125946, + "starter_score": 2443.815838992282, + "unique_policy_evaluations": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/metadata.json index 6faccf9..907c0cc 100644 --- a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:a3ec10ab61a65c03ca17e6460147a3178e222bd1c48964727ea9daae54717ebe", - "digest_compatibility_note": null, - "finished_at": 1783941641.828137, + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", "harness": "opencode", "model": { "display_name": "GPT-5.4 Mini", @@ -12,16 +10,18 @@ "provider_reasoning_effort": "xhigh" }, "normalized_score": 0.0, + "normalized_score_stddev": 0.0, "provider_reasoning_effort": "xhigh", - "source_run_id": "leaderboard/runs/article_suite/20260713T111944Z/gpt-5.4-mini", + "raw_score": 0.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.0, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "task": "simulation_heuristics_atari57_v1", - "task_digest": "sha256:a3ec10ab61a65c03ca17e6460147a3178e222bd1c48964727ea9daae54717ebe", - "token_usage": { - "final_input_tokens": 461.0, - "final_output_tokens": 240.0, - "input_tokens": 461.0, - "output_tokens": 240.0, - "total_tokens": 10941.0 - }, - "tool_calls": 16 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/score.json index 7788108..29c4ffc 100644 --- a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/score.json @@ -1,15 +1,59 @@ { - "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", - "evaluation": null, - "interaction_budget": { - "completed_trajectories": 0, - "counted_env_steps": 0, - "planned_trajectories": 342, - "target_env_steps": 6840000000 - }, + "benchmark": "learning_beyond_gradients_article_suite", "normalized_score": 0.0, - "protocol_complete": false, + "normalized_score_stddev": 0.0, + "raw_score_imputation_count": 0, "reference_score": 0.8283015254994576, "score": 0.0, - "starter_score": 0.0 + "score_stddev": 0.0, + "starter_score": 0.0, + "task": "simulation_heuristics_atari57_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "trial": 1 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "trial": 2 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "trial": 3 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "trial": 4 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05", + "trial": 5 + } + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-01/metadata.json new file mode 100644 index 0000000..70a7e86 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "digest_compatibility_note": null, + "finished_at": 1784074742.653706, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "task": "simulation_heuristics_atari57_v1", + "task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "token_usage": { + "final_input_tokens": 5807.0, + "final_output_tokens": 235.0, + "input_tokens": 5807.0, + "output_tokens": 235.0, + "total_tokens": 16282.0 + }, + "tool_calls": 17, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-01/score.json new file mode 100644 index 0000000..7788108 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-01/score.json @@ -0,0 +1,15 @@ +{ + "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", + "evaluation": null, + "interaction_budget": { + "completed_trajectories": 0, + "counted_env_steps": 0, + "planned_trajectories": 342, + "target_env_steps": 6840000000 + }, + "normalized_score": 0.0, + "protocol_complete": false, + "reference_score": 0.8283015254994576, + "score": 0.0, + "starter_score": 0.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-02/metadata.json new file mode 100644 index 0000000..16bad36 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "digest_compatibility_note": null, + "finished_at": 1784074623.04269, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "task": "simulation_heuristics_atari57_v1", + "task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "token_usage": { + "final_input_tokens": 651.0, + "final_output_tokens": 207.0, + "input_tokens": 651.0, + "output_tokens": 207.0, + "total_tokens": 15706.0 + }, + "tool_calls": 13, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-02/score.json new file mode 100644 index 0000000..7788108 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-02/score.json @@ -0,0 +1,15 @@ +{ + "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", + "evaluation": null, + "interaction_budget": { + "completed_trajectories": 0, + "counted_env_steps": 0, + "planned_trajectories": 342, + "target_env_steps": 6840000000 + }, + "normalized_score": 0.0, + "protocol_complete": false, + "reference_score": 0.8283015254994576, + "score": 0.0, + "starter_score": 0.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-03/metadata.json new file mode 100644 index 0000000..401447c --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "digest_compatibility_note": null, + "finished_at": 1784076792.366179, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "task": "simulation_heuristics_atari57_v1", + "task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "token_usage": { + "final_input_tokens": 5482.0, + "final_output_tokens": 145.0, + "input_tokens": 5482.0, + "output_tokens": 145.0, + "total_tokens": 17915.0 + }, + "tool_calls": 16, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-03/score.json new file mode 100644 index 0000000..7788108 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-03/score.json @@ -0,0 +1,15 @@ +{ + "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", + "evaluation": null, + "interaction_budget": { + "completed_trajectories": 0, + "counted_env_steps": 0, + "planned_trajectories": 342, + "target_env_steps": 6840000000 + }, + "normalized_score": 0.0, + "protocol_complete": false, + "reference_score": 0.8283015254994576, + "score": 0.0, + "starter_score": 0.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-04/metadata.json new file mode 100644 index 0000000..727efa5 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-04/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "digest_compatibility_note": null, + "finished_at": 1784079105.716872, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "task": "simulation_heuristics_atari57_v1", + "task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "token_usage": { + "final_input_tokens": 573.0, + "final_output_tokens": 197.0, + "input_tokens": 573.0, + "output_tokens": 197.0, + "total_tokens": 15618.0 + }, + "tool_calls": 11, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-04/score.json new file mode 100644 index 0000000..7788108 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-04/score.json @@ -0,0 +1,15 @@ +{ + "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", + "evaluation": null, + "interaction_budget": { + "completed_trajectories": 0, + "counted_env_steps": 0, + "planned_trajectories": 342, + "target_env_steps": 6840000000 + }, + "normalized_score": 0.0, + "protocol_complete": false, + "reference_score": 0.8283015254994576, + "score": 0.0, + "starter_score": 0.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-05/metadata.json new file mode 100644 index 0000000..895da78 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-05/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "digest_compatibility_note": null, + "finished_at": 1784081904.218542, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05", + "task": "simulation_heuristics_atari57_v1", + "task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "token_usage": { + "final_input_tokens": 5648.0, + "final_output_tokens": 237.0, + "input_tokens": 5648.0, + "output_tokens": 237.0, + "total_tokens": 23293.0 + }, + "tool_calls": 22, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-05/score.json new file mode 100644 index 0000000..7788108 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/trial-05/score.json @@ -0,0 +1,15 @@ +{ + "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", + "evaluation": null, + "interaction_budget": { + "completed_trajectories": 0, + "counted_env_steps": 0, + "planned_trajectories": 342, + "target_env_steps": 6840000000 + }, + "normalized_score": 0.0, + "protocol_complete": false, + "reference_score": 0.8283015254994576, + "score": 0.0, + "starter_score": 0.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/metadata.json index 9d0c949..7c41d08 100644 --- a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:77215337f89f950f83b8f7436f799959c2ddccd4102552c6302b13d30cc57199", - "digest_compatibility_note": null, - "finished_at": 1783935048.629668, + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", "harness": "opencode", "model": { "display_name": "GPT-5.4 Mini", @@ -11,17 +9,19 @@ "provider": "azure", "provider_reasoning_effort": "xhigh" }, - "normalized_score": 2.51572327, + "normalized_score": -1.3417190774, + "normalized_score_stddev": 9.228605826577622, "provider_reasoning_effort": "xhigh", - "source_run_id": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "raw_score": 380.6, + "raw_score_imputation_count": 0, + "raw_score_stddev": 44.0204497932495, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02/simulation_heuristics_breakout_ram_v1", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "task": "simulation_heuristics_breakout_ram_v1", - "task_digest": "sha256:77215337f89f950f83b8f7436f799959c2ddccd4102552c6302b13d30cc57199", - "token_usage": { - "final_input_tokens": 932.0, - "final_output_tokens": 181.0, - "input_tokens": 932.0, - "output_tokens": 181.0, - "total_tokens": 15449.0 - }, - "tool_calls": 17 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/score.json index c41aa06..277b98c 100644 --- a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/score.json @@ -1,80 +1,59 @@ { - "normalized_score": 2.51572327, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": -1.3417190774, + "normalized_score_stddev": 9.228605826577622, + "raw_score_imputation_count": 0, "reference_score": 864.0, - "score": 399.0, + "score": 380.6, + "score_stddev": 44.0204497932495, "starter_score": 387.0, - "suites": { - "hidden_nominal": { - "completion_rate": 0.0, - "episodes": [ - { - "final_lives": 3, - "invalid_action": false, - "length": 108000, - "mean_action_latency_ms": 0.0051444856271027, - "observation_mode": "ram", - "policy_error": null, - "return": 399.0, - "seed": 101, - "terminated": false, - "truncated": true, - "variant": "nominal", - "warmup_steps": 0 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 399.0, - "max_steps": 108000, - "mean_action_latency_ms": 0.0051444856271027, - "mean_return": 399.0, - "min_return": 399.0, - "observation_mode": "ram", - "policy_path": "submitted_artifact", - "score": 399.0 + "task": "simulation_heuristics_breakout_ram_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 0.209643606, + "observed_raw_score": 388.0, + "raw_score": 388.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "trial": 1 }, - "hidden_shifted_start": { - "completion_rate": 0.0, - "episodes": [ - { - "final_lives": 3, - "invalid_action": false, - "length": 108000, - "mean_action_latency_ms": 0.005121370694487942, - "observation_mode": "ram", - "policy_error": null, - "return": 399.0, - "seed": 211, - "terminated": false, - "truncated": true, - "variant": "left_6", - "warmup_steps": 6 - }, - { - "final_lives": 3, - "invalid_action": false, - "length": 108000, - "mean_action_latency_ms": 0.004679645820201754, - "observation_mode": "ram", - "policy_error": null, - "return": 399.0, - "seed": 211, - "terminated": false, - "truncated": true, - "variant": "right_6", - "warmup_steps": 6 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 399.0, - "max_steps": 108000, - "mean_action_latency_ms": 0.004900508257344848, - "mean_return": 399.0, - "min_return": 399.0, - "observation_mode": "ram", - "policy_path": "submitted_artifact", - "score": 399.0 + { + "normalized_score": -0.628930818, + "observed_raw_score": 384.0, + "raw_score": 384.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02/simulation_heuristics_breakout_ram_v1", + "trial": 2 + }, + { + "normalized_score": 3.983228512, + "observed_raw_score": 406.0, + "raw_score": 406.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "trial": 3 + }, + { + "normalized_score": 6.708595388, + "observed_raw_score": 419.0, + "raw_score": 419.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "trial": 4 + }, + { + "normalized_score": -16.981132075, + "observed_raw_score": 306.0, + "raw_score": 306.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05", + "trial": 5 } - }, - "target_gap": -465.0, - "target_score": 864.0 + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-01/metadata.json new file mode 100644 index 0000000..cb77ad5 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "digest_compatibility_note": null, + "finished_at": 1784074742.653706, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.209643606, + "observed_raw_score": 388.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 388.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "task": "simulation_heuristics_breakout_ram_v1", + "task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "token_usage": { + "final_input_tokens": 542.0, + "final_output_tokens": 214.0, + "input_tokens": 542.0, + "output_tokens": 214.0, + "total_tokens": 16628.0 + }, + "tool_calls": 18, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-01/score.json new file mode 100644 index 0000000..bd905f0 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-01/score.json @@ -0,0 +1,80 @@ +{ + "normalized_score": 0.209643606, + "reference_score": 864.0, + "score": 388.0, + "starter_score": 387.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 5, + "invalid_action": false, + "length": 108000, + "mean_action_latency_ms": 0.0046152717959894625, + "observation_mode": "ram", + "policy_error": null, + "return": 385.0, + "seed": 101, + "terminated": false, + "truncated": true, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 385.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.0046152717959894625, + "mean_return": 385.0, + "min_return": 385.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 385.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 5, + "invalid_action": false, + "length": 108000, + "mean_action_latency_ms": 0.004780716426601565, + "observation_mode": "ram", + "policy_error": null, + "return": 385.0, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "left_6", + "warmup_steps": 6 + }, + { + "final_lives": 4, + "invalid_action": false, + "length": 108000, + "mean_action_latency_ms": 0.004903455652917425, + "observation_mode": "ram", + "policy_error": null, + "return": 397.0, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 397.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.004842086039759495, + "mean_return": 391.0, + "min_return": 385.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 391.0 + } + }, + "target_gap": -476.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-02/metadata.json new file mode 100644 index 0000000..2dd3d22 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "digest_compatibility_note": null, + "finished_at": 1784075014.388146, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -0.628930818, + "observed_raw_score": 384.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 384.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02/simulation_heuristics_breakout_ram_v1", + "task": "simulation_heuristics_breakout_ram_v1", + "task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "token_usage": { + "final_input_tokens": 782.0, + "final_output_tokens": 192.0, + "input_tokens": 782.0, + "output_tokens": 192.0, + "total_tokens": 17358.0 + }, + "tool_calls": 20, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-02/score.json new file mode 100644 index 0000000..caaf46c --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-02/score.json @@ -0,0 +1,80 @@ +{ + "normalized_score": -0.628930818, + "reference_score": 864.0, + "score": 384.0, + "starter_score": 387.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 3, + "invalid_action": false, + "length": 108000, + "mean_action_latency_ms": 0.007308058147491129, + "observation_mode": "ram", + "policy_error": null, + "return": 384.0, + "seed": 101, + "terminated": false, + "truncated": true, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 384.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.007308058147491129, + "mean_return": 384.0, + "min_return": 384.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 384.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 3, + "invalid_action": false, + "length": 108000, + "mean_action_latency_ms": 0.0061370774034272746, + "observation_mode": "ram", + "policy_error": null, + "return": 384.0, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "left_6", + "warmup_steps": 6 + }, + { + "final_lives": 3, + "invalid_action": false, + "length": 108000, + "mean_action_latency_ms": 0.006589179272177043, + "observation_mode": "ram", + "policy_error": null, + "return": 384.0, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 384.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.006363128337802159, + "mean_return": 384.0, + "min_return": 384.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 384.0 + } + }, + "target_gap": -480.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-03/metadata.json new file mode 100644 index 0000000..72d5dbb --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "digest_compatibility_note": null, + "finished_at": 1784076792.366179, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 3.983228512, + "observed_raw_score": 406.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 406.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "task": "simulation_heuristics_breakout_ram_v1", + "task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "token_usage": { + "final_input_tokens": 2251.0, + "final_output_tokens": 194.0, + "input_tokens": 2251.0, + "output_tokens": 194.0, + "total_tokens": 16781.0 + }, + "tool_calls": 17, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-03/score.json new file mode 100644 index 0000000..b6899a0 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-03/score.json @@ -0,0 +1,80 @@ +{ + "normalized_score": 3.983228512, + "reference_score": 864.0, + "score": 406.0, + "starter_score": 387.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 1, + "invalid_action": false, + "length": 108000, + "mean_action_latency_ms": 0.005329795358530073, + "observation_mode": "ram", + "policy_error": null, + "return": 406.0, + "seed": 101, + "terminated": false, + "truncated": true, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 406.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.005329795358530073, + "mean_return": 406.0, + "min_return": 406.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 406.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 1, + "invalid_action": false, + "length": 108000, + "mean_action_latency_ms": 0.005526979553030321, + "observation_mode": "ram", + "policy_error": null, + "return": 406.0, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "left_6", + "warmup_steps": 6 + }, + { + "final_lives": 1, + "invalid_action": false, + "length": 108000, + "mean_action_latency_ms": 0.005741247135490455, + "observation_mode": "ram", + "policy_error": null, + "return": 406.0, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 406.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.005634113344260388, + "mean_return": 406.0, + "min_return": 406.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 406.0 + } + }, + "target_gap": -458.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-04/metadata.json new file mode 100644 index 0000000..43b215a --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-04/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "digest_compatibility_note": null, + "finished_at": 1784079105.716872, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 6.708595388, + "observed_raw_score": 419.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 419.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "task": "simulation_heuristics_breakout_ram_v1", + "task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "token_usage": { + "final_input_tokens": 661.0, + "final_output_tokens": 166.0, + "input_tokens": 661.0, + "output_tokens": 166.0, + "total_tokens": 16187.0 + }, + "tool_calls": 22, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-04/score.json new file mode 100644 index 0000000..62d0025 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-04/score.json @@ -0,0 +1,80 @@ +{ + "normalized_score": 6.708595388, + "reference_score": 864.0, + "score": 419.0, + "starter_score": 387.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 9298, + "mean_action_latency_ms": 0.006597048398063218, + "observation_mode": "ram", + "policy_error": null, + "return": 419.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 419.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.006597048398063218, + "mean_return": 419.0, + "min_return": 419.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 419.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 9292, + "mean_action_latency_ms": 0.00712134984806597, + "observation_mode": "ram", + "policy_error": null, + "return": 419.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "left_6", + "warmup_steps": 6 + }, + { + "final_lives": 0, + "invalid_action": false, + "length": 9292, + "mean_action_latency_ms": 0.007826795627523793, + "observation_mode": "ram", + "policy_error": null, + "return": 419.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 419.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.007474072737794881, + "mean_return": 419.0, + "min_return": 419.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 419.0 + } + }, + "target_gap": -445.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-05/metadata.json new file mode 100644 index 0000000..06ebc44 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-05/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "digest_compatibility_note": null, + "finished_at": 1784081904.218542, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -16.981132075, + "observed_raw_score": 306.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 306.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05", + "task": "simulation_heuristics_breakout_ram_v1", + "task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "token_usage": { + "final_input_tokens": 999.0, + "final_output_tokens": 197.0, + "input_tokens": 999.0, + "output_tokens": 197.0, + "total_tokens": 15020.0 + }, + "tool_calls": 20, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-05/score.json new file mode 100644 index 0000000..1935c20 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/trial-05/score.json @@ -0,0 +1,80 @@ +{ + "normalized_score": -16.981132075, + "reference_score": 864.0, + "score": 306.0, + "starter_score": 387.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 7818, + "mean_action_latency_ms": 0.006595597406823058, + "observation_mode": "ram", + "policy_error": null, + "return": 306.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 306.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.006595597406823058, + "mean_return": 306.0, + "min_return": 306.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 306.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 7812, + "mean_action_latency_ms": 0.006847919219569768, + "observation_mode": "ram", + "policy_error": null, + "return": 306.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "left_6", + "warmup_steps": 6 + }, + { + "final_lives": 0, + "invalid_action": false, + "length": 7812, + "mean_action_latency_ms": 0.00622298520085098, + "observation_mode": "ram", + "policy_error": null, + "return": 306.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 306.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.006535452210210374, + "mean_return": 306.0, + "min_return": 306.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 306.0 + } + }, + "target_gap": -558.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/metadata.json index dcb0bef..b445645 100644 --- a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:7e126380af5345a4bfa0c9d29e45e68b8f1aa20d5d655d12187744100b24441a", - "digest_compatibility_note": null, - "finished_at": 1783935048.629668, + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", "harness": "opencode", "model": { "display_name": "GPT-5.4 Mini", @@ -11,17 +9,19 @@ "provider": "azure", "provider_reasoning_effort": "xhigh" }, - "normalized_score": 15.523465704, + "normalized_score": 14.332129963999998, + "normalized_score_stddev": 51.9448986087478, "provider_reasoning_effort": "xhigh", - "source_run_id": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "raw_score": 389.4, + "raw_score_imputation_count": 0, + "raw_score_stddev": 287.77473829368694, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "task": "simulation_heuristics_breakout_rgb_v1", - "task_digest": "sha256:7e126380af5345a4bfa0c9d29e45e68b8f1aa20d5d655d12187744100b24441a", - "token_usage": { - "final_input_tokens": 859.0, - "final_output_tokens": 202.0, - "input_tokens": 859.0, - "output_tokens": 202.0, - "total_tokens": 23077.0 - }, - "tool_calls": 21 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/score.json index 5d3ff02..836363a 100644 --- a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/score.json @@ -1,66 +1,59 @@ { - "normalized_score": 15.523465704, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": 14.332129963999998, + "normalized_score_stddev": 51.9448986087478, + "raw_score_imputation_count": 0, "reference_score": 864.0, - "score": 396.0, + "score": 389.4, + "score_stddev": 287.77473829368694, "starter_score": 310.0, - "suites": { - "hidden_nominal": { - "completion_rate": 0.0, - "episodes": [ - { - "final_lives": 0, - "invalid_action": false, - "length": 11673, - "mean_action_latency_ms": 0.4428296772221238, - "observation_mode": "rgb", - "policy_error": null, - "return": 396.0, - "seed": 101, - "terminated": true, - "truncated": false, - "variant": "nominal", - "warmup_steps": 0 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 396.0, - "max_steps": 30000, - "mean_action_latency_ms": 0.4428296772221238, - "mean_return": 396.0, - "min_return": 396.0, - "observation_mode": "rgb", - "policy_path": "submitted_artifact", - "score": 396.0 + "task": "simulation_heuristics_breakout_rgb_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 8.664259928, + "observed_raw_score": 358.0, + "raw_score": 358.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "trial": 1 }, - "hidden_shifted_start": { - "completion_rate": 0.0, - "episodes": [ - { - "final_lives": 0, - "invalid_action": false, - "length": 11667, - "mean_action_latency_ms": 0.4506273232825088, - "observation_mode": "rgb", - "policy_error": null, - "return": 396.0, - "seed": 211, - "terminated": true, - "truncated": false, - "variant": "right_6", - "warmup_steps": 6 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 396.0, - "max_steps": 30000, - "mean_action_latency_ms": 0.4506273232825088, - "mean_return": 396.0, - "min_return": 396.0, - "observation_mode": "rgb", - "policy_path": "submitted_artifact", - "score": 396.0 + { + "normalized_score": 95.848375451, + "observed_raw_score": 841.0, + "raw_score": 841.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "trial": 2 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 310.0, + "raw_score": 310.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "trial": 3 + }, + { + "normalized_score": -48.194945848, + "observed_raw_score": 43.0, + "raw_score": 43.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "trial": 4 + }, + { + "normalized_score": 15.342960289, + "observed_raw_score": 395.0, + "raw_score": 395.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05", + "trial": 5 } - }, - "target_gap": -468.0, - "target_score": 864.0 + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-01/metadata.json new file mode 100644 index 0000000..510347e --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "digest_compatibility_note": null, + "finished_at": 1784074742.653706, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 8.664259928, + "observed_raw_score": 358.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 358.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "task": "simulation_heuristics_breakout_rgb_v1", + "task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "token_usage": { + "final_input_tokens": 505.0, + "final_output_tokens": 161.0, + "input_tokens": 505.0, + "output_tokens": 161.0, + "total_tokens": 15002.0 + }, + "tool_calls": 8, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-01/score.json new file mode 100644 index 0000000..18b7932 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-01/score.json @@ -0,0 +1,66 @@ +{ + "normalized_score": 8.664259928, + "reference_score": 864.0, + "score": 358.0, + "starter_score": 310.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 9466, + "mean_action_latency_ms": 0.45474840671661515, + "observation_mode": "rgb", + "policy_error": null, + "return": 358.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 358.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.45474840671661515, + "mean_return": 358.0, + "min_return": 358.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 358.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 9460, + "mean_action_latency_ms": 0.45666814748252926, + "observation_mode": "rgb", + "policy_error": null, + "return": 358.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 358.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.45666814748252926, + "mean_return": 358.0, + "min_return": 358.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 358.0 + } + }, + "target_gap": -506.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-02/metadata.json new file mode 100644 index 0000000..b62066c --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "digest_compatibility_note": null, + "finished_at": 1784074623.04269, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 95.848375451, + "observed_raw_score": 841.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 841.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "task": "simulation_heuristics_breakout_rgb_v1", + "task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "token_usage": { + "final_input_tokens": 3353.0, + "final_output_tokens": 179.0, + "input_tokens": 3353.0, + "output_tokens": 179.0, + "total_tokens": 17868.0 + }, + "tool_calls": 9, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-02/score.json new file mode 100644 index 0000000..0225f17 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-02/score.json @@ -0,0 +1,66 @@ +{ + "normalized_score": 95.848375451, + "reference_score": 864.0, + "score": 841.0, + "starter_score": 310.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 19937, + "mean_action_latency_ms": 0.46153924593151396, + "observation_mode": "rgb", + "policy_error": null, + "return": 841.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 841.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.46153924593151396, + "mean_return": 841.0, + "min_return": 841.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 841.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 19931, + "mean_action_latency_ms": 0.46838430003603465, + "observation_mode": "rgb", + "policy_error": null, + "return": 841.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 841.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.46838430003603465, + "mean_return": 841.0, + "min_return": 841.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 841.0 + } + }, + "target_gap": -23.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-03/metadata.json new file mode 100644 index 0000000..dd72a79 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "digest_compatibility_note": null, + "finished_at": 1784076792.366179, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 310.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 310.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "task": "simulation_heuristics_breakout_rgb_v1", + "task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "token_usage": { + "final_input_tokens": 356.0, + "final_output_tokens": 123.0, + "input_tokens": 356.0, + "output_tokens": 123.0, + "total_tokens": 19423.0 + }, + "tool_calls": 13, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-03/score.json new file mode 100644 index 0000000..39ddc82 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-03/score.json @@ -0,0 +1,66 @@ +{ + "normalized_score": 0.0, + "reference_score": 864.0, + "score": 310.0, + "starter_score": 310.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 11358, + "mean_action_latency_ms": 0.4277758664658336, + "observation_mode": "rgb", + "policy_error": null, + "return": 310.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 310.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.4277758664658336, + "mean_return": 310.0, + "min_return": 310.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 310.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 11352, + "mean_action_latency_ms": 0.42319861327032504, + "observation_mode": "rgb", + "policy_error": null, + "return": 310.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 310.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.42319861327032504, + "mean_return": 310.0, + "min_return": 310.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 310.0 + } + }, + "target_gap": -554.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-04/metadata.json new file mode 100644 index 0000000..e2d0019 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-04/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "digest_compatibility_note": null, + "finished_at": 1784079105.716872, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -48.194945848, + "observed_raw_score": 43.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 43.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "task": "simulation_heuristics_breakout_rgb_v1", + "task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "token_usage": { + "final_input_tokens": 416.0, + "final_output_tokens": 150.0, + "input_tokens": 416.0, + "output_tokens": 150.0, + "total_tokens": 18998.0 + }, + "tool_calls": 11, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-04/score.json new file mode 100644 index 0000000..cb26108 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-04/score.json @@ -0,0 +1,66 @@ +{ + "normalized_score": -48.194945848, + "reference_score": 864.0, + "score": 43.0, + "starter_score": 310.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 4641, + "mean_action_latency_ms": 0.4387987776386715, + "observation_mode": "rgb", + "policy_error": null, + "return": 43.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 43.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.4387987776386715, + "mean_return": 43.0, + "min_return": 43.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 43.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 4635, + "mean_action_latency_ms": 0.4768407749898225, + "observation_mode": "rgb", + "policy_error": null, + "return": 43.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 43.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.4768407749898225, + "mean_return": 43.0, + "min_return": 43.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 43.0 + } + }, + "target_gap": -821.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-05/metadata.json new file mode 100644 index 0000000..136892c --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-05/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "digest_compatibility_note": null, + "finished_at": 1784081904.218542, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 15.342960289, + "observed_raw_score": 395.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 395.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05", + "task": "simulation_heuristics_breakout_rgb_v1", + "task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "token_usage": { + "final_input_tokens": 3633.0, + "final_output_tokens": 160.0, + "input_tokens": 3633.0, + "output_tokens": 160.0, + "total_tokens": 18641.0 + }, + "tool_calls": 11, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-05/score.json new file mode 100644 index 0000000..a04328c --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/trial-05/score.json @@ -0,0 +1,66 @@ +{ + "normalized_score": 15.342960289, + "reference_score": 864.0, + "score": 395.0, + "starter_score": 310.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 10022, + "mean_action_latency_ms": 0.6255766486879524, + "observation_mode": "rgb", + "policy_error": null, + "return": 395.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 395.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.6255766486879524, + "mean_return": 395.0, + "min_return": 395.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 395.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 10016, + "mean_action_latency_ms": 0.4841433544182025, + "observation_mode": "rgb", + "policy_error": null, + "return": 395.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 395.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.4841433544182025, + "mean_return": 395.0, + "min_return": 395.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 395.0 + } + }, + "target_gap": -469.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/metadata.json index f000def..e02b822 100644 --- a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:c7e3b5e466431f4908c1e96cc7de80191f7a7c5ab5a63b3eea4d708f7f447fbf", - "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", - "finished_at": 1783937429.008938, + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", "harness": "opencode", "model": { "display_name": "GPT-5.4 Mini", @@ -11,17 +9,19 @@ "provider": "azure", "provider_reasoning_effort": "xhigh" }, - "normalized_score": -8.167582379, + "normalized_score": -39.8837988972, + "normalized_score_stddev": 19.209845675650264, "provider_reasoning_effort": "xhigh", - "source_run_id": "leaderboard/runs/article_suite/20260713T093314Z/gpt-5.4-mini", + "raw_score": 1015.6002588284422, + "raw_score_imputation_count": 0, + "raw_score_stddev": 1438.0428685581014, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "task": "simulation_heuristics_halfcheetah_v1", - "task_digest": "sha256:80c439f53e4ab964f9d7443cd7fb8f25cf6645a0bc288b0496b871c1800ebe78", - "token_usage": { - "final_input_tokens": 1370.0, - "final_output_tokens": 203.0, - "input_tokens": 1370.0, - "output_tokens": 203.0, - "total_tokens": 16933.0 - }, - "tool_calls": 12 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/score.json index 4b8932e..97f6fdc 100644 --- a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/score.json @@ -1,98 +1,59 @@ { - "hidden_nominal": { - "episodes": [ - { - "control_cost": 344.8222642093897, - "forward_reward": 3834.116914743516, - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.06746797123923898, - "policy_error": null, - "return": 3489.2946505341256, - "seed": 211, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": 191.6951879081835, - "x_velocity": 5.427317695066039 - }, - { - "control_cost": 346.87259566783905, - "forward_reward": 3854.082422006625, - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.06828817958012223, - "policy_error": null, - "return": 3507.209826338786, - "seed": 307, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": 192.73070516626822, - "x_velocity": 6.038759707036823 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 3507.209826338786, - "max_steps": 1000, - "mean_action_latency_ms": 0.0678780754096806, - "mean_control_cost": 345.84742993861437, - "mean_forward_reward": 3844.099668375071, - "mean_return": 3498.2522384364556, - "mean_x_position": 192.21294653722586, - "min_return": 3489.2946505341256, - "policy_path": "submitted_artifact", - "score": 3498.2522384364556 - }, - "hidden_nominal_weight": 0.7, - "hidden_robustness": { - "episodes": [ - { - "control_cost": 345.874118052423, - "forward_reward": 3808.7975607576186, - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.06775542348623276, - "policy_error": null, - "return": 3462.9234427051965, - "seed": 503, - "terminated": false, - "truncated": true, - "variant": "light_low_friction", - "x_position": 190.42219966048805, - "x_velocity": 1.4021610504933653 - }, - { - "control_cost": 367.54270024597645, - "forward_reward": 3178.5489849598293, - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.06758225383237004, - "policy_error": null, - "return": 2811.006284713854, - "seed": 503, - "terminated": false, - "truncated": true, - "variant": "heavy_weak_actuators", - "x_position": 158.90977087059895, - "x_velocity": 3.0848840894137 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 3462.9234427051965, - "max_steps": 1000, - "mean_action_latency_ms": 0.0676688386593014, - "mean_control_cost": 356.7084091491997, - "mean_forward_reward": 3493.673272858724, - "mean_return": 3136.964863709525, - "mean_x_position": 174.6659852655435, - "min_return": 2811.006284713854, - "policy_path": "submitted_artifact", - "score": 3136.964863709525 - }, - "hidden_robustness_weight": 0.3, - "normalized_score": -8.167582379, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": -39.8837988972, + "normalized_score_stddev": 19.209845675650264, + "raw_score_imputation_count": 0, "reference_score": 11487.256451471967, - "score": 3389.8660260183765, - "starter_score": 4001.2886158917113 + "score": 1015.6002588284422, + "score_stddev": 1438.0428685581014, + "starter_score": 4001.2886158917113, + "task": "simulation_heuristics_halfcheetah_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": -53.572769951, + "observed_raw_score": -9.15171128521311, + "raw_score": -9.15171128521311, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "trial": 1 + }, + { + "normalized_score": -12.137156309, + "observed_raw_score": 3092.704998440495, + "raw_score": 3092.704998440495, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "trial": 2 + }, + { + "normalized_score": -53.770506118, + "observed_raw_score": -23.954177164674096, + "raw_score": -23.954177164674096, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "trial": 3 + }, + { + "normalized_score": -27.203049655, + "observed_raw_score": 1964.877068442624, + "raw_score": 1964.877068442624, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "trial": 4 + }, + { + "normalized_score": -52.735512453, + "observed_raw_score": 53.52511570897907, + "raw_score": 53.52511570897907, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05", + "trial": 5 + } + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-01/metadata.json new file mode 100644 index 0000000..37fe11b --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "digest_compatibility_note": null, + "finished_at": 1784074742.653706, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -53.572769951, + "observed_raw_score": -9.15171128521311, + "provider_reasoning_effort": "xhigh", + "raw_score": -9.15171128521311, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "task": "simulation_heuristics_halfcheetah_v1", + "task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "token_usage": { + "final_input_tokens": 1058.0, + "final_output_tokens": 280.0, + "input_tokens": 1058.0, + "output_tokens": 280.0, + "total_tokens": 19770.0 + }, + "tool_calls": 18, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-01/score.json new file mode 100644 index 0000000..9ba5c34 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-01/score.json @@ -0,0 +1,98 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "control_cost": 19.53834575926885, + "forward_reward": 20.48329593233583, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.03969835233874619, + "policy_error": null, + "return": 0.9449501730669523, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 1.0135069676245076, + "x_velocity": 0.3814383608081456 + }, + { + "control_cost": 15.40235303575173, + "forward_reward": -2.8159707668228857, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.03457550541497767, + "policy_error": null, + "return": -18.218323802574634, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": -0.11421447240419799, + "x_velocity": -0.3419840797560057 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 0.9449501730669523, + "max_steps": 1000, + "mean_action_latency_ms": 0.03713692887686193, + "mean_control_cost": 17.47034939751029, + "mean_forward_reward": 8.833662582756473, + "mean_return": -8.63668681475384, + "mean_x_position": 0.4496462476101548, + "min_return": -18.218323802574634, + "policy_path": "submitted_artifact", + "score": -8.63668681475384 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "control_cost": 16.704875929281116, + "forward_reward": 12.449459638731577, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.03616986330598593, + "policy_error": null, + "return": -4.255416290549544, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 0.6047946045439015, + "x_velocity": 0.1057544067678566 + }, + { + "control_cost": 17.191544962581247, + "forward_reward": 0.7400911538946445, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.03489358793012798, + "policy_error": null, + "return": -16.45145380868661, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 0.019326180302054793, + "x_velocity": 0.3807459688298116 + } + ], + "invalid_episode_rate": 0.0, + "max_return": -4.255416290549544, + "max_steps": 1000, + "mean_action_latency_ms": 0.03553172561805695, + "mean_control_cost": 16.94821044593118, + "mean_forward_reward": 6.594775396313111, + "mean_return": -10.353435049618076, + "mean_x_position": 0.3120603924229782, + "min_return": -16.45145380868661, + "policy_path": "submitted_artifact", + "score": -10.353435049618076 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": -53.572769951, + "reference_score": 11487.256451471967, + "score": -9.15171128521311, + "starter_score": 4001.2886158917113 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-02/metadata.json new file mode 100644 index 0000000..7aeb83b --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "digest_compatibility_note": null, + "finished_at": 1784074623.04269, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -12.137156309, + "observed_raw_score": 3092.704998440495, + "provider_reasoning_effort": "xhigh", + "raw_score": 3092.704998440495, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "task": "simulation_heuristics_halfcheetah_v1", + "task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "token_usage": { + "final_input_tokens": 987.0, + "final_output_tokens": 225.0, + "input_tokens": 987.0, + "output_tokens": 225.0, + "total_tokens": 17084.0 + }, + "tool_calls": 16, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-02/score.json new file mode 100644 index 0000000..706d51c --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-02/score.json @@ -0,0 +1,98 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "control_cost": 305.24595556408167, + "forward_reward": 3678.4608247679944, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 1.4314969836268574, + "policy_error": null, + "return": 3373.2148692039127, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 183.9123834094072, + "x_velocity": 5.180105467356659 + }, + { + "control_cost": 341.0270830988884, + "forward_reward": 3378.504294999591, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 1.454640245065093, + "policy_error": null, + "return": 3037.4772119007034, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 168.95179881591653, + "x_velocity": 3.7480712162186745 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 3373.2148692039127, + "max_steps": 1000, + "mean_action_latency_ms": 1.4430686143459752, + "mean_control_cost": 323.13651933148503, + "mean_forward_reward": 3528.4825598837924, + "mean_return": 3205.3460405523083, + "mean_x_position": 176.43209111266185, + "min_return": 3037.4772119007034, + "policy_path": "submitted_artifact", + "score": 3205.3460405523083 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "control_cost": 316.4261013865471, + "forward_reward": 3542.825622244609, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 1.4034700258634984, + "policy_error": null, + "return": 3226.399520858062, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 177.12360273483776, + "x_velocity": 4.668825507724834 + }, + { + "control_cost": 320.11440293490887, + "forward_reward": 2753.466682436041, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 1.4270926392637193, + "policy_error": null, + "return": 2433.352279501132, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 137.65565574440942, + "x_velocity": 3.683240843554857 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 3226.399520858062, + "max_steps": 1000, + "mean_action_latency_ms": 1.4152813325636089, + "mean_control_cost": 318.270252160728, + "mean_forward_reward": 3148.146152340325, + "mean_return": 2829.875900179597, + "mean_x_position": 157.3896292396236, + "min_return": 2433.352279501132, + "policy_path": "submitted_artifact", + "score": 2829.875900179597 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": -12.137156309, + "reference_score": 11487.256451471967, + "score": 3092.704998440495, + "starter_score": 4001.2886158917113 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-03/metadata.json new file mode 100644 index 0000000..7d8ef70 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "digest_compatibility_note": null, + "finished_at": 1784076792.366179, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -53.770506118, + "observed_raw_score": -23.954177164674096, + "provider_reasoning_effort": "xhigh", + "raw_score": -23.954177164674096, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "task": "simulation_heuristics_halfcheetah_v1", + "task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "token_usage": { + "final_input_tokens": 1211.0, + "final_output_tokens": 235.0, + "input_tokens": 1211.0, + "output_tokens": 235.0, + "total_tokens": 19366.0 + }, + "tool_calls": 17, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-03/score.json new file mode 100644 index 0000000..b6aa425 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-03/score.json @@ -0,0 +1,98 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "control_cost": 300.1487913131714, + "forward_reward": 202.69393928681197, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.02377027226611972, + "policy_error": null, + "return": -97.45485202635925, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 10.124039135348323, + "x_velocity": -0.5472069099421868 + }, + { + "control_cost": 285.96953331679106, + "forward_reward": 254.28358693272796, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.023790671257302165, + "policy_error": null, + "return": -31.685946384063044, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 12.740763412573347, + "x_velocity": 0.3039916204056923 + } + ], + "invalid_episode_rate": 0.0, + "max_return": -31.685946384063044, + "max_steps": 1000, + "mean_action_latency_ms": 0.023780471761710942, + "mean_control_cost": 293.0591623149812, + "mean_forward_reward": 228.48876310976996, + "mean_return": -64.57039920521115, + "mean_x_position": 11.432401273960835, + "min_return": -97.45485202635925, + "policy_path": "submitted_artifact", + "score": -64.57039920521115 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "control_cost": 333.74386498332024, + "forward_reward": 365.5635468069191, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.024653877830132842, + "policy_error": null, + "return": 31.81968182359931, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 18.2604989629533, + "x_velocity": 0.6806847999852901 + }, + { + "control_cost": 326.35960227251053, + "forward_reward": 436.1739356420694, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.02376141818240285, + "policy_error": null, + "return": 109.8143333695587, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 21.791018404710783, + "x_velocity": -0.48323668331188246 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 109.8143333695587, + "max_steps": 1000, + "mean_action_latency_ms": 0.024207648006267846, + "mean_control_cost": 330.0517336279154, + "mean_forward_reward": 400.8687412244942, + "mean_return": 70.817007596579, + "mean_x_position": 20.02575868383204, + "min_return": 31.81968182359931, + "policy_path": "submitted_artifact", + "score": 70.817007596579 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": -53.770506118, + "reference_score": 11487.256451471967, + "score": -23.954177164674096, + "starter_score": 4001.2886158917113 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-04/metadata.json new file mode 100644 index 0000000..4af4919 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-04/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "digest_compatibility_note": null, + "finished_at": 1784079105.716872, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -27.203049655, + "observed_raw_score": 1964.877068442624, + "provider_reasoning_effort": "xhigh", + "raw_score": 1964.877068442624, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "task": "simulation_heuristics_halfcheetah_v1", + "task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "token_usage": { + "final_input_tokens": 1077.0, + "final_output_tokens": 226.0, + "input_tokens": 1077.0, + "output_tokens": 226.0, + "total_tokens": 18199.0 + }, + "tool_calls": 17, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-04/score.json new file mode 100644 index 0000000..a87db63 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-04/score.json @@ -0,0 +1,98 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "control_cost": 404.52249928563833, + "forward_reward": 2399.0693167475824, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.08948082011193037, + "policy_error": null, + "return": 1994.5468174619423, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 119.94280800838672, + "x_velocity": 1.8023743211231424 + }, + { + "control_cost": 404.0115480609238, + "forward_reward": 2399.8158037950166, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.095644909190014, + "policy_error": null, + "return": 1995.804255734091, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 120.01737425568766, + "x_velocity": 1.831299591341348 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 1995.804255734091, + "max_steps": 1000, + "mean_action_latency_ms": 0.09256286465097219, + "mean_control_cost": 404.2670236732811, + "mean_forward_reward": 2399.4425602712995, + "mean_return": 1995.1755365980166, + "mean_x_position": 119.98009113203719, + "min_return": 1994.5468174619423, + "policy_path": "submitted_artifact", + "score": 1995.1755365980166 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "control_cost": 403.4608011431992, + "forward_reward": 2646.1086004662097, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.08898329525254667, + "policy_error": null, + "return": 2242.6477993230096, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 132.28775164591758, + "x_velocity": 2.9110843054303674 + }, + { + "control_cost": 402.04308273643255, + "forward_reward": 1947.756568906841, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.09446040284819901, + "policy_error": null, + "return": 1545.713486170408, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 97.37015006794935, + "x_velocity": 1.8825078142764653 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 2242.6477993230096, + "max_steps": 1000, + "mean_action_latency_ms": 0.09172184905037284, + "mean_control_cost": 402.7519419398159, + "mean_forward_reward": 2296.9325846865254, + "mean_return": 1894.1806427467088, + "mean_x_position": 114.82895085693346, + "min_return": 1545.713486170408, + "policy_path": "submitted_artifact", + "score": 1894.1806427467088 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": -27.203049655, + "reference_score": 11487.256451471967, + "score": 1964.877068442624, + "starter_score": 4001.2886158917113 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-05/metadata.json new file mode 100644 index 0000000..2609b97 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-05/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "digest_compatibility_note": null, + "finished_at": 1784081904.218542, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -52.735512453, + "observed_raw_score": 53.52511570897907, + "provider_reasoning_effort": "xhigh", + "raw_score": 53.52511570897907, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05", + "task": "simulation_heuristics_halfcheetah_v1", + "task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "token_usage": { + "final_input_tokens": 1245.0, + "final_output_tokens": 213.0, + "input_tokens": 1245.0, + "output_tokens": 213.0, + "total_tokens": 21938.0 + }, + "tool_calls": 16, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-05/score.json new file mode 100644 index 0000000..a609919 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/trial-05/score.json @@ -0,0 +1,98 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "control_cost": 21.065339737804607, + "forward_reward": 79.29325042535999, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.24139313492923975, + "policy_error": null, + "return": 58.22791068755541, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 3.9540046922757175, + "x_velocity": 0.20219197257589955 + }, + { + "control_cost": 21.083589614136145, + "forward_reward": 77.07591265678883, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.23547821794636548, + "policy_error": null, + "return": 55.99232304265263, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 3.880379698776389, + "x_velocity": 0.2019426829676266 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 58.22791068755541, + "max_steps": 1000, + "mean_action_latency_ms": 0.2384356764378026, + "mean_control_cost": 21.074464675970376, + "mean_forward_reward": 78.18458154107441, + "mean_return": 57.11011686510402, + "mean_x_position": 3.917192195526053, + "min_return": 55.99232304265263, + "policy_path": "submitted_artifact", + "score": 57.11011686510402 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "control_cost": 19.409622317412868, + "forward_reward": 54.47101878826277, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.23511882545426488, + "policy_error": null, + "return": 35.061396470849886, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 2.705872562020459, + "x_velocity": 0.09250549261443558 + }, + { + "control_cost": 23.10210151481442, + "forward_reward": 78.36093106667289, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.2396400817669928, + "policy_error": null, + "return": 55.25882955185853, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 3.900368175940971, + "x_velocity": 0.1225159928213048 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 55.25882955185853, + "max_steps": 1000, + "mean_action_latency_ms": 0.23737945361062884, + "mean_control_cost": 21.255861916113645, + "mean_forward_reward": 66.41597492746783, + "mean_return": 45.16011301135421, + "mean_x_position": 3.303120368980715, + "min_return": 35.061396470849886, + "policy_path": "submitted_artifact", + "score": 45.16011301135421 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": -52.735512453, + "reference_score": 11487.256451471967, + "score": 53.52511570897907, + "starter_score": 4001.2886158917113 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/metadata.json index a046715..ee0cfce 100644 --- a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:33650c821735e31eea00f2ba46ae09b5d19134820c2ec005e8e96d92cb0b66d0", - "digest_compatibility_note": null, - "finished_at": 1783935048.629668, + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", "harness": "opencode", "model": { "display_name": "GPT-5.4 Mini", @@ -12,16 +10,18 @@ "provider_reasoning_effort": "xhigh" }, "normalized_score": 0.0, + "normalized_score_stddev": 0.0, "provider_reasoning_effort": "xhigh", - "source_run_id": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "raw_score": 0.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.0, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "task": "simulation_heuristics_montezuma_v1", - "task_digest": "sha256:33650c821735e31eea00f2ba46ae09b5d19134820c2ec005e8e96d92cb0b66d0", - "token_usage": { - "final_input_tokens": 542.0, - "final_output_tokens": 161.0, - "input_tokens": 542.0, - "output_tokens": 161.0, - "total_tokens": 12991.0 - }, - "tool_calls": 15 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/score.json index 3677853..136d942 100644 --- a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/score.json @@ -1,216 +1,59 @@ { + "benchmark": "learning_beyond_gradients_article_suite", "normalized_score": 0.0, + "normalized_score_stddev": 0.0, + "raw_score_imputation_count": 0, "reference_score": 400.0, "score": 0.0, + "score_stddev": 0.0, "starter_score": 0.0, - "suites": { - "hidden_seeds": { - "evaluation": { - "capped_mean_score": 0.0, - "episodes": [ - { - "bootstrap_steps": 0, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.15504447456735831, - "policy_error": null, - "policy_steps": 2600, - "pre_policy_noops": 0, - "return": 0.0, - "seed": 10007, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "canonical" - }, - { - "bootstrap_steps": 0, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.1614375047099132, - "policy_error": null, - "policy_steps": 2600, - "pre_policy_noops": 0, - "return": 0.0, - "seed": 10103, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "canonical" - } - ], - "invalid_episode_rate": 0.0, - "max_steps": 2600, - "mean_action_latency_ms": 0.15824098963863575, - "mean_return": 0.0, - "policy_path": "submitted_artifact", - "recovery_success_rate": 0.0, - "score": 0.0, - "target_score": 400.0, - "target_success_rate": 0.0 - }, - "score": 0.0, - "weight": 0.1 + "task": "simulation_heuristics_montezuma_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "trial": 1 }, - "noop_variants": { - "evaluation": { - "capped_mean_score": 0.0, - "episodes": [ - { - "bootstrap_steps": 0, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.1589538881674977, - "policy_error": null, - "policy_steps": 2597, - "pre_policy_noops": 3, - "return": 0.0, - "seed": 10001, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "startup_noop_3" - }, - { - "bootstrap_steps": 0, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.2664268070095333, - "policy_error": null, - "policy_steps": 2587, - "pre_policy_noops": 13, - "return": 0.0, - "seed": 10001, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "startup_noop_13" - } - ], - "invalid_episode_rate": 0.0, - "max_steps": 2600, - "mean_action_latency_ms": 0.21269034758851552, - "mean_return": 0.0, - "policy_path": "submitted_artifact", - "recovery_success_rate": 0.0, - "score": 0.0, - "target_score": 400.0, - "target_success_rate": 0.0 - }, - "score": 0.0, - "weight": 0.1 + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "trial": 2 }, - "recovery": { - "evaluation": { - "capped_mean_score": 0.0, - "episodes": [ - { - "bootstrap_steps": 512, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.1545967847008737, - "policy_error": null, - "policy_steps": 2084, - "pre_policy_noops": 4, - "return": 0.0, - "seed": 10001, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "recover_512_noop_4" - }, - { - "bootstrap_steps": 512, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.1483132426913541, - "policy_error": null, - "policy_steps": 2080, - "pre_policy_noops": 8, - "return": 0.0, - "seed": 10001, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "recover_512_noop_8" - }, - { - "bootstrap_steps": 768, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.34355820039582385, - "policy_error": null, - "policy_steps": 1830, - "pre_policy_noops": 2, - "return": 0.0, - "seed": 10001, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "recover_768_noop_2" - }, - { - "bootstrap_steps": 900, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.2770369280787075, - "policy_error": null, - "policy_steps": 1700, - "pre_policy_noops": 0, - "return": 0.0, - "seed": 10001, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "recover_900" - } - ], - "invalid_episode_rate": 0.0, - "max_steps": 2600, - "mean_action_latency_ms": 0.2308762889666898, - "mean_return": 0.0, - "policy_path": "submitted_artifact", - "recovery_success_rate": 0.0, - "score": 0.0, - "target_score": 400.0, - "target_success_rate": 0.0 - }, - "score": 0.0, - "weight": 0.55 + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "trial": 3 }, - "reproduction": { - "evaluation": { - "capped_mean_score": 0.0, - "episodes": [ - { - "bootstrap_steps": 0, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.15375497965858534, - "policy_error": null, - "policy_steps": 2600, - "pre_policy_noops": 0, - "return": 0.0, - "seed": 10001, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "canonical" - } - ], - "invalid_episode_rate": 0.0, - "max_steps": 2600, - "mean_action_latency_ms": 0.15375497965858534, - "mean_return": 0.0, - "policy_path": "submitted_artifact", - "recovery_success_rate": 0.0, - "score": 0.0, - "target_score": 400.0, - "target_success_rate": 0.0 - }, - "score": 0.0, - "weight": 0.25 + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "trial": 4 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05", + "trial": 5 } - }, - "target_score": 400.0 + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-01/metadata.json new file mode 100644 index 0000000..38320ff --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "digest_compatibility_note": null, + "finished_at": 1784074742.653706, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "task": "simulation_heuristics_montezuma_v1", + "task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "token_usage": { + "final_input_tokens": 674.0, + "final_output_tokens": 192.0, + "input_tokens": 674.0, + "output_tokens": 192.0, + "total_tokens": 14690.0 + }, + "tool_calls": 24, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-01/score.json new file mode 100644 index 0000000..2e7ef9f --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-01/score.json @@ -0,0 +1,216 @@ +{ + "normalized_score": 0.0, + "reference_score": 400.0, + "score": 0.0, + "starter_score": 0.0, + "suites": { + "hidden_seeds": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 793, + "mean_action_latency_ms": 0.13810291317905785, + "policy_error": null, + "policy_steps": 793, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10007, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "canonical" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 793, + "mean_action_latency_ms": 0.27633429719851565, + "policy_error": null, + "policy_steps": 793, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10103, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.20721860518878676, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "noop_variants": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 769, + "mean_action_latency_ms": 0.332880131450455, + "policy_error": null, + "policy_steps": 766, + "pre_policy_noops": 3, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "startup_noop_3" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 801, + "mean_action_latency_ms": 0.2730418201110463, + "policy_error": null, + "policy_steps": 788, + "pre_policy_noops": 13, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "startup_noop_13" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.30296097578075065, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "recovery": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.2462163969969123, + "policy_error": null, + "policy_steps": 2084, + "pre_policy_noops": 4, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_4" + }, + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.14897178838369796, + "policy_error": null, + "policy_steps": 2080, + "pre_policy_noops": 8, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_8" + }, + { + "bootstrap_steps": 768, + "invalid_action": false, + "length": 1333, + "mean_action_latency_ms": 0.272245857646262, + "policy_error": null, + "policy_steps": 563, + "pre_policy_noops": 2, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "recover_768_noop_2" + }, + { + "bootstrap_steps": 900, + "invalid_action": false, + "length": 1569, + "mean_action_latency_ms": 0.366762724199704, + "policy_error": null, + "policy_steps": 669, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "recover_900" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.2585491918066441, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.55 + }, + "reproduction": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 793, + "mean_action_latency_ms": 0.22760812019339727, + "policy_error": null, + "policy_steps": 793, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.22760812019339727, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.25 + } + }, + "target_score": 400.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-02/metadata.json new file mode 100644 index 0000000..867df8a --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "digest_compatibility_note": null, + "finished_at": 1784074623.04269, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "task": "simulation_heuristics_montezuma_v1", + "task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "token_usage": { + "final_input_tokens": 694.0, + "final_output_tokens": 320.0, + "input_tokens": 694.0, + "output_tokens": 320.0, + "total_tokens": 16886.0 + }, + "tool_calls": 18, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-02/score.json new file mode 100644 index 0000000..28d4c8f --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-02/score.json @@ -0,0 +1,216 @@ +{ + "normalized_score": 0.0, + "reference_score": 400.0, + "score": 0.0, + "starter_score": 0.0, + "suites": { + "hidden_seeds": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 2.2590444011327167, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10007, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 2.237814278073179, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10103, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 2.2484293396029478, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "noop_variants": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 2.261234643432668, + "policy_error": null, + "policy_steps": 2597, + "pre_policy_noops": 3, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_3" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 2.24331060765008, + "policy_error": null, + "policy_steps": 2587, + "pre_policy_noops": 13, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_13" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 2.252272625541374, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "recovery": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 2.2613830950903044, + "policy_error": null, + "policy_steps": 2084, + "pre_policy_noops": 4, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_4" + }, + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 2.2555219408017226, + "policy_error": null, + "policy_steps": 2080, + "pre_policy_noops": 8, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_8" + }, + { + "bootstrap_steps": 768, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 2.244413713415667, + "policy_error": null, + "policy_steps": 1830, + "pre_policy_noops": 2, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_768_noop_2" + }, + { + "bootstrap_steps": 900, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 2.2363532552331247, + "policy_error": null, + "policy_steps": 1700, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_900" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 2.2494180011352047, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.55 + }, + "reproduction": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 2.2355375501613777, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 2.2355375501613777, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.25 + } + }, + "target_score": 400.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-03/metadata.json new file mode 100644 index 0000000..f0873f9 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "digest_compatibility_note": null, + "finished_at": 1784076792.366179, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "task": "simulation_heuristics_montezuma_v1", + "task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "token_usage": { + "final_input_tokens": 951.0, + "final_output_tokens": 235.0, + "input_tokens": 951.0, + "output_tokens": 235.0, + "total_tokens": 17570.0 + }, + "tool_calls": 14, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-03/score.json new file mode 100644 index 0000000..c6f9efc --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-03/score.json @@ -0,0 +1,216 @@ +{ + "normalized_score": 0.0, + "reference_score": 400.0, + "score": 0.0, + "starter_score": 0.0, + "suites": { + "hidden_seeds": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 545, + "mean_action_latency_ms": 0.3433585999994923, + "policy_error": null, + "policy_steps": 545, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10007, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "canonical" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 545, + "mean_action_latency_ms": 0.3635533705114498, + "policy_error": null, + "policy_steps": 545, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10103, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.35345598525547106, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "noop_variants": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 545, + "mean_action_latency_ms": 0.3476791857910167, + "policy_error": null, + "policy_steps": 542, + "pre_policy_noops": 3, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "startup_noop_3" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 581, + "mean_action_latency_ms": 0.39047193442943545, + "policy_error": null, + "policy_steps": 568, + "pre_policy_noops": 13, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "startup_noop_13" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.3690755601102261, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "recovery": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.3457488382642735, + "policy_error": null, + "policy_steps": 2084, + "pre_policy_noops": 4, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_4" + }, + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.3476481597037771, + "policy_error": null, + "policy_steps": 2080, + "pre_policy_noops": 8, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_8" + }, + { + "bootstrap_steps": 768, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.3545664379352836, + "policy_error": null, + "policy_steps": 1830, + "pre_policy_noops": 2, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_768_noop_2" + }, + { + "bootstrap_steps": 900, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.3445286229800652, + "policy_error": null, + "policy_steps": 1700, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_900" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.34812301472084983, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.55 + }, + "reproduction": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 545, + "mean_action_latency_ms": 0.3702154062654174, + "policy_error": null, + "policy_steps": 545, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.3702154062654174, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.25 + } + }, + "target_score": 400.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-04/metadata.json new file mode 100644 index 0000000..7c0ffd4 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-04/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "digest_compatibility_note": null, + "finished_at": 1784079105.716872, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "task": "simulation_heuristics_montezuma_v1", + "task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "token_usage": { + "final_input_tokens": 317.0, + "final_output_tokens": 173.0, + "input_tokens": 317.0, + "output_tokens": 173.0, + "total_tokens": 14826.0 + }, + "tool_calls": 18, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-04/score.json new file mode 100644 index 0000000..6a8f6ff --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-04/score.json @@ -0,0 +1,216 @@ +{ + "normalized_score": 0.0, + "reference_score": 400.0, + "score": 0.0, + "starter_score": 0.0, + "suites": { + "hidden_seeds": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 525, + "mean_action_latency_ms": 0.35121889208399115, + "policy_error": null, + "policy_steps": 525, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10007, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "canonical" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 525, + "mean_action_latency_ms": 0.3174069279893523, + "policy_error": null, + "policy_steps": 525, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10103, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.33431291003667174, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "noop_variants": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.3104727892947452, + "policy_error": null, + "policy_steps": 2597, + "pre_policy_noops": 3, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_3" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.3048902937107776, + "policy_error": null, + "policy_steps": 2587, + "pre_policy_noops": 13, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_13" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.3076815415027614, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "recovery": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.31454834786302727, + "policy_error": null, + "policy_steps": 2084, + "pre_policy_noops": 4, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_4" + }, + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.301926501113205, + "policy_error": null, + "policy_steps": 2080, + "pre_policy_noops": 8, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_8" + }, + { + "bootstrap_steps": 768, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.3054561355543104, + "policy_error": null, + "policy_steps": 1830, + "pre_policy_noops": 2, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_768_noop_2" + }, + { + "bootstrap_steps": 900, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.3009652088889304, + "policy_error": null, + "policy_steps": 1700, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_900" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.30572404835486827, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.55 + }, + "reproduction": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 525, + "mean_action_latency_ms": 0.31950572195152444, + "policy_error": null, + "policy_steps": 525, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.31950572195152444, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.25 + } + }, + "target_score": 400.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-05/metadata.json new file mode 100644 index 0000000..4252f9d --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-05/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "digest_compatibility_note": null, + "finished_at": 1784081904.218542, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05", + "task": "simulation_heuristics_montezuma_v1", + "task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "token_usage": { + "final_input_tokens": 514.0, + "final_output_tokens": 258.0, + "input_tokens": 514.0, + "output_tokens": 258.0, + "total_tokens": 12548.0 + }, + "tool_calls": 16, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-05/score.json new file mode 100644 index 0000000..0a1e64b --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/trial-05/score.json @@ -0,0 +1,216 @@ +{ + "normalized_score": 0.0, + "reference_score": 400.0, + "score": 0.0, + "starter_score": 0.0, + "suites": { + "hidden_seeds": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.2897980182359998, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10007, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.26959146256558597, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10103, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.2796947404007929, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "noop_variants": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.22053315331474446, + "policy_error": null, + "policy_steps": 2597, + "pre_policy_noops": 3, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_3" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.29671448237897347, + "policy_error": null, + "policy_steps": 2587, + "pre_policy_noops": 13, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_13" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.25862381784685895, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "recovery": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.25202852757397376, + "policy_error": null, + "policy_steps": 2084, + "pre_policy_noops": 4, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_4" + }, + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.23053955206146035, + "policy_error": null, + "policy_steps": 2080, + "pre_policy_noops": 8, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_8" + }, + { + "bootstrap_steps": 768, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.4031258870598153, + "policy_error": null, + "policy_steps": 1830, + "pre_policy_noops": 2, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_768_noop_2" + }, + { + "bootstrap_steps": 900, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.2294785449938739, + "policy_error": null, + "policy_steps": 1700, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_900" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.27879312792228084, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.55 + }, + "reproduction": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.257155328290537, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.257155328290537, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.25 + } + }, + "target_score": 400.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/metadata.json index d7ba401..767dd8b 100644 --- a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:f0b6f90f1d8072e75f0e10f90f4b51e51e31b3590bb2c2736778ba155b23389f", - "digest_compatibility_note": null, - "finished_at": 1783935048.629668, + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", "harness": "opencode", "model": { "display_name": "GPT-5.4 Mini", @@ -11,17 +9,19 @@ "provider": "azure", "provider_reasoning_effort": "xhigh" }, - "normalized_score": -75.0, + "normalized_score": -40.8333333332, + "normalized_score_stddev": 67.46912874308083, "provider_reasoning_effort": "xhigh", - "source_run_id": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "raw_score": -12.8, + "raw_score_imputation_count": 0, + "raw_score_stddev": 16.192590898309017, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "task": "simulation_heuristics_pong_ram_v1", - "task_digest": "sha256:f0b6f90f1d8072e75f0e10f90f4b51e51e31b3590bb2c2736778ba155b23389f", - "token_usage": { - "final_input_tokens": 568.0, - "final_output_tokens": 241.0, - "input_tokens": 568.0, - "output_tokens": 241.0, - "total_tokens": 14121.0 - }, - "tool_calls": 17 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/score.json index bd2984d..6e60517 100644 --- a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/score.json @@ -1,133 +1,59 @@ { - "normalized_score": -75.0, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": -40.8333333332, + "normalized_score_stddev": 67.46912874308083, + "raw_score_imputation_count": 0, "reference_score": 21.0, - "score": -21.0, + "score": -12.8, + "score_stddev": 16.192590898309017, "starter_score": -2.9999999999999996, - "suite_weights": { - "hidden_nominal": 0.7, - "hidden_reset_robustness": 0.3 - }, - "suites": { - "hidden_nominal": { - "episodes": [ - { - "invalid_action": false, - "length": 3055, - "mean_action_latency_ms": 0.009747649163446724, - "points_against": 21, - "points_for": 0, - "policy_error": null, - "score": -21.0, - "seed": 101, - "target_reached": false, - "terminated": true, - "truncated": false, - "variant": "article_nominal", - "won": false - }, - { - "invalid_action": false, - "length": 3055, - "mean_action_latency_ms": 0.009773229286928224, - "points_against": 21, - "points_for": 0, - "policy_error": null, - "score": -21.0, - "seed": 211, - "target_reached": false, - "terminated": true, - "truncated": false, - "variant": "article_nominal", - "won": false - }, - { - "invalid_action": false, - "length": 3055, - "mean_action_latency_ms": 0.010469134549462463, - "points_against": 21, - "points_for": 0, - "policy_error": null, - "score": -21.0, - "seed": 307, - "target_reached": false, - "terminated": true, - "truncated": false, - "variant": "article_nominal", - "won": false - } - ], - "invalid_episode_rate": 0.0, - "max_score": -21.0, - "max_steps": 27000, - "mean_action_latency_ms": 0.009996670999945804, - "mean_score": -21.0, - "min_score": -21.0, - "policy_path": "submitted_artifact", - "score": -21.0, - "target_score": 21.0, - "target_score_rate": 0.0, - "win_rate": 0.0 + "task": "simulation_heuristics_pong_ram_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": -75.0, + "observed_raw_score": -21.0, + "raw_score": -21.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "trial": 1 }, - "hidden_reset_robustness": { - "episodes": [ - { - "invalid_action": false, - "length": 3055, - "mean_action_latency_ms": 0.010082433564182975, - "points_against": 21, - "points_for": 0, - "policy_error": null, - "score": -21.0, - "seed": 503, - "target_reached": false, - "terminated": true, - "truncated": false, - "variant": "randomized_reset", - "won": false - }, - { - "invalid_action": false, - "length": 3051, - "mean_action_latency_ms": 0.012134256769949708, - "points_against": 21, - "points_for": 0, - "policy_error": null, - "score": -21.0, - "seed": 601, - "target_reached": false, - "terminated": true, - "truncated": false, - "variant": "randomized_reset", - "won": false - }, - { - "invalid_action": false, - "length": 3052, - "mean_action_latency_ms": 0.011448162823895951, - "points_against": 21, - "points_for": 0, - "policy_error": null, - "score": -21.0, - "seed": 701, - "target_reached": false, - "terminated": true, - "truncated": false, - "variant": "randomized_reset", - "won": false - } - ], - "invalid_episode_rate": 0.0, - "max_score": -21.0, - "max_steps": 27000, - "mean_action_latency_ms": 0.011221617719342878, - "mean_score": -21.0, - "min_score": -21.0, - "policy_path": "submitted_artifact", - "score": -21.0, - "target_score": 21.0, - "target_score_rate": 0.0, - "win_rate": 0.0 + { + "normalized_score": -75.0, + "observed_raw_score": -21.0, + "raw_score": -21.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "trial": 2 + }, + { + "normalized_score": -58.333333333, + "observed_raw_score": -17.0, + "raw_score": -17.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "trial": 3 + }, + { + "normalized_score": -75.0, + "observed_raw_score": -21.0, + "raw_score": -21.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "trial": 4 + }, + { + "normalized_score": 79.166666667, + "observed_raw_score": 16.0, + "raw_score": 16.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05", + "trial": 5 } - }, - "target_score": 21.0 + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-01/metadata.json new file mode 100644 index 0000000..0117bd0 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "digest_compatibility_note": null, + "finished_at": 1784074742.653706, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -75.0, + "observed_raw_score": -21.0, + "provider_reasoning_effort": "xhigh", + "raw_score": -21.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "task": "simulation_heuristics_pong_ram_v1", + "task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "token_usage": { + "final_input_tokens": 497.0, + "final_output_tokens": 207.0, + "input_tokens": 497.0, + "output_tokens": 207.0, + "total_tokens": 17600.0 + }, + "tool_calls": 20, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-01/score.json new file mode 100644 index 0000000..c339c14 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-01/score.json @@ -0,0 +1,133 @@ +{ + "normalized_score": -75.0, + "reference_score": 21.0, + "score": -21.0, + "starter_score": -2.9999999999999996, + "suite_weights": { + "hidden_nominal": 0.7, + "hidden_reset_robustness": 0.3 + }, + "suites": { + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 3055, + "mean_action_latency_ms": 0.006232374087097992, + "points_against": 21, + "points_for": 0, + "policy_error": null, + "score": -21.0, + "seed": 101, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": false + }, + { + "invalid_action": false, + "length": 3055, + "mean_action_latency_ms": 0.005362869382099707, + "points_against": 21, + "points_for": 0, + "policy_error": null, + "score": -21.0, + "seed": 211, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": false + }, + { + "invalid_action": false, + "length": 3055, + "mean_action_latency_ms": 0.0055636384077796955, + "points_against": 21, + "points_for": 0, + "policy_error": null, + "score": -21.0, + "seed": 307, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": false + } + ], + "invalid_episode_rate": 0.0, + "max_score": -21.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.005719627292325798, + "mean_score": -21.0, + "min_score": -21.0, + "policy_path": "submitted_artifact", + "score": -21.0, + "target_score": 21.0, + "target_score_rate": 0.0, + "win_rate": 0.0 + }, + "hidden_reset_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 3055, + "mean_action_latency_ms": 0.00551823692765154, + "points_against": 21, + "points_for": 0, + "policy_error": null, + "score": -21.0, + "seed": 503, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": false + }, + { + "invalid_action": false, + "length": 3051, + "mean_action_latency_ms": 0.006575906381831526, + "points_against": 21, + "points_for": 0, + "policy_error": null, + "score": -21.0, + "seed": 601, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": false + }, + { + "invalid_action": false, + "length": 3052, + "mean_action_latency_ms": 0.005233440493864555, + "points_against": 21, + "points_for": 0, + "policy_error": null, + "score": -21.0, + "seed": 701, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": false + } + ], + "invalid_episode_rate": 0.0, + "max_score": -21.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.00577586126778254, + "mean_score": -21.0, + "min_score": -21.0, + "policy_path": "submitted_artifact", + "score": -21.0, + "target_score": 21.0, + "target_score_rate": 0.0, + "win_rate": 0.0 + } + }, + "target_score": 21.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-02/metadata.json new file mode 100644 index 0000000..a351970 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "digest_compatibility_note": null, + "finished_at": 1784074623.04269, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -75.0, + "observed_raw_score": -21.0, + "provider_reasoning_effort": "xhigh", + "raw_score": -21.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "task": "simulation_heuristics_pong_ram_v1", + "task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "token_usage": { + "final_input_tokens": 720.0, + "final_output_tokens": 190.0, + "input_tokens": 720.0, + "output_tokens": 190.0, + "total_tokens": 17806.0 + }, + "tool_calls": 21, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-02/score.json new file mode 100644 index 0000000..3c2f846 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-02/score.json @@ -0,0 +1,133 @@ +{ + "normalized_score": -75.0, + "reference_score": 21.0, + "score": -21.0, + "starter_score": -2.9999999999999996, + "suite_weights": { + "hidden_nominal": 0.7, + "hidden_reset_robustness": 0.3 + }, + "suites": { + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 3055, + "mean_action_latency_ms": 0.012658657079315615, + "points_against": 21, + "points_for": 0, + "policy_error": null, + "score": -21.0, + "seed": 101, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": false + }, + { + "invalid_action": false, + "length": 3055, + "mean_action_latency_ms": 0.013793323725559513, + "points_against": 21, + "points_for": 0, + "policy_error": null, + "score": -21.0, + "seed": 211, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": false + }, + { + "invalid_action": false, + "length": 3055, + "mean_action_latency_ms": 0.013321648396051652, + "points_against": 21, + "points_for": 0, + "policy_error": null, + "score": -21.0, + "seed": 307, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": false + } + ], + "invalid_episode_rate": 0.0, + "max_score": -21.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.013257876400308927, + "mean_score": -21.0, + "min_score": -21.0, + "policy_path": "submitted_artifact", + "score": -21.0, + "target_score": 21.0, + "target_score_rate": 0.0, + "win_rate": 0.0 + }, + "hidden_reset_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 3055, + "mean_action_latency_ms": 0.012896045716250781, + "points_against": 21, + "points_for": 0, + "policy_error": null, + "score": -21.0, + "seed": 503, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": false + }, + { + "invalid_action": false, + "length": 3051, + "mean_action_latency_ms": 0.013689050204888285, + "points_against": 21, + "points_for": 0, + "policy_error": null, + "score": -21.0, + "seed": 601, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": false + }, + { + "invalid_action": false, + "length": 3052, + "mean_action_latency_ms": 0.013577351227602867, + "points_against": 21, + "points_for": 0, + "policy_error": null, + "score": -21.0, + "seed": 701, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": false + } + ], + "invalid_episode_rate": 0.0, + "max_score": -21.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.013387482382913977, + "mean_score": -21.0, + "min_score": -21.0, + "policy_path": "submitted_artifact", + "score": -21.0, + "target_score": 21.0, + "target_score_rate": 0.0, + "win_rate": 0.0 + } + }, + "target_score": 21.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-03/metadata.json new file mode 100644 index 0000000..77fa72c --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "digest_compatibility_note": null, + "finished_at": 1784076792.366179, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -58.333333333, + "observed_raw_score": -17.0, + "provider_reasoning_effort": "xhigh", + "raw_score": -17.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "task": "simulation_heuristics_pong_ram_v1", + "task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "token_usage": { + "final_input_tokens": 845.0, + "final_output_tokens": 262.0, + "input_tokens": 845.0, + "output_tokens": 262.0, + "total_tokens": 13907.0 + }, + "tool_calls": 15, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-03/score.json new file mode 100644 index 0000000..7ad0866 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-03/score.json @@ -0,0 +1,133 @@ +{ + "normalized_score": -58.333333333, + "reference_score": 21.0, + "score": -17.0, + "starter_score": -2.9999999999999996, + "suite_weights": { + "hidden_nominal": 0.7, + "hidden_reset_robustness": 0.3 + }, + "suites": { + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 4815, + "mean_action_latency_ms": 0.003980811460955244, + "points_against": 21, + "points_for": 4, + "policy_error": null, + "score": -17.0, + "seed": 101, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": false + }, + { + "invalid_action": false, + "length": 4815, + "mean_action_latency_ms": 0.003966486017652019, + "points_against": 21, + "points_for": 4, + "policy_error": null, + "score": -17.0, + "seed": 211, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": false + }, + { + "invalid_action": false, + "length": 4815, + "mean_action_latency_ms": 0.004236061374539836, + "points_against": 21, + "points_for": 4, + "policy_error": null, + "score": -17.0, + "seed": 307, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": false + } + ], + "invalid_episode_rate": 0.0, + "max_score": -17.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.0040611196177157, + "mean_score": -17.0, + "min_score": -17.0, + "policy_path": "submitted_artifact", + "score": -17.0, + "target_score": 21.0, + "target_score_rate": 0.0, + "win_rate": 0.0 + }, + "hidden_reset_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 4815, + "mean_action_latency_ms": 0.0039586330002883635, + "points_against": 21, + "points_for": 4, + "policy_error": null, + "score": -17.0, + "seed": 503, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": false + }, + { + "invalid_action": false, + "length": 4811, + "mean_action_latency_ms": 0.0035986335014270487, + "points_against": 21, + "points_for": 4, + "policy_error": null, + "score": -17.0, + "seed": 601, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": false + }, + { + "invalid_action": false, + "length": 4812, + "mean_action_latency_ms": 0.0033883651964682297, + "points_against": 21, + "points_for": 4, + "policy_error": null, + "score": -17.0, + "seed": 701, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": false + } + ], + "invalid_episode_rate": 0.0, + "max_score": -17.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.0036485438993945473, + "mean_score": -17.0, + "min_score": -17.0, + "policy_path": "submitted_artifact", + "score": -17.0, + "target_score": 21.0, + "target_score_rate": 0.0, + "win_rate": 0.0 + } + }, + "target_score": 21.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-04/metadata.json new file mode 100644 index 0000000..5bd783f --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-04/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "digest_compatibility_note": null, + "finished_at": 1784079105.716872, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -75.0, + "observed_raw_score": -21.0, + "provider_reasoning_effort": "xhigh", + "raw_score": -21.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "task": "simulation_heuristics_pong_ram_v1", + "task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "token_usage": { + "final_input_tokens": 544.0, + "final_output_tokens": 229.0, + "input_tokens": 544.0, + "output_tokens": 229.0, + "total_tokens": 14597.0 + }, + "tool_calls": 13, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-04/score.json new file mode 100644 index 0000000..b536a9a --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-04/score.json @@ -0,0 +1,133 @@ +{ + "normalized_score": -75.0, + "reference_score": 21.0, + "score": -21.0, + "starter_score": -2.9999999999999996, + "suite_weights": { + "hidden_nominal": 0.7, + "hidden_reset_robustness": 0.3 + }, + "suites": { + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 3055, + "mean_action_latency_ms": 0.003980824673539195, + "points_against": 21, + "points_for": 0, + "policy_error": null, + "score": -21.0, + "seed": 101, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": false + }, + { + "invalid_action": false, + "length": 3055, + "mean_action_latency_ms": 0.00418389548353985, + "points_against": 21, + "points_for": 0, + "policy_error": null, + "score": -21.0, + "seed": 211, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": false + }, + { + "invalid_action": false, + "length": 3055, + "mean_action_latency_ms": 0.003770847405488382, + "points_against": 21, + "points_for": 0, + "policy_error": null, + "score": -21.0, + "seed": 307, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": false + } + ], + "invalid_episode_rate": 0.0, + "max_score": -21.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.003978522520855809, + "mean_score": -21.0, + "min_score": -21.0, + "policy_path": "submitted_artifact", + "score": -21.0, + "target_score": 21.0, + "target_score_rate": 0.0, + "win_rate": 0.0 + }, + "hidden_reset_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 3055, + "mean_action_latency_ms": 0.0040017647985765275, + "points_against": 21, + "points_for": 0, + "policy_error": null, + "score": -21.0, + "seed": 503, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": false + }, + { + "invalid_action": false, + "length": 3051, + "mean_action_latency_ms": 0.0035915751866715455, + "points_against": 21, + "points_for": 0, + "policy_error": null, + "score": -21.0, + "seed": 601, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": false + }, + { + "invalid_action": false, + "length": 3052, + "mean_action_latency_ms": 0.0035856373417087455, + "points_against": 21, + "points_for": 0, + "policy_error": null, + "score": -21.0, + "seed": 701, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": false + } + ], + "invalid_episode_rate": 0.0, + "max_score": -21.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.003726325775652273, + "mean_score": -21.0, + "min_score": -21.0, + "policy_path": "submitted_artifact", + "score": -21.0, + "target_score": 21.0, + "target_score_rate": 0.0, + "win_rate": 0.0 + } + }, + "target_score": 21.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-05/metadata.json new file mode 100644 index 0000000..ed0b350 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-05/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "digest_compatibility_note": null, + "finished_at": 1784081904.218542, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 79.166666667, + "observed_raw_score": 16.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 16.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05", + "task": "simulation_heuristics_pong_ram_v1", + "task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "token_usage": { + "final_input_tokens": 885.0, + "final_output_tokens": 198.0, + "input_tokens": 885.0, + "output_tokens": 198.0, + "total_tokens": 13371.0 + }, + "tool_calls": 14, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-05/score.json new file mode 100644 index 0000000..2da675d --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/trial-05/score.json @@ -0,0 +1,133 @@ +{ + "normalized_score": 79.166666667, + "reference_score": 21.0, + "score": 16.0, + "starter_score": -2.9999999999999996, + "suite_weights": { + "hidden_nominal": 0.7, + "hidden_reset_robustness": 0.3 + }, + "suites": { + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.004337354916734276, + "points_against": 0, + "points_for": 16, + "policy_error": null, + "score": 16.0, + "seed": 101, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.003737011928697703, + "points_against": 0, + "points_for": 16, + "policy_error": null, + "score": 16.0, + "seed": 211, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.003354957568700667, + "points_against": 0, + "points_for": 16, + "policy_error": null, + "score": 16.0, + "seed": 307, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "article_nominal", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 16.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.003809774804710882, + "mean_score": 16.0, + "min_score": 16.0, + "policy_path": "submitted_artifact", + "score": 16.0, + "target_score": 21.0, + "target_score_rate": 0.0, + "win_rate": 1.0 + }, + "hidden_reset_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.003438978649986287, + "points_against": 0, + "points_for": 16, + "policy_error": null, + "score": 16.0, + "seed": 503, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.003652865886343298, + "points_against": 0, + "points_for": 16, + "policy_error": null, + "score": 16.0, + "seed": 601, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.004092430954592096, + "points_against": 0, + "points_for": 16, + "policy_error": null, + "score": 16.0, + "seed": 701, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "randomized_reset", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 16.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.003728091830307227, + "mean_score": 16.0, + "min_score": 16.0, + "policy_path": "submitted_artifact", + "score": 16.0, + "target_score": 21.0, + "target_score_rate": 0.0, + "win_rate": 1.0 + } + }, + "target_score": 21.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/metadata.json index d00892d..2ad5ef5 100644 --- a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:316e198d8fa71e705c8b7fa54f41faf64e1f3c0ba5c4c92b8116b4bf78c5712b", - "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", - "finished_at": 1783938535.150049, + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", "harness": "opencode", "model": { "display_name": "GPT-5.4 Mini", @@ -11,17 +9,19 @@ "provider": "azure", "provider_reasoning_effort": "xhigh" }, - "normalized_score": -161.116433137, + "normalized_score": -186.0627378054, + "normalized_score_stddev": 3.359451433761567, "provider_reasoning_effort": "xhigh", - "source_run_id": "leaderboard/runs/article_suite/20260713T102707Z/gpt-5.4-mini", + "raw_score": -0.25834800247848033, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.012638255798059739, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "task": "simulation_heuristics_vizdoom_d1_v1", - "task_digest": "sha256:b3431b238bec4e66af6189d30bcd15d5cd227144dfe8bf0dd1011aa5416c1436", - "token_usage": { - "final_input_tokens": 1327.0, - "final_output_tokens": 193.0, - "input_tokens": 1327.0, - "output_tokens": 193.0, - "total_tokens": 16368.0 - }, - "tool_calls": 14 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/score.json index d960adc..dd29cbd 100644 --- a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/score.json @@ -1,211 +1,59 @@ { - "anchor_calibration": { - "envpool_version": "1.1.1", - "mode": "numeric", - "platform": "linux-x86_64" - }, - "isolation": { - "mode": "subprocess_per_suite", - "suite_process_count": 2, - "unique_working_directories": true - }, - "normalized_score": -161.116433137, - "reference_score": 0.8178199748694897, - "resource_cleanup": { - "removed_bytes": 0, - "removed_resources": 0, - "runs": [ - { - "removed_bytes": 0, - "removed_resources": 0, - "terminated_processes": 0 - }, - { - "removed_bytes": 0, - "removed_resources": 0, - "terminated_processes": 0 - } - ], - "terminated_processes": 0 - }, - "score": -0.16450000800192355, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": -186.0627378054, + "normalized_score_stddev": 3.359451433761567, + "raw_score_imputation_count": 0, + "reference_score": 0.8178199748694898, + "score": -0.25834800247848033, + "score_stddev": 0.012638255798059739, "starter_score": 0.44161998964846133, - "suites": { - "hidden_nominal": { - "batch_seed": 101, - "envpool_version": "1.1.1", - "episodes": [ - { - "final_variables": { - "HEALTH": 4.0 - }, - "invalid_action": false, - "lane": 0, - "length": 768, - "mean_action_latency_ms": 0.439295722268677, - "policy_error": null, - "return": -0.10400001332163811, - "seed": 101, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "HEALTH": 4.0 - }, - "invalid_action": false, - "lane": 1, - "length": 480, - "mean_action_latency_ms": 0.38578944901625317, - "policy_error": null, - "return": -0.2160000056028366, - "seed": 102, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "HEALTH": 4.0 - }, - "invalid_action": false, - "lane": 2, - "length": 576, - "mean_action_latency_ms": 0.3996389051381912, - "policy_error": null, - "return": -0.1680000089108944, - "seed": 103, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "HEALTH": 5.0 - }, - "invalid_action": false, - "lane": 3, - "length": 800, - "mean_action_latency_ms": 0.3952940576709807, - "policy_error": null, - "return": -0.08300001174211502, - "seed": 104, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "HEALTH": 4.0 - }, - "invalid_action": false, - "lane": 4, - "length": 448, - "mean_action_latency_ms": 0.3939860575233719, - "policy_error": null, - "return": -0.23200000450015068, - "seed": 105, - "terminated": true, - "truncated": false - } - ], - "frame_skip": 1, - "invalid_episode_rate": 0.0, - "max_return": -0.08300001174211502, - "max_steps": 2100, - "mean_action_latency_ms": 0.4028008383234948, - "mean_return": -0.16060000881552697, - "min_return": -0.23200000450015068, - "policy_path": "submitted_artifact", - "render_height": 180, - "render_width": 240, - "scenario": "d1", - "score": -0.16060000881552697 + "task": "simulation_heuristics_vizdoom_d1_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": -180.053168387, + "observed_raw_score": -0.2357400032132864, + "raw_score": -0.2357400032132864, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "trial": 1 }, - "hidden_secondary": { - "batch_seed": 607, - "envpool_version": "1.1.1", - "episodes": [ - { - "final_variables": { - "HEALTH": 4.0 - }, - "invalid_action": false, - "lane": 0, - "length": 576, - "mean_action_latency_ms": 0.42993988285565543, - "policy_error": null, - "return": -0.20000000670552254, - "seed": 607, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "HEALTH": 4.0 - }, - "invalid_action": false, - "lane": 1, - "length": 512, - "mean_action_latency_ms": 0.39957215631147847, - "policy_error": null, - "return": -0.2160000056028366, - "seed": 608, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "HEALTH": 8.0 - }, - "invalid_action": false, - "lane": 2, - "length": 768, - "mean_action_latency_ms": 0.4345782654127106, - "policy_error": null, - "return": -0.052000004798173904, - "seed": 609, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "HEALTH": 4.0 - }, - "invalid_action": false, - "lane": 3, - "length": 480, - "mean_action_latency_ms": 0.3747861715964973, - "policy_error": null, - "return": -0.2160000056028366, - "seed": 610, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "HEALTH": 4.0 - }, - "invalid_action": false, - "lane": 4, - "length": 544, - "mean_action_latency_ms": 0.39103973026880445, - "policy_error": null, - "return": -0.18400000780820847, - "seed": 611, - "terminated": true, - "truncated": false - } - ], - "frame_skip": 1, - "invalid_episode_rate": 0.0, - "max_return": -0.052000004798173904, - "max_steps": 2100, - "mean_action_latency_ms": 0.4059832412890293, - "mean_return": -0.17360000610351561, - "min_return": -0.2160000056028366, - "policy_path": "submitted_artifact", - "render_height": 180, - "render_width": 240, - "scenario": "d1", - "score": -0.17360000610351561 + { + "normalized_score": -187.56513016, + "observed_raw_score": -0.2640000022947788, + "raw_score": -0.2640000022947788, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "trial": 2 + }, + { + "normalized_score": -187.56513016, + "observed_raw_score": -0.2640000022947788, + "raw_score": -0.2640000022947788, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "trial": 3 + }, + { + "normalized_score": -187.56513016, + "observed_raw_score": -0.2640000022947788, + "raw_score": -0.2640000022947788, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "trial": 4 + }, + { + "normalized_score": -187.56513016, + "observed_raw_score": -0.2640000022947788, + "raw_score": -0.2640000022947788, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05", + "trial": 5 } - } + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-01/metadata.json new file mode 100644 index 0000000..d906837 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "digest_compatibility_note": null, + "finished_at": 1784074742.653706, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -180.053168387, + "observed_raw_score": -0.2357400032132864, + "provider_reasoning_effort": "xhigh", + "raw_score": -0.2357400032132864, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "task": "simulation_heuristics_vizdoom_d1_v1", + "task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "token_usage": { + "final_input_tokens": 921.0, + "final_output_tokens": 262.0, + "input_tokens": 921.0, + "output_tokens": 262.0, + "total_tokens": 11423.0 + }, + "tool_calls": 11, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-01/score.json new file mode 100644 index 0000000..a29a557 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-01/score.json @@ -0,0 +1,211 @@ +{ + "anchor_calibration": { + "envpool_version": "1.1.1", + "mode": "numeric", + "platform": "linux-x86_64" + }, + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": -180.053168387, + "reference_score": 0.8178199748694897, + "resource_cleanup": { + "removed_bytes": 0, + "removed_resources": 0, + "runs": [ + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + }, + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + } + ], + "terminated_processes": 0 + }, + "score": -0.2357400032132864, + "starter_score": 0.44161998964846133, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 0, + "length": 384, + "mean_action_latency_ms": 0.2117169351549819, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 1, + "length": 448, + "mean_action_latency_ms": 0.1866291942340987, + "policy_error": null, + "return": -0.23200000450015068, + "seed": 102, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 5.0 + }, + "invalid_action": false, + "lane": 2, + "length": 480, + "mean_action_latency_ms": 0.1824474660679698, + "policy_error": null, + "return": -0.22700000181794167, + "seed": 103, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 5.0 + }, + "invalid_action": false, + "lane": 3, + "length": 480, + "mean_action_latency_ms": 0.18646556418389082, + "policy_error": null, + "return": -0.2110000029206276, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 4, + "length": 416, + "mean_action_latency_ms": 0.18574434440009868, + "policy_error": null, + "return": -0.24800000339746475, + "seed": 105, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": -0.2110000029206276, + "max_steps": 2100, + "mean_action_latency_ms": 0.19060070080820798, + "mean_return": -0.2364000029861927, + "min_return": -0.2640000022947788, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": -0.2364000029861927 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 0, + "length": 384, + "mean_action_latency_ms": 0.25260150626612204, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 607, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 1, + "length": 384, + "mean_action_latency_ms": 0.23529956039662162, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 608, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 5.0 + }, + "invalid_action": false, + "lane": 2, + "length": 480, + "mean_action_latency_ms": 0.24159828511377177, + "policy_error": null, + "return": -0.2110000029206276, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 3, + "length": 512, + "mean_action_latency_ms": 0.2397390708210878, + "policy_error": null, + "return": -0.2160000056028366, + "seed": 610, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 4, + "length": 480, + "mean_action_latency_ms": 0.2332081513789793, + "policy_error": null, + "return": -0.2160000056028366, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": -0.2110000029206276, + "max_steps": 2100, + "mean_action_latency_ms": 0.24048931479531652, + "mean_return": -0.23420000374317168, + "min_return": -0.2640000022947788, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": -0.23420000374317168 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-02/metadata.json new file mode 100644 index 0000000..8cf2768 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "digest_compatibility_note": null, + "finished_at": 1784074623.04269, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -187.56513016, + "observed_raw_score": -0.2640000022947788, + "provider_reasoning_effort": "xhigh", + "raw_score": -0.2640000022947788, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "task": "simulation_heuristics_vizdoom_d1_v1", + "task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "token_usage": { + "final_input_tokens": 1505.0, + "final_output_tokens": 220.0, + "input_tokens": 1505.0, + "output_tokens": 220.0, + "total_tokens": 14013.0 + }, + "tool_calls": 12, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-02/score.json new file mode 100644 index 0000000..af33e91 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-02/score.json @@ -0,0 +1,211 @@ +{ + "anchor_calibration": { + "envpool_version": "1.1.1", + "mode": "numeric", + "platform": "linux-x86_64" + }, + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": -187.56513016, + "reference_score": 0.8178199748694897, + "resource_cleanup": { + "removed_bytes": 0, + "removed_resources": 0, + "runs": [ + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + }, + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + } + ], + "terminated_processes": 0 + }, + "score": -0.2640000022947788, + "starter_score": 0.44161998964846133, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 0, + "length": 384, + "mean_action_latency_ms": 0.46621569102474797, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 1, + "length": 384, + "mean_action_latency_ms": 0.44232957588974386, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 102, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 2, + "length": 384, + "mean_action_latency_ms": 0.44008545228280127, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 103, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 3, + "length": 384, + "mean_action_latency_ms": 0.43773456612446654, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 4, + "length": 384, + "mean_action_latency_ms": 0.43688927447268117, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 105, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": -0.2640000022947788, + "max_steps": 2100, + "mean_action_latency_ms": 0.4446509119588882, + "mean_return": -0.2640000022947788, + "min_return": -0.2640000022947788, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": -0.2640000022947788 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 0, + "length": 384, + "mean_action_latency_ms": 0.32195636110069853, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 607, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 1, + "length": 384, + "mean_action_latency_ms": 0.2923051021449889, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 608, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 2, + "length": 384, + "mean_action_latency_ms": 0.2899604636089255, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 3, + "length": 384, + "mean_action_latency_ms": 0.2888124339127292, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 610, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 4, + "length": 384, + "mean_action_latency_ms": 0.28818423258295905, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": -0.2640000022947788, + "max_steps": 2100, + "mean_action_latency_ms": 0.2962437186700603, + "mean_return": -0.2640000022947788, + "min_return": -0.2640000022947788, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": -0.2640000022947788 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-03/metadata.json new file mode 100644 index 0000000..f806f9b --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "digest_compatibility_note": null, + "finished_at": 1784076792.366179, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -187.56513016, + "observed_raw_score": -0.2640000022947788, + "provider_reasoning_effort": "xhigh", + "raw_score": -0.2640000022947788, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "task": "simulation_heuristics_vizdoom_d1_v1", + "task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "token_usage": { + "final_input_tokens": 1478.0, + "final_output_tokens": 227.0, + "input_tokens": 1478.0, + "output_tokens": 227.0, + "total_tokens": 11945.0 + }, + "tool_calls": 10, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-03/score.json new file mode 100644 index 0000000..b165728 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-03/score.json @@ -0,0 +1,211 @@ +{ + "anchor_calibration": { + "envpool_version": "1.1.1", + "mode": "numeric", + "platform": "linux-x86_64" + }, + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": -187.56513016, + "reference_score": 0.8178199748694897, + "resource_cleanup": { + "removed_bytes": 0, + "removed_resources": 0, + "runs": [ + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + }, + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + } + ], + "terminated_processes": 0 + }, + "score": -0.2640000022947788, + "starter_score": 0.44161998964846133, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 0, + "length": 384, + "mean_action_latency_ms": 0.2687672773996989, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 1, + "length": 384, + "mean_action_latency_ms": 0.25103507020200294, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 102, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 2, + "length": 384, + "mean_action_latency_ms": 0.2492256365561237, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 103, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 3, + "length": 384, + "mean_action_latency_ms": 0.24633660602072874, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 4, + "length": 384, + "mean_action_latency_ms": 0.24550990686596683, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 105, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": -0.2640000022947788, + "max_steps": 2100, + "mean_action_latency_ms": 0.25217489940890425, + "mean_return": -0.2640000022947788, + "min_return": -0.2640000022947788, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": -0.2640000022947788 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 0, + "length": 384, + "mean_action_latency_ms": 0.2688381161230306, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 607, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 1, + "length": 384, + "mean_action_latency_ms": 0.2502202308581521, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 608, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 2, + "length": 384, + "mean_action_latency_ms": 0.24534330198851725, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 3, + "length": 384, + "mean_action_latency_ms": 0.24609865310291448, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 610, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 4, + "length": 384, + "mean_action_latency_ms": 0.2457198716001585, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": -0.2640000022947788, + "max_steps": 2100, + "mean_action_latency_ms": 0.2512440347345546, + "mean_return": -0.2640000022947788, + "min_return": -0.2640000022947788, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": -0.2640000022947788 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-04/metadata.json new file mode 100644 index 0000000..581a69f --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-04/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "digest_compatibility_note": null, + "finished_at": 1784079105.716872, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -187.56513016, + "observed_raw_score": -0.2640000022947788, + "provider_reasoning_effort": "xhigh", + "raw_score": -0.2640000022947788, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "task": "simulation_heuristics_vizdoom_d1_v1", + "task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "token_usage": { + "final_input_tokens": 765.0, + "final_output_tokens": 195.0, + "input_tokens": 765.0, + "output_tokens": 195.0, + "total_tokens": 12736.0 + }, + "tool_calls": 12, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-04/score.json new file mode 100644 index 0000000..c0c331e --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-04/score.json @@ -0,0 +1,211 @@ +{ + "anchor_calibration": { + "envpool_version": "1.1.1", + "mode": "numeric", + "platform": "linux-x86_64" + }, + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": -187.56513016, + "reference_score": 0.8178199748694897, + "resource_cleanup": { + "removed_bytes": 0, + "removed_resources": 0, + "runs": [ + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + }, + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + } + ], + "terminated_processes": 0 + }, + "score": -0.2640000022947788, + "starter_score": 0.44161998964846133, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 0, + "length": 384, + "mean_action_latency_ms": 0.2397822584801664, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 1, + "length": 384, + "mean_action_latency_ms": 0.2204639264770473, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 102, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 2, + "length": 384, + "mean_action_latency_ms": 0.21643378325582793, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 103, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 3, + "length": 384, + "mean_action_latency_ms": 0.21570341777987778, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 4, + "length": 384, + "mean_action_latency_ms": 0.21629892095612982, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 105, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": -0.2640000022947788, + "max_steps": 2100, + "mean_action_latency_ms": 0.22173646138980985, + "mean_return": -0.2640000022947788, + "min_return": -0.2640000022947788, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": -0.2640000022947788 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 0, + "length": 384, + "mean_action_latency_ms": 0.26224593360287446, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 607, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 1, + "length": 384, + "mean_action_latency_ms": 0.23886664712335914, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 608, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 2, + "length": 384, + "mean_action_latency_ms": 0.23468153085559607, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 3, + "length": 384, + "mean_action_latency_ms": 0.23422976664733142, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 610, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 4, + "length": 384, + "mean_action_latency_ms": 0.24144447039968023, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": -0.2640000022947788, + "max_steps": 2100, + "mean_action_latency_ms": 0.24229366972576827, + "mean_return": -0.2640000022947788, + "min_return": -0.2640000022947788, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": -0.2640000022947788 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-05/metadata.json new file mode 100644 index 0000000..f697e5a --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-05/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "digest_compatibility_note": null, + "finished_at": 1784081904.218542, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -187.56513016, + "observed_raw_score": -0.2640000022947788, + "provider_reasoning_effort": "xhigh", + "raw_score": -0.2640000022947788, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05", + "task": "simulation_heuristics_vizdoom_d1_v1", + "task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "token_usage": { + "final_input_tokens": 1720.0, + "final_output_tokens": 217.0, + "input_tokens": 1720.0, + "output_tokens": 217.0, + "total_tokens": 12177.0 + }, + "tool_calls": 12, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-05/score.json new file mode 100644 index 0000000..b029134 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/trial-05/score.json @@ -0,0 +1,211 @@ +{ + "anchor_calibration": { + "envpool_version": "1.1.1", + "mode": "numeric", + "platform": "linux-x86_64" + }, + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": -187.56513016, + "reference_score": 0.8178199748694897, + "resource_cleanup": { + "removed_bytes": 0, + "removed_resources": 0, + "runs": [ + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + }, + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + } + ], + "terminated_processes": 0 + }, + "score": -0.2640000022947788, + "starter_score": 0.44161998964846133, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 0, + "length": 384, + "mean_action_latency_ms": 0.12425181921571493, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 1, + "length": 384, + "mean_action_latency_ms": 0.1077580915686364, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 102, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 2, + "length": 384, + "mean_action_latency_ms": 0.10544837762912114, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 103, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 3, + "length": 384, + "mean_action_latency_ms": 0.10459061013534665, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 4, + "length": 384, + "mean_action_latency_ms": 0.10554475011304021, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 105, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": -0.2640000022947788, + "max_steps": 2100, + "mean_action_latency_ms": 0.10951872973237187, + "mean_return": -0.2640000022947788, + "min_return": -0.2640000022947788, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": -0.2640000022947788 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 0, + "length": 384, + "mean_action_latency_ms": 0.11629477376118302, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 607, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 1, + "length": 384, + "mean_action_latency_ms": 0.09714766444327931, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 608, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 2, + "length": 384, + "mean_action_latency_ms": 0.0932229255946974, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 3, + "length": 384, + "mean_action_latency_ms": 0.0936128538645183, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 610, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 4, + "length": 384, + "mean_action_latency_ms": 0.09199571892774354, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": -0.2640000022947788, + "max_steps": 2100, + "mean_action_latency_ms": 0.09845478731828432, + "mean_return": -0.2640000022947788, + "min_return": -0.2640000022947788, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": -0.2640000022947788 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/metadata.json index cf37cb7..2b2aa10 100644 --- a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:29434bdcdce19a6e0cb24d77c800d172d49a9529875ffd4201cf928e242f4e62", - "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", - "finished_at": 1783939368.238364, + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", "harness": "opencode", "model": { "display_name": "GPT-5.4 Mini", @@ -11,17 +9,19 @@ "provider": "azure", "provider_reasoning_effort": "xhigh" }, - "normalized_score": -9.859625668, + "normalized_score": 16.5307486632, + "normalized_score_stddev": 45.27484310055022, "provider_reasoning_effort": "xhigh", - "source_run_id": "leaderboard/runs/article_suite/20260713T104034Z/gpt-5.4-mini", + "raw_score": 81.36, + "raw_score_imputation_count": 0, + "raw_score_stddev": 135.46233055724386, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01/simulation_heuristics_vizdoom_d3_v1", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "task": "simulation_heuristics_vizdoom_d3_v1", - "task_digest": "sha256:fcc6ed05673b7981aa17d60ca4e0d355fbcb57a6cbca293c0fa201ab97cdd081", - "token_usage": { - "final_input_tokens": 610.0, - "final_output_tokens": 258.0, - "input_tokens": 610.0, - "output_tokens": 258.0, - "total_tokens": 13668.0 - }, - "tool_calls": 11 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/score.json index c0dfc03..1996be4 100644 --- a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/score.json @@ -1,229 +1,59 @@ { - "isolation": { - "mode": "subprocess_per_suite", - "suite_process_count": 2, - "unique_working_directories": true - }, - "normalized_score": -9.859625668, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": 16.5307486632, + "normalized_score_stddev": 45.27484310055022, + "raw_score_imputation_count": 0, "reference_score": 331.1, - "score": 2.4, + "score": 81.36, + "score_stddev": 135.46233055724386, "starter_score": 31.9, - "suites": { - "hidden_nominal": { - "batch_seed": 101, - "envpool_version": "1.1.1", - "episodes": [ - { - "final_variables": { - "AMMO2": 16.0, - "DAMAGECOUNT": 0.0, - "HEALTH": 23.0, - "HITCOUNT": 0.0, - "KILLCOUNT": 0.0 - }, - "invalid_action": false, - "lane": 0, - "length": 414, - "mean_action_latency_ms": 5.564870929192518, - "policy_error": null, - "return": 0.0, - "seed": 101, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 36.0, - "DAMAGECOUNT": 0.0, - "HEALTH": 1.0, - "HITCOUNT": 0.0, - "KILLCOUNT": 0.0 - }, - "invalid_action": false, - "lane": 1, - "length": 519, - "mean_action_latency_ms": 5.470503561256134, - "policy_error": null, - "return": 0.0, - "seed": 102, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 28.0, - "DAMAGECOUNT": 0.0, - "HEALTH": 7.0, - "HITCOUNT": 0.0, - "KILLCOUNT": 0.0 - }, - "invalid_action": false, - "lane": 2, - "length": 236, - "mean_action_latency_ms": 5.521233540849161, - "policy_error": null, - "return": 0.0, - "seed": 103, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 14.0, - "DAMAGECOUNT": 0.0, - "HEALTH": 1.0, - "HITCOUNT": 0.0, - "KILLCOUNT": 0.0 - }, - "invalid_action": false, - "lane": 3, - "length": 207, - "mean_action_latency_ms": 5.482445196541035, - "policy_error": null, - "return": 0.0, - "seed": 104, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 16.0, - "DAMAGECOUNT": 0.0, - "HEALTH": 13.0, - "HITCOUNT": 0.0, - "KILLCOUNT": 0.0 - }, - "invalid_action": false, - "lane": 4, - "length": 166, - "mean_action_latency_ms": 5.500560115843293, - "policy_error": null, - "return": 0.0, - "seed": 105, - "terminated": true, - "truncated": false - } - ], - "frame_skip": 2, - "invalid_episode_rate": 0.0, - "max_return": 0.0, - "max_steps": 1050, - "mean_action_latency_ms": 5.507922668736429, - "mean_return": 0.0, - "min_return": 0.0, - "policy_path": "submitted_artifact", - "render_height": 480, - "render_width": 640, - "scenario": "d3", - "score": 0.0 + "task": "simulation_heuristics_vizdoom_d3_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 97.159090909, + "observed_raw_score": 322.6, + "raw_score": 322.6, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01/simulation_heuristics_vizdoom_d3_v1", + "trial": 1 }, - "hidden_secondary": { - "batch_seed": 607, - "envpool_version": "1.1.1", - "episodes": [ - { - "final_variables": { - "AMMO2": 29.0, - "DAMAGECOUNT": 0.0, - "HEALTH": 13.0, - "HITCOUNT": 0.0, - "KILLCOUNT": 0.0 - }, - "invalid_action": false, - "lane": 0, - "length": 88, - "mean_action_latency_ms": 5.623010484586385, - "policy_error": null, - "return": 0.0, - "seed": 607, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 20.0, - "DAMAGECOUNT": 0.0, - "HEALTH": 100.0, - "HITCOUNT": 0.0, - "KILLCOUNT": 0.0 - }, - "invalid_action": false, - "lane": 1, - "length": 1050, - "mean_action_latency_ms": 5.629530707817702, - "policy_error": null, - "return": 0.0, - "seed": 608, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "AMMO2": 27.0, - "DAMAGECOUNT": 20.0, - "HEALTH": 4.0, - "HITCOUNT": 2.0, - "KILLCOUNT": 2.0 - }, - "invalid_action": false, - "lane": 2, - "length": 762, - "mean_action_latency_ms": 5.546016788496355, - "policy_error": null, - "return": 40.0, - "seed": 609, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 15.0, - "DAMAGECOUNT": 0.0, - "HEALTH": 12.0, - "HITCOUNT": 0.0, - "KILLCOUNT": 0.0 - }, - "invalid_action": false, - "lane": 3, - "length": 493, - "mean_action_latency_ms": 5.572860722352598, - "policy_error": null, - "return": 0.0, - "seed": 610, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 30.0, - "DAMAGECOUNT": 0.0, - "HEALTH": 13.0, - "HITCOUNT": 0.0, - "KILLCOUNT": 0.0 - }, - "invalid_action": false, - "lane": 4, - "length": 224, - "mean_action_latency_ms": 5.576010119901704, - "policy_error": null, - "return": 0.0, - "seed": 611, - "terminated": true, - "truncated": false - } - ], - "frame_skip": 2, - "invalid_episode_rate": 0.0, - "max_return": 40.0, - "max_steps": 1050, - "mean_action_latency_ms": 5.589485764630949, - "mean_return": 8.0, - "min_return": 0.0, - "policy_path": "submitted_artifact", - "render_height": 480, - "render_width": 640, - "scenario": "d3", - "score": 8.0 + { + "normalized_score": -10.160427807, + "observed_raw_score": 1.5, + "raw_score": 1.5, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "trial": 2 + }, + { + "normalized_score": 1.771390374, + "observed_raw_score": 37.2, + "raw_score": 37.2, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "trial": 3 + }, + { + "normalized_score": -2.506684492, + "observed_raw_score": 24.4, + "raw_score": 24.4, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "trial": 4 + }, + { + "normalized_score": -3.609625668, + "observed_raw_score": 21.1, + "raw_score": 21.1, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05", + "trial": 5 } - } + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-01/metadata.json new file mode 100644 index 0000000..d117398 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "digest_compatibility_note": null, + "finished_at": 1784075921.181849, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 97.159090909, + "observed_raw_score": 322.6, + "provider_reasoning_effort": "xhigh", + "raw_score": 322.6, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01/simulation_heuristics_vizdoom_d3_v1", + "task": "simulation_heuristics_vizdoom_d3_v1", + "task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "token_usage": { + "final_input_tokens": 634.0, + "final_output_tokens": 199.0, + "input_tokens": 634.0, + "output_tokens": 199.0, + "total_tokens": 19777.0 + }, + "tool_calls": 18, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-01/score.json new file mode 100644 index 0000000..f7355ea --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-01/score.json @@ -0,0 +1,229 @@ +{ + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 97.159090909, + "reference_score": 331.1, + "score": 322.6, + "starter_score": 31.9, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 17.0, + "DAMAGECOUNT": 75.0, + "HEALTH": 100.0, + "HITCOUNT": 9.0, + "KILLCOUNT": 8.0 + }, + "invalid_action": false, + "lane": 0, + "length": 1050, + "mean_action_latency_ms": 5.646619502277601, + "policy_error": null, + "return": 155.0, + "seed": 101, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 29.0, + "DAMAGECOUNT": 265.0, + "HEALTH": 100.0, + "HITCOUNT": 27.0, + "KILLCOUNT": 23.0 + }, + "invalid_action": false, + "lane": 1, + "length": 1050, + "mean_action_latency_ms": 6.30188389193444, + "policy_error": null, + "return": 495.0, + "seed": 102, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 55.0, + "DAMAGECOUNT": 115.0, + "HEALTH": 100.0, + "HITCOUNT": 13.0, + "KILLCOUNT": 11.0 + }, + "invalid_action": false, + "lane": 2, + "length": 1050, + "mean_action_latency_ms": 6.649908533408528, + "policy_error": null, + "return": 225.0, + "seed": 103, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 56.0, + "DAMAGECOUNT": 300.0, + "HEALTH": 100.0, + "HITCOUNT": 26.0, + "KILLCOUNT": 24.0 + }, + "invalid_action": false, + "lane": 3, + "length": 1050, + "mean_action_latency_ms": 6.607993830527578, + "policy_error": null, + "return": 540.0, + "seed": 104, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 11.0, + "DAMAGECOUNT": 160.0, + "HEALTH": 88.0, + "HITCOUNT": 15.0, + "KILLCOUNT": 14.0 + }, + "invalid_action": false, + "lane": 4, + "length": 1050, + "mean_action_latency_ms": 6.9045043878612065, + "policy_error": null, + "return": 300.0, + "seed": 105, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 540.0, + "max_steps": 1050, + "mean_action_latency_ms": 6.4221820292018705, + "mean_return": 343.0, + "min_return": 155.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 343.0 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 62.0, + "DAMAGECOUNT": 180.0, + "HEALTH": 4.0, + "HITCOUNT": 14.0, + "KILLCOUNT": 14.0 + }, + "invalid_action": false, + "lane": 0, + "length": 831, + "mean_action_latency_ms": 7.889954388930622, + "policy_error": null, + "return": 320.0, + "seed": 607, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 16.0, + "DAMAGECOUNT": 205.0, + "HEALTH": 1.0, + "HITCOUNT": 20.0, + "KILLCOUNT": 18.0 + }, + "invalid_action": false, + "lane": 1, + "length": 857, + "mean_action_latency_ms": 6.821388263546599, + "policy_error": null, + "return": 385.0, + "seed": 608, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 130.0, + "HEALTH": 46.0, + "HITCOUNT": 12.0, + "KILLCOUNT": 11.0 + }, + "invalid_action": false, + "lane": 2, + "length": 1050, + "mean_action_latency_ms": 7.051365478407769, + "policy_error": null, + "return": 240.0, + "seed": 609, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 64.0, + "DAMAGECOUNT": 145.0, + "HEALTH": 100.0, + "HITCOUNT": 13.0, + "KILLCOUNT": 13.0 + }, + "invalid_action": false, + "lane": 3, + "length": 1050, + "mean_action_latency_ms": 6.179718293604397, + "policy_error": null, + "return": 275.0, + "seed": 610, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 14.0, + "DAMAGECOUNT": 85.0, + "HEALTH": 85.0, + "HITCOUNT": 7.0, + "KILLCOUNT": 7.0 + }, + "invalid_action": false, + "lane": 4, + "length": 1050, + "mean_action_latency_ms": 5.647325260298593, + "policy_error": null, + "return": 155.0, + "seed": 611, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 385.0, + "max_steps": 1050, + "mean_action_latency_ms": 6.717950336957595, + "mean_return": 275.0, + "min_return": 155.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 275.0 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-02/metadata.json new file mode 100644 index 0000000..06b01f4 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "digest_compatibility_note": null, + "finished_at": 1784074623.04269, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -10.160427807, + "observed_raw_score": 1.5, + "provider_reasoning_effort": "xhigh", + "raw_score": 1.5, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "task": "simulation_heuristics_vizdoom_d3_v1", + "task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "token_usage": { + "final_input_tokens": 632.0, + "final_output_tokens": 194.0, + "input_tokens": 632.0, + "output_tokens": 194.0, + "total_tokens": 14138.0 + }, + "tool_calls": 15, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-02/score.json new file mode 100644 index 0000000..8066d0b --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-02/score.json @@ -0,0 +1,229 @@ +{ + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": -10.160427807, + "reference_score": 331.1, + "score": 1.5, + "starter_score": 31.9, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 11.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 4.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 0, + "length": 73, + "mean_action_latency_ms": 2.4930866409654486, + "policy_error": null, + "return": 0.0, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 4.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 1, + "length": 232, + "mean_action_latency_ms": 2.702273717471238, + "policy_error": null, + "return": 0.0, + "seed": 102, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 5.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 7.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 2, + "length": 110, + "mean_action_latency_ms": 2.412408150055192, + "policy_error": null, + "return": 0.0, + "seed": 103, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 4.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 10.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 3, + "length": 117, + "mean_action_latency_ms": 2.4189213211210365, + "policy_error": null, + "return": 0.0, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 5.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 4.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 4, + "length": 115, + "mean_action_latency_ms": 2.3962137491806694, + "policy_error": null, + "return": 0.0, + "seed": 105, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 0.0, + "max_steps": 1050, + "mean_action_latency_ms": 2.4845807157587165, + "mean_return": 0.0, + "min_return": 0.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 0.0 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 15.0, + "HEALTH": 7.0, + "HITCOUNT": 1.0, + "KILLCOUNT": 1.0 + }, + "invalid_action": false, + "lane": 0, + "length": 183, + "mean_action_latency_ms": 2.4072949474300844, + "policy_error": null, + "return": 25.0, + "seed": 607, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 6.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 10.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 1, + "length": 106, + "mean_action_latency_ms": 2.2758428218229763, + "policy_error": null, + "return": 0.0, + "seed": 608, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 7.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 2, + "length": 367, + "mean_action_latency_ms": 2.3025390603236997, + "policy_error": null, + "return": 0.0, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 13.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 3, + "length": 371, + "mean_action_latency_ms": 2.312555146024555, + "policy_error": null, + "return": 0.0, + "seed": 610, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 3.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 13.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 4, + "length": 137, + "mean_action_latency_ms": 2.342233692642546, + "policy_error": null, + "return": 0.0, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 25.0, + "max_steps": 1050, + "mean_action_latency_ms": 2.328093133648772, + "mean_return": 5.0, + "min_return": 0.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 5.0 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-03/metadata.json new file mode 100644 index 0000000..d727fd6 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "digest_compatibility_note": null, + "finished_at": 1784076792.366179, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 1.771390374, + "observed_raw_score": 37.2, + "provider_reasoning_effort": "xhigh", + "raw_score": 37.2, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "task": "simulation_heuristics_vizdoom_d3_v1", + "task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "token_usage": { + "final_input_tokens": 796.0, + "final_output_tokens": 213.0, + "input_tokens": 796.0, + "output_tokens": 213.0, + "total_tokens": 12785.0 + }, + "tool_calls": 12, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-03/score.json new file mode 100644 index 0000000..3ee9221 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-03/score.json @@ -0,0 +1,229 @@ +{ + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 1.771390374, + "reference_score": 331.1, + "score": 37.2, + "starter_score": 31.9, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 20.0, + "HEALTH": 10.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 0, + "length": 315, + "mean_action_latency_ms": 5.8433850251493, + "policy_error": null, + "return": 40.0, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 20.0, + "HEALTH": 12.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 1, + "length": 445, + "mean_action_latency_ms": 5.737529586205322, + "policy_error": null, + "return": 40.0, + "seed": 102, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 3.0, + "DAMAGECOUNT": 15.0, + "HEALTH": 1.0, + "HITCOUNT": 1.0, + "KILLCOUNT": 1.0 + }, + "invalid_action": false, + "lane": 2, + "length": 375, + "mean_action_latency_ms": 5.722608546415965, + "policy_error": null, + "return": 25.0, + "seed": 103, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 10.0, + "HEALTH": 10.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 3, + "length": 233, + "mean_action_latency_ms": 5.6318166014206765, + "policy_error": null, + "return": 30.0, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 25.0, + "HEALTH": 4.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 4, + "length": 398, + "mean_action_latency_ms": 5.6471056468672485, + "policy_error": null, + "return": 45.0, + "seed": 105, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 45.0, + "max_steps": 1050, + "mean_action_latency_ms": 5.716489081211703, + "mean_return": 36.0, + "min_return": 25.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 36.0 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 5.0, + "DAMAGECOUNT": 20.0, + "HEALTH": 1.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 0, + "length": 158, + "mean_action_latency_ms": 5.699313799791698, + "policy_error": null, + "return": 40.0, + "seed": 607, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 15.0, + "HEALTH": 10.0, + "HITCOUNT": 1.0, + "KILLCOUNT": 1.0 + }, + "invalid_action": false, + "lane": 1, + "length": 412, + "mean_action_latency_ms": 5.59018411248633, + "policy_error": null, + "return": 25.0, + "seed": 608, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 3.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 13.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 2, + "length": 329, + "mean_action_latency_ms": 5.516373368382092, + "policy_error": null, + "return": 0.0, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 40.0, + "HEALTH": 10.0, + "HITCOUNT": 3.0, + "KILLCOUNT": 3.0 + }, + "invalid_action": false, + "lane": 3, + "length": 990, + "mean_action_latency_ms": 6.544470527406895, + "policy_error": null, + "return": 70.0, + "seed": 610, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 35.0, + "HEALTH": 10.0, + "HITCOUNT": 3.0, + "KILLCOUNT": 3.0 + }, + "invalid_action": false, + "lane": 4, + "length": 258, + "mean_action_latency_ms": 5.524764198434445, + "policy_error": null, + "return": 65.0, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 70.0, + "max_steps": 1050, + "mean_action_latency_ms": 5.775021201300293, + "mean_return": 40.0, + "min_return": 0.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 40.0 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-04/metadata.json new file mode 100644 index 0000000..38609fe --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-04/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "digest_compatibility_note": null, + "finished_at": 1784079105.716872, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -2.506684492, + "observed_raw_score": 24.4, + "provider_reasoning_effort": "xhigh", + "raw_score": 24.4, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "task": "simulation_heuristics_vizdoom_d3_v1", + "task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "token_usage": { + "final_input_tokens": 1817.0, + "final_output_tokens": 217.0, + "input_tokens": 1817.0, + "output_tokens": 217.0, + "total_tokens": 20466.0 + }, + "tool_calls": 19, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-04/score.json new file mode 100644 index 0000000..261e93d --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-04/score.json @@ -0,0 +1,229 @@ +{ + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": -2.506684492, + "reference_score": 331.1, + "score": 24.4, + "starter_score": 31.9, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 5.0, + "HEALTH": 3.0, + "HITCOUNT": 1.0, + "KILLCOUNT": 1.0 + }, + "invalid_action": false, + "lane": 0, + "length": 216, + "mean_action_latency_ms": 7.614488799676851, + "policy_error": null, + "return": 15.0, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 19.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 1, + "length": 426, + "mean_action_latency_ms": 5.6243633562830135, + "policy_error": null, + "return": 0.0, + "seed": 102, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 1.0, + "DAMAGECOUNT": 10.0, + "HEALTH": 16.0, + "HITCOUNT": 1.0, + "KILLCOUNT": 1.0 + }, + "invalid_action": false, + "lane": 2, + "length": 246, + "mean_action_latency_ms": 5.558255226994918, + "policy_error": null, + "return": 20.0, + "seed": 103, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 1.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 10.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 3, + "length": 173, + "mean_action_latency_ms": 5.456319261837557, + "policy_error": null, + "return": 0.0, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 5.0, + "HEALTH": 7.0, + "HITCOUNT": 1.0, + "KILLCOUNT": 1.0 + }, + "invalid_action": false, + "lane": 4, + "length": 191, + "mean_action_latency_ms": 5.487486447027217, + "policy_error": null, + "return": 15.0, + "seed": 105, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 20.0, + "max_steps": 1050, + "mean_action_latency_ms": 5.948182618363911, + "mean_return": 10.0, + "min_return": 0.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 10.0 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 70.0, + "HEALTH": 4.0, + "HITCOUNT": 5.0, + "KILLCOUNT": 5.0 + }, + "invalid_action": false, + "lane": 0, + "length": 439, + "mean_action_latency_ms": 7.574730444538295, + "policy_error": null, + "return": 120.0, + "seed": 607, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 5.0, + "DAMAGECOUNT": 30.0, + "HEALTH": 22.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 1, + "length": 190, + "mean_action_latency_ms": 5.557355441545186, + "policy_error": null, + "return": 50.0, + "seed": 608, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 6.0, + "DAMAGECOUNT": 40.0, + "HEALTH": 10.0, + "HITCOUNT": 4.0, + "KILLCOUNT": 4.0 + }, + "invalid_action": false, + "lane": 2, + "length": 528, + "mean_action_latency_ms": 5.772683924684922, + "policy_error": null, + "return": 80.0, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 15.0, + "HEALTH": 1.0, + "HITCOUNT": 1.0, + "KILLCOUNT": 1.0 + }, + "invalid_action": false, + "lane": 3, + "length": 343, + "mean_action_latency_ms": 5.514197790066633, + "policy_error": null, + "return": 25.0, + "seed": 610, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 5.0, + "HEALTH": 7.0, + "HITCOUNT": 1.0, + "KILLCOUNT": 1.0 + }, + "invalid_action": false, + "lane": 4, + "length": 578, + "mean_action_latency_ms": 5.525901990984551, + "policy_error": null, + "return": 15.0, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 120.0, + "max_steps": 1050, + "mean_action_latency_ms": 5.988973918363917, + "mean_return": 58.0, + "min_return": 15.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 58.0 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-05/metadata.json new file mode 100644 index 0000000..ce7e2fe --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-05/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "digest_compatibility_note": null, + "finished_at": 1784081904.218542, + "harness": "opencode", + "model": { + "display_name": "GPT-5.4 Mini", + "id": "gpt-5.4-mini", + "model": "azure/gpt-5.4-mini", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -3.609625668, + "observed_raw_score": 21.1, + "provider_reasoning_effort": "xhigh", + "raw_score": 21.1, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05", + "task": "simulation_heuristics_vizdoom_d3_v1", + "task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "token_usage": { + "final_input_tokens": 558.0, + "final_output_tokens": 251.0, + "input_tokens": 558.0, + "output_tokens": 251.0, + "total_tokens": 14633.0 + }, + "tool_calls": 9, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-05/score.json new file mode 100644 index 0000000..731a679 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/trial-05/score.json @@ -0,0 +1,229 @@ +{ + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": -3.609625668, + "reference_score": 331.1, + "score": 21.1, + "starter_score": 31.9, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 4.0, + "DAMAGECOUNT": 5.0, + "HEALTH": 15.0, + "HITCOUNT": 1.0, + "KILLCOUNT": 1.0 + }, + "invalid_action": false, + "lane": 0, + "length": 264, + "mean_action_latency_ms": 13.738082735940363, + "policy_error": null, + "return": 15.0, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 4.0, + "DAMAGECOUNT": 5.0, + "HEALTH": 22.0, + "HITCOUNT": 1.0, + "KILLCOUNT": 1.0 + }, + "invalid_action": false, + "lane": 1, + "length": 160, + "mean_action_latency_ms": 12.262725713662803, + "policy_error": null, + "return": 15.0, + "seed": 102, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 7.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 13.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 2, + "length": 121, + "mean_action_latency_ms": 12.116436715707305, + "policy_error": null, + "return": 0.0, + "seed": 103, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 4.0, + "DAMAGECOUNT": 30.0, + "HEALTH": 1.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 3, + "length": 444, + "mean_action_latency_ms": 13.201536837260466, + "policy_error": null, + "return": 50.0, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 4.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 4.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 4, + "length": 288, + "mean_action_latency_ms": 12.298387151935863, + "policy_error": null, + "return": 0.0, + "seed": 105, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 50.0, + "max_steps": 1050, + "mean_action_latency_ms": 12.72343383090136, + "mean_return": 16.0, + "min_return": 0.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 16.0 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 4.0, + "DAMAGECOUNT": 60.0, + "HEALTH": 18.0, + "HITCOUNT": 5.0, + "KILLCOUNT": 5.0 + }, + "invalid_action": false, + "lane": 0, + "length": 266, + "mean_action_latency_ms": 13.774102035545765, + "policy_error": null, + "return": 110.0, + "seed": 607, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 4.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 6.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 1, + "length": 351, + "mean_action_latency_ms": 12.674972669691103, + "policy_error": null, + "return": 0.0, + "seed": 608, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 4.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 22.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 2, + "length": 402, + "mean_action_latency_ms": 12.53730497920691, + "policy_error": null, + "return": 0.0, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 4.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 1.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 3, + "length": 673, + "mean_action_latency_ms": 13.176179193458443, + "policy_error": null, + "return": 0.0, + "seed": 610, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 4.0, + "DAMAGECOUNT": 25.0, + "HEALTH": 1.0, + "HITCOUNT": 3.0, + "KILLCOUNT": 3.0 + }, + "invalid_action": false, + "lane": 4, + "length": 496, + "mean_action_latency_ms": 12.794351145144432, + "policy_error": null, + "return": 55.0, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 110.0, + "max_steps": 1050, + "mean_action_latency_ms": 12.99138200460933, + "mean_return": 33.0, + "min_return": 0.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 33.0 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/metadata.json index 5a0d184..b376b48 100644 --- a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:5cf3805d59b03cb9d0bebb6f154eab38beb655918b8321500d692f5010a00cb2", - "digest_compatibility_note": "Score-equivalent to the current task. Later changes add a fail-closed verifier timeout plus a CI-only smoke config and documentation; the publication scoring config is unchanged.", - "finished_at": 1783939074.870018, + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", "harness": "opencode", "model": { "display_name": "GPT-5.5", @@ -11,17 +9,19 @@ "provider": "azure", "provider_reasoning_effort": "xhigh" }, - "normalized_score": -16.891796576, + "normalized_score": -12.5116242462, + "normalized_score_stddev": 8.756962566015595, "provider_reasoning_effort": "xhigh", - "source_run_id": "leaderboard/runs/article_suite/20260713T095807Z/gpt-5.5", + "raw_score": 2006.0944869926293, + "raw_score_imputation_count": 0, + "raw_score_stddev": 306.36385957656836, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05/tasks-81ec8d98a4" + ], "task": "simulation_heuristics_ant_v1", - "task_digest": "sha256:bbb533da0cb86459f4d49dee667e6c73ac54c0188bc40e54e911d50ef3c3bc38", - "token_usage": { - "final_input_tokens": 1098.0, - "final_output_tokens": 223.0, - "input_tokens": 1098.0, - "output_tokens": 223.0, - "total_tokens": 31017.0 - }, - "tool_calls": 32 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/score.json index d4f8a15..845c316 100644 --- a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/score.json @@ -1,141 +1,59 @@ { - "hidden_nominal": { - "episodes": [ - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.028467876109061763, - "policy_error": null, - "return": 1960.7456949187047, - "seed": 101, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": 53.32761124336437, - "x_velocity": 0.9206927285512734 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.028923347155796364, - "policy_error": null, - "return": 877.2195257753193, - "seed": 211, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": -0.9485613953689006, - "x_velocity": -1.8377400222706064 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.029911679826909676, - "policy_error": null, - "return": 2424.075753957914, - "seed": 307, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": 76.49771178866004, - "x_velocity": 1.573669182193953 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.031103011569939554, - "policy_error": null, - "return": 1210.9204369966183, - "seed": 401, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": 15.788498223386553, - "x_velocity": -1.7502006181826246 - } - ], - "fall_rate": 0.0, - "invalid_episode_rate": 0.0, - "max_return": 2424.075753957914, - "max_steps": 1000, - "mean_action_latency_ms": 0.02960147866542684, - "mean_return": 1618.240352912139, - "mean_x_position": 36.166314965010514, - "min_return": 877.2195257753193, - "policy_path": "submitted_artifact", - "score": 1618.240352912139 - }, - "hidden_nominal_weight": 0.7, - "hidden_robustness": { - "episodes": [ - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.03534062029211782, - "policy_error": null, - "return": 2816.5356939643866, - "seed": 503, - "terminated": false, - "truncated": true, - "variant": "light_low_friction", - "x_position": 96.03292689641613, - "x_velocity": 1.711305903083371 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.028797698294511065, - "policy_error": null, - "return": 2494.2487539071335, - "seed": 601, - "terminated": false, - "truncated": true, - "variant": "light_low_friction", - "x_position": 80.00959841427752, - "x_velocity": 2.2284234213796594 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.02896833320846781, - "policy_error": null, - "return": 2413.6787614355176, - "seed": 503, - "terminated": false, - "truncated": true, - "variant": "heavy_weak_actuators", - "x_position": 76.42935346107426, - "x_velocity": 1.1236195076369881 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.029127599817002192, - "policy_error": null, - "return": 1876.672105142364, - "seed": 601, - "terminated": false, - "truncated": true, - "variant": "heavy_weak_actuators", - "x_position": 49.60350134337408, - "x_velocity": 1.681621357020333 - } - ], - "fall_rate": 0.0, - "invalid_episode_rate": 0.0, - "max_return": 2816.5356939643866, - "max_steps": 1000, - "mean_action_latency_ms": 0.030558562903024722, - "mean_return": 2400.2838286123506, - "mean_x_position": 75.51884502878549, - "min_return": 1876.672105142364, - "policy_path": "submitted_artifact", - "score": 2400.2838286123506 - }, - "hidden_robustness_weight": 0.3, - "normalized_score": -16.891796576, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": -12.5116242462, + "normalized_score_stddev": 8.756962566015595, + "raw_score_imputation_count": 0, "reference_score": 5942.333244755838, - "score": 1852.8533956222025, + "score": 2006.0944869926293, + "score_stddev": 306.36385957656836, "starter_score": 2443.815838992282, - "unique_policy_evaluations": 3 + "task": "simulation_heuristics_ant_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": -6.720960755, + "observed_raw_score": 2208.681857129609, + "raw_score": 2208.681857129609, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "trial": 1 + }, + { + "normalized_score": -24.004147668, + "observed_raw_score": 1604.0265547373397, + "raw_score": 1604.0265547373397, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "trial": 2 + }, + { + "normalized_score": -13.99811598, + "observed_raw_score": 1954.0893149644958, + "raw_score": 1954.0893149644958, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "trial": 3 + }, + { + "normalized_score": -1.398078834, + "observed_raw_score": 2394.903807623008, + "raw_score": 2394.903807623008, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "trial": 4 + }, + { + "normalized_score": -16.436817994, + "observed_raw_score": 1868.7709005086936, + "raw_score": 1868.7709005086936, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05/tasks-81ec8d98a4", + "trial": 5 + } + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-01/metadata.json new file mode 100644 index 0000000..5d37c9c --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "digest_compatibility_note": null, + "finished_at": 1784074045.3527198, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -6.720960755, + "observed_raw_score": 2208.681857129609, + "provider_reasoning_effort": "xhigh", + "raw_score": 2208.681857129609, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "task": "simulation_heuristics_ant_v1", + "task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "token_usage": { + "final_input_tokens": 1609.0, + "final_output_tokens": 282.0, + "input_tokens": 1609.0, + "output_tokens": 282.0, + "total_tokens": 17763.0 + }, + "tool_calls": 18, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-01/score.json new file mode 100644 index 0000000..f9829f0 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-01/score.json @@ -0,0 +1,141 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.01907168049365282, + "policy_error": null, + "return": 1601.122188022158, + "seed": 101, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 34.68738055006754, + "x_velocity": 0.03588238448003267 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.019829126074910164, + "policy_error": null, + "return": 2522.5963748778736, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 80.66747834675905, + "x_velocity": 1.4342086254438868 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.020496094599366188, + "policy_error": null, + "return": 2449.9497004980203, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 77.08201897354911, + "x_velocity": 2.2766027143123324 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.021072883624583483, + "policy_error": null, + "return": 2282.0994021923943, + "seed": 401, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 68.69290428167095, + "x_velocity": 1.3902943972050252 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 2522.5963748778736, + "max_steps": 1000, + "mean_action_latency_ms": 0.020117446198128164, + "mean_return": 2213.9419163976117, + "mean_x_position": 65.28244553801166, + "min_return": 1601.122188022158, + "policy_path": "submitted_artifact", + "score": 2213.9419163976117 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.019811073318123817, + "policy_error": null, + "return": 2139.659168170233, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 61.46160671564114, + "x_velocity": 2.353856976070574 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.020937751047313213, + "policy_error": null, + "return": 2654.0438351392863, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 87.2676711933084, + "x_velocity": 1.7041576644825795 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.01916940649971366, + "policy_error": null, + "return": 1606.122754418626, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 35.238836088463565, + "x_velocity": -2.113524265761697 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.021226889919489622, + "policy_error": null, + "return": 2385.8077842889347, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 74.29669634060585, + "x_velocity": 2.19176681246978 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 2654.0438351392863, + "max_steps": 1000, + "mean_action_latency_ms": 0.020286280196160078, + "mean_return": 2196.40838550427, + "mean_x_position": 64.56620258450474, + "min_return": 1606.122754418626, + "policy_path": "submitted_artifact", + "score": 2196.40838550427 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": -6.720960755, + "reference_score": 5942.333244755838, + "score": 2208.681857129609, + "starter_score": 2443.815838992282, + "unique_policy_evaluations": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-02/metadata.json new file mode 100644 index 0000000..3254321 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "digest_compatibility_note": null, + "finished_at": 1784076544.549837, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -24.004147668, + "observed_raw_score": 1604.0265547373397, + "provider_reasoning_effort": "xhigh", + "raw_score": 1604.0265547373397, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "task": "simulation_heuristics_ant_v1", + "task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "token_usage": { + "final_input_tokens": 981.0, + "final_output_tokens": 215.0, + "input_tokens": 981.0, + "output_tokens": 215.0, + "total_tokens": 22700.0 + }, + "tool_calls": 28, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-02/score.json new file mode 100644 index 0000000..bf36169 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-02/score.json @@ -0,0 +1,141 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.016911959275603294, + "policy_error": null, + "return": 176.89411957893154, + "seed": 101, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": -36.57923389874495, + "x_velocity": -1.211375745728418 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.016091985628008842, + "policy_error": null, + "return": 2742.000049861795, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 91.59436514230818, + "x_velocity": 2.3429770537646277 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.016246113926172256, + "policy_error": null, + "return": 2179.0043651418314, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 63.461422980729935, + "x_velocity": 0.07659975052902723 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.01582847349345684, + "policy_error": null, + "return": 193.37175661799213, + "seed": 401, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": -35.81493950294066, + "x_velocity": 1.2937601888094719 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 2742.000049861795, + "max_steps": 1000, + "mean_action_latency_ms": 0.01626963308081031, + "mean_return": 1322.8175728001377, + "mean_x_position": 20.665403680338123, + "min_return": 176.89411957893154, + "policy_path": "submitted_artifact", + "score": 1322.8175728001377 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.015734322369098663, + "policy_error": null, + "return": 3042.151126609854, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 106.51731857664652, + "x_velocity": 0.8138932089968876 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.016522204503417015, + "policy_error": null, + "return": 3029.5377487205033, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 105.97738514657962, + "x_velocity": 2.069749974360775 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.016038810834288597, + "policy_error": null, + "return": 556.1223655505021, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": -17.350484247551925, + "x_velocity": -1.5662514507855008 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.015960564836859703, + "policy_error": null, + "return": 2412.91214281572, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 75.5912068676289, + "x_velocity": 0.8768194616473579 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 3042.151126609854, + "max_steps": 1000, + "mean_action_latency_ms": 0.016063975635915995, + "mean_return": 2260.1808459241447, + "mean_x_position": 67.68385658582578, + "min_return": 556.1223655505021, + "policy_path": "submitted_artifact", + "score": 2260.1808459241447 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": -24.004147668, + "reference_score": 5942.333244755838, + "score": 1604.0265547373397, + "starter_score": 2443.815838992282, + "unique_policy_evaluations": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-03/metadata.json new file mode 100644 index 0000000..d89edf6 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "digest_compatibility_note": null, + "finished_at": 1784080509.825609, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -13.99811598, + "observed_raw_score": 1954.0893149644958, + "provider_reasoning_effort": "xhigh", + "raw_score": 1954.0893149644958, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "task": "simulation_heuristics_ant_v1", + "task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "token_usage": { + "final_input_tokens": 2383.0, + "final_output_tokens": 202.0, + "input_tokens": 2383.0, + "output_tokens": 202.0, + "total_tokens": 36889.0 + }, + "tool_calls": 17, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-03/score.json new file mode 100644 index 0000000..33e7ca8 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-03/score.json @@ -0,0 +1,141 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.01656520925462246, + "policy_error": null, + "return": 2988.258529384216, + "seed": 101, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 104.16707560042252, + "x_velocity": 2.363546281986828 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.016722097992897034, + "policy_error": null, + "return": 664.7747350845099, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": -12.103239548783908, + "x_velocity": -2.8122828618410622 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.016976559534668922, + "policy_error": null, + "return": 2600.9601787588417, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 84.73117309184968, + "x_velocity": 2.761741228316339 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.017433492466807365, + "policy_error": null, + "return": 2115.4072197880546, + "seed": 401, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 60.47143469033601, + "x_velocity": 1.0406742624414278 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 2988.258529384216, + "max_steps": 1000, + "mean_action_latency_ms": 0.016924339812248945, + "mean_return": 2092.3501657539055, + "mean_x_position": 59.31661095845608, + "min_return": 664.7747350845099, + "policy_path": "submitted_artifact", + "score": 2092.3501657539055 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.016585947945713997, + "policy_error": null, + "return": 2809.5973334222112, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 95.14214630390696, + "x_velocity": 2.915600143846291 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.016511891037225723, + "policy_error": null, + "return": 1257.6715210493046, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 17.62910901932937, + "x_velocity": 1.9249602732435989 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.016465576365590096, + "policy_error": null, + "return": 2625.276354645639, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 86.2152805144325, + "x_velocity": 1.8760103105140047 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.016470259055495262, + "policy_error": null, + "return": -166.62255662699602, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": -53.28291766302929, + "x_velocity": -1.8836210637655881 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 2809.5973334222112, + "max_steps": 1000, + "mean_action_latency_ms": 0.01650841860100627, + "mean_return": 1631.4806631225397, + "mean_x_position": 36.42590454365988, + "min_return": -166.62255662699602, + "policy_path": "submitted_artifact", + "score": 1631.4806631225397 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": -13.99811598, + "reference_score": 5942.333244755838, + "score": 1954.0893149644958, + "starter_score": 2443.815838992282, + "unique_policy_evaluations": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-04/metadata.json new file mode 100644 index 0000000..93a1317 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-04/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "digest_compatibility_note": null, + "finished_at": 1784093477.189311, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -1.398078834, + "observed_raw_score": 2394.903807623008, + "provider_reasoning_effort": "xhigh", + "raw_score": 2394.903807623008, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "task": "simulation_heuristics_ant_v1", + "task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "token_usage": { + "final_input_tokens": 4200.0, + "final_output_tokens": 244.0, + "input_tokens": 4200.0, + "output_tokens": 244.0, + "total_tokens": 20316.0 + }, + "tool_calls": 23, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-04/score.json new file mode 100644 index 0000000..908c7ae --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-04/score.json @@ -0,0 +1,141 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.015127851627767086, + "policy_error": null, + "return": 2632.738449893457, + "seed": 101, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 86.59133294253101, + "x_velocity": 2.1494054053925993 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.014626999851316214, + "policy_error": null, + "return": 2759.2392216673875, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 92.80698215403275, + "x_velocity": 1.9592731817380127 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.014990258030593395, + "policy_error": null, + "return": 2468.526557576306, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 78.33519158866122, + "x_velocity": 1.6909310063846306 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.01611954253166914, + "policy_error": null, + "return": 1606.5678228059596, + "seed": 401, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 35.227475531807826, + "x_velocity": -0.4673749187759313 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 2759.2392216673875, + "max_steps": 1000, + "mean_action_latency_ms": 0.015216163010336459, + "mean_return": 2366.768012985777, + "mean_x_position": 73.2402455542582, + "min_return": 1606.5678228059596, + "policy_path": "submitted_artifact", + "score": 2366.768012985777 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.014720248524099588, + "policy_error": null, + "return": 2560.1209086767167, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 82.81068500616887, + "x_velocity": 0.48071101945112105 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.015089788008481264, + "policy_error": null, + "return": 2779.6893993691415, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 93.91444720784214, + "x_velocity": 0.6902947820412919 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.015854522120207548, + "policy_error": null, + "return": 2273.636836518902, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 68.94608289857779, + "x_velocity": 1.8331548234507977 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.015502279158681631, + "policy_error": null, + "return": 2228.768835874762, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 66.80544008518254, + "x_velocity": 0.6067153490798205 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 2779.6893993691415, + "max_steps": 1000, + "mean_action_latency_ms": 0.015291709452867508, + "mean_return": 2460.553995109881, + "mean_x_position": 78.11916379944284, + "min_return": 2228.768835874762, + "policy_path": "submitted_artifact", + "score": 2460.553995109881 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": -1.398078834, + "reference_score": 5942.333244755838, + "score": 2394.903807623008, + "starter_score": 2443.815838992282, + "unique_policy_evaluations": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-05/metadata.json new file mode 100644 index 0000000..8e5241a --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-05/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "digest_compatibility_note": null, + "finished_at": 1784097871.7138891, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -16.436817994, + "observed_raw_score": 1868.7709005086936, + "provider_reasoning_effort": "xhigh", + "raw_score": 1868.7709005086936, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05/tasks-81ec8d98a4", + "task": "simulation_heuristics_ant_v1", + "task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "token_usage": { + "final_input_tokens": 610.0, + "final_output_tokens": 234.0, + "input_tokens": 610.0, + "output_tokens": 234.0, + "total_tokens": 26444.0 + }, + "tool_calls": 24, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-05/score.json new file mode 100644 index 0000000..fee26b9 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/trial-05/score.json @@ -0,0 +1,141 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.017913746647536755, + "policy_error": null, + "return": 2389.016477394929, + "seed": 101, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 74.22281166790762, + "x_velocity": 2.508672920822619 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.017680784221738577, + "policy_error": null, + "return": 156.76339833426255, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": -37.47828895680252, + "x_velocity": -1.507505088291623 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.021180227864533663, + "policy_error": null, + "return": 2007.3412485317763, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 55.070020477479396, + "x_velocity": 1.0145651610720563 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.017246118281036615, + "policy_error": null, + "return": 2601.2188091356065, + "seed": 401, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 84.79597747329657, + "x_velocity": 1.858617530554909 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 2601.2188091356065, + "max_steps": 1000, + "mean_action_latency_ms": 0.018505219253711402, + "mean_return": 1788.5849833491436, + "mean_x_position": 44.15263016547027, + "min_return": 156.76339833426255, + "policy_path": "submitted_artifact", + "score": 1788.5849833491436 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.018348641227930784, + "policy_error": null, + "return": 2745.016426505356, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 91.85677954590513, + "x_velocity": -0.8914465365953106 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.018894037697464228, + "policy_error": null, + "return": 2724.6856817722264, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 90.91680012853115, + "x_velocity": -0.9347595834074696 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.01803864585235715, + "policy_error": null, + "return": 817.9996690088209, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": -3.757484319159554, + "x_velocity": -1.8454673649829978 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.019061703700572252, + "policy_error": null, + "return": 1935.7837182375044, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 52.223617067519896, + "x_velocity": -2.1286202783832664 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 2745.016426505356, + "max_steps": 1000, + "mean_action_latency_ms": 0.018585757119581103, + "mean_return": 2055.871373880977, + "mean_x_position": 57.80992810569916, + "min_return": 817.9996690088209, + "policy_path": "submitted_artifact", + "score": 2055.871373880977 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": -16.436817994, + "reference_score": 5942.333244755838, + "score": 1868.7709005086936, + "starter_score": 2443.815838992282, + "unique_policy_evaluations": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/metadata.json index 36d1330..ce3a997 100644 --- a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:a3ec10ab61a65c03ca17e6460147a3178e222bd1c48964727ea9daae54717ebe", - "digest_compatibility_note": null, - "finished_at": 1783941481.68291, + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", "harness": "opencode", "model": { "display_name": "GPT-5.5", @@ -12,16 +10,18 @@ "provider_reasoning_effort": "xhigh" }, "normalized_score": 0.0, + "normalized_score_stddev": 0.0, "provider_reasoning_effort": "xhigh", - "source_run_id": "leaderboard/runs/article_suite/20260713T111604Z/gpt-5.5", + "raw_score": 0.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.0, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], "task": "simulation_heuristics_atari57_v1", - "task_digest": "sha256:a3ec10ab61a65c03ca17e6460147a3178e222bd1c48964727ea9daae54717ebe", - "token_usage": { - "final_input_tokens": 5376.0, - "final_output_tokens": 324.0, - "input_tokens": 5376.0, - "output_tokens": 324.0, - "total_tokens": 19012.0 - }, - "tool_calls": 13 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/score.json index 7788108..c2ca3ce 100644 --- a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/score.json @@ -1,15 +1,59 @@ { - "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", - "evaluation": null, - "interaction_budget": { - "completed_trajectories": 0, - "counted_env_steps": 0, - "planned_trajectories": 342, - "target_env_steps": 6840000000 - }, + "benchmark": "learning_beyond_gradients_article_suite", "normalized_score": 0.0, - "protocol_complete": false, + "normalized_score_stddev": 0.0, + "raw_score_imputation_count": 0, "reference_score": 0.8283015254994576, "score": 0.0, - "starter_score": 0.0 + "score_stddev": 0.0, + "starter_score": 0.0, + "task": "simulation_heuristics_atari57_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "trial": 1 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "trial": 2 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "trial": 3 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "trial": 4 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05", + "trial": 5 + } + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-01/metadata.json new file mode 100644 index 0000000..03fcbbb --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "digest_compatibility_note": null, + "finished_at": 1784074045.3527198, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "task": "simulation_heuristics_atari57_v1", + "task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "token_usage": { + "final_input_tokens": 5475.0, + "final_output_tokens": 286.0, + "input_tokens": 5475.0, + "output_tokens": 286.0, + "total_tokens": 20097.0 + }, + "tool_calls": 17, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-01/score.json new file mode 100644 index 0000000..7788108 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-01/score.json @@ -0,0 +1,15 @@ +{ + "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", + "evaluation": null, + "interaction_budget": { + "completed_trajectories": 0, + "counted_env_steps": 0, + "planned_trajectories": 342, + "target_env_steps": 6840000000 + }, + "normalized_score": 0.0, + "protocol_complete": false, + "reference_score": 0.8283015254994576, + "score": 0.0, + "starter_score": 0.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-02/metadata.json new file mode 100644 index 0000000..4fadd6e --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "digest_compatibility_note": null, + "finished_at": 1784076544.549837, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "task": "simulation_heuristics_atari57_v1", + "task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "token_usage": { + "final_input_tokens": 5698.0, + "final_output_tokens": 300.0, + "input_tokens": 5698.0, + "output_tokens": 300.0, + "total_tokens": 19310.0 + }, + "tool_calls": 14, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-02/score.json new file mode 100644 index 0000000..7788108 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-02/score.json @@ -0,0 +1,15 @@ +{ + "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", + "evaluation": null, + "interaction_budget": { + "completed_trajectories": 0, + "counted_env_steps": 0, + "planned_trajectories": 342, + "target_env_steps": 6840000000 + }, + "normalized_score": 0.0, + "protocol_complete": false, + "reference_score": 0.8283015254994576, + "score": 0.0, + "starter_score": 0.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-03/metadata.json new file mode 100644 index 0000000..dec7816 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "digest_compatibility_note": null, + "finished_at": 1784080509.825609, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "task": "simulation_heuristics_atari57_v1", + "task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "token_usage": { + "final_input_tokens": 5847.0, + "final_output_tokens": 283.0, + "input_tokens": 5847.0, + "output_tokens": 283.0, + "total_tokens": 19442.0 + }, + "tool_calls": 14, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-03/score.json new file mode 100644 index 0000000..7788108 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-03/score.json @@ -0,0 +1,15 @@ +{ + "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", + "evaluation": null, + "interaction_budget": { + "completed_trajectories": 0, + "counted_env_steps": 0, + "planned_trajectories": 342, + "target_env_steps": 6840000000 + }, + "normalized_score": 0.0, + "protocol_complete": false, + "reference_score": 0.8283015254994576, + "score": 0.0, + "starter_score": 0.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-04/metadata.json new file mode 100644 index 0000000..b7c2149 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-04/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "digest_compatibility_note": null, + "finished_at": 1784093477.189311, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "task": "simulation_heuristics_atari57_v1", + "task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "token_usage": { + "final_input_tokens": 5923.0, + "final_output_tokens": 258.0, + "input_tokens": 5923.0, + "output_tokens": 258.0, + "total_tokens": 20005.0 + }, + "tool_calls": 14, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-04/score.json new file mode 100644 index 0000000..7788108 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-04/score.json @@ -0,0 +1,15 @@ +{ + "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", + "evaluation": null, + "interaction_budget": { + "completed_trajectories": 0, + "counted_env_steps": 0, + "planned_trajectories": 342, + "target_env_steps": 6840000000 + }, + "normalized_score": 0.0, + "protocol_complete": false, + "reference_score": 0.8283015254994576, + "score": 0.0, + "starter_score": 0.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-05/metadata.json new file mode 100644 index 0000000..11dfb29 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-05/metadata.json @@ -0,0 +1,25 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "digest_compatibility_note": null, + "finished_at": 1784095422.450964, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05", + "task": "simulation_heuristics_atari57_v1", + "task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "token_usage": null, + "tool_calls": 14, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-05/score.json new file mode 100644 index 0000000..7788108 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/trial-05/score.json @@ -0,0 +1,15 @@ +{ + "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", + "evaluation": null, + "interaction_budget": { + "completed_trajectories": 0, + "counted_env_steps": 0, + "planned_trajectories": 342, + "target_env_steps": 6840000000 + }, + "normalized_score": 0.0, + "protocol_complete": false, + "reference_score": 0.8283015254994576, + "score": 0.0, + "starter_score": 0.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/metadata.json index 9c46b09..c192b1a 100644 --- a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:77215337f89f950f83b8f7436f799959c2ddccd4102552c6302b13d30cc57199", - "digest_compatibility_note": null, - "finished_at": 1783933771.613736, + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", "harness": "opencode", "model": { "display_name": "GPT-5.5", @@ -11,17 +9,19 @@ "provider": "azure", "provider_reasoning_effort": "xhigh" }, - "normalized_score": 95.597484277, + "normalized_score": 20.9224318658, + "normalized_score_stddev": 27.49546024414613, "provider_reasoning_effort": "xhigh", - "source_run_id": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "raw_score": 486.8, + "raw_score_imputation_count": 0, + "raw_score_stddev": 131.153345363357, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-e6fa3546aa", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], "task": "simulation_heuristics_breakout_ram_v1", - "task_digest": "sha256:77215337f89f950f83b8f7436f799959c2ddccd4102552c6302b13d30cc57199", - "token_usage": { - "final_input_tokens": 2395.0, - "final_output_tokens": 237.0, - "input_tokens": 2395.0, - "output_tokens": 237.0, - "total_tokens": 21064.0 - }, - "tool_calls": 26 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/score.json index 7867301..016ee15 100644 --- a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/score.json @@ -1,80 +1,59 @@ { - "normalized_score": 95.597484277, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": 20.9224318658, + "normalized_score_stddev": 27.49546024414613, + "raw_score_imputation_count": 0, "reference_score": 864.0, - "score": 843.0, + "score": 486.8, + "score_stddev": 131.153345363357, "starter_score": 387.0, - "suites": { - "hidden_nominal": { - "completion_rate": 0.0, - "episodes": [ - { - "final_lives": 2, - "invalid_action": false, - "length": 108000, - "mean_action_latency_ms": 0.004359432254676466, - "observation_mode": "ram", - "policy_error": null, - "return": 843.0, - "seed": 101, - "terminated": false, - "truncated": true, - "variant": "nominal", - "warmup_steps": 0 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 843.0, - "max_steps": 108000, - "mean_action_latency_ms": 0.004359432254676466, - "mean_return": 843.0, - "min_return": 843.0, - "observation_mode": "ram", - "policy_path": "submitted_artifact", - "score": 843.0 + "task": "simulation_heuristics_breakout_ram_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 5.870020964, + "observed_raw_score": 415.0, + "raw_score": 415.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "trial": 1 }, - "hidden_shifted_start": { - "completion_rate": 0.0, - "episodes": [ - { - "final_lives": 2, - "invalid_action": false, - "length": 108000, - "mean_action_latency_ms": 0.004456932001091816, - "observation_mode": "ram", - "policy_error": null, - "return": 843.0, - "seed": 211, - "terminated": false, - "truncated": true, - "variant": "left_6", - "warmup_steps": 6 - }, - { - "final_lives": 2, - "invalid_action": false, - "length": 108000, - "mean_action_latency_ms": 0.00464326129467399, - "observation_mode": "ram", - "policy_error": null, - "return": 843.0, - "seed": 211, - "terminated": false, - "truncated": true, - "variant": "right_6", - "warmup_steps": 6 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 843.0, - "max_steps": 108000, - "mean_action_latency_ms": 0.004550096647882904, - "mean_return": 843.0, - "min_return": 843.0, - "observation_mode": "ram", - "policy_path": "submitted_artifact", - "score": 843.0 + { + "normalized_score": 70.020964361, + "observed_raw_score": 721.0, + "raw_score": 721.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "trial": 2 + }, + { + "normalized_score": 10.062893082, + "observed_raw_score": 435.0, + "raw_score": 435.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "trial": 3 + }, + { + "normalized_score": 9.224318658, + "observed_raw_score": 431.0, + "raw_score": 431.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-e6fa3546aa", + "trial": 4 + }, + { + "normalized_score": 9.433962264, + "observed_raw_score": 432.0, + "raw_score": 432.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05", + "trial": 5 } - }, - "target_gap": -21.0, - "target_score": 864.0 + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-01/metadata.json new file mode 100644 index 0000000..61fcf5d --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "digest_compatibility_note": null, + "finished_at": 1784074045.3527198, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 5.870020964, + "observed_raw_score": 415.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 415.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "task": "simulation_heuristics_breakout_ram_v1", + "task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "token_usage": { + "final_input_tokens": 636.0, + "final_output_tokens": 164.0, + "input_tokens": 636.0, + "output_tokens": 164.0, + "total_tokens": 32544.0 + }, + "tool_calls": 30, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-01/score.json new file mode 100644 index 0000000..0f2266a --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-01/score.json @@ -0,0 +1,80 @@ +{ + "normalized_score": 5.870020964, + "reference_score": 864.0, + "score": 415.0, + "starter_score": 387.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 2, + "invalid_action": false, + "length": 108000, + "mean_action_latency_ms": 0.0034241528784925185, + "observation_mode": "ram", + "policy_error": null, + "return": 415.0, + "seed": 101, + "terminated": false, + "truncated": true, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 415.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.0034241528784925185, + "mean_return": 415.0, + "min_return": 415.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 415.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 2, + "invalid_action": false, + "length": 108000, + "mean_action_latency_ms": 0.0031579665718945086, + "observation_mode": "ram", + "policy_error": null, + "return": 415.0, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "left_6", + "warmup_steps": 6 + }, + { + "final_lives": 2, + "invalid_action": false, + "length": 108000, + "mean_action_latency_ms": 0.0031295226007286045, + "observation_mode": "ram", + "policy_error": null, + "return": 415.0, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 415.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.0031437445863115566, + "mean_return": 415.0, + "min_return": 415.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 415.0 + } + }, + "target_gap": -449.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-02/metadata.json new file mode 100644 index 0000000..80e1c0b --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "digest_compatibility_note": null, + "finished_at": 1784076544.549837, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 70.020964361, + "observed_raw_score": 721.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 721.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "task": "simulation_heuristics_breakout_ram_v1", + "task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "token_usage": { + "final_input_tokens": 1147.0, + "final_output_tokens": 202.0, + "input_tokens": 1147.0, + "output_tokens": 202.0, + "total_tokens": 18245.0 + }, + "tool_calls": 23, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-02/score.json new file mode 100644 index 0000000..3fcb35f --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-02/score.json @@ -0,0 +1,80 @@ +{ + "normalized_score": 70.020964361, + "reference_score": 864.0, + "score": 721.0, + "starter_score": 387.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 16972, + "mean_action_latency_ms": 0.01102048132613375, + "observation_mode": "ram", + "policy_error": null, + "return": 721.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 721.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.01102048132613375, + "mean_return": 721.0, + "min_return": 721.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 721.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 16966, + "mean_action_latency_ms": 0.010819149619800772, + "observation_mode": "ram", + "policy_error": null, + "return": 721.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "left_6", + "warmup_steps": 6 + }, + { + "final_lives": 0, + "invalid_action": false, + "length": 16966, + "mean_action_latency_ms": 0.01013604416318179, + "observation_mode": "ram", + "policy_error": null, + "return": 721.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 721.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.01047759689149128, + "mean_return": 721.0, + "min_return": 721.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 721.0 + } + }, + "target_gap": -143.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-03/metadata.json new file mode 100644 index 0000000..1ccdf1c --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "digest_compatibility_note": null, + "finished_at": 1784080509.825609, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 10.062893082, + "observed_raw_score": 435.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 435.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "task": "simulation_heuristics_breakout_ram_v1", + "task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "token_usage": { + "final_input_tokens": 655.0, + "final_output_tokens": 187.0, + "input_tokens": 655.0, + "output_tokens": 187.0, + "total_tokens": 18250.0 + }, + "tool_calls": 21, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-03/score.json new file mode 100644 index 0000000..b0d4d6e --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-03/score.json @@ -0,0 +1,80 @@ +{ + "normalized_score": 10.062893082, + "reference_score": 864.0, + "score": 435.0, + "starter_score": 387.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 12657, + "mean_action_latency_ms": 0.008401450276271214, + "observation_mode": "ram", + "policy_error": null, + "return": 435.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 435.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.008401450276271214, + "mean_return": 435.0, + "min_return": 435.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 435.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 12651, + "mean_action_latency_ms": 0.00870571632308363, + "observation_mode": "ram", + "policy_error": null, + "return": 435.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "left_6", + "warmup_steps": 6 + }, + { + "final_lives": 0, + "invalid_action": false, + "length": 12651, + "mean_action_latency_ms": 0.011082768084531566, + "observation_mode": "ram", + "policy_error": null, + "return": 435.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 435.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.009894242203807597, + "mean_return": 435.0, + "min_return": 435.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 435.0 + } + }, + "target_gap": -429.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-04/metadata.json new file mode 100644 index 0000000..7ecbf1d --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-04/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "digest_compatibility_note": null, + "finished_at": 1784098096.514206, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 9.224318658, + "observed_raw_score": 431.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 431.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-e6fa3546aa", + "task": "simulation_heuristics_breakout_ram_v1", + "task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "token_usage": { + "final_input_tokens": 396.0, + "final_output_tokens": 185.0, + "input_tokens": 396.0, + "output_tokens": 185.0, + "total_tokens": 31813.0 + }, + "tool_calls": 29, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-04/score.json new file mode 100644 index 0000000..7da200f --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-04/score.json @@ -0,0 +1,80 @@ +{ + "normalized_score": 9.224318658, + "reference_score": 864.0, + "score": 431.0, + "starter_score": 387.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 12418, + "mean_action_latency_ms": 0.009699299414641406, + "observation_mode": "ram", + "policy_error": null, + "return": 431.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 431.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.009699299414641406, + "mean_return": 431.0, + "min_return": 431.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 431.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 12412, + "mean_action_latency_ms": 0.009239044875120237, + "observation_mode": "ram", + "policy_error": null, + "return": 431.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "left_6", + "warmup_steps": 6 + }, + { + "final_lives": 0, + "invalid_action": false, + "length": 12412, + "mean_action_latency_ms": 0.010169134677882698, + "observation_mode": "ram", + "policy_error": null, + "return": 431.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 431.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.009704089776501466, + "mean_return": 431.0, + "min_return": 431.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 431.0 + } + }, + "target_gap": -433.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-05/metadata.json new file mode 100644 index 0000000..5daa07c --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-05/metadata.json @@ -0,0 +1,25 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "digest_compatibility_note": null, + "finished_at": 1784095422.450964, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 9.433962264, + "observed_raw_score": 432.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 432.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05", + "task": "simulation_heuristics_breakout_ram_v1", + "task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "token_usage": null, + "tool_calls": 36, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-05/score.json new file mode 100644 index 0000000..a39dcd2 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/trial-05/score.json @@ -0,0 +1,80 @@ +{ + "normalized_score": 9.433962264, + "reference_score": 864.0, + "score": 432.0, + "starter_score": 387.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 12130, + "mean_action_latency_ms": 0.011443472041242493, + "observation_mode": "ram", + "policy_error": null, + "return": 432.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 432.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.011443472041242493, + "mean_return": 432.0, + "min_return": 432.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 432.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 12124, + "mean_action_latency_ms": 0.011673750535897476, + "observation_mode": "ram", + "policy_error": null, + "return": 432.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "left_6", + "warmup_steps": 6 + }, + { + "final_lives": 0, + "invalid_action": false, + "length": 12124, + "mean_action_latency_ms": 0.011795816782656695, + "observation_mode": "ram", + "policy_error": null, + "return": 432.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 432.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.011734783659277086, + "mean_return": 432.0, + "min_return": 432.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 432.0 + } + }, + "target_gap": -432.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/metadata.json index 44a7093..8ad11aa 100644 --- a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:7e126380af5345a4bfa0c9d29e45e68b8f1aa20d5d655d12187744100b24441a", - "digest_compatibility_note": null, - "finished_at": 1783933771.613736, + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", "harness": "opencode", "model": { "display_name": "GPT-5.5", @@ -11,17 +9,19 @@ "provider": "azure", "provider_reasoning_effort": "xhigh" }, - "normalized_score": 70.758122744, + "normalized_score": 60.86642599300001, + "normalized_score_stddev": 22.118506348049813, "provider_reasoning_effort": "xhigh", - "source_run_id": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "raw_score": 647.2, + "raw_score_imputation_count": 0, + "raw_score_stddev": 122.53652516698847, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-885696cfa4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], "task": "simulation_heuristics_breakout_rgb_v1", - "task_digest": "sha256:7e126380af5345a4bfa0c9d29e45e68b8f1aa20d5d655d12187744100b24441a", - "token_usage": { - "final_input_tokens": 580.0, - "final_output_tokens": 197.0, - "input_tokens": 580.0, - "output_tokens": 197.0, - "total_tokens": 19209.0 - }, - "tool_calls": 25 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/score.json index 31e15c4..28986ca 100644 --- a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/score.json @@ -1,66 +1,59 @@ { - "normalized_score": 70.758122744, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": 60.86642599300001, + "normalized_score_stddev": 22.118506348049813, + "raw_score_imputation_count": 0, "reference_score": 864.0, - "score": 702.0, + "score": 647.2, + "score_stddev": 122.53652516698847, "starter_score": 310.0, - "suites": { - "hidden_nominal": { - "completion_rate": 0.0, - "episodes": [ - { - "final_lives": 0, - "invalid_action": false, - "length": 18811, - "mean_action_latency_ms": 0.4204832990456023, - "observation_mode": "rgb", - "policy_error": null, - "return": 702.0, - "seed": 101, - "terminated": true, - "truncated": false, - "variant": "nominal", - "warmup_steps": 0 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 702.0, - "max_steps": 30000, - "mean_action_latency_ms": 0.4204832990456023, - "mean_return": 702.0, - "min_return": 702.0, - "observation_mode": "rgb", - "policy_path": "submitted_artifact", - "score": 702.0 + "task": "simulation_heuristics_breakout_rgb_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 21.299638989, + "observed_raw_score": 428.0, + "raw_score": 428.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "trial": 1 }, - "hidden_shifted_start": { - "completion_rate": 0.0, - "episodes": [ - { - "final_lives": 0, - "invalid_action": false, - "length": 18805, - "mean_action_latency_ms": 0.42227563768678744, - "observation_mode": "rgb", - "policy_error": null, - "return": 702.0, - "seed": 211, - "terminated": true, - "truncated": false, - "variant": "right_6", - "warmup_steps": 6 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 702.0, - "max_steps": 30000, - "mean_action_latency_ms": 0.42227563768678744, - "mean_return": 702.0, - "min_return": 702.0, - "observation_mode": "rgb", - "policy_path": "submitted_artifact", - "score": 702.0 + { + "normalized_score": 70.758122744, + "observed_raw_score": 702.0, + "raw_score": 702.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "trial": 2 + }, + { + "normalized_score": 70.758122744, + "observed_raw_score": 702.0, + "raw_score": 702.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "trial": 3 + }, + { + "normalized_score": 70.758122744, + "observed_raw_score": 702.0, + "raw_score": 702.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-885696cfa4", + "trial": 4 + }, + { + "normalized_score": 70.758122744, + "observed_raw_score": 702.0, + "raw_score": 702.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05", + "trial": 5 } - }, - "target_gap": -162.0, - "target_score": 864.0 + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-01/metadata.json new file mode 100644 index 0000000..a29fd10 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "digest_compatibility_note": null, + "finished_at": 1784074045.3527198, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 21.299638989, + "observed_raw_score": 428.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 428.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "task": "simulation_heuristics_breakout_rgb_v1", + "task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "token_usage": { + "final_input_tokens": 1288.0, + "final_output_tokens": 168.0, + "input_tokens": 1288.0, + "output_tokens": 168.0, + "total_tokens": 17840.0 + }, + "tool_calls": 21, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-01/score.json new file mode 100644 index 0000000..2259765 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-01/score.json @@ -0,0 +1,66 @@ +{ + "normalized_score": 21.299638989, + "reference_score": 864.0, + "score": 428.0, + "starter_score": 310.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 14382, + "mean_action_latency_ms": 0.4408362879364866, + "observation_mode": "rgb", + "policy_error": null, + "return": 428.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 428.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.4408362879364866, + "mean_return": 428.0, + "min_return": 428.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 428.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 14376, + "mean_action_latency_ms": 0.4348243627143722, + "observation_mode": "rgb", + "policy_error": null, + "return": 428.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 428.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.4348243627143722, + "mean_return": 428.0, + "min_return": 428.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 428.0 + } + }, + "target_gap": -436.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-02/metadata.json new file mode 100644 index 0000000..4305d0f --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "digest_compatibility_note": null, + "finished_at": 1784076544.549837, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 70.758122744, + "observed_raw_score": 702.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 702.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "task": "simulation_heuristics_breakout_rgb_v1", + "task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "token_usage": { + "final_input_tokens": 1446.0, + "final_output_tokens": 195.0, + "input_tokens": 1446.0, + "output_tokens": 195.0, + "total_tokens": 19049.0 + }, + "tool_calls": 24, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-02/score.json new file mode 100644 index 0000000..97e96ca --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-02/score.json @@ -0,0 +1,66 @@ +{ + "normalized_score": 70.758122744, + "reference_score": 864.0, + "score": 702.0, + "starter_score": 310.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18811, + "mean_action_latency_ms": 0.4385001127637876, + "observation_mode": "rgb", + "policy_error": null, + "return": 702.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 702.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.4385001127637876, + "mean_return": 702.0, + "min_return": 702.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 702.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18805, + "mean_action_latency_ms": 0.45143717766695596, + "observation_mode": "rgb", + "policy_error": null, + "return": 702.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 702.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.45143717766695596, + "mean_return": 702.0, + "min_return": 702.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 702.0 + } + }, + "target_gap": -162.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-03/metadata.json new file mode 100644 index 0000000..50da221 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "digest_compatibility_note": null, + "finished_at": 1784080509.825609, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 70.758122744, + "observed_raw_score": 702.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 702.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "task": "simulation_heuristics_breakout_rgb_v1", + "task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "token_usage": { + "final_input_tokens": 871.0, + "final_output_tokens": 214.0, + "input_tokens": 871.0, + "output_tokens": 214.0, + "total_tokens": 17981.0 + }, + "tool_calls": 22, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-03/score.json new file mode 100644 index 0000000..706fef8 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-03/score.json @@ -0,0 +1,66 @@ +{ + "normalized_score": 70.758122744, + "reference_score": 864.0, + "score": 702.0, + "starter_score": 310.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18811, + "mean_action_latency_ms": 0.5747133783865797, + "observation_mode": "rgb", + "policy_error": null, + "return": 702.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 702.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.5747133783865797, + "mean_return": 702.0, + "min_return": 702.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 702.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18805, + "mean_action_latency_ms": 0.5044946598066862, + "observation_mode": "rgb", + "policy_error": null, + "return": 702.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 702.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.5044946598066862, + "mean_return": 702.0, + "min_return": 702.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 702.0 + } + }, + "target_gap": -162.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-04/metadata.json new file mode 100644 index 0000000..3ff7d6c --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-04/metadata.json @@ -0,0 +1,25 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "digest_compatibility_note": null, + "finished_at": 1784095422.450964, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 70.758122744, + "observed_raw_score": 702.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 702.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-885696cfa4", + "task": "simulation_heuristics_breakout_rgb_v1", + "task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "token_usage": null, + "tool_calls": 27, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-04/score.json new file mode 100644 index 0000000..57b87dd --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-04/score.json @@ -0,0 +1,66 @@ +{ + "normalized_score": 70.758122744, + "reference_score": 864.0, + "score": 702.0, + "starter_score": 310.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18811, + "mean_action_latency_ms": 0.5009699954273192, + "observation_mode": "rgb", + "policy_error": null, + "return": 702.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 702.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.5009699954273192, + "mean_return": 702.0, + "min_return": 702.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 702.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18805, + "mean_action_latency_ms": 0.507553582934267, + "observation_mode": "rgb", + "policy_error": null, + "return": 702.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 702.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.507553582934267, + "mean_return": 702.0, + "min_return": 702.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 702.0 + } + }, + "target_gap": -162.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-05/metadata.json new file mode 100644 index 0000000..670c72d --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-05/metadata.json @@ -0,0 +1,25 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "digest_compatibility_note": null, + "finished_at": 1784095422.450964, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 70.758122744, + "observed_raw_score": 702.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 702.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05", + "task": "simulation_heuristics_breakout_rgb_v1", + "task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "token_usage": null, + "tool_calls": 17, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-05/score.json new file mode 100644 index 0000000..dfc4a64 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/trial-05/score.json @@ -0,0 +1,66 @@ +{ + "normalized_score": 70.758122744, + "reference_score": 864.0, + "score": 702.0, + "starter_score": 310.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18811, + "mean_action_latency_ms": 0.551382433317107, + "observation_mode": "rgb", + "policy_error": null, + "return": 702.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 702.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.551382433317107, + "mean_return": 702.0, + "min_return": 702.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 702.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18805, + "mean_action_latency_ms": 0.5641856102772578, + "observation_mode": "rgb", + "policy_error": null, + "return": 702.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 702.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.5641856102772578, + "mean_return": 702.0, + "min_return": 702.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 702.0 + } + }, + "target_gap": -162.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/metadata.json index 5affbbc..5a86d6c 100644 --- a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:c7e3b5e466431f4908c1e96cc7de80191f7a7c5ab5a63b3eea4d708f7f447fbf", - "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", - "finished_at": 1783941357.9386308, + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", "harness": "opencode", "model": { "display_name": "GPT-5.5", @@ -11,17 +9,19 @@ "provider": "azure", "provider_reasoning_effort": "xhigh" }, - "normalized_score": 2.73755349, + "normalized_score": 9.5689381094, + "normalized_score_stddev": 9.330884765209642, "provider_reasoning_effort": "xhigh", - "source_run_id": "leaderboard/runs/article_suite/20260713T103810Z/gpt-5.5", + "raw_score": 4717.6162449605235, + "raw_score_imputation_count": 0, + "raw_score_stddev": 698.5070322948952, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01/simulation_heuristics_halfcheetah_v1", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-e6fa3546aa", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05/tasks-81ec8d98a4" + ], "task": "simulation_heuristics_halfcheetah_v1", - "task_digest": "sha256:80c439f53e4ab964f9d7443cd7fb8f25cf6645a0bc288b0496b871c1800ebe78", - "token_usage": { - "final_input_tokens": 1233.0, - "final_output_tokens": 233.0, - "input_tokens": 1233.0, - "output_tokens": 233.0, - "total_tokens": 18362.0 - }, - "tool_calls": 21 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/score.json index 36ef619..943293a 100644 --- a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/score.json @@ -1,98 +1,59 @@ { - "hidden_nominal": { - "episodes": [ - { - "control_cost": 330.4261677414179, - "forward_reward": 4605.109353712759, - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.11520281340926886, - "policy_error": null, - "return": 4274.683185971341, - "seed": 211, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": 230.2448098566455, - "x_velocity": 4.320877724728021 - }, - { - "control_cost": 331.2794958502054, - "forward_reward": 4739.932284020166, - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.11275607440620661, - "policy_error": null, - "return": 4408.652788169957, - "seed": 307, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": 237.02319826694526, - "x_velocity": 4.455080651779344 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 4408.652788169957, - "max_steps": 1000, - "mean_action_latency_ms": 0.11397944390773773, - "mean_control_cost": 330.85283179581165, - "mean_forward_reward": 4672.520818866462, - "mean_return": 4341.667987070648, - "mean_x_position": 233.6340040617954, - "min_return": 4274.683185971341, - "policy_path": "submitted_artifact", - "score": 4341.667987070648 - }, - "hidden_nominal_weight": 0.7, - "hidden_robustness": { - "episodes": [ - { - "control_cost": 328.19089092314243, - "forward_reward": 4836.63682109343, - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.11922981264069676, - "policy_error": null, - "return": 4508.4459301702855, - "seed": 503, - "terminated": false, - "truncated": true, - "variant": "light_low_friction", - "x_position": 241.81416267727866, - "x_velocity": 4.403407883231224 - }, - { - "control_cost": 331.44420075416565, - "forward_reward": 3603.354261605028, - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 0.11692689219489694, - "policy_error": null, - "return": 3271.9100608508643, - "seed": 503, - "terminated": false, - "truncated": true, - "variant": "heavy_weak_actuators", - "x_position": 180.15003470285865, - "x_velocity": 3.3799933403413216 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 4508.4459301702855, - "max_steps": 1000, - "mean_action_latency_ms": 0.11807835241779685, - "mean_control_cost": 329.81754583865404, - "mean_forward_reward": 4219.995541349229, - "mean_return": 3890.177995510575, - "mean_x_position": 210.98209869006865, - "min_return": 3271.9100608508643, - "policy_path": "submitted_artifact", - "score": 3890.177995510575 - }, - "hidden_robustness_weight": 0.3, - "normalized_score": 2.73755349, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": 9.5689381094, + "normalized_score_stddev": 9.330884765209642, + "raw_score_imputation_count": 0, "reference_score": 11487.256451471967, - "score": 4206.220989602626, - "starter_score": 4001.2886158917113 + "score": 4717.6162449605235, + "score_stddev": 698.5070322948952, + "starter_score": 4001.2886158917113, + "task": "simulation_heuristics_halfcheetah_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 0.0, + "observed_raw_score": 4001.2886158917113, + "raw_score": 4001.2886158917113, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01/simulation_heuristics_halfcheetah_v1", + "trial": 1 + }, + { + "normalized_score": 21.272433217, + "observed_raw_score": 5593.736124362126, + "raw_score": 5593.736124362126, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "trial": 2 + }, + { + "normalized_score": 12.773449459, + "observed_raw_score": 4957.504933873245, + "raw_score": 4957.504933873245, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "trial": 3 + }, + { + "normalized_score": 13.798807871, + "observed_raw_score": 5034.262934783822, + "raw_score": 5034.262934783822, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-e6fa3546aa", + "trial": 4 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 4001.2886158917113, + "raw_score": 4001.2886158917113, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05/tasks-81ec8d98a4", + "trial": 5 + } + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-01/metadata.json new file mode 100644 index 0000000..a09a064 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "digest_compatibility_note": null, + "finished_at": 1784076184.464024, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 4001.2886158917113, + "provider_reasoning_effort": "xhigh", + "raw_score": 4001.2886158917113, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01/simulation_heuristics_halfcheetah_v1", + "task": "simulation_heuristics_halfcheetah_v1", + "task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "token_usage": { + "final_input_tokens": 1006.0, + "final_output_tokens": 269.0, + "input_tokens": 1006.0, + "output_tokens": 269.0, + "total_tokens": 17659.0 + }, + "tool_calls": 16, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-01/score.json new file mode 100644 index 0000000..3a2a25f --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-01/score.json @@ -0,0 +1,98 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "control_cost": 373.2381326109171, + "forward_reward": 5105.140023715441, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.06185706681571901, + "policy_error": null, + "return": 4731.901891104524, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 255.24634335677956, + "x_velocity": 5.862027122819839 + }, + { + "control_cost": 374.2015422284603, + "forward_reward": 5151.263870973642, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.061849022982642055, + "policy_error": null, + "return": 4777.062328745182, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 257.5897776146191, + "x_velocity": 5.590444847813859 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 4777.062328745182, + "max_steps": 1000, + "mean_action_latency_ms": 0.06185304489918053, + "mean_control_cost": 373.7198374196887, + "mean_forward_reward": 5128.201947344542, + "mean_return": 4754.482109924853, + "mean_x_position": 256.41806048569936, + "min_return": 4731.901891104524, + "policy_path": "submitted_artifact", + "score": 4754.482109924853 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "control_cost": 375.5750959441066, + "forward_reward": 1358.951199769031, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.06245467532426119, + "policy_error": null, + "return": 983.3761038249263, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 67.92988161105906, + "x_velocity": -0.39402351510640576 + }, + { + "control_cost": 372.8621143847704, + "forward_reward": 3877.160270188608, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.06294826534576714, + "policy_error": null, + "return": 3504.298155803837, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 193.8403351320377, + "x_velocity": 3.7236471583694275 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 3504.298155803837, + "max_steps": 1000, + "mean_action_latency_ms": 0.06270147033501416, + "mean_control_cost": 374.2186051644385, + "mean_forward_reward": 2618.0557349788196, + "mean_return": 2243.8371298143816, + "mean_x_position": 130.88510837154837, + "min_return": 983.3761038249263, + "policy_path": "submitted_artifact", + "score": 2243.8371298143816 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": 0.0, + "reference_score": 11487.256451471967, + "score": 4001.2886158917113, + "starter_score": 4001.2886158917113 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-02/metadata.json new file mode 100644 index 0000000..343d4a5 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "digest_compatibility_note": null, + "finished_at": 1784076544.549837, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 21.272433217, + "observed_raw_score": 5593.736124362126, + "provider_reasoning_effort": "xhigh", + "raw_score": 5593.736124362126, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "task": "simulation_heuristics_halfcheetah_v1", + "task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "token_usage": { + "final_input_tokens": 1301.0, + "final_output_tokens": 237.0, + "input_tokens": 1301.0, + "output_tokens": 237.0, + "total_tokens": 18946.0 + }, + "tool_calls": 19, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-02/score.json new file mode 100644 index 0000000..24612ca --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-02/score.json @@ -0,0 +1,98 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "control_cost": 389.9041979163885, + "forward_reward": 6129.455169240355, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 18.366634750040248, + "policy_error": null, + "return": 5739.550971323966, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 306.4621006330258, + "x_velocity": 5.48738598103033 + }, + { + "control_cost": 387.7622296139598, + "forward_reward": 6073.480018252979, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 18.500966612016782, + "policy_error": null, + "return": 5685.717788639015, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 303.70058497858537, + "x_velocity": 5.556400990860339 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 5739.550971323966, + "max_steps": 1000, + "mean_action_latency_ms": 18.433800681028515, + "mean_control_cost": 388.83321376517415, + "mean_forward_reward": 6101.467593746667, + "mean_return": 5712.634379981491, + "mean_x_position": 305.08134280580555, + "min_return": 5685.717788639015, + "policy_path": "submitted_artifact", + "score": 5712.634379981491 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "control_cost": 382.8844804689288, + "forward_reward": 6753.125424069593, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 18.9403802130837, + "policy_error": null, + "return": 6370.240943600667, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 337.63859282608706, + "x_velocity": 5.959405323121231 + }, + { + "control_cost": 386.8188530355692, + "forward_reward": 4649.191631935449, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 18.887637481559068, + "policy_error": null, + "return": 4262.372778899883, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 232.44190321937975, + "x_velocity": 4.405383592115184 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 6370.240943600667, + "max_steps": 1000, + "mean_action_latency_ms": 18.914008847321384, + "mean_control_cost": 384.851666752249, + "mean_forward_reward": 5701.158528002521, + "mean_return": 5316.306861250276, + "mean_x_position": 285.0402480227334, + "min_return": 4262.372778899883, + "policy_path": "submitted_artifact", + "score": 5316.306861250276 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": 21.272433217, + "reference_score": 11487.256451471967, + "score": 5593.736124362126, + "starter_score": 4001.2886158917113 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-03/metadata.json new file mode 100644 index 0000000..acad8e3 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "digest_compatibility_note": null, + "finished_at": 1784080509.825609, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 12.773449459, + "observed_raw_score": 4957.504933873245, + "provider_reasoning_effort": "xhigh", + "raw_score": 4957.504933873245, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "task": "simulation_heuristics_halfcheetah_v1", + "task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "token_usage": { + "final_input_tokens": 1235.0, + "final_output_tokens": 302.0, + "input_tokens": 1235.0, + "output_tokens": 302.0, + "total_tokens": 23553.0 + }, + "tool_calls": 27, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-03/score.json new file mode 100644 index 0000000..1d74777 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-03/score.json @@ -0,0 +1,98 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "control_cost": 391.0229788720608, + "forward_reward": 5487.729589842253, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 72.73117655958049, + "policy_error": null, + "return": 5096.706610970185, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 274.3758216631199, + "x_velocity": 5.380790936589212 + }, + { + "control_cost": 389.5616500079632, + "forward_reward": 5502.412911971809, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 71.79770914255641, + "policy_error": null, + "return": 5112.851261963848, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 275.1472296645273, + "x_velocity": 5.932817167795463 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 5112.851261963848, + "max_steps": 1000, + "mean_action_latency_ms": 72.26444285106845, + "mean_control_cost": 390.292314440012, + "mean_forward_reward": 5495.071250907031, + "mean_return": 5104.778936467016, + "mean_x_position": 274.7615256638236, + "min_return": 5096.706610970185, + "policy_path": "submitted_artifact", + "score": 5104.778936467016 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "control_cost": 382.84815073013306, + "forward_reward": 5729.417378280265, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 71.2625126359053, + "policy_error": null, + "return": 5346.569227550136, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 286.45319053662064, + "x_velocity": 5.490495912032788 + }, + { + "control_cost": 391.19067096710205, + "forward_reward": 4272.352632392531, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 73.40940277907066, + "policy_error": null, + "return": 3881.1619614254264, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 213.59995324223374, + "x_velocity": 4.58688107957856 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 5346.569227550136, + "max_steps": 1000, + "mean_action_latency_ms": 72.33595770748798, + "mean_control_cost": 387.01941084861755, + "mean_forward_reward": 5000.8850053363985, + "mean_return": 4613.865594487781, + "mean_x_position": 250.0265718894272, + "min_return": 3881.1619614254264, + "policy_path": "submitted_artifact", + "score": 4613.865594487781 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": 12.773449459, + "reference_score": 11487.256451471967, + "score": 4957.504933873245, + "starter_score": 4001.2886158917113 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-04/metadata.json new file mode 100644 index 0000000..3d078f1 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-04/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "digest_compatibility_note": null, + "finished_at": 1784098096.514206, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 13.798807871, + "observed_raw_score": 5034.262934783822, + "provider_reasoning_effort": "xhigh", + "raw_score": 5034.262934783822, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-e6fa3546aa", + "task": "simulation_heuristics_halfcheetah_v1", + "task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "token_usage": { + "final_input_tokens": 574.0, + "final_output_tokens": 244.0, + "input_tokens": 574.0, + "output_tokens": 244.0, + "total_tokens": 16690.0 + }, + "tool_calls": 15, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-04/score.json new file mode 100644 index 0000000..b534dab --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-04/score.json @@ -0,0 +1,98 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "control_cost": 389.2573233023286, + "forward_reward": 5493.9502136172605, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 43.46089743619086, + "policy_error": null, + "return": 5104.692890314932, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 274.68685285187087, + "x_velocity": 6.162579444455217 + }, + { + "control_cost": 392.1595486998558, + "forward_reward": 5511.954288018311, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 43.07861266256077, + "policy_error": null, + "return": 5119.7947393184495, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 275.62429846685274, + "x_velocity": 6.0135998260103385 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 5119.7947393184495, + "max_steps": 1000, + "mean_action_latency_ms": 43.26975504937582, + "mean_control_cost": 390.7084360010922, + "mean_forward_reward": 5502.952250817785, + "mean_return": 5112.243814816691, + "mean_x_position": 275.15557565936183, + "min_return": 5104.692890314932, + "policy_path": "submitted_artifact", + "score": 5112.243814816691 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "control_cost": 384.98944222927094, + "forward_reward": 5954.823330647336, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 43.20365892414702, + "policy_error": null, + "return": 5569.833888418068, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 297.72348815497446, + "x_velocity": 6.496560809741385 + }, + { + "control_cost": 401.1867419555783, + "forward_reward": 4535.967949618429, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 44.20063610264333, + "policy_error": null, + "return": 4134.7812076628525, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 226.78071910352855, + "x_velocity": 5.22504840646882 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 5569.833888418068, + "max_steps": 1000, + "mean_action_latency_ms": 43.702147513395175, + "mean_control_cost": 393.08809209242463, + "mean_forward_reward": 5245.395640132882, + "mean_return": 4852.3075480404605, + "mean_x_position": 262.2521036292515, + "min_return": 4134.7812076628525, + "policy_path": "submitted_artifact", + "score": 4852.3075480404605 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": 13.798807871, + "reference_score": 11487.256451471967, + "score": 5034.262934783822, + "starter_score": 4001.2886158917113 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-05/metadata.json new file mode 100644 index 0000000..95e8c94 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-05/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "digest_compatibility_note": null, + "finished_at": 1784097871.7138891, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 4001.2886158917113, + "provider_reasoning_effort": "xhigh", + "raw_score": 4001.2886158917113, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05/tasks-81ec8d98a4", + "task": "simulation_heuristics_halfcheetah_v1", + "task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "token_usage": { + "final_input_tokens": 1336.0, + "final_output_tokens": 257.0, + "input_tokens": 1336.0, + "output_tokens": 257.0, + "total_tokens": 17977.0 + }, + "tool_calls": 17, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-05/score.json new file mode 100644 index 0000000..8e6dbf4 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/trial-05/score.json @@ -0,0 +1,98 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "control_cost": 373.2381326109171, + "forward_reward": 5105.140023715441, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.021765077952295542, + "policy_error": null, + "return": 4731.901891104524, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 255.24634335677956, + "x_velocity": 5.862027122819839 + }, + { + "control_cost": 374.2015422284603, + "forward_reward": 5151.263870973642, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.02172576985321939, + "policy_error": null, + "return": 4777.062328745182, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 257.5897776146191, + "x_velocity": 5.590444847813859 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 4777.062328745182, + "max_steps": 1000, + "mean_action_latency_ms": 0.021745423902757466, + "mean_control_cost": 373.7198374196887, + "mean_forward_reward": 5128.201947344542, + "mean_return": 4754.482109924853, + "mean_x_position": 256.41806048569936, + "min_return": 4731.901891104524, + "policy_path": "submitted_artifact", + "score": 4754.482109924853 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "control_cost": 375.5750959441066, + "forward_reward": 1358.951199769031, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.021822939394041896, + "policy_error": null, + "return": 983.3761038249263, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 67.92988161105906, + "x_velocity": -0.39402351510640576 + }, + { + "control_cost": 372.8621143847704, + "forward_reward": 3877.160270188608, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.021858201827853918, + "policy_error": null, + "return": 3504.298155803837, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 193.8403351320377, + "x_velocity": 3.7236471583694275 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 3504.298155803837, + "max_steps": 1000, + "mean_action_latency_ms": 0.021840570610947907, + "mean_control_cost": 374.2186051644385, + "mean_forward_reward": 2618.0557349788196, + "mean_return": 2243.8371298143816, + "mean_x_position": 130.88510837154837, + "min_return": 983.3761038249263, + "policy_path": "submitted_artifact", + "score": 2243.8371298143816 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": 0.0, + "reference_score": 11487.256451471967, + "score": 4001.2886158917113, + "starter_score": 4001.2886158917113 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/metadata.json index b30e569..12dcf3c 100644 --- a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:33650c821735e31eea00f2ba46ae09b5d19134820c2ec005e8e96d92cb0b66d0", - "digest_compatibility_note": null, - "finished_at": 1783933771.613736, + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", "harness": "opencode", "model": { "display_name": "GPT-5.5", @@ -11,17 +9,19 @@ "provider": "azure", "provider_reasoning_effort": "xhigh" }, - "normalized_score": 100.0, + "normalized_score": 0.0, + "normalized_score_stddev": 0.0, "provider_reasoning_effort": "xhigh", - "source_run_id": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "raw_score": 0.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.0, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], "task": "simulation_heuristics_montezuma_v1", - "task_digest": "sha256:33650c821735e31eea00f2ba46ae09b5d19134820c2ec005e8e96d92cb0b66d0", - "token_usage": { - "final_input_tokens": 2141.0, - "final_output_tokens": 318.0, - "input_tokens": 2141.0, - "output_tokens": 318.0, - "total_tokens": 43419.0 - }, - "tool_calls": 60 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/score.json index 9fda219..7057b63 100644 --- a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/score.json @@ -1,216 +1,59 @@ { - "normalized_score": 100.0, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": 0.0, + "normalized_score_stddev": 0.0, + "raw_score_imputation_count": 0, "reference_score": 400.0, - "score": 400.0, + "score": 0.0, + "score_stddev": 0.0, "starter_score": 0.0, - "suites": { - "hidden_seeds": { - "evaluation": { - "capped_mean_score": 400.0, - "episodes": [ - { - "bootstrap_steps": 0, - "invalid_action": false, - "length": 1769, - "mean_action_latency_ms": 0.003403249586821276, - "policy_error": null, - "policy_steps": 1769, - "pre_policy_noops": 0, - "return": 400.0, - "seed": 10007, - "target_reached": true, - "terminated": true, - "truncated": false, - "variant": "canonical" - }, - { - "bootstrap_steps": 0, - "invalid_action": false, - "length": 1769, - "mean_action_latency_ms": 0.003197897434099843, - "policy_error": null, - "policy_steps": 1769, - "pre_policy_noops": 0, - "return": 400.0, - "seed": 10103, - "target_reached": true, - "terminated": true, - "truncated": false, - "variant": "canonical" - } - ], - "invalid_episode_rate": 0.0, - "max_steps": 2600, - "mean_action_latency_ms": 0.0033005735104605595, - "mean_return": 400.0, - "policy_path": "submitted_artifact", - "recovery_success_rate": 0.0, - "score": 400.0, - "target_score": 400.0, - "target_success_rate": 1.0 - }, - "score": 400.0, - "weight": 0.1 + "task": "simulation_heuristics_montezuma_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "trial": 1 }, - "noop_variants": { - "evaluation": { - "capped_mean_score": 400.0, - "episodes": [ - { - "bootstrap_steps": 0, - "invalid_action": false, - "length": 1773, - "mean_action_latency_ms": 0.003588716315347596, - "policy_error": null, - "policy_steps": 1770, - "pre_policy_noops": 3, - "return": 400.0, - "seed": 10001, - "target_reached": true, - "terminated": true, - "truncated": false, - "variant": "startup_noop_3" - }, - { - "bootstrap_steps": 0, - "invalid_action": false, - "length": 1769, - "mean_action_latency_ms": 0.0032576125598849904, - "policy_error": null, - "policy_steps": 1756, - "pre_policy_noops": 13, - "return": 400.0, - "seed": 10001, - "target_reached": true, - "terminated": true, - "truncated": false, - "variant": "startup_noop_13" - } - ], - "invalid_episode_rate": 0.0, - "max_steps": 2600, - "mean_action_latency_ms": 0.0034231644376162933, - "mean_return": 400.0, - "policy_path": "submitted_artifact", - "recovery_success_rate": 1.0, - "score": 400.0, - "target_score": 400.0, - "target_success_rate": 1.0 - }, - "score": 400.0, - "weight": 0.1 + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "trial": 2 }, - "recovery": { - "evaluation": { - "capped_mean_score": 400.0, - "episodes": [ - { - "bootstrap_steps": 512, - "invalid_action": false, - "length": 1773, - "mean_action_latency_ms": 0.0038068125123719705, - "policy_error": null, - "policy_steps": 1257, - "pre_policy_noops": 4, - "return": 400.0, - "seed": 10001, - "target_reached": true, - "terminated": true, - "truncated": false, - "variant": "recover_512_noop_4" - }, - { - "bootstrap_steps": 512, - "invalid_action": false, - "length": 1769, - "mean_action_latency_ms": 0.0031122399435890877, - "policy_error": null, - "policy_steps": 1249, - "pre_policy_noops": 8, - "return": 400.0, - "seed": 10001, - "target_reached": true, - "terminated": true, - "truncated": false, - "variant": "recover_512_noop_8" - }, - { - "bootstrap_steps": 768, - "invalid_action": false, - "length": 1773, - "mean_action_latency_ms": 0.005221254343644215, - "policy_error": null, - "policy_steps": 1003, - "pre_policy_noops": 2, - "return": 400.0, - "seed": 10001, - "target_reached": true, - "terminated": true, - "truncated": false, - "variant": "recover_768_noop_2" - }, - { - "bootstrap_steps": 900, - "invalid_action": false, - "length": 2173, - "mean_action_latency_ms": 0.002957832211597829, - "policy_error": null, - "policy_steps": 1273, - "pre_policy_noops": 0, - "return": 400.0, - "seed": 10001, - "target_reached": true, - "terminated": true, - "truncated": false, - "variant": "recover_900" - } - ], - "invalid_episode_rate": 0.0, - "max_steps": 2600, - "mean_action_latency_ms": 0.003774534752800776, - "mean_return": 400.0, - "policy_path": "submitted_artifact", - "recovery_success_rate": 1.0, - "score": 400.0, - "target_score": 400.0, - "target_success_rate": 1.0 - }, - "score": 400.0, - "weight": 0.55 + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "trial": 3 }, - "reproduction": { - "evaluation": { - "capped_mean_score": 400.0, - "episodes": [ - { - "bootstrap_steps": 0, - "invalid_action": false, - "length": 1769, - "mean_action_latency_ms": 0.0035391858299281247, - "policy_error": null, - "policy_steps": 1769, - "pre_policy_noops": 0, - "return": 400.0, - "seed": 10001, - "target_reached": true, - "terminated": true, - "truncated": false, - "variant": "canonical" - } - ], - "invalid_episode_rate": 0.0, - "max_steps": 2600, - "mean_action_latency_ms": 0.0035391858299281247, - "mean_return": 400.0, - "policy_path": "submitted_artifact", - "recovery_success_rate": 0.0, - "score": 400.0, - "target_score": 400.0, - "target_success_rate": 1.0 - }, - "score": 400.0, - "weight": 0.25 + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "trial": 4 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05", + "trial": 5 } - }, - "target_score": 400.0 + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-01/metadata.json new file mode 100644 index 0000000..f711341 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "digest_compatibility_note": null, + "finished_at": 1784074045.3527198, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "task": "simulation_heuristics_montezuma_v1", + "task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "token_usage": { + "final_input_tokens": 923.0, + "final_output_tokens": 177.0, + "input_tokens": 923.0, + "output_tokens": 177.0, + "total_tokens": 16460.0 + }, + "tool_calls": 22, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-01/score.json new file mode 100644 index 0000000..947de2d --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-01/score.json @@ -0,0 +1,216 @@ +{ + "normalized_score": 0.0, + "reference_score": 400.0, + "score": 0.0, + "starter_score": 0.0, + "suites": { + "hidden_seeds": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 545, + "mean_action_latency_ms": 0.27821460405552606, + "policy_error": null, + "policy_steps": 545, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10007, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "canonical" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 545, + "mean_action_latency_ms": 0.41895271407081447, + "policy_error": null, + "policy_steps": 545, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10103, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.34858365906317024, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "noop_variants": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 545, + "mean_action_latency_ms": 0.4387473261389345, + "policy_error": null, + "policy_steps": 542, + "pre_policy_noops": 3, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "startup_noop_3" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.2953987347216255, + "policy_error": null, + "policy_steps": 2587, + "pre_policy_noops": 13, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_13" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.36707303043028, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "recovery": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 1021, + "mean_action_latency_ms": 0.22750515675190652, + "policy_error": null, + "policy_steps": 505, + "pre_policy_noops": 4, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "recover_512_noop_4" + }, + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 1025, + "mean_action_latency_ms": 0.42445713857022843, + "policy_error": null, + "policy_steps": 505, + "pre_policy_noops": 8, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "recover_512_noop_8" + }, + { + "bootstrap_steps": 768, + "invalid_action": false, + "length": 1237, + "mean_action_latency_ms": 0.43852114800674885, + "policy_error": null, + "policy_steps": 467, + "pre_policy_noops": 2, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "recover_768_noop_2" + }, + { + "bootstrap_steps": 900, + "invalid_action": false, + "length": 1365, + "mean_action_latency_ms": 0.4381000149434292, + "policy_error": null, + "policy_steps": 465, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "recover_900" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.3821458645680782, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.55 + }, + "reproduction": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 545, + "mean_action_latency_ms": 0.44048703788275567, + "policy_error": null, + "policy_steps": 545, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.44048703788275567, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.25 + } + }, + "target_score": 400.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-02/metadata.json new file mode 100644 index 0000000..a919819 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "digest_compatibility_note": null, + "finished_at": 1784076544.549837, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "task": "simulation_heuristics_montezuma_v1", + "task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "token_usage": { + "final_input_tokens": 1109.0, + "final_output_tokens": 169.0, + "input_tokens": 1109.0, + "output_tokens": 169.0, + "total_tokens": 18174.0 + }, + "tool_calls": 20, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-02/score.json new file mode 100644 index 0000000..96cc5a1 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-02/score.json @@ -0,0 +1,216 @@ +{ + "normalized_score": 0.0, + "reference_score": 400.0, + "score": 0.0, + "starter_score": 0.0, + "suites": { + "hidden_seeds": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 901, + "mean_action_latency_ms": 0.15087826895867615, + "policy_error": null, + "policy_steps": 901, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10007, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "canonical" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 901, + "mean_action_latency_ms": 0.13973402401037538, + "policy_error": null, + "policy_steps": 901, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10103, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.14530614648452578, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "noop_variants": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 741, + "mean_action_latency_ms": 0.23926611861017177, + "policy_error": null, + "policy_steps": 738, + "pre_policy_noops": 3, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "startup_noop_3" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 749, + "mean_action_latency_ms": 0.25171598352481733, + "policy_error": null, + "policy_steps": 736, + "pre_policy_noops": 13, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "startup_noop_13" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.24549105106749455, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "recovery": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 1185, + "mean_action_latency_ms": 0.1082429693669348, + "policy_error": null, + "policy_steps": 669, + "pre_policy_noops": 4, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "recover_512_noop_4" + }, + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 1185, + "mean_action_latency_ms": 0.11131790102480497, + "policy_error": null, + "policy_steps": 665, + "pre_policy_noops": 8, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "recover_512_noop_8" + }, + { + "bootstrap_steps": 768, + "invalid_action": false, + "length": 1293, + "mean_action_latency_ms": 0.19445866844964074, + "policy_error": null, + "policy_steps": 523, + "pre_policy_noops": 2, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "recover_768_noop_2" + }, + { + "bootstrap_steps": 900, + "invalid_action": false, + "length": 1365, + "mean_action_latency_ms": 0.25221905270491235, + "policy_error": null, + "policy_steps": 465, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "recover_900" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.16655964788657324, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.55 + }, + "reproduction": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 901, + "mean_action_latency_ms": 0.13561565795919542, + "policy_error": null, + "policy_steps": 901, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.13561565795919542, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.25 + } + }, + "target_score": 400.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-03/metadata.json new file mode 100644 index 0000000..2101b42 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "digest_compatibility_note": null, + "finished_at": 1784080509.825609, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "task": "simulation_heuristics_montezuma_v1", + "task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "token_usage": { + "final_input_tokens": 1090.0, + "final_output_tokens": 181.0, + "input_tokens": 1090.0, + "output_tokens": 181.0, + "total_tokens": 24311.0 + }, + "tool_calls": 23, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-03/score.json new file mode 100644 index 0000000..c49e4a4 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-03/score.json @@ -0,0 +1,216 @@ +{ + "normalized_score": 0.0, + "reference_score": 400.0, + "score": 0.0, + "starter_score": 0.0, + "suites": { + "hidden_seeds": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.09567196051088664, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10007, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.10887690354138613, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10103, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.10227443202613638, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "noop_variants": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.09863816058599563, + "policy_error": null, + "policy_steps": 2597, + "pre_policy_noops": 3, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_3" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.10525766500190137, + "policy_error": null, + "policy_steps": 2587, + "pre_policy_noops": 13, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_13" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.10194791279394849, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "recovery": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.09369574649877946, + "policy_error": null, + "policy_steps": 2084, + "pre_policy_noops": 4, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_4" + }, + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.09461750965923645, + "policy_error": null, + "policy_steps": 2080, + "pre_policy_noops": 8, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_8" + }, + { + "bootstrap_steps": 768, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.09385450040829964, + "policy_error": null, + "policy_steps": 1830, + "pre_policy_noops": 2, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_768_noop_2" + }, + { + "bootstrap_steps": 900, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.10466819122324095, + "policy_error": null, + "policy_steps": 1700, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_900" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.09670898694738912, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.55 + }, + "reproduction": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.09800665026817185, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.09800665026817185, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.25 + } + }, + "target_score": 400.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-04/metadata.json new file mode 100644 index 0000000..6cb78b2 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-04/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "digest_compatibility_note": null, + "finished_at": 1784093477.189311, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "task": "simulation_heuristics_montezuma_v1", + "task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "token_usage": { + "final_input_tokens": 0.0, + "final_output_tokens": 0.0, + "input_tokens": 0.0, + "output_tokens": 0.0, + "total_tokens": 0.0 + }, + "tool_calls": 9, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-04/score.json new file mode 100644 index 0000000..a6d99c3 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-04/score.json @@ -0,0 +1,216 @@ +{ + "normalized_score": 0.0, + "reference_score": 400.0, + "score": 0.0, + "starter_score": 0.0, + "suites": { + "hidden_seeds": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.15601597448739296, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10007, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.11907760976240613, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10103, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.13754679212489954, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "noop_variants": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.16236112535166244, + "policy_error": null, + "policy_steps": 2597, + "pre_policy_noops": 3, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_3" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.14516854209441635, + "policy_error": null, + "policy_steps": 2587, + "pre_policy_noops": 13, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_13" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.1537648337230394, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "recovery": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.0939244203780726, + "policy_error": null, + "policy_steps": 2084, + "pre_policy_noops": 4, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_4" + }, + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.09359156258422165, + "policy_error": null, + "policy_steps": 2080, + "pre_policy_noops": 8, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_8" + }, + { + "bootstrap_steps": 768, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.1827507849803062, + "policy_error": null, + "policy_steps": 1830, + "pre_policy_noops": 2, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_768_noop_2" + }, + { + "bootstrap_steps": 900, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.09532441782272037, + "policy_error": null, + "policy_steps": 1700, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_900" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.1163977964413302, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.55 + }, + "reproduction": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.1588235599382852, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.1588235599382852, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.25 + } + }, + "target_score": 400.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-05/metadata.json new file mode 100644 index 0000000..95cd0eb --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-05/metadata.json @@ -0,0 +1,25 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "digest_compatibility_note": null, + "finished_at": 1784095422.450964, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05", + "task": "simulation_heuristics_montezuma_v1", + "task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "token_usage": null, + "tool_calls": 40, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-05/score.json new file mode 100644 index 0000000..d5d2c8a --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/trial-05/score.json @@ -0,0 +1,216 @@ +{ + "normalized_score": 0.0, + "reference_score": 400.0, + "score": 0.0, + "starter_score": 0.0, + "suites": { + "hidden_seeds": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.10522190078448218, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10007, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.14796666380089635, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10103, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.12659428229268926, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "noop_variants": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.10761198039752247, + "policy_error": null, + "policy_steps": 2597, + "pre_policy_noops": 3, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_3" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.15897829457109483, + "policy_error": null, + "policy_steps": 2587, + "pre_policy_noops": 13, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_13" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.13329513748430866, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "recovery": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.10332824830359094, + "policy_error": null, + "policy_steps": 2084, + "pre_policy_noops": 4, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_4" + }, + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.10210778150044811, + "policy_error": null, + "policy_steps": 2080, + "pre_policy_noops": 8, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_8" + }, + { + "bootstrap_steps": 768, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.16739498022333033, + "policy_error": null, + "policy_steps": 1830, + "pre_policy_noops": 2, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_768_noop_2" + }, + { + "bootstrap_steps": 900, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.141322282690774, + "policy_error": null, + "policy_steps": 1700, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_900" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.12853832317953584, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.55 + }, + "reproduction": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.14858248822677594, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.14858248822677594, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.25 + } + }, + "target_score": 400.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/metadata.json index 1ee464b..45d5555 100644 --- a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:f0b6f90f1d8072e75f0e10f90f4b51e51e31b3590bb2c2736778ba155b23389f", - "digest_compatibility_note": null, - "finished_at": 1783933771.613736, + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", "harness": "opencode", "model": { "display_name": "GPT-5.5", @@ -11,17 +9,19 @@ "provider": "azure", "provider_reasoning_effort": "xhigh" }, - "normalized_score": 45.833333333, + "normalized_score": 49.1666666668, + "normalized_score_stddev": 40.44028519521613, "provider_reasoning_effort": "xhigh", - "source_run_id": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "raw_score": 8.8, + "raw_score_imputation_count": 0, + "raw_score_stddev": 9.705668446840743, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-885696cfa4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], "task": "simulation_heuristics_pong_ram_v1", - "task_digest": "sha256:f0b6f90f1d8072e75f0e10f90f4b51e51e31b3590bb2c2736778ba155b23389f", - "token_usage": { - "final_input_tokens": 739.0, - "final_output_tokens": 248.0, - "input_tokens": 739.0, - "output_tokens": 248.0, - "total_tokens": 18907.0 - }, - "tool_calls": 31 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/score.json index 4135a2c..0dd946e 100644 --- a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/score.json @@ -1,133 +1,59 @@ { - "normalized_score": 45.833333333, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": 49.1666666668, + "normalized_score_stddev": 40.44028519521613, + "raw_score_imputation_count": 0, "reference_score": 21.0, - "score": 8.0, + "score": 8.8, + "score_stddev": 9.705668446840743, "starter_score": -2.9999999999999996, - "suite_weights": { - "hidden_nominal": 0.7, - "hidden_reset_robustness": 0.3 - }, - "suites": { - "hidden_nominal": { - "episodes": [ - { - "invalid_action": false, - "length": 27000, - "mean_action_latency_ms": 0.003850699022964195, - "points_against": 1, - "points_for": 9, - "policy_error": null, - "score": 8.0, - "seed": 101, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "article_nominal", - "won": true - }, - { - "invalid_action": false, - "length": 27000, - "mean_action_latency_ms": 0.0040996284396560105, - "points_against": 1, - "points_for": 9, - "policy_error": null, - "score": 8.0, - "seed": 211, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "article_nominal", - "won": true - }, - { - "invalid_action": false, - "length": 27000, - "mean_action_latency_ms": 0.004162275025414096, - "points_against": 1, - "points_for": 9, - "policy_error": null, - "score": 8.0, - "seed": 307, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "article_nominal", - "won": true - } - ], - "invalid_episode_rate": 0.0, - "max_score": 8.0, - "max_steps": 27000, - "mean_action_latency_ms": 0.004037534162678101, - "mean_score": 8.0, - "min_score": 8.0, - "policy_path": "submitted_artifact", - "score": 8.0, - "target_score": 21.0, - "target_score_rate": 0.0, - "win_rate": 1.0 + "task": "simulation_heuristics_pong_ram_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 41.666666667, + "observed_raw_score": 7.0, + "raw_score": 7.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "trial": 1 }, - "hidden_reset_robustness": { - "episodes": [ - { - "invalid_action": false, - "length": 27000, - "mean_action_latency_ms": 0.003967505638246183, - "points_against": 1, - "points_for": 9, - "policy_error": null, - "score": 8.0, - "seed": 503, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "randomized_reset", - "won": true - }, - { - "invalid_action": false, - "length": 27000, - "mean_action_latency_ms": 0.0037646840705915733, - "points_against": 1, - "points_for": 9, - "policy_error": null, - "score": 8.0, - "seed": 601, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "randomized_reset", - "won": true - }, - { - "invalid_action": false, - "length": 27000, - "mean_action_latency_ms": 0.0034993514073667705, - "points_against": 1, - "points_for": 9, - "policy_error": null, - "score": 8.0, - "seed": 701, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "randomized_reset", - "won": true - } - ], - "invalid_episode_rate": 0.0, - "max_score": 8.0, - "max_steps": 27000, - "mean_action_latency_ms": 0.003743847038734842, - "mean_score": 8.0, - "min_score": 8.0, - "policy_path": "submitted_artifact", - "score": 8.0, - "target_score": 21.0, - "target_score_rate": 0.0, - "win_rate": 1.0 + { + "normalized_score": 79.166666667, + "observed_raw_score": 16.0, + "raw_score": 16.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "trial": 2 + }, + { + "normalized_score": 100.0, + "observed_raw_score": 21.0, + "raw_score": 21.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "trial": 3 + }, + { + "normalized_score": 25.0, + "observed_raw_score": 2.9999999999999996, + "raw_score": 2.9999999999999996, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-885696cfa4", + "trial": 4 + }, + { + "normalized_score": 0.0, + "observed_raw_score": -2.9999999999999996, + "raw_score": -2.9999999999999996, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05", + "trial": 5 } - }, - "target_score": 21.0 + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-01/metadata.json new file mode 100644 index 0000000..192bbf0 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "digest_compatibility_note": null, + "finished_at": 1784074045.3527198, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 41.666666667, + "observed_raw_score": 7.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 7.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "task": "simulation_heuristics_pong_ram_v1", + "task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "token_usage": { + "final_input_tokens": 2559.0, + "final_output_tokens": 157.0, + "input_tokens": 2559.0, + "output_tokens": 157.0, + "total_tokens": 14492.0 + }, + "tool_calls": 16, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-01/score.json new file mode 100644 index 0000000..3597401 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-01/score.json @@ -0,0 +1,133 @@ +{ + "normalized_score": 41.666666667, + "reference_score": 21.0, + "score": 7.0, + "starter_score": -2.9999999999999996, + "suite_weights": { + "hidden_nominal": 0.7, + "hidden_reset_robustness": 0.3 + }, + "suites": { + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.0030477160104998837, + "points_against": 3, + "points_for": 10, + "policy_error": null, + "score": 7.0, + "seed": 101, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.0030376952986612363, + "points_against": 3, + "points_for": 10, + "policy_error": null, + "score": 7.0, + "seed": 211, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.002719435444170678, + "points_against": 3, + "points_for": 10, + "policy_error": null, + "score": 7.0, + "seed": 307, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "article_nominal", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 7.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.002934948917777266, + "mean_score": 7.0, + "min_score": 7.0, + "policy_path": "submitted_artifact", + "score": 7.0, + "target_score": 21.0, + "target_score_rate": 0.0, + "win_rate": 1.0 + }, + "hidden_reset_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.0026732335186184005, + "points_against": 3, + "points_for": 10, + "policy_error": null, + "score": 7.0, + "seed": 503, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.0025382887882490954, + "points_against": 3, + "points_for": 10, + "policy_error": null, + "score": 7.0, + "seed": 601, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.0027301118157252117, + "points_against": 3, + "points_for": 10, + "policy_error": null, + "score": 7.0, + "seed": 701, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "randomized_reset", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 7.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.002647211374197569, + "mean_score": 7.0, + "min_score": 7.0, + "policy_path": "submitted_artifact", + "score": 7.0, + "target_score": 21.0, + "target_score_rate": 0.0, + "win_rate": 1.0 + } + }, + "target_score": 21.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-02/metadata.json new file mode 100644 index 0000000..c611355 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "digest_compatibility_note": null, + "finished_at": 1784076544.549837, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 79.166666667, + "observed_raw_score": 16.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 16.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "task": "simulation_heuristics_pong_ram_v1", + "task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "token_usage": { + "final_input_tokens": 777.0, + "final_output_tokens": 255.0, + "input_tokens": 777.0, + "output_tokens": 255.0, + "total_tokens": 17416.0 + }, + "tool_calls": 16, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-02/score.json new file mode 100644 index 0000000..8954f61 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-02/score.json @@ -0,0 +1,133 @@ +{ + "normalized_score": 79.166666667, + "reference_score": 21.0, + "score": 16.0, + "starter_score": -2.9999999999999996, + "suite_weights": { + "hidden_nominal": 0.7, + "hidden_reset_robustness": 0.3 + }, + "suites": { + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 14182, + "mean_action_latency_ms": 0.004015189461620713, + "points_against": 5, + "points_for": 21, + "policy_error": null, + "score": 16.0, + "seed": 101, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 14182, + "mean_action_latency_ms": 0.004561139019346698, + "points_against": 5, + "points_for": 21, + "policy_error": null, + "score": 16.0, + "seed": 211, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 14182, + "mean_action_latency_ms": 0.004372412831766945, + "points_against": 5, + "points_for": 21, + "policy_error": null, + "score": 16.0, + "seed": 307, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 16.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.004316247104244785, + "mean_score": 16.0, + "min_score": 16.0, + "policy_path": "submitted_artifact", + "score": 16.0, + "target_score": 21.0, + "target_score_rate": 0.0, + "win_rate": 1.0 + }, + "hidden_reset_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 14182, + "mean_action_latency_ms": 0.004795075821983278, + "points_against": 5, + "points_for": 21, + "policy_error": null, + "score": 16.0, + "seed": 503, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 14178, + "mean_action_latency_ms": 0.004969168962701358, + "points_against": 5, + "points_for": 21, + "policy_error": null, + "score": 16.0, + "seed": 601, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 14179, + "mean_action_latency_ms": 0.004090428445560466, + "points_against": 5, + "points_for": 21, + "policy_error": null, + "score": 16.0, + "seed": 701, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 16.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.004618224410081701, + "mean_score": 16.0, + "min_score": 16.0, + "policy_path": "submitted_artifact", + "score": 16.0, + "target_score": 21.0, + "target_score_rate": 0.0, + "win_rate": 1.0 + } + }, + "target_score": 21.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-03/metadata.json new file mode 100644 index 0000000..39edb92 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "digest_compatibility_note": null, + "finished_at": 1784080509.825609, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 100.0, + "observed_raw_score": 21.0, + "provider_reasoning_effort": "xhigh", + "raw_score": 21.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "task": "simulation_heuristics_pong_ram_v1", + "task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "token_usage": { + "final_input_tokens": 1704.0, + "final_output_tokens": 239.0, + "input_tokens": 1704.0, + "output_tokens": 239.0, + "total_tokens": 17303.0 + }, + "tool_calls": 19, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-03/score.json new file mode 100644 index 0000000..832b1a7 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-03/score.json @@ -0,0 +1,133 @@ +{ + "normalized_score": 100.0, + "reference_score": 21.0, + "score": 21.0, + "starter_score": -2.9999999999999996, + "suite_weights": { + "hidden_nominal": 0.7, + "hidden_reset_robustness": 0.3 + }, + "suites": { + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 12942, + "mean_action_latency_ms": 0.004304664006842158, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 101, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 12942, + "mean_action_latency_ms": 0.0040340984637112585, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 211, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 12942, + "mean_action_latency_ms": 0.004495031698609067, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 307, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 21.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.004277931389720827, + "mean_score": 21.0, + "min_score": 21.0, + "policy_path": "submitted_artifact", + "score": 21.0, + "target_score": 21.0, + "target_score_rate": 1.0, + "win_rate": 1.0 + }, + "hidden_reset_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 12942, + "mean_action_latency_ms": 0.0044422488009954065, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 503, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 12938, + "mean_action_latency_ms": 0.004413906020692667, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 601, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 12939, + "mean_action_latency_ms": 0.004178940527836689, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 701, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 21.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.004345031783174921, + "mean_score": 21.0, + "min_score": 21.0, + "policy_path": "submitted_artifact", + "score": 21.0, + "target_score": 21.0, + "target_score_rate": 1.0, + "win_rate": 1.0 + } + }, + "target_score": 21.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-04/metadata.json new file mode 100644 index 0000000..1199610 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-04/metadata.json @@ -0,0 +1,25 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "digest_compatibility_note": null, + "finished_at": 1784095422.450964, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 25.0, + "observed_raw_score": 2.9999999999999996, + "provider_reasoning_effort": "xhigh", + "raw_score": 2.9999999999999996, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-885696cfa4", + "task": "simulation_heuristics_pong_ram_v1", + "task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "token_usage": null, + "tool_calls": 22, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-04/score.json new file mode 100644 index 0000000..3d4bf6c --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-04/score.json @@ -0,0 +1,133 @@ +{ + "normalized_score": 25.0, + "reference_score": 21.0, + "score": 2.9999999999999996, + "starter_score": -2.9999999999999996, + "suite_weights": { + "hidden_nominal": 0.7, + "hidden_reset_robustness": 0.3 + }, + "suites": { + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.003091757802327198, + "points_against": 10, + "points_for": 13, + "policy_error": null, + "score": 3.0, + "seed": 101, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.003134694601478124, + "points_against": 10, + "points_for": 13, + "policy_error": null, + "score": 3.0, + "seed": 211, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.0029207480622938384, + "points_against": 10, + "points_for": 13, + "policy_error": null, + "score": 3.0, + "seed": 307, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "article_nominal", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 3.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.0030490668220330536, + "mean_score": 3.0, + "min_score": 3.0, + "policy_path": "submitted_artifact", + "score": 3.0, + "target_score": 21.0, + "target_score_rate": 0.0, + "win_rate": 1.0 + }, + "hidden_reset_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.0028929879794035245, + "points_against": 10, + "points_for": 13, + "policy_error": null, + "score": 3.0, + "seed": 503, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.003008703041511277, + "points_against": 10, + "points_for": 13, + "policy_error": null, + "score": 3.0, + "seed": 601, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.0030335993031729703, + "points_against": 10, + "points_for": 13, + "policy_error": null, + "score": 3.0, + "seed": 701, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "randomized_reset", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 3.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.0029784301080292576, + "mean_score": 3.0, + "min_score": 3.0, + "policy_path": "submitted_artifact", + "score": 3.0, + "target_score": 21.0, + "target_score_rate": 0.0, + "win_rate": 1.0 + } + }, + "target_score": 21.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-05/metadata.json new file mode 100644 index 0000000..2535570 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-05/metadata.json @@ -0,0 +1,25 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "digest_compatibility_note": null, + "finished_at": 1784095422.450964, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": -2.9999999999999996, + "provider_reasoning_effort": "xhigh", + "raw_score": -2.9999999999999996, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05", + "task": "simulation_heuristics_pong_ram_v1", + "task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "token_usage": null, + "tool_calls": 20, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-05/score.json new file mode 100644 index 0000000..086f9ce --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/trial-05/score.json @@ -0,0 +1,133 @@ +{ + "normalized_score": 0.0, + "reference_score": 21.0, + "score": -2.9999999999999996, + "starter_score": -2.9999999999999996, + "suite_weights": { + "hidden_nominal": 0.7, + "hidden_reset_robustness": 0.3 + }, + "suites": { + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.00246426814528941, + "points_against": 5, + "points_for": 2, + "policy_error": null, + "score": -3.0, + "seed": 101, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "article_nominal", + "won": false + }, + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.00251024576007492, + "points_against": 5, + "points_for": 2, + "policy_error": null, + "score": -3.0, + "seed": 211, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "article_nominal", + "won": false + }, + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.0023182847588840458, + "points_against": 5, + "points_for": 2, + "policy_error": null, + "score": -3.0, + "seed": 307, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "article_nominal", + "won": false + } + ], + "invalid_episode_rate": 0.0, + "max_score": -3.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.002430932888082792, + "mean_score": -3.0, + "min_score": -3.0, + "policy_path": "submitted_artifact", + "score": -3.0, + "target_score": 21.0, + "target_score_rate": 0.0, + "win_rate": 0.0 + }, + "hidden_reset_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.0024129147382660043, + "points_against": 5, + "points_for": 2, + "policy_error": null, + "score": -3.0, + "seed": 503, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "randomized_reset", + "won": false + }, + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.002361140683879731, + "points_against": 5, + "points_for": 2, + "policy_error": null, + "score": -3.0, + "seed": 601, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "randomized_reset", + "won": false + }, + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.002365137812578016, + "points_against": 5, + "points_for": 2, + "policy_error": null, + "score": -3.0, + "seed": 701, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "randomized_reset", + "won": false + } + ], + "invalid_episode_rate": 0.0, + "max_score": -3.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.0023797310782412502, + "mean_score": -3.0, + "min_score": -3.0, + "policy_path": "submitted_artifact", + "score": -3.0, + "target_score": 21.0, + "target_score_rate": 0.0, + "win_rate": 0.0 + } + }, + "target_score": 21.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/metadata.json index 7015a52..e6e8413 100644 --- a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:316e198d8fa71e705c8b7fa54f41faf64e1f3c0ba5c4c92b8116b4bf78c5712b", - "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", - "finished_at": 1783938341.5167592, + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", "harness": "opencode", "model": { "display_name": "GPT-5.5", @@ -11,17 +9,19 @@ "provider": "azure", "provider_reasoning_effort": "xhigh" }, - "normalized_score": 85.199359782, + "normalized_score": 47.33864911959999, + "normalized_score_stddev": 95.13699952993868, "provider_reasoning_effort": "xhigh", - "source_run_id": "leaderboard/runs/article_suite/20260713T102218Z/gpt-5.5", + "raw_score": 0.6197079806402326, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.3579053781713541, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], "task": "simulation_heuristics_vizdoom_d1_v1", - "task_digest": "sha256:b3431b238bec4e66af6189d30bcd15d5cd227144dfe8bf0dd1011aa5416c1436", - "token_usage": { - "final_input_tokens": 1575.0, - "final_output_tokens": 196.0, - "input_tokens": 1575.0, - "output_tokens": 196.0, - "total_tokens": 22251.0 - }, - "tool_calls": 25 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/score.json index abc6f9f..13260c9 100644 --- a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/score.json @@ -1,211 +1,59 @@ { - "anchor_calibration": { - "envpool_version": "1.1.1", - "mode": "numeric", - "platform": "linux-x86_64" - }, - "isolation": { - "mode": "subprocess_per_suite", - "suite_process_count": 2, - "unique_working_directories": true - }, - "normalized_score": 85.199359782, - "reference_score": 0.8178199748694897, - "resource_cleanup": { - "removed_bytes": 0, - "removed_resources": 0, - "runs": [ - { - "removed_bytes": 0, - "removed_resources": 0, - "terminated_processes": 0 - }, - { - "removed_bytes": 0, - "removed_resources": 0, - "terminated_processes": 0 - } - ], - "terminated_processes": 0 - }, - "score": 0.7621399685554205, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": 47.33864911959999, + "normalized_score_stddev": 95.13699952993868, + "raw_score_imputation_count": 0, + "reference_score": 0.8178199748694898, + "score": 0.6197079806402326, + "score_stddev": 0.3579053781713541, "starter_score": 0.44161998964846133, - "suites": { - "hidden_nominal": { - "batch_seed": 101, - "envpool_version": "1.1.1", - "episodes": [ - { - "final_variables": { - "HEALTH": 37.0 - }, - "invalid_action": false, - "lane": 0, - "length": 2100, - "mean_action_latency_ms": 0.12978367596155121, - "policy_error": null, - "return": 0.7009999677538872, - "seed": 101, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "HEALTH": 52.0 - }, - "invalid_action": false, - "lane": 1, - "length": 2100, - "mean_action_latency_ms": 0.12354861146637372, - "policy_error": null, - "return": 0.7919999733567238, - "seed": 102, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "HEALTH": 52.0 - }, - "invalid_action": false, - "lane": 2, - "length": 2100, - "mean_action_latency_ms": 0.12090237012931279, - "policy_error": null, - "return": 0.8079999694600701, - "seed": 103, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "HEALTH": 100.0 - }, - "invalid_action": false, - "lane": 3, - "length": 2100, - "mean_action_latency_ms": 0.12010271705332257, - "policy_error": null, - "return": 0.9839999675750732, - "seed": 104, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "HEALTH": 60.0 - }, - "invalid_action": false, - "lane": 4, - "length": 2100, - "mean_action_latency_ms": 0.1198997339677243, - "policy_error": null, - "return": 0.8319999575614929, - "seed": 105, - "terminated": false, - "truncated": true - } - ], - "frame_skip": 1, - "invalid_episode_rate": 0.0, - "max_return": 0.9839999675750732, - "max_steps": 2100, - "mean_action_latency_ms": 0.12284742171565691, - "mean_return": 0.8233999671414495, - "min_return": 0.7009999677538872, - "policy_path": "submitted_artifact", - "render_height": 180, - "render_width": 240, - "scenario": "d1", - "score": 0.8233999671414495 + "task": "simulation_heuristics_vizdoom_d1_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": -117.814998349, + "observed_raw_score": -0.0016000167280435552, + "raw_score": -0.0016000167280435552, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "trial": 1 }, - "hidden_secondary": { - "batch_seed": 607, - "envpool_version": "1.1.1", - "episodes": [ - { - "final_variables": { - "HEALTH": 60.0 - }, - "invalid_action": false, - "lane": 0, - "length": 2100, - "mean_action_latency_ms": 0.13575854578188487, - "policy_error": null, - "return": 0.7359999641776085, - "seed": 607, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "HEALTH": 52.0 - }, - "invalid_action": false, - "lane": 1, - "length": 2100, - "mean_action_latency_ms": 0.12331685051321983, - "policy_error": null, - "return": 0.7759999595582485, - "seed": 608, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "HEALTH": 44.0 - }, - "invalid_action": false, - "lane": 2, - "length": 2100, - "mean_action_latency_ms": 0.12160280719399452, - "policy_error": null, - "return": 0.7199999764561653, - "seed": 609, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "HEALTH": 100.0 - }, - "invalid_action": false, - "lane": 3, - "length": 2100, - "mean_action_latency_ms": 0.11989811878828775, - "policy_error": null, - "return": 1.063999965786934, - "seed": 610, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "HEALTH": 4.0 - }, - "invalid_action": false, - "lane": 4, - "length": 512, - "mean_action_latency_ms": 0.11529984476510435, - "policy_error": null, - "return": -0.20000000670552254, - "seed": 611, - "terminated": true, - "truncated": false - } - ], - "frame_skip": 1, - "invalid_episode_rate": 0.0, - "max_return": 1.063999965786934, - "max_steps": 2100, - "mean_action_latency_ms": 0.12317523340849826, - "mean_return": 0.6191999718546868, - "min_return": -0.20000000670552254, - "policy_path": "submitted_artifact", - "render_height": 180, - "render_width": 240, - "scenario": "d1", - "score": 0.6191999718546868 + { + "normalized_score": 85.486445859, + "observed_raw_score": 0.7632199863344431, + "raw_score": 0.7632199863344431, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "trial": 2 + }, + { + "normalized_score": 104.662416785, + "observed_raw_score": 0.8353599861264229, + "raw_score": 0.8353599861264229, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "trial": 3 + }, + { + "normalized_score": 111.839447567, + "observed_raw_score": 0.8623599748685956, + "raw_score": 0.8623599748685956, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "trial": 4 + }, + { + "normalized_score": 52.519933736, + "observed_raw_score": 0.6391999725997448, + "raw_score": 0.6391999725997448, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05", + "trial": 5 } - } + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-01/metadata.json new file mode 100644 index 0000000..76be973 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "digest_compatibility_note": null, + "finished_at": 1784074045.3527198, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -117.814998349, + "observed_raw_score": -0.0016000167280435552, + "provider_reasoning_effort": "xhigh", + "raw_score": -0.0016000167280435552, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "task": "simulation_heuristics_vizdoom_d1_v1", + "task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "token_usage": { + "final_input_tokens": 1573.0, + "final_output_tokens": 195.0, + "input_tokens": 1573.0, + "output_tokens": 195.0, + "total_tokens": 19176.0 + }, + "tool_calls": 19, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-01/score.json new file mode 100644 index 0000000..9b09ea9 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-01/score.json @@ -0,0 +1,211 @@ +{ + "anchor_calibration": { + "envpool_version": "1.1.1", + "mode": "numeric", + "platform": "linux-x86_64" + }, + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": -117.814998349, + "reference_score": 0.8178199748694897, + "resource_cleanup": { + "removed_bytes": 0, + "removed_resources": 0, + "runs": [ + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + }, + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + } + ], + "terminated_processes": 0 + }, + "score": -0.0016000167280435552, + "starter_score": 0.44161998964846133, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 0, + "length": 544, + "mean_action_latency_ms": 0.19580094308099327, + "policy_error": null, + "return": -0.18400000780820847, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 1, + "length": 1024, + "mean_action_latency_ms": 0.20769537695741747, + "policy_error": null, + "return": 0.023999977856874466, + "seed": 102, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 2, + "length": 1312, + "mean_action_latency_ms": 0.21352932402273503, + "policy_error": null, + "return": 0.11999997869133949, + "seed": 103, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 3, + "length": 1408, + "mean_action_latency_ms": 0.20920986505436964, + "policy_error": null, + "return": 0.11999997869133949, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 4, + "length": 1152, + "mean_action_latency_ms": 0.20680191745567653, + "policy_error": null, + "return": 0.10399998351931572, + "seed": 105, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 0.11999997869133949, + "max_steps": 2100, + "mean_action_latency_ms": 0.2066074853142384, + "mean_return": 0.03679998219013214, + "min_return": -0.18400000780820847, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.03679998219013214 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 0, + "length": 1120, + "mean_action_latency_ms": 0.27209268882870674, + "policy_error": null, + "return": 0.08799997344613075, + "seed": 607, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 1, + "length": 512, + "mean_action_latency_ms": 0.23011269149719737, + "policy_error": null, + "return": -0.20000000670552254, + "seed": 608, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 2, + "length": 1056, + "mean_action_latency_ms": 0.25887539694932377, + "policy_error": null, + "return": 0.05599997565150261, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 3, + "length": 480, + "mean_action_latency_ms": 0.22971342550590634, + "policy_error": null, + "return": -0.2160000056028366, + "seed": 610, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 4, + "length": 544, + "mean_action_latency_ms": 0.22190529853105545, + "policy_error": null, + "return": -0.18400000780820847, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 0.08799997344613075, + "max_steps": 2100, + "mean_action_latency_ms": 0.24253990026243794, + "mean_return": -0.09120001420378684, + "min_return": -0.2160000056028366, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": -0.09120001420378684 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-02/metadata.json new file mode 100644 index 0000000..f7ad2c1 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "digest_compatibility_note": null, + "finished_at": 1784076544.549837, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 85.486445859, + "observed_raw_score": 0.7632199863344431, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.7632199863344431, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "task": "simulation_heuristics_vizdoom_d1_v1", + "task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "token_usage": { + "final_input_tokens": 1510.0, + "final_output_tokens": 193.0, + "input_tokens": 1510.0, + "output_tokens": 193.0, + "total_tokens": 18087.0 + }, + "tool_calls": 18, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-02/score.json new file mode 100644 index 0000000..de9f494 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-02/score.json @@ -0,0 +1,211 @@ +{ + "anchor_calibration": { + "envpool_version": "1.1.1", + "mode": "numeric", + "platform": "linux-x86_64" + }, + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 85.486445859, + "reference_score": 0.8178199748694897, + "resource_cleanup": { + "removed_bytes": 0, + "removed_resources": 0, + "runs": [ + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + }, + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + } + ], + "terminated_processes": 0 + }, + "score": 0.7632199863344431, + "starter_score": 0.44161998964846133, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 72.0 + }, + "invalid_action": false, + "lane": 0, + "length": 2100, + "mean_action_latency_ms": 0.2783612401357719, + "policy_error": null, + "return": 0.9399999864399433, + "seed": 101, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 3.0 + }, + "invalid_action": false, + "lane": 1, + "length": 1184, + "mean_action_latency_ms": 0.2447489422519465, + "policy_error": null, + "return": 0.13099998980760574, + "seed": 102, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 47.0 + }, + "invalid_action": false, + "lane": 2, + "length": 2100, + "mean_action_latency_ms": 0.2512248401485738, + "policy_error": null, + "return": 0.8149999864399433, + "seed": 103, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 70.0 + }, + "invalid_action": false, + "lane": 3, + "length": 2100, + "mean_action_latency_ms": 0.2478282047169549, + "policy_error": null, + "return": 0.9299999810755253, + "seed": 104, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 64.0 + }, + "invalid_action": false, + "lane": 4, + "length": 2100, + "mean_action_latency_ms": 0.24941567242855117, + "policy_error": null, + "return": 0.8839999884366989, + "seed": 105, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 0.9399999864399433, + "max_steps": 2100, + "mean_action_latency_ms": 0.2543157799363597, + "mean_return": 0.7399999864399434, + "min_return": 0.13099998980760574, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.7399999864399434 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 35.0 + }, + "invalid_action": false, + "lane": 0, + "length": 2100, + "mean_action_latency_ms": 0.2670958895413649, + "policy_error": null, + "return": 0.7389999851584435, + "seed": 607, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 47.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.24034427212817328, + "policy_error": null, + "return": 0.7989999875426292, + "seed": 608, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 46.0 + }, + "invalid_action": false, + "lane": 2, + "length": 2100, + "mean_action_latency_ms": 0.23726920641603924, + "policy_error": null, + "return": 0.8099999837577343, + "seed": 609, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 72.0 + }, + "invalid_action": false, + "lane": 3, + "length": 2100, + "mean_action_latency_ms": 0.23713585787585803, + "policy_error": null, + "return": 0.9239999875426292, + "seed": 610, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 47.0 + }, + "invalid_action": false, + "lane": 4, + "length": 2100, + "mean_action_latency_ms": 0.2356169292969363, + "policy_error": null, + "return": 0.8149999864399433, + "seed": 611, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 0.9239999875426292, + "max_steps": 2100, + "mean_action_latency_ms": 0.2434924310516743, + "mean_return": 0.8173999860882759, + "min_return": 0.7389999851584435, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.8173999860882759 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-03/metadata.json new file mode 100644 index 0000000..6f6efff --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "digest_compatibility_note": null, + "finished_at": 1784080509.825609, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 104.662416785, + "observed_raw_score": 0.8353599861264229, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.8353599861264229, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "task": "simulation_heuristics_vizdoom_d1_v1", + "task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "token_usage": { + "final_input_tokens": 1355.0, + "final_output_tokens": 198.0, + "input_tokens": 1355.0, + "output_tokens": 198.0, + "total_tokens": 15377.0 + }, + "tool_calls": 14, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-03/score.json new file mode 100644 index 0000000..404bdc8 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-03/score.json @@ -0,0 +1,211 @@ +{ + "anchor_calibration": { + "envpool_version": "1.1.1", + "mode": "numeric", + "platform": "linux-x86_64" + }, + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 104.662416785, + "reference_score": 0.8178199748694897, + "resource_cleanup": { + "removed_bytes": 0, + "removed_resources": 0, + "runs": [ + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + }, + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + } + ], + "terminated_processes": 0 + }, + "score": 0.8353599861264229, + "starter_score": 0.44161998964846133, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 54.0 + }, + "invalid_action": false, + "lane": 0, + "length": 2100, + "mean_action_latency_ms": 0.29605321320039885, + "policy_error": null, + "return": 0.8339999876916409, + "seed": 101, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 47.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.2735343894788197, + "policy_error": null, + "return": 0.8149999864399433, + "seed": 102, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 74.0 + }, + "invalid_action": false, + "lane": 2, + "length": 2100, + "mean_action_latency_ms": 0.26558944689376013, + "policy_error": null, + "return": 0.9499999843537807, + "seed": 103, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 47.0 + }, + "invalid_action": false, + "lane": 3, + "length": 2100, + "mean_action_latency_ms": 0.2647762293262141, + "policy_error": null, + "return": 0.8149999864399433, + "seed": 104, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 47.0 + }, + "invalid_action": false, + "lane": 4, + "length": 2100, + "mean_action_latency_ms": 0.26202865388421787, + "policy_error": null, + "return": 0.8149999864399433, + "seed": 105, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 0.9499999843537807, + "max_steps": 2100, + "mean_action_latency_ms": 0.27239638655668214, + "mean_return": 0.8457999862730503, + "min_return": 0.8149999864399433, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.8457999862730503 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 43.0 + }, + "invalid_action": false, + "lane": 0, + "length": 2100, + "mean_action_latency_ms": 0.3139959178155377, + "policy_error": null, + "return": 0.7949999831616879, + "seed": 607, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 47.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.28281007760337423, + "policy_error": null, + "return": 0.8149999864399433, + "seed": 608, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 47.0 + }, + "invalid_action": false, + "lane": 2, + "length": 2100, + "mean_action_latency_ms": 0.27199451855960344, + "policy_error": null, + "return": 0.8149999864399433, + "seed": 609, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 47.0 + }, + "invalid_action": false, + "lane": 3, + "length": 2100, + "mean_action_latency_ms": 0.27482333903511363, + "policy_error": null, + "return": 0.8149999864399433, + "seed": 610, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 47.0 + }, + "invalid_action": false, + "lane": 4, + "length": 2100, + "mean_action_latency_ms": 0.27485757711387815, + "policy_error": null, + "return": 0.8149999864399433, + "seed": 611, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 0.8149999864399433, + "max_steps": 2100, + "mean_action_latency_ms": 0.28369628602550145, + "mean_return": 0.8109999857842922, + "min_return": 0.7949999831616879, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.8109999857842922 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-04/metadata.json new file mode 100644 index 0000000..fd858fa --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-04/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "digest_compatibility_note": null, + "finished_at": 1784093477.189311, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 111.839447567, + "observed_raw_score": 0.8623599748685956, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.8623599748685956, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "task": "simulation_heuristics_vizdoom_d1_v1", + "task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "token_usage": { + "final_input_tokens": 434.0, + "final_output_tokens": 192.0, + "input_tokens": 434.0, + "output_tokens": 192.0, + "total_tokens": 17010.0 + }, + "tool_calls": 18, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-04/score.json new file mode 100644 index 0000000..c42ead0 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-04/score.json @@ -0,0 +1,211 @@ +{ + "anchor_calibration": { + "envpool_version": "1.1.1", + "mode": "numeric", + "platform": "linux-x86_64" + }, + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 111.839447567, + "reference_score": 0.8178199748694897, + "resource_cleanup": { + "removed_bytes": 0, + "removed_resources": 0, + "runs": [ + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + }, + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + } + ], + "terminated_processes": 0 + }, + "score": 0.8623599748685956, + "starter_score": 0.44161998964846133, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 0, + "length": 1760, + "mean_action_latency_ms": 0.3213713421824981, + "policy_error": null, + "return": 0.391999963670969, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.3558971963468052, + "policy_error": null, + "return": 1.0159999802708626, + "seed": 102, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 94.0 + }, + "invalid_action": false, + "lane": 2, + "length": 2100, + "mean_action_latency_ms": 0.33849141072659267, + "policy_error": null, + "return": 1.0499999709427357, + "seed": 103, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 3, + "length": 1920, + "mean_action_latency_ms": 0.30980391214446473, + "policy_error": null, + "return": 0.47199997678399086, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 4, + "length": 2100, + "mean_action_latency_ms": 0.3215396058346544, + "policy_error": null, + "return": 1.0799999795854092, + "seed": 105, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 1.0799999795854092, + "max_steps": 2100, + "mean_action_latency_ms": 0.32942069344700303, + "mean_return": 0.8019999742507935, + "min_return": 0.391999963670969, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.8019999742507935 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 84.0 + }, + "invalid_action": false, + "lane": 0, + "length": 2100, + "mean_action_latency_ms": 0.378185502652611, + "policy_error": null, + "return": 0.9679999761283398, + "seed": 607, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.3418168319123132, + "policy_error": null, + "return": 1.0319999810308218, + "seed": 608, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 76.0 + }, + "invalid_action": false, + "lane": 2, + "length": 2100, + "mean_action_latency_ms": 0.3283377231231758, + "policy_error": null, + "return": 0.9439999759197235, + "seed": 609, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 3, + "length": 2100, + "mean_action_latency_ms": 0.3206995306980042, + "policy_error": null, + "return": 1.0479999743402004, + "seed": 610, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 92.0 + }, + "invalid_action": false, + "lane": 4, + "length": 2100, + "mean_action_latency_ms": 0.30782170061554226, + "policy_error": null, + "return": 1.0239999741315842, + "seed": 611, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 1.0479999743402004, + "max_steps": 2100, + "mean_action_latency_ms": 0.33537225780032937, + "mean_return": 1.003199976310134, + "min_return": 0.9439999759197235, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 1.003199976310134 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-05/metadata.json new file mode 100644 index 0000000..6108b07 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-05/metadata.json @@ -0,0 +1,25 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "digest_compatibility_note": null, + "finished_at": 1784095422.450964, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 52.519933736, + "observed_raw_score": 0.6391999725997448, + "provider_reasoning_effort": "xhigh", + "raw_score": 0.6391999725997448, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05", + "task": "simulation_heuristics_vizdoom_d1_v1", + "task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "token_usage": null, + "tool_calls": 22, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-05/score.json new file mode 100644 index 0000000..9bc0c7a --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/trial-05/score.json @@ -0,0 +1,211 @@ +{ + "anchor_calibration": { + "envpool_version": "1.1.1", + "mode": "numeric", + "platform": "linux-x86_64" + }, + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 52.519933736, + "reference_score": 0.8178199748694897, + "resource_cleanup": { + "removed_bytes": 0, + "removed_resources": 0, + "runs": [ + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + }, + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + } + ], + "terminated_processes": 0 + }, + "score": 0.6391999725997448, + "starter_score": 0.44161998964846133, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 46.0 + }, + "invalid_action": false, + "lane": 0, + "length": 2100, + "mean_action_latency_ms": 0.3108608829123633, + "policy_error": null, + "return": 0.7299999631941319, + "seed": 101, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 92.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.2905094499389331, + "policy_error": null, + "return": 0.9439999535679817, + "seed": 102, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 2, + "length": 2100, + "mean_action_latency_ms": 0.2850152498909405, + "policy_error": null, + "return": 0.9679999761283398, + "seed": 103, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 100.0 + }, + "invalid_action": false, + "lane": 3, + "length": 2100, + "mean_action_latency_ms": 0.2804609734032835, + "policy_error": null, + "return": 0.9679999761283398, + "seed": 104, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 6.0 + }, + "invalid_action": false, + "lane": 4, + "length": 864, + "mean_action_latency_ms": 0.2749363753806662, + "policy_error": null, + "return": -0.03000001236796379, + "seed": 105, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 0.9679999761283398, + "max_steps": 2100, + "mean_action_latency_ms": 0.2883565863052373, + "mean_return": 0.7159999713301659, + "min_return": -0.03000001236796379, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.7159999713301659 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 6.0 + }, + "invalid_action": false, + "lane": 0, + "length": 576, + "mean_action_latency_ms": 0.20063967935533988, + "policy_error": null, + "return": -0.15800000354647636, + "seed": 607, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 38.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.2630051580213365, + "policy_error": null, + "return": 0.6899999678134918, + "seed": 608, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 92.0 + }, + "invalid_action": false, + "lane": 2, + "length": 2100, + "mean_action_latency_ms": 0.2476548980034533, + "policy_error": null, + "return": 0.9439999721944332, + "seed": 609, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 92.0 + }, + "invalid_action": false, + "lane": 3, + "length": 2100, + "mean_action_latency_ms": 0.23808318412020094, + "policy_error": null, + "return": 0.9599999524652958, + "seed": 610, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 4, + "length": 640, + "mean_action_latency_ms": 0.16739944694563746, + "policy_error": null, + "return": -0.13600001111626625, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 0.9599999524652958, + "max_steps": 2100, + "mean_action_latency_ms": 0.22335647328919359, + "mean_return": 0.45999997556209565, + "min_return": -0.15800000354647636, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.45999997556209565 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/metadata.json index f1e4a55..0783317 100644 --- a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:29434bdcdce19a6e0cb24d77c800d172d49a9529875ffd4201cf928e242f4e62", - "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", - "finished_at": 1783933771.613736, + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", "harness": "opencode", "model": { "display_name": "GPT-5.5", @@ -11,17 +9,19 @@ "provider": "azure", "provider_reasoning_effort": "xhigh" }, - "normalized_score": 5.481283422, + "normalized_score": 23.2620320856, + "normalized_score_stddev": 41.04765230004701, "provider_reasoning_effort": "xhigh", - "source_run_id": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "raw_score": 101.49999999999999, + "raw_score_imputation_count": 0, + "raw_score_stddev": 122.81457568220475, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], "task": "simulation_heuristics_vizdoom_d3_v1", - "task_digest": "sha256:fcc6ed05673b7981aa17d60ca4e0d355fbcb57a6cbca293c0fa201ab97cdd081", - "token_usage": { - "final_input_tokens": 1848.0, - "final_output_tokens": 220.0, - "input_tokens": 1848.0, - "output_tokens": 220.0, - "total_tokens": 17940.0 - }, - "tool_calls": 14 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/score.json index f7ec5f1..bbdefc7 100644 --- a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/score.json @@ -1,229 +1,59 @@ { - "isolation": { - "mode": "subprocess_per_suite", - "suite_process_count": 2, - "unique_working_directories": true - }, - "normalized_score": 5.481283422, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": 23.2620320856, + "normalized_score_stddev": 41.04765230004701, + "raw_score_imputation_count": 0, "reference_score": 331.1, - "score": 48.3, + "score": 101.49999999999999, + "score_stddev": 122.81457568220475, "starter_score": 31.9, - "suites": { - "hidden_nominal": { - "batch_seed": 101, - "envpool_version": "1.1.1", - "episodes": [ - { - "final_variables": { - "AMMO2": 1.0, - "DAMAGECOUNT": 15.0, - "HEALTH": 19.0, - "HITCOUNT": 1.0, - "KILLCOUNT": 1.0 - }, - "invalid_action": false, - "lane": 0, - "length": 147, - "mean_action_latency_ms": 1.9462588211508836, - "policy_error": null, - "return": 25.0, - "seed": 101, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 0.0, - "DAMAGECOUNT": 20.0, - "HEALTH": 4.0, - "HITCOUNT": 2.0, - "KILLCOUNT": 2.0 - }, - "invalid_action": false, - "lane": 1, - "length": 257, - "mean_action_latency_ms": 1.9732051119498242, - "policy_error": null, - "return": 40.0, - "seed": 102, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 2.0, - "DAMAGECOUNT": 0.0, - "HEALTH": 16.0, - "HITCOUNT": 0.0, - "KILLCOUNT": 0.0 - }, - "invalid_action": false, - "lane": 2, - "length": 130, - "mean_action_latency_ms": 1.8855262404451003, - "policy_error": null, - "return": 0.0, - "seed": 103, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 0.0, - "DAMAGECOUNT": 40.0, - "HEALTH": 7.0, - "HITCOUNT": 4.0, - "KILLCOUNT": 4.0 - }, - "invalid_action": false, - "lane": 3, - "length": 505, - "mean_action_latency_ms": 2.334480810135898, - "policy_error": null, - "return": 80.0, - "seed": 104, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 0.0, - "DAMAGECOUNT": 30.0, - "HEALTH": 12.0, - "HITCOUNT": 2.0, - "KILLCOUNT": 2.0 - }, - "invalid_action": false, - "lane": 4, - "length": 379, - "mean_action_latency_ms": 1.9117984675008577, - "policy_error": null, - "return": 50.0, - "seed": 105, - "terminated": true, - "truncated": false - } - ], - "frame_skip": 2, - "invalid_episode_rate": 0.0, - "max_return": 80.0, - "max_steps": 1050, - "mean_action_latency_ms": 2.0102538902365126, - "mean_return": 39.0, - "min_return": 0.0, - "policy_path": "submitted_artifact", - "render_height": 480, - "render_width": 640, - "scenario": "d3", - "score": 39.0 + "task": "simulation_heuristics_vizdoom_d3_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 22.326203209, + "observed_raw_score": 98.69999999999999, + "raw_score": 98.69999999999999, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "trial": 1 }, - "hidden_secondary": { - "batch_seed": 607, - "envpool_version": "1.1.1", - "episodes": [ - { - "final_variables": { - "AMMO2": 0.0, - "DAMAGECOUNT": 90.0, - "HEALTH": 10.0, - "HITCOUNT": 7.0, - "KILLCOUNT": 7.0 - }, - "invalid_action": false, - "lane": 0, - "length": 328, - "mean_action_latency_ms": 1.9002374625060616, - "policy_error": null, - "return": 160.0, - "seed": 607, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 0.0, - "DAMAGECOUNT": 15.0, - "HEALTH": 10.0, - "HITCOUNT": 1.0, - "KILLCOUNT": 1.0 - }, - "invalid_action": false, - "lane": 1, - "length": 295, - "mean_action_latency_ms": 1.9393673419194706, - "policy_error": null, - "return": 25.0, - "seed": 608, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 0.0, - "DAMAGECOUNT": 10.0, - "HEALTH": 9.0, - "HITCOUNT": 2.0, - "KILLCOUNT": 2.0 - }, - "invalid_action": false, - "lane": 2, - "length": 648, - "mean_action_latency_ms": 2.3452048074959975, - "policy_error": null, - "return": 30.0, - "seed": 609, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 0.0, - "DAMAGECOUNT": 0.0, - "HEALTH": 16.0, - "HITCOUNT": 0.0, - "KILLCOUNT": 0.0 - }, - "invalid_action": false, - "lane": 3, - "length": 415, - "mean_action_latency_ms": 1.9054093229842473, - "policy_error": null, - "return": 0.0, - "seed": 610, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 0.0, - "DAMAGECOUNT": 75.0, - "HEALTH": 19.0, - "HITCOUNT": 6.0, - "KILLCOUNT": 6.0 - }, - "invalid_action": false, - "lane": 4, - "length": 422, - "mean_action_latency_ms": 1.8646283432735369, - "policy_error": null, - "return": 135.0, - "seed": 611, - "terminated": true, - "truncated": false - } - ], - "frame_skip": 2, - "invalid_episode_rate": 0.0, - "max_return": 160.0, - "max_steps": 1050, - "mean_action_latency_ms": 1.990969455635863, - "mean_return": 70.0, - "min_return": 0.0, - "policy_path": "submitted_artifact", - "render_height": 480, - "render_width": 640, - "scenario": "d3", - "score": 70.0 + { + "normalized_score": 94.585561497, + "observed_raw_score": 314.9, + "raw_score": 314.9, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "trial": 2 + }, + { + "normalized_score": -0.601604278, + "observed_raw_score": 30.099999999999998, + "raw_score": 30.099999999999998, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "trial": 3 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 31.9, + "raw_score": 31.9, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "trial": 4 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 31.9, + "raw_score": 31.9, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05", + "trial": 5 } - } + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-01/metadata.json new file mode 100644 index 0000000..aa559d9 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "digest_compatibility_note": null, + "finished_at": 1784074045.3527198, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 22.326203209, + "observed_raw_score": 98.69999999999999, + "provider_reasoning_effort": "xhigh", + "raw_score": 98.69999999999999, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "task": "simulation_heuristics_vizdoom_d3_v1", + "task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "token_usage": { + "final_input_tokens": 2072.0, + "final_output_tokens": 198.0, + "input_tokens": 2072.0, + "output_tokens": 198.0, + "total_tokens": 23774.0 + }, + "tool_calls": 18, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-01/score.json new file mode 100644 index 0000000..e211dc6 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-01/score.json @@ -0,0 +1,229 @@ +{ + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 22.326203209, + "reference_score": 331.1, + "score": 98.69999999999999, + "starter_score": 31.9, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 100.0, + "HEALTH": 6.0, + "HITCOUNT": 10.0, + "KILLCOUNT": 10.0 + }, + "invalid_action": false, + "lane": 0, + "length": 498, + "mean_action_latency_ms": 3.368008126276564, + "policy_error": null, + "return": 200.0, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 5.0, + "HEALTH": 4.0, + "HITCOUNT": 1.0, + "KILLCOUNT": 1.0 + }, + "invalid_action": false, + "lane": 1, + "length": 457, + "mean_action_latency_ms": 3.243444515983177, + "policy_error": null, + "return": 15.0, + "seed": 102, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 45.0, + "HEALTH": 100.0, + "HITCOUNT": 6.0, + "KILLCOUNT": 6.0 + }, + "invalid_action": false, + "lane": 2, + "length": 1050, + "mean_action_latency_ms": 3.8129528860251107, + "policy_error": null, + "return": 105.0, + "seed": 103, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 30.0, + "HEALTH": 1.0, + "HITCOUNT": 3.0, + "KILLCOUNT": 3.0 + }, + "invalid_action": false, + "lane": 3, + "length": 635, + "mean_action_latency_ms": 3.2886894141125866, + "policy_error": null, + "return": 60.0, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 65.0, + "HEALTH": 6.0, + "HITCOUNT": 5.0, + "KILLCOUNT": 5.0 + }, + "invalid_action": false, + "lane": 4, + "length": 376, + "mean_action_latency_ms": 3.308834567507531, + "policy_error": null, + "return": 115.0, + "seed": 105, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 200.0, + "max_steps": 1050, + "mean_action_latency_ms": 3.4043859019809934, + "mean_return": 99.0, + "min_return": 15.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 99.0 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 35.0, + "HEALTH": 21.0, + "HITCOUNT": 3.0, + "KILLCOUNT": 3.0 + }, + "invalid_action": false, + "lane": 0, + "length": 227, + "mean_action_latency_ms": 3.420247807114135, + "policy_error": null, + "return": 65.0, + "seed": 607, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 100.0, + "HEALTH": 92.0, + "HITCOUNT": 10.0, + "KILLCOUNT": 10.0 + }, + "invalid_action": false, + "lane": 1, + "length": 1050, + "mean_action_latency_ms": 3.6867654359056834, + "policy_error": null, + "return": 200.0, + "seed": 608, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 55.0, + "HEALTH": 6.0, + "HITCOUNT": 5.0, + "KILLCOUNT": 5.0 + }, + "invalid_action": false, + "lane": 2, + "length": 784, + "mean_action_latency_ms": 3.4631100169630074, + "policy_error": null, + "return": 105.0, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 16.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 3, + "length": 491, + "mean_action_latency_ms": 3.30673801559052, + "policy_error": null, + "return": 0.0, + "seed": 610, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 60.0, + "HEALTH": 36.0, + "HITCOUNT": 6.0, + "KILLCOUNT": 6.0 + }, + "invalid_action": false, + "lane": 4, + "length": 422, + "mean_action_latency_ms": 3.29475750113833, + "policy_error": null, + "return": 120.0, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 200.0, + "max_steps": 1050, + "mean_action_latency_ms": 3.434323755342335, + "mean_return": 98.0, + "min_return": 0.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 98.0 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-02/metadata.json new file mode 100644 index 0000000..6122ae2 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "digest_compatibility_note": null, + "finished_at": 1784076544.549837, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 94.585561497, + "observed_raw_score": 314.9, + "provider_reasoning_effort": "xhigh", + "raw_score": 314.9, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "task": "simulation_heuristics_vizdoom_d3_v1", + "task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "token_usage": { + "final_input_tokens": 1955.0, + "final_output_tokens": 215.0, + "input_tokens": 1955.0, + "output_tokens": 215.0, + "total_tokens": 19066.0 + }, + "tool_calls": 14, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-02/score.json new file mode 100644 index 0000000..dd93d20 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-02/score.json @@ -0,0 +1,229 @@ +{ + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 94.585561497, + "reference_score": 331.1, + "score": 314.9, + "starter_score": 31.9, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 45.0, + "HEALTH": 13.0, + "HITCOUNT": 5.0, + "KILLCOUNT": 5.0 + }, + "invalid_action": false, + "lane": 0, + "length": 938, + "mean_action_latency_ms": 3.965730248674401, + "policy_error": null, + "return": 95.0, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 40.0, + "DAMAGECOUNT": 175.0, + "HEALTH": 100.0, + "HITCOUNT": 18.0, + "KILLCOUNT": 14.0 + }, + "invalid_action": false, + "lane": 1, + "length": 1050, + "mean_action_latency_ms": 3.922919499732199, + "policy_error": null, + "return": 315.0, + "seed": 102, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 11.0, + "DAMAGECOUNT": 195.0, + "HEALTH": 76.0, + "HITCOUNT": 19.0, + "KILLCOUNT": 17.0 + }, + "invalid_action": false, + "lane": 2, + "length": 1050, + "mean_action_latency_ms": 3.884811369436128, + "policy_error": null, + "return": 365.0, + "seed": 103, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 47.0, + "DAMAGECOUNT": 230.0, + "HEALTH": 94.0, + "HITCOUNT": 20.0, + "KILLCOUNT": 20.0 + }, + "invalid_action": false, + "lane": 3, + "length": 1050, + "mean_action_latency_ms": 3.8977514668589546, + "policy_error": null, + "return": 430.0, + "seed": 104, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 6.0, + "DAMAGECOUNT": 255.0, + "HEALTH": 100.0, + "HITCOUNT": 27.0, + "KILLCOUNT": 23.0 + }, + "invalid_action": false, + "lane": 4, + "length": 1050, + "mean_action_latency_ms": 3.8629698895272755, + "policy_error": null, + "return": 485.0, + "seed": 105, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 485.0, + "max_steps": 1050, + "mean_action_latency_ms": 3.906836494845792, + "mean_return": 338.0, + "min_return": 95.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 338.0 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 95.0, + "HEALTH": 16.0, + "HITCOUNT": 11.0, + "KILLCOUNT": 11.0 + }, + "invalid_action": false, + "lane": 0, + "length": 640, + "mean_action_latency_ms": 3.9674829866271466, + "policy_error": null, + "return": 205.0, + "seed": 607, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 115.0, + "HEALTH": 16.0, + "HITCOUNT": 9.0, + "KILLCOUNT": 9.0 + }, + "invalid_action": false, + "lane": 1, + "length": 274, + "mean_action_latency_ms": 4.022217783940969, + "policy_error": null, + "return": 205.0, + "seed": 608, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 8.0, + "DAMAGECOUNT": 170.0, + "HEALTH": 100.0, + "HITCOUNT": 19.0, + "KILLCOUNT": 17.0 + }, + "invalid_action": false, + "lane": 2, + "length": 1050, + "mean_action_latency_ms": 3.9119946034181687, + "policy_error": null, + "return": 340.0, + "seed": 609, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 7.0, + "DAMAGECOUNT": 200.0, + "HEALTH": 67.0, + "HITCOUNT": 24.0, + "KILLCOUNT": 19.0 + }, + "invalid_action": false, + "lane": 3, + "length": 1050, + "mean_action_latency_ms": 3.8772569596767426, + "policy_error": null, + "return": 390.0, + "seed": 610, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 85.0, + "HEALTH": 2.0, + "HITCOUNT": 8.0, + "KILLCOUNT": 8.0 + }, + "invalid_action": false, + "lane": 4, + "length": 544, + "mean_action_latency_ms": 3.9501719943740787, + "policy_error": null, + "return": 165.0, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 390.0, + "max_steps": 1050, + "mean_action_latency_ms": 3.9458248656074213, + "mean_return": 261.0, + "min_return": 165.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 261.0 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-03/metadata.json new file mode 100644 index 0000000..33ef5ba --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "digest_compatibility_note": null, + "finished_at": 1784080509.825609, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": -0.601604278, + "observed_raw_score": 30.099999999999998, + "provider_reasoning_effort": "xhigh", + "raw_score": 30.099999999999998, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "task": "simulation_heuristics_vizdoom_d3_v1", + "task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "token_usage": { + "final_input_tokens": 2152.0, + "final_output_tokens": 211.0, + "input_tokens": 2152.0, + "output_tokens": 211.0, + "total_tokens": 24379.0 + }, + "tool_calls": 20, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-03/score.json new file mode 100644 index 0000000..75a79d1 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-03/score.json @@ -0,0 +1,229 @@ +{ + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": -0.601604278, + "reference_score": 331.1, + "score": 30.099999999999998, + "starter_score": 31.9, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 10.0, + "HEALTH": 3.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 0, + "length": 736, + "mean_action_latency_ms": 2.8825574959425824, + "policy_error": null, + "return": 30.0, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 20.0, + "HEALTH": 10.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 1, + "length": 596, + "mean_action_latency_ms": 2.692667692399665, + "policy_error": null, + "return": 40.0, + "seed": 102, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 10.0, + "HEALTH": 19.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 2, + "length": 286, + "mean_action_latency_ms": 2.0376152222181534, + "policy_error": null, + "return": 30.0, + "seed": 103, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 1.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 3, + "length": 184, + "mean_action_latency_ms": 2.0164204963847348, + "policy_error": null, + "return": 0.0, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 20.0, + "HEALTH": 4.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 4, + "length": 279, + "mean_action_latency_ms": 2.018733878075863, + "policy_error": null, + "return": 40.0, + "seed": 105, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 40.0, + "max_steps": 1050, + "mean_action_latency_ms": 2.3295989570041997, + "mean_return": 28.0, + "min_return": 0.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 28.0 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 3.0, + "DAMAGECOUNT": 40.0, + "HEALTH": 2.0, + "HITCOUNT": 5.0, + "KILLCOUNT": 5.0 + }, + "invalid_action": false, + "lane": 0, + "length": 471, + "mean_action_latency_ms": 2.0396869684986756, + "policy_error": null, + "return": 90.0, + "seed": 607, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 18.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 1, + "length": 242, + "mean_action_latency_ms": 2.002615250708643, + "policy_error": null, + "return": 0.0, + "seed": 608, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 13.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 2, + "length": 471, + "mean_action_latency_ms": 2.006273400315307, + "policy_error": null, + "return": 0.0, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 1.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 3, + "length": 593, + "mean_action_latency_ms": 2.4311050873180666, + "policy_error": null, + "return": 0.0, + "seed": 610, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 45.0, + "HEALTH": 2.0, + "HITCOUNT": 4.0, + "KILLCOUNT": 4.0 + }, + "invalid_action": false, + "lane": 4, + "length": 387, + "mean_action_latency_ms": 2.013533710663325, + "policy_error": null, + "return": 85.0, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 90.0, + "max_steps": 1050, + "mean_action_latency_ms": 2.098642883500803, + "mean_return": 35.0, + "min_return": 0.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 35.0 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-04/metadata.json new file mode 100644 index 0000000..f42a7c6 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-04/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "digest_compatibility_note": null, + "finished_at": 1784093477.189311, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 31.9, + "provider_reasoning_effort": "xhigh", + "raw_score": 31.9, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "task": "simulation_heuristics_vizdoom_d3_v1", + "task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "token_usage": { + "final_input_tokens": 1882.0, + "final_output_tokens": 131.0, + "input_tokens": 1882.0, + "output_tokens": 131.0, + "total_tokens": 25053.0 + }, + "tool_calls": 19, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-04/score.json new file mode 100644 index 0000000..412e35c --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-04/score.json @@ -0,0 +1,229 @@ +{ + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 0.0, + "reference_score": 331.1, + "score": 31.9, + "starter_score": 31.9, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 10.0, + "HEALTH": 10.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 0, + "length": 711, + "mean_action_latency_ms": 2.0906845508496974, + "policy_error": null, + "return": 30.0, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 20.0, + "HEALTH": 12.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 1, + "length": 980, + "mean_action_latency_ms": 2.2012136914596265, + "policy_error": null, + "return": 40.0, + "seed": 102, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 10.0, + "HEALTH": 16.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 2, + "length": 317, + "mean_action_latency_ms": 2.145109210480651, + "policy_error": null, + "return": 30.0, + "seed": 103, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 13.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 3, + "length": 179, + "mean_action_latency_ms": 2.084528767196826, + "policy_error": null, + "return": 0.0, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 20.0, + "HEALTH": 10.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 4, + "length": 306, + "mean_action_latency_ms": 2.123713895094161, + "policy_error": null, + "return": 40.0, + "seed": 105, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 40.0, + "max_steps": 1050, + "mean_action_latency_ms": 2.1290500230161924, + "mean_return": 28.0, + "min_return": 0.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 28.0 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 35.0, + "HEALTH": 90.0, + "HITCOUNT": 4.0, + "KILLCOUNT": 4.0 + }, + "invalid_action": false, + "lane": 0, + "length": 1050, + "mean_action_latency_ms": 2.4963120300145376, + "policy_error": null, + "return": 75.0, + "seed": 607, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 10.0, + "HEALTH": 10.0, + "HITCOUNT": 1.0, + "KILLCOUNT": 1.0 + }, + "invalid_action": false, + "lane": 1, + "length": 262, + "mean_action_latency_ms": 2.000286364714608, + "policy_error": null, + "return": 20.0, + "seed": 608, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 4.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 2, + "length": 473, + "mean_action_latency_ms": 2.091843603340314, + "policy_error": null, + "return": 0.0, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 19.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 3, + "length": 400, + "mean_action_latency_ms": 1.9805391505360603, + "policy_error": null, + "return": 0.0, + "seed": 610, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 60.0, + "HEALTH": 19.0, + "HITCOUNT": 5.0, + "KILLCOUNT": 5.0 + }, + "invalid_action": false, + "lane": 4, + "length": 373, + "mean_action_latency_ms": 1.939606520749289, + "policy_error": null, + "return": 110.0, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 110.0, + "max_steps": 1050, + "mean_action_latency_ms": 2.1017175338709615, + "mean_return": 41.0, + "min_return": 0.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 41.0 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-05/metadata.json new file mode 100644 index 0000000..a89ac3d --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-05/metadata.json @@ -0,0 +1,25 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "digest_compatibility_note": null, + "finished_at": 1784095422.450964, + "harness": "opencode", + "model": { + "display_name": "GPT-5.5", + "id": "gpt-5.5", + "model": "azure/gpt-5.5", + "provider": "azure", + "provider_reasoning_effort": "xhigh" + }, + "normalized_score": 0.0, + "observed_raw_score": 31.9, + "provider_reasoning_effort": "xhigh", + "raw_score": 31.9, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05", + "task": "simulation_heuristics_vizdoom_d3_v1", + "task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "token_usage": null, + "tool_calls": 16, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-05/score.json new file mode 100644 index 0000000..1342a4b --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/trial-05/score.json @@ -0,0 +1,229 @@ +{ + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 0.0, + "reference_score": 331.1, + "score": 31.9, + "starter_score": 31.9, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 10.0, + "HEALTH": 10.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 0, + "length": 711, + "mean_action_latency_ms": 2.1708207899172764, + "policy_error": null, + "return": 30.0, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 20.0, + "HEALTH": 12.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 1, + "length": 980, + "mean_action_latency_ms": 2.1338099081601416, + "policy_error": null, + "return": 40.0, + "seed": 102, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 10.0, + "HEALTH": 16.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 2, + "length": 317, + "mean_action_latency_ms": 2.134885497931802, + "policy_error": null, + "return": 30.0, + "seed": 103, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 13.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 3, + "length": 179, + "mean_action_latency_ms": 2.1322682375015494, + "policy_error": null, + "return": 0.0, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 20.0, + "HEALTH": 10.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 4, + "length": 306, + "mean_action_latency_ms": 2.1491556556201448, + "policy_error": null, + "return": 40.0, + "seed": 105, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 40.0, + "max_steps": 1050, + "mean_action_latency_ms": 2.144188017826183, + "mean_return": 28.0, + "min_return": 0.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 28.0 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 35.0, + "HEALTH": 90.0, + "HITCOUNT": 4.0, + "KILLCOUNT": 4.0 + }, + "invalid_action": false, + "lane": 0, + "length": 1050, + "mean_action_latency_ms": 2.089796236583165, + "policy_error": null, + "return": 75.0, + "seed": 607, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 10.0, + "HEALTH": 10.0, + "HITCOUNT": 1.0, + "KILLCOUNT": 1.0 + }, + "invalid_action": false, + "lane": 1, + "length": 262, + "mean_action_latency_ms": 2.1605209113304853, + "policy_error": null, + "return": 20.0, + "seed": 608, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 4.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 2, + "length": 473, + "mean_action_latency_ms": 2.126526977496722, + "policy_error": null, + "return": 0.0, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 19.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 3, + "length": 400, + "mean_action_latency_ms": 2.056230138987303, + "policy_error": null, + "return": 0.0, + "seed": 610, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 60.0, + "HEALTH": 19.0, + "HITCOUNT": 5.0, + "KILLCOUNT": 5.0 + }, + "invalid_action": false, + "lane": 4, + "length": 373, + "mean_action_latency_ms": 2.052008196831389, + "policy_error": null, + "return": 110.0, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 110.0, + "max_steps": 1050, + "mean_action_latency_ms": 2.097016492245813, + "mean_return": 41.0, + "min_return": 0.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 41.0 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/metadata.json index 660774b..f674300 100644 --- a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:5cf3805d59b03cb9d0bebb6f154eab38beb655918b8321500d692f5010a00cb2", - "digest_compatibility_note": "Score-equivalent to the current task. Later changes add a fail-closed verifier timeout plus a CI-only smoke config and documentation; the publication scoring config is unchanged.", - "finished_at": 1783939823.3358562, + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", "harness": "opencode", "model": { "display_name": "GPT-5.6 Sol", @@ -11,17 +9,19 @@ "provider": "azure", "provider_reasoning_effort": "max" }, - "normalized_score": 39.203600385, + "normalized_score": -4.9009087386, + "normalized_score_stddev": 11.991465810903719, "provider_reasoning_effort": "max", - "source_run_id": "leaderboard/runs/article_suite/20260713T093815Z/gpt-5.6-sol", + "raw_score": 2272.356693724495, + "raw_score_imputation_count": 0, + "raw_score_stddev": 419.52351859557746, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "task": "simulation_heuristics_ant_v1", - "task_digest": "sha256:bbb533da0cb86459f4d49dee667e6c73ac54c0188bc40e54e911d50ef3c3bc38", - "token_usage": { - "final_input_tokens": 416.0, - "final_output_tokens": 196.0, - "input_tokens": 416.0, - "output_tokens": 196.0, - "total_tokens": 36964.0 - }, - "tool_calls": 37 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/score.json index 3582d4f..61ed20f 100644 --- a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/score.json @@ -1,141 +1,59 @@ { - "hidden_nominal": { - "episodes": [ - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 133.12361360079376, - "policy_error": null, - "return": 2944.6798391671878, - "seed": 101, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": 112.93341277596396, - "x_velocity": -0.05774537124608514 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 130.92819425201742, - "policy_error": null, - "return": 4189.938284380654, - "seed": 211, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": 176.7363723697072, - "x_velocity": 5.390771978175053 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 133.01312479103217, - "policy_error": null, - "return": 4054.5561540797858, - "seed": 307, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": 169.74913616212385, - "x_velocity": 3.097238615771971 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 128.64087908767397, - "policy_error": null, - "return": 4052.711831067477, - "seed": 401, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": 169.84904705908204, - "x_velocity": 2.148484367220931 - } - ], - "fall_rate": 0.0, - "invalid_episode_rate": 0.0, - "max_return": 4189.938284380654, - "max_steps": 1000, - "mean_action_latency_ms": 131.42645293287933, - "mean_return": 3810.471527173776, - "mean_x_position": 157.31699209171927, - "min_return": 2944.6798391671878, - "policy_path": "submitted_artifact", - "score": 3810.471527173776 - }, - "hidden_nominal_weight": 0.7, - "hidden_robustness": { - "episodes": [ - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 129.90234417081228, - "policy_error": null, - "return": 4416.461572320053, - "seed": 503, - "terminated": false, - "truncated": true, - "variant": "light_low_friction", - "x_position": 187.29214144677127, - "x_velocity": 3.390203653403887 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 128.21582087071147, - "policy_error": null, - "return": 4370.866265431624, - "seed": 601, - "terminated": false, - "truncated": true, - "variant": "light_low_friction", - "x_position": 185.1144372257778, - "x_velocity": 3.11121379282838 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 129.91995375533588, - "policy_error": null, - "return": 3731.5669383240584, - "seed": 503, - "terminated": false, - "truncated": true, - "variant": "heavy_weak_actuators", - "x_position": 153.54017346584874, - "x_velocity": 3.9302016156398167 - }, - { - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 131.77032847257215, - "policy_error": null, - "return": 2788.179265576913, - "seed": 601, - "terminated": false, - "truncated": true, - "variant": "heavy_weak_actuators", - "x_position": 106.42804578295254, - "x_velocity": 2.7356958324236302 - } - ], - "fall_rate": 0.0, - "invalid_episode_rate": 0.0, - "max_return": 4416.461572320053, - "max_steps": 1000, - "mean_action_latency_ms": 129.95211181735795, - "mean_return": 3826.768510413162, - "mean_x_position": 158.0936994803376, - "min_return": 2788.179265576913, - "policy_path": "submitted_artifact", - "score": 3826.768510413162 - }, - "hidden_robustness_weight": 0.3, - "normalized_score": 39.203600385, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": -4.9009087386, + "normalized_score_stddev": 11.991465810903719, + "raw_score_imputation_count": 0, "reference_score": 5942.333244755838, - "score": 3815.360622145591, + "score": 2272.356693724495, + "score_stddev": 419.52351859557746, "starter_score": 2443.815838992282, - "unique_policy_evaluations": 3 + "task": "simulation_heuristics_ant_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 9.173888471, + "observed_raw_score": 2764.765923919426, + "raw_score": 2764.765923919426, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "trial": 1 + }, + { + "normalized_score": -7.591479, + "observed_raw_score": 2178.226624807645, + "raw_score": 2178.226624807645, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "trial": 2 + }, + { + "normalized_score": -11.713338376, + "observed_raw_score": 2034.0226571116652, + "raw_score": 2034.0226571116652, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "trial": 3 + }, + { + "normalized_score": -19.696444908, + "observed_raw_score": 1754.7322855694497, + "raw_score": 1754.7322855694497, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04/tasks-81ec8d98a4", + "trial": 4 + }, + { + "normalized_score": 5.32283012, + "observed_raw_score": 2630.03597721429, + "raw_score": 2630.03597721429, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05", + "trial": 5 + } + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-01/metadata.json new file mode 100644 index 0000000..25ac64b --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "digest_compatibility_note": null, + "finished_at": 1784075101.71677, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 9.173888471, + "observed_raw_score": 2764.765923919426, + "provider_reasoning_effort": "max", + "raw_score": 2764.765923919426, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "task": "simulation_heuristics_ant_v1", + "task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "token_usage": { + "final_input_tokens": 374.0, + "final_output_tokens": 210.0, + "input_tokens": 374.0, + "output_tokens": 210.0, + "total_tokens": 21576.0 + }, + "tool_calls": 27, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-01/score.json new file mode 100644 index 0000000..3d28619 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-01/score.json @@ -0,0 +1,141 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 46.019714021123946, + "policy_error": null, + "return": 2949.322870014556, + "seed": 101, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 104.78259776691343, + "x_velocity": 1.0957534864547824 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 47.32066801702604, + "policy_error": null, + "return": 2925.031969122408, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 103.51131861843047, + "x_velocity": 2.3111608990888044 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 47.05008286237717, + "policy_error": null, + "return": 2884.4059581318525, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 101.60425460079155, + "x_velocity": 1.6999644592368668 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 44.82955207768828, + "policy_error": null, + "return": 2016.633773953487, + "seed": 401, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 58.19575045615417, + "x_velocity": 1.398747935893283 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 2949.322870014556, + "max_steps": 1000, + "mean_action_latency_ms": 46.30500424455386, + "mean_return": 2693.8486428055758, + "mean_x_position": 92.02348036057239, + "min_return": 2016.633773953487, + "policy_path": "submitted_artifact", + "score": 2693.8486428055758 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 44.357171959243715, + "policy_error": null, + "return": 3089.2579725253718, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 111.71315475597991, + "x_velocity": 2.158725891256097 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 45.45596221461892, + "policy_error": null, + "return": 2808.0848523662758, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 97.84109478638925, + "x_velocity": -0.2925011593802651 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 45.69658788526431, + "policy_error": null, + "return": 2857.02579552167, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 100.50025867847285, + "x_velocity": 2.8428404053954637 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 47.05479347752407, + "policy_error": null, + "return": 2966.589698993657, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 106.07191251292033, + "x_velocity": 1.9826592504307428 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 3089.2579725253718, + "max_steps": 1000, + "mean_action_latency_ms": 45.641128884162754, + "mean_return": 2930.2395798517437, + "mean_x_position": 104.03160518344059, + "min_return": 2808.0848523662758, + "policy_path": "submitted_artifact", + "score": 2930.2395798517437 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": 9.173888471, + "reference_score": 5942.333244755838, + "score": 2764.765923919426, + "starter_score": 2443.815838992282, + "unique_policy_evaluations": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-02/metadata.json new file mode 100644 index 0000000..ceaec6a --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "digest_compatibility_note": null, + "finished_at": 1784078883.680485, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": -7.591479, + "observed_raw_score": 2178.226624807645, + "provider_reasoning_effort": "max", + "raw_score": 2178.226624807645, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "task": "simulation_heuristics_ant_v1", + "task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "token_usage": { + "final_input_tokens": 539.0, + "final_output_tokens": 172.0, + "input_tokens": 539.0, + "output_tokens": 172.0, + "total_tokens": 38087.0 + }, + "tool_calls": 37, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-02/score.json new file mode 100644 index 0000000..0a2274f --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-02/score.json @@ -0,0 +1,141 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.02975405380129814, + "policy_error": null, + "return": 2000.6286232348173, + "seed": 101, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 53.80113931757859, + "x_velocity": 2.1311493378351543 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.028919238597154617, + "policy_error": null, + "return": 2569.3091871731217, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 82.17737194908375, + "x_velocity": 2.354063933416626 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.03253542631864548, + "policy_error": null, + "return": 2385.6946277480392, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 73.04151306245869, + "x_velocity": 1.1981546903319895 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.03688942268490791, + "policy_error": null, + "return": 2368.655642993632, + "seed": 401, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 72.19982332507476, + "x_velocity": 1.1704044847220985 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 2569.3091871731217, + "max_steps": 1000, + "mean_action_latency_ms": 0.03202453535050154, + "mean_return": 2331.0720202874027, + "mean_x_position": 70.30496191354895, + "min_return": 2000.6286232348173, + "policy_path": "submitted_artifact", + "score": 2331.0720202874027 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.02802887186408043, + "policy_error": null, + "return": 2798.32044246107, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 93.54063165430142, + "x_velocity": 1.2473198151258202 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.03125184401869774, + "policy_error": null, + "return": 358.1267611570017, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": -28.309368557338463, + "x_velocity": -1.156389118354042 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.027832284569740295, + "policy_error": null, + "return": 1812.2106051749915, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 44.58481769343802, + "x_velocity": -1.9142496670555431 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.034463025629520416, + "policy_error": null, + "return": 2317.6916659597805, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 69.94711534445288, + "x_velocity": 0.8705855141025154 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 2798.32044246107, + "max_steps": 1000, + "mean_action_latency_ms": 0.03039400652050972, + "mean_return": 1821.587368688211, + "mean_x_position": 44.94079903371346, + "min_return": 358.1267611570017, + "policy_path": "submitted_artifact", + "score": 1821.587368688211 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": -7.591479, + "reference_score": 5942.333244755838, + "score": 2178.226624807645, + "starter_score": 2443.815838992282, + "unique_policy_evaluations": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-03/metadata.json new file mode 100644 index 0000000..dcfa35e --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "digest_compatibility_note": null, + "finished_at": 1784092762.840873, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": -11.713338376, + "observed_raw_score": 2034.0226571116652, + "provider_reasoning_effort": "max", + "raw_score": 2034.0226571116652, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "task": "simulation_heuristics_ant_v1", + "task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "token_usage": { + "final_input_tokens": 719.0, + "final_output_tokens": 129.0, + "input_tokens": 719.0, + "output_tokens": 129.0, + "total_tokens": 21840.0 + }, + "tool_calls": 34, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-03/score.json new file mode 100644 index 0000000..4f4f7b3 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-03/score.json @@ -0,0 +1,141 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.017353050410747528, + "policy_error": null, + "return": 2080.5543068950337, + "seed": 101, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 60.626678078165895, + "x_velocity": -0.20195544825952538 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.018392615020275116, + "policy_error": null, + "return": 2138.706422712988, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 63.44907774332245, + "x_velocity": 0.2710931594036481 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.0175800584256649, + "policy_error": null, + "return": 2176.081199440905, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 65.34472145474142, + "x_velocity": 0.7585728870071762 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.01700751483440399, + "policy_error": null, + "return": 1939.887151650598, + "seed": 401, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 53.53862811029294, + "x_velocity": 1.4427638736164283 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 2176.081199440905, + "max_steps": 1000, + "mean_action_latency_ms": 0.017583309672772884, + "mean_return": 2083.8072701748815, + "mean_x_position": 60.73977634663067, + "min_return": 1939.887151650598, + "policy_path": "submitted_artifact", + "score": 2083.8072701748815 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.016563743352890015, + "policy_error": null, + "return": 2104.812808980764, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 61.7917287401649, + "x_velocity": 1.6486042617829355 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.01746513694524765, + "policy_error": null, + "return": 1688.2270242222276, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 41.068687559404765, + "x_velocity": 1.9024071234746032 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.016550857573747635, + "policy_error": null, + "return": 2157.7229841192566, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 64.80208213222305, + "x_velocity": 0.22495850458341238 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.017036084085702896, + "policy_error": null, + "return": 1720.6714225343967, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 43.03552789392859, + "x_velocity": 1.2683653157641572 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 2157.7229841192566, + "max_steps": 1000, + "mean_action_latency_ms": 0.01690395548939705, + "mean_return": 1917.8585599641613, + "mean_x_position": 52.67450658143032, + "min_return": 1688.2270242222276, + "policy_path": "submitted_artifact", + "score": 1917.8585599641613 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": -11.713338376, + "reference_score": 5942.333244755838, + "score": 2034.0226571116652, + "starter_score": 2443.815838992282, + "unique_policy_evaluations": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-04/metadata.json new file mode 100644 index 0000000..7db93be --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-04/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "digest_compatibility_note": null, + "finished_at": 1784100567.131854, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": -19.696444908, + "observed_raw_score": 1754.7322855694497, + "provider_reasoning_effort": "max", + "raw_score": 1754.7322855694497, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04/tasks-81ec8d98a4", + "task": "simulation_heuristics_ant_v1", + "task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "token_usage": { + "final_input_tokens": 601.0, + "final_output_tokens": 185.0, + "input_tokens": 601.0, + "output_tokens": 185.0, + "total_tokens": 22290.0 + }, + "tool_calls": 28, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-04/score.json new file mode 100644 index 0000000..d056ced --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-04/score.json @@ -0,0 +1,141 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 119.50873717106879, + "policy_error": null, + "return": 1207.6233402008957, + "seed": 101, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 35.19708013114659, + "x_velocity": 4.326067445795161 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 118.88042288413271, + "policy_error": null, + "return": 2224.5950062725537, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 86.86975832747764, + "x_velocity": 0.7609041259428295 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 120.67298788204789, + "policy_error": null, + "return": 2114.4738526999563, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 81.06652564488044, + "x_velocity": 1.8010710504256622 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 118.82429872313514, + "policy_error": null, + "return": 1775.110582039617, + "seed": 401, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 64.35707795946001, + "x_velocity": 3.540525530772527 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 2224.5950062725537, + "max_steps": 1000, + "mean_action_latency_ms": 119.47161166509613, + "mean_return": 1830.4506953032555, + "mean_x_position": 66.87261051574117, + "min_return": 1207.6233402008957, + "policy_path": "submitted_artifact", + "score": 1830.4506953032555 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 120.90635462431237, + "policy_error": null, + "return": 1835.9059030476835, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 66.37537159228599, + "x_velocity": 0.757499837334592 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 124.13967195758596, + "policy_error": null, + "return": 1539.2394495086737, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 52.57240081740299, + "x_velocity": 1.0963517619310892 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 120.76446598116308, + "policy_error": null, + "return": 1636.622896590629, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 56.09285461517814, + "x_velocity": -0.0012046484749816955 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 120.55753978481516, + "policy_error": null, + "return": 1300.4557356152952, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 40.4254284572598, + "x_velocity": 0.888905532432176 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 1835.9059030476835, + "max_steps": 1000, + "mean_action_latency_ms": 121.59200808696914, + "mean_return": 1578.0559961905703, + "mean_x_position": 53.86651387053173, + "min_return": 1300.4557356152952, + "policy_path": "submitted_artifact", + "score": 1578.0559961905703 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": -19.696444908, + "reference_score": 5942.333244755838, + "score": 1754.7322855694497, + "starter_score": 2443.815838992282, + "unique_policy_evaluations": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-05/metadata.json new file mode 100644 index 0000000..dac34cf --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-05/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "digest_compatibility_note": null, + "finished_at": 1784098371.002007, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 5.32283012, + "observed_raw_score": 2630.03597721429, + "provider_reasoning_effort": "max", + "raw_score": 2630.03597721429, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05", + "task": "simulation_heuristics_ant_v1", + "task_digest": "sha256:c338003eab742d732cfd8b1cb13fe3d6a5b8fdd6f58842f6ba2991314d6ab974", + "token_usage": { + "final_input_tokens": 399.0, + "final_output_tokens": 146.0, + "input_tokens": 399.0, + "output_tokens": 146.0, + "total_tokens": 32289.0 + }, + "tool_calls": 44, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-05/score.json new file mode 100644 index 0000000..7a7480b --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/trial-05/score.json @@ -0,0 +1,141 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.021075081080198288, + "policy_error": null, + "return": 2655.214324046493, + "seed": 101, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 87.34248805124248, + "x_velocity": 1.2954003061059893 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.02215452864766121, + "policy_error": null, + "return": 2491.1272178193153, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 78.99889794956486, + "x_velocity": 2.4052931688149215 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.02063225954771042, + "policy_error": null, + "return": 2809.121812321686, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 95.01913506115089, + "x_velocity": 1.277566742010947 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.020623520016670227, + "policy_error": null, + "return": 2489.0478363540433, + "seed": 401, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 78.93877933298222, + "x_velocity": 1.550839455894959 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 2809.121812321686, + "max_steps": 1000, + "mean_action_latency_ms": 0.021121347323060036, + "mean_return": 2611.127797635384, + "mean_x_position": 85.07482509873512, + "min_return": 2489.0478363540433, + "policy_path": "submitted_artifact", + "score": 2611.127797635384 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.021803636103868484, + "policy_error": null, + "return": 2956.346962224474, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 102.33613093046802, + "x_velocity": 2.177513901407906 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.01981833577156067, + "policy_error": null, + "return": 2968.6505183129248, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 103.04152944539119, + "x_velocity": 2.6376314203852758 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.021427400410175323, + "policy_error": null, + "return": 2459.79401547148, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 77.75460994453468, + "x_velocity": 1.8289635675867544 + }, + { + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 0.020313870161771774, + "policy_error": null, + "return": 2311.828755584742, + "seed": 601, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 70.39480461449014, + "x_velocity": 1.4809266915034414 + } + ], + "fall_rate": 0.0, + "invalid_episode_rate": 0.0, + "max_return": 2968.6505183129248, + "max_steps": 1000, + "mean_action_latency_ms": 0.020840810611844063, + "mean_return": 2674.155062898405, + "mean_x_position": 88.381768733721, + "min_return": 2311.828755584742, + "policy_path": "submitted_artifact", + "score": 2674.155062898405 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": 5.32283012, + "reference_score": 5942.333244755838, + "score": 2630.03597721429, + "starter_score": 2443.815838992282, + "unique_policy_evaluations": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/metadata.json index acdbc6b..3b0752f 100644 --- a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:a3ec10ab61a65c03ca17e6460147a3178e222bd1c48964727ea9daae54717ebe", - "digest_compatibility_note": null, - "finished_at": 1783941310.238979, + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", "harness": "opencode", "model": { "display_name": "GPT-5.6 Sol", @@ -12,16 +10,18 @@ "provider_reasoning_effort": "max" }, "normalized_score": 0.0, + "normalized_score_stddev": 0.0, "provider_reasoning_effort": "max", - "source_run_id": "leaderboard/runs/article_suite/20260713T111156Z/gpt-5.6-sol", + "raw_score": 0.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.0, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "task": "simulation_heuristics_atari57_v1", - "task_digest": "sha256:a3ec10ab61a65c03ca17e6460147a3178e222bd1c48964727ea9daae54717ebe", - "token_usage": { - "final_input_tokens": 708.0, - "final_output_tokens": 205.0, - "input_tokens": 708.0, - "output_tokens": 205.0, - "total_tokens": 31121.0 - }, - "tool_calls": 26 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/score.json index 7788108..44c323f 100644 --- a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/score.json @@ -1,15 +1,59 @@ { - "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", - "evaluation": null, - "interaction_budget": { - "completed_trajectories": 0, - "counted_env_steps": 0, - "planned_trajectories": 342, - "target_env_steps": 6840000000 - }, + "benchmark": "learning_beyond_gradients_article_suite", "normalized_score": 0.0, - "protocol_complete": false, + "normalized_score_stddev": 0.0, + "raw_score_imputation_count": 0, "reference_score": 0.8283015254994576, "score": 0.0, - "starter_score": 0.0 + "score_stddev": 0.0, + "starter_score": 0.0, + "task": "simulation_heuristics_atari57_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "trial": 1 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "trial": 2 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "trial": 3 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "trial": 4 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05", + "trial": 5 + } + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-01/metadata.json new file mode 100644 index 0000000..e91beb3 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "digest_compatibility_note": null, + "finished_at": 1784075101.71677, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "max", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "task": "simulation_heuristics_atari57_v1", + "task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "token_usage": { + "final_input_tokens": 708.0, + "final_output_tokens": 204.0, + "input_tokens": 708.0, + "output_tokens": 204.0, + "total_tokens": 45456.0 + }, + "tool_calls": 29, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-01/score.json new file mode 100644 index 0000000..3e4b071 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-01/score.json @@ -0,0 +1,15 @@ +{ + "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", + "evaluation": null, + "interaction_budget": { + "completed_trajectories": 0, + "counted_env_steps": 357852, + "planned_trajectories": 342, + "target_env_steps": 6840000000 + }, + "normalized_score": 0.0, + "protocol_complete": false, + "reference_score": 0.8283015254994576, + "score": 0.0, + "starter_score": 0.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-02/metadata.json new file mode 100644 index 0000000..cc2f047 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "digest_compatibility_note": null, + "finished_at": 1784078883.680485, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "max", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "task": "simulation_heuristics_atari57_v1", + "task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "token_usage": { + "final_input_tokens": 805.0, + "final_output_tokens": 197.0, + "input_tokens": 805.0, + "output_tokens": 197.0, + "total_tokens": 28138.0 + }, + "tool_calls": 30, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-02/score.json new file mode 100644 index 0000000..7788108 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-02/score.json @@ -0,0 +1,15 @@ +{ + "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", + "evaluation": null, + "interaction_budget": { + "completed_trajectories": 0, + "counted_env_steps": 0, + "planned_trajectories": 342, + "target_env_steps": 6840000000 + }, + "normalized_score": 0.0, + "protocol_complete": false, + "reference_score": 0.8283015254994576, + "score": 0.0, + "starter_score": 0.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-03/metadata.json new file mode 100644 index 0000000..179e322 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "digest_compatibility_note": null, + "finished_at": 1784092762.840873, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "max", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "task": "simulation_heuristics_atari57_v1", + "task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "token_usage": { + "final_input_tokens": 603.0, + "final_output_tokens": 219.0, + "input_tokens": 603.0, + "output_tokens": 219.0, + "total_tokens": 33590.0 + }, + "tool_calls": 25, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-03/score.json new file mode 100644 index 0000000..c85df7b --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-03/score.json @@ -0,0 +1,15 @@ +{ + "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", + "evaluation": null, + "interaction_budget": { + "completed_trajectories": 0, + "counted_env_steps": 142698, + "planned_trajectories": 342, + "target_env_steps": 6840000000 + }, + "normalized_score": 0.0, + "protocol_complete": false, + "reference_score": 0.8283015254994576, + "score": 0.0, + "starter_score": 0.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-04/metadata.json new file mode 100644 index 0000000..d2f0725 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-04/metadata.json @@ -0,0 +1,25 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "digest_compatibility_note": null, + "finished_at": 1784095422.450964, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "max", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "task": "simulation_heuristics_atari57_v1", + "task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "token_usage": null, + "tool_calls": 29, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-04/score.json new file mode 100644 index 0000000..7788108 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-04/score.json @@ -0,0 +1,15 @@ +{ + "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", + "evaluation": null, + "interaction_budget": { + "completed_trajectories": 0, + "counted_env_steps": 0, + "planned_trajectories": 342, + "target_env_steps": 6840000000 + }, + "normalized_score": 0.0, + "protocol_complete": false, + "reference_score": 0.8283015254994576, + "score": 0.0, + "starter_score": 0.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-05/metadata.json new file mode 100644 index 0000000..823fee7 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-05/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "digest_compatibility_note": null, + "finished_at": 1784098371.002007, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "max", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05", + "task": "simulation_heuristics_atari57_v1", + "task_digest": "sha256:b3b516b56ca8c457a9abd3b7628d00741db0f22dc261cb174df008fafad35b8c", + "token_usage": { + "final_input_tokens": 806.0, + "final_output_tokens": 202.0, + "input_tokens": 806.0, + "output_tokens": 202.0, + "total_tokens": 32752.0 + }, + "tool_calls": 25, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-05/score.json new file mode 100644 index 0000000..5c6aefa --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/trial-05/score.json @@ -0,0 +1,15 @@ +{ + "disqualification_reason": "Official suite requires 342 completed search-ledger records; found 0.", + "evaluation": null, + "interaction_budget": { + "completed_trajectories": 0, + "counted_env_steps": 379624, + "planned_trajectories": 342, + "target_env_steps": 6840000000 + }, + "normalized_score": 0.0, + "protocol_complete": false, + "reference_score": 0.8283015254994576, + "score": 0.0, + "starter_score": 0.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/metadata.json index 7241d05..ff610e7 100644 --- a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:77215337f89f950f83b8f7436f799959c2ddccd4102552c6302b13d30cc57199", - "digest_compatibility_note": null, - "finished_at": 1783931627.734036, + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", "harness": "opencode", "model": { "display_name": "GPT-5.6 Sol", @@ -11,17 +9,19 @@ "provider": "azure", "provider_reasoning_effort": "max" }, - "normalized_score": 100.0, + "normalized_score": 43.899371069400004, + "normalized_score_stddev": 47.34804435501101, "provider_reasoning_effort": "max", - "source_run_id": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "raw_score": 596.4, + "raw_score_imputation_count": 0, + "raw_score_stddev": 225.8501715739884, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "task": "simulation_heuristics_breakout_ram_v1", - "task_digest": "sha256:77215337f89f950f83b8f7436f799959c2ddccd4102552c6302b13d30cc57199", - "token_usage": { - "final_input_tokens": 536.0, - "final_output_tokens": 151.0, - "input_tokens": 536.0, - "output_tokens": 151.0, - "total_tokens": 34479.0 - }, - "tool_calls": 53 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/score.json index 806d20b..b3465d4 100644 --- a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/score.json @@ -1,80 +1,59 @@ { - "normalized_score": 100.0, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": 43.899371069400004, + "normalized_score_stddev": 47.34804435501101, + "raw_score_imputation_count": 0, "reference_score": 864.0, - "score": 864.0, + "score": 596.4, + "score_stddev": 225.8501715739884, "starter_score": 387.0, - "suites": { - "hidden_nominal": { - "completion_rate": 1.0, - "episodes": [ - { - "final_lives": 5, - "invalid_action": false, - "length": 108000, - "mean_action_latency_ms": 0.021162176155485213, - "observation_mode": "ram", - "policy_error": null, - "return": 864.0, - "seed": 101, - "terminated": false, - "truncated": true, - "variant": "nominal", - "warmup_steps": 0 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 864.0, - "max_steps": 108000, - "mean_action_latency_ms": 0.021162176155485213, - "mean_return": 864.0, - "min_return": 864.0, - "observation_mode": "ram", - "policy_path": "submitted_artifact", - "score": 864.0 + "task": "simulation_heuristics_breakout_ram_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 100.0, + "observed_raw_score": 864.0, + "raw_score": 864.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "trial": 1 }, - "hidden_shifted_start": { - "completion_rate": 1.0, - "episodes": [ - { - "final_lives": 5, - "invalid_action": false, - "length": 108000, - "mean_action_latency_ms": 0.022908275248482823, - "observation_mode": "ram", - "policy_error": null, - "return": 864.0, - "seed": 211, - "terminated": false, - "truncated": true, - "variant": "left_6", - "warmup_steps": 6 - }, - { - "final_lives": 5, - "invalid_action": false, - "length": 108000, - "mean_action_latency_ms": 0.02272433058048288, - "observation_mode": "ram", - "policy_error": null, - "return": 864.0, - "seed": 211, - "terminated": false, - "truncated": true, - "variant": "right_6", - "warmup_steps": 6 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 864.0, - "max_steps": 108000, - "mean_action_latency_ms": 0.022816302914482854, - "mean_return": 864.0, - "min_return": 864.0, - "observation_mode": "ram", - "policy_path": "submitted_artifact", - "score": 864.0 + { + "normalized_score": 6.918238994, + "observed_raw_score": 420.0, + "raw_score": 420.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "trial": 2 + }, + { + "normalized_score": 3.5639413, + "observed_raw_score": 404.0, + "raw_score": 404.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "trial": 3 + }, + { + "normalized_score": 18.448637317, + "observed_raw_score": 475.0, + "raw_score": 475.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "trial": 4 + }, + { + "normalized_score": 90.566037736, + "observed_raw_score": 819.0, + "raw_score": 819.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05", + "trial": 5 } - }, - "target_gap": 0.0, - "target_score": 864.0 + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-01/metadata.json new file mode 100644 index 0000000..896bd23 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "digest_compatibility_note": null, + "finished_at": 1784075101.71677, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 100.0, + "observed_raw_score": 864.0, + "provider_reasoning_effort": "max", + "raw_score": 864.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "task": "simulation_heuristics_breakout_ram_v1", + "task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "token_usage": { + "final_input_tokens": 623.0, + "final_output_tokens": 154.0, + "input_tokens": 623.0, + "output_tokens": 154.0, + "total_tokens": 43785.0 + }, + "tool_calls": 59, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-01/score.json new file mode 100644 index 0000000..136a279 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-01/score.json @@ -0,0 +1,80 @@ +{ + "normalized_score": 100.0, + "reference_score": 864.0, + "score": 864.0, + "starter_score": 387.0, + "suites": { + "hidden_nominal": { + "completion_rate": 1.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 53024, + "mean_action_latency_ms": 0.018633130110050487, + "observation_mode": "ram", + "policy_error": null, + "return": 864.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 864.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.018633130110050487, + "mean_return": 864.0, + "min_return": 864.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 864.0 + }, + "hidden_shifted_start": { + "completion_rate": 1.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 53018, + "mean_action_latency_ms": 0.02392616088074654, + "observation_mode": "ram", + "policy_error": null, + "return": 864.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "left_6", + "warmup_steps": 6 + }, + { + "final_lives": 0, + "invalid_action": false, + "length": 53018, + "mean_action_latency_ms": 0.022209750019665093, + "observation_mode": "ram", + "policy_error": null, + "return": 864.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 864.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.023067955450205818, + "mean_return": 864.0, + "min_return": 864.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 864.0 + } + }, + "target_gap": 0.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-02/metadata.json new file mode 100644 index 0000000..054b318 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "digest_compatibility_note": null, + "finished_at": 1784078883.680485, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 6.918238994, + "observed_raw_score": 420.0, + "provider_reasoning_effort": "max", + "raw_score": 420.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "task": "simulation_heuristics_breakout_ram_v1", + "task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "token_usage": { + "final_input_tokens": 388.0, + "final_output_tokens": 117.0, + "input_tokens": 388.0, + "output_tokens": 117.0, + "total_tokens": 23545.0 + }, + "tool_calls": 39, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-02/score.json new file mode 100644 index 0000000..0aa9b4c --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-02/score.json @@ -0,0 +1,80 @@ +{ + "normalized_score": 6.918238994, + "reference_score": 864.0, + "score": 420.0, + "starter_score": 387.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 16635, + "mean_action_latency_ms": 0.008863277469073325, + "observation_mode": "ram", + "policy_error": null, + "return": 420.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 420.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.008863277469073325, + "mean_return": 420.0, + "min_return": 420.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 420.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 16629, + "mean_action_latency_ms": 0.008907414241568741, + "observation_mode": "ram", + "policy_error": null, + "return": 420.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "left_6", + "warmup_steps": 6 + }, + { + "final_lives": 0, + "invalid_action": false, + "length": 16629, + "mean_action_latency_ms": 0.007831716617765685, + "observation_mode": "ram", + "policy_error": null, + "return": 420.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 420.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.008369565429667214, + "mean_return": 420.0, + "min_return": 420.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 420.0 + } + }, + "target_gap": -444.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-03/metadata.json new file mode 100644 index 0000000..8198f19 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "digest_compatibility_note": null, + "finished_at": 1784092762.840873, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 3.5639413, + "observed_raw_score": 404.0, + "provider_reasoning_effort": "max", + "raw_score": 404.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "task": "simulation_heuristics_breakout_ram_v1", + "task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "token_usage": { + "final_input_tokens": 470.0, + "final_output_tokens": 139.0, + "input_tokens": 470.0, + "output_tokens": 139.0, + "total_tokens": 37473.0 + }, + "tool_calls": 64, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-03/score.json new file mode 100644 index 0000000..ede3d6f --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-03/score.json @@ -0,0 +1,80 @@ +{ + "normalized_score": 3.5639413, + "reference_score": 864.0, + "score": 404.0, + "starter_score": 387.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 1, + "invalid_action": false, + "length": 108000, + "mean_action_latency_ms": 0.0077363104016209645, + "observation_mode": "ram", + "policy_error": null, + "return": 404.0, + "seed": 101, + "terminated": false, + "truncated": true, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 404.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.0077363104016209645, + "mean_return": 404.0, + "min_return": 404.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 404.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 1, + "invalid_action": false, + "length": 108000, + "mean_action_latency_ms": 0.008410930400714278, + "observation_mode": "ram", + "policy_error": null, + "return": 404.0, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "left_6", + "warmup_steps": 6 + }, + { + "final_lives": 1, + "invalid_action": false, + "length": 108000, + "mean_action_latency_ms": 0.00835073453857115, + "observation_mode": "ram", + "policy_error": null, + "return": 404.0, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 404.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.008380832469642715, + "mean_return": 404.0, + "min_return": 404.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 404.0 + } + }, + "target_gap": -460.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-04/metadata.json new file mode 100644 index 0000000..0123d69 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-04/metadata.json @@ -0,0 +1,25 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "digest_compatibility_note": null, + "finished_at": 1784095422.450964, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 18.448637317, + "observed_raw_score": 475.0, + "provider_reasoning_effort": "max", + "raw_score": 475.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "task": "simulation_heuristics_breakout_ram_v1", + "task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "token_usage": null, + "tool_calls": 95, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-04/score.json new file mode 100644 index 0000000..c802e30 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-04/score.json @@ -0,0 +1,80 @@ +{ + "normalized_score": 18.448637317, + "reference_score": 864.0, + "score": 475.0, + "starter_score": 387.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 23884, + "mean_action_latency_ms": 0.005037961625969005, + "observation_mode": "ram", + "policy_error": null, + "return": 475.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 475.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.005037961625969005, + "mean_return": 475.0, + "min_return": 475.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 475.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 23878, + "mean_action_latency_ms": 0.005554480693310147, + "observation_mode": "ram", + "policy_error": null, + "return": 475.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "left_6", + "warmup_steps": 6 + }, + { + "final_lives": 0, + "invalid_action": false, + "length": 23878, + "mean_action_latency_ms": 0.005160711673857908, + "observation_mode": "ram", + "policy_error": null, + "return": 475.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 475.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.005357596183584027, + "mean_return": 475.0, + "min_return": 475.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 475.0 + } + }, + "target_gap": -389.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-05/metadata.json new file mode 100644 index 0000000..c840d3a --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-05/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "digest_compatibility_note": null, + "finished_at": 1784098371.002007, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 90.566037736, + "observed_raw_score": 819.0, + "provider_reasoning_effort": "max", + "raw_score": 819.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05", + "task": "simulation_heuristics_breakout_ram_v1", + "task_digest": "sha256:12eb4ca8814c602a2fbf709f0472c90c8758f3c1ffd0131329b734ca1d8e6b20", + "token_usage": { + "final_input_tokens": 663.0, + "final_output_tokens": 137.0, + "input_tokens": 663.0, + "output_tokens": 137.0, + "total_tokens": 41760.0 + }, + "tool_calls": 62, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-05/score.json new file mode 100644 index 0000000..1137524 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/trial-05/score.json @@ -0,0 +1,80 @@ +{ + "normalized_score": 90.566037736, + "reference_score": 864.0, + "score": 819.0, + "starter_score": 387.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 16581, + "mean_action_latency_ms": 0.004663191669241495, + "observation_mode": "ram", + "policy_error": null, + "return": 819.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 819.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.004663191669241495, + "mean_return": 819.0, + "min_return": 819.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 819.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 16575, + "mean_action_latency_ms": 0.004178550527016682, + "observation_mode": "ram", + "policy_error": null, + "return": 819.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "left_6", + "warmup_steps": 6 + }, + { + "final_lives": 0, + "invalid_action": false, + "length": 16575, + "mean_action_latency_ms": 0.004286238668011684, + "observation_mode": "ram", + "policy_error": null, + "return": 819.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 819.0, + "max_steps": 108000, + "mean_action_latency_ms": 0.004232394597514183, + "mean_return": 819.0, + "min_return": 819.0, + "observation_mode": "ram", + "policy_path": "submitted_artifact", + "score": 819.0 + } + }, + "target_gap": -45.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/metadata.json index 80d6c8c..6f6a8ea 100644 --- a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:7e126380af5345a4bfa0c9d29e45e68b8f1aa20d5d655d12187744100b24441a", - "digest_compatibility_note": null, - "finished_at": 1783931627.734036, + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", "harness": "opencode", "model": { "display_name": "GPT-5.6 Sol", @@ -11,17 +9,19 @@ "provider": "azure", "provider_reasoning_effort": "max" }, - "normalized_score": 70.758122744, + "normalized_score": 60.86642599300001, + "normalized_score_stddev": 22.118506348049813, "provider_reasoning_effort": "max", - "source_run_id": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "raw_score": 647.2, + "raw_score_imputation_count": 0, + "raw_score_stddev": 122.53652516698847, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "task": "simulation_heuristics_breakout_rgb_v1", - "task_digest": "sha256:7e126380af5345a4bfa0c9d29e45e68b8f1aa20d5d655d12187744100b24441a", - "token_usage": { - "final_input_tokens": 708.0, - "final_output_tokens": 118.0, - "input_tokens": 708.0, - "output_tokens": 118.0, - "total_tokens": 21306.0 - }, - "tool_calls": 33 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/score.json index 51a1a30..14560ce 100644 --- a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/score.json @@ -1,66 +1,59 @@ { - "normalized_score": 70.758122744, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": 60.86642599300001, + "normalized_score_stddev": 22.118506348049813, + "raw_score_imputation_count": 0, "reference_score": 864.0, - "score": 702.0, + "score": 647.2, + "score_stddev": 122.53652516698847, "starter_score": 310.0, - "suites": { - "hidden_nominal": { - "completion_rate": 0.0, - "episodes": [ - { - "final_lives": 0, - "invalid_action": false, - "length": 18811, - "mean_action_latency_ms": 0.48869839894144823, - "observation_mode": "rgb", - "policy_error": null, - "return": 702.0, - "seed": 101, - "terminated": true, - "truncated": false, - "variant": "nominal", - "warmup_steps": 0 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 702.0, - "max_steps": 30000, - "mean_action_latency_ms": 0.48869839894144823, - "mean_return": 702.0, - "min_return": 702.0, - "observation_mode": "rgb", - "policy_path": "submitted_artifact", - "score": 702.0 + "task": "simulation_heuristics_breakout_rgb_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 70.758122744, + "observed_raw_score": 702.0, + "raw_score": 702.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "trial": 1 }, - "hidden_shifted_start": { - "completion_rate": 0.0, - "episodes": [ - { - "final_lives": 0, - "invalid_action": false, - "length": 18805, - "mean_action_latency_ms": 0.5275690000493359, - "observation_mode": "rgb", - "policy_error": null, - "return": 702.0, - "seed": 211, - "terminated": true, - "truncated": false, - "variant": "right_6", - "warmup_steps": 6 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 702.0, - "max_steps": 30000, - "mean_action_latency_ms": 0.5275690000493359, - "mean_return": 702.0, - "min_return": 702.0, - "observation_mode": "rgb", - "policy_path": "submitted_artifact", - "score": 702.0 + { + "normalized_score": 70.758122744, + "observed_raw_score": 702.0, + "raw_score": 702.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "trial": 2 + }, + { + "normalized_score": 70.758122744, + "observed_raw_score": 702.0, + "raw_score": 702.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "trial": 3 + }, + { + "normalized_score": 70.758122744, + "observed_raw_score": 702.0, + "raw_score": 702.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "trial": 4 + }, + { + "normalized_score": 21.299638989, + "observed_raw_score": 428.0, + "raw_score": 428.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05", + "trial": 5 } - }, - "target_gap": -162.0, - "target_score": 864.0 + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-01/metadata.json new file mode 100644 index 0000000..a2552db --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "digest_compatibility_note": null, + "finished_at": 1784075101.71677, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 70.758122744, + "observed_raw_score": 702.0, + "provider_reasoning_effort": "max", + "raw_score": 702.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "task": "simulation_heuristics_breakout_rgb_v1", + "task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "token_usage": { + "final_input_tokens": 745.0, + "final_output_tokens": 114.0, + "input_tokens": 745.0, + "output_tokens": 114.0, + "total_tokens": 29531.0 + }, + "tool_calls": 62, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-01/score.json new file mode 100644 index 0000000..4cbc55e --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-01/score.json @@ -0,0 +1,66 @@ +{ + "normalized_score": 70.758122744, + "reference_score": 864.0, + "score": 702.0, + "starter_score": 310.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18811, + "mean_action_latency_ms": 0.44321123277459057, + "observation_mode": "rgb", + "policy_error": null, + "return": 702.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 702.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.44321123277459057, + "mean_return": 702.0, + "min_return": 702.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 702.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18805, + "mean_action_latency_ms": 0.44798922251560325, + "observation_mode": "rgb", + "policy_error": null, + "return": 702.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 702.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.44798922251560325, + "mean_return": 702.0, + "min_return": 702.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 702.0 + } + }, + "target_gap": -162.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-02/metadata.json new file mode 100644 index 0000000..fbf6fb4 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "digest_compatibility_note": null, + "finished_at": 1784078883.680485, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 70.758122744, + "observed_raw_score": 702.0, + "provider_reasoning_effort": "max", + "raw_score": 702.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "task": "simulation_heuristics_breakout_rgb_v1", + "task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "token_usage": { + "final_input_tokens": 741.0, + "final_output_tokens": 135.0, + "input_tokens": 741.0, + "output_tokens": 135.0, + "total_tokens": 24428.0 + }, + "tool_calls": 41, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-02/score.json new file mode 100644 index 0000000..c6d847f --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-02/score.json @@ -0,0 +1,66 @@ +{ + "normalized_score": 70.758122744, + "reference_score": 864.0, + "score": 702.0, + "starter_score": 310.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18811, + "mean_action_latency_ms": 0.44180829986754827, + "observation_mode": "rgb", + "policy_error": null, + "return": 702.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 702.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.44180829986754827, + "mean_return": 702.0, + "min_return": 702.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 702.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18805, + "mean_action_latency_ms": 0.4404976308282402, + "observation_mode": "rgb", + "policy_error": null, + "return": 702.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 702.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.4404976308282402, + "mean_return": 702.0, + "min_return": 702.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 702.0 + } + }, + "target_gap": -162.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-03/metadata.json new file mode 100644 index 0000000..0f27660 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "digest_compatibility_note": null, + "finished_at": 1784092762.840873, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 70.758122744, + "observed_raw_score": 702.0, + "provider_reasoning_effort": "max", + "raw_score": 702.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "task": "simulation_heuristics_breakout_rgb_v1", + "task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "token_usage": { + "final_input_tokens": 557.0, + "final_output_tokens": 134.0, + "input_tokens": 557.0, + "output_tokens": 134.0, + "total_tokens": 32435.0 + }, + "tool_calls": 64, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-03/score.json new file mode 100644 index 0000000..a613b2e --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-03/score.json @@ -0,0 +1,66 @@ +{ + "normalized_score": 70.758122744, + "reference_score": 864.0, + "score": 702.0, + "starter_score": 310.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18811, + "mean_action_latency_ms": 0.4548370397729069, + "observation_mode": "rgb", + "policy_error": null, + "return": 702.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 702.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.4548370397729069, + "mean_return": 702.0, + "min_return": 702.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 702.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18805, + "mean_action_latency_ms": 0.4476128681420456, + "observation_mode": "rgb", + "policy_error": null, + "return": 702.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 702.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.4476128681420456, + "mean_return": 702.0, + "min_return": 702.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 702.0 + } + }, + "target_gap": -162.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-04/metadata.json new file mode 100644 index 0000000..ff38a32 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-04/metadata.json @@ -0,0 +1,25 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "digest_compatibility_note": null, + "finished_at": 1784095422.450964, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 70.758122744, + "observed_raw_score": 702.0, + "provider_reasoning_effort": "max", + "raw_score": 702.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "task": "simulation_heuristics_breakout_rgb_v1", + "task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "token_usage": null, + "tool_calls": 66, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-04/score.json new file mode 100644 index 0000000..d45313b --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-04/score.json @@ -0,0 +1,66 @@ +{ + "normalized_score": 70.758122744, + "reference_score": 864.0, + "score": 702.0, + "starter_score": 310.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18811, + "mean_action_latency_ms": 0.4421635477171263, + "observation_mode": "rgb", + "policy_error": null, + "return": 702.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 702.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.4421635477171263, + "mean_return": 702.0, + "min_return": 702.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 702.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 18805, + "mean_action_latency_ms": 0.44192217272572365, + "observation_mode": "rgb", + "policy_error": null, + "return": 702.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 702.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.44192217272572365, + "mean_return": 702.0, + "min_return": 702.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 702.0 + } + }, + "target_gap": -162.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-05/metadata.json new file mode 100644 index 0000000..5920402 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-05/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "digest_compatibility_note": null, + "finished_at": 1784098371.002007, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 21.299638989, + "observed_raw_score": 428.0, + "provider_reasoning_effort": "max", + "raw_score": 428.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05", + "task": "simulation_heuristics_breakout_rgb_v1", + "task_digest": "sha256:dae0fbd989a518bd7dbdcd1c40d9a347bb6f835ac9cbc50957e7c5a63ec1db1e", + "token_usage": { + "final_input_tokens": 23803.0, + "final_output_tokens": 110.0, + "input_tokens": 23803.0, + "output_tokens": 110.0, + "total_tokens": 23913.0 + }, + "tool_calls": 45, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-05/score.json new file mode 100644 index 0000000..c67e1c3 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/trial-05/score.json @@ -0,0 +1,66 @@ +{ + "normalized_score": 21.299638989, + "reference_score": 864.0, + "score": 428.0, + "starter_score": 310.0, + "suites": { + "hidden_nominal": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 14382, + "mean_action_latency_ms": 0.499704248760338, + "observation_mode": "rgb", + "policy_error": null, + "return": 428.0, + "seed": 101, + "terminated": true, + "truncated": false, + "variant": "nominal", + "warmup_steps": 0 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 428.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.499704248760338, + "mean_return": 428.0, + "min_return": 428.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 428.0 + }, + "hidden_shifted_start": { + "completion_rate": 0.0, + "episodes": [ + { + "final_lives": 0, + "invalid_action": false, + "length": 14376, + "mean_action_latency_ms": 0.49861123625912823, + "observation_mode": "rgb", + "policy_error": null, + "return": 428.0, + "seed": 211, + "terminated": true, + "truncated": false, + "variant": "right_6", + "warmup_steps": 6 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 428.0, + "max_steps": 30000, + "mean_action_latency_ms": 0.49861123625912823, + "mean_return": 428.0, + "min_return": 428.0, + "observation_mode": "rgb", + "policy_path": "submitted_artifact", + "score": 428.0 + } + }, + "target_gap": -436.0, + "target_score": 864.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/metadata.json index 71b273c..536ab9c 100644 --- a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:c7e3b5e466431f4908c1e96cc7de80191f7a7c5ab5a63b3eea4d708f7f447fbf", - "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", - "finished_at": 1783935158.328714, + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", "harness": "opencode", "model": { "display_name": "GPT-5.6 Sol", @@ -11,17 +9,19 @@ "provider": "azure", "provider_reasoning_effort": "max" }, - "normalized_score": 15.873072595, + "normalized_score": 31.8442100886, + "normalized_score_stddev": 3.8873875677992533, "provider_reasoning_effort": "max", - "source_run_id": "leaderboard/runs/article_suite/20260713T084340Z/gpt-5.6-sol", + "raw_score": 6385.135940620764, + "raw_score_imputation_count": 0, + "raw_score_stddev": 291.0085829715705, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "task": "simulation_heuristics_halfcheetah_v1", - "task_digest": "sha256:80c439f53e4ab964f9d7443cd7fb8f25cf6645a0bc288b0496b871c1800ebe78", - "token_usage": { - "final_input_tokens": 670.0, - "final_output_tokens": 192.0, - "input_tokens": 670.0, - "output_tokens": 192.0, - "total_tokens": 19806.0 - }, - "tool_calls": 28 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/score.json index 47fb6eb..522af16 100644 --- a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/score.json @@ -1,98 +1,59 @@ { - "hidden_nominal": { - "episodes": [ - { - "control_cost": 356.07088327407837, - "forward_reward": 5632.4692681876395, - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 88.74183517647907, - "policy_error": null, - "return": 5276.398384913561, - "seed": 211, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": 281.61280558038936, - "x_velocity": 6.870092442627187 - }, - { - "control_cost": 354.76760932803154, - "forward_reward": 5644.934512193883, - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 87.7370754101139, - "policy_error": null, - "return": 5290.16690286585, - "seed": 307, - "terminated": false, - "truncated": true, - "variant": "nominal", - "x_position": 282.2733096756311, - "x_velocity": 5.513374251299842 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 5290.16690286585, - "max_steps": 1000, - "mean_action_latency_ms": 88.23945529329649, - "mean_control_cost": 355.41924630105495, - "mean_forward_reward": 5638.701890190761, - "mean_return": 5283.2826438897055, - "mean_x_position": 281.94305762801025, - "min_return": 5276.398384913561, - "policy_path": "submitted_artifact", - "score": 5283.2826438897055 - }, - "hidden_nominal_weight": 0.7, - "hidden_robustness": { - "episodes": [ - { - "control_cost": 356.7619220018387, - "forward_reward": 6306.763750673376, - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 87.32058335322654, - "policy_error": null, - "return": 5950.001828671538, - "seed": 503, - "terminated": false, - "truncated": true, - "variant": "light_low_friction", - "x_position": 315.32050915627565, - "x_velocity": 6.34920209270735 - }, - { - "control_cost": 354.2664739340544, - "forward_reward": 4345.89047307381, - "invalid_action": false, - "length": 1000, - "mean_action_latency_ms": 89.54298742787796, - "policy_error": null, - "return": 3991.623999139752, - "seed": 503, - "terminated": false, - "truncated": true, - "variant": "heavy_weak_actuators", - "x_position": 217.2768452762976, - "x_velocity": 4.8277047961250705 - } - ], - "invalid_episode_rate": 0.0, - "max_return": 5950.001828671538, - "max_steps": 1000, - "mean_action_latency_ms": 88.43178539055225, - "mean_control_cost": 355.51419796794653, - "mean_forward_reward": 5326.327111873594, - "mean_return": 4970.812913905645, - "mean_x_position": 266.2986772162866, - "min_return": 3991.623999139752, - "policy_path": "submitted_artifact", - "score": 4970.812913905645 - }, - "hidden_robustness_weight": 0.3, - "normalized_score": 15.873072595, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": 31.8442100886, + "normalized_score_stddev": 3.8873875677992533, + "raw_score_imputation_count": 0, "reference_score": 11487.256451471967, - "score": 5189.541724894487, - "starter_score": 4001.2886158917113 + "score": 6385.135940620764, + "score_stddev": 291.0085829715705, + "starter_score": 4001.2886158917113, + "task": "simulation_heuristics_halfcheetah_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 26.12140084, + "observed_raw_score": 5956.728280971269, + "raw_score": 5956.728280971269, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "trial": 1 + }, + { + "normalized_score": 33.28729024, + "observed_raw_score": 6493.16445658394, + "raw_score": 6493.16445658394, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "trial": 2 + }, + { + "normalized_score": 36.605449762, + "observed_raw_score": 6741.560811161888, + "raw_score": 6741.560811161888, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "trial": 3 + }, + { + "normalized_score": 32.792734613, + "observed_raw_score": 6456.142181402096, + "raw_score": 6456.142181402096, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04/tasks-81ec8d98a4", + "trial": 4 + }, + { + "normalized_score": 30.414174988, + "observed_raw_score": 6278.083972984627, + "raw_score": 6278.083972984627, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05", + "trial": 5 + } + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-01/metadata.json new file mode 100644 index 0000000..5f51c1b --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "digest_compatibility_note": null, + "finished_at": 1784075101.71677, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 26.12140084, + "observed_raw_score": 5956.728280971269, + "provider_reasoning_effort": "max", + "raw_score": 5956.728280971269, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "task": "simulation_heuristics_halfcheetah_v1", + "task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "token_usage": { + "final_input_tokens": 666.0, + "final_output_tokens": 258.0, + "input_tokens": 666.0, + "output_tokens": 258.0, + "total_tokens": 18844.0 + }, + "tool_calls": 22, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-01/score.json new file mode 100644 index 0000000..a1559fb --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-01/score.json @@ -0,0 +1,98 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "control_cost": 415.5962042212486, + "forward_reward": 6502.7550215291085, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 11.819874366978183, + "policy_error": null, + "return": 6087.15881730786, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 325.1270932474631, + "x_velocity": 7.454664376824296 + }, + { + "control_cost": 415.76147669553757, + "forward_reward": 6405.91186391007, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 11.884192822035402, + "policy_error": null, + "return": 5990.15038721453, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 320.3221772614403, + "x_velocity": 7.396573180181561 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 6087.15881730786, + "max_steps": 1000, + "mean_action_latency_ms": 11.852033594506793, + "mean_control_cost": 415.6788404583931, + "mean_forward_reward": 6454.33344271959, + "mean_return": 6038.654602261195, + "mean_x_position": 322.7246352544517, + "min_return": 5990.15038721453, + "policy_path": "submitted_artifact", + "score": 6038.654602261195 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "control_cost": 406.1905238777399, + "forward_reward": 7183.572041837751, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 11.910023966571316, + "policy_error": null, + "return": 6777.381517960015, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 359.16092371449446, + "x_velocity": 6.435041780863084 + }, + { + "control_cost": 419.91816034913063, + "forward_reward": 5173.670371645333, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 11.844803968910128, + "policy_error": null, + "return": 4753.752211296202, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 258.6658402048741, + "x_velocity": 5.467666635602541 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 6777.381517960015, + "max_steps": 1000, + "mean_action_latency_ms": 11.877413967740722, + "mean_control_cost": 413.05434211343527, + "mean_forward_reward": 6178.621206741542, + "mean_return": 5765.566864628108, + "mean_x_position": 308.9133819596843, + "min_return": 4753.752211296202, + "policy_path": "submitted_artifact", + "score": 5765.566864628108 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": 26.12140084, + "reference_score": 11487.256451471967, + "score": 5956.728280971269, + "starter_score": 4001.2886158917113 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-02/metadata.json new file mode 100644 index 0000000..5df3418 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "digest_compatibility_note": null, + "finished_at": 1784078883.680485, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 33.28729024, + "observed_raw_score": 6493.16445658394, + "provider_reasoning_effort": "max", + "raw_score": 6493.16445658394, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "task": "simulation_heuristics_halfcheetah_v1", + "task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "token_usage": { + "final_input_tokens": 622.0, + "final_output_tokens": 230.0, + "input_tokens": 622.0, + "output_tokens": 230.0, + "total_tokens": 29012.0 + }, + "tool_calls": 38, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-02/score.json new file mode 100644 index 0000000..dc90ea1 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-02/score.json @@ -0,0 +1,98 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "control_cost": 502.87058797478676, + "forward_reward": 6998.176594556877, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 113.71166357165202, + "policy_error": null, + "return": 6495.306006582092, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 349.8981718988517, + "x_velocity": 7.701369162706499 + }, + { + "control_cost": 502.3115586936474, + "forward_reward": 7177.082110275604, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 113.92410149797797, + "policy_error": null, + "return": 6674.77055158196, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 358.88068957971745, + "x_velocity": 8.290412499586969 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 6674.77055158196, + "max_steps": 1000, + "mean_action_latency_ms": 113.817882534815, + "mean_control_cost": 502.59107333421707, + "mean_forward_reward": 7087.62935241624, + "mean_return": 6585.038279082026, + "mean_x_position": 354.38943073928453, + "min_return": 6495.306006582092, + "policy_path": "submitted_artifact", + "score": 6585.038279082026 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "control_cost": 500.8375596702099, + "forward_reward": 7998.785070339316, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 115.16809724690393, + "policy_error": null, + "return": 7497.9475106691025, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 399.9215751395729, + "x_velocity": 8.264049591045989 + }, + { + "control_cost": 501.7664304077625, + "forward_reward": 5561.403327915474, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 112.91522677824832, + "policy_error": null, + "return": 5059.636897507711, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 278.0524880183809, + "x_velocity": 6.648779737562803 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 7497.9475106691025, + "max_steps": 1000, + "mean_action_latency_ms": 114.04166201257613, + "mean_control_cost": 501.3019950389862, + "mean_forward_reward": 6780.094199127395, + "mean_return": 6278.792204088407, + "mean_x_position": 338.98703157897694, + "min_return": 5059.636897507711, + "policy_path": "submitted_artifact", + "score": 6278.792204088407 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": 33.28729024, + "reference_score": 11487.256451471967, + "score": 6493.16445658394, + "starter_score": 4001.2886158917113 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-03/metadata.json new file mode 100644 index 0000000..89047d6 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "digest_compatibility_note": null, + "finished_at": 1784092762.840873, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 36.605449762, + "observed_raw_score": 6741.560811161888, + "provider_reasoning_effort": "max", + "raw_score": 6741.560811161888, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "task": "simulation_heuristics_halfcheetah_v1", + "task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "token_usage": { + "final_input_tokens": 19401.0, + "final_output_tokens": 208.0, + "input_tokens": 19401.0, + "output_tokens": 208.0, + "total_tokens": 19609.0 + }, + "tool_calls": 28, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-03/score.json new file mode 100644 index 0000000..dde7bf4 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-03/score.json @@ -0,0 +1,98 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "control_cost": 413.70311142131686, + "forward_reward": 7273.056633979042, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 235.5579943174962, + "policy_error": null, + "return": 6859.353522557723, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 363.6421738699595, + "x_velocity": 7.513700414274276 + }, + { + "control_cost": 414.3573304899037, + "forward_reward": 7214.063268255348, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 232.4452954849694, + "policy_error": null, + "return": 6799.705937765444, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 360.7297474787042, + "x_velocity": 7.491474814705178 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 6859.353522557723, + "max_steps": 1000, + "mean_action_latency_ms": 234.0016449012328, + "mean_control_cost": 414.0302209556103, + "mean_forward_reward": 7243.559951117195, + "mean_return": 6829.529730161584, + "mean_x_position": 362.18596067433185, + "min_return": 6799.705937765444, + "policy_path": "submitted_artifact", + "score": 6829.529730161584 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "control_cost": 402.5017847828567, + "forward_reward": 8038.404671273793, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 230.55053228023462, + "policy_error": null, + "return": 7635.902886490934, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 401.90255518629664, + "x_velocity": 10.140904580088216 + }, + { + "control_cost": 424.0220176950097, + "forward_reward": 5860.7191315292675, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 227.22194750304334, + "policy_error": null, + "return": 5436.69711383426, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 293.0182781990708, + "x_velocity": 6.801563749094552 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 7635.902886490934, + "max_steps": 1000, + "mean_action_latency_ms": 228.88623989163898, + "mean_control_cost": 413.2619012389332, + "mean_forward_reward": 6949.56190140153, + "mean_return": 6536.300000162597, + "mean_x_position": 347.4604166926837, + "min_return": 5436.69711383426, + "policy_path": "submitted_artifact", + "score": 6536.300000162597 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": 36.605449762, + "reference_score": 11487.256451471967, + "score": 6741.560811161888, + "starter_score": 4001.2886158917113 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-04/metadata.json new file mode 100644 index 0000000..2e2bfc1 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-04/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "digest_compatibility_note": null, + "finished_at": 1784100567.131854, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 32.792734613, + "observed_raw_score": 6456.142181402096, + "provider_reasoning_effort": "max", + "raw_score": 6456.142181402096, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04/tasks-81ec8d98a4", + "task": "simulation_heuristics_halfcheetah_v1", + "task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "token_usage": { + "final_input_tokens": 345.0, + "final_output_tokens": 211.0, + "input_tokens": 345.0, + "output_tokens": 211.0, + "total_tokens": 19500.0 + }, + "tool_calls": 28, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-04/score.json new file mode 100644 index 0000000..cb62930 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-04/score.json @@ -0,0 +1,98 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "control_cost": 448.42896580696106, + "forward_reward": 6964.515616776878, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 348.3865849059075, + "policy_error": null, + "return": 6516.086650969921, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 348.215123009852, + "x_velocity": 7.1219693932192705 + }, + { + "control_cost": 450.80686481297016, + "forward_reward": 7027.654462952563, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 350.67184246890247, + "policy_error": null, + "return": 6576.847598139597, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 351.4093072135651, + "x_velocity": 7.233277812611013 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 6576.847598139597, + "max_steps": 1000, + "mean_action_latency_ms": 349.529213687405, + "mean_control_cost": 449.6179153099656, + "mean_forward_reward": 6996.085039864721, + "mean_return": 6546.467124554759, + "mean_x_position": 349.81221511170855, + "min_return": 6516.086650969921, + "policy_path": "submitted_artifact", + "score": 6546.467124554759 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "control_cost": 454.4385256394744, + "forward_reward": 7964.6520265067265, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 344.35145055130124, + "policy_error": null, + "return": 7510.213500867252, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 398.2149229479433, + "x_velocity": 8.458794698348157 + }, + { + "control_cost": 448.25406950712204, + "forward_reward": 5428.808530064973, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 343.7038985583931, + "policy_error": null, + "return": 4980.554460557849, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 271.42274812585606, + "x_velocity": 5.967667982744695 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 7510.213500867252, + "max_steps": 1000, + "mean_action_latency_ms": 344.0276745548472, + "mean_control_cost": 451.3462975732982, + "mean_forward_reward": 6696.73027828585, + "mean_return": 6245.3839807125505, + "mean_x_position": 334.81883553689966, + "min_return": 4980.554460557849, + "policy_path": "submitted_artifact", + "score": 6245.3839807125505 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": 32.792734613, + "reference_score": 11487.256451471967, + "score": 6456.142181402096, + "starter_score": 4001.2886158917113 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-05/metadata.json new file mode 100644 index 0000000..2732c2a --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-05/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "digest_compatibility_note": null, + "finished_at": 1784098371.002007, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 30.414174988, + "observed_raw_score": 6278.083972984627, + "provider_reasoning_effort": "max", + "raw_score": 6278.083972984627, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05", + "task": "simulation_heuristics_halfcheetah_v1", + "task_digest": "sha256:aab8a7d4a73765939684373673d03a7811f56e06c4160beee887b3e3e6743d3b", + "token_usage": { + "final_input_tokens": 2487.0, + "final_output_tokens": 213.0, + "input_tokens": 2487.0, + "output_tokens": 213.0, + "total_tokens": 22668.0 + }, + "tool_calls": 27, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-05/score.json new file mode 100644 index 0000000..3919596 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/trial-05/score.json @@ -0,0 +1,98 @@ +{ + "hidden_nominal": { + "episodes": [ + { + "control_cost": 417.77286276221275, + "forward_reward": 6787.942582700975, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 27.935350572690368, + "policy_error": null, + "return": 6370.169719938763, + "seed": 211, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 339.38647130605716, + "x_velocity": 7.2753363280675085 + }, + { + "control_cost": 419.6935954988003, + "forward_reward": 6868.99355346121, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 28.34423817321658, + "policy_error": null, + "return": 6449.299957962409, + "seed": 307, + "terminated": false, + "truncated": true, + "variant": "nominal", + "x_position": 343.47626173899687, + "x_velocity": 7.074323089389433 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 6449.299957962409, + "max_steps": 1000, + "mean_action_latency_ms": 28.139794372953475, + "mean_control_cost": 418.7332291305065, + "mean_forward_reward": 6828.4680680810925, + "mean_return": 6409.734838950586, + "mean_x_position": 341.431366522527, + "min_return": 6370.169719938763, + "policy_path": "submitted_artifact", + "score": 6409.734838950586 + }, + "hidden_nominal_weight": 0.7, + "hidden_robustness": { + "episodes": [ + { + "control_cost": 413.7057405412197, + "forward_reward": 7503.509984149641, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 28.837490713223815, + "policy_error": null, + "return": 7089.804243608423, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "light_low_friction", + "x_position": 375.1578208300892, + "x_velocity": 7.891027106650781 + }, + { + "control_cost": 416.16760087013245, + "forward_reward": 5268.160595389818, + "invalid_action": false, + "length": 1000, + "mean_action_latency_ms": 28.359201481565833, + "policy_error": null, + "return": 4851.992994519687, + "seed": 503, + "terminated": false, + "truncated": true, + "variant": "heavy_weak_actuators", + "x_position": 263.3903513920981, + "x_velocity": 5.685887964065333 + } + ], + "invalid_episode_rate": 0.0, + "max_return": 7089.804243608423, + "max_steps": 1000, + "mean_action_latency_ms": 28.598346097394824, + "mean_control_cost": 414.9366707056761, + "mean_forward_reward": 6385.83528976973, + "mean_return": 5970.898619064055, + "mean_x_position": 319.27408611109365, + "min_return": 4851.992994519687, + "policy_path": "submitted_artifact", + "score": 5970.898619064055 + }, + "hidden_robustness_weight": 0.3, + "normalized_score": 30.414174988, + "reference_score": 11487.256451471967, + "score": 6278.083972984627, + "starter_score": 4001.2886158917113 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/metadata.json index 5033390..86c2774 100644 --- a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:33650c821735e31eea00f2ba46ae09b5d19134820c2ec005e8e96d92cb0b66d0", - "digest_compatibility_note": null, - "finished_at": 1783931627.734036, + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", "harness": "opencode", "model": { "display_name": "GPT-5.6 Sol", @@ -11,17 +9,19 @@ "provider": "azure", "provider_reasoning_effort": "max" }, - "normalized_score": 0.0, + "normalized_score": 74.5, + "normalized_score_stddev": 43.31570154112709, "provider_reasoning_effort": "max", - "source_run_id": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "raw_score": 298.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 173.26280616450836, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "task": "simulation_heuristics_montezuma_v1", - "task_digest": "sha256:33650c821735e31eea00f2ba46ae09b5d19134820c2ec005e8e96d92cb0b66d0", - "token_usage": { - "final_input_tokens": 892.0, - "final_output_tokens": 35.0, - "input_tokens": 892.0, - "output_tokens": 35.0, - "total_tokens": 13215.0 - }, - "tool_calls": 18 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/score.json index bbc2347..445d813 100644 --- a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/score.json @@ -1,216 +1,59 @@ { - "normalized_score": 0.0, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": 74.5, + "normalized_score_stddev": 43.31570154112709, + "raw_score_imputation_count": 0, "reference_score": 400.0, - "score": 0.0, + "score": 298.0, + "score_stddev": 173.26280616450836, "starter_score": 0.0, - "suites": { - "hidden_seeds": { - "evaluation": { - "capped_mean_score": 0.0, - "episodes": [ - { - "bootstrap_steps": 0, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.10412707590148784, - "policy_error": null, - "policy_steps": 2600, - "pre_policy_noops": 0, - "return": 0.0, - "seed": 10007, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "canonical" - }, - { - "bootstrap_steps": 0, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.10631888453365089, - "policy_error": null, - "policy_steps": 2600, - "pre_policy_noops": 0, - "return": 0.0, - "seed": 10103, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "canonical" - } - ], - "invalid_episode_rate": 0.0, - "max_steps": 2600, - "mean_action_latency_ms": 0.10522298021756937, - "mean_return": 0.0, - "policy_path": "submitted_artifact", - "recovery_success_rate": 0.0, - "score": 0.0, - "target_score": 400.0, - "target_success_rate": 0.0 - }, - "score": 0.0, - "weight": 0.1 + "task": "simulation_heuristics_montezuma_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "raw_score": 0.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "trial": 1 }, - "noop_variants": { - "evaluation": { - "capped_mean_score": 0.0, - "episodes": [ - { - "bootstrap_steps": 0, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.10265886916421801, - "policy_error": null, - "policy_steps": 2597, - "pre_policy_noops": 3, - "return": 0.0, - "seed": 10001, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "startup_noop_3" - }, - { - "bootstrap_steps": 0, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.10074814882796873, - "policy_error": null, - "policy_steps": 2587, - "pre_policy_noops": 13, - "return": 0.0, - "seed": 10001, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "startup_noop_13" - } - ], - "invalid_episode_rate": 0.0, - "max_steps": 2600, - "mean_action_latency_ms": 0.10170350899609337, - "mean_return": 0.0, - "policy_path": "submitted_artifact", - "recovery_success_rate": 0.0, - "score": 0.0, - "target_score": 400.0, - "target_success_rate": 0.0 - }, - "score": 0.0, - "weight": 0.1 + { + "normalized_score": 100.0, + "observed_raw_score": 400.0, + "raw_score": 400.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "trial": 2 }, - "recovery": { - "evaluation": { - "capped_mean_score": 0.0, - "episodes": [ - { - "bootstrap_steps": 512, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.0991505951992229, - "policy_error": null, - "policy_steps": 2084, - "pre_policy_noops": 4, - "return": 0.0, - "seed": 10001, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "recover_512_noop_4" - }, - { - "bootstrap_steps": 512, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.09907003707037522, - "policy_error": null, - "policy_steps": 2080, - "pre_policy_noops": 8, - "return": 0.0, - "seed": 10001, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "recover_512_noop_8" - }, - { - "bootstrap_steps": 768, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.11110870922086098, - "policy_error": null, - "policy_steps": 1830, - "pre_policy_noops": 2, - "return": 0.0, - "seed": 10001, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "recover_768_noop_2" - }, - { - "bootstrap_steps": 900, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.10168849173864788, - "policy_error": null, - "policy_steps": 1700, - "pre_policy_noops": 0, - "return": 0.0, - "seed": 10001, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "recover_900" - } - ], - "invalid_episode_rate": 0.0, - "max_steps": 2600, - "mean_action_latency_ms": 0.10275445830727675, - "mean_return": 0.0, - "policy_path": "submitted_artifact", - "recovery_success_rate": 0.0, - "score": 0.0, - "target_score": 400.0, - "target_success_rate": 0.0 - }, - "score": 0.0, - "weight": 0.55 + { + "normalized_score": 100.0, + "observed_raw_score": 400.0, + "raw_score": 400.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "trial": 3 }, - "reproduction": { - "evaluation": { - "capped_mean_score": 0.0, - "episodes": [ - { - "bootstrap_steps": 0, - "invalid_action": false, - "length": 2600, - "mean_action_latency_ms": 0.10425583515405798, - "policy_error": null, - "policy_steps": 2600, - "pre_policy_noops": 0, - "return": 0.0, - "seed": 10001, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "canonical" - } - ], - "invalid_episode_rate": 0.0, - "max_steps": 2600, - "mean_action_latency_ms": 0.10425583515405798, - "mean_return": 0.0, - "policy_path": "submitted_artifact", - "recovery_success_rate": 0.0, - "score": 0.0, - "target_score": 400.0, - "target_success_rate": 0.0 - }, - "score": 0.0, - "weight": 0.25 + { + "normalized_score": 100.0, + "observed_raw_score": 400.0, + "raw_score": 400.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "trial": 4 + }, + { + "normalized_score": 72.5, + "observed_raw_score": 290.0, + "raw_score": 290.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05", + "trial": 5 } - }, - "target_score": 400.0 + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-01/metadata.json new file mode 100644 index 0000000..6065114 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "digest_compatibility_note": null, + "finished_at": 1784075101.71677, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 0.0, + "observed_raw_score": 0.0, + "provider_reasoning_effort": "max", + "raw_score": 0.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "task": "simulation_heuristics_montezuma_v1", + "task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "token_usage": { + "final_input_tokens": 13326.0, + "final_output_tokens": 32.0, + "input_tokens": 13326.0, + "output_tokens": 32.0, + "total_tokens": 41006.0 + }, + "tool_calls": 49, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-01/score.json new file mode 100644 index 0000000..e339367 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-01/score.json @@ -0,0 +1,216 @@ +{ + "normalized_score": 0.0, + "reference_score": 400.0, + "score": 0.0, + "starter_score": 0.0, + "suites": { + "hidden_seeds": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.2316676582603787, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10007, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.13895810071307307, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10103, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.18531287948672587, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "noop_variants": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.21699025431231186, + "policy_error": null, + "policy_steps": 2597, + "pre_policy_noops": 3, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_3" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.1656160129008834, + "policy_error": null, + "policy_steps": 2587, + "pre_policy_noops": 13, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "startup_noop_13" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.19130313360659762, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.1 + }, + "recovery": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.09572550137357468, + "policy_error": null, + "policy_steps": 2084, + "pre_policy_noops": 4, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_4" + }, + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.09617516737377557, + "policy_error": null, + "policy_steps": 2080, + "pre_policy_noops": 8, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_512_noop_8" + }, + { + "bootstrap_steps": 768, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.09545464198425656, + "policy_error": null, + "policy_steps": 1830, + "pre_policy_noops": 2, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_768_noop_2" + }, + { + "bootstrap_steps": 900, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.17371646525776563, + "policy_error": null, + "policy_steps": 1700, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_900" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.11526794399734311, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.55 + }, + "reproduction": { + "evaluation": { + "capped_mean_score": 0.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.24029507731588987, + "policy_error": null, + "policy_steps": 2600, + "pre_policy_noops": 0, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.24029507731588987, + "mean_return": 0.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 0.0, + "target_score": 400.0, + "target_success_rate": 0.0 + }, + "score": 0.0, + "weight": 0.25 + } + }, + "target_score": 400.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-02/metadata.json new file mode 100644 index 0000000..5722190 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "digest_compatibility_note": null, + "finished_at": 1784078883.680485, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 100.0, + "observed_raw_score": 400.0, + "provider_reasoning_effort": "max", + "raw_score": 400.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "task": "simulation_heuristics_montezuma_v1", + "task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "token_usage": { + "final_input_tokens": 753.0, + "final_output_tokens": 138.0, + "input_tokens": 753.0, + "output_tokens": 138.0, + "total_tokens": 41851.0 + }, + "tool_calls": 49, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-02/score.json new file mode 100644 index 0000000..be074ac --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-02/score.json @@ -0,0 +1,216 @@ +{ + "normalized_score": 100.0, + "reference_score": 400.0, + "score": 400.0, + "starter_score": 0.0, + "suites": { + "hidden_seeds": { + "evaluation": { + "capped_mean_score": 400.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.4489696188474473, + "policy_error": null, + "policy_steps": 1769, + "pre_policy_noops": 0, + "return": 400.0, + "seed": 10007, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "canonical" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.012120476530611077, + "policy_error": null, + "policy_steps": 1769, + "pre_policy_noops": 0, + "return": 400.0, + "seed": 10103, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.2305450476890292, + "mean_return": 400.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 400.0, + "target_score": 400.0, + "target_success_rate": 1.0 + }, + "score": 400.0, + "weight": 0.1 + }, + "noop_variants": { + "evaluation": { + "capped_mean_score": 400.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.12976303076981183, + "policy_error": null, + "policy_steps": 1766, + "pre_policy_noops": 3, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "startup_noop_3" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.012106056630628392, + "policy_error": null, + "policy_steps": 1756, + "pre_policy_noops": 13, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "startup_noop_13" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.0709345437002201, + "mean_return": 400.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 1.0, + "score": 400.0, + "target_score": 400.0, + "target_success_rate": 1.0 + }, + "score": 400.0, + "weight": 0.1 + }, + "recovery": { + "evaluation": { + "capped_mean_score": 400.0, + "episodes": [ + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 1773, + "mean_action_latency_ms": 0.026373654809930824, + "policy_error": null, + "policy_steps": 1257, + "pre_policy_noops": 4, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "recover_512_noop_4" + }, + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.018692677363371542, + "policy_error": null, + "policy_steps": 1249, + "pre_policy_noops": 8, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "recover_512_noop_8" + }, + { + "bootstrap_steps": 768, + "invalid_action": false, + "length": 1773, + "mean_action_latency_ms": 0.03854719042622021, + "policy_error": null, + "policy_steps": 1003, + "pre_policy_noops": 2, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "recover_768_noop_2" + }, + { + "bootstrap_steps": 900, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.028355980304823235, + "policy_error": null, + "policy_steps": 869, + "pre_policy_noops": 0, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "recover_900" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.02799237572608645, + "mean_return": 400.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 1.0, + "score": 400.0, + "target_score": 400.0, + "target_success_rate": 1.0 + }, + "score": 400.0, + "weight": 0.55 + }, + "reproduction": { + "evaluation": { + "capped_mean_score": 400.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.037769407298207415, + "policy_error": null, + "policy_steps": 1769, + "pre_policy_noops": 0, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.037769407298207415, + "mean_return": 400.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 400.0, + "target_score": 400.0, + "target_success_rate": 1.0 + }, + "score": 400.0, + "weight": 0.25 + } + }, + "target_score": 400.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-03/metadata.json new file mode 100644 index 0000000..05b24fe --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "digest_compatibility_note": null, + "finished_at": 1784092762.840873, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 100.0, + "observed_raw_score": 400.0, + "provider_reasoning_effort": "max", + "raw_score": 400.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "task": "simulation_heuristics_montezuma_v1", + "task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "token_usage": { + "final_input_tokens": 671.0, + "final_output_tokens": 170.0, + "input_tokens": 671.0, + "output_tokens": 170.0, + "total_tokens": 29001.0 + }, + "tool_calls": 60, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-03/score.json new file mode 100644 index 0000000..a7478bb --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-03/score.json @@ -0,0 +1,216 @@ +{ + "normalized_score": 100.0, + "reference_score": 400.0, + "score": 400.0, + "starter_score": 0.0, + "suites": { + "hidden_seeds": { + "evaluation": { + "capped_mean_score": 400.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.0037183640803773233, + "policy_error": null, + "policy_steps": 1769, + "pre_policy_noops": 0, + "return": 400.0, + "seed": 10007, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "canonical" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.004860456957107483, + "policy_error": null, + "policy_steps": 1769, + "pre_policy_noops": 0, + "return": 400.0, + "seed": 10103, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.004289410518742403, + "mean_return": 400.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 400.0, + "target_score": 400.0, + "target_success_rate": 1.0 + }, + "score": 400.0, + "weight": 0.1 + }, + "noop_variants": { + "evaluation": { + "capped_mean_score": 400.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.012799707876301095, + "policy_error": null, + "policy_steps": 1766, + "pre_policy_noops": 3, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "startup_noop_3" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.013479607334627785, + "policy_error": null, + "policy_steps": 1756, + "pre_policy_noops": 13, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "startup_noop_13" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.01313965760546444, + "mean_return": 400.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 1.0, + "score": 400.0, + "target_score": 400.0, + "target_success_rate": 1.0 + }, + "score": 400.0, + "weight": 0.1 + }, + "recovery": { + "evaluation": { + "capped_mean_score": 400.0, + "episodes": [ + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 1773, + "mean_action_latency_ms": 0.020382573565819382, + "policy_error": null, + "policy_steps": 1257, + "pre_policy_noops": 4, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "recover_512_noop_4" + }, + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.02002937728919393, + "policy_error": null, + "policy_steps": 1249, + "pre_policy_noops": 8, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "recover_512_noop_8" + }, + { + "bootstrap_steps": 768, + "invalid_action": false, + "length": 1773, + "mean_action_latency_ms": 0.024168840112723595, + "policy_error": null, + "policy_steps": 1003, + "pre_policy_noops": 2, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "recover_768_noop_2" + }, + { + "bootstrap_steps": 900, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.024828632491864736, + "policy_error": null, + "policy_steps": 869, + "pre_policy_noops": 0, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "recover_900" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.02235235586490041, + "mean_return": 400.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 1.0, + "score": 400.0, + "target_score": 400.0, + "target_success_rate": 1.0 + }, + "score": 400.0, + "weight": 0.55 + }, + "reproduction": { + "evaluation": { + "capped_mean_score": 400.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.005501331529833288, + "policy_error": null, + "policy_steps": 1769, + "pre_policy_noops": 0, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.005501331529833288, + "mean_return": 400.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 400.0, + "target_score": 400.0, + "target_success_rate": 1.0 + }, + "score": 400.0, + "weight": 0.25 + } + }, + "target_score": 400.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-04/metadata.json new file mode 100644 index 0000000..11f68f4 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-04/metadata.json @@ -0,0 +1,25 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "digest_compatibility_note": null, + "finished_at": 1784095422.450964, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 100.0, + "observed_raw_score": 400.0, + "provider_reasoning_effort": "max", + "raw_score": 400.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "task": "simulation_heuristics_montezuma_v1", + "task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "token_usage": null, + "tool_calls": 47, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-04/score.json new file mode 100644 index 0000000..25e80ec --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-04/score.json @@ -0,0 +1,216 @@ +{ + "normalized_score": 100.0, + "reference_score": 400.0, + "score": 400.0, + "starter_score": 0.0, + "suites": { + "hidden_seeds": { + "evaluation": { + "capped_mean_score": 400.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.03794879639924516, + "policy_error": null, + "policy_steps": 1769, + "pre_policy_noops": 0, + "return": 400.0, + "seed": 10007, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "canonical" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.028250547475510562, + "policy_error": null, + "policy_steps": 1769, + "pre_policy_noops": 0, + "return": 400.0, + "seed": 10103, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.03309967193737786, + "mean_return": 400.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 400.0, + "target_score": 400.0, + "target_success_rate": 1.0 + }, + "score": 400.0, + "weight": 0.1 + }, + "noop_variants": { + "evaluation": { + "capped_mean_score": 400.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 1773, + "mean_action_latency_ms": 0.3512181140387715, + "policy_error": null, + "policy_steps": 1770, + "pre_policy_noops": 3, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "startup_noop_3" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 1785, + "mean_action_latency_ms": 0.037615131754974745, + "policy_error": null, + "policy_steps": 1772, + "pre_policy_noops": 13, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "startup_noop_13" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.19441662289687311, + "mean_return": 400.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 1.0, + "score": 400.0, + "target_score": 400.0, + "target_success_rate": 1.0 + }, + "score": 400.0, + "weight": 0.1 + }, + "recovery": { + "evaluation": { + "capped_mean_score": 400.0, + "episodes": [ + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.03204773739872222, + "policy_error": null, + "policy_steps": 1253, + "pre_policy_noops": 4, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "recover_512_noop_4" + }, + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.5535182991982985, + "policy_error": null, + "policy_steps": 1249, + "pre_policy_noops": 8, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "recover_512_noop_8" + }, + { + "bootstrap_steps": 768, + "invalid_action": false, + "length": 1773, + "mean_action_latency_ms": 0.045125789853074735, + "policy_error": null, + "policy_steps": 1003, + "pre_policy_noops": 2, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "recover_768_noop_2" + }, + { + "bootstrap_steps": 900, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.05509714800671549, + "policy_error": null, + "policy_steps": 869, + "pre_policy_noops": 0, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "recover_900" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.17144724361420274, + "mean_return": 400.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 1.0, + "score": 400.0, + "target_score": 400.0, + "target_success_rate": 1.0 + }, + "score": 400.0, + "weight": 0.55 + }, + "reproduction": { + "evaluation": { + "capped_mean_score": 400.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.1465841797967544, + "policy_error": null, + "policy_steps": 1769, + "pre_policy_noops": 0, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.1465841797967544, + "mean_return": 400.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 400.0, + "target_score": 400.0, + "target_success_rate": 1.0 + }, + "score": 400.0, + "weight": 0.25 + } + }, + "target_score": 400.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-05/metadata.json new file mode 100644 index 0000000..71fde67 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-05/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "digest_compatibility_note": null, + "finished_at": 1784098371.002007, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 72.5, + "observed_raw_score": 290.0, + "provider_reasoning_effort": "max", + "raw_score": 290.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05", + "task": "simulation_heuristics_montezuma_v1", + "task_digest": "sha256:518afe1cd1a1794ad99057a695c46008f38ebee53f9ed7e2dce017cdecc08dca", + "token_usage": { + "final_input_tokens": 651.0, + "final_output_tokens": 236.0, + "input_tokens": 651.0, + "output_tokens": 236.0, + "total_tokens": 58743.0 + }, + "tool_calls": 64, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-05/score.json new file mode 100644 index 0000000..25f71ee --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/trial-05/score.json @@ -0,0 +1,216 @@ +{ + "normalized_score": 72.5, + "reference_score": 400.0, + "score": 290.0, + "starter_score": 0.0, + "suites": { + "hidden_seeds": { + "evaluation": { + "capped_mean_score": 400.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.3241568813263662, + "policy_error": null, + "policy_steps": 1769, + "pre_policy_noops": 0, + "return": 400.0, + "seed": 10007, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "canonical" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.32422510017848843, + "policy_error": null, + "policy_steps": 1769, + "pre_policy_noops": 0, + "return": 400.0, + "seed": 10103, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.3241909907524273, + "mean_return": 400.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 400.0, + "target_score": 400.0, + "target_success_rate": 1.0 + }, + "score": 400.0, + "weight": 0.1 + }, + "noop_variants": { + "evaluation": { + "capped_mean_score": 400.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.3241713796631514, + "policy_error": null, + "policy_steps": 1766, + "pre_policy_noops": 3, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "startup_noop_3" + }, + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.32027439065256363, + "policy_error": null, + "policy_steps": 1756, + "pre_policy_noops": 13, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "startup_noop_13" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.3222228851578575, + "mean_return": 400.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 1.0, + "score": 400.0, + "target_score": 400.0, + "target_success_rate": 1.0 + }, + "score": 400.0, + "weight": 0.1 + }, + "recovery": { + "evaluation": { + "capped_mean_score": 200.0, + "episodes": [ + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.3290024676329716, + "policy_error": null, + "policy_steps": 1253, + "pre_policy_noops": 4, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "recover_512_noop_4" + }, + { + "bootstrap_steps": 512, + "invalid_action": false, + "length": 1677, + "mean_action_latency_ms": 0.3302508102634933, + "policy_error": null, + "policy_steps": 1157, + "pre_policy_noops": 8, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "recover_512_noop_8" + }, + { + "bootstrap_steps": 768, + "invalid_action": false, + "length": 2600, + "mean_action_latency_ms": 0.32738516733172485, + "policy_error": null, + "policy_steps": 1830, + "pre_policy_noops": 2, + "return": 0.0, + "seed": 10001, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "recover_768_noop_2" + }, + { + "bootstrap_steps": 900, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.33947648250115275, + "policy_error": null, + "policy_steps": 869, + "pre_policy_noops": 0, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "recover_900" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.3315287319323356, + "mean_return": 200.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.5, + "score": 200.0, + "target_score": 400.0, + "target_success_rate": 0.5 + }, + "score": 200.0, + "weight": 0.55 + }, + "reproduction": { + "evaluation": { + "capped_mean_score": 400.0, + "episodes": [ + { + "bootstrap_steps": 0, + "invalid_action": false, + "length": 1769, + "mean_action_latency_ms": 0.32747153970878834, + "policy_error": null, + "policy_steps": 1769, + "pre_policy_noops": 0, + "return": 400.0, + "seed": 10001, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "canonical" + } + ], + "invalid_episode_rate": 0.0, + "max_steps": 2600, + "mean_action_latency_ms": 0.32747153970878834, + "mean_return": 400.0, + "policy_path": "submitted_artifact", + "recovery_success_rate": 0.0, + "score": 400.0, + "target_score": 400.0, + "target_success_rate": 1.0 + }, + "score": 400.0, + "weight": 0.25 + } + }, + "target_score": 400.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/metadata.json index 19ace4e..3b9c969 100644 --- a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:f0b6f90f1d8072e75f0e10f90f4b51e51e31b3590bb2c2736778ba155b23389f", - "digest_compatibility_note": null, - "finished_at": 1783931627.734036, + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", "harness": "opencode", "model": { "display_name": "GPT-5.6 Sol", @@ -11,17 +9,19 @@ "provider": "azure", "provider_reasoning_effort": "max" }, - "normalized_score": 52.916666667, + "normalized_score": 91.6666666666, + "normalized_score_stddev": 9.771699499561077, "provider_reasoning_effort": "max", - "source_run_id": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "raw_score": 19.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 2.345207879911715, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "task": "simulation_heuristics_pong_ram_v1", - "task_digest": "sha256:f0b6f90f1d8072e75f0e10f90f4b51e51e31b3590bb2c2736778ba155b23389f", - "token_usage": { - "final_input_tokens": 530.0, - "final_output_tokens": 124.0, - "input_tokens": 530.0, - "output_tokens": 124.0, - "total_tokens": 19598.0 - }, - "tool_calls": 39 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/score.json index 8c61862..b9b9520 100644 --- a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/score.json @@ -1,133 +1,59 @@ { - "normalized_score": 52.916666667, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": 91.6666666666, + "normalized_score_stddev": 9.771699499561077, + "raw_score_imputation_count": 0, "reference_score": 21.0, - "score": 9.7, + "score": 19.0, + "score_stddev": 2.345207879911715, "starter_score": -2.9999999999999996, - "suite_weights": { - "hidden_nominal": 0.7, - "hidden_reset_robustness": 0.3 - }, - "suites": { - "hidden_nominal": { - "episodes": [ - { - "invalid_action": false, - "length": 27000, - "mean_action_latency_ms": 0.0035600767867570676, - "points_against": 2, - "points_for": 11, - "policy_error": null, - "score": 9.0, - "seed": 101, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "article_nominal", - "won": true - }, - { - "invalid_action": false, - "length": 27000, - "mean_action_latency_ms": 0.0040982644573819855, - "points_against": 2, - "points_for": 11, - "policy_error": null, - "score": 9.0, - "seed": 211, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "article_nominal", - "won": true - }, - { - "invalid_action": false, - "length": 27000, - "mean_action_latency_ms": 0.004250124277960923, - "points_against": 2, - "points_for": 11, - "policy_error": null, - "score": 9.0, - "seed": 307, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "article_nominal", - "won": true - } - ], - "invalid_episode_rate": 0.0, - "max_score": 9.0, - "max_steps": 27000, - "mean_action_latency_ms": 0.003969488507366658, - "mean_score": 9.0, - "min_score": 9.0, - "policy_path": "submitted_artifact", - "score": 9.0, - "target_score": 21.0, - "target_score_rate": 0.0, - "win_rate": 1.0 + "task": "simulation_heuristics_pong_ram_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 79.166666667, + "observed_raw_score": 16.0, + "raw_score": 16.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "trial": 1 }, - "hidden_reset_robustness": { - "episodes": [ - { - "invalid_action": false, - "length": 27000, - "mean_action_latency_ms": 0.004065527891119321, - "points_against": 2, - "points_for": 11, - "policy_error": null, - "score": 9.0, - "seed": 503, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "randomized_reset", - "won": true - }, - { - "invalid_action": false, - "length": 27000, - "mean_action_latency_ms": 0.004138178468026497, - "points_against": 0, - "points_for": 4, - "policy_error": null, - "score": 4.0, - "seed": 601, - "target_reached": false, - "terminated": false, - "truncated": true, - "variant": "randomized_reset", - "won": true - }, - { - "invalid_action": false, - "length": 26459, - "mean_action_latency_ms": 0.003632126502959365, - "points_against": 0, - "points_for": 21, - "policy_error": null, - "score": 21.0, - "seed": 701, - "target_reached": true, - "terminated": true, - "truncated": false, - "variant": "randomized_reset", - "won": true - } - ], - "invalid_episode_rate": 0.0, - "max_score": 21.0, - "max_steps": 27000, - "mean_action_latency_ms": 0.0039452776207017275, - "mean_score": 11.333333333333334, - "min_score": 4.0, - "policy_path": "submitted_artifact", - "score": 11.333333333333334, - "target_score": 21.0, - "target_score_rate": 0.3333333333333333, - "win_rate": 1.0 + { + "normalized_score": 100.0, + "observed_raw_score": 21.0, + "raw_score": 21.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "trial": 2 + }, + { + "normalized_score": 83.333333333, + "observed_raw_score": 17.0, + "raw_score": 17.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "trial": 3 + }, + { + "normalized_score": 95.833333333, + "observed_raw_score": 20.0, + "raw_score": 20.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "trial": 4 + }, + { + "normalized_score": 100.0, + "observed_raw_score": 21.0, + "raw_score": 21.0, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05", + "trial": 5 } - }, - "target_score": 21.0 + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-01/metadata.json new file mode 100644 index 0000000..370e532 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "digest_compatibility_note": null, + "finished_at": 1784075101.71677, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 79.166666667, + "observed_raw_score": 16.0, + "provider_reasoning_effort": "max", + "raw_score": 16.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "task": "simulation_heuristics_pong_ram_v1", + "task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "token_usage": { + "final_input_tokens": 789.0, + "final_output_tokens": 132.0, + "input_tokens": 789.0, + "output_tokens": 132.0, + "total_tokens": 33689.0 + }, + "tool_calls": 39, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-01/score.json new file mode 100644 index 0000000..b9d786b --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-01/score.json @@ -0,0 +1,133 @@ +{ + "normalized_score": 79.166666667, + "reference_score": 21.0, + "score": 16.0, + "starter_score": -2.9999999999999996, + "suite_weights": { + "hidden_nominal": 0.7, + "hidden_reset_robustness": 0.3 + }, + "suites": { + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.004031659992135785, + "points_against": 0, + "points_for": 16, + "policy_error": null, + "score": 16.0, + "seed": 101, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.0032283716395290363, + "points_against": 0, + "points_for": 16, + "policy_error": null, + "score": 16.0, + "seed": 211, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.00395186485601934, + "points_against": 0, + "points_for": 16, + "policy_error": null, + "score": 16.0, + "seed": 307, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "article_nominal", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 16.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.003737298829228054, + "mean_score": 16.0, + "min_score": 16.0, + "policy_path": "submitted_artifact", + "score": 16.0, + "target_score": 21.0, + "target_score_rate": 0.0, + "win_rate": 1.0 + }, + "hidden_reset_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.004118698455082874, + "points_against": 0, + "points_for": 16, + "policy_error": null, + "score": 16.0, + "seed": 503, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.003954343411519572, + "points_against": 0, + "points_for": 16, + "policy_error": null, + "score": 16.0, + "seed": 601, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 27000, + "mean_action_latency_ms": 0.003702660970803764, + "points_against": 0, + "points_for": 16, + "policy_error": null, + "score": 16.0, + "seed": 701, + "target_reached": false, + "terminated": false, + "truncated": true, + "variant": "randomized_reset", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 16.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.003925234279135403, + "mean_score": 16.0, + "min_score": 16.0, + "policy_path": "submitted_artifact", + "score": 16.0, + "target_score": 21.0, + "target_score_rate": 0.0, + "win_rate": 1.0 + } + }, + "target_score": 21.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-02/metadata.json new file mode 100644 index 0000000..9b4ad1c --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "digest_compatibility_note": null, + "finished_at": 1784078883.680485, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 100.0, + "observed_raw_score": 21.0, + "provider_reasoning_effort": "max", + "raw_score": 21.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "task": "simulation_heuristics_pong_ram_v1", + "task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "token_usage": { + "final_input_tokens": 541.0, + "final_output_tokens": 166.0, + "input_tokens": 541.0, + "output_tokens": 166.0, + "total_tokens": 26307.0 + }, + "tool_calls": 29, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-02/score.json new file mode 100644 index 0000000..21996c2 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-02/score.json @@ -0,0 +1,133 @@ +{ + "normalized_score": 100.0, + "reference_score": 21.0, + "score": 21.0, + "starter_score": -2.9999999999999996, + "suite_weights": { + "hidden_nominal": 0.7, + "hidden_reset_robustness": 0.3 + }, + "suites": { + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 13022, + "mean_action_latency_ms": 0.00400013646725671, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 101, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 13022, + "mean_action_latency_ms": 0.0038056247733902994, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 211, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 13022, + "mean_action_latency_ms": 0.003781334522495373, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 307, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 21.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.003862365254380794, + "mean_score": 21.0, + "min_score": 21.0, + "policy_path": "submitted_artifact", + "score": 21.0, + "target_score": 21.0, + "target_score_rate": 1.0, + "win_rate": 1.0 + }, + "hidden_reset_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 13022, + "mean_action_latency_ms": 0.0030782288812201586, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 503, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 13018, + "mean_action_latency_ms": 0.0036527639285547243, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 601, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 13019, + "mean_action_latency_ms": 0.003410475861147528, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 701, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 21.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.003380489556974137, + "mean_score": 21.0, + "min_score": 21.0, + "policy_path": "submitted_artifact", + "score": 21.0, + "target_score": 21.0, + "target_score_rate": 1.0, + "win_rate": 1.0 + } + }, + "target_score": 21.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-03/metadata.json new file mode 100644 index 0000000..6d93fc6 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "digest_compatibility_note": null, + "finished_at": 1784092762.840873, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 83.333333333, + "observed_raw_score": 17.0, + "provider_reasoning_effort": "max", + "raw_score": 17.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "task": "simulation_heuristics_pong_ram_v1", + "task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "token_usage": { + "final_input_tokens": 492.0, + "final_output_tokens": 162.0, + "input_tokens": 492.0, + "output_tokens": 162.0, + "total_tokens": 39566.0 + }, + "tool_calls": 57, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-03/score.json new file mode 100644 index 0000000..45ac878 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-03/score.json @@ -0,0 +1,133 @@ +{ + "normalized_score": 83.333333333, + "reference_score": 21.0, + "score": 17.0, + "starter_score": -2.9999999999999996, + "suite_weights": { + "hidden_nominal": 0.7, + "hidden_reset_robustness": 0.3 + }, + "suites": { + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 15718, + "mean_action_latency_ms": 0.002908375447968886, + "points_against": 4, + "points_for": 21, + "policy_error": null, + "score": 17.0, + "seed": 101, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 15718, + "mean_action_latency_ms": 0.0028746912026815074, + "points_against": 4, + "points_for": 21, + "policy_error": null, + "score": 17.0, + "seed": 211, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 15718, + "mean_action_latency_ms": 0.00319398247931574, + "points_against": 4, + "points_for": 21, + "policy_error": null, + "score": 17.0, + "seed": 307, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 17.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.0029923497099887107, + "mean_score": 17.0, + "min_score": 17.0, + "policy_path": "submitted_artifact", + "score": 17.0, + "target_score": 21.0, + "target_score_rate": 0.0, + "win_rate": 1.0 + }, + "hidden_reset_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 15718, + "mean_action_latency_ms": 0.0024105928788362855, + "points_against": 4, + "points_for": 21, + "policy_error": null, + "score": 17.0, + "seed": 503, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 15714, + "mean_action_latency_ms": 0.0024329361331442757, + "points_against": 4, + "points_for": 21, + "policy_error": null, + "score": 17.0, + "seed": 601, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 15715, + "mean_action_latency_ms": 0.0024574487865672836, + "points_against": 4, + "points_for": 21, + "policy_error": null, + "score": 17.0, + "seed": 701, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 17.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.002433659266182615, + "mean_score": 17.0, + "min_score": 17.0, + "policy_path": "submitted_artifact", + "score": 17.0, + "target_score": 21.0, + "target_score_rate": 0.0, + "win_rate": 1.0 + } + }, + "target_score": 21.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-04/metadata.json new file mode 100644 index 0000000..dc48679 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-04/metadata.json @@ -0,0 +1,25 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "digest_compatibility_note": null, + "finished_at": 1784095422.450964, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 95.833333333, + "observed_raw_score": 20.0, + "provider_reasoning_effort": "max", + "raw_score": 20.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "task": "simulation_heuristics_pong_ram_v1", + "task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "token_usage": null, + "tool_calls": 37, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-04/score.json new file mode 100644 index 0000000..21a440e --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-04/score.json @@ -0,0 +1,133 @@ +{ + "normalized_score": 95.833333333, + "reference_score": 21.0, + "score": 20.0, + "starter_score": -2.9999999999999996, + "suite_weights": { + "hidden_nominal": 0.7, + "hidden_reset_robustness": 0.3 + }, + "suites": { + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 23278, + "mean_action_latency_ms": 0.0029596734219629597, + "points_against": 1, + "points_for": 21, + "policy_error": null, + "score": 20.0, + "seed": 101, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 23278, + "mean_action_latency_ms": 0.0030998745725129835, + "points_against": 1, + "points_for": 21, + "policy_error": null, + "score": 20.0, + "seed": 211, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 23278, + "mean_action_latency_ms": 0.0028895405471623865, + "points_against": 1, + "points_for": 21, + "policy_error": null, + "score": 20.0, + "seed": 307, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 20.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.002983029513879443, + "mean_score": 20.0, + "min_score": 20.0, + "policy_path": "submitted_artifact", + "score": 20.0, + "target_score": 21.0, + "target_score_rate": 0.0, + "win_rate": 1.0 + }, + "hidden_reset_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 23278, + "mean_action_latency_ms": 0.002942564800618192, + "points_against": 1, + "points_for": 21, + "policy_error": null, + "score": 20.0, + "seed": 503, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 19758, + "mean_action_latency_ms": 0.0032649286893496284, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 601, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 26541, + "mean_action_latency_ms": 0.0030508183122106786, + "points_against": 2, + "points_for": 21, + "policy_error": null, + "score": 19.0, + "seed": 701, + "target_reached": false, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 21.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.0030861039340594996, + "mean_score": 20.0, + "min_score": 19.0, + "policy_path": "submitted_artifact", + "score": 20.0, + "target_score": 21.0, + "target_score_rate": 0.3333333333333333, + "win_rate": 1.0 + } + }, + "target_score": 21.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-05/metadata.json new file mode 100644 index 0000000..fefdec2 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-05/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "digest_compatibility_note": null, + "finished_at": 1784098371.002007, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 100.0, + "observed_raw_score": 21.0, + "provider_reasoning_effort": "max", + "raw_score": 21.0, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05", + "task": "simulation_heuristics_pong_ram_v1", + "task_digest": "sha256:23dc66f3d54d291790ea8a04459e3848966a82854ac3bea5107b0b5240e74457", + "token_usage": { + "final_input_tokens": 811.0, + "final_output_tokens": 194.0, + "input_tokens": 811.0, + "output_tokens": 194.0, + "total_tokens": 22509.0 + }, + "tool_calls": 20, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-05/score.json new file mode 100644 index 0000000..3ae9f13 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/trial-05/score.json @@ -0,0 +1,133 @@ +{ + "normalized_score": 100.0, + "reference_score": 21.0, + "score": 21.0, + "starter_score": -2.9999999999999996, + "suite_weights": { + "hidden_nominal": 0.7, + "hidden_reset_robustness": 0.3 + }, + "suites": { + "hidden_nominal": { + "episodes": [ + { + "invalid_action": false, + "length": 13022, + "mean_action_latency_ms": 0.0030071182777803903, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 101, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 13022, + "mean_action_latency_ms": 0.00275014456073453, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 211, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + }, + { + "invalid_action": false, + "length": 13022, + "mean_action_latency_ms": 0.002694509229650674, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 307, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "article_nominal", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 21.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.002817257356055198, + "mean_score": 21.0, + "min_score": 21.0, + "policy_path": "submitted_artifact", + "score": 21.0, + "target_score": 21.0, + "target_score_rate": 1.0, + "win_rate": 1.0 + }, + "hidden_reset_robustness": { + "episodes": [ + { + "invalid_action": false, + "length": 13022, + "mean_action_latency_ms": 0.0029419475734820885, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 503, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 13018, + "mean_action_latency_ms": 0.0027743870923849345, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 601, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + }, + { + "invalid_action": false, + "length": 13019, + "mean_action_latency_ms": 0.0028401848212186664, + "points_against": 0, + "points_for": 21, + "policy_error": null, + "score": 21.0, + "seed": 701, + "target_reached": true, + "terminated": true, + "truncated": false, + "variant": "randomized_reset", + "won": true + } + ], + "invalid_episode_rate": 0.0, + "max_score": 21.0, + "max_steps": 27000, + "mean_action_latency_ms": 0.0028521731623618964, + "mean_score": 21.0, + "min_score": 21.0, + "policy_path": "submitted_artifact", + "score": 21.0, + "target_score": 21.0, + "target_score_rate": 1.0, + "win_rate": 1.0 + } + }, + "target_score": 21.0 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/metadata.json index acad2e8..05ddaab 100644 --- a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:316e198d8fa71e705c8b7fa54f41faf64e1f3c0ba5c4c92b8116b4bf78c5712b", - "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", - "finished_at": 1783938003.487764, + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", "harness": "opencode", "model": { "display_name": "GPT-5.6 Sol", @@ -11,17 +9,19 @@ "provider": "azure", "provider_reasoning_effort": "max" }, - "normalized_score": 49.409889366, + "normalized_score": 56.834663982799995, + "normalized_score_stddev": 33.111283305681766, "provider_reasoning_effort": "max", - "source_run_id": "leaderboard/runs/article_suite/20260713T101636Z/gpt-5.6-sol", + "raw_score": 0.6554319871515035, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.12456464290218099, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "task": "simulation_heuristics_vizdoom_d1_v1", - "task_digest": "sha256:b3431b238bec4e66af6189d30bcd15d5cd227144dfe8bf0dd1011aa5416c1436", - "token_usage": { - "final_input_tokens": 589.0, - "final_output_tokens": 136.0, - "input_tokens": 589.0, - "output_tokens": 136.0, - "total_tokens": 22229.0 - }, - "tool_calls": 25 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/score.json index d210cac..0ebcf21 100644 --- a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/score.json @@ -1,211 +1,59 @@ { - "anchor_calibration": { - "envpool_version": "1.1.1", - "mode": "numeric", - "platform": "linux-x86_64" - }, - "isolation": { - "mode": "subprocess_per_suite", - "suite_process_count": 2, - "unique_working_directories": true - }, - "normalized_score": 49.409889366, - "reference_score": 0.8178199748694897, - "resource_cleanup": { - "removed_bytes": 0, - "removed_resources": 0, - "runs": [ - { - "removed_bytes": 0, - "removed_resources": 0, - "terminated_processes": 0 - }, - { - "removed_bytes": 0, - "removed_resources": 0, - "terminated_processes": 0 - } - ], - "terminated_processes": 0 - }, - "score": 0.62749998614192, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": 56.834663982799995, + "normalized_score_stddev": 33.111283305681766, + "raw_score_imputation_count": 0, + "reference_score": 0.8178199748694898, + "score": 0.6554319871515035, + "score_stddev": 0.12456464290218099, "starter_score": 0.44161998964846133, - "suites": { - "hidden_nominal": { - "batch_seed": 101, - "envpool_version": "1.1.1", - "episodes": [ - { - "final_variables": { - "HEALTH": 19.0 - }, - "invalid_action": false, - "lane": 0, - "length": 2100, - "mean_action_latency_ms": 0.17381823754736356, - "policy_error": null, - "return": 0.6429999805986881, - "seed": 101, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "HEALTH": 3.0 - }, - "invalid_action": false, - "lane": 1, - "length": 1088, - "mean_action_latency_ms": 0.14961455770603874, - "policy_error": null, - "return": 0.08299999311566353, - "seed": 102, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "HEALTH": 22.0 - }, - "invalid_action": false, - "lane": 2, - "length": 2100, - "mean_action_latency_ms": 0.15601739109981627, - "policy_error": null, - "return": 0.6899999864399433, - "seed": 103, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "HEALTH": 71.0 - }, - "invalid_action": false, - "lane": 3, - "length": 2100, - "mean_action_latency_ms": 0.1531444631871723, - "policy_error": null, - "return": 0.9349999837577343, - "seed": 104, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "HEALTH": 47.0 - }, - "invalid_action": false, - "lane": 4, - "length": 2100, - "mean_action_latency_ms": 0.15155099775819553, - "policy_error": null, - "return": 0.8149999864399433, - "seed": 105, - "terminated": false, - "truncated": true - } - ], - "frame_skip": 1, - "invalid_episode_rate": 0.0, - "max_return": 0.9349999837577343, - "max_steps": 2100, - "mean_action_latency_ms": 0.15682912945971728, - "mean_return": 0.6331999860703945, - "min_return": 0.08299999311566353, - "policy_path": "submitted_artifact", - "render_height": 180, - "render_width": 240, - "scenario": "d1", - "score": 0.6331999860703945 + "task": "simulation_heuristics_vizdoom_d1_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 42.828284045, + "observed_raw_score": 0.6027399878948927, + "raw_score": 0.6027399878948927, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "trial": 1 }, - "hidden_secondary": { - "batch_seed": 607, - "envpool_version": "1.1.1", - "episodes": [ - { - "final_variables": { - "HEALTH": 38.0 - }, - "invalid_action": false, - "lane": 0, - "length": 2100, - "mean_action_latency_ms": 0.14928735703939483, - "policy_error": null, - "return": 0.769999984651804, - "seed": 607, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "HEALTH": 47.0 - }, - "invalid_action": false, - "lane": 1, - "length": 2100, - "mean_action_latency_ms": 0.13680784031748772, - "policy_error": null, - "return": 0.7989999875426292, - "seed": 608, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "HEALTH": 37.0 - }, - "invalid_action": false, - "lane": 2, - "length": 2100, - "mean_action_latency_ms": 0.13356554454990796, - "policy_error": null, - "return": 0.764999981969595, - "seed": 609, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "HEALTH": 47.0 - }, - "invalid_action": false, - "lane": 3, - "length": 2100, - "mean_action_latency_ms": 0.1317734359985306, - "policy_error": null, - "return": 0.8149999864399433, - "seed": 610, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "HEALTH": 6.0 - }, - "invalid_action": false, - "lane": 4, - "length": 736, - "mean_action_latency_ms": 0.12118354662442984, - "policy_error": null, - "return": -0.078000009059906, - "seed": 611, - "terminated": true, - "truncated": false - } - ], - "frame_skip": 1, - "invalid_episode_rate": 0.0, - "max_return": 0.8149999864399433, - "max_steps": 2100, - "mean_action_latency_ms": 0.1345235449059502, - "mean_return": 0.6141999863088131, - "min_return": -0.078000009059906, - "policy_path": "submitted_artifact", - "render_height": 180, - "render_width": 240, - "scenario": "d1", - "score": 0.6141999863088131 + { + "normalized_score": 95.055823417, + "observed_raw_score": 0.7992199832946062, + "raw_score": 0.7992199832946062, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "trial": 2 + }, + { + "normalized_score": 31.015418049, + "observed_raw_score": 0.5582999877631664, + "raw_score": 0.5582999877631664, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "trial": 3 + }, + { + "normalized_score": 89.73418684, + "observed_raw_score": 0.7791999872773886, + "raw_score": 0.7791999872773886, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "trial": 4 + }, + { + "normalized_score": 25.539607563, + "observed_raw_score": 0.5376999895274639, + "raw_score": 0.5376999895274639, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05", + "trial": 5 } - } + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-01/metadata.json new file mode 100644 index 0000000..6ebc6f5 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "digest_compatibility_note": null, + "finished_at": 1784075101.71677, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 42.828284045, + "observed_raw_score": 0.6027399878948927, + "provider_reasoning_effort": "max", + "raw_score": 0.6027399878948927, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "task": "simulation_heuristics_vizdoom_d1_v1", + "task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "token_usage": { + "final_input_tokens": 675.0, + "final_output_tokens": 123.0, + "input_tokens": 675.0, + "output_tokens": 123.0, + "total_tokens": 23326.0 + }, + "tool_calls": 28, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-01/score.json new file mode 100644 index 0000000..deeef80 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-01/score.json @@ -0,0 +1,211 @@ +{ + "anchor_calibration": { + "envpool_version": "1.1.1", + "mode": "numeric", + "platform": "linux-x86_64" + }, + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 42.828284045, + "reference_score": 0.8178199748694897, + "resource_cleanup": { + "removed_bytes": 22517424, + "removed_resources": 27, + "runs": [ + { + "removed_bytes": 22517424, + "removed_resources": 27, + "terminated_processes": 0 + }, + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + } + ], + "terminated_processes": 0 + }, + "score": 0.6027399878948927, + "starter_score": 0.44161998964846133, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 47.0 + }, + "invalid_action": false, + "lane": 0, + "length": 2100, + "mean_action_latency_ms": 0.09277379051560447, + "policy_error": null, + "return": 0.7989999875426292, + "seed": 101, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 42.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.05938445528348287, + "policy_error": null, + "return": 0.7899999804794788, + "seed": 102, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 2.0 + }, + "invalid_action": false, + "lane": 2, + "length": 1792, + "mean_action_latency_ms": 0.08098577278100752, + "policy_error": null, + "return": 0.4299999885261059, + "seed": 103, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 47.0 + }, + "invalid_action": false, + "lane": 3, + "length": 2100, + "mean_action_latency_ms": 0.07177316539344333, + "policy_error": null, + "return": 0.7989999875426292, + "seed": 104, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 47.0 + }, + "invalid_action": false, + "lane": 4, + "length": 2100, + "mean_action_latency_ms": 0.08024889266207105, + "policy_error": null, + "return": 0.7989999875426292, + "seed": 105, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 0.7989999875426292, + "max_steps": 2100, + "mean_action_latency_ms": 0.07703321532712185, + "mean_return": 0.7233999863266944, + "min_return": 0.4299999885261059, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.7233999863266944 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 5.0 + }, + "invalid_action": false, + "lane": 0, + "length": 1280, + "mean_action_latency_ms": 0.09665053948992863, + "policy_error": null, + "return": 0.188999991863966, + "seed": 607, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 47.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.06673190387941542, + "policy_error": null, + "return": 0.8149999864399433, + "seed": 608, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 3.0 + }, + "invalid_action": false, + "lane": 2, + "length": 1088, + "mean_action_latency_ms": 0.09983103989404352, + "policy_error": null, + "return": 0.08299999311566353, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 47.0 + }, + "invalid_action": false, + "lane": 3, + "length": 2100, + "mean_action_latency_ms": 0.07967882035743623, + "policy_error": null, + "return": 0.7829999886453152, + "seed": 610, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 4.0 + }, + "invalid_action": false, + "lane": 4, + "length": 384, + "mean_action_latency_ms": 0.10957056656479836, + "policy_error": null, + "return": -0.2640000022947788, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 0.8149999864399433, + "max_steps": 2100, + "mean_action_latency_ms": 0.09049257403712443, + "mean_return": 0.32119999155402185, + "min_return": -0.2640000022947788, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.32119999155402185 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-02/metadata.json new file mode 100644 index 0000000..bfd409f --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "digest_compatibility_note": null, + "finished_at": 1784078883.680485, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 95.055823417, + "observed_raw_score": 0.7992199832946062, + "provider_reasoning_effort": "max", + "raw_score": 0.7992199832946062, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "task": "simulation_heuristics_vizdoom_d1_v1", + "task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "token_usage": { + "final_input_tokens": 491.0, + "final_output_tokens": 191.0, + "input_tokens": 491.0, + "output_tokens": 191.0, + "total_tokens": 22186.0 + }, + "tool_calls": 24, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-02/score.json new file mode 100644 index 0000000..dbb3da8 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-02/score.json @@ -0,0 +1,211 @@ +{ + "anchor_calibration": { + "envpool_version": "1.1.1", + "mode": "numeric", + "platform": "linux-x86_64" + }, + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 95.055823417, + "reference_score": 0.8178199748694897, + "resource_cleanup": { + "removed_bytes": 0, + "removed_resources": 0, + "runs": [ + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + }, + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + } + ], + "terminated_processes": 0 + }, + "score": 0.7992199832946062, + "starter_score": 0.44161998964846133, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 7.0 + }, + "invalid_action": false, + "lane": 0, + "length": 1152, + "mean_action_latency_ms": 0.2847819770168927, + "policy_error": null, + "return": 0.13499998673796654, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 72.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.3129474126866886, + "policy_error": null, + "return": 0.9239999782294035, + "seed": 102, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 60.0 + }, + "invalid_action": false, + "lane": 2, + "length": 2100, + "mean_action_latency_ms": 0.2952394217607521, + "policy_error": null, + "return": 0.8799999840557575, + "seed": 103, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 60.0 + }, + "invalid_action": false, + "lane": 3, + "length": 2100, + "mean_action_latency_ms": 0.29309217418943134, + "policy_error": null, + "return": 0.8479999843984842, + "seed": 104, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 72.0 + }, + "invalid_action": false, + "lane": 4, + "length": 2100, + "mean_action_latency_ms": 0.2893664146817866, + "policy_error": null, + "return": 0.9399999864399433, + "seed": 105, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 0.9399999864399433, + "max_steps": 2100, + "mean_action_latency_ms": 0.2950854800671102, + "mean_return": 0.745399983972311, + "min_return": 0.13499998673796654, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.745399983972311 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 63.0 + }, + "invalid_action": false, + "lane": 0, + "length": 2100, + "mean_action_latency_ms": 0.3169753970134826, + "policy_error": null, + "return": 0.8789999838918447, + "seed": 607, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 46.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.29031046976645786, + "policy_error": null, + "return": 0.8099999837577343, + "seed": 608, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 59.0 + }, + "invalid_action": false, + "lane": 2, + "length": 2100, + "mean_action_latency_ms": 0.28571441326112973, + "policy_error": null, + "return": 0.8749999813735485, + "seed": 609, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 88.0 + }, + "invalid_action": false, + "lane": 3, + "length": 2100, + "mean_action_latency_ms": 0.28346478584266843, + "policy_error": null, + "return": 1.0199999809265137, + "seed": 610, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 92.0 + }, + "invalid_action": false, + "lane": 4, + "length": 2100, + "mean_action_latency_ms": 0.28227492546041805, + "policy_error": null, + "return": 1.0399999786168337, + "seed": 611, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 1.0399999786168337, + "max_steps": 2100, + "mean_action_latency_ms": 0.2917479982688313, + "mean_return": 0.924799981713295, + "min_return": 0.8099999837577343, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.924799981713295 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-03/metadata.json new file mode 100644 index 0000000..6bef491 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "digest_compatibility_note": null, + "finished_at": 1784092762.840873, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 31.015418049, + "observed_raw_score": 0.5582999877631664, + "provider_reasoning_effort": "max", + "raw_score": 0.5582999877631664, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "task": "simulation_heuristics_vizdoom_d1_v1", + "task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "token_usage": { + "final_input_tokens": 0.0, + "final_output_tokens": 0.0, + "input_tokens": 0.0, + "output_tokens": 0.0, + "total_tokens": 0.0 + }, + "tool_calls": 1, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-03/score.json new file mode 100644 index 0000000..2693627 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-03/score.json @@ -0,0 +1,211 @@ +{ + "anchor_calibration": { + "envpool_version": "1.1.1", + "mode": "numeric", + "platform": "linux-x86_64" + }, + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 31.015418049, + "reference_score": 0.8178199748694897, + "resource_cleanup": { + "removed_bytes": 0, + "removed_resources": 0, + "runs": [ + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + }, + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + } + ], + "terminated_processes": 0 + }, + "score": 0.5582999877631664, + "starter_score": 0.44161998964846133, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 68.0 + }, + "invalid_action": false, + "lane": 0, + "length": 2100, + "mean_action_latency_ms": 0.09957417490936461, + "policy_error": null, + "return": 0.9199999831616879, + "seed": 101, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 6.0 + }, + "invalid_action": false, + "lane": 1, + "length": 1216, + "mean_action_latency_ms": 0.1304904075560013, + "policy_error": null, + "return": 0.16199998930096626, + "seed": 102, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 58.0 + }, + "invalid_action": false, + "lane": 2, + "length": 2100, + "mean_action_latency_ms": 0.08596905906285558, + "policy_error": null, + "return": 0.8699999824166298, + "seed": 103, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 6.0 + }, + "invalid_action": false, + "lane": 3, + "length": 1376, + "mean_action_latency_ms": 0.11495853433125587, + "policy_error": null, + "return": 0.24199999123811722, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 69.0 + }, + "invalid_action": false, + "lane": 4, + "length": 2100, + "mean_action_latency_ms": 0.08597019705034438, + "policy_error": null, + "return": 0.8929999880492687, + "seed": 105, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 0.9199999831616879, + "max_steps": 2100, + "mean_action_latency_ms": 0.10339247458196435, + "mean_return": 0.617399986833334, + "min_return": 0.16199998930096626, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.617399986833334 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 7.0 + }, + "invalid_action": false, + "lane": 0, + "length": 672, + "mean_action_latency_ms": 0.2247366133988613, + "policy_error": null, + "return": -0.10500000417232513, + "seed": 607, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 69.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.13988662688505082, + "policy_error": null, + "return": 0.9089999869465828, + "seed": 608, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 68.0 + }, + "invalid_action": false, + "lane": 2, + "length": 2100, + "mean_action_latency_ms": 0.07088755479171163, + "policy_error": null, + "return": 0.9199999831616879, + "seed": 609, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 3.0 + }, + "invalid_action": false, + "lane": 3, + "length": 1888, + "mean_action_latency_ms": 0.1139322570051556, + "policy_error": null, + "return": 0.4829999879002571, + "seed": 610, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 7.0 + }, + "invalid_action": false, + "lane": 4, + "length": 672, + "mean_action_latency_ms": 0.13459125988274104, + "policy_error": null, + "return": -0.10500000417232513, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 0.9199999831616879, + "max_steps": 2100, + "mean_action_latency_ms": 0.13680686239270407, + "mean_return": 0.4203999899327755, + "min_return": -0.10500000417232513, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.4203999899327755 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-04/metadata.json new file mode 100644 index 0000000..b2749b2 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-04/metadata.json @@ -0,0 +1,25 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "digest_compatibility_note": null, + "finished_at": 1784095422.450964, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 89.73418684, + "observed_raw_score": 0.7791999872773886, + "provider_reasoning_effort": "max", + "raw_score": 0.7791999872773886, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "task": "simulation_heuristics_vizdoom_d1_v1", + "task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "token_usage": null, + "tool_calls": 23, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-04/score.json new file mode 100644 index 0000000..f897a50 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-04/score.json @@ -0,0 +1,211 @@ +{ + "anchor_calibration": { + "envpool_version": "1.1.1", + "mode": "numeric", + "platform": "linux-x86_64" + }, + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 89.73418684, + "reference_score": 0.8178199748694897, + "resource_cleanup": { + "removed_bytes": 0, + "removed_resources": 0, + "runs": [ + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + }, + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + } + ], + "terminated_processes": 0 + }, + "score": 0.7791999872773886, + "starter_score": 0.44161998964846133, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 72.0 + }, + "invalid_action": false, + "lane": 0, + "length": 2100, + "mean_action_latency_ms": 0.04258775019219944, + "policy_error": null, + "return": 0.9239999875426292, + "seed": 101, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 57.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.04289436641903151, + "policy_error": null, + "return": 0.8649999834597111, + "seed": 102, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 66.0 + }, + "invalid_action": false, + "lane": 2, + "length": 2100, + "mean_action_latency_ms": 0.03522284949819247, + "policy_error": null, + "return": 0.9099999852478504, + "seed": 103, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 72.0 + }, + "invalid_action": false, + "lane": 3, + "length": 2100, + "mean_action_latency_ms": 0.033290026088555656, + "policy_error": null, + "return": 0.9399999864399433, + "seed": 104, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 72.0 + }, + "invalid_action": false, + "lane": 4, + "length": 2100, + "mean_action_latency_ms": 0.03839936639581408, + "policy_error": null, + "return": 0.9079999886453152, + "seed": 105, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 0.9399999864399433, + "max_steps": 2100, + "mean_action_latency_ms": 0.038478871718758634, + "mean_return": 0.9093999862670898, + "min_return": 0.8649999834597111, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.9093999862670898 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 44.0 + }, + "invalid_action": false, + "lane": 0, + "length": 2100, + "mean_action_latency_ms": 0.04541670637471335, + "policy_error": null, + "return": 0.7999999858438969, + "seed": 607, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 47.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.03715894140657924, + "policy_error": null, + "return": 0.8149999864399433, + "seed": 608, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 2.0 + }, + "invalid_action": false, + "lane": 2, + "length": 992, + "mean_action_latency_ms": 0.04850396878957268, + "policy_error": null, + "return": 0.0299999937415123, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 8.0 + }, + "invalid_action": false, + "lane": 3, + "length": 768, + "mean_action_latency_ms": 0.08625746462106083, + "policy_error": null, + "return": -0.052000004798173904, + "seed": 610, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 44.0 + }, + "invalid_action": false, + "lane": 4, + "length": 2100, + "mean_action_latency_ms": 0.04557695151084945, + "policy_error": null, + "return": 0.7839999869465828, + "seed": 611, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 0.8149999864399433, + "max_steps": 2100, + "mean_action_latency_ms": 0.0525828065405551, + "mean_return": 0.4753999896347523, + "min_return": -0.052000004798173904, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.4753999896347523 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-05/metadata.json new file mode 100644 index 0000000..eed9519 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-05/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "digest_compatibility_note": null, + "finished_at": 1784098371.002007, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 25.539607563, + "observed_raw_score": 0.5376999895274639, + "provider_reasoning_effort": "max", + "raw_score": 0.5376999895274639, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05", + "task": "simulation_heuristics_vizdoom_d1_v1", + "task_digest": "sha256:ea30e0252670bc77855fc732868f29c080f8773c1e0f9b72180289c568c59238", + "token_usage": { + "final_input_tokens": 817.0, + "final_output_tokens": 159.0, + "input_tokens": 817.0, + "output_tokens": 159.0, + "total_tokens": 19408.0 + }, + "tool_calls": 20, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-05/score.json new file mode 100644 index 0000000..7360d83 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/trial-05/score.json @@ -0,0 +1,211 @@ +{ + "anchor_calibration": { + "envpool_version": "1.1.1", + "mode": "numeric", + "platform": "linux-x86_64" + }, + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 25.539607563, + "reference_score": 0.8178199748694897, + "resource_cleanup": { + "removed_bytes": 0, + "removed_resources": 0, + "runs": [ + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + }, + { + "removed_bytes": 0, + "removed_resources": 0, + "terminated_processes": 0 + } + ], + "terminated_processes": 0 + }, + "score": 0.5376999895274639, + "starter_score": 0.44161998964846133, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 72.0 + }, + "invalid_action": false, + "lane": 0, + "length": 2100, + "mean_action_latency_ms": 0.21995443318571364, + "policy_error": null, + "return": 0.9399999864399433, + "seed": 101, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 55.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.2012049202762899, + "policy_error": null, + "return": 0.8229999877512455, + "seed": 102, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 47.0 + }, + "invalid_action": false, + "lane": 2, + "length": 2100, + "mean_action_latency_ms": 0.19551001311767668, + "policy_error": null, + "return": 0.7989999875426292, + "seed": 103, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 5.0 + }, + "invalid_action": false, + "lane": 3, + "length": 480, + "mean_action_latency_ms": 0.17379071020210782, + "policy_error": null, + "return": -0.2110000029206276, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 8.0 + }, + "invalid_action": false, + "lane": 4, + "length": 1568, + "mean_action_latency_ms": 0.1861551985600773, + "policy_error": null, + "return": 0.3479999899864197, + "seed": 105, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 0.9399999864399433, + "max_steps": 2100, + "mean_action_latency_ms": 0.19532305506837305, + "mean_return": 0.539799989759922, + "min_return": -0.2110000029206276, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.539799989759922 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "HEALTH": 2.0 + }, + "invalid_action": false, + "lane": 0, + "length": 992, + "mean_action_latency_ms": 0.2011167666604442, + "policy_error": null, + "return": 0.0299999937415123, + "seed": 607, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 47.0 + }, + "invalid_action": false, + "lane": 1, + "length": 2100, + "mean_action_latency_ms": 0.17853501740665662, + "policy_error": null, + "return": 0.8149999864399433, + "seed": 608, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 5.0 + }, + "invalid_action": false, + "lane": 2, + "length": 1280, + "mean_action_latency_ms": 0.16564973775530234, + "policy_error": null, + "return": 0.188999991863966, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "HEALTH": 47.0 + }, + "invalid_action": false, + "lane": 3, + "length": 2100, + "mean_action_latency_ms": 0.16895033153040068, + "policy_error": null, + "return": 0.8149999864399433, + "seed": 610, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "HEALTH": 47.0 + }, + "invalid_action": false, + "lane": 4, + "length": 2100, + "mean_action_latency_ms": 0.16575174317473457, + "policy_error": null, + "return": 0.8149999864399433, + "seed": 611, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 1, + "invalid_episode_rate": 0.0, + "max_return": 0.8149999864399433, + "max_steps": 2100, + "mean_action_latency_ms": 0.17600071930550767, + "mean_return": 0.5327999889850616, + "min_return": 0.0299999937415123, + "policy_path": "submitted_artifact", + "render_height": 180, + "render_width": 240, + "scenario": "d1", + "score": 0.5327999889850616 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/metadata.json index d2ae053..b16b3de 100644 --- a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/metadata.json +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/metadata.json @@ -1,8 +1,6 @@ { "benchmark": "learning_beyond_gradients_article_suite", - "current_task_digest": "sha256:29434bdcdce19a6e0cb24d77c800d172d49a9529875ffd4201cf928e242f4e62", - "digest_compatibility_note": "Score-equivalent to the current task. The later loader fix registers submitted modules before execution so postponed dataclass annotations import correctly; scoring is unchanged.", - "finished_at": 1783932419.4689522, + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", "harness": "opencode", "model": { "display_name": "GPT-5.6 Sol", @@ -11,17 +9,19 @@ "provider": "azure", "provider_reasoning_effort": "max" }, - "normalized_score": 26.236631016, + "normalized_score": 16.3034759358, + "normalized_score_stddev": 23.995771403261536, "provider_reasoning_effort": "max", - "source_run_id": "leaderboard/runs/article_suite/20260713T083940Z/gpt-5.6-sol", + "raw_score": 80.67999999999999, + "raw_score_imputation_count": 0, + "raw_score_stddev": 71.79534803871347, + "source_run_ids": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "task": "simulation_heuristics_vizdoom_d3_v1", - "task_digest": "sha256:fcc6ed05673b7981aa17d60ca4e0d355fbcb57a6cbca293c0fa201ab97cdd081", - "token_usage": { - "final_input_tokens": 425.0, - "final_output_tokens": 156.0, - "input_tokens": 425.0, - "output_tokens": 156.0, - "total_tokens": 23109.0 - }, - "tool_calls": 29 + "trial_count": 5 } diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/score.json index 54904fa..c1a882f 100644 --- a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/score.json +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/score.json @@ -1,229 +1,59 @@ { - "isolation": { - "mode": "subprocess_per_suite", - "suite_process_count": 2, - "unique_working_directories": true - }, - "normalized_score": 26.236631016, + "benchmark": "learning_beyond_gradients_article_suite", + "normalized_score": 16.3034759358, + "normalized_score_stddev": 23.995771403261536, + "raw_score_imputation_count": 0, "reference_score": 331.1, - "score": 110.39999999999998, + "score": 80.67999999999999, + "score_stddev": 71.79534803871347, "starter_score": 31.9, - "suites": { - "hidden_nominal": { - "batch_seed": 101, - "envpool_version": "1.1.1", - "episodes": [ - { - "final_variables": { - "AMMO2": 0.0, - "DAMAGECOUNT": 70.0, - "HEALTH": 10.0, - "HITCOUNT": 8.0, - "KILLCOUNT": 7.0 - }, - "invalid_action": false, - "lane": 0, - "length": 1027, - "mean_action_latency_ms": 3.1377216359813986, - "policy_error": null, - "return": 140.0, - "seed": 101, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 22.0, - "DAMAGECOUNT": 40.0, - "HEALTH": 64.0, - "HITCOUNT": 5.0, - "KILLCOUNT": 5.0 - }, - "invalid_action": false, - "lane": 1, - "length": 1050, - "mean_action_latency_ms": 3.0818451521918178, - "policy_error": null, - "return": 90.0, - "seed": 102, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "AMMO2": 0.0, - "DAMAGECOUNT": 40.0, - "HEALTH": 2.0, - "HITCOUNT": 5.0, - "KILLCOUNT": 4.0 - }, - "invalid_action": false, - "lane": 2, - "length": 1009, - "mean_action_latency_ms": 3.0280790251979237, - "policy_error": null, - "return": 80.0, - "seed": 103, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 10.0, - "DAMAGECOUNT": 45.0, - "HEALTH": 78.0, - "HITCOUNT": 4.0, - "KILLCOUNT": 4.0 - }, - "invalid_action": false, - "lane": 3, - "length": 1050, - "mean_action_latency_ms": 3.0736763611535673, - "policy_error": null, - "return": 85.0, - "seed": 104, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "AMMO2": 2.0, - "DAMAGECOUNT": 90.0, - "HEALTH": 14.0, - "HITCOUNT": 7.0, - "KILLCOUNT": 7.0 - }, - "invalid_action": false, - "lane": 4, - "length": 929, - "mean_action_latency_ms": 3.054685383848144, - "policy_error": null, - "return": 160.0, - "seed": 105, - "terminated": true, - "truncated": false - } - ], - "frame_skip": 2, - "invalid_episode_rate": 0.0, - "max_return": 160.0, - "max_steps": 1050, - "mean_action_latency_ms": 3.07520151167457, - "mean_return": 111.0, - "min_return": 80.0, - "policy_path": "submitted_artifact", - "render_height": 480, - "render_width": 640, - "scenario": "d3", - "score": 111.0 + "task": "simulation_heuristics_vizdoom_d3_v1", + "trial_count": 5, + "trials": [ + { + "normalized_score": 40.106951872, + "observed_raw_score": 151.89999999999998, + "raw_score": 151.89999999999998, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-01/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "trial": 1 }, - "hidden_secondary": { - "batch_seed": 607, - "envpool_version": "1.1.1", - "episodes": [ - { - "final_variables": { - "AMMO2": 14.0, - "DAMAGECOUNT": 30.0, - "HEALTH": 19.0, - "HITCOUNT": 2.0, - "KILLCOUNT": 2.0 - }, - "invalid_action": false, - "lane": 0, - "length": 240, - "mean_action_latency_ms": 3.0935825085180113, - "policy_error": null, - "return": 50.0, - "seed": 607, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 6.0, - "DAMAGECOUNT": 55.0, - "HEALTH": 1.0, - "HITCOUNT": 4.0, - "KILLCOUNT": 4.0 - }, - "invalid_action": false, - "lane": 1, - "length": 715, - "mean_action_latency_ms": 3.0578368642185065, - "policy_error": null, - "return": 95.0, - "seed": 608, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 0.0, - "DAMAGECOUNT": 20.0, - "HEALTH": 9.0, - "HITCOUNT": 3.0, - "KILLCOUNT": 2.0 - }, - "invalid_action": false, - "lane": 2, - "length": 956, - "mean_action_latency_ms": 3.0342543454878284, - "policy_error": null, - "return": 40.0, - "seed": 609, - "terminated": true, - "truncated": false - }, - { - "final_variables": { - "AMMO2": 9.0, - "DAMAGECOUNT": 35.0, - "HEALTH": 61.0, - "HITCOUNT": 4.0, - "KILLCOUNT": 4.0 - }, - "invalid_action": false, - "lane": 3, - "length": 1050, - "mean_action_latency_ms": 3.052990203418414, - "policy_error": null, - "return": 75.0, - "seed": 610, - "terminated": false, - "truncated": true - }, - { - "final_variables": { - "AMMO2": 4.0, - "DAMAGECOUNT": 145.0, - "HEALTH": 2.0, - "HITCOUNT": 14.0, - "KILLCOUNT": 14.0 - }, - "invalid_action": false, - "lane": 4, - "length": 992, - "mean_action_latency_ms": 3.0088798643367727, - "policy_error": null, - "return": 285.0, - "seed": 611, - "terminated": true, - "truncated": false - } - ], - "frame_skip": 2, - "invalid_episode_rate": 0.0, - "max_return": 285.0, - "max_steps": 1050, - "mean_action_latency_ms": 3.0495087571959067, - "mean_return": 109.0, - "min_return": 40.0, - "policy_path": "submitted_artifact", - "render_height": 480, - "render_width": 640, - "scenario": "d3", - "score": 109.0 + { + "normalized_score": -3.442513369, + "observed_raw_score": 21.599999999999998, + "raw_score": 21.599999999999998, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-02/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "trial": 2 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 31.9, + "raw_score": 31.9, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-03/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "trial": 3 + }, + { + "normalized_score": 0.0, + "observed_raw_score": 31.9, + "raw_score": 31.9, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-04/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "trial": 4 + }, + { + "normalized_score": 44.852941176, + "observed_raw_score": 166.1, + "raw_score": 166.1, + "raw_score_source": "observed", + "score_path": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-05/score.json", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05", + "trial": 5 } - } + ] } diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-01/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-01/metadata.json new file mode 100644 index 0000000..22dea8f --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-01/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "digest_compatibility_note": null, + "finished_at": 1784075101.71677, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 40.106951872, + "observed_raw_score": 151.89999999999998, + "provider_reasoning_effort": "max", + "raw_score": 151.89999999999998, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "task": "simulation_heuristics_vizdoom_d3_v1", + "task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "token_usage": { + "final_input_tokens": 854.0, + "final_output_tokens": 171.0, + "input_tokens": 854.0, + "output_tokens": 171.0, + "total_tokens": 30721.0 + }, + "tool_calls": 35, + "trial": 1 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-01/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-01/score.json new file mode 100644 index 0000000..dfd9ffa --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-01/score.json @@ -0,0 +1,229 @@ +{ + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 40.106951872, + "reference_score": 331.1, + "score": 151.89999999999998, + "starter_score": 31.9, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 20.0, + "HEALTH": 10.0, + "HITCOUNT": 4.0, + "KILLCOUNT": 4.0 + }, + "invalid_action": false, + "lane": 0, + "length": 754, + "mean_action_latency_ms": 3.0050282255407987, + "policy_error": null, + "return": 60.0, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 60.0, + "HEALTH": 4.0, + "HITCOUNT": 5.0, + "KILLCOUNT": 5.0 + }, + "invalid_action": false, + "lane": 1, + "length": 736, + "mean_action_latency_ms": 3.0707310804206394, + "policy_error": null, + "return": 110.0, + "seed": 102, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 70.0, + "HEALTH": 2.0, + "HITCOUNT": 9.0, + "KILLCOUNT": 9.0 + }, + "invalid_action": false, + "lane": 2, + "length": 737, + "mean_action_latency_ms": 3.0704634200928167, + "policy_error": null, + "return": 160.0, + "seed": 103, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 42.0, + "DAMAGECOUNT": 95.0, + "HEALTH": 1.0, + "HITCOUNT": 10.0, + "KILLCOUNT": 9.0 + }, + "invalid_action": false, + "lane": 3, + "length": 968, + "mean_action_latency_ms": 2.989207848537067, + "policy_error": null, + "return": 185.0, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 30.0, + "HEALTH": 10.0, + "HITCOUNT": 3.0, + "KILLCOUNT": 3.0 + }, + "invalid_action": false, + "lane": 4, + "length": 596, + "mean_action_latency_ms": 3.0182157299362573, + "policy_error": null, + "return": 60.0, + "seed": 105, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 185.0, + "max_steps": 1050, + "mean_action_latency_ms": 3.030729260905516, + "mean_return": 115.0, + "min_return": 60.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 115.0 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 2.0, + "DAMAGECOUNT": 165.0, + "HEALTH": 100.0, + "HITCOUNT": 16.0, + "KILLCOUNT": 14.0 + }, + "invalid_action": false, + "lane": 0, + "length": 1050, + "mean_action_latency_ms": 3.06140120895136, + "policy_error": null, + "return": 305.0, + "seed": 607, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 26.0, + "DAMAGECOUNT": 90.0, + "HEALTH": 22.0, + "HITCOUNT": 7.0, + "KILLCOUNT": 7.0 + }, + "invalid_action": false, + "lane": 1, + "length": 466, + "mean_action_latency_ms": 2.9988041699188464, + "policy_error": null, + "return": 160.0, + "seed": 608, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 46.0, + "DAMAGECOUNT": 95.0, + "HEALTH": 16.0, + "HITCOUNT": 9.0, + "KILLCOUNT": 9.0 + }, + "invalid_action": false, + "lane": 2, + "length": 1021, + "mean_action_latency_ms": 3.027502381690686, + "policy_error": null, + "return": 185.0, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 18.0, + "DAMAGECOUNT": 50.0, + "HEALTH": 1.0, + "HITCOUNT": 5.0, + "KILLCOUNT": 5.0 + }, + "invalid_action": false, + "lane": 3, + "length": 916, + "mean_action_latency_ms": 3.0297691645060043, + "policy_error": null, + "return": 100.0, + "seed": 610, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 20.0, + "DAMAGECOUNT": 220.0, + "HEALTH": 100.0, + "HITCOUNT": 25.0, + "KILLCOUNT": 22.0 + }, + "invalid_action": false, + "lane": 4, + "length": 1050, + "mean_action_latency_ms": 3.0611898253361383, + "policy_error": null, + "return": 440.0, + "seed": 611, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 440.0, + "max_steps": 1050, + "mean_action_latency_ms": 3.0357333500806067, + "mean_return": 238.0, + "min_return": 100.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 238.0 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-02/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-02/metadata.json new file mode 100644 index 0000000..91e7836 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-02/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "digest_compatibility_note": null, + "finished_at": 1784078883.680485, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": -3.442513369, + "observed_raw_score": 21.599999999999998, + "provider_reasoning_effort": "max", + "raw_score": 21.599999999999998, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "task": "simulation_heuristics_vizdoom_d3_v1", + "task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "token_usage": { + "final_input_tokens": 379.0, + "final_output_tokens": 167.0, + "input_tokens": 379.0, + "output_tokens": 167.0, + "total_tokens": 22562.0 + }, + "tool_calls": 22, + "trial": 2 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-02/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-02/score.json new file mode 100644 index 0000000..274b10d --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-02/score.json @@ -0,0 +1,229 @@ +{ + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": -3.442513369, + "reference_score": 331.1, + "score": 21.599999999999998, + "starter_score": 31.9, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 5.0, + "HEALTH": 13.0, + "HITCOUNT": 1.0, + "KILLCOUNT": 1.0 + }, + "invalid_action": false, + "lane": 0, + "length": 171, + "mean_action_latency_ms": 4.125108822570209, + "policy_error": null, + "return": 15.0, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 5.0, + "HEALTH": 13.0, + "HITCOUNT": 1.0, + "KILLCOUNT": 1.0 + }, + "invalid_action": false, + "lane": 1, + "length": 277, + "mean_action_latency_ms": 4.5146709610624, + "policy_error": null, + "return": 15.0, + "seed": 102, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 4.0, + "DAMAGECOUNT": 5.0, + "HEALTH": 4.0, + "HITCOUNT": 1.0, + "KILLCOUNT": 1.0 + }, + "invalid_action": false, + "lane": 2, + "length": 183, + "mean_action_latency_ms": 3.889043108035958, + "policy_error": null, + "return": 15.0, + "seed": 103, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 10.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 3, + "length": 187, + "mean_action_latency_ms": 4.112742303367605, + "policy_error": null, + "return": 0.0, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 2.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 13.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 4, + "length": 210, + "mean_action_latency_ms": 3.873070010117122, + "policy_error": null, + "return": 0.0, + "seed": 105, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 15.0, + "max_steps": 1050, + "mean_action_latency_ms": 4.102927041030659, + "mean_return": 9.0, + "min_return": 0.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 9.0 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 75.0, + "HEALTH": 7.0, + "HITCOUNT": 7.0, + "KILLCOUNT": 7.0 + }, + "invalid_action": false, + "lane": 0, + "length": 714, + "mean_action_latency_ms": 5.57378194808793, + "policy_error": null, + "return": 145.0, + "seed": 607, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 1.0, + "DAMAGECOUNT": 45.0, + "HEALTH": 1.0, + "HITCOUNT": 3.0, + "KILLCOUNT": 3.0 + }, + "invalid_action": false, + "lane": 1, + "length": 463, + "mean_action_latency_ms": 5.170079523427955, + "policy_error": null, + "return": 75.0, + "seed": 608, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 19.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 2, + "length": 444, + "mean_action_latency_ms": 5.691396475240991, + "policy_error": null, + "return": 0.0, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 1.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 3, + "length": 272, + "mean_action_latency_ms": 5.352805930135005, + "policy_error": null, + "return": 0.0, + "seed": 610, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 15.0, + "HEALTH": 19.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 4, + "length": 310, + "mean_action_latency_ms": 5.015822499990463, + "policy_error": null, + "return": 35.0, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 145.0, + "max_steps": 1050, + "mean_action_latency_ms": 5.360777275376469, + "mean_return": 51.0, + "min_return": 0.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 51.0 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-03/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-03/metadata.json new file mode 100644 index 0000000..8eeddb8 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-03/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "digest_compatibility_note": null, + "finished_at": 1784092762.840873, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 0.0, + "observed_raw_score": 31.9, + "provider_reasoning_effort": "max", + "raw_score": 31.9, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "task": "simulation_heuristics_vizdoom_d3_v1", + "task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "token_usage": { + "final_input_tokens": 516.0, + "final_output_tokens": 100.0, + "input_tokens": 516.0, + "output_tokens": 100.0, + "total_tokens": 26216.0 + }, + "tool_calls": 23, + "trial": 3 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-03/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-03/score.json new file mode 100644 index 0000000..6aaa0ce --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-03/score.json @@ -0,0 +1,229 @@ +{ + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 0.0, + "reference_score": 331.1, + "score": 31.9, + "starter_score": 31.9, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 10.0, + "HEALTH": 10.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 0, + "length": 711, + "mean_action_latency_ms": 2.0836446952970724, + "policy_error": null, + "return": 30.0, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 20.0, + "HEALTH": 12.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 1, + "length": 980, + "mean_action_latency_ms": 2.0975400332589538, + "policy_error": null, + "return": 40.0, + "seed": 102, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 10.0, + "HEALTH": 16.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 2, + "length": 317, + "mean_action_latency_ms": 2.065451903862156, + "policy_error": null, + "return": 30.0, + "seed": 103, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 13.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 3, + "length": 179, + "mean_action_latency_ms": 2.033258387662845, + "policy_error": null, + "return": 0.0, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 20.0, + "HEALTH": 10.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 4, + "length": 306, + "mean_action_latency_ms": 2.0744595676660538, + "policy_error": null, + "return": 40.0, + "seed": 105, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 40.0, + "max_steps": 1050, + "mean_action_latency_ms": 2.070870917549416, + "mean_return": 28.0, + "min_return": 0.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 28.0 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 35.0, + "HEALTH": 90.0, + "HITCOUNT": 4.0, + "KILLCOUNT": 4.0 + }, + "invalid_action": false, + "lane": 0, + "length": 1050, + "mean_action_latency_ms": 1.997888364962169, + "policy_error": null, + "return": 75.0, + "seed": 607, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 10.0, + "HEALTH": 10.0, + "HITCOUNT": 1.0, + "KILLCOUNT": 1.0 + }, + "invalid_action": false, + "lane": 1, + "length": 262, + "mean_action_latency_ms": 2.115637646945378, + "policy_error": null, + "return": 20.0, + "seed": 608, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 4.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 2, + "length": 473, + "mean_action_latency_ms": 2.0683587353526915, + "policy_error": null, + "return": 0.0, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 19.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 3, + "length": 400, + "mean_action_latency_ms": 2.0626725628972054, + "policy_error": null, + "return": 0.0, + "seed": 610, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 60.0, + "HEALTH": 19.0, + "HITCOUNT": 5.0, + "KILLCOUNT": 5.0 + }, + "invalid_action": false, + "lane": 4, + "length": 373, + "mean_action_latency_ms": 2.069716169511345, + "policy_error": null, + "return": 110.0, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 110.0, + "max_steps": 1050, + "mean_action_latency_ms": 2.0628546959337575, + "mean_return": 41.0, + "min_return": 0.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 41.0 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-04/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-04/metadata.json new file mode 100644 index 0000000..f531486 --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-04/metadata.json @@ -0,0 +1,25 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "digest_compatibility_note": null, + "finished_at": 1784095422.450964, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 0.0, + "observed_raw_score": 31.9, + "provider_reasoning_effort": "max", + "raw_score": 31.9, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "task": "simulation_heuristics_vizdoom_d3_v1", + "task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "token_usage": null, + "tool_calls": 23, + "trial": 4 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-04/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-04/score.json new file mode 100644 index 0000000..fd038dd --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-04/score.json @@ -0,0 +1,229 @@ +{ + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 0.0, + "reference_score": 331.1, + "score": 31.9, + "starter_score": 31.9, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 10.0, + "HEALTH": 10.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 0, + "length": 711, + "mean_action_latency_ms": 2.1637215707372515, + "policy_error": null, + "return": 30.0, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 20.0, + "HEALTH": 12.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 1, + "length": 980, + "mean_action_latency_ms": 2.1701510197349956, + "policy_error": null, + "return": 40.0, + "seed": 102, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 10.0, + "HEALTH": 16.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 2, + "length": 317, + "mean_action_latency_ms": 2.150934824150067, + "policy_error": null, + "return": 30.0, + "seed": 103, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 13.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 3, + "length": 179, + "mean_action_latency_ms": 2.12456362956729, + "policy_error": null, + "return": 0.0, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 20.0, + "HEALTH": 10.0, + "HITCOUNT": 2.0, + "KILLCOUNT": 2.0 + }, + "invalid_action": false, + "lane": 4, + "length": 306, + "mean_action_latency_ms": 2.176532607062969, + "policy_error": null, + "return": 40.0, + "seed": 105, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 40.0, + "max_steps": 1050, + "mean_action_latency_ms": 2.1571807302505146, + "mean_return": 28.0, + "min_return": 0.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 28.0 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 35.0, + "HEALTH": 90.0, + "HITCOUNT": 4.0, + "KILLCOUNT": 4.0 + }, + "invalid_action": false, + "lane": 0, + "length": 1050, + "mean_action_latency_ms": 2.0629346193302247, + "policy_error": null, + "return": 75.0, + "seed": 607, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 10.0, + "HEALTH": 10.0, + "HITCOUNT": 1.0, + "KILLCOUNT": 1.0 + }, + "invalid_action": false, + "lane": 1, + "length": 262, + "mean_action_latency_ms": 2.085507470574088, + "policy_error": null, + "return": 20.0, + "seed": 608, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 4.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 2, + "length": 473, + "mean_action_latency_ms": 2.0426902880628317, + "policy_error": null, + "return": 0.0, + "seed": 609, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 0.0, + "HEALTH": 19.0, + "HITCOUNT": 0.0, + "KILLCOUNT": 0.0 + }, + "invalid_action": false, + "lane": 3, + "length": 400, + "mean_action_latency_ms": 2.0749272871762514, + "policy_error": null, + "return": 0.0, + "seed": 610, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 60.0, + "HEALTH": 19.0, + "HITCOUNT": 5.0, + "KILLCOUNT": 5.0 + }, + "invalid_action": false, + "lane": 4, + "length": 373, + "mean_action_latency_ms": 2.0402100027086907, + "policy_error": null, + "return": 110.0, + "seed": 611, + "terminated": true, + "truncated": false + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 110.0, + "max_steps": 1050, + "mean_action_latency_ms": 2.061253933570417, + "mean_return": 41.0, + "min_return": 0.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 41.0 + } + } +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-05/metadata.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-05/metadata.json new file mode 100644 index 0000000..d2453ec --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-05/metadata.json @@ -0,0 +1,31 @@ +{ + "benchmark": "learning_beyond_gradients_article_suite", + "current_task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "digest_compatibility_note": null, + "finished_at": 1784098371.002007, + "harness": "opencode", + "model": { + "display_name": "GPT-5.6 Sol", + "id": "gpt-5.6-sol", + "model": "azure/gpt-5.6-sol", + "provider": "azure", + "provider_reasoning_effort": "max" + }, + "normalized_score": 44.852941176, + "observed_raw_score": 166.1, + "provider_reasoning_effort": "max", + "raw_score": 166.1, + "raw_score_source": "observed", + "source_run_id": "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05", + "task": "simulation_heuristics_vizdoom_d3_v1", + "task_digest": "sha256:a1da044451acdb98fa331f2742def1693b3233098571b4d5aee5ecd43124617c", + "token_usage": { + "final_input_tokens": 543.0, + "final_output_tokens": 169.0, + "input_tokens": 543.0, + "output_tokens": 169.0, + "total_tokens": 22728.0 + }, + "tool_calls": 22, + "trial": 5 +} diff --git a/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-05/score.json b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-05/score.json new file mode 100644 index 0000000..f372e2b --- /dev/null +++ b/leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/trial-05/score.json @@ -0,0 +1,229 @@ +{ + "isolation": { + "mode": "subprocess_per_suite", + "suite_process_count": 2, + "unique_working_directories": true + }, + "normalized_score": 44.852941176, + "reference_score": 331.1, + "score": 166.1, + "starter_score": 31.9, + "suites": { + "hidden_nominal": { + "batch_seed": 101, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 70.0, + "HEALTH": 15.0, + "HITCOUNT": 5.0, + "KILLCOUNT": 5.0 + }, + "invalid_action": false, + "lane": 0, + "length": 744, + "mean_action_latency_ms": 2.425014847509002, + "policy_error": null, + "return": 120.0, + "seed": 101, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 80.0, + "HEALTH": 100.0, + "HITCOUNT": 9.0, + "KILLCOUNT": 9.0 + }, + "invalid_action": false, + "lane": 1, + "length": 1050, + "mean_action_latency_ms": 2.3797496940408434, + "policy_error": null, + "return": 170.0, + "seed": 102, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 40.0, + "HEALTH": 3.0, + "HITCOUNT": 5.0, + "KILLCOUNT": 5.0 + }, + "invalid_action": false, + "lane": 2, + "length": 691, + "mean_action_latency_ms": 2.3115585125475993, + "policy_error": null, + "return": 90.0, + "seed": 103, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 5.0, + "DAMAGECOUNT": 85.0, + "HEALTH": 3.0, + "HITCOUNT": 11.0, + "KILLCOUNT": 10.0 + }, + "invalid_action": false, + "lane": 3, + "length": 757, + "mean_action_latency_ms": 2.3620919111619063, + "policy_error": null, + "return": 185.0, + "seed": 104, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 110.0, + "HEALTH": 100.0, + "HITCOUNT": 11.0, + "KILLCOUNT": 10.0 + }, + "invalid_action": false, + "lane": 4, + "length": 1050, + "mean_action_latency_ms": 2.3375741782642545, + "policy_error": null, + "return": 210.0, + "seed": 105, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 210.0, + "max_steps": 1050, + "mean_action_latency_ms": 2.3631978287047213, + "mean_return": 155.0, + "min_return": 90.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 155.0 + }, + "hidden_secondary": { + "batch_seed": 607, + "envpool_version": "1.1.1", + "episodes": [ + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 135.0, + "HEALTH": 2.0, + "HITCOUNT": 10.0, + "KILLCOUNT": 10.0 + }, + "invalid_action": false, + "lane": 0, + "length": 1029, + "mean_action_latency_ms": 2.120358226829877, + "policy_error": null, + "return": 235.0, + "seed": 607, + "terminated": true, + "truncated": false + }, + { + "final_variables": { + "AMMO2": 6.0, + "DAMAGECOUNT": 80.0, + "HEALTH": 52.0, + "HITCOUNT": 8.0, + "KILLCOUNT": 8.0 + }, + "invalid_action": false, + "lane": 1, + "length": 1050, + "mean_action_latency_ms": 2.1108544440496537, + "policy_error": null, + "return": 160.0, + "seed": 608, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 80.0, + "HEALTH": 100.0, + "HITCOUNT": 7.0, + "KILLCOUNT": 7.0 + }, + "invalid_action": false, + "lane": 2, + "length": 1050, + "mean_action_latency_ms": 2.055742321979432, + "policy_error": null, + "return": 150.0, + "seed": 609, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 0.0, + "DAMAGECOUNT": 70.0, + "HEALTH": 41.0, + "HITCOUNT": 8.0, + "KILLCOUNT": 7.0 + }, + "invalid_action": false, + "lane": 3, + "length": 1050, + "mean_action_latency_ms": 2.111986737166132, + "policy_error": null, + "return": 140.0, + "seed": 610, + "terminated": false, + "truncated": true + }, + { + "final_variables": { + "AMMO2": 6.0, + "DAMAGECOUNT": 135.0, + "HEALTH": 61.0, + "HITCOUNT": 15.0, + "KILLCOUNT": 14.0 + }, + "invalid_action": false, + "lane": 4, + "length": 1050, + "mean_action_latency_ms": 2.0210168226843788, + "policy_error": null, + "return": 275.0, + "seed": 611, + "terminated": false, + "truncated": true + } + ], + "frame_skip": 2, + "invalid_episode_rate": 0.0, + "max_return": 275.0, + "max_steps": 1050, + "mean_action_latency_ms": 2.083991710541895, + "mean_return": 192.0, + "min_return": 140.0, + "policy_path": "submitted_artifact", + "render_height": 480, + "render_width": 640, + "scenario": "d3", + "score": 192.0 + } + } +} diff --git a/leaderboard/article_suite_task_leaderboards.png b/leaderboard/article_suite_task_leaderboards.png index dd44da9..18766cf 100644 Binary files a/leaderboard/article_suite_task_leaderboards.png and b/leaderboard/article_suite_task_leaderboards.png differ diff --git a/scripts/build_article_suite_leaderboard.py b/scripts/build_article_suite_leaderboard.py index 16ff80e..f1741d7 100644 --- a/scripts/build_article_suite_leaderboard.py +++ b/scripts/build_article_suite_leaderboard.py @@ -16,6 +16,7 @@ REPO_ROOT = Path(__file__).resolve().parents[1] MODELS_PATH = REPO_ROOT / "experiments" / "article_suite" / "models.toml" +PROTOCOL_PATH = REPO_ROOT / "experiments" / "article_suite" / "protocol.toml" TASKS = ( "simulation_heuristics_ant_v1", "simulation_heuristics_pong_ram_v1", @@ -149,6 +150,63 @@ def _result_files(model_root: Path) -> list[Path]: return candidates +def _individual_result_rows(model_root: Path) -> list[dict[str, Any]]: + candidates = sorted( + (model_root / "jobs").glob("*/*/result.json"), + key=lambda path: path.stat().st_mtime, + ) + rows = [] + for path in candidates: + result = json.loads(path.read_text()) + task_name = result.get("task_name") + rewards = result.get("rewards") + reward = rewards.get("reward") if isinstance(rewards, dict) else None + error = result.get("error") + verifier_error = result.get("verifier_error") + if error is None and verifier_error is not None: + error = { + "error": result.get("verifier_error_category") + or "verifier_error", + "message": verifier_error, + } + rows.append( + { + "info": { + "task_name": task_name, + "rollout_dir": str(path.parent), + }, + "reward": reward, + "error": error, + "metrics": {"n_tool_calls": result.get("n_tool_calls")}, + "token_usage": result.get("token_usage"), + "total_tool_calls": result.get("n_tool_calls"), + } + ) + return rows + + +def _task_digests_for_run(model_root: Path) -> dict[str, str]: + manifest_path = model_root / "task_manifest.json" + if manifest_path.is_file(): + manifest = json.loads(manifest_path.read_text()) + return { + item.get("task_id"): item.get("digest") + for item in manifest.get("tasks", []) + if isinstance(item, dict) + and isinstance(item.get("task_id"), str) + and isinstance(item.get("digest"), str) + } + + digests: dict[str, str] = {} + for path in (model_root / "jobs").glob("*/*/result.json"): + result = json.loads(path.read_text()) + task_name = result.get("task_name") + digest = result.get("task_digest") + if isinstance(task_name, str) and isinstance(digest, str): + digests[task_name] = digest + return digests + + def _load_results( model_root: Path, *, @@ -156,16 +214,24 @@ def _load_results( allow_partial_errors: bool = False, ) -> dict[str, dict[str, Any]]: files = _result_files(model_root) - if len(files) != 1: + if len(files) > 1: raise RuntimeError( - f"{model_root} must contain exactly one top-level results.jsonl; " + f"{model_root} must contain at most one top-level results.jsonl; " f"found {len(files)}" ) + if files: + source_rows = [ + json.loads(line) + for line in files[0].read_text().splitlines() + if line.strip() + ] + else: + source_rows = _individual_result_rows(model_root) + if not source_rows: + raise RuntimeError(f"{model_root} contains no result rows") rows: dict[str, dict[str, Any]] = {} - for line in files[0].read_text().splitlines(): - if not line.strip(): - continue - row = json.loads(line) + invalid_messages: dict[str, str] = {} + for row in source_rows: task_name = row.get("info", {}).get("task_name") if task_name not in expected_tasks: continue @@ -183,21 +249,30 @@ def _load_results( isinstance(error, dict) and error.get("error") == "missing_llm_trajectory" ) and not has_score_details: - if allow_partial_errors: - continue - raise RuntimeError(f"{model_root.name}/{task_name} contains an error") + invalid_messages[task_name] = ( + f"{model_root.name}/{task_name} contains an error" + ) + continue reward = row.get("reward") if ( not isinstance(reward, int | float) or isinstance(reward, bool) or not math.isfinite(float(reward)) ): - raise RuntimeError( + invalid_messages[task_name] = ( f"{model_root.name}/{task_name} has invalid reward {reward!r}" ) + continue rows[task_name] = row + invalid_messages.pop(task_name, None) missing = sorted(set(expected_tasks) - set(rows)) if missing and not allow_partial_errors: + invalid_task = next( + (task for task in missing if task in invalid_messages), + None, + ) + if invalid_task is not None: + raise RuntimeError(invalid_messages[invalid_task]) raise RuntimeError( f"{model_root.name} is missing article-suite tasks: " + ", ".join(missing) @@ -251,6 +326,105 @@ def _expected_models() -> dict[str, dict[str, Any]]: return {model["id"]: model for model in payload["models"]} +def _protocol() -> dict[str, Any]: + return tomllib.loads(PROTOCOL_PATH.read_text()) + + +def _finite_number(value: Any) -> bool: + return ( + isinstance(value, int | float) + and not isinstance(value, bool) + and math.isfinite(float(value)) + ) + + +def _fallback_task_anchors( + task_results: dict[ + str, + dict[ + str, + dict[int, tuple[dict[str, Any], dict[str, Any], Path, str]], + ], + ], +) -> dict[str, tuple[float, float]]: + candidates: dict[str, list[tuple[float, float]]] = { + task: [] for task in TASKS + } + for model_results in task_results.values(): + for task, trial_results in model_results.items(): + for _, result_row, _, _ in trial_results.values(): + rollout_dir = Path(result_row["info"]["rollout_dir"]) + score_path = rollout_dir / "verifier" / "genesis-score.json" + if not score_path.is_file(): + continue + score = json.loads(score_path.read_text()) + starter = score.get("starter_score") + reference = score.get("reference_score") + if _finite_number(starter) and _finite_number(reference): + candidates[task].append( + (float(starter), float(reference)) + ) + anchors = {} + for task, values in candidates.items(): + if not values: + raise RuntimeError(f"{task} has no numeric starter/reference anchors") + anchors[task] = ( + statistics.fmean(value[0] for value in values), + statistics.fmean(value[1] for value in values), + ) + return anchors + + +def _resolve_raw_score( + score: dict[str, Any], + *, + normalized_score: float, + fallback_anchor: tuple[float, float], +) -> tuple[float, float, float, str, float | None]: + raw = score.get("score") + starter = score.get("starter_score") + reference = score.get("reference_score") + if ( + _finite_number(raw) + and _finite_number(starter) + and _finite_number(reference) + ): + return ( + float(raw), + float(starter), + float(reference), + "observed", + float(raw), + ) + if ( + score.get("verifier_timeout") is True + and normalized_score == 0.0 + ): + fallback_starter, fallback_reference = fallback_anchor + return ( + fallback_starter, + fallback_starter, + fallback_reference, + "starter_anchor_equivalent_for_fail_closed_timeout", + None, + ) + raise RuntimeError( + "score artifact has no finite raw score and is not a fail-closed " + "verifier timeout" + ) + + +def _execution_protocol(protocol: dict[str, Any]) -> dict[str, Any]: + return { + key: protocol.get(key) + for key in ( + "trials", + "agent_timeout_multiplier", + "baseline_agent_timeout_sec", + ) + } + + def _rank_rows( rows: list[dict[str, Any]], *, @@ -285,13 +459,61 @@ def _interquartile_mean(scores: list[float]) -> float: return math.fsum(retained) / len(retained) -def _aggregate_task_scores(task_scores: dict[str, float]) -> dict[str, float]: +def _aggregate_task_scores( + task_scores: dict[str, float], + *, + trial_task_scores: dict[int, dict[str, float]] | None = None, +) -> dict[str, Any]: scores = [float(task_scores[task]) for task in TASKS] - return { + aggregate: dict[str, Any] = { "final_normalized_score": _interquartile_mean(scores), "arithmetic_mean_normalized_score": math.fsum(scores) / len(scores), "median_normalized_score": float(statistics.median(scores)), } + if trial_task_scores: + pooled_scores = [ + float(scores_for_trial[task]) + for _, scores_for_trial in sorted(trial_task_scores.items()) + for task in TASKS + ] + trial_final_scores = { + trial: _interquartile_mean( + [float(task_scores_for_trial[task]) for task in TASKS] + ) + for trial, task_scores_for_trial in sorted( + trial_task_scores.items() + ) + } + values = list(trial_final_scores.values()) + aggregate.update( + { + "final_normalized_score": _interquartile_mean(pooled_scores), + "arithmetic_mean_normalized_score": statistics.fmean( + pooled_scores + ), + "median_normalized_score": float( + statistics.median(pooled_scores) + ), + "final_normalized_score_stddev": statistics.stdev(values), + "mean_trial_iqm_normalized_score": statistics.fmean(values), + "trial_final_normalized_scores": trial_final_scores, + "trial_task_normalized_scores": trial_task_scores, + } + ) + else: + aggregate.update( + { + "final_normalized_score_stddev": 0.0, + "mean_trial_iqm_normalized_score": aggregate[ + "final_normalized_score" + ], + "trial_final_normalized_scores": { + 1: aggregate["final_normalized_score"] + }, + "trial_task_normalized_scores": {1: task_scores}, + } + ) + return aggregate def _build_leaderboards( @@ -307,11 +529,20 @@ def _build_leaderboards( "provider": row["provider"], "provider_label": row["provider_label"], "harness": row["harness"], + "sandbox": row["sandbox"], "provider_reasoning_effort": row[ "provider_reasoning_effort" ], "normalized_score": row["task_scores"][task], + "normalized_score_stddev": row[ + "task_score_stddevs" + ][task], "raw_score": row["raw_task_scores"][task], + "raw_score_stddev": row["raw_task_score_stddevs"][task], + "raw_score_imputation_count": row[ + "raw_task_score_imputation_counts" + ][task], + "trial_count": row["trial_count"], "starter_score": row["task_anchors"][task][ "starter_score" ], @@ -344,6 +575,7 @@ def _build_leaderboards( "provider": row["provider"], "provider_label": row["provider_label"], "harness": row["harness"], + "sandbox": row["sandbox"], "provider_reasoning_effort": row[ "provider_reasoning_effort" ], @@ -352,6 +584,13 @@ def _build_leaderboards( "arithmetic_mean_normalized_score" ], "median_normalized_score": row["median_normalized_score"], + "mean_trial_iqm_normalized_score": row[ + "mean_trial_iqm_normalized_score" + ], + "final_normalized_score_stddev": row[ + "final_normalized_score_stddev" + ], + "trial_count": row["trial_count"], "positive_display_score": row["final_normalized_score"] + FINAL_DISPLAY_OFFSET, "average_normalized_score": row["average_normalized_score"], @@ -388,6 +627,15 @@ def _leaderboard_relative_path(path: str) -> str: return detail_path.as_posix() +def _published_artifact_path(path: Path, *, output_parent: Path) -> str: + path = path.resolve() + output_parent = output_parent.resolve() + try: + return str(path.relative_to(REPO_ROOT)) + except ValueError: + return str(path.relative_to(output_parent)) + + def _render_article_suite_markdown( leaderboards: list[dict[str, Any]], ) -> str: @@ -401,20 +649,22 @@ def _render_article_suite_markdown( "The first image contains the nine independently ranked task " "leaderboards. The second image contains the final cross-task ranking.", "", - "The nine task panels use each environment's native raw score. " - "The final scientific metric remains unbounded IQM.", + "The nine task panels report five-trial mean ± sample standard " + "deviation for each environment's native raw score.", "", - "## Nine task leaderboards", + "## Leaderboards", "", "![Nine task-specific GenesisBench leaderboards]" "(article_suite_task_leaderboards.png)", "", "## Final normalized score", "", - "The primary score is the interquartile mean (IQM): sort the nine task " - "scores, remove the lowest two and highest two, then average the middle " - "five. The image uses a plot-only positive display index equal to " - "`IQM + 100`; raw IQM, arithmetic mean, and median remain in the JSON.", + "The final score is RLiable-style IQM over the complete 5 × 9 " + "trial-task score matrix: flatten all 45 normalized scores, trim the " + "lowest 11 and highest 11, and average the middle 23. The displayed " + "± value is the sample standard deviation of the five per-trial IQMs. " + "The image uses a plot-only positive display index equal to `IQM + " + "100`; raw metrics remain in the JSON.", "", "![Final GenesisBench article-suite leaderboard]" "(article_suite_final_leaderboard.png)", @@ -473,15 +723,7 @@ def _latest_task_results( ): continue model_root = metadata_path.parent - manifest_path = model_root / "task_manifest.json" - if not manifest_path.is_file(): - continue - manifest = json.loads(manifest_path.read_text()) - manifest_digests = { - item.get("task_id"): item.get("digest") - for item in manifest.get("tasks", []) - if isinstance(item, dict) - } + manifest_digests = _task_digests_for_run(model_root) if any( not isinstance(manifest_digests.get(task), str) for task in configured_tasks @@ -508,146 +750,400 @@ def _latest_task_results( return selected +def _trial_batch_results( + batch_root: Path, + *, + expected_models: set[str], + expected_trials: int, +) -> dict[ + str, + dict[str, dict[int, tuple[dict[str, Any], dict[str, Any], Path, str]]], +]: + selected: dict[ + str, + dict[str, dict[int, tuple[dict[str, Any], dict[str, Any], Path, str]]], + ] = {} + metadata_paths = [ + *batch_root.glob("*/trial-*/run_metadata.json"), + *batch_root.glob("*/trial-*/*/run_metadata.json"), + ] + for metadata_path in metadata_paths: + metadata = json.loads(metadata_path.read_text()) + model_id = metadata.get("model", {}).get("id") + trial = metadata.get("trial") + configured_tasks = metadata.get("tasks") + finished_at = metadata.get("finished_at") + if ( + model_id not in expected_models + or not isinstance(trial, int) + or not 1 <= trial <= expected_trials + or not isinstance(configured_tasks, list) + or not configured_tasks + or not all(task in TASKS for task in configured_tasks) + or not isinstance(finished_at, int | float) + or metadata.get("dry_run") + ): + continue + model_root = metadata_path.parent + manifest_digests = _task_digests_for_run(model_root) + try: + rows = _load_results( + model_root, + expected_tasks=tuple(configured_tasks), + allow_partial_errors=True, + ) + except RuntimeError: + continue + for task, row in rows.items(): + digest = manifest_digests.get(task) + if not isinstance(digest, str): + continue + task_trials = selected.setdefault(model_id, {}).setdefault( + task, + {}, + ) + current = task_trials.get(trial) + candidate = ( + metadata, + row, + model_root, + digest, + ) + if ( + current is None + or float(metadata["finished_at"]) + > float(current[0]["finished_at"]) + ): + task_trials[trial] = candidate + return selected + + +def _trial_batch_missing( + results: dict[ + str, + dict[str, dict[int, tuple[dict[str, Any], dict[str, Any], Path, str]]], + ], + *, + expected_models: set[str], + expected_trials: int, +) -> list[str]: + missing = [] + for model_id in sorted(expected_models): + for task in TASKS: + trials = results.get(model_id, {}).get(task, {}) + for trial in range(1, expected_trials + 1): + if trial not in trials: + missing.append(f"{model_id}/{task}/trial-{trial:02d}") + return missing + + +def _latest_complete_model_batches( + runs_root: Path, + *, + expected_models: set[str], + protocol: dict[str, Any], +) -> tuple[ + dict[str, Path], + dict[ + str, + dict[ + str, + dict[int, tuple[dict[str, Any], dict[str, Any], Path, str]], + ], + ], +]: + expected_trials = int(protocol["trials"]) + manifests = [] + for manifest_path in runs_root.glob("*/batch_manifest.json"): + manifest = json.loads(manifest_path.read_text()) + if _execution_protocol( + manifest.get("protocol", {}) + ) == _execution_protocol(protocol): + manifests.append(manifest_path.parent) + + selected_batches: dict[str, Path] = {} + selected_results: dict[ + str, + dict[ + str, + dict[int, tuple[dict[str, Any], dict[str, Any], Path, str]], + ], + ] = {} + for model_id in sorted(expected_models): + candidates = [] + incomplete: list[tuple[Path, int]] = [] + for batch_root in manifests: + results = _trial_batch_results( + batch_root, + expected_models={model_id}, + expected_trials=expected_trials, + ) + missing = _trial_batch_missing( + results, + expected_models={model_id}, + expected_trials=expected_trials, + ) + if missing: + incomplete.append((batch_root, len(missing))) + continue + finished_at = max( + entry[0]["finished_at"] + for task_results in results[model_id].values() + for entry in task_results.values() + ) + candidates.append( + (float(finished_at), batch_root, results[model_id]) + ) + if not candidates: + detail = ( + ", ".join( + f"{path.name}: {missing_count} missing" + for path, missing_count in sorted(incomplete)[-3:] + ) + or "no matching protocol batches" + ) + raise RuntimeError( + f"No complete {expected_trials}-trial article-suite batch is " + f"available for {model_id}: {detail}" + ) + _, batch_root, model_results = max( + candidates, + key=lambda item: item[0], + ) + selected_batches[model_id] = batch_root + selected_results[model_id] = model_results + return selected_batches, selected_results + + def main() -> None: args = parse_args() runs_root = args.runs_root.resolve() expected_models = _expected_models() - task_results = _latest_task_results(runs_root) - missing_models = sorted(set(expected_models) - set(task_results)) - if missing_models: - raise RuntimeError( - "Missing completed article-suite model runs: " - + ", ".join(missing_models) - ) + protocol = _protocol() + trial_count = int(protocol["trials"]) + model_batch_roots, task_results = _latest_complete_model_batches( + runs_root, + expected_models=set(expected_models), + protocol=protocol, + ) + fallback_task_anchors = _fallback_task_anchors(task_results) ranked: list[dict[str, Any]] = [] submissions_root = args.output.parent / "article_suite_submissions" for model_id in expected_models: model_results = task_results[model_id] - missing_tasks = sorted(set(TASKS) - set(model_results)) - if missing_tasks: - raise RuntimeError( - f"{model_id} is missing completed task results: " - + ", ".join(missing_tasks) - ) - stale_tasks = [ - task - for task in TASKS - if _digest_compatibility_note( - task, - model_results[task][3], - task_digest(REPO_ROOT / "tasks" / task), - ) - is None - and model_results[task][3] - != task_digest(REPO_ROOT / "tasks" / task) - ] - if stale_tasks: - raise RuntimeError( - f"{model_id} has stale task digests: " - + ", ".join(stale_tasks) - ) - latest_task = max( - model_results.values(), - key=lambda item: item[0], - ) - metadata = json.loads( - (latest_task[2] / "run_metadata.json").read_text() - ) - task_scores = { - task: _normalized_task_score(model_results[task][1]) - for task in TASKS - } + representative_metadata = model_results[TASKS[0]][1][0] + task_scores: dict[str, float] = {} + task_score_stddevs: dict[str, float] = {} raw_task_scores: dict[str, float] = {} + raw_task_score_stddevs: dict[str, float] = {} + raw_task_score_imputation_counts: dict[str, int] = {} task_anchors: dict[str, dict[str, float]] = {} submission_details: dict[str, str] = {} + source_runs: dict[str, list[str]] = {} + trial_task_scores = { + trial: {} for trial in range(1, trial_count + 1) + } for task in TASKS: - _, result_row, model_root, digest = model_results[task] + trial_results = model_results[task] current_digest = task_digest(REPO_ROOT / "tasks" / task) - compatibility_note = _digest_compatibility_note( - task, - digest, - current_digest, - ) - task_metadata = json.loads( - (model_root / "run_metadata.json").read_text() - ) - rollout_dir = Path(result_row["info"]["rollout_dir"]) - source_score = rollout_dir / "verifier" / "genesis-score.json" - if not source_score.is_file(): - raise RuntimeError( - f"{model_id}/{task} has no verifier genesis-score.json" - ) destination = submissions_root / model_id / task + if destination.exists(): + shutil.rmtree(destination) destination.mkdir(parents=True, exist_ok=True) - sanitized_score = _sanitize_score_paths( - json.loads(source_score.read_text()) - ) - raw_score = sanitized_score.get("score") - starter_score = sanitized_score.get("starter_score") - reference_score = sanitized_score.get("reference_score") - for name, value in ( - ("score", raw_score), - ("starter_score", starter_score), - ("reference_score", reference_score), - ): - if ( - not isinstance(value, int | float) - or isinstance(value, bool) - or not math.isfinite(float(value)) - ): + normalized_values = [] + raw_values = [] + starter_values = [] + reference_values = [] + raw_score_imputation_count = 0 + trial_records = [] + for trial in range(1, trial_count + 1): + task_metadata, result_row, model_root, digest = trial_results[ + trial + ] + compatibility_note = _digest_compatibility_note( + task, + digest, + current_digest, + ) + if digest != current_digest and compatibility_note is None: + raise RuntimeError( + f"{model_id}/{task}/trial-{trial:02d} has stale " + f"task digest {digest}" + ) + rollout_dir = Path(result_row["info"]["rollout_dir"]) + source_score = ( + rollout_dir / "verifier" / "genesis-score.json" + ) + if not source_score.is_file(): raise RuntimeError( - f"{model_id}/{task} has invalid {name} {value!r}" + f"{model_id}/{task}/trial-{trial:02d} has no " + "verifier genesis-score.json" + ) + sanitized_score = _sanitize_score_paths( + json.loads(source_score.read_text()) + ) + normalized = _normalized_task_score(result_row) + try: + ( + raw_score, + starter_score, + reference_score, + raw_score_source, + observed_raw_score, + ) = _resolve_raw_score( + sanitized_score, + normalized_score=normalized, + fallback_anchor=fallback_task_anchors[task], + ) + except RuntimeError as exc: + raise RuntimeError( + f"{model_id}/{task}/trial-{trial:02d}: {exc}" + ) from exc + if raw_score_source != "observed": + raw_score_imputation_count += 1 + trial_destination = destination / f"trial-{trial:02d}" + trial_destination.mkdir(parents=True, exist_ok=True) + trial_score_path = trial_destination / "score.json" + trial_score_path.write_text( + json.dumps( + sanitized_score, + indent=2, + sort_keys=True, + ) + + "\n" + ) + source_run = str(model_root.relative_to(REPO_ROOT)) + trial_metadata = { + "benchmark": "learning_beyond_gradients_article_suite", + "task": task, + "trial": trial, + "task_digest": digest, + "current_task_digest": current_digest, + "digest_compatibility_note": compatibility_note, + "model": task_metadata["model"], + "harness": task_metadata["harness"], + "provider_reasoning_effort": task_metadata[ + "provider_reasoning_effort" + ], + "normalized_score": normalized, + "raw_score": float(raw_score), + "observed_raw_score": observed_raw_score, + "raw_score_source": raw_score_source, + "source_run_id": source_run, + "finished_at": task_metadata["finished_at"], + "tool_calls": result_row.get("metrics", {}).get( + "n_tool_calls", + result_row.get("total_tool_calls"), + ), + "token_usage": result_row.get("token_usage"), + } + (trial_destination / "metadata.json").write_text( + json.dumps( + trial_metadata, + indent=2, + sort_keys=True, ) - raw_task_scores[task] = float(raw_score) + + "\n" + ) + normalized_values.append(normalized) + raw_values.append(float(raw_score)) + starter_values.append(float(starter_score)) + reference_values.append(float(reference_score)) + trial_task_scores[trial][task] = normalized + trial_records.append( + { + "trial": trial, + "normalized_score": normalized, + "raw_score": float(raw_score), + "observed_raw_score": observed_raw_score, + "raw_score_source": raw_score_source, + "score_path": _published_artifact_path( + trial_score_path, + output_parent=args.output.parent, + ), + "source_run_id": source_run, + } + ) + + task_scores[task] = statistics.fmean(normalized_values) + task_score_stddevs[task] = statistics.stdev(normalized_values) + raw_task_scores[task] = statistics.fmean(raw_values) + raw_task_score_stddevs[task] = statistics.stdev(raw_values) + raw_task_score_imputation_counts[task] = ( + raw_score_imputation_count + ) task_anchors[task] = { - "starter_score": float(starter_score), - "reference_score": float(reference_score), + "starter_score": statistics.fmean(starter_values), + "reference_score": statistics.fmean(reference_values), } - (destination / "score.json").write_text( - json.dumps(sanitized_score, indent=2, sort_keys=True) + "\n" + aggregate_score = { + "benchmark": "learning_beyond_gradients_article_suite", + "task": task, + "trial_count": trial_count, + "normalized_score": task_scores[task], + "normalized_score_stddev": task_score_stddevs[task], + "score": raw_task_scores[task], + "score_stddev": raw_task_score_stddevs[task], + "raw_score_imputation_count": raw_score_imputation_count, + "starter_score": task_anchors[task]["starter_score"], + "reference_score": task_anchors[task]["reference_score"], + "trials": trial_records, + } + aggregate_score_path = destination / "score.json" + aggregate_score_path.write_text( + json.dumps(aggregate_score, indent=2, sort_keys=True) + "\n" ) - source_run = str(model_root.relative_to(REPO_ROOT)) metadata_payload = { "benchmark": "learning_beyond_gradients_article_suite", "task": task, - "task_digest": digest, + "trial_count": trial_count, "current_task_digest": current_digest, - "digest_compatibility_note": compatibility_note, - "model": task_metadata["model"], - "harness": task_metadata["harness"], - "provider_reasoning_effort": task_metadata[ + "model": representative_metadata["model"], + "harness": representative_metadata["harness"], + "provider_reasoning_effort": representative_metadata[ "provider_reasoning_effort" ], "normalized_score": task_scores[task], - "source_run_id": source_run, - "finished_at": task_metadata["finished_at"], - "tool_calls": result_row.get("metrics", {}).get( - "n_tool_calls", - result_row.get("total_tool_calls"), - ), - "token_usage": result_row.get("token_usage"), + "normalized_score_stddev": task_score_stddevs[task], + "raw_score": raw_task_scores[task], + "raw_score_stddev": raw_task_score_stddevs[task], + "raw_score_imputation_count": raw_score_imputation_count, + "source_run_ids": [ + record["source_run_id"] for record in trial_records + ], } (destination / "metadata.json").write_text( json.dumps(metadata_payload, indent=2, sort_keys=True) + "\n" ) submission_details[task] = str( - (destination / "score.json").relative_to(REPO_ROOT) + _published_artifact_path( + aggregate_score_path, + output_parent=args.output.parent, + ) ) - aggregate_scores = _aggregate_task_scores(task_scores) + source_runs[task] = [ + record["source_run_id"] for record in trial_records + ] + aggregate_scores = _aggregate_task_scores( + task_scores, + trial_task_scores=trial_task_scores, + ) ranked.append( { - "model_id": metadata["model"]["id"], - "model": metadata["model"]["display_name"], - "model_route": metadata["model"]["model"], - "provider": metadata["model"]["provider"], + "model_id": representative_metadata["model"]["id"], + "model": representative_metadata["model"]["display_name"], + "model_route": representative_metadata["model"]["model"], + "provider": representative_metadata["model"]["provider"], "provider_label": PROVIDER_LABELS.get( - metadata["model"]["provider"], - metadata["model"]["provider"], + representative_metadata["model"]["provider"], + representative_metadata["model"]["provider"], ), - "harness": metadata["harness"], - "provider_reasoning_effort": metadata[ + "harness": representative_metadata["harness"], + "sandbox": representative_metadata["sandbox"], + "provider_reasoning_effort": representative_metadata[ "provider_reasoning_effort" ], + "trial_count": trial_count, **aggregate_scores, # Backward-compatible alias for consumers of the first # published schema. This is not the primary ranking metric. @@ -655,25 +1151,42 @@ def main() -> None: "arithmetic_mean_normalized_score" ], "task_scores": task_scores, + "task_score_stddevs": task_score_stddevs, "raw_task_scores": raw_task_scores, + "raw_task_score_stddevs": raw_task_score_stddevs, + "raw_task_score_imputation_counts": ( + raw_task_score_imputation_counts + ), "task_anchors": task_anchors, "submission_details": submission_details, - "source_runs": { - task: str( - model_results[task][2].relative_to(REPO_ROOT) - ) - for task in TASKS - }, + "source_runs": source_runs, } ) ranked = _rank_rows( ranked, score_key="final_normalized_score", ) + sandboxes = sorted({row["sandbox"] for row in ranked}) leaderboards = _build_leaderboards(ranked) + pooled_score_count = trial_count * len(TASKS) + pooled_trim_count = int(pooled_score_count * IQM_TRIM_FRACTION) payload = { "benchmark": "learning_beyond_gradients_article_suite", + "batch_id": ( + next(iter(model_batch_roots.values())).name + if len(set(model_batch_roots.values())) == 1 + else "per-model-batches" + ), + "batch_ids": { + model_id: batch_root.name + for model_id, batch_root in model_batch_roots.items() + }, + "protocol": protocol, + "execution": { + "sandbox": sandboxes[0] if len(sandboxes) == 1 else "mixed", + "sandboxes": sandboxes, + }, "task_count": len(TASKS), "leaderboard_count": len(leaderboards), "tasks": list(TASKS), @@ -681,17 +1194,25 @@ def main() -> None: "primary_metric": "interquartile_mean", "primary_field": "final_normalized_score", "trim_fraction_per_tail": IQM_TRIM_FRACTION, - "trimmed_score_count_per_tail": int( - len(TASKS) * IQM_TRIM_FRACTION - ), - "retained_score_count": len(TASKS) - - 2 * int(len(TASKS) * IQM_TRIM_FRACTION), + "pooled_score_count": pooled_score_count, + "trimmed_score_count_per_tail": pooled_trim_count, + "retained_score_count": pooled_score_count + - 2 * pooled_trim_count, "score_bounds": "unbounded", "secondary_fields": [ "arithmetic_mean_normalized_score", "median_normalized_score", + "mean_trial_iqm_normalized_score", ], - "uncertainty": "not_estimated_single_run_per_model_task", + "task_estimator": "mean_across_five_trials", + "task_variability": "sample_standard_deviation", + "trial_estimator": "iqm_across_nine_tasks", + "final_estimator": ( + "iqm_across_all_45_trial_task_normalized_scores" + ), + "final_variability": ( + "sample_standard_deviation_of_five_trial_iqms" + ), "display_transform": { "type": "additive_offset", "offset": FINAL_DISPLAY_OFFSET, @@ -715,6 +1236,7 @@ def main() -> None: "provider": row["provider"], "provider_label": row["provider_label"], "harness": row["harness"], + "sandbox": row["sandbox"], "provider_reasoning_effort": row[ "provider_reasoning_effort" ], @@ -724,17 +1246,14 @@ def main() -> None: }, "task_digest_compatibility": TASK_DIGEST_COMPATIBILITY, "source_runs": { - model_id: { - task: str(task_results[model_id][task][2].relative_to(REPO_ROOT)) - for task in TASKS - } - for model_id in expected_models + row["model_id"]: row["source_runs"] for row in ranked }, "generated_at": datetime.fromtimestamp( max( - item[0] + entry[0]["finished_at"] for model_results in task_results.values() - for item in model_results.values() + for task_trials in model_results.values() + for entry in task_trials.values() ), UTC, ).isoformat(), diff --git a/scripts/plot_article_suite_leaderboards.py b/scripts/plot_article_suite_leaderboards.py index 06bcc1e..3fcd3d6 100644 --- a/scripts/plot_article_suite_leaderboards.py +++ b/scripts/plot_article_suite_leaderboards.py @@ -67,13 +67,6 @@ def _score_limits(task_boards: list[dict[str, Any]]) -> tuple[float, float]: return lower, upper -def _bar_label_x(score: float, span: float) -> tuple[float, str]: - offset = span * 0.012 - if score >= 0: - return score + offset, "left" - return score - offset, "right" - - def _plot_task_leaderboards( task_boards: list[dict[str, Any]], output: Path, @@ -87,19 +80,38 @@ def _plot_task_leaderboards( rows = board["rows"] names = [row["model"] for row in rows] scores = [float(row["raw_score"]) for row in rows] + score_stddevs = [ + float(row.get("raw_score_stddev", 0.0)) for row in rows + ] starter_score = float(rows[0]["starter_score"]) reference_score = float(rows[0]["reference_score"]) - all_values = [*scores, starter_score, reference_score] + all_values = [ + *(score - stddev for score, stddev in zip(scores, score_stddevs)), + *(score + stddev for score, stddev in zip(scores, score_stddevs)), + starter_score, + reference_score, + ] value_min = min(all_values) value_max = max(all_values) - padding = max((value_max - value_min) * 0.16, abs(value_max) * 0.05, 1.0) + padding = max( + (value_max - value_min) * 0.35, + max(abs(value_min), abs(value_max)) * 0.08, + 1.0, + ) lower = min(0.0, value_min - padding) upper = max(0.0, value_max + padding) span = upper - lower colors = [ MODEL_COLORS.get(row["model_id"], "#666666") for row in rows ] - bars = axis.barh(names, scores, color=colors, height=0.62) + bars = axis.barh( + names, + scores, + xerr=score_stddevs, + color=colors, + height=0.62, + error_kw={"ecolor": "#333333", "elinewidth": 0.8, "capsize": 2}, + ) axis.invert_yaxis() axis.axvline( starter_score, @@ -134,23 +146,27 @@ def _plot_task_leaderboards( axis.spines[["top", "right", "left"]].set_visible(False) axis.tick_params(axis="y", length=0, labelsize=9.5) axis.tick_params(axis="x", labelsize=8.5) - for bar, score in zip(bars, scores, strict=True): - if score < lower + span * 0.08: - x = score + span * 0.012 + for bar, score, score_stddev in zip( + bars, + scores, + score_stddevs, + strict=True, + ): + if score >= 0: + x = score + score_stddev + span * 0.012 alignment = "left" - label_color = "#ffffff" else: - x, alignment = _bar_label_x(score, span) - label_color = "#171717" + x = score - score_stddev - span * 0.012 + alignment = "right" axis.text( x, bar.get_y() + bar.get_height() / 2, - f"{score:.1f}", + f"{score:.1f} ± {score_stddev:.1f}", va="center", ha=alignment, - fontsize=9, + fontsize=8.5, fontweight="bold", - color=label_color, + color="#171717", ) figure.suptitle( @@ -164,7 +180,8 @@ def _plot_task_leaderboards( figure.text( 0.055, 0.955, - "Native raw score for each environment · task-specific axes", + "Native raw score mean ± sample SD · task-specific axes · " + "raw-null fail-closed trials use starter-equivalent anchors", fontsize=11, color="#666666", ) @@ -207,6 +224,9 @@ def _plot_final_leaderboard( rows = final_board["rows"] names = [f"{row['rank']} {row['model']}" for row in rows] scores = [float(row["positive_display_score"]) for row in rows] + score_stddevs = [ + float(row.get("final_normalized_score_stddev", 0.0)) for row in rows + ] colors = [ MODEL_COLORS.get(row["model_id"], "#666666") for row in rows ] @@ -214,10 +234,30 @@ def _plot_final_leaderboard( figure, axis = plt.subplots(figsize=(12, 7.2), dpi=160) figure.patch.set_facecolor("#fafafa") axis.set_facecolor("#fafafa") - bars = axis.barh(names, scores, color=colors, height=0.62) + bars = axis.barh( + names, + scores, + xerr=score_stddevs, + color=colors, + height=0.62, + error_kw={"ecolor": "#333333", "elinewidth": 1.0, "capsize": 3}, + ) axis.invert_yaxis() lower = 0.0 - upper = max(160.0, math.ceil((max(scores) + 10.0) / 10.0) * 10.0) + upper = max( + 160.0, + math.ceil( + ( + max( + score + stddev + for score, stddev in zip(scores, score_stddevs) + ) + + 10.0 + ) + / 10.0 + ) + * 10.0, + ) span = upper - lower axis.set_xlim(lower, upper) axis.axvline( @@ -233,14 +273,21 @@ def _plot_final_leaderboard( axis.tick_params(axis="x", labelsize=10) axis.set_xlabel("Positive display index (IQM + 100)", fontsize=11) - for bar, row, score in zip(bars, rows, scores, strict=True): + for bar, row, score, score_stddev in zip( + bars, + rows, + scores, + score_stddevs, + strict=True, + ): label_color = ( "#171717" if row["model_id"] == "gpt-5.4-mini" else "#ffffff" ) axis.text( score - span * 0.015, bar.get_y() + bar.get_height() / 2, - f"{score:.2f} (IQM {row['final_normalized_score']:.2f})", + f"{score:.2f} ± {score_stddev:.2f} " + f"(IQM {row['final_normalized_score']:.2f})", va="center", ha="right", fontsize=10.5, diff --git a/scripts/run_article_suite.py b/scripts/run_article_suite.py index 032286d..d5ea14e 100644 --- a/scripts/run_article_suite.py +++ b/scripts/run_article_suite.py @@ -2,8 +2,10 @@ from __future__ import annotations import argparse +import hashlib import json import os +import shutil import subprocess import time import tomllib @@ -15,6 +17,7 @@ REPO_ROOT = Path(__file__).resolve().parents[1] MODELS_PATH = REPO_ROOT / "experiments" / "article_suite" / "models.toml" +PROTOCOL_PATH = REPO_ROOT / "experiments" / "article_suite" / "protocol.toml" TASKS = ( "simulation_heuristics_ant_v1", "simulation_heuristics_pong_ram_v1", @@ -43,6 +46,7 @@ def _default_env_path() -> Path: def parse_args() -> argparse.Namespace: + protocol = _protocol() parser = argparse.ArgumentParser( description="Run the nine-task article suite through OpenCode." ) @@ -62,15 +66,61 @@ def parse_args() -> argparse.Namespace: default=REPO_ROOT / "leaderboard" / "runs" / "article_suite", ) parser.add_argument("--concurrency", type=int, default=1) + parser.add_argument( + "--trials", + type=int, + default=int(protocol["trials"]), + help="Independent full-suite trials per model.", + ) + parser.add_argument( + "--trial", + action="append", + type=int, + help="Run only this 1-indexed trial; repeat for multiple trials.", + ) + parser.add_argument( + "--batch-id", + help="Stable batch directory name for resumable multi-invocation runs.", + ) + parser.add_argument( + "--force", + action="store_true", + help="Delete and rerun selected trial directories even if completed.", + ) + parser.add_argument( + "--fail-fast", + action="store_true", + help="Stop after the first failed model trial instead of continuing.", + ) parser.add_argument( "--sandbox", choices=("docker", "daytona"), - default="docker", + default="daytona", + ) + parser.add_argument( + "--docker-platform", + default="linux/amd64", + help="Calibrated target for optional local-Docker runs.", ) parser.add_argument("--dry-run", action="store_true") return parser.parse_args() +def _protocol() -> dict[str, Any]: + return tomllib.loads(PROTOCOL_PATH.read_text()) + + +def _execution_protocol(protocol: dict[str, Any]) -> dict[str, Any]: + return { + key: protocol.get(key) + for key in ( + "trials", + "agent_timeout_multiplier", + "baseline_agent_timeout_sec", + ) + } + + def _read_env(path: Path) -> dict[str, str]: result: dict[str, str] = {} if not path.is_file(): @@ -119,6 +169,77 @@ def _validate_tasks(tasks: tuple[str, ...]) -> None: ) +def _task_agent_timeout(task: str) -> int: + import yaml + + task_path = REPO_ROOT / "tasks" / task / "task.md" + parts = task_path.read_text().split("---", 2) + if len(parts) != 3: + raise RuntimeError(f"{task} has no YAML front matter") + document = yaml.safe_load(parts[1]) + timeout = document.get("agent", {}).get("timeout_sec") + if not isinstance(timeout, int) or timeout <= 0: + raise RuntimeError(f"{task} has invalid agent.timeout_sec {timeout!r}") + return timeout + + +def _validate_protocol(tasks: tuple[str, ...], trials: int) -> dict[str, Any]: + protocol = _protocol() + expected_trials = int(protocol["trials"]) + if trials != expected_trials: + raise ValueError( + f"--trials must match protocol.toml: expected " + f"{expected_trials}, got {trials}" + ) + multiplier = protocol["agent_timeout_multiplier"] + baselines = protocol["baseline_agent_timeout_sec"] + for task in tasks: + expected = int(baselines[task]) * int(multiplier) + actual = _task_agent_timeout(task) + if actual != expected: + raise RuntimeError( + f"{task} agent timeout {actual} does not match protocol " + f"{baselines[task]} x {multiplier} = {expected}" + ) + return protocol + + +def _selected_trials( + requested: list[int] | None, + *, + trial_count: int, +) -> tuple[int, ...]: + if requested is None: + return tuple(range(1, trial_count + 1)) + selected = tuple(dict.fromkeys(requested)) + invalid = [trial for trial in selected if not 1 <= trial <= trial_count] + if invalid: + raise ValueError( + f"--trial must be between 1 and {trial_count}: {invalid}" + ) + return selected + + +def _completed_trial(metadata_path: Path) -> bool: + if not metadata_path.is_file(): + return False + metadata = json.loads(metadata_path.read_text()) + return ( + metadata.get("status") == "completed" + and metadata.get("return_code") == 0 + and not metadata.get("dry_run") + ) + + +def _task_scope(tasks: tuple[str, ...]) -> str: + if tasks == TASKS: + return "full-suite" + if len(tasks) == 1: + return tasks[0] + digest = hashlib.sha256("\n".join(tasks).encode()).hexdigest()[:10] + return f"tasks-{digest}" + + def _require_credentials(model: dict[str, Any], env: dict[str, str]) -> None: required = { "azure": ("AZURE_API_ENDPOINT", "AZURE_API_KEY"), @@ -315,10 +436,22 @@ def _docker_ready() -> bool: return result.returncode == 0 +def _configure_docker_platform( + run_env: dict[str, str], + *, + platform: str, +) -> None: + if not platform.strip(): + raise ValueError("--docker-platform must not be empty") + run_env["DOCKER_DEFAULT_PLATFORM"] = platform + + def main() -> None: args = parse_args() tasks = tuple(args.task or TASKS) _validate_tasks(tasks) + protocol = _validate_protocol(tasks, args.trials) + selected_trials = _selected_trials(args.trial, trial_count=args.trials) models = _select_models(args.model, args.all_models) provider_env = os.environ.copy() provider_env.update(_read_env(args.env_file.expanduser())) @@ -326,8 +459,8 @@ def main() -> None: if not provider_env.get("DAYTONA_API_KEY"): raise RuntimeError("DAYTONA_API_KEY is required for --sandbox daytona") provider_env.setdefault("DAYTONA_TARGET", "us") - timestamp = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ") - batch_root = args.output_root.resolve() / timestamp + batch_id = args.batch_id or datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ") + batch_root = args.output_root.resolve() / batch_id batch_root.mkdir(parents=True, exist_ok=True) if args.sandbox == "docker" and not args.dry_run and not _docker_ready(): @@ -335,33 +468,56 @@ def main() -> None: "Docker is not ready. Start Docker Desktop before running the " "authoritative article-suite evaluation." ) + if args.sandbox == "docker": + _configure_docker_platform( + provider_env, + platform=args.docker_platform, + ) + manifest_path = batch_root / "batch_manifest.json" batch_manifest: dict[str, Any] = { "suite": "learning_beyond_gradients_article", + "batch_id": batch_id, + "protocol": protocol, "harness": "opencode", "tasks": list(tasks), "created_at": datetime.now(UTC).isoformat(), - "models": [], + "trial_count": args.trials, + "runs": [], } + if manifest_path.is_file(): + existing = json.loads(manifest_path.read_text()) + if _execution_protocol( + existing.get("protocol", {}) + ) != _execution_protocol(protocol): + raise RuntimeError( + f"{batch_root} uses incompatible trial or timeout settings" + ) + batch_manifest["created_at"] = existing.get( + "created_at", + batch_manifest["created_at"], + ) + batch_manifest["runs"] = list(existing.get("runs", [])) + + run_entries = { + ( + entry["model_id"], + entry["trial"], + entry.get("task_scope", "full-suite"), + ): { + **entry, + "task_scope": entry.get("task_scope", "full-suite"), + } + for entry in batch_manifest["runs"] + } + failures: list[tuple[str, int, int]] = [] for model in models: _require_credentials(model, provider_env) - model_root = batch_root / model["id"] - jobs_dir = model_root / "jobs" - model_root.mkdir(parents=True, exist_ok=True) run_env = _scoped_provider_env(model, provider_env) azure_resource_name = None if model["provider"] == "azure": azure_resource_name = _azure_resource_name(run_env) run_env["AZURE_RESOURCE_NAME"] = azure_resource_name - command = build_command( - model=model, - jobs_dir=jobs_dir, - sandbox=args.sandbox, - concurrency=args.concurrency, - artifact_dir=model_root, - tasks=tasks, - azure_resource_name=azure_resource_name, - ) run_env["BENCHFLOW_REASONING_EFFORT"] = model[ "provider_reasoning_effort" ] @@ -369,86 +525,152 @@ def main() -> None: run_env["BENCHFLOW_DAYTONA_PTY_READLINE_TIMEOUT"] = str( model["daytona_pty_readline_timeout_sec"] ) - started_at = time.time() - metadata = { - "model": model, - "harness": "opencode", - "sandbox": args.sandbox, - "tasks": list(tasks), - "command": command, - "provider_env_keys": sorted( - key - for key in ( - "AZURE_API_ENDPOINT", - "AZURE_API_KEY", - "CLAUDE_CODE_OAUTH_TOKEN", + for trial in selected_trials: + trial_root = batch_root / model["id"] / f"trial-{trial:02d}" + task_scope = _task_scope(tasks) + run_root = ( + trial_root + if task_scope == "full-suite" + else trial_root / task_scope + ) + metadata_path = run_root / "run_metadata.json" + if args.force and run_root.exists() and not args.dry_run: + shutil.rmtree(run_root) + if not args.force and _completed_trial(metadata_path): + print( + f"skip completed {model['id']} trial {trial} " + f"scope {task_scope}" ) - if run_env.get(key) - ), - "provider_reasoning_effort": model["provider_reasoning_effort"], - "started_at": started_at, - "finished_at": None, - "elapsed_seconds": None, - "return_code": None, - "status": "running", - "dry_run": args.dry_run, - } - metadata_path = model_root / "run_metadata.json" - metadata_path.write_text( - json.dumps(metadata, indent=2, sort_keys=True) + "\n" - ) - batch_entry = { - "id": model["id"], - "run_metadata": str(metadata_path.relative_to(batch_root)), - "return_code": None, - "status": "running", - } - batch_manifest["models"].append(batch_entry) - (batch_root / "batch_manifest.json").write_text( - json.dumps(batch_manifest, indent=2, sort_keys=True) + "\n" - ) - return_code = 0 - status = "completed" - try: - if args.dry_run: - print(" ".join(command)) - else: - return_code = subprocess.run( - command, - cwd=REPO_ROOT, - env=run_env, - check=False, - ).returncode - if return_code != 0: - status = "failed" - except KeyboardInterrupt: - return_code = 130 - status = "interrupted" - raise - finally: - metadata.update( - { - "finished_at": time.time(), - "elapsed_seconds": time.time() - started_at, - "return_code": return_code, - "status": status, - } + continue + + jobs_dir = run_root / "jobs" + run_root.mkdir(parents=True, exist_ok=True) + command = build_command( + model=model, + jobs_dir=jobs_dir, + sandbox=args.sandbox, + concurrency=args.concurrency, + artifact_dir=run_root, + tasks=tasks, + azure_resource_name=azure_resource_name, ) + started_at = time.time() + metadata = { + "batch_id": batch_id, + "trial": trial, + "trial_count": args.trials, + "task_scope": task_scope, + "protocol": protocol, + "model": model, + "harness": "opencode", + "sandbox": args.sandbox, + "docker_platform": ( + args.docker_platform if args.sandbox == "docker" else None + ), + "tasks": list(tasks), + "task_agent_timeout_sec": { + task: _task_agent_timeout(task) for task in tasks + }, + "command": command, + "provider_env_keys": sorted( + key + for key in ( + "AZURE_API_ENDPOINT", + "AZURE_API_KEY", + "CLAUDE_CODE_OAUTH_TOKEN", + ) + if run_env.get(key) + ), + "provider_reasoning_effort": model[ + "provider_reasoning_effort" + ], + "started_at": started_at, + "finished_at": None, + "elapsed_seconds": None, + "return_code": None, + "status": "running", + "dry_run": args.dry_run, + } metadata_path.write_text( json.dumps(metadata, indent=2, sort_keys=True) + "\n" ) - batch_entry["return_code"] = return_code - batch_entry["status"] = status - (batch_root / "batch_manifest.json").write_text( + batch_entry = run_entries.setdefault( + (model["id"], trial, task_scope), + { + "model_id": model["id"], + "trial": trial, + "task_scope": task_scope, + "run_metadata": str( + metadata_path.relative_to(batch_root) + ), + }, + ) + batch_entry.update({"return_code": None, "status": "running"}) + batch_manifest["runs"] = sorted( + run_entries.values(), + key=lambda entry: ( + entry["model_id"], + entry["trial"], + entry["task_scope"], + ), + ) + manifest_path.write_text( json.dumps(batch_manifest, indent=2, sort_keys=True) + "\n" ) - if return_code != 0: - raise SystemExit(return_code) - (batch_root / "batch_manifest.json").write_text( + return_code = 0 + status = "completed" + try: + if args.dry_run: + print(" ".join(command)) + else: + return_code = subprocess.run( + command, + cwd=REPO_ROOT, + env=run_env, + check=False, + ).returncode + if return_code != 0: + status = "failed" + except KeyboardInterrupt: + return_code = 130 + status = "interrupted" + raise + finally: + metadata.update( + { + "finished_at": time.time(), + "elapsed_seconds": time.time() - started_at, + "return_code": return_code, + "status": status, + } + ) + metadata_path.write_text( + json.dumps(metadata, indent=2, sort_keys=True) + "\n" + ) + batch_entry.update( + {"return_code": return_code, "status": status} + ) + manifest_path.write_text( + json.dumps(batch_manifest, indent=2, sort_keys=True) + "\n" + ) + if return_code != 0: + failures.append((model["id"], trial, return_code)) + if args.fail_fast: + raise SystemExit(return_code) + + batch_manifest["status"] = "failed" if failures else "completed" + batch_manifest["finished_at"] = datetime.now(UTC).isoformat() + batch_manifest["failures"] = [ + {"model_id": model_id, "trial": trial, "return_code": return_code} + for model_id, trial, return_code in failures + ] + manifest_path.write_text( json.dumps(batch_manifest, indent=2, sort_keys=True) + "\n" ) print(batch_root) + if failures: + raise SystemExit(1) if __name__ == "__main__": diff --git a/tasks/simulation_heuristics_ant_v1/task.md b/tasks/simulation_heuristics_ant_v1/task.md index 3f286a0..cad6b7a 100644 --- a/tasks/simulation_heuristics_ant_v1/task.md +++ b/tasks/simulation_heuristics_ant_v1/task.md @@ -37,7 +37,7 @@ metadata: anchors: verifier/anchors.json supports_private_config: true agent: - timeout_sec: 1800 + timeout_sec: 5400 user: agent network_mode: public verifier: diff --git a/tasks/simulation_heuristics_atari57_v1/task.md b/tasks/simulation_heuristics_atari57_v1/task.md index 06a536e..77f2835 100644 --- a/tasks/simulation_heuristics_atari57_v1/task.md +++ b/tasks/simulation_heuristics_atari57_v1/task.md @@ -46,7 +46,7 @@ metadata: anchors: verifier/anchors.json supports_private_config: true agent: - timeout_sec: 604800 + timeout_sec: 1814400 user: agent network_mode: no-network verifier: diff --git a/tasks/simulation_heuristics_breakout_ram_v1/task.md b/tasks/simulation_heuristics_breakout_ram_v1/task.md index fbb6c13..6ed5020 100644 --- a/tasks/simulation_heuristics_breakout_ram_v1/task.md +++ b/tasks/simulation_heuristics_breakout_ram_v1/task.md @@ -36,7 +36,7 @@ metadata: anchors: verifier/anchors.json supports_private_config: true agent: - timeout_sec: 1800 + timeout_sec: 5400 user: agent network_mode: public verifier: diff --git a/tasks/simulation_heuristics_breakout_rgb_v1/task.md b/tasks/simulation_heuristics_breakout_rgb_v1/task.md index 3ad7fad..d4db822 100644 --- a/tasks/simulation_heuristics_breakout_rgb_v1/task.md +++ b/tasks/simulation_heuristics_breakout_rgb_v1/task.md @@ -37,7 +37,7 @@ metadata: anchors: verifier/anchors.json supports_private_config: true agent: - timeout_sec: 1800 + timeout_sec: 5400 user: agent network_mode: public verifier: diff --git a/tasks/simulation_heuristics_halfcheetah_v1/task.md b/tasks/simulation_heuristics_halfcheetah_v1/task.md index ad935f1..b0c30c3 100644 --- a/tasks/simulation_heuristics_halfcheetah_v1/task.md +++ b/tasks/simulation_heuristics_halfcheetah_v1/task.md @@ -36,7 +36,7 @@ metadata: anchors: verifier/anchors.json supports_private_config: true agent: - timeout_sec: 3600 + timeout_sec: 10800 user: agent network_mode: public verifier: diff --git a/tasks/simulation_heuristics_montezuma_v1/task.md b/tasks/simulation_heuristics_montezuma_v1/task.md index 7fac50a..b32c0d6 100644 --- a/tasks/simulation_heuristics_montezuma_v1/task.md +++ b/tasks/simulation_heuristics_montezuma_v1/task.md @@ -37,7 +37,7 @@ metadata: anchors: verifier/anchors.json supports_private_config: true agent: - timeout_sec: 1800 + timeout_sec: 5400 user: agent network_mode: public verifier: diff --git a/tasks/simulation_heuristics_pong_ram_v1/task.md b/tasks/simulation_heuristics_pong_ram_v1/task.md index ff5aad7..7f1be32 100644 --- a/tasks/simulation_heuristics_pong_ram_v1/task.md +++ b/tasks/simulation_heuristics_pong_ram_v1/task.md @@ -36,7 +36,7 @@ metadata: anchors: verifier/anchors.json supports_private_config: true agent: - timeout_sec: 1800 + timeout_sec: 5400 user: agent network_mode: public verifier: diff --git a/tasks/simulation_heuristics_vizdoom_d1_v1/task.md b/tasks/simulation_heuristics_vizdoom_d1_v1/task.md index 3f53b4c..f84f211 100644 --- a/tasks/simulation_heuristics_vizdoom_d1_v1/task.md +++ b/tasks/simulation_heuristics_vizdoom_d1_v1/task.md @@ -37,7 +37,7 @@ metadata: anchors: verifier/anchors.json supports_private_config: true agent: - timeout_sec: 1800 + timeout_sec: 5400 user: agent network_mode: public verifier: diff --git a/tasks/simulation_heuristics_vizdoom_d3_v1/task.md b/tasks/simulation_heuristics_vizdoom_d3_v1/task.md index d2f35fa..3e862fd 100644 --- a/tasks/simulation_heuristics_vizdoom_d3_v1/task.md +++ b/tasks/simulation_heuristics_vizdoom_d3_v1/task.md @@ -37,7 +37,7 @@ metadata: anchors: verifier/anchors.json supports_private_config: true agent: - timeout_sec: 1800 + timeout_sec: 5400 user: agent network_mode: public verifier: diff --git a/tests/test_article_suite_tooling.py b/tests/test_article_suite_tooling.py index b26b828..0d40804 100644 --- a/tests/test_article_suite_tooling.py +++ b/tests/test_article_suite_tooling.py @@ -2,6 +2,8 @@ import json from pathlib import Path +import statistics +import sys import pytest @@ -17,6 +19,60 @@ def test_article_suite_declares_exactly_nine_unique_tasks() -> None: assert "simulation_heuristics_ant_v1" in runner.TASKS +def test_article_suite_protocol_uses_five_trials_and_triple_timeouts() -> None: + protocol = runner._protocol() + + assert protocol["version"] == "2.1" + assert protocol["trials"] == 5 + assert protocol["agent_timeout_multiplier"] == 3 + runner._validate_protocol(runner.TASKS, protocol["trials"]) + with pytest.raises(ValueError, match="must match protocol.toml"): + runner._validate_protocol(runner.TASKS, 4) + + +def test_article_suite_defaults_to_daytona( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr( + sys, + "argv", + ["run_article_suite.py", "--model", "gpt-5.6-sol"], + ) + + assert runner.parse_args().sandbox == "daytona" + + +def test_execution_protocol_ignores_posthoc_aggregation_changes() -> None: + current = runner._protocol() + prior_scoring = { + **current, + "version": "2.0", + "aggregation": { + "final_score": "arithmetic_mean_across_trial_scores" + }, + } + incompatible_timeout = { + **current, + "agent_timeout_multiplier": 2, + } + + assert runner._execution_protocol(prior_scoring) == ( + runner._execution_protocol(current) + ) + assert runner._execution_protocol(incompatible_timeout) != ( + runner._execution_protocol(current) + ) + + +def test_trial_storage_scope_supports_full_and_partial_resumes() -> None: + assert runner._task_scope(runner.TASKS) == "full-suite" + assert ( + runner._task_scope(("simulation_heuristics_vizdoom_d1_v1",)) + == "simulation_heuristics_vizdoom_d1_v1" + ) + assert runner._task_scope(runner.TASKS[:2]).startswith("tasks-") + + def test_command_uses_opencode_and_fail_closed_artifacts(tmp_path: Path) -> None: model = { "id": "gpt-5.6-sol", @@ -115,14 +171,14 @@ def test_opencode_config_pins_claude_oauth_plugin() -> None: assert definition["variants"]["max"]["thinking"]["budgetTokens"] == 31_999 -def test_claude_command_can_disable_idle_watchdog(tmp_path: Path) -> None: +def test_claude_command_uses_bounded_long_idle_watchdog(tmp_path: Path) -> None: model = { "id": "claude-opus-4.8", "display_name": "Claude Opus 4.8", "model": "anthropic/claude-opus-4-8", "provider": "claude_oauth", "provider_reasoning_effort": "max", - "agent_idle_timeout_sec": 0, + "agent_idle_timeout_sec": 3600, "daytona_pty_readline_timeout_sec": 3600, } @@ -135,13 +191,14 @@ def test_claude_command_can_disable_idle_watchdog(tmp_path: Path) -> None: tasks=(runner.TASKS[0],), ) - assert command[command.index("--agent-idle-timeout") + 1] == "0" + assert command[command.index("--agent-idle-timeout") + 1] == "3600" def test_claude_model_declares_long_daytona_pty_timeout() -> None: models = runner._models() claude = next(model for model in models if model["id"] == "claude-opus-4.8") + assert claude["agent_idle_timeout_sec"] == 3600 assert claude["daytona_pty_readline_timeout_sec"] == 3600 @@ -183,6 +240,14 @@ def test_provider_environment_is_least_privilege() -> None: assert "AZURE_API_ENDPOINT" not in claude +def test_optional_local_docker_runs_pin_calibrated_amd64_platform() -> None: + env: dict[str, str] = {} + + runner._configure_docker_platform(env, platform="linux/amd64") + + assert env["DOCKER_DEFAULT_PLATFORM"] == "linux/amd64" + + def test_aggregate_loader_requires_every_task(tmp_path: Path) -> None: model_root = tmp_path / "gpt-5.6-sol" job = model_root / "jobs" / "run" @@ -230,6 +295,132 @@ def test_aggregate_loader_rejects_task_errors(tmp_path: Path) -> None: assert len(partial) == 8 +def test_aggregate_loader_selects_successful_retry_after_failed_attempt( + tmp_path: Path, +) -> None: + model_root = tmp_path / "gpt-5.4-mini" + job = model_root / "jobs" / "run" + failed_rollout = job / "failed" + successful_rollout = job / "successful" + verifier = successful_rollout / "verifier" + verifier.mkdir(parents=True) + task = leaderboard.TASKS[0] + (verifier / "genesis-score.json").write_text( + json.dumps({"normalized_score": 12.5}) + ) + rows = [ + { + "info": { + "task_name": task, + "rollout_dir": str(failed_rollout), + }, + "reward": None, + "error": {"error": "verifier_timeout"}, + }, + { + "info": { + "task_name": task, + "rollout_dir": str(successful_rollout), + }, + "reward": 0.125, + "error": None, + }, + ] + (job / "results.jsonl").write_text( + "\n".join(json.dumps(row) for row in rows) + "\n" + ) + + loaded = leaderboard._load_results( + model_root, + expected_tasks=(task,), + ) + + assert leaderboard._normalized_task_score(loaded[task]) == 12.5 + + +def test_aggregate_loader_recovers_individual_results_after_interruption( + tmp_path: Path, +) -> None: + model_root = tmp_path / "claude-opus-4.8" + rollout = model_root / "jobs" / "run" / "task-attempt" + verifier = rollout / "verifier" + verifier.mkdir(parents=True) + task = leaderboard.TASKS[0] + (verifier / "genesis-score.json").write_text( + json.dumps({"normalized_score": 33.0}) + ) + (rollout / "result.json").write_text( + json.dumps( + { + "task_name": task, + "rewards": {"reward": 0.33}, + "error": None, + "verifier_error": None, + "n_tool_calls": 12, + "token_usage": None, + } + ) + ) + + loaded = leaderboard._load_results( + model_root, + expected_tasks=(task,), + ) + + assert leaderboard._normalized_task_score(loaded[task]) == 33.0 + assert loaded[task]["metrics"]["n_tool_calls"] == 12 + + +def test_trial_batch_recovers_digest_from_individual_results( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + task = "task-a" + monkeypatch.setattr(leaderboard, "TASKS", (task,)) + batch = tmp_path / "batch" + model_root = batch / "model-a" / "trial-01" + rollout = model_root / "jobs" / "run" / "attempt" + verifier = rollout / "verifier" + verifier.mkdir(parents=True) + (model_root / "run_metadata.json").write_text( + json.dumps( + { + "model": {"id": "model-a"}, + "trial": 1, + "tasks": [task], + "status": "interrupted", + "return_code": 130, + "dry_run": False, + "finished_at": 1.0, + } + ) + ) + (verifier / "genesis-score.json").write_text( + json.dumps({"normalized_score": 44.0}) + ) + (rollout / "result.json").write_text( + json.dumps( + { + "task_name": task, + "task_digest": "sha256:task-a", + "rewards": {"reward": 0.44}, + "error": None, + "verifier_error": None, + } + ) + ) + + results = leaderboard._trial_batch_results( + batch, + expected_models={"model-a"}, + expected_trials=1, + ) + + entry = results["model-a"][task][1] + assert entry[3] == "sha256:task-a" + assert leaderboard._normalized_task_score(entry[1]) == 44.0 + + def test_aggregate_loader_accepts_direct_provider_training_export_warning( tmp_path: Path, ) -> None: @@ -420,11 +611,375 @@ def test_latest_task_results_merges_partial_runs(tmp_path: Path) -> None: assert selected[model_id][leaderboard.TASKS[1]][3] == "sha256:digest-2" +def test_latest_complete_trial_batch_requires_every_trial( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + tasks = ("task-a", "task-b") + monkeypatch.setattr(leaderboard, "TASKS", tasks) + protocol = {"version": "test", "trials": 2} + batch = tmp_path / "batch-1" + batch.mkdir() + (batch / "batch_manifest.json").write_text( + json.dumps({"protocol": protocol}) + ) + + for trial in (1, 2): + trial_root = batch / "model-a" / f"trial-{trial:02d}" + job = trial_root / "jobs" / "run" + job.mkdir(parents=True) + metadata = { + "model": {"id": "model-a"}, + "trial": trial, + "tasks": list(tasks), + "status": "completed", + "return_code": 0, + "dry_run": False, + "finished_at": float(trial), + } + (trial_root / "run_metadata.json").write_text( + json.dumps(metadata) + ) + (trial_root / "task_manifest.json").write_text( + json.dumps( + { + "tasks": [ + {"task_id": task, "digest": f"sha256:{task}"} + for task in tasks + ] + } + ) + ) + rows = [] + for index, task in enumerate(tasks): + rollout = trial_root / f"rollout-{index}" + verifier = rollout / "verifier" + verifier.mkdir(parents=True) + (verifier / "genesis-score.json").write_text( + json.dumps({"normalized_score": trial * 10 + index}) + ) + rows.append( + { + "info": { + "task_name": task, + "rollout_dir": str(rollout), + }, + "reward": 0.5, + "error": None, + } + ) + (job / "results.jsonl").write_text( + "\n".join(json.dumps(row) for row in rows) + "\n" + ) + + selected_batches, results = leaderboard._latest_complete_model_batches( + tmp_path, + expected_models={"model-a"}, + protocol=protocol, + ) + + assert selected_batches == {"model-a": batch} + assert set(results["model-a"]["task-a"]) == {1, 2} + assert ( + leaderboard._normalized_task_score( + results["model-a"]["task-b"][2][1] + ) + == 21 + ) + + (batch / "model-a" / "trial-02" / "run_metadata.json").unlink() + with pytest.raises(RuntimeError, match="2 missing"): + leaderboard._latest_complete_model_batches( + tmp_path, + expected_models={"model-a"}, + protocol=protocol, + ) + + +def test_latest_complete_model_batches_support_parallel_model_shards( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + tasks = ("task-a",) + monkeypatch.setattr(leaderboard, "TASKS", tasks) + protocol = {"version": "test", "trials": 2} + + for model_index, model_id in enumerate(("model-a", "model-b"), start=1): + batch = tmp_path / f"batch-{model_id}" + batch.mkdir() + (batch / "batch_manifest.json").write_text( + json.dumps({"protocol": protocol}) + ) + for trial in (1, 2): + trial_root = batch / model_id / f"trial-{trial:02d}" + job = trial_root / "jobs" / "run" + rollout = trial_root / f"rollout-{trial}" + verifier = rollout / "verifier" + job.mkdir(parents=True) + verifier.mkdir(parents=True) + (trial_root / "run_metadata.json").write_text( + json.dumps( + { + "model": {"id": model_id}, + "trial": trial, + "tasks": list(tasks), + "status": "completed", + "return_code": 0, + "dry_run": False, + "finished_at": float(model_index * 10 + trial), + } + ) + ) + (trial_root / "task_manifest.json").write_text( + json.dumps( + { + "tasks": [ + { + "task_id": "task-a", + "digest": f"sha256:{model_id}", + } + ] + } + ) + ) + (verifier / "genesis-score.json").write_text( + json.dumps({"normalized_score": model_index * 10 + trial}) + ) + (job / "results.jsonl").write_text( + json.dumps( + { + "info": { + "task_name": "task-a", + "rollout_dir": str(rollout), + }, + "reward": 0.5, + "error": None, + } + ) + + "\n" + ) + + selected_batches, results = ( + leaderboard._latest_complete_model_batches( + tmp_path, + expected_models={"model-a", "model-b"}, + protocol=protocol, + ) + ) + + assert selected_batches == { + "model-a": tmp_path / "batch-model-a", + "model-b": tmp_path / "batch-model-b", + } + assert set(results) == {"model-a", "model-b"} + assert set(results["model-a"]["task-a"]) == {1, 2} + assert set(results["model-b"]["task-a"]) == {1, 2} + + +def test_failed_full_trial_combines_with_successful_task_repair( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + tasks = ("task-a", "task-b") + monkeypatch.setattr(leaderboard, "TASKS", tasks) + protocol = {"version": "test", "trials": 1} + batch = tmp_path / "batch" + batch.mkdir() + (batch / "batch_manifest.json").write_text( + json.dumps({"protocol": protocol}) + ) + + full_root = batch / "model-a" / "trial-01" + full_job = full_root / "jobs" / "run" + full_job.mkdir(parents=True) + (full_root / "run_metadata.json").write_text( + json.dumps( + { + "model": {"id": "model-a"}, + "trial": 1, + "tasks": list(tasks), + "status": "failed", + "return_code": 1, + "dry_run": False, + "finished_at": 1.0, + } + ) + ) + (full_root / "task_manifest.json").write_text( + json.dumps( + { + "tasks": [ + {"task_id": task, "digest": f"sha256:{task}"} + for task in tasks + ] + } + ) + ) + rollout_a = full_root / "rollout-a" + verifier_a = rollout_a / "verifier" + verifier_a.mkdir(parents=True) + (verifier_a / "genesis-score.json").write_text( + json.dumps({"normalized_score": 10.0}) + ) + (full_job / "results.jsonl").write_text( + "\n".join( + [ + json.dumps( + { + "info": { + "task_name": "task-a", + "rollout_dir": str(rollout_a), + }, + "reward": 0.1, + "error": None, + } + ), + json.dumps( + { + "info": {"task_name": "task-b"}, + "reward": None, + "error": {"error": "verifier_failure"}, + } + ), + ] + ) + + "\n" + ) + + repair_root = full_root / "task-b" + repair_job = repair_root / "jobs" / "run" + repair_job.mkdir(parents=True) + (repair_root / "run_metadata.json").write_text( + json.dumps( + { + "model": {"id": "model-a"}, + "trial": 1, + "tasks": ["task-b"], + "status": "completed", + "return_code": 0, + "dry_run": False, + "finished_at": 2.0, + } + ) + ) + (repair_root / "task_manifest.json").write_text( + json.dumps( + { + "tasks": [ + {"task_id": "task-b", "digest": "sha256:task-b"} + ] + } + ) + ) + rollout_b = repair_root / "rollout-b" + verifier_b = rollout_b / "verifier" + verifier_b.mkdir(parents=True) + (verifier_b / "genesis-score.json").write_text( + json.dumps({"normalized_score": 20.0}) + ) + (repair_job / "results.jsonl").write_text( + json.dumps( + { + "info": { + "task_name": "task-b", + "rollout_dir": str(rollout_b), + }, + "reward": 0.2, + "error": None, + } + ) + + "\n" + ) + + selected_batches, results = ( + leaderboard._latest_complete_model_batches( + tmp_path, + expected_models={"model-a"}, + protocol=protocol, + ) + ) + + assert selected_batches == {"model-a": batch} + assert leaderboard._normalized_task_score( + results["model-a"]["task-a"][1][1] + ) == 10.0 + assert leaderboard._normalized_task_score( + results["model-a"]["task-b"][1][1] + ) == 20.0 + + def test_iqm_matches_rliable_25_percent_trimmed_mean() -> None: assert leaderboard._interquartile_mean(list(range(9))) == 4.0 assert leaderboard._interquartile_mean( [-100.0, -50.0, 1.0, 2.0, 3.0, 4.0, 5.0, 100.0, 500.0] ) == 3.0 + assert leaderboard._interquartile_mean(list(range(45))) == 22.0 + + +def test_fail_closed_timeout_uses_starter_equivalent_raw_score() -> None: + raw, starter, reference, source, observed = ( + leaderboard._resolve_raw_score( + { + "score": None, + "normalized_score": 0.0, + "verifier_timeout": True, + }, + normalized_score=0.0, + fallback_anchor=(12.0, 34.0), + ) + ) + + assert (raw, starter, reference) == (12.0, 12.0, 34.0) + assert source == "starter_anchor_equivalent_for_fail_closed_timeout" + assert observed is None + + +def test_five_trial_final_score_pools_runs_and_tasks( + monkeypatch: pytest.MonkeyPatch, +) -> None: + tasks = tuple(f"task-{index}" for index in range(9)) + monkeypatch.setattr(leaderboard, "TASKS", tasks) + trial_scores = { + trial: { + task: (1000.0 if trial == 4 else 0.0) + for task in tasks + } + for trial in range(5) + } + task_means = { + task: statistics.fmean( + trial_scores[trial][task] for trial in trial_scores + ) + for task in tasks + } + + aggregate = leaderboard._aggregate_task_scores( + task_means, + trial_task_scores=trial_scores, + ) + pooled = [ + trial_scores[trial][task] + for trial in sorted(trial_scores) + for task in tasks + ] + trial_iqms = [ + leaderboard._interquartile_mean( + [trial_scores[trial][task] for task in tasks] + ) + for trial in sorted(trial_scores) + ] + + assert aggregate["final_normalized_score"] == ( + leaderboard._interquartile_mean(pooled) + ) + assert aggregate["mean_trial_iqm_normalized_score"] == ( + statistics.fmean(trial_iqms) + ) + assert aggregate["final_normalized_score"] == 0.0 + assert aggregate["mean_trial_iqm_normalized_score"] == 200.0 + assert aggregate["final_normalized_score_stddev"] == statistics.stdev( + trial_iqms + ) def test_offline_report_builds_nine_task_boards_then_final() -> None: @@ -442,20 +997,31 @@ def test_offline_report_builds_nine_task_boards_then_final() -> None: task_scores[leaderboard.TASKS[1]] = 5.0 aggregates = leaderboard._aggregate_task_scores(task_scores) rows.append( - { - "model_id": model_id, - "model": model, - "model_route": f"test/{model_id}", - "provider": "test", - "provider_label": "Test provider", - "harness": "opencode", + { + "model_id": model_id, + "model": model, + "model_route": f"test/{model_id}", + "provider": "test", + "provider_label": "Test provider", + "harness": "opencode", + "sandbox": "daytona", "provider_reasoning_effort": "max", + "trial_count": 5, **aggregates, "average_normalized_score": aggregates[ "arithmetic_mean_normalized_score" ], "task_scores": task_scores, + "task_score_stddevs": { + task: 1.0 for task in leaderboard.TASKS + }, "raw_task_scores": task_scores, + "raw_task_score_stddevs": { + task: 2.0 for task in leaderboard.TASKS + }, + "raw_task_score_imputation_counts": { + task: 0 for task in leaderboard.TASKS + }, "task_anchors": { task: { "starter_score": 0.0, diff --git a/tests/test_leaderboard_artifacts.py b/tests/test_leaderboard_artifacts.py index b964960..fca47c9 100644 --- a/tests/test_leaderboard_artifacts.py +++ b/tests/test_leaderboard_artifacts.py @@ -4,6 +4,7 @@ import math from pathlib import Path import re +import statistics import struct @@ -62,8 +63,26 @@ def test_article_suite_leaderboard_is_complete_and_self_contained() -> None: assert payload["leaderboards"][-1]["id"] == "final" assert payload["aggregation"]["primary_metric"] == "interquartile_mean" assert payload["aggregation"]["trim_fraction_per_tail"] == 0.25 - assert payload["aggregation"]["trimmed_score_count_per_tail"] == 2 - assert payload["aggregation"]["retained_score_count"] == 5 + if "protocol" in payload: + assert payload["protocol"]["version"] == "2.1" + assert payload["protocol"]["trials"] == 5 + assert payload["protocol"]["agent_timeout_multiplier"] == 3 + assert payload["execution"] == { + "sandbox": "daytona", + "sandboxes": ["daytona"], + } + assert payload["aggregation"]["pooled_score_count"] == 45 + assert payload["aggregation"]["trimmed_score_count_per_tail"] == 11 + assert payload["aggregation"]["retained_score_count"] == 23 + assert set(payload["batch_ids"]) == { + "gpt-5.6-sol", + "gpt-5.5", + "claude-opus-4.8", + "gpt-5.4-mini", + } + else: + assert payload["aggregation"]["trimmed_score_count_per_tail"] == 2 + assert payload["aggregation"]["retained_score_count"] == 5 assert ( payload["inference_settings"]["cross_provider_comparability"] == "labels_are_not_a_shared_numeric_compute_scale" @@ -89,12 +108,45 @@ def test_article_suite_leaderboard_is_complete_and_self_contained() -> None: for row in payload["rows"]: assert row["harness"] == "opencode" + if row.get("sandbox") is not None: + assert row["sandbox"] == "daytona" assert set(row["task_scores"]) == set(payload["tasks"]) assert set(row["raw_task_scores"]) == set(payload["tasks"]) assert set(row["task_anchors"]) == set(payload["tasks"]) assert set(row["submission_details"]) == set(payload["tasks"]) - sorted_scores = sorted(row["task_scores"].values()) - expected_iqm = sum(sorted_scores[2:7]) / 5 + trial_final_scores = row.get("trial_final_normalized_scores") + if trial_final_scores is None: + sorted_scores = sorted(row["task_scores"].values()) + expected_iqm = sum(sorted_scores[2:7]) / 5 + else: + assert row["trial_count"] == 5 + assert len(trial_final_scores) == 5 + trial_values = list(trial_final_scores.values()) + trial_task_scores = row["trial_task_normalized_scores"] + assert len(trial_task_scores) == 5 + pooled_scores = [ + trial_scores[task] + for _, trial_scores in sorted( + trial_task_scores.items(), + key=lambda item: int(item[0]), + ) + for task in payload["tasks"] + ] + ordered_scores = sorted(pooled_scores) + trim_count = int(len(ordered_scores) * 0.25) + expected_iqm = statistics.fmean( + ordered_scores[ + trim_count : len(ordered_scores) - trim_count + ] + ) + assert math.isclose( + row["final_normalized_score_stddev"], + statistics.stdev(trial_values), + ) + assert math.isclose( + row["mean_trial_iqm_normalized_score"], + statistics.fmean(trial_values), + ) assert math.isclose( row["final_normalized_score"], expected_iqm, @@ -120,6 +172,28 @@ def test_article_suite_leaderboard_is_complete_and_self_contained() -> None: assert metadata["harness"] == "opencode" assert metadata["normalized_score"] == row["task_scores"][task] assert score["normalized_score"] == row["task_scores"][task] + if row.get("trial_count") is not None: + assert metadata["trial_count"] == 5 + assert score["trial_count"] == 5 + assert len(score["trials"]) == 5 + assert metadata["normalized_score_stddev"] == row[ + "task_score_stddevs" + ][task] + assert score["normalized_score_stddev"] == row[ + "task_score_stddevs" + ][task] + assert metadata["raw_score_stddev"] == row[ + "raw_task_score_stddevs" + ][task] + assert score["score_stddev"] == row[ + "raw_task_score_stddevs" + ][task] + assert metadata["raw_score_imputation_count"] == row[ + "raw_task_score_imputation_counts" + ][task] + assert score["raw_score_imputation_count"] == row[ + "raw_task_score_imputation_counts" + ][task] def assert_no_absolute_artifact_paths(value: object) -> None: if isinstance(value, dict): @@ -152,6 +226,14 @@ def assert_no_absolute_artifact_paths(value: object) -> None: assert row["reference_score"] == model_row["task_anchors"][ board["id"] ]["reference_score"] + if model_row.get("trial_count") is not None: + assert row["trial_count"] == 5 + assert row["normalized_score_stddev"] == model_row[ + "task_score_stddevs" + ][board["id"]] + assert row["raw_score_stddev"] == model_row[ + "raw_task_score_stddevs" + ][board["id"]] final_board = payload["leaderboards"][-1] assert [ @@ -165,12 +247,12 @@ def assert_no_absolute_artifact_paths(value: object) -> None: ) for row in final_board["rows"] ) - assert [row["model"] for row in payload["rows"]] == [ + assert {row["model"] for row in payload["rows"]} == { "GPT-5.5", "GPT-5.6 Sol", "Claude Opus 4.8", "GPT-5.4 Mini", - ] + } markdown = ARTICLE_SUITE_MARKDOWN.read_text() assert "article_suite_task_leaderboards.png" in markdown @@ -188,10 +270,17 @@ def assert_no_absolute_artifact_paths(value: object) -> None: assert "assets/article_suite_final_leaderboard.png" not in website assert '

Leaderboards

' in website assert "Task-level view" not in website + assert "Four coding models, nine article-derived tasks" not in website + assert "The task panels show each environment's native raw score" not in website + assert "Pooled IQM across all 45 trial-task scores" in website + assert "grid-template-columns:5.2rem minmax(0,1fr) 5.1rem" in website + assert 'class="chart-error-range"' in website + assert 'class="final-error-range"' in website assert 'id="taskLeaderboards"' in website assert 'id="finalLeaderboard"' in website assert 'id="inferenceSettings"' in website assert "not a shared numeric compute scale" in website + assert "' · '+esc(row.harness)+' · '+esc(row.sandbox)" in website assert 'id="loopScore"' in website assert 'id="loopNote"' in website assert "loopScore.textContent=t.score" in website diff --git a/website/README.md b/website/README.md index dbf3ec8..d2d357c 100644 --- a/website/README.md +++ b/website/README.md @@ -34,9 +34,14 @@ Offline README/report images are generated separately by: uv run python scripts/plot_article_suite_leaderboards.py ``` -The HTML task panels use native raw environment scores. The final HTML plot -uses a fixed positive display index (`IQM + 100`) while retaining raw IQM as -the official ranking metric. +The HTML task panels show five-trial mean ± sample standard deviation for +native raw environment scores. The final HTML plot shows RLiable-style pooled +IQM over all 45 trial-task scores, plus the sample standard deviation of the +five per-trial IQMs, and uses a fixed positive display index (`IQM + 100`) while +retaining raw IQM as the official ranking metric. + +Inference cards also display the exact provider route, OpenCode harness, +reasoning setting, and Daytona sandbox used by each published model. ## Deploy — genesisbench.benchflow.ai (Vercel, same as clawsbench) diff --git a/website/assets/article_suite.json b/website/assets/article_suite.json index b490741..37b8aa1 100644 --- a/website/assets/article_suite.json +++ b/website/assets/article_suite.json @@ -7,20 +7,39 @@ "ranking_field": "final_normalized_score", "type": "additive_offset" }, + "final_estimator": "iqm_across_all_45_trial_task_normalized_scores", + "final_variability": "sample_standard_deviation_of_five_trial_iqms", + "pooled_score_count": 45, "primary_field": "final_normalized_score", "primary_metric": "interquartile_mean", - "retained_score_count": 5, + "retained_score_count": 23, "score_bounds": "unbounded", "secondary_fields": [ "arithmetic_mean_normalized_score", - "median_normalized_score" + "median_normalized_score", + "mean_trial_iqm_normalized_score" ], + "task_estimator": "mean_across_five_trials", + "task_variability": "sample_standard_deviation", + "trial_estimator": "iqm_across_nine_tasks", "trim_fraction_per_tail": 0.25, - "trimmed_score_count_per_tail": 2, - "uncertainty": "not_estimated_single_run_per_model_task" + "trimmed_score_count_per_tail": 11 + }, + "batch_id": "per-model-batches", + "batch_ids": { + "claude-opus-4.8": "article-suite-v2-daytona-20260714-claude", + "gpt-5.4-mini": "article-suite-v2-daytona-20260714-gpt54mini", + "gpt-5.5": "article-suite-v2-daytona-20260714-gpt55", + "gpt-5.6-sol": "article-suite-v2-daytona-20260714-gpt56" }, "benchmark": "learning_beyond_gradients_article_suite", - "generated_at": "2026-07-13T15:34:22.260689+00:00", + "execution": { + "sandbox": "daytona", + "sandboxes": [ + "daytona" + ] + }, + "generated_at": "2026-07-15T12:56:18.690710+00:00", "inference_settings": { "cross_provider_comparability": "labels_are_not_a_shared_numeric_compute_scale", "field": "provider_reasoning_effort", @@ -28,12 +47,13 @@ "models": [ { "harness": "opencode", - "model": "GPT-5.5", - "model_id": "gpt-5.5", - "model_route": "azure/gpt-5.5", - "provider": "azure", - "provider_label": "Azure direct", - "provider_reasoning_effort": "xhigh" + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max", + "sandbox": "daytona" }, { "harness": "opencode", @@ -42,16 +62,18 @@ "model_route": "azure/gpt-5.6-sol", "provider": "azure", "provider_label": "Azure direct", - "provider_reasoning_effort": "max" + "provider_reasoning_effort": "max", + "sandbox": "daytona" }, { "harness": "opencode", - "model": "Claude Opus 4.8", - "model_id": "claude-opus-4.8", - "model_route": "anthropic/claude-opus-4-8", - "provider": "claude_oauth", - "provider_label": "Claude OAuth via pinned OpenCode plugin", - "provider_reasoning_effort": "max" + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "sandbox": "daytona" }, { "harness": "opencode", @@ -60,7 +82,8 @@ "model_route": "azure/gpt-5.4-mini", "provider": "azure", "provider_label": "Azure direct", - "provider_reasoning_effort": "xhigh" + "provider_reasoning_effort": "xhigh", + "sandbox": "daytona" } ] }, @@ -77,67 +100,111 @@ "rows": [ { "harness": "opencode", - "model": "GPT-5.6 Sol", - "model_id": "gpt-5.6-sol", - "model_route": "azure/gpt-5.6-sol", - "normalized_score": 39.203600385, - "provider": "azure", - "provider_label": "Azure direct", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "normalized_score": 1.820348169, + "normalized_score_stddev": 4.070422248601275, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 1, - "raw_score": 3815.360622145591, + "raw_score": 2507.501036532837, + "raw_score_imputation_count": 4, + "raw_score_stddev": 142.40443086118393, "reference_score": 5942.333244755838, - "source_run": "leaderboard/runs/article_suite/20260713T093815Z/gpt-5.6-sol", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], "starter_score": 2443.815838992282, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", - "model": "Claude Opus 4.8", - "model_id": "claude-opus-4.8", - "model_route": "anthropic/claude-opus-4-8", - "normalized_score": 14.050353128, - "provider": "claude_oauth", - "provider_label": "Claude OAuth via pinned OpenCode plugin", + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", + "normalized_score": -4.9009087386, + "normalized_score_stddev": 11.991465810903719, + "provider": "azure", + "provider_label": "Azure direct", "provider_reasoning_effort": "max", "rank": 2, - "raw_score": 2935.3698887625987, + "raw_score": 2272.356693724495, + "raw_score_imputation_count": 0, + "raw_score_stddev": 419.52351859557746, "reference_score": 5942.333244755838, - "source_run": "leaderboard/runs/article_suite/20260713T142919Z/claude-opus-4.8", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "starter_score": 2443.815838992282, - "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.5", "model_id": "gpt-5.5", "model_route": "azure/gpt-5.5", - "normalized_score": -16.891796576, + "normalized_score": -12.5116242462, + "normalized_score_stddev": 8.756962566015595, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 3, - "raw_score": 1852.8533956222025, + "raw_score": 2006.0944869926293, + "raw_score_imputation_count": 0, + "raw_score_stddev": 306.36385957656836, "reference_score": 5942.333244755838, - "source_run": "leaderboard/runs/article_suite/20260713T095807Z/gpt-5.5", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05/tasks-81ec8d98a4" + ], "starter_score": 2443.815838992282, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", "model_route": "azure/gpt-5.4-mini", - "normalized_score": -31.35954984, + "normalized_score": -47.232499900200004, + "normalized_score_stddev": 10.40778807635678, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 4, - "raw_score": 1346.6965294546999, + "raw_score": 791.3786088082715, + "raw_score_imputation_count": 0, + "raw_score_stddev": 364.1182774109385, "reference_score": 5942.333244755838, - "source_run": "leaderboard/runs/article_suite/20260713T104315Z/gpt-5.4-mini", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "starter_score": 2443.815838992282, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/score.json", + "trial_count": 5 } ] }, @@ -155,64 +222,108 @@ "model": "Claude Opus 4.8", "model_id": "claude-opus-4.8", "model_route": "anthropic/claude-opus-4-8", - "normalized_score": 100.0, + "normalized_score": 99.6666666666, + "normalized_score_stddev": 0.7453559926490017, "provider": "claude_oauth", "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 1, - "raw_score": 21.0, + "raw_score": 20.919999999999998, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.17888543819998254, "reference_score": 21.0, - "source_run": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], "starter_score": -2.9999999999999996, - "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.6 Sol", "model_id": "gpt-5.6-sol", "model_route": "azure/gpt-5.6-sol", - "normalized_score": 52.916666667, + "normalized_score": 91.6666666666, + "normalized_score_stddev": 9.771699499561077, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "max", "rank": 2, - "raw_score": 9.7, + "raw_score": 19.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 2.345207879911715, "reference_score": 21.0, - "source_run": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "starter_score": -2.9999999999999996, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_pong_ram_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.5", "model_id": "gpt-5.5", "model_route": "azure/gpt-5.5", - "normalized_score": 45.833333333, + "normalized_score": 49.1666666668, + "normalized_score_stddev": 40.44028519521613, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 3, - "raw_score": 8.0, + "raw_score": 8.8, + "raw_score_imputation_count": 0, + "raw_score_stddev": 9.705668446840743, "reference_score": 21.0, - "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-885696cfa4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], "starter_score": -2.9999999999999996, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", "model_route": "azure/gpt-5.4-mini", - "normalized_score": -75.0, + "normalized_score": -40.8333333332, + "normalized_score_stddev": 67.46912874308083, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 4, - "raw_score": -21.0, + "raw_score": -12.8, + "raw_score_imputation_count": 0, + "raw_score_stddev": 16.192590898309017, "reference_score": 21.0, - "source_run": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "starter_score": -2.9999999999999996, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_pong_ram_v1/score.json", + "trial_count": 5 } ] }, @@ -225,69 +336,113 @@ "unit": "points" }, "rows": [ + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "normalized_score": 94.004192872, + "normalized_score_stddev": 7.012257406411853, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max", + "rank": 1, + "raw_score": 835.4, + "raw_score_imputation_count": 0, + "raw_score_stddev": 33.448467827390836, + "reference_score": 864.0, + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "starter_score": 387.0, + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/score.json", + "trial_count": 5 + }, { "harness": "opencode", "model": "GPT-5.6 Sol", "model_id": "gpt-5.6-sol", "model_route": "azure/gpt-5.6-sol", - "normalized_score": 100.0, + "normalized_score": 43.899371069400004, + "normalized_score_stddev": 47.34804435501101, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "max", - "rank": 1, - "raw_score": 864.0, + "rank": 2, + "raw_score": 596.4, + "raw_score_imputation_count": 0, + "raw_score_stddev": 225.8501715739884, "reference_score": 864.0, - "source_run": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "starter_score": 387.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_ram_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.5", "model_id": "gpt-5.5", "model_route": "azure/gpt-5.5", - "normalized_score": 95.597484277, + "normalized_score": 20.9224318658, + "normalized_score_stddev": 27.49546024414613, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", - "rank": 2, - "raw_score": 843.0, + "rank": 3, + "raw_score": 486.8, + "raw_score_imputation_count": 0, + "raw_score_stddev": 131.153345363357, "reference_score": 864.0, - "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-e6fa3546aa", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], "starter_score": 387.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", "model_route": "azure/gpt-5.4-mini", - "normalized_score": 2.51572327, + "normalized_score": -1.3417190774, + "normalized_score_stddev": 9.228605826577622, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", - "rank": 3, - "raw_score": 399.0, - "reference_score": 864.0, - "source_run": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", - "starter_score": 387.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/score.json" - }, - { - "harness": "opencode", - "model": "Claude Opus 4.8", - "model_id": "claude-opus-4.8", - "model_route": "anthropic/claude-opus-4-8", - "normalized_score": 0.0, - "provider": "claude_oauth", - "provider_label": "Claude OAuth via pinned OpenCode plugin", - "provider_reasoning_effort": "max", "rank": 4, - "raw_score": 387.0, + "raw_score": 380.6, + "raw_score_imputation_count": 0, + "raw_score_stddev": 44.0204497932495, "reference_score": 864.0, - "source_run": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02/simulation_heuristics_breakout_ram_v1", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "starter_score": 387.0, - "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_ram_v1/score.json", + "trial_count": 5 } ] }, @@ -300,69 +455,113 @@ "unit": "points" }, "rows": [ + { + "harness": "opencode", + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "normalized_score": 89.2779783394, + "normalized_score_stddev": 13.287454681897577, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max", + "rank": 1, + "raw_score": 804.6, + "raw_score_imputation_count": 0, + "raw_score_stddev": 73.61249893869926, + "reference_score": 864.0, + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "starter_score": 310.0, + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/score.json", + "trial_count": 5 + }, { "harness": "opencode", "model": "GPT-5.5", "model_id": "gpt-5.5", "model_route": "azure/gpt-5.5", - "normalized_score": 70.758122744, + "normalized_score": 60.86642599300001, + "normalized_score_stddev": 22.118506348049813, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", - "rank": 1, - "raw_score": 702.0, + "rank": 2, + "raw_score": 647.2, + "raw_score_imputation_count": 0, + "raw_score_stddev": 122.53652516698847, "reference_score": 864.0, - "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-885696cfa4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], "starter_score": 310.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.6 Sol", "model_id": "gpt-5.6-sol", "model_route": "azure/gpt-5.6-sol", - "normalized_score": 70.758122744, + "normalized_score": 60.86642599300001, + "normalized_score_stddev": 22.118506348049813, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "max", - "rank": 1, - "raw_score": 702.0, + "rank": 2, + "raw_score": 647.2, + "raw_score_imputation_count": 0, + "raw_score_stddev": 122.53652516698847, "reference_score": 864.0, - "source_run": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "starter_score": 310.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_breakout_rgb_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", "model_route": "azure/gpt-5.4-mini", - "normalized_score": 15.523465704, + "normalized_score": 14.332129963999998, + "normalized_score_stddev": 51.9448986087478, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", - "rank": 3, - "raw_score": 396.0, - "reference_score": 864.0, - "source_run": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", - "starter_score": 310.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/score.json" - }, - { - "harness": "opencode", - "model": "Claude Opus 4.8", - "model_id": "claude-opus-4.8", - "model_route": "anthropic/claude-opus-4-8", - "normalized_score": 0.0, - "provider": "claude_oauth", - "provider_label": "Claude OAuth via pinned OpenCode plugin", - "provider_reasoning_effort": "max", "rank": 4, - "raw_score": 310.0, + "raw_score": 389.4, + "raw_score_imputation_count": 0, + "raw_score_stddev": 287.77473829368694, "reference_score": 864.0, - "source_run": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "starter_score": 310.0, - "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_breakout_rgb_v1/score.json", + "trial_count": 5 } ] }, @@ -380,64 +579,108 @@ "model": "Claude Opus 4.8", "model_id": "claude-opus-4.8", "model_route": "anthropic/claude-opus-4-8", - "normalized_score": 26.986707946, + "normalized_score": 40.934675074999994, + "normalized_score_stddev": 24.40596858830019, "provider": "claude_oauth", "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 1, - "raw_score": 6021.504892591717, + "raw_score": 7065.645225601496, + "raw_score_imputation_count": 0, + "raw_score_stddev": 1827.0229584820154, "reference_score": 11487.256451471967, - "source_run": "leaderboard/runs/article_suite/20260713T111714Z/claude-opus-4.8", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], "starter_score": 4001.2886158917113, - "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.6 Sol", "model_id": "gpt-5.6-sol", "model_route": "azure/gpt-5.6-sol", - "normalized_score": 15.873072595, + "normalized_score": 31.8442100886, + "normalized_score_stddev": 3.8873875677992533, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "max", "rank": 2, - "raw_score": 5189.541724894487, + "raw_score": 6385.135940620764, + "raw_score_imputation_count": 0, + "raw_score_stddev": 291.0085829715705, "reference_score": 11487.256451471967, - "source_run": "leaderboard/runs/article_suite/20260713T084340Z/gpt-5.6-sol", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "starter_score": 4001.2886158917113, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_halfcheetah_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.5", "model_id": "gpt-5.5", "model_route": "azure/gpt-5.5", - "normalized_score": 2.73755349, + "normalized_score": 9.5689381094, + "normalized_score_stddev": 9.330884765209642, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 3, - "raw_score": 4206.220989602626, + "raw_score": 4717.6162449605235, + "raw_score_imputation_count": 0, + "raw_score_stddev": 698.5070322948952, "reference_score": 11487.256451471967, - "source_run": "leaderboard/runs/article_suite/20260713T103810Z/gpt-5.5", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01/simulation_heuristics_halfcheetah_v1", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-e6fa3546aa", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05/tasks-81ec8d98a4" + ], "starter_score": 4001.2886158917113, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", "model_route": "azure/gpt-5.4-mini", - "normalized_score": -8.167582379, + "normalized_score": -39.8837988972, + "normalized_score_stddev": 19.209845675650264, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 4, - "raw_score": 3389.8660260183765, + "raw_score": 1015.6002588284422, + "raw_score_imputation_count": 0, + "raw_score_stddev": 1438.0428685581014, "reference_score": 11487.256451471967, - "source_run": "leaderboard/runs/article_suite/20260713T093314Z/gpt-5.4-mini", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "starter_score": 4001.2886158917113, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_halfcheetah_v1/score.json", + "trial_count": 5 } ] }, @@ -455,64 +698,108 @@ "model": "Claude Opus 4.8", "model_id": "claude-opus-4.8", "model_route": "anthropic/claude-opus-4-8", - "normalized_score": 95.055812742, + "normalized_score": 107.1451288772, + "normalized_score_stddev": 14.745599637347764, "provider": "claude_oauth", "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 1, - "raw_score": 0.7992199431359768, - "reference_score": 0.8178199748694897, - "source_run": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "raw_score": 0.8446999486498534, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.05547294365604739, + "reference_score": 0.8178199748694898, + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], "starter_score": 0.44161998964846133, - "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", - "model": "GPT-5.5", - "model_id": "gpt-5.5", - "model_route": "azure/gpt-5.5", - "normalized_score": 85.199359782, + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", + "normalized_score": 56.834663982799995, + "normalized_score_stddev": 33.111283305681766, "provider": "azure", "provider_label": "Azure direct", - "provider_reasoning_effort": "xhigh", + "provider_reasoning_effort": "max", "rank": 2, - "raw_score": 0.7621399685554205, - "reference_score": 0.8178199748694897, - "source_run": "leaderboard/runs/article_suite/20260713T102218Z/gpt-5.5", + "raw_score": 0.6554319871515035, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.12456464290218099, + "reference_score": 0.8178199748694898, + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "starter_score": 0.44161998964846133, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", - "model": "GPT-5.6 Sol", - "model_id": "gpt-5.6-sol", - "model_route": "azure/gpt-5.6-sol", - "normalized_score": 49.409889366, + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "normalized_score": 47.33864911959999, + "normalized_score_stddev": 95.13699952993868, "provider": "azure", "provider_label": "Azure direct", - "provider_reasoning_effort": "max", + "provider_reasoning_effort": "xhigh", "rank": 3, - "raw_score": 0.62749998614192, - "reference_score": 0.8178199748694897, - "source_run": "leaderboard/runs/article_suite/20260713T101636Z/gpt-5.6-sol", + "raw_score": 0.6197079806402326, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.3579053781713541, + "reference_score": 0.8178199748694898, + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], "starter_score": 0.44161998964846133, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d1_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", "model_route": "azure/gpt-5.4-mini", - "normalized_score": -161.116433137, + "normalized_score": -186.0627378054, + "normalized_score_stddev": 3.359451433761567, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 4, - "raw_score": -0.16450000800192355, - "reference_score": 0.8178199748694897, - "source_run": "leaderboard/runs/article_suite/20260713T102707Z/gpt-5.4-mini", + "raw_score": -0.25834800247848033, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.012638255798059739, + "reference_score": 0.8178199748694898, + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "starter_score": 0.44161998964846133, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d1_v1/score.json", + "trial_count": 5 } ] }, @@ -530,64 +817,108 @@ "model": "Claude Opus 4.8", "model_id": "claude-opus-4.8", "model_route": "anthropic/claude-opus-4-8", - "normalized_score": 122.259358289, + "normalized_score": 96.4371657756, + "normalized_score_stddev": 42.45851366513837, "provider": "claude_oauth", "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 1, - "raw_score": 397.7, + "raw_score": 320.44, + "raw_score_imputation_count": 0, + "raw_score_stddev": 127.03587288636228, "reference_score": 331.1, - "source_run": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], "starter_score": 31.9, - "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", - "model": "GPT-5.6 Sol", - "model_id": "gpt-5.6-sol", - "model_route": "azure/gpt-5.6-sol", - "normalized_score": 26.236631016, + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "normalized_score": 23.2620320856, + "normalized_score_stddev": 41.04765230004701, "provider": "azure", "provider_label": "Azure direct", - "provider_reasoning_effort": "max", + "provider_reasoning_effort": "xhigh", "rank": 2, - "raw_score": 110.39999999999998, + "raw_score": 101.49999999999999, + "raw_score_imputation_count": 0, + "raw_score_stddev": 122.81457568220475, "reference_score": 331.1, - "source_run": "leaderboard/runs/article_suite/20260713T083940Z/gpt-5.6-sol", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], "starter_score": 31.9, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", - "model": "GPT-5.5", - "model_id": "gpt-5.5", - "model_route": "azure/gpt-5.5", - "normalized_score": 5.481283422, + "model": "GPT-5.4 Mini", + "model_id": "gpt-5.4-mini", + "model_route": "azure/gpt-5.4-mini", + "normalized_score": 16.5307486632, + "normalized_score_stddev": 45.27484310055022, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 3, - "raw_score": 48.3, + "raw_score": 81.36, + "raw_score_imputation_count": 0, + "raw_score_stddev": 135.46233055724386, "reference_score": 331.1, - "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01/simulation_heuristics_vizdoom_d3_v1", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "starter_score": 31.9, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", - "model": "GPT-5.4 Mini", - "model_id": "gpt-5.4-mini", - "model_route": "azure/gpt-5.4-mini", - "normalized_score": -9.859625668, + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", + "normalized_score": 16.3034759358, + "normalized_score_stddev": 23.995771403261536, "provider": "azure", "provider_label": "Azure direct", - "provider_reasoning_effort": "xhigh", + "provider_reasoning_effort": "max", "rank": 4, - "raw_score": 2.4, + "raw_score": 80.67999999999999, + "raw_score_imputation_count": 0, + "raw_score_stddev": 71.79534803871347, "reference_score": 331.1, - "source_run": "leaderboard/runs/article_suite/20260713T104034Z/gpt-5.4-mini", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "starter_score": 31.9, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_vizdoom_d3_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_vizdoom_d3_v1/score.json", + "trial_count": 5 } ] }, @@ -606,15 +937,26 @@ "model_id": "claude-opus-4.8", "model_route": "anthropic/claude-opus-4-8", "normalized_score": 0.0, + "normalized_score_stddev": 0.0, "provider": "claude_oauth", "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 1, "raw_score": 0.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.0, "reference_score": 0.8283015254994576, - "source_run": "leaderboard/runs/article_suite/20260713T123222Z/claude-opus-4.8", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], "starter_score": 0.0, - "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", @@ -622,15 +964,26 @@ "model_id": "gpt-5.4-mini", "model_route": "azure/gpt-5.4-mini", "normalized_score": 0.0, + "normalized_score_stddev": 0.0, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 1, "raw_score": 0.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.0, "reference_score": 0.8283015254994576, - "source_run": "leaderboard/runs/article_suite/20260713T111944Z/gpt-5.4-mini", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "starter_score": 0.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_atari57_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", @@ -638,15 +991,26 @@ "model_id": "gpt-5.5", "model_route": "azure/gpt-5.5", "normalized_score": 0.0, + "normalized_score_stddev": 0.0, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 1, "raw_score": 0.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.0, "reference_score": 0.8283015254994576, - "source_run": "leaderboard/runs/article_suite/20260713T111604Z/gpt-5.5", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], "starter_score": 0.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", @@ -654,15 +1018,26 @@ "model_id": "gpt-5.6-sol", "model_route": "azure/gpt-5.6-sol", "normalized_score": 0.0, + "normalized_score_stddev": 0.0, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "max", "rank": 1, "raw_score": 0.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.0, "reference_score": 0.8283015254994576, - "source_run": "leaderboard/runs/article_suite/20260713T111156Z/gpt-5.6-sol", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "starter_score": 0.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_atari57_v1/score.json", + "trial_count": 5 } ] }, @@ -677,19 +1052,30 @@ "rows": [ { "harness": "opencode", - "model": "GPT-5.5", - "model_id": "gpt-5.5", - "model_route": "azure/gpt-5.5", - "normalized_score": 100.0, + "model": "GPT-5.6 Sol", + "model_id": "gpt-5.6-sol", + "model_route": "azure/gpt-5.6-sol", + "normalized_score": 74.5, + "normalized_score_stddev": 43.31570154112709, "provider": "azure", "provider_label": "Azure direct", - "provider_reasoning_effort": "xhigh", + "provider_reasoning_effort": "max", "rank": 1, - "raw_score": 400.0, + "raw_score": 298.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 173.26280616450836, "reference_score": 400.0, - "source_run": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], "starter_score": 0.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", @@ -697,15 +1083,26 @@ "model_id": "claude-opus-4.8", "model_route": "anthropic/claude-opus-4-8", "normalized_score": 0.0, + "normalized_score_stddev": 0.0, "provider": "claude_oauth", "provider_label": "Claude OAuth via pinned OpenCode plugin", "provider_reasoning_effort": "max", "rank": 2, "raw_score": 0.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.0, "reference_score": 400.0, - "source_run": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], "starter_score": 0.0, - "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", @@ -713,31 +1110,53 @@ "model_id": "gpt-5.4-mini", "model_route": "azure/gpt-5.4-mini", "normalized_score": 0.0, + "normalized_score_stddev": 0.0, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 2, "raw_score": 0.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.0, "reference_score": 400.0, - "source_run": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], "starter_score": 0.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_montezuma_v1/score.json", + "trial_count": 5 }, { "harness": "opencode", - "model": "GPT-5.6 Sol", - "model_id": "gpt-5.6-sol", - "model_route": "azure/gpt-5.6-sol", + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", "normalized_score": 0.0, + "normalized_score_stddev": 0.0, "provider": "azure", "provider_label": "Azure direct", - "provider_reasoning_effort": "max", + "provider_reasoning_effort": "xhigh", "rank": 2, "raw_score": 0.0, + "raw_score_imputation_count": 0, + "raw_score_stddev": 0.0, "reference_score": 400.0, - "source_run": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", + "sandbox": "daytona", + "source_run": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], "starter_score": 0.0, - "submission_detail": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_montezuma_v1/score.json" + "submission_detail": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/score.json", + "trial_count": 5 } ] }, @@ -754,114 +1173,232 @@ "metric": "final_normalized_score", "rows": [ { - "arithmetic_mean_normalized_score": 43.19059338577778, - "average_normalized_score": 43.19059338577778, - "final_normalized_score": 42.0019305542, + "arithmetic_mean_normalized_score": 58.80957286386666, + "average_normalized_score": 58.80957286386666, + "final_normalized_score": 62.03374323269565, + "final_normalized_score_stddev": 10.38022421100604, "harness": "opencode", - "median_normalized_score": 45.833333333, - "model": "GPT-5.5", - "model_id": "gpt-5.5", - "model_route": "azure/gpt-5.5", - "positive_display_score": 142.0019305542, - "provider": "azure", - "provider_label": "Azure direct", - "provider_reasoning_effort": "xhigh", - "rank": 1 + "mean_trial_iqm_normalized_score": 61.137710440759996, + "median_normalized_score": 79.783393502, + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "positive_display_score": 162.03374323269566, + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max", + "rank": 1, + "sandbox": "daytona", + "trial_count": 5 }, { - "arithmetic_mean_normalized_score": 39.37755364144445, - "average_normalized_score": 39.37755364144445, - "final_normalized_score": 36.7279720058, + "arithmetic_mean_normalized_score": 41.223767221955555, + "average_normalized_score": 41.223767221955555, + "final_normalized_score": 36.651764607913044, + "final_normalized_score_stddev": 5.524217450335815, "harness": "opencode", - "median_normalized_score": 39.203600385, + "mean_trial_iqm_normalized_score": 37.73164397808, + "median_normalized_score": 32.792734613, "model": "GPT-5.6 Sol", "model_id": "gpt-5.6-sol", "model_route": "azure/gpt-5.6-sol", - "positive_display_score": 136.72797200579998, + "positive_display_score": 136.65176460791304, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "max", - "rank": 2 + "rank": 2, + "sandbox": "daytona", + "trial_count": 5 }, { - "arithmetic_mean_normalized_score": 39.81691467833333, - "average_normalized_score": 39.81691467833333, - "final_normalized_score": 27.218574763200003, + "arithmetic_mean_normalized_score": 22.06816884377778, + "average_normalized_score": 22.06816884377778, + "final_normalized_score": 10.662970787652174, + "final_normalized_score_stddev": 18.67984640949238, "harness": "opencode", - "median_normalized_score": 14.050353128, - "model": "Claude Opus 4.8", - "model_id": "claude-opus-4.8", - "model_route": "anthropic/claude-opus-4-8", - "positive_display_score": 127.2185747632, - "provider": "claude_oauth", - "provider_label": "Claude OAuth via pinned OpenCode plugin", - "provider_reasoning_effort": "max", - "rank": 3 + "mean_trial_iqm_normalized_score": 16.7775760408, + "median_normalized_score": 5.870020964, + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "positive_display_score": 110.66297078765217, + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", + "rank": 3, + "sandbox": "daytona", + "trial_count": 5 }, { - "arithmetic_mean_normalized_score": -29.71822245, - "average_normalized_score": -29.71822245, - "final_normalized_score": -9.8773515774, + "arithmetic_mean_normalized_score": -31.610134487355555, + "average_normalized_score": -31.610134487355555, + "final_normalized_score": -17.411123288956524, + "final_normalized_score_stddev": 5.902995817222085, "harness": "opencode", - "median_normalized_score": -8.167582379, + "mean_trial_iqm_normalized_score": -18.58872338544, + "median_normalized_score": -3.609625668, "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", "model_route": "azure/gpt-5.4-mini", - "positive_display_score": 90.1226484226, + "positive_display_score": 82.58887671104347, "provider": "azure", "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", - "rank": 4 + "rank": 4, + "sandbox": "daytona", + "trial_count": 5 } ] } ], + "protocol": { + "agent_timeout_multiplier": 3, + "aggregation": { + "final_score": "interquartile_mean_across_all_trial_task_scores", + "final_variability": "sample_standard_deviation_of_trial_level_iqms", + "task_score": "arithmetic_mean_across_trials", + "task_variability": "sample_standard_deviation", + "trial_score": "interquartile_mean_across_nine_tasks" + }, + "baseline_agent_timeout_sec": { + "simulation_heuristics_ant_v1": 1800, + "simulation_heuristics_atari57_v1": 604800, + "simulation_heuristics_breakout_ram_v1": 1800, + "simulation_heuristics_breakout_rgb_v1": 1800, + "simulation_heuristics_halfcheetah_v1": 3600, + "simulation_heuristics_montezuma_v1": 1800, + "simulation_heuristics_pong_ram_v1": 1800, + "simulation_heuristics_vizdoom_d1_v1": 1800, + "simulation_heuristics_vizdoom_d3_v1": 1800 + }, + "trials": 5, + "version": "2.1" + }, "rows": [ { - "arithmetic_mean_normalized_score": 43.19059338577778, - "average_normalized_score": 43.19059338577778, - "final_normalized_score": 42.0019305542, + "arithmetic_mean_normalized_score": 58.80957286386666, + "average_normalized_score": 58.80957286386666, + "final_normalized_score": 62.03374323269565, + "final_normalized_score_stddev": 10.38022421100604, "harness": "opencode", - "median_normalized_score": 45.833333333, - "model": "GPT-5.5", - "model_id": "gpt-5.5", - "model_route": "azure/gpt-5.5", - "provider": "azure", - "provider_label": "Azure direct", - "provider_reasoning_effort": "xhigh", + "mean_trial_iqm_normalized_score": 61.137710440759996, + "median_normalized_score": 79.783393502, + "model": "Claude Opus 4.8", + "model_id": "claude-opus-4.8", + "model_route": "anthropic/claude-opus-4-8", + "provider": "claude_oauth", + "provider_label": "Claude OAuth via pinned OpenCode plugin", + "provider_reasoning_effort": "max", "rank": 1, + "raw_task_score_imputation_counts": { + "simulation_heuristics_ant_v1": 4, + "simulation_heuristics_atari57_v1": 0, + "simulation_heuristics_breakout_ram_v1": 0, + "simulation_heuristics_breakout_rgb_v1": 0, + "simulation_heuristics_halfcheetah_v1": 0, + "simulation_heuristics_montezuma_v1": 0, + "simulation_heuristics_pong_ram_v1": 0, + "simulation_heuristics_vizdoom_d1_v1": 0, + "simulation_heuristics_vizdoom_d3_v1": 0 + }, + "raw_task_score_stddevs": { + "simulation_heuristics_ant_v1": 142.40443086118393, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 33.448467827390836, + "simulation_heuristics_breakout_rgb_v1": 73.61249893869926, + "simulation_heuristics_halfcheetah_v1": 1827.0229584820154, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 0.17888543819998254, + "simulation_heuristics_vizdoom_d1_v1": 0.05547294365604739, + "simulation_heuristics_vizdoom_d3_v1": 127.03587288636228 + }, "raw_task_scores": { - "simulation_heuristics_ant_v1": 1852.8533956222025, + "simulation_heuristics_ant_v1": 2507.501036532837, "simulation_heuristics_atari57_v1": 0.0, - "simulation_heuristics_breakout_ram_v1": 843.0, - "simulation_heuristics_breakout_rgb_v1": 702.0, - "simulation_heuristics_halfcheetah_v1": 4206.220989602626, - "simulation_heuristics_montezuma_v1": 400.0, - "simulation_heuristics_pong_ram_v1": 8.0, - "simulation_heuristics_vizdoom_d1_v1": 0.7621399685554205, - "simulation_heuristics_vizdoom_d3_v1": 48.3 + "simulation_heuristics_breakout_ram_v1": 835.4, + "simulation_heuristics_breakout_rgb_v1": 804.6, + "simulation_heuristics_halfcheetah_v1": 7065.645225601496, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 20.919999999999998, + "simulation_heuristics_vizdoom_d1_v1": 0.8446999486498534, + "simulation_heuristics_vizdoom_d3_v1": 320.44 }, + "sandbox": "daytona", "source_runs": { - "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T095807Z/gpt-5.5", - "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T111604Z/gpt-5.5", - "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", - "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", - "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T103810Z/gpt-5.5", - "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", - "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", - "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T102218Z/gpt-5.5", - "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5" + "simulation_heuristics_ant_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_atari57_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_breakout_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_breakout_rgb_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_halfcheetah_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_montezuma_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_pong_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_vizdoom_d1_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_vizdoom_d3_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ] }, "submission_details": { - "simulation_heuristics_ant_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/score.json", - "simulation_heuristics_atari57_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/score.json", - "simulation_heuristics_breakout_ram_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/score.json", - "simulation_heuristics_breakout_rgb_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/score.json", - "simulation_heuristics_halfcheetah_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/score.json", - "simulation_heuristics_montezuma_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/score.json", - "simulation_heuristics_pong_ram_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/score.json", - "simulation_heuristics_vizdoom_d1_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/score.json", - "simulation_heuristics_vizdoom_d3_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/score.json" + "simulation_heuristics_ant_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/score.json", + "simulation_heuristics_atari57_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/score.json", + "simulation_heuristics_breakout_ram_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/score.json", + "simulation_heuristics_breakout_rgb_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/score.json", + "simulation_heuristics_halfcheetah_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/score.json", + "simulation_heuristics_montezuma_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/score.json", + "simulation_heuristics_pong_ram_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/score.json", + "simulation_heuristics_vizdoom_d1_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/score.json", + "simulation_heuristics_vizdoom_d3_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/score.json" }, "task_anchors": { "simulation_heuristics_ant_v1": { @@ -893,7 +1430,7 @@ "starter_score": -2.9999999999999996 }, "simulation_heuristics_vizdoom_d1_v1": { - "reference_score": 0.8178199748694897, + "reference_score": 0.8178199748694898, "starter_score": 0.44161998964846133 }, "simulation_heuristics_vizdoom_d3_v1": { @@ -901,24 +1438,102 @@ "starter_score": 31.9 } }, + "task_score_stddevs": { + "simulation_heuristics_ant_v1": 4.070422248601275, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 7.012257406411853, + "simulation_heuristics_breakout_rgb_v1": 13.287454681897577, + "simulation_heuristics_halfcheetah_v1": 24.40596858830019, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 0.7453559926490017, + "simulation_heuristics_vizdoom_d1_v1": 14.745599637347764, + "simulation_heuristics_vizdoom_d3_v1": 42.45851366513837 + }, "task_scores": { - "simulation_heuristics_ant_v1": -16.891796576, + "simulation_heuristics_ant_v1": 1.820348169, "simulation_heuristics_atari57_v1": 0.0, - "simulation_heuristics_breakout_ram_v1": 95.597484277, - "simulation_heuristics_breakout_rgb_v1": 70.758122744, - "simulation_heuristics_halfcheetah_v1": 2.73755349, - "simulation_heuristics_montezuma_v1": 100.0, - "simulation_heuristics_pong_ram_v1": 45.833333333, - "simulation_heuristics_vizdoom_d1_v1": 85.199359782, - "simulation_heuristics_vizdoom_d3_v1": 5.481283422 + "simulation_heuristics_breakout_ram_v1": 94.004192872, + "simulation_heuristics_breakout_rgb_v1": 89.2779783394, + "simulation_heuristics_halfcheetah_v1": 40.934675074999994, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 99.6666666666, + "simulation_heuristics_vizdoom_d1_v1": 107.1451288772, + "simulation_heuristics_vizdoom_d3_v1": 96.4371657756 + }, + "trial_count": 5, + "trial_final_normalized_scores": { + "1": 59.1255877926, + "2": 66.2620495092, + "3": 44.389717685200004, + "4": 71.6451968772, + "5": 64.2660003396 + }, + "trial_task_normalized_scores": { + "1": { + "simulation_heuristics_ant_v1": 0.0, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 98.742138365, + "simulation_heuristics_breakout_rgb_v1": 70.758122744, + "simulation_heuristics_halfcheetah_v1": 41.669121704, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 100.0, + "simulation_heuristics_vizdoom_d1_v1": 124.779364311, + "simulation_heuristics_vizdoom_d3_v1": 84.45855615 + }, + "2": { + "simulation_heuristics_ant_v1": 9.101740845, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 100.0, + "simulation_heuristics_breakout_rgb_v1": 79.783393502, + "simulation_heuristics_halfcheetah_v1": 42.425113199, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 100.0, + "simulation_heuristics_vizdoom_d1_v1": 100.2179605, + "simulation_heuristics_vizdoom_d3_v1": 114.104278075 + }, + "3": { + "simulation_heuristics_ant_v1": 0.0, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 98.322851153, + "simulation_heuristics_breakout_rgb_v1": 100.0, + "simulation_heuristics_halfcheetah_v1": 9.827504681, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 98.333333333, + "simulation_heuristics_vizdoom_d1_v1": 85.890478581, + "simulation_heuristics_vizdoom_d3_v1": 27.907754011 + }, + "4": { + "simulation_heuristics_ant_v1": 0.0, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 84.696016771, + "simulation_heuristics_breakout_rgb_v1": 95.848375451, + "simulation_heuristics_halfcheetah_v1": 77.681592164, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 100.0, + "simulation_heuristics_vizdoom_d1_v1": 113.721417769, + "simulation_heuristics_vizdoom_d3_v1": 134.35828877 + }, + "5": { + "simulation_heuristics_ant_v1": 0.0, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 88.259958071, + "simulation_heuristics_breakout_rgb_v1": 100.0, + "simulation_heuristics_halfcheetah_v1": 33.070043627, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 100.0, + "simulation_heuristics_vizdoom_d1_v1": 111.116423225, + "simulation_heuristics_vizdoom_d3_v1": 121.356951872 + } } }, { - "arithmetic_mean_normalized_score": 39.37755364144445, - "average_normalized_score": 39.37755364144445, - "final_normalized_score": 36.7279720058, + "arithmetic_mean_normalized_score": 41.223767221955555, + "average_normalized_score": 41.223767221955555, + "final_normalized_score": 36.651764607913044, + "final_normalized_score_stddev": 5.524217450335815, "harness": "opencode", - "median_normalized_score": 39.203600385, + "mean_trial_iqm_normalized_score": 37.73164397808, + "median_normalized_score": 32.792734613, "model": "GPT-5.6 Sol", "model_id": "gpt-5.6-sol", "model_route": "azure/gpt-5.6-sol", @@ -926,27 +1541,104 @@ "provider_label": "Azure direct", "provider_reasoning_effort": "max", "rank": 2, + "raw_task_score_imputation_counts": { + "simulation_heuristics_ant_v1": 0, + "simulation_heuristics_atari57_v1": 0, + "simulation_heuristics_breakout_ram_v1": 0, + "simulation_heuristics_breakout_rgb_v1": 0, + "simulation_heuristics_halfcheetah_v1": 0, + "simulation_heuristics_montezuma_v1": 0, + "simulation_heuristics_pong_ram_v1": 0, + "simulation_heuristics_vizdoom_d1_v1": 0, + "simulation_heuristics_vizdoom_d3_v1": 0 + }, + "raw_task_score_stddevs": { + "simulation_heuristics_ant_v1": 419.52351859557746, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 225.8501715739884, + "simulation_heuristics_breakout_rgb_v1": 122.53652516698847, + "simulation_heuristics_halfcheetah_v1": 291.0085829715705, + "simulation_heuristics_montezuma_v1": 173.26280616450836, + "simulation_heuristics_pong_ram_v1": 2.345207879911715, + "simulation_heuristics_vizdoom_d1_v1": 0.12456464290218099, + "simulation_heuristics_vizdoom_d3_v1": 71.79534803871347 + }, "raw_task_scores": { - "simulation_heuristics_ant_v1": 3815.360622145591, + "simulation_heuristics_ant_v1": 2272.356693724495, "simulation_heuristics_atari57_v1": 0.0, - "simulation_heuristics_breakout_ram_v1": 864.0, - "simulation_heuristics_breakout_rgb_v1": 702.0, - "simulation_heuristics_halfcheetah_v1": 5189.541724894487, - "simulation_heuristics_montezuma_v1": 0.0, - "simulation_heuristics_pong_ram_v1": 9.7, - "simulation_heuristics_vizdoom_d1_v1": 0.62749998614192, - "simulation_heuristics_vizdoom_d3_v1": 110.39999999999998 + "simulation_heuristics_breakout_ram_v1": 596.4, + "simulation_heuristics_breakout_rgb_v1": 647.2, + "simulation_heuristics_halfcheetah_v1": 6385.135940620764, + "simulation_heuristics_montezuma_v1": 298.0, + "simulation_heuristics_pong_ram_v1": 19.0, + "simulation_heuristics_vizdoom_d1_v1": 0.6554319871515035, + "simulation_heuristics_vizdoom_d3_v1": 80.67999999999999 }, + "sandbox": "daytona", "source_runs": { - "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T093815Z/gpt-5.6-sol", - "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T111156Z/gpt-5.6-sol", - "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", - "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", - "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T084340Z/gpt-5.6-sol", - "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", - "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", - "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T101636Z/gpt-5.6-sol", - "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T083940Z/gpt-5.6-sol" + "simulation_heuristics_ant_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_atari57_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_breakout_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_breakout_rgb_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_halfcheetah_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_montezuma_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_pong_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_vizdoom_d1_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_vizdoom_d3_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ] }, "submission_details": { "simulation_heuristics_ant_v1": "leaderboard/article_suite_submissions/gpt-5.6-sol/simulation_heuristics_ant_v1/score.json", @@ -989,7 +1681,7 @@ "starter_score": -2.9999999999999996 }, "simulation_heuristics_vizdoom_d1_v1": { - "reference_score": 0.8178199748694897, + "reference_score": 0.8178199748694898, "starter_score": 0.44161998964846133 }, "simulation_heuristics_vizdoom_d3_v1": { @@ -997,63 +1689,218 @@ "starter_score": 31.9 } }, + "task_score_stddevs": { + "simulation_heuristics_ant_v1": 11.991465810903719, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 47.34804435501101, + "simulation_heuristics_breakout_rgb_v1": 22.118506348049813, + "simulation_heuristics_halfcheetah_v1": 3.8873875677992533, + "simulation_heuristics_montezuma_v1": 43.31570154112709, + "simulation_heuristics_pong_ram_v1": 9.771699499561077, + "simulation_heuristics_vizdoom_d1_v1": 33.111283305681766, + "simulation_heuristics_vizdoom_d3_v1": 23.995771403261536 + }, "task_scores": { - "simulation_heuristics_ant_v1": 39.203600385, + "simulation_heuristics_ant_v1": -4.9009087386, "simulation_heuristics_atari57_v1": 0.0, - "simulation_heuristics_breakout_ram_v1": 100.0, - "simulation_heuristics_breakout_rgb_v1": 70.758122744, - "simulation_heuristics_halfcheetah_v1": 15.873072595, - "simulation_heuristics_montezuma_v1": 0.0, - "simulation_heuristics_pong_ram_v1": 52.916666667, - "simulation_heuristics_vizdoom_d1_v1": 49.409889366, - "simulation_heuristics_vizdoom_d3_v1": 26.236631016 + "simulation_heuristics_breakout_ram_v1": 43.899371069400004, + "simulation_heuristics_breakout_rgb_v1": 60.86642599300001, + "simulation_heuristics_halfcheetah_v1": 31.8442100886, + "simulation_heuristics_montezuma_v1": 74.5, + "simulation_heuristics_pong_ram_v1": 91.6666666666, + "simulation_heuristics_vizdoom_d1_v1": 56.834663982799995, + "simulation_heuristics_vizdoom_d3_v1": 16.3034759358 + }, + "trial_count": 5, + "trial_final_normalized_scores": { + "1": 37.797729594399996, + "2": 41.203895079, + "3": 28.388586371000002, + "4": 42.346736302800004, + "5": 38.921272543200004 + }, + "trial_task_normalized_scores": { + "1": { + "simulation_heuristics_ant_v1": 9.173888471, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 100.0, + "simulation_heuristics_breakout_rgb_v1": 70.758122744, + "simulation_heuristics_halfcheetah_v1": 26.12140084, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 79.166666667, + "simulation_heuristics_vizdoom_d1_v1": 42.828284045, + "simulation_heuristics_vizdoom_d3_v1": 40.106951872 + }, + "2": { + "simulation_heuristics_ant_v1": -7.591479, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 6.918238994, + "simulation_heuristics_breakout_rgb_v1": 70.758122744, + "simulation_heuristics_halfcheetah_v1": 33.28729024, + "simulation_heuristics_montezuma_v1": 100.0, + "simulation_heuristics_pong_ram_v1": 100.0, + "simulation_heuristics_vizdoom_d1_v1": 95.055823417, + "simulation_heuristics_vizdoom_d3_v1": -3.442513369 + }, + "3": { + "simulation_heuristics_ant_v1": -11.713338376, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 3.5639413, + "simulation_heuristics_breakout_rgb_v1": 70.758122744, + "simulation_heuristics_halfcheetah_v1": 36.605449762, + "simulation_heuristics_montezuma_v1": 100.0, + "simulation_heuristics_pong_ram_v1": 83.333333333, + "simulation_heuristics_vizdoom_d1_v1": 31.015418049, + "simulation_heuristics_vizdoom_d3_v1": 0.0 + }, + "4": { + "simulation_heuristics_ant_v1": -19.696444908, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 18.448637317, + "simulation_heuristics_breakout_rgb_v1": 70.758122744, + "simulation_heuristics_halfcheetah_v1": 32.792734613, + "simulation_heuristics_montezuma_v1": 100.0, + "simulation_heuristics_pong_ram_v1": 95.833333333, + "simulation_heuristics_vizdoom_d1_v1": 89.73418684, + "simulation_heuristics_vizdoom_d3_v1": 0.0 + }, + "5": { + "simulation_heuristics_ant_v1": 5.32283012, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 90.566037736, + "simulation_heuristics_breakout_rgb_v1": 21.299638989, + "simulation_heuristics_halfcheetah_v1": 30.414174988, + "simulation_heuristics_montezuma_v1": 72.5, + "simulation_heuristics_pong_ram_v1": 100.0, + "simulation_heuristics_vizdoom_d1_v1": 25.539607563, + "simulation_heuristics_vizdoom_d3_v1": 44.852941176 + } } }, { - "arithmetic_mean_normalized_score": 39.81691467833333, - "average_normalized_score": 39.81691467833333, - "final_normalized_score": 27.218574763200003, + "arithmetic_mean_normalized_score": 22.06816884377778, + "average_normalized_score": 22.06816884377778, + "final_normalized_score": 10.662970787652174, + "final_normalized_score_stddev": 18.67984640949238, "harness": "opencode", - "median_normalized_score": 14.050353128, - "model": "Claude Opus 4.8", - "model_id": "claude-opus-4.8", - "model_route": "anthropic/claude-opus-4-8", - "provider": "claude_oauth", - "provider_label": "Claude OAuth via pinned OpenCode plugin", - "provider_reasoning_effort": "max", + "mean_trial_iqm_normalized_score": 16.7775760408, + "median_normalized_score": 5.870020964, + "model": "GPT-5.5", + "model_id": "gpt-5.5", + "model_route": "azure/gpt-5.5", + "provider": "azure", + "provider_label": "Azure direct", + "provider_reasoning_effort": "xhigh", "rank": 3, + "raw_task_score_imputation_counts": { + "simulation_heuristics_ant_v1": 0, + "simulation_heuristics_atari57_v1": 0, + "simulation_heuristics_breakout_ram_v1": 0, + "simulation_heuristics_breakout_rgb_v1": 0, + "simulation_heuristics_halfcheetah_v1": 0, + "simulation_heuristics_montezuma_v1": 0, + "simulation_heuristics_pong_ram_v1": 0, + "simulation_heuristics_vizdoom_d1_v1": 0, + "simulation_heuristics_vizdoom_d3_v1": 0 + }, + "raw_task_score_stddevs": { + "simulation_heuristics_ant_v1": 306.36385957656836, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 131.153345363357, + "simulation_heuristics_breakout_rgb_v1": 122.53652516698847, + "simulation_heuristics_halfcheetah_v1": 698.5070322948952, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 9.705668446840743, + "simulation_heuristics_vizdoom_d1_v1": 0.3579053781713541, + "simulation_heuristics_vizdoom_d3_v1": 122.81457568220475 + }, "raw_task_scores": { - "simulation_heuristics_ant_v1": 2935.3698887625987, + "simulation_heuristics_ant_v1": 2006.0944869926293, "simulation_heuristics_atari57_v1": 0.0, - "simulation_heuristics_breakout_ram_v1": 387.0, - "simulation_heuristics_breakout_rgb_v1": 310.0, - "simulation_heuristics_halfcheetah_v1": 6021.504892591717, + "simulation_heuristics_breakout_ram_v1": 486.8, + "simulation_heuristics_breakout_rgb_v1": 647.2, + "simulation_heuristics_halfcheetah_v1": 4717.6162449605235, "simulation_heuristics_montezuma_v1": 0.0, - "simulation_heuristics_pong_ram_v1": 21.0, - "simulation_heuristics_vizdoom_d1_v1": 0.7992199431359768, - "simulation_heuristics_vizdoom_d3_v1": 397.7 + "simulation_heuristics_pong_ram_v1": 8.8, + "simulation_heuristics_vizdoom_d1_v1": 0.6197079806402326, + "simulation_heuristics_vizdoom_d3_v1": 101.49999999999999 }, + "sandbox": "daytona", "source_runs": { - "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T142919Z/claude-opus-4.8", - "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T123222Z/claude-opus-4.8", - "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", - "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", - "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T111714Z/claude-opus-4.8", - "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", - "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", - "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", - "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8" + "simulation_heuristics_ant_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05/tasks-81ec8d98a4" + ], + "simulation_heuristics_atari57_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_breakout_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-e6fa3546aa", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_breakout_rgb_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-885696cfa4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_halfcheetah_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01/simulation_heuristics_halfcheetah_v1", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-e6fa3546aa", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05/tasks-81ec8d98a4" + ], + "simulation_heuristics_montezuma_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_pong_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-885696cfa4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_vizdoom_d1_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_vizdoom_d3_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ] }, "submission_details": { - "simulation_heuristics_ant_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_ant_v1/score.json", - "simulation_heuristics_atari57_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_atari57_v1/score.json", - "simulation_heuristics_breakout_ram_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_ram_v1/score.json", - "simulation_heuristics_breakout_rgb_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_breakout_rgb_v1/score.json", - "simulation_heuristics_halfcheetah_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_halfcheetah_v1/score.json", - "simulation_heuristics_montezuma_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_montezuma_v1/score.json", - "simulation_heuristics_pong_ram_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_pong_ram_v1/score.json", - "simulation_heuristics_vizdoom_d1_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d1_v1/score.json", - "simulation_heuristics_vizdoom_d3_v1": "leaderboard/article_suite_submissions/claude-opus-4.8/simulation_heuristics_vizdoom_d3_v1/score.json" + "simulation_heuristics_ant_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_ant_v1/score.json", + "simulation_heuristics_atari57_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_atari57_v1/score.json", + "simulation_heuristics_breakout_ram_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_ram_v1/score.json", + "simulation_heuristics_breakout_rgb_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_breakout_rgb_v1/score.json", + "simulation_heuristics_halfcheetah_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_halfcheetah_v1/score.json", + "simulation_heuristics_montezuma_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_montezuma_v1/score.json", + "simulation_heuristics_pong_ram_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_pong_ram_v1/score.json", + "simulation_heuristics_vizdoom_d1_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d1_v1/score.json", + "simulation_heuristics_vizdoom_d3_v1": "leaderboard/article_suite_submissions/gpt-5.5/simulation_heuristics_vizdoom_d3_v1/score.json" }, "task_anchors": { "simulation_heuristics_ant_v1": { @@ -1085,7 +1932,7 @@ "starter_score": -2.9999999999999996 }, "simulation_heuristics_vizdoom_d1_v1": { - "reference_score": 0.8178199748694897, + "reference_score": 0.8178199748694898, "starter_score": 0.44161998964846133 }, "simulation_heuristics_vizdoom_d3_v1": { @@ -1093,24 +1940,102 @@ "starter_score": 31.9 } }, + "task_score_stddevs": { + "simulation_heuristics_ant_v1": 8.756962566015595, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 27.49546024414613, + "simulation_heuristics_breakout_rgb_v1": 22.118506348049813, + "simulation_heuristics_halfcheetah_v1": 9.330884765209642, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 40.44028519521613, + "simulation_heuristics_vizdoom_d1_v1": 95.13699952993868, + "simulation_heuristics_vizdoom_d3_v1": 41.04765230004701 + }, "task_scores": { - "simulation_heuristics_ant_v1": 14.050353128, + "simulation_heuristics_ant_v1": -12.5116242462, "simulation_heuristics_atari57_v1": 0.0, - "simulation_heuristics_breakout_ram_v1": 0.0, - "simulation_heuristics_breakout_rgb_v1": 0.0, - "simulation_heuristics_halfcheetah_v1": 26.986707946, + "simulation_heuristics_breakout_ram_v1": 20.9224318658, + "simulation_heuristics_breakout_rgb_v1": 60.86642599300001, + "simulation_heuristics_halfcheetah_v1": 9.5689381094, "simulation_heuristics_montezuma_v1": 0.0, - "simulation_heuristics_pong_ram_v1": 100.0, - "simulation_heuristics_vizdoom_d1_v1": 95.055812742, - "simulation_heuristics_vizdoom_d3_v1": 122.259358289 + "simulation_heuristics_pong_ram_v1": 49.1666666668, + "simulation_heuristics_vizdoom_d1_v1": 47.33864911959999, + "simulation_heuristics_vizdoom_d3_v1": 23.2620320856 + }, + "trial_count": 5, + "trial_final_normalized_scores": { + "1": 5.4339319906, + "2": 48.2436373978, + "3": 18.718893057000002, + "4": 9.604625305799999, + "5": 1.8867924528 + }, + "trial_task_normalized_scores": { + "1": { + "simulation_heuristics_ant_v1": -6.720960755, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 5.870020964, + "simulation_heuristics_breakout_rgb_v1": 21.299638989, + "simulation_heuristics_halfcheetah_v1": 0.0, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 41.666666667, + "simulation_heuristics_vizdoom_d1_v1": -117.814998349, + "simulation_heuristics_vizdoom_d3_v1": 22.326203209 + }, + "2": { + "simulation_heuristics_ant_v1": -24.004147668, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 70.020964361, + "simulation_heuristics_breakout_rgb_v1": 70.758122744, + "simulation_heuristics_halfcheetah_v1": 21.272433217, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 79.166666667, + "simulation_heuristics_vizdoom_d1_v1": 85.486445859, + "simulation_heuristics_vizdoom_d3_v1": 94.585561497 + }, + "3": { + "simulation_heuristics_ant_v1": -13.99811598, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 10.062893082, + "simulation_heuristics_breakout_rgb_v1": 70.758122744, + "simulation_heuristics_halfcheetah_v1": 12.773449459, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 100.0, + "simulation_heuristics_vizdoom_d1_v1": 104.662416785, + "simulation_heuristics_vizdoom_d3_v1": -0.601604278 + }, + "4": { + "simulation_heuristics_ant_v1": -1.398078834, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 9.224318658, + "simulation_heuristics_breakout_rgb_v1": 70.758122744, + "simulation_heuristics_halfcheetah_v1": 13.798807871, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 25.0, + "simulation_heuristics_vizdoom_d1_v1": 111.839447567, + "simulation_heuristics_vizdoom_d3_v1": 0.0 + }, + "5": { + "simulation_heuristics_ant_v1": -16.436817994, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 9.433962264, + "simulation_heuristics_breakout_rgb_v1": 70.758122744, + "simulation_heuristics_halfcheetah_v1": 0.0, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 0.0, + "simulation_heuristics_vizdoom_d1_v1": 52.519933736, + "simulation_heuristics_vizdoom_d3_v1": 0.0 + } } }, { - "arithmetic_mean_normalized_score": -29.71822245, - "average_normalized_score": -29.71822245, - "final_normalized_score": -9.8773515774, + "arithmetic_mean_normalized_score": -31.610134487355555, + "average_normalized_score": -31.610134487355555, + "final_normalized_score": -17.411123288956524, + "final_normalized_score_stddev": 5.902995817222085, "harness": "opencode", - "median_normalized_score": -8.167582379, + "mean_trial_iqm_normalized_score": -18.58872338544, + "median_normalized_score": -3.609625668, "model": "GPT-5.4 Mini", "model_id": "gpt-5.4-mini", "model_route": "azure/gpt-5.4-mini", @@ -1118,27 +2043,104 @@ "provider_label": "Azure direct", "provider_reasoning_effort": "xhigh", "rank": 4, + "raw_task_score_imputation_counts": { + "simulation_heuristics_ant_v1": 0, + "simulation_heuristics_atari57_v1": 0, + "simulation_heuristics_breakout_ram_v1": 0, + "simulation_heuristics_breakout_rgb_v1": 0, + "simulation_heuristics_halfcheetah_v1": 0, + "simulation_heuristics_montezuma_v1": 0, + "simulation_heuristics_pong_ram_v1": 0, + "simulation_heuristics_vizdoom_d1_v1": 0, + "simulation_heuristics_vizdoom_d3_v1": 0 + }, + "raw_task_score_stddevs": { + "simulation_heuristics_ant_v1": 364.1182774109385, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 44.0204497932495, + "simulation_heuristics_breakout_rgb_v1": 287.77473829368694, + "simulation_heuristics_halfcheetah_v1": 1438.0428685581014, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 16.192590898309017, + "simulation_heuristics_vizdoom_d1_v1": 0.012638255798059739, + "simulation_heuristics_vizdoom_d3_v1": 135.46233055724386 + }, "raw_task_scores": { - "simulation_heuristics_ant_v1": 1346.6965294546999, + "simulation_heuristics_ant_v1": 791.3786088082715, "simulation_heuristics_atari57_v1": 0.0, - "simulation_heuristics_breakout_ram_v1": 399.0, - "simulation_heuristics_breakout_rgb_v1": 396.0, - "simulation_heuristics_halfcheetah_v1": 3389.8660260183765, + "simulation_heuristics_breakout_ram_v1": 380.6, + "simulation_heuristics_breakout_rgb_v1": 389.4, + "simulation_heuristics_halfcheetah_v1": 1015.6002588284422, "simulation_heuristics_montezuma_v1": 0.0, - "simulation_heuristics_pong_ram_v1": -21.0, - "simulation_heuristics_vizdoom_d1_v1": -0.16450000800192355, - "simulation_heuristics_vizdoom_d3_v1": 2.4 + "simulation_heuristics_pong_ram_v1": -12.8, + "simulation_heuristics_vizdoom_d1_v1": -0.25834800247848033, + "simulation_heuristics_vizdoom_d3_v1": 81.36 }, + "sandbox": "daytona", "source_runs": { - "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T104315Z/gpt-5.4-mini", - "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T111944Z/gpt-5.4-mini", - "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", - "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", - "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T093314Z/gpt-5.4-mini", - "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", - "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", - "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T102707Z/gpt-5.4-mini", - "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T104034Z/gpt-5.4-mini" + "simulation_heuristics_ant_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_atari57_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_breakout_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02/simulation_heuristics_breakout_ram_v1", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_breakout_rgb_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_halfcheetah_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_montezuma_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_pong_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_vizdoom_d1_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_vizdoom_d3_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01/simulation_heuristics_vizdoom_d3_v1", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ] }, "submission_details": { "simulation_heuristics_ant_v1": "leaderboard/article_suite_submissions/gpt-5.4-mini/simulation_heuristics_ant_v1/score.json", @@ -1181,7 +2183,7 @@ "starter_score": -2.9999999999999996 }, "simulation_heuristics_vizdoom_d1_v1": { - "reference_score": 0.8178199748694897, + "reference_score": 0.8178199748694898, "starter_score": 0.44161998964846133 }, "simulation_heuristics_vizdoom_d3_v1": { @@ -1189,63 +2191,355 @@ "starter_score": 31.9 } }, + "task_score_stddevs": { + "simulation_heuristics_ant_v1": 10.40778807635678, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 9.228605826577622, + "simulation_heuristics_breakout_rgb_v1": 51.9448986087478, + "simulation_heuristics_halfcheetah_v1": 19.209845675650264, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 67.46912874308083, + "simulation_heuristics_vizdoom_d1_v1": 3.359451433761567, + "simulation_heuristics_vizdoom_d3_v1": 45.27484310055022 + }, "task_scores": { - "simulation_heuristics_ant_v1": -31.35954984, + "simulation_heuristics_ant_v1": -47.232499900200004, "simulation_heuristics_atari57_v1": 0.0, - "simulation_heuristics_breakout_ram_v1": 2.51572327, - "simulation_heuristics_breakout_rgb_v1": 15.523465704, - "simulation_heuristics_halfcheetah_v1": -8.167582379, + "simulation_heuristics_breakout_ram_v1": -1.3417190774, + "simulation_heuristics_breakout_rgb_v1": 14.332129963999998, + "simulation_heuristics_halfcheetah_v1": -39.8837988972, "simulation_heuristics_montezuma_v1": 0.0, - "simulation_heuristics_pong_ram_v1": -75.0, - "simulation_heuristics_vizdoom_d1_v1": -161.116433137, - "simulation_heuristics_vizdoom_d3_v1": -9.859625668 + "simulation_heuristics_pong_ram_v1": -40.8333333332, + "simulation_heuristics_vizdoom_d1_v1": -186.0627378054, + "simulation_heuristics_vizdoom_d3_v1": 16.5307486632 + }, + "trial_count": 5, + "trial_final_normalized_scores": { + "1": -17.9990639644, + "2": -14.669657405, + "3": -18.885235435600002, + "4": -28.268315785200002, + "5": -13.121344337 + }, + "trial_task_normalized_scores": { + "1": { + "simulation_heuristics_ant_v1": -36.632193477, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 0.209643606, + "simulation_heuristics_breakout_rgb_v1": 8.664259928, + "simulation_heuristics_halfcheetah_v1": -53.572769951, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": -75.0, + "simulation_heuristics_vizdoom_d1_v1": -180.053168387, + "simulation_heuristics_vizdoom_d3_v1": 97.159090909 + }, + "2": { + "simulation_heuristics_ant_v1": -50.421772091, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": -0.628930818, + "simulation_heuristics_breakout_rgb_v1": 95.848375451, + "simulation_heuristics_halfcheetah_v1": -12.137156309, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": -75.0, + "simulation_heuristics_vizdoom_d1_v1": -187.56513016, + "simulation_heuristics_vizdoom_d3_v1": -10.160427807 + }, + "3": { + "simulation_heuristics_ant_v1": -40.65567106, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 3.983228512, + "simulation_heuristics_breakout_rgb_v1": 0.0, + "simulation_heuristics_halfcheetah_v1": -53.770506118, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": -58.333333333, + "simulation_heuristics_vizdoom_d1_v1": -187.56513016, + "simulation_heuristics_vizdoom_d3_v1": 1.771390374 + }, + "4": { + "simulation_heuristics_ant_v1": -63.436898931, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": 6.708595388, + "simulation_heuristics_breakout_rgb_v1": -48.194945848, + "simulation_heuristics_halfcheetah_v1": -27.203049655, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": -75.0, + "simulation_heuristics_vizdoom_d1_v1": -187.56513016, + "simulation_heuristics_vizdoom_d3_v1": -2.506684492 + }, + "5": { + "simulation_heuristics_ant_v1": -45.015963942, + "simulation_heuristics_atari57_v1": 0.0, + "simulation_heuristics_breakout_ram_v1": -16.981132075, + "simulation_heuristics_breakout_rgb_v1": 15.342960289, + "simulation_heuristics_halfcheetah_v1": -52.735512453, + "simulation_heuristics_montezuma_v1": 0.0, + "simulation_heuristics_pong_ram_v1": 79.166666667, + "simulation_heuristics_vizdoom_d1_v1": -187.56513016, + "simulation_heuristics_vizdoom_d3_v1": -3.609625668 + } } } ], "source_runs": { "claude-opus-4.8": { - "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T142919Z/claude-opus-4.8", - "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T123222Z/claude-opus-4.8", - "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", - "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", - "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T111714Z/claude-opus-4.8", - "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", - "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T090958Z/claude-opus-4.8", - "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8", - "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T110612Z/claude-opus-4.8" + "simulation_heuristics_ant_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_atari57_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_breakout_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_breakout_rgb_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_halfcheetah_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_montezuma_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_pong_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_vizdoom_d1_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ], + "simulation_heuristics_vizdoom_d3_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-02/tasks-e7ee9d1b4f", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-claude/claude-opus-4.8/trial-05" + ] }, "gpt-5.4-mini": { - "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T104315Z/gpt-5.4-mini", - "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T111944Z/gpt-5.4-mini", - "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", - "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", - "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T093314Z/gpt-5.4-mini", - "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", - "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T091939Z/gpt-5.4-mini", - "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T102707Z/gpt-5.4-mini", - "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T104034Z/gpt-5.4-mini" + "simulation_heuristics_ant_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_atari57_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_breakout_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02/simulation_heuristics_breakout_ram_v1", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_breakout_rgb_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_halfcheetah_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_montezuma_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_pong_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_vizdoom_d1_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ], + "simulation_heuristics_vizdoom_d3_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-01/simulation_heuristics_vizdoom_d3_v1", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt54mini/gpt-5.4-mini/trial-05" + ] }, "gpt-5.5": { - "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T095807Z/gpt-5.5", - "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T111604Z/gpt-5.5", - "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", - "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", - "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T103810Z/gpt-5.5", - "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", - "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5", - "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T102218Z/gpt-5.5", - "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T084111Z/gpt-5.5" + "simulation_heuristics_ant_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05/tasks-81ec8d98a4" + ], + "simulation_heuristics_atari57_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_breakout_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-e6fa3546aa", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_breakout_rgb_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-885696cfa4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_halfcheetah_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01/simulation_heuristics_halfcheetah_v1", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-e6fa3546aa", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05/tasks-81ec8d98a4" + ], + "simulation_heuristics_montezuma_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_pong_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04/tasks-885696cfa4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_vizdoom_d1_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ], + "simulation_heuristics_vizdoom_d3_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt55/gpt-5.5/trial-05" + ] }, "gpt-5.6-sol": { - "simulation_heuristics_ant_v1": "leaderboard/runs/article_suite/20260713T093815Z/gpt-5.6-sol", - "simulation_heuristics_atari57_v1": "leaderboard/runs/article_suite/20260713T111156Z/gpt-5.6-sol", - "simulation_heuristics_breakout_ram_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", - "simulation_heuristics_breakout_rgb_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", - "simulation_heuristics_halfcheetah_v1": "leaderboard/runs/article_suite/20260713T084340Z/gpt-5.6-sol", - "simulation_heuristics_montezuma_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", - "simulation_heuristics_pong_ram_v1": "leaderboard/runs/article_suite/20260713T080935Z/gpt-5.6-sol", - "simulation_heuristics_vizdoom_d1_v1": "leaderboard/runs/article_suite/20260713T101636Z/gpt-5.6-sol", - "simulation_heuristics_vizdoom_d3_v1": "leaderboard/runs/article_suite/20260713T083940Z/gpt-5.6-sol" + "simulation_heuristics_ant_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_atari57_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_breakout_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_breakout_rgb_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_halfcheetah_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04/tasks-81ec8d98a4", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_montezuma_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_pong_ram_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_vizdoom_d1_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ], + "simulation_heuristics_vizdoom_d3_v1": [ + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-01", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-02", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-03", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-04", + "leaderboard/runs/article_suite/article-suite-v2-daytona-20260714-gpt56/gpt-5.6-sol/trial-05" + ] } }, "task_count": 9, diff --git a/website/index.html b/website/index.html index 6c22c69..37b2b53 100644 --- a/website/index.html +++ b/website/index.html @@ -251,7 +251,7 @@ background:var(--paper);overflow:hidden} .chart-card h4{font-size:.9875rem;font-weight:600;letter-spacing:-.01em} .chart-metric{display:block;margin-top:.12rem;font-size:.7rem;color:var(--ink-soft);line-height:1.35} -.chart-body{display:grid;grid-template-columns:5.2rem minmax(0,1fr) 3.7rem; +.chart-body{display:grid;grid-template-columns:5.2rem minmax(0,1fr) 5.1rem; gap:.35rem;align-items:stretch;min-width:0;margin-top:.8rem} .chart-labels,.chart-values,.chart-bars{display:grid;grid-template-rows:repeat(4,16px);gap:.48rem} .chart-model{font-size:.69rem;line-height:16px;color:var(--ink);overflow-wrap:anywhere} @@ -265,6 +265,12 @@ .chart-bars{position:relative;z-index:1} .chart-bar-row{position:relative;height:16px} .chart-bar{position:absolute;top:2px;height:12px;min-width:2px;border-radius:1px;background:#555} +.chart-error-range,.final-error-range{position:absolute;height:1px;background:#222;opacity:.7;z-index:2} +.chart-error-range{top:7px}.final-error-range{top:14px} +.chart-error-range::before,.chart-error-range::after, +.final-error-range::before,.final-error-range::after{content:"";position:absolute;top:-2px;width:1px;height:5px;background:#222} +.chart-error-range::before,.final-error-range::before{left:0} +.chart-error-range::after,.final-error-range::after{right:0} .model-gpt-5-5{background:#111}.model-gpt-5-6-sol{background:#4b4b4b} .model-claude-opus-4-8{background:#858585}.model-gpt-5-4-mini{background:#b7b7b7} .chart-anchors{display:flex;justify-content:space-between;gap:.6rem;margin-top:.7rem; @@ -286,7 +292,7 @@ .result-actions a:hover{text-decoration-color:var(--forest)} @media(max-width:680px){ .leaderboard-grid{grid-template-columns:1fr} - .chart-body{grid-template-columns:5.2rem minmax(0,1fr) 3.7rem} + .chart-body{grid-template-columns:5.2rem minmax(0,1fr) 5.1rem} .final-row{grid-template-columns:1fr} .final-plot{height:26px}.final-bar{height:18px} .final-value{text-align:left;display:flex;align-items:baseline;gap:.6rem} @@ -481,8 +487,6 @@

Real-world operation

Results

-

Four coding models, nine article-derived tasks, one robust cross-task score.

-

The task panels show each environment's native raw score. Cross-task ranking first normalizes each task so the public starter scores 0 and the trusted article-level reference scores 100, then uses the interquartile mean: remove the lowest two and highest two task scores and average the middle five.