Add article-derived nine-task experiment suite

.env.example

@@ -3,11 +3,12 @@ AZURE_API_ENDPOINT=https://your-resource.openai.azure.com/
 AZURE_API_KEY=
 AZURE_API_VERSION=preview
 
-# Claude subscription route used by OpenHands ACPAgent
+# Claude subscription route used by OpenCode's pinned OAuth plugin
 CLAUDE_CODE_OAUTH_TOKEN=
 
-# Optional Docker/runtime overrides
-GENESISBENCH_DOCKER_IMAGE=genesisbench-simulation-heuristics-ant-v1-runner:latest
-GENESISBENCH_DOCKER_PLATFORM=
+# Optional Daytona sandbox route
+DAYTONA_API_KEY=
+DAYTONA_TARGET=us
+
+# Runtime overrides
 GENESISBENCH_ENV_FILE=.env
-GENESISBENCH_OPENHANDS_PYTHON=

.github/workflows/ci.yml

@@ -41,15 +41,20 @@ jobs:
           tasks/simulation_heuristics_ant_v1
           --level publication-grade
 
-      - name: Run native BenchFlow oracle
-        run: >-
-          uv run bench eval run
-          --tasks-dir tasks/simulation_heuristics_ant_v1
-          --agent oracle
-          --sandbox docker
-          --context-root .
-          --jobs-dir /tmp/genesisbench-oracle
-          --concurrency 1
+      - name: Run native BenchFlow oracle smoke
+        shell: bash
+        run: |
+          full_config="$(mktemp)"
+          cp tasks/simulation_heuristics_ant_v1/verifier/config.toml "$full_config"
+          trap 'cp "$full_config" tasks/simulation_heuristics_ant_v1/verifier/config.toml; rm -f "$full_config"' EXIT
+          cp tasks/simulation_heuristics_ant_v1/verifier/config_smoke.toml tasks/simulation_heuristics_ant_v1/verifier/config.toml
+          uv run bench eval run \
+            --tasks-dir tasks/simulation_heuristics_ant_v1 \
+            --agent oracle \
+            --sandbox docker \
+            --context-root . \
+            --jobs-dir /tmp/genesisbench-oracle \
+            --concurrency 1
 
       - name: Audit packaged leaderboard
         run: uv run python scripts/audit_simulation_heuristics_ant_v1_leaderboard.py

.gitignore

@@ -9,6 +9,7 @@
 __pycache__/
 *.pyc
 *.egg-info/
+_vizdoom.ini
 build/
 dist/
 

README.md

@@ -22,10 +22,10 @@ instruction-tuned language model.
 
 ## Reference Task: Simulation Heuristics Ant v1
 
-`tasks/simulation_heuristics_ant_v1/` is the first executable task and the canonical example for
-future contributors. An agent receives a weak rhythmic CPG/PD controller for
-Gymnasium `Ant-v5`, repeatedly edits and evaluates it, and submits
-`final_policy/policy.py`.
+`tasks/simulation_heuristics_ant_v1/` is the first executable task and the
+canonical package example. The complete article-derived suite contains nine
+tasks spanning MuJoCo locomotion, Atari RAM and vision control, VizDoom,
+long-horizon recovery, and the aggregate Atari57 workflow.
 
 The package follows BenchFlow `0.6.5`'s native `task.md` format
 (`schema_version: "1.3"`, document version `"0.6"`).
@@ -76,15 +76,16 @@ uv run python scripts/prepare_task.py \
   --force
 ```
 
-The prepared OpenHands workspace deliberately excludes `verifier/`, `oracle/`,
+The prepared OpenCode workspace deliberately excludes `verifier/`, `oracle/`,
 and `evidence/`.
 
-## OpenHands Experiment
+## OpenCode Article-Suite Experiment
 
-Build the isolated runner:
+OpenCode is the default and only leaderboard harness for the nine-task suite.
+Install the Daytona dependency when using the hosted sandbox:
 
 ```bash
-sh scripts/build_simulation_heuristics_ant_v1_runner_image.sh
+uv sync --extra dev --extra sandbox-daytona
 ```
 
 Configure credentials:
@@ -93,24 +94,51 @@ Configure credentials:
 cp .env.example .env
 ```
 
-Run one agent:
+Run one model across all nine tasks:
 
 ```bash
-uv run python scripts/run_simulation_heuristics_ant_v1_experiment.py \
-  --model gpt-5.6-sol \
-  --minutes 30
+uv run python scripts/run_article_suite.py \
+  --model gpt-5.6-sol
 ```
 
-See `experiments/simulation_heuristics_ant_v1/README.md` for model routes, fairness controls, artifact
-layout, and leaderboard regeneration.
+Run all four canonical models and rebuild the aggregate leaderboard:
 
-## Current Leaderboard
+```bash
+uv run python scripts/run_article_suite.py \
+  --all-models
+uv run python scripts/build_article_suite_leaderboard.py
+```
+
+See `experiments/article_suite/README.md` for the exact model routes, task
+manifest, isolation controls, and scoring contract. The task-by-task research
+mapping is documented in `docs/learning-beyond-gradients-suite.md`.
+
+## Article-Suite Leaderboard
+
+The first OpenCode sweep across all nine article-derived tasks:
+
+| Rank | Model | Nine-task average |
+| ---: | --- | ---: |
+| 1 | GPT-5.5 | 43.19 |
+| 2 | Claude Opus 4.8 | 39.82 |
+| 3 | GPT-5.6 Sol | 39.38 |
+| 4 | GPT-5.4 Mini | -29.72 |
+
+See [`leaderboard/ARTICLE_SUITE.md`](leaderboard/ARTICLE_SUITE.md) for every
+per-task score and `leaderboard/article_suite.json` for the machine-readable
+leaderboard.
+
+Scores are unbounded normalized values: `0` matches the public starter and
+`100` matches the trusted article-level reference. Negative scores are genuine
+regressions; scores above `100` exceed the reference.
+
+## Legacy Ant-Only Leaderboard
 
 ![GenesisBench Simulation Heuristics Ant v1 leaderboard](leaderboard/simulation_heuristics_ant_v1_leaderboard.png)
 
-The first four-model OpenHands sweep used equal 30-minute budgets and each
-model's highest supported reasoning setting. Machine-readable results and
-packaged policies are in `leaderboard/`.
+The table below is the historical Ant-only OpenHands sweep. It remains for
+provenance; new GenesisBench leaderboard runs use OpenCode and the nine-task
+article suite.
 
 | Rank | Model | Hidden-suite score |
 | ---: | --- | ---: |

THIRD_PARTY_NOTICES.md

@@ -24,3 +24,26 @@ The Apache License 2.0 text is included in `LICENSES/Apache-2.0.txt`.
 
 GenesisBench as a combined project is distributed under GPL-3.0. Apache-2.0 is
 compatible with GPLv3; upstream notices remain applicable to the derived files.
+
+## Learning Beyond Gradients Atari controllers
+
+The Pong RAM and Breakout RAM/RGB task anchors retain or adapt controller
+structure and parameterization from the corresponding public article
+artifacts:
+
+- `tasks/simulation_heuristics_pong_ram_v1/{starter_policy,oracle}/policy.py`
+- `tasks/simulation_heuristics_pong_ram_v1/verifier/anchor_policies/*.py`
+- `tasks/simulation_heuristics_breakout_ram_v1/{starter_policy,oracle}/policy.py`
+- `tasks/simulation_heuristics_breakout_ram_v1/verifier/anchor_policies/*.py`
+- `tasks/simulation_heuristics_breakout_rgb_v1/{starter_policy,oracle}/policy.py`
+- `tasks/simulation_heuristics_breakout_rgb_v1/verifier/anchor_policies/*.py`
+
+Upstream:
+
+- Project: `Trinkle23897/learning-beyond-gradients`
+- Sources: `atari/pong/heuristic_pong.py` and
+  `atari/breakout/heuristic_breakout.py`
+- Reviewed revision: `3555c2956c257d49a5015b782cbe485b14fd659e`
+- Copyright notice retained in derived files:
+  `Copyright 2021 Garena Online Private Limited`
+- License: Apache License 2.0

docs/learning-beyond-gradients-suite.md

@@ -0,0 +1,77 @@
+# Learning Beyond Gradients task suite
+
+GenesisBench represents each experiment-family row in the article as one
+native task package. Together with the original Ant task, the suite contains
+nine tasks.
+
+| Task | Article contract | Trusted reproduction |
+| --- | --- | --- |
+| `simulation_heuristics_ant_v1` | CPG/PD through adaptive residual MPC | Gymnasium mean 5895.932216 with exact source/action parity |
+| `simulation_heuristics_pong_ram_v1` | RAM-only Pong, score 21 | 21 on hidden seeds and reset variants |
+| `simulation_heuristics_breakout_ram_v1` | `387 → 507 → 839 → 864` | 864 on nominal and shifted starts |
+| `simulation_heuristics_breakout_rgb_v1` | `310 → 428 → 864` pixel transfer | 864 on nominal and shifted starts |
+| `simulation_heuristics_halfcheetah_v1` | staged-tree MPC, seeds 100–104 | mean 11836.693449819431 |
+| `simulation_heuristics_vizdoom_d1_v1` | screen CV + `HEALTH` | mean 0.9440999741666019 |
+| `simulation_heuristics_vizdoom_d3_v1` | screen CV + five public variables | returns `[545,475,480,440,690,500,600,595,530,715]` |
+| `simulation_heuristics_atari57_v1` | 57 games × 2 modes × 3 searches | complete 342-slot and 6.84B-frame protocol |
+| `simulation_heuristics_montezuma_v1` | 400-point macro route plus recovery | 400 in 1,769 steps with recovery variants |
+
+## Common benchmark boundary
+
+Each task follows the original Ant package:
+
+```text
+fixed public starter
+→ OpenCode research in an isolated sandbox
+→ public development evaluator
+→ standardized final artifact
+→ hidden verifier
+→ starter/reference normalized score
+```
+
+The task workspace never contains `verifier/`, `oracle/`, or `evidence/`.
+
+## Atari57 exception
+
+Atari57 is one aggregate task rather than 114 standalone tasks. Its artifact
+resolves one policy for every `(game, observation mode, repeat index)` tuple:
+
+```text
+57 × 2 × 3 = 342 policy/search slots
+```
+
+An official result is eligible only when the interaction ledger records all
+342 searches at 20,000,000 frames each. Incomplete submissions return a clean
+zero and are not presented as article reproductions.
+
+## Model and harness contract
+
+All new leaderboard runs use the BenchFlow `opencode` ACP harness. The canonical
+matrix is:
+
+| Model | Provider | Effort |
+| --- | --- | --- |
+| GPT-5.6 Sol | Azure | `max` |
+| GPT-5.5 | Azure | `xhigh` |
+| Claude Opus 4.8 | Claude OAuth through pinned OpenCode plugin | `max` |
+| GPT-5.4 Mini | Azure | `xhigh` |
+
+OpenCode talks directly to the provider because BenchFlow 0.6.5's
+chat-completions gateway cannot faithfully transform Azure GPT-5.6 Sol tool
+calls. BenchFlow continues to own task staging, Daytona/Docker isolation, ACP
+trajectory capture, timing, and verifier execution.
+
+## Aggregate score
+
+The article-suite leaderboard reports every normalized task score and their
+unweighted arithmetic mean:
+
+```text
+average = sum(nine normalized task scores) / 9
+```
+
+The runner and resumable leaderboard builder live in:
+
+- `scripts/run_article_suite.py`
+- `scripts/build_article_suite_leaderboard.py`
+- `experiments/article_suite/`

docs/simulation-heuristics-ant-v1-task-design.md

@@ -30,9 +30,34 @@ That experiment evolved an interpretable controller through:
 - warm-started planning and adaptive gait timing.
 
 The public artifacts report a five-episode mean near `6005` in EnvPool's
-`Ant-v5`. GenesisBench does not ask agents to reproduce that exact public
-solution on the exact public setup because copying a completed controller would
-not measure autonomous research.
+`Ant-v5`: mean `6005.521`, minimum `5776.805`, and maximum `6146.208`.
+
+GenesisBench keeps the public weak CPG/PD starter so agents still face an
+authentic policy-improvement problem. The trusted oracle and normalization
+anchor now reproduce the article's final controller instead of using an
+unrelated stronger rhythmic policy.
+
+That reference combines:
+
+- speed-adaptive phase increment and stance duty;
+- higher-harmonic CPG joint targets;
+- posture and heading feedback;
+- a 10-step copied-MuJoCo planning horizon;
+- 96 residual-action candidates per external step;
+- temporal residual smoothing and warm-start plan decay;
+- forward, control, posture, yaw, height, health, and terminal-velocity
+  objective terms.
+
+The source used EnvPool `1.1.1`; GenesisBench uses Gymnasium. The supplied
+`ant_envpool.xml` and Gymnasium's Ant XML parse to the same MuJoCo model, but
+the two runtimes use different reset random-number streams. Machine-readable
+source hashes and both result families live in
+`tasks/simulation_heuristics_ant_v1/evidence/source_provenance.json`.
+
+On the development host, both the imported source policy and the GenesisBench
+adaptation produced the same Gymnasium seed-`0..4` returns: mean
+`5895.932216`, minimum `5791.444245`, and maximum `6131.400491`. A separate
+50-step action-parity probe had maximum absolute difference `0.0`.
 
 ## Benchmark translation
 
@@ -81,6 +106,17 @@ It must expose `Policy` or `make_policy` and produce finite eight-dimensional
 actions from observations. The final evaluator imports this artifact in a clean
 process after the agent exits.
 
+Policies may optionally expose:
+
+```python
+configure_simulator(model_xml_path=..., frame_skip=...)
+```
+
+The evaluator supplies a copied model matching the current episode. It never
+passes the live `Env`, mutable scored `MjData`, reward, `info`, or hidden suite
+configuration. This preserves the article's model-based planning capability
+without letting a policy step or mutate the scored simulator.
+
 ## Hidden evaluation
 
 The reproducibility suite evaluates:
@@ -116,6 +152,11 @@ policies on the same platform as the submission. This keeps `0` and `100`
 stable across small MuJoCo platform differences. A hosted private suite may
 instead inject fixed scores through its private anchors file.
 
+The reference and oracle policy files are byte-identical. The verifier caches
+evaluations by `policy.py` SHA-256 fingerprint so an oracle run evaluates the
+expensive MPC controller once rather than once as the submission and again as
+the reference anchor.
+
 Interpretation:
 
 - `0`: matches the starter;
@@ -141,11 +182,23 @@ authoritative interaction meter before making sample-efficiency claims across
 agents.
 
 Internal MPC transitions must also remain separate from external environment
-steps in any future accounting.
+steps in any future accounting. The final reference performs roughly
+`96 x 10` copied-model environment steps for every scored external action, so
+full hidden evaluation is intentionally a long-running publication check.
+
+Ordinary unit tests use an injected one-step hidden config. The full
+five-seed, 1,000-step reproduction is opt-in through:
+
+```bash
+GENESISBENCH_RUN_SLOW_ANT_MPC=1 \
+  uv run pytest -q tests/test_simulation_heuristics_ant_v1.py
+```
 
 ## Benchmark integrity
 
 - The agent container receives the public task but not `verifier/`.
+- The agent receives only a copied model path for optional MPC, never the live
+  scored environment.
 - Credentials are supplied through a temporary mode-`0600` file and removed
   after agent startup.
 - Final scores come from the independently imported final policy.