Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 0 additions & 27 deletions containers/simulation_heuristics_ant_v1/Dockerfile

This file was deleted.

6 changes: 3 additions & 3 deletions docs/simulation-heuristics-ant-v1-task-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -199,8 +199,8 @@ GENESISBENCH_RUN_SLOW_ANT_MPC=1 \
- The agent container receives the public task but not `verifier/`.
- The agent receives only a copied model path for optional MPC, never the live
scored environment.
- Credentials are supplied through a temporary mode-`0600` file and removed
after agent startup.
- Provider credentials are scoped to the selected model route and are not
copied into the public task workspace or committed artifacts.
- Final scores come from the independently imported final policy.
- Invalid actions receive a fixed failure result instead of crashing scoring.
- Public packaged score files contain only relative paths.
Expand All @@ -213,7 +213,7 @@ Simulation Heuristics Ant v1 demonstrates the full GenesisBench task lifecycle:
```text
task scaffold
→ public workspace preparation
→ autonomous OpenHands run
→ autonomous BenchFlow/OpenCode run
→ clean final evaluation
→ packaged submission
→ audited leaderboard
Expand Down
117 changes: 0 additions & 117 deletions experiments/simulation_heuristics_ant_v1/README.md

This file was deleted.

29 changes: 0 additions & 29 deletions experiments/simulation_heuristics_ant_v1/models.toml

This file was deleted.

4 changes: 4 additions & 0 deletions leaderboard/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@ nine task-specific rankings followed by the final IQM ranking.

![GenesisBench Simulation Heuristics Ant v1 leaderboard](simulation_heuristics_ant_v1_leaderboard.png)

This is an immutable historical OpenHands result. The retired runner is no
longer shipped in the current tree; see [`REPORT.md`](REPORT.md) for provenance
and the Git-history source pointer.

| Rank | Agent model | Harness | Reasoning | Score | Normalized | Nominal | Robust | Fall rate |
| ---: | --- | --- | --- | ---: | ---: | ---: | ---: | ---: |
| 1 | GPT-5.6 Sol | OpenHands SDK | xhigh | 3417.86 | 150.24 | 3399.54 | 3460.62 | 0.0% |
Expand Down
7 changes: 6 additions & 1 deletion leaderboard/REPORT.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ surviving for more steps.
- Machine-readable leaderboard: `leaderboard/simulation_heuristics_ant_v1.json`
- Packaged policies and scores: `leaderboard/submissions/`
- Deterministic verifier: `tasks/simulation_heuristics_ant_v1/verifier/evaluate_hidden.py`
- Run and audit commands: `experiments/simulation_heuristics_ant_v1/README.md`
- Artifact audit: `scripts/audit_simulation_heuristics_ant_v1_leaderboard.py`

Every packaged policy was re-evaluated after packaging, and all four scores
matched their recorded score exactly.
Expand All @@ -65,6 +65,11 @@ Raw OpenHands trajectories are intentionally excluded from the source
distribution because they are large and can contain provider-specific runtime
metadata. Packaged metadata retains a non-secret source run identifier.

The executable OpenHands runner was retired after the nine-task OpenCode suite
became canonical. Its final source snapshot remains available in Git history at
commit `8f05ed4`; the current tree retains only the immutable published
artifacts, verifier, audit, and this methodology report.

## Limitations

- This is one agent run per model, not a multi-trial estimate of model quality.
Expand Down
18 changes: 0 additions & 18 deletions scripts/build_simulation_heuristics_ant_v1_runner_image.sh

This file was deleted.

Loading