Skip to content

Publish five-trial Daytona article-suite leaderboard#22

Merged
bingran-you merged 15 commits into
mainfrom
bry/five-trial-leaderboard
Jul 15, 2026
Merged

Publish five-trial Daytona article-suite leaderboard#22
bingran-you merged 15 commits into
mainfrom
bry/five-trial-leaderboard

Conversation

@bingran-you

Copy link
Copy Markdown
Collaborator

Summary

  • run all four canonical models on all nine article-derived tasks in Daytona
  • increase every task's agent timeout to 3x its original limit
  • publish five independent trials per model-task pair (180 accepted cells)
  • compute the final score as RLiable-style IQM over the pooled 5 x 9 normalized score matrix
  • publish mean and sample standard deviation for native task scores and per-trial IQMs
  • add 180 trial-level score files and 180 matching provenance metadata files
  • make Daytona the authoritative default sandbox and record exact provider-specific inference settings
  • update the README, detailed leaderboard report, scoring docs, website JSON, and native HTML/CSS plots
  • add responsive uncertainty whiskers and fix clipped leaderboard values on desktop and mobile

Final pooled IQM

Rank Model IQM Trial-IQM SD Positive display index
1 Claude Opus 4.8 62.03 10.38 162.03
2 GPT-5.6 Sol 36.65 5.52 136.65
3 GPT-5.5 10.66 18.68 110.66
4 GPT-5.4 Mini -17.41 5.90 82.59

Validation

  • uv run python scripts/validate_tasks.py
  • uv run bench tasks check tasks/simulation_heuristics_ant_v1 --level publication-grade
  • uv run python scripts/audit_simulation_heuristics_ant_v1_leaderboard.py
  • uv run ruff check .
  • uv run pytest -q (106 passed, 18 skipped)
  • uv build
  • CI-equivalent Docker oracle smoke (1/1, 100%)
  • localhost HTML audit at desktop and 390px mobile width: zero clipped values or uncertainty ranges

@vercel

vercel Bot commented Jul 15, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
genesis-bench Ready Ready Preview, Comment Jul 15, 2026 1:52pm

Request Review

@bingran-you bingran-you merged commit eaa70f3 into main Jul 15, 2026
3 checks passed
@bingran-you bingran-you deleted the bry/five-trial-leaderboard branch July 15, 2026 13:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant