Revise article scoring and publish HTML leaderboards#19
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e032bfc483
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "positive_display_score": row["final_normalized_score"] | ||
| + FINAL_DISPLAY_OFFSET, |
There was a problem hiding this comment.
Handle final display scores below zero
For any future model whose IQM is <= -100, which is allowed because the normalized scores are explicitly unbounded, this + 100 transform produces a non-positive positive_display_score. The new HTML and PNG final leaderboards both treat this field as a bar width/value from a zero lower bound, so a valid very-low-scoring model would be clipped or omitted instead of showing its meaningful negative gap; use a plotting domain that can include negative display values or derive an offset that is guaranteed for the current cohort.
Useful? React with 👍 / 👎.
Summary
IQM + 100display index for the final plotmax/xhigh)Inference settings
azure/gpt-5.6-solmaxazure/gpt-5.5xhighanthropic/claude-opus-4-8via Claude OAuthmaxazure/gpt-5.4-minixhighThese are provider-specific categorical labels, not a shared numeric compute scale.
Validation
uv run pytest -q— 92 passed, 18 skippeduv run ruff check .uv builduv run python scripts/validate_tasks.py— all nine tasks valid