Add article-derived nine-task experiment suite#15
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4860523799
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if spec is None or spec.loader is None: | ||
| raise RuntimeError(f"Unable to import policy from {policy_path}") | ||
| module = importlib.util.module_from_spec(spec) | ||
| spec.loader.exec_module(module) |
There was a problem hiding this comment.
When a submission defines @dataclass classes with from __future__ import annotations, dataclasses looks up the defining module in sys.modules during class creation. This loader executes the spec without first inserting the module, so otherwise valid single-file policies fail during import with an AttributeError before any action is evaluated; the same copied pattern appears in several of the new evaluators. Please mirror the Ant/Pong loader by registering the module around exec_module.
Useful? React with 👍 / 👎.
1 participant
Summary
Impact
GenesisBench can now evaluate the complete article-derived suite rather than only the Ant experiment. The committed leaderboard is reproducible from the checked-in per-model score artifacts.
Validation
uv run ruff check .uv run pytest -q— 90 passed, 18 skippeduv build