Skip to content

feat(spaceship-titanic): add evaluator#32

Open
skyzer wants to merge 1 commit into
hexo-ai:mainfrom
skyzer:feat/spaceship-titanic-evaluator
Open

feat(spaceship-titanic): add evaluator#32
skyzer wants to merge 1 commit into
hexo-ai:mainfrom
skyzer:feat/spaceship-titanic-evaluator

Conversation

@skyzer

@skyzer skyzer commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a bundled evaluator for the spaceship-titanic task
  • score Kaggle-style PassengerId,Transported CSV submissions against private labels
  • write the canonical results.json artifact with accuracy, counts, extras, and per-passenger details
  • add regression tests for the evaluator contract and missing/invalid/extra predictions

Why

spaceship-titanic ships with public train/test data, a sample submission, private labels, and a reference agent, but no evaluate.py. That means SIA can run the task but has no first-party evaluator to feed scores back into the generation loop.

Test plan

  • python3 -m pytest tests/test_spaceship_titanic_evaluator.py -q
  • local CLI smoke test against bundled sample_submission.csv
  • python3 -m ruff check .
  • python3 -m pytest -q

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant