Skip to content

fix: validate n >= 1 in the MMLU/GSM8K slice loaders (fixes #8)#12

Merged
bamdadd merged 1 commit into
bamdadd:mainfrom
Nitjsefnie-OSC:fix-8-slice-size-validation
Jul 17, 2026
Merged

fix: validate n >= 1 in the MMLU/GSM8K slice loaders (fixes #8)#12
bamdadd merged 1 commit into
bamdadd:mainfrom
Nitjsefnie-OSC:fix-8-slice-size-validation

Conversation

@Nitjsefnie

Copy link
Copy Markdown
Contributor

Fixes #8.

Both load_mmlu_slice and load_gsm8k_slice now raise ValueError("n must be >= 1, got {n}") at the very top — before _cache_path(...) (which mkdirs the cache root) and before the function-local from datasets import load_dataset — so the guard is reachable with no download, no network, and no side effects, per the issue's ordering requirement.

Tests: four parametrized cases in tests/test_metrics.py (n=0 and n=-3 for each loader), pytest.raises(ValueError), no existing test touched.

Verification

  • ruff check ., ruff format --check ., mypy src, pytest -q all green (56 tests, 2 pre-existing unrelated skips — torch/matplotlib extras).
  • Mutation-proven twice (implementer, then independently re-run before opening this PR): removing only the mmlu guard makes exactly the two mmlu tests fail — and the failure mode is itself informative: ModuleNotFoundError: No module named 'datasets' at metrics.py:584, i.e. without the guard the call sails past the cache check into the gated import, which is precisely the no-network property the guard exists to protect. gsm8k tests unaffected by the mmlu mutation; restore → all green.

One boundary note: the guards reject n < 1 but deliberately not non-int types (out of #8's stated scope), and score_mmlu([], []) still returns 0.0 — the issue fixes the misleading-0% at the loader boundary, which is what this does.


🤖 This PR was written with AI assistance (implementation: Kimi K3; verification and review: Claude), directed by a human-supervised workflow.

n=0 or n<0 previously yielded an empty slice that scored as a
misleading 0% accuracy. Both loaders now raise a clear ValueError
at the top of the function, before the cache check and before the
gated datasets import, so the guard needs no download or network.

Co-Authored-By: Kimi K3 <noreply@kimi.com>
@bamdadd

bamdadd commented Jul 17, 2026

Copy link
Copy Markdown
Owner

Thanks Peter — verified locally: ruff, mypy, and the full metrics suite all green, and the guard fires before any cache/datasets access as your test asserts. The ValueError("n must be >= 1, got {n}") reads cleanly and doesn't touch the valid-n path, so existing behavior is unchanged. Merging.

One small follow-up if you're up for it: an accept-path test (n=1 against a pre-seeded tmp cache_dir) would pin the happy path without needing network — nice complement to the reject cases here. Not blocking. Appreciate the fix!

@bamdadd
bamdadd merged commit 9ff0018 into bamdadd:main Jul 17, 2026
1 check passed
bamdadd pushed a commit that referenced this pull request Jul 17, 2026
Add accept-path tests for load_mmlu_slice / load_gsm8k_slice: seed a tmp cache_dir with the exact JSON file each loader looks for, call with n=1, and assert one item of the right dataclass with the promised field shapes. Network is blocked via a socket monkeypatch so a cache miss fails loudly rather than downloading; the cache hit means the gated datasets import is never reached. Complements the reject-path tests from #12.

Follow-up to #12.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Validate slice size (n >= 1) in the MMLU / GSM8K side-effect loaders

2 participants