test: pin the n=1 accept path for the MMLU/GSM8K slice loaders (no network)

[Nitjsefnie]

Follow-up to #12, taking you up on the suggestion in its merge note:

an accept-path test (n=1 against a pre-seeded tmp cache_dir) would pin the happy path without needing network — nice complement to the reject cases here.

What this adds

Two tests in tests/test_metrics.py, placed directly after the existing reject-case tests and following their conventions:

• test_load_mmlu_slice_accepts_n_equal_1
• test_load_gsm8k_slice_accepts_n_equal_1

Each seeds a pytest tmp_path cache dir with the exact JSON cache file the loader looks for (mmlu_1_0.json / gsm8k_1_0.json), calls the real loader with n=1, and asserts it returns exactly one item of the right dataclass with the promised field shapes (MMLU: str question, list[str] choices, int answer; GSM8K: str question/answer).

No network, verified rather than assumed

Both tests monkeypatch socket.socket to raise, so a network attempt fails loudly instead of silently passing on a machine that happens to be online. They were additionally run under a sitecustomize.py that blocks socket.socket process-wide — 2 passed. Because the cache hits, the gated datasets import is never reached, so the tests also work without the data extra installed.

The tests bite

Mutating the loaders so their cache-hit path yields no rows makes both new tests fail (assert 0 == 1); restoring makes them pass. So they pin the accept path rather than just decorating it.

Gates

ruff check src tests → 0 · mypy src → 0 · pytest → 0 (56 passed, 2 skipped). No production code changed; the diff is one test file (+57).

One thing I noticed while writing these (not fixed here)

On a cache hit both loaders return every row in the cache file, without checking the row count against n (src/steerbench/metrics.py:578 and :619). n is part of the cache key, so this is invisible in normal use — but a hand-edited or stale cache file would make load_*_slice(n=1) return however many rows the file holds, silently violating the contract these very tests assert. I kept it out of scope since the merge note asked only for the accept path; happy to send a follow-up that slices to n (or validates and re-fetches) if you want it — just say which behavior you would prefer.

---

Implemented with AI assistance (Kimi K2.7 Code), reviewed, and independently mutation-verified by me before submitting.

[bamdadd]

Independently reproduced your mutation check here — dropping the cache-hit rows makes both new tests fail with assert 0 == 1, so they genuinely pin the accept path. ruff/mypy/pytest all green locally (network-blocked via the socket monkeypatch, cache hit means datasets is never imported). Clean complement to the reject cases in #12 — merging.

Good catch on the cache-hit-not-sliced-to-n contract gap; that's a real latent bug, not out of scope noise. Filed it as a good-first-issue: #14 — and yes please, I'd take the follow-up that slices to n on the cache-hit path (with a too-many-rows fixture). Thanks Peter.