On a cache hit, load_mmlu_slice / load_gsm8k_slice return every row in the cache file without checking the count against n (src/steerbench/metrics.py:578 and :619). Because n is part of the cache key this is invisible in normal use, but a hand-edited or stale cache file makes load_*_slice(n=1) return however many rows the file holds — silently violating the size contract.
Fix: on the cache-hit path, slice the loaded rows to n (and/or validate the count and re-fetch on mismatch). The accept-path tests added in #13 already assert the n=1 shape and would extend naturally to a too-many-rows fixture.
Surfaced by @Nitjsefnie while writing #13.
On a cache hit,
load_mmlu_slice/load_gsm8k_slicereturn every row in the cache file without checking the count againstn(src/steerbench/metrics.py:578 and :619). Becausenis part of the cache key this is invisible in normal use, but a hand-edited or stale cache file makesload_*_slice(n=1)return however many rows the file holds — silently violating the size contract.Fix: on the cache-hit path, slice the loaded rows to
n(and/or validate the count and re-fetch on mismatch). The accept-path tests added in #13 already assert the n=1 shape and would extend naturally to a too-many-rows fixture.Surfaced by @Nitjsefnie while writing #13.