fix(swebench): sample the pilot across each repo's id range, not its head

[eranra]

The pilot's purpose is to predict a 250-task full run from 50 tasks. The first version couldn't: it allocated per repo correctly, then took ids[:n] — the head of each repo's slice. The full tier is sorted by instance id, and SWE-bench ids track upstream PR numbers, so it drew almost entirely from the oldest issues.

Measured on a 0=first / 1=last scale, the picks' mean relative position was 0.08:

repo	full id-range	pilot BEFORE	pilot AFTER
django	9296–17084	10554–11880	9296–16819
sympy	11618–24443	11618–13647	11618–24443
scikit-learn	10844–26194	10844–12682	10844–26194
matplotlib	13989–26342	13989–21568	13989–26342

mean relative position 0.08 → 0.50

I had called that selection "stratified" without qualifying it, which overstated it — proportional across repos, but systematically age-skewed within them.

Picks are now spread evenly across each repo's ordering. Still fully deterministic (no RNG), so the file stays reproducible and reviewable in a diff.

Also

Promotes the generator from a throwaway to ci/benchmarks/swebench/utils/make_pilot.py, beside the existing select_candidates.py. It asserts its own invariants: exact size, unique ids, every id present in the full tier.

Deliberately unchanged

The floor of 1 per repo, which tilts django to ~38% against full's ~46%. That's the price of guaranteeing no repo's build/test toolchain can hide from the pilot.

Scope of the fix

This fixes representativeness of the sample, not of the statistic. At n=50, p≈0.8 the 95% interval is ~±0.11 versus ~±0.05 at n=250. The pilot predicts cost and runtime well — tasks are independent, so both scale linearly — but gives only a ballpark for reward.

🤖 Generated with Claude Code

[skillberry-bot]

❌ Automatic Labeling Failed

An error occurred while trying to automatically label this pull request. Please check the workflow logs for details and add labels manually.