ci: check the sdist can actually be consumed

[d-v-b]

🤖 AI text below 🤖

Split out of the sdist allowlist PR (#289 / zarr-developers#4261) so the packaging change and the CI change can be reviewed separately.

Stacked on claude/zarr-sdist-allowlist-414698, not main — check_sdist_contents.py asserts packages/ is absent from the sdist, which is only true once the allowlist lands. Based on main this PR would be red by construction. Merge the allowlist first; the diff here is CI-only either way.

Why

The allowlist is the safe default — a new subpackage under packages/ stays out of the zarr sdist unless someone opts it in — but it inverts the failure mode. A blocklist ships too much silently; an allowlist can ship too little, by adding a file consumers need without a matching entry.

Test distribution artifacts previously ran ls dist and asserted nothing. Two checks replace it, split by cost.

test_dist_pypi — seconds, runs on PRs

ci/check_sdist_contents.py asserts:

• every testpaths entry from pyproject.toml ships
• packages/ does not
• the paths packagers read but no test opens: LICENSE.txt, PKG-INFO, py.typed, and the hatch-vcs _version.py

The testpaths assertion is the one that earns its keep — it's derived from config maintained for other reasons, not restated, so it can't rot into a second allowlist to forget. Pointed at the pre-allowlist sdist it reports both defects that were actually live:

testpaths entries missing from the sdist
  docs/user-guide
  `pytest` on an unpacked sdist will fail to collect these.
Paths that must not ship in the zarr sdist
  packages/zarr-http-server/... (370 total)

test_dist_conda — ~10 min, not on PRs

Builds the sdist with the real conda-forge recipe, fetched from the feedstock at run time and repointed at our freshly built tarball by ci/conda_recipe_for_sdist.py. conda-forge is the consumer most exposed to a missing file: it builds from the sdist rather than the wheel, and runs our suite from a directory holding only what its recipe copies out of the tarball.

Fetching beats reproducing. A vendored copy of that recipe would go stale the moment the feedstock changed, and silently — the same class of bug this job exists to catch. Only the fields naming which tarball to build are rewritten; requirements, tests, pip_check and license_file execute as conda-forge wrote them.

Worth knowing: the recipe's requirements are already regenerated from our pyproject.toml by grayskull on each version bump (bot: inspection: update-grayskull in the feedstock's conda-forge.yml) — the 3.3.0 bump moved typing_extensions >=4.13 → >=4.14 with no human involved. The tests: block is hand-maintained there, which is why running it beats reproducing it.

Why it's skipped on pull_request

It runs the whole suite from an unpacked sdist. Measured on CI: 591.95s of a 613s job — conda's build and env solve are ~22 seconds of that. The cost is inherent to testing an sdist, so there's no cheaper variant of the same coverage.

On push to main it still catches a bad merge within ~10 minutes, attributed to one commit, and it gates upload_pypi, so no release can publish an sdist conda cannot consume. workflow_dispatch runs it against a branch on demand. Keeping it off PRs also stops a recipe change in the feedstock from reddening unrelated PRs.

Release gating is unchanged in effect: on a release event neither job is skipped, so upload_pypi still requires both.

One trap worth flagging

The generated recipe uses a url: source, not path:. path: sources honour .gitignore, and src/zarr/_version.py is gitignored (.gitignore:69). It's written into the sdist by the hatch-vcs hook precisely because an unpacked sdist has no git history to derive a version from — so a path: source silently drops it and the build fails. Noted in a comment in the script.

Verification

Ran on Linux CI before the trigger was narrowed, with the recipe fetched live:

✔ python imports test passed!
✔ pip check passed!
6884 passed, 475 skipped, 4 xfailed in 591.95s
✔ all tests passed!

No changes/ fragment: CI-only, nothing user-visible for the release notes.