ci: cache the pooch example datasets on the e2e jobs - #128
Merged
Conversation
Three slow-slice specs load pyxem's sped_ag; every uncached runner re-downloaded it from Zenodo per job, and a stalled download eats the spec's whole budget (the likely mechanism behind the sped_ag_grid job that hung to the kill with no diagnostics). actions/cache on ~/.cache/ pyxem, lock-keyed with a restore-keys fallback -- the pattern the hyperspy org's shared pooch-cache workflows use; pooch checksums verify integrity on every load either way.
CSSFrancis
marked this pull request as ready for review
August 9, 2026 00:57
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three slow-slice e2e specs (
sped_ag_grid,progressive_signal_preview,progressive_orientation_preview) load pyxem's sped_ag, which pooch downloads from Zenodo on first use — true "first run then cached" locally, but every CI runner is a first run, so each of those jobs re-fetched it every time. Beyond the wasted minutes, a stalled Zenodo download eats the spec's entire budget, which is the most plausible mechanism behind the earliersped_ag_gridjob that hung to the job kill with zero diagnostics.One
actions/cachestep on~/.cache/pyxem(the pooch os_cache on the ubuntu runners), keyed onuv.lockwith arestore-keysprefix fallback so it stays warm across unrelated dependency bumps and re-saves on a pyxem bump. Integrity is pooch's job — checksums verify on every load regardless of where the file came from. Same approach as the hyperspy org's shared pooch-cache workflows (hyperspy/.github), minus the registry-prefetch machinery we don't need for a single in-package fetch.The python test jobs are untouched: the migrated suite deliberately uses the bundled synthetic datasets and downloads nothing.