Improve test data generation and download progress#191
Improve test data generation and download progress#191Sprengmeister-dev wants to merge 1 commit intobio-learn:masterfrom
Conversation
|
I can see you're trying to solve real pain points around download feedback and robustness. Did you run into this issue yourself or did someone you know experience this? some concerns:
I'd suggest instead: Split this into smaller PRs:
|
|
Could you elaborate more on what problem you're trying to solve with these changes to cached download? I am concerned about the added complexity and we don't have any testing around it to cover any possible failure modes. |
|
Closing for now pending more information |
Summary
This PR improves test data generation by downloading the GEO series matrix directly with progress output and more robust error handling. It also upgrades the download helper to support resumable downloads, progress reporting, and forced re-downloads.
Changes
cached_downloadwith progress output in the test data generator.DataSource.cached_downloadwith.partfiles, HTTP range resume, and optional progress output.