Phase 1: STAC sampler — real COG fetch + map-style datasets - #1
Merged
Conversation
Introduce the body-agnostic data layer foundation for the Phase 1 STAC sampler: EndpointError (pointing back at data/endpoints.py), the single STAC API root, and TargetGrid — the common geographic raster every layer is reprojected onto. Add rasterio and pystac-client as core dependencies.
read_window opens a Cloud Optimized GeoTIFF, reprojects the region covering a TargetGrid onto it via WarpedVRT (any source projection to the geographic target grid), applies band scale/offset for physical values, and returns a boolean validity mask for nodata and out-of-coverage pixels.
find_asset_hrefs queries the USGS ARD catalog by collection and bbox through a memoized pystac-client with exponential backoff on rate-limit and transient errors, returning one asset href per intersecting item. Failures surface as EndpointError.
WindowCache stores reprojected (image, mask) arrays under a hash of the layer id and target grid, writing atomically via a temp file and replace. Entries are re-fetchable and safe to delete; the cache is never load-bearing.
InstrumentDataset.read now samples a real ground window, searches the USGS ARD catalog for the covering COGs, reprojects and mosaics them onto a common geographic grid, and caches the result — replacing the Phase 0 synthetic tensors. KaguyaTC is backed by the kaguya_terrain_camera_usgs_dtms collection (dtm + orthoimage assets); LROCWAC (no STAC collection exists) is replaced by KaguyaTCImagery over the stereoscopic radiance observations. The catalog, registry, and exports follow. Note: LRO WAC has no STAC backing in this catalog; a later phase can add it from a non-STAC (COG/WMS) source.
Update AGENTS.md, README, and the docs site to the real per-instrument API (KaguyaTC + KaguyaTCImagery), mark Phase 0 done and Phase 1 current, switch the type-check command to ty, and note the deferred non-STAC LRO WAC source for a later phase. Replace the live-test placeholder with a real USGS ARD catalog reachability check and an end-to-end two-layer patch fetch.
Add a contributor guide grounded in the real uv/ruff/ty workflow and the unit-vs-live test policy, plus GitHub issue forms for bug reports, feature requests, and new dataset/body proposals (with a config pointing at Discussions and the guide).
Switch from IterableDataset to a map-style Dataset: each index deterministically samples one window, so moondata[i] works and is reproducible, len() is unchanged, and DataLoader shuffling, samplers, and random_split now work. This also fixes a latent bug where multi-worker DataLoader duplicated patches across workers. Document that patch_size sets the (C, patch_size, patch_size) output size.
TechnicToms
added
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 1: STAC sampler — real COG fetch + map-style datasets
Closes Phase 0 and implements the Phase 1 core:
InstrumentDataset.read(bbox)nowfetches real Cloud Optimized GeoTIFFs from the USGS ARD STAC catalog, reprojects
them onto a common geographic grid, applies scale/offset, mosaics overlapping
items, and caches the result — no more synthetic tensors.
New body-agnostic data layer (
src/astrofetch/data/)endpoints.py— the single external URL (STAC API root); every COG href is discovered at request time.errors.py—EndpointError, pointing back atdata/endpoints.py.grid.py—TargetGrid, the common IAU-2015 geographic raster all layers reproject onto.raster.py— windowed COG reads viaWarpedVRT(any source projection → target grid), scale/offset applied, boolean validity mask.stac.py— politepystac-clientsearch with exponential backoff.cache.py— disposable on-disk window cache, never load-bearing.Moon datasets
KaguyaTCnow reads thekaguya_terrain_camera_usgs_dtmscollection (dtm+orthoimageassets).LROCWAC→KaguyaTCImagery. The USGS catalog has no LRO WAC collection, so the fictional class is replaced by real Kaguya TC stereoscopic radiance imagery. Adding LRO WAC from a non-STAC source is noted for a later phase.IterableDatasetto a map-styleDataset:moondata[i]works and is reproducible,len()unchanged, andDataLoader(shuffle=True), samplers, andrandom_splitnow work. Also fixes a latent multi-worker duplication bug.Docs & project
CONTRIBUTING.md+ GitHub issue forms (bug / feature / new dataset).ty.rasterio,pystac-client.Testing
ruff,ruff format --check,ty check,pytest,mkdocs build --strict.Notes
rasterio'sWarpedVRT.reademits a NumPy 2.5DeprecationWarning(upstream, harmless).Files: 32 changed, +2166 / −145 (11 commits).