feat: wider PDS ODE dataset roster (Phase 2) - #2
Merged
Conversation
The USGS ARD STAC catalog has no LROC, LOLA, or other lunar collections beyond Kaguya TC. Add the PDS Orbital Data Explorer API root plus the LRO WAC mosaic and LOLA/SLDEM2015 DEM URLs as the new external endpoints.
Mirrors data/stac.py: a single retrying, backed-off HTTP session, with failures normalized into EndpointError. Handles ODE's JSON quirks (a lone match comes back as a dict instead of a list, an empty result is the string "No Products Found", errors are HTTP 200 responses with an error message in the body) and converts the internal -180..180 longitude convention to ODE's 0-360 westernlon/easternlon at the query boundary. requests becomes a direct dependency (previously only pulled in transitively via pystac-client).
read_window gains an optional nodata_override, for the rare PDS product that omits a nodata value from its label even though its raster does not cover its full requested extent (unwarped pixels would otherwise read back as valid zeros). Add read_full: reads a raster's own native pixels with no reprojection onto a TargetGrid, for sources with no map projection to warp to (raw camera-frame swaths). Used by the upcoming raw-granule datasets.
Split the shared product-dataset machinery (validation, the read loop, caching) out of InstrumentDataset into a new _ProductDataset base, so it can back three source backends instead of one: - InstrumentDataset: USGS ARD STAC search (unchanged behavior) - ODEInstrumentDataset: PDS Orbital Data Explorer search, for instruments the STAC catalog doesn't carry. Adds footprint_sampling for instruments that cover only a handful of named sites rather than the whole Moon, drawing windows from inside real product footprints instead of uniformly over the bbox. - MosaicDataset: reads one well-known archive URL directly, for instruments published as a single global file. Ships four new datasets: LROCNACDTM (LRO LROC NAC stereo DTM sites -- elevation, orthoimage, pixel confidence; the color-coded slope/shade SDP products are rendered visualizations, not quantitative rasters, so are intentionally not offered), LROCWACMosaic (LRO WAC 100m global mosaic), LOLA (global gridded DEM), and SLDEM2015 (LOLA + Kaguya TC DEM). The NAC DTM file patterns target each product's data file (.TIF/.IMG) directly rather than its detached PDS4 .xml label: GDAL's PDS4 driver mis-parses the resolution unit in these labels into a near-zero pixel size, while opening the data file directly reads correct georeferencing, scale, and nodata (verified live against the Apollo 15 site).
A deliberately different, documented contract from the windowed instrument datasets: no bbox windowing, no reprojection, no ISIS/SPICE -- each item is one raw PDS product read in its own native camera/instrument geometry. GranuleDataset does one eager ODE search in __init__ (so len() needs no network call) and guards against reading a whole gigapixel strip by default (max_pixels raises with a rows= hint instead). Ships LROCNACRaw and LROCWACRaw (calibrated NAC/WAC strips) and M3 (Chandrayaan-1 Moon Mineralogy Mapper L1B radiance, with its lon/lat/ elevation geolocation backplane via extra_patterns). M3's ENVI-format data file must be opened directly -- unlike the PDS-labeled NAC/WAC strips, its driver rejects the .HDR header file and wants the .IMG data file itself.
LayerSpec grows a source discriminator ("stac", "ode", or "mosaic") plus
the fields each backend needs, dispatched in _spec by asset spec type.
Probe grows a granules field for the experimental raw-granule dataset
classes, which sit outside the layer/product contract entirely and are
not part of LAYERS.
Adds an "lro" probe (nac_dtm, wac_mosaic, lola, sldem2015 instruments;
nac_raw, wac_raw granules) and a "chandrayaan1" probe (m3 granule only).
LROCNACDTM, LROCWACMosaic, LOLA, SLDEM2015, ODEInstrumentDataset, MosaicDataset, and the granule classes (GranuleDataset, LROCNACRaw, LROCWACRaw, M3) become part of the public API in both astrofetch.moon and the top-level astrofetch namespace.
Recorded JSON fixtures under tests/fixtures/ode/ cover the response quirks the client normalizes: a single match as a dict instead of a list, an empty result, and an error body. Also covers longitude conversion (including the full-Moon bbox that would otherwise collapse to a zero-width query), pagination, and match_files/find_file_urls filtering. The HTTP session is stubbed; no network in this suite.
Extends the existing mocked-network fixture to also stub ode.find_file_urls and ode.query_products. New coverage: the ODE and mosaic sample contracts, that MosaicDataset never searches ODE, footprint-sampling determinism and containment, its fallback to uniform sampling with no footprints, and registry/catalog agreement for the new source types.
Local GeoTIFFs plus a stubbed ode.query_products, no network. Covers length, sample keys, extra_patterns backplanes, negative/out-of-range indexing, the missing-file error, rows= partial reads, and the max_pixels guard.
One targeted test per source against real archives: NAC DTM (PDS ODE search plus footprint sampling), the WAC mosaic, LOLA, SLDEM2015, and the NAC-raw/M3 granule paths. Deselected by default; run manually with pytest tests/live -m live. Each also serves as the live verification for that dataset's filename-pattern regexes.
Add reference/granules.md (the experimental contract, size guidance) and register it in the mkdocs nav. reference/data.md and datasets.md gain an intro distinguishing the three sources (STAC/ODE/fixed mosaic) and entries for the new classes. index.md gets a quickstart example beyond the STAC catalog. roadmap.md reflects Phase 1 as complete and Phase 2's new-source work as delivered, with what's still open called out.
Architecture tree and design rules now reflect data/ode.py and moon/granules.py. Amends the "no PDS granule access" non-goal to describe what actually shipped (raw, camera-geometry only, experimental, excluded from the quantitative registry) rather than leaving it contradicted by the code. Updates the roadmap's current-phase banner and Phase 2 section to separate delivered work from what's still open.
Every current ODE-backed dataset reads a file typed "Product", but some archives (e.g. ShadowCam's DTM confidence maps) ship the file we need under a different ODE role such as "Referenced". Add a file_type field to ODEAsset (default "Product", so all existing datasets are unaffected) and thread it through ODEInstrumentDataset._hrefs and LayerSpec/_spec so the registry stays consistent with what each dataset actually reads.
Search PDS ODE (product type MOSDDR) for the LRO Mini-RF global mosaics: circular polarization ratio and same-/opposite-sense circular received power, 128 px/degree, detached PDS3 label (same read path as LOLA). The label declares MISSING_CONSTANT as a float64 sentinel inside a float32 band; GDAL's overflowing cast leaves src.nodata unset, so out-of-coverage pixels silently read back as a large-negative overflow artifact marked valid. Verified live by reading raw pixels and pinning the exact bit pattern as a nodata_override, the same class of archive quirk as the LROCNACDTM PDS4-label bug.
Add ODE's productid wildcard filter to query_products/find_file_urls (and a matching ODEAsset.product_id/LayerSpec.product_id field), for product types where a bbox-only search buries the products actually wanted among thousands of unrelated ones and no reasonable max_products cap would ever reach them. DivinerGDR needed this immediately: LRO Diviner rock abundance and regolith temperature are mission-cumulative global mosaics republished periodically under product type GDR_L3, which also carries per-orbit bolometric temperature outnumbering every other parameter combined. productid narrows the search to the specific dated product wanted (the most complete date verified live, 2016-09-13) instead of paging through the whole product type. TBOL itself is intentionally not offered, since mosaicking single-orbit epochs would misrepresent the data.
Search PDS ODE (product types SDWDTM and SDWTIO) for the LROC WAC GLD100 global 100 m DTM and the WAC TiO2 abundance map. GLD100's product type is dominated by WAC_CSHADE, a rendered shaded-relief product (excluded per rule 3) that would otherwise swamp a bbox-only search, so it uses the same productid narrowing added for DivinerGDR. Both open their .IMG data file directly rather than the PDS4 .xml label, following the LROCNACDTM precedent for this im-ldi archive.
Search PDS ODE for two more LROC WAC products: the global morphology mosaic as 8 searched E-family quadrant tiles (product type BDRWGL, the tiled sibling of the fixed-URL LROCWACMosaic -- the same type also carries a whole-globe O-prefixed family, not offered here), and the empirically-normalized 7-color reflectance (product type MDREMP), one product per band pinned to its 64 px/degree tiling. Both open their .IMG data file directly, following the LROCNACDTM precedent for the im-ldi archive.
Search PDS ODE (product type BDRROI) for LROC NAC region-of-interest mosaics: named sites such as craters and poles, footprint-sampled like LROCNACDTM since coverage is a few hundred sites, not the whole Moon. Native-resolution mosaics can reach ~14 GB, far too large for windowed reads, so only the downsampled 5 m and 20 m products are offered. The same product type also carries unrelated WAC_ROI mosaics from the WAC camera; productid narrows the ODE search to the NAC family.
Search PDS ODE (product types CMOS and DTM) for KPLO ShadowCam controlled mosaics and stereo DTMs of permanently shadowed polar regions. Genuine Cloud Optimized GeoTIFFs, no PDS4-label quirk to work around. footprint_sampling is on, matching the other named-site instruments: coverage is a handful of PSR sites, not the whole Moon. The DTM product type also carries rendered slope/shaded-relief/color visualizations (excluded per rule 3); confidence is the one non-elevation product offered and is typed "Referenced" rather than "Product" in ODE (verified live 2026-07-21), exercising the file_type override added earlier in this phase. New kplo probe in the catalog, since this is the first Korea Pathfinder Lunar Orbiter dataset.
… probe Search PDS ODE (product type MDIM) for the Clementine UVVIS 5-band and NIR 6-band basemaps, one product per band indexing the same per-tile file. Sinusoidal-projected, attached PDS3 label -- the first attached-label source in this project, so its georeferencing was verified live before shipping: opened a tile directly and compared its bounds against the product's own ODE footprint, which agreed to within expected rounding (unlike the LROCNACDTM PDS4-label case, this archive's driver checked out). New clementine probe in the catalog, since these are the first Clementine datasets.
Add reference pages for the ten new datasets (MiniRF, DivinerGDR, WACGLD100, WACTiO2, LROCWACGlobal, LROCWACColor, LROCNACROI, ShadowCam, ClementineUVVIS, ClementineNIR), a ShadowCam quickstart example, and an updated Phase 2 roadmap summary. AGENTS.md: record the ODEAsset.product_id search-narrowing pattern as a domain note, list the roster items deliberately dropped with their reasons (Kaguya MI's host ignores Range headers, Kaguya LALT superseded by SLDEM2015, Diviner TBOL mixes epochs, and several redundant/ follow-up products), and add the Mini-RF nodata-overflow-artifact and Clementine attached-label-verification live findings to risks.
TechnicToms
added
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ODEInstrumentDatasetclasses backed by the NASA PDS Orbital Data Explorer:MiniRF,DivinerGDR,WACGLD100,WACTiO2,LROCWACGlobal,LROCWACColor,LROCNACROI,ShadowCam,ClementineUVVIS,ClementineNIRMOONcatalog (kplo,clementine) and extendlrowith the new instrumentsproduct_id(ODE wildcard) search narrowing todata/ode.py, needed to find dated/pinned products buried in large per-ptresult setsfile_typesupport toODEAsset/LayerSpecfor non-ProductODE file roles (e.g. ShadowCam'sReferencedconfidence rasters)AGENTS.mdanddocs/Test plan
uv run pytest— full unit suite greenuv run pytest tests/live -m live— full live suite (18 tests) passes against real PDS/USGS/JAXA serversuv run ruff format && uv run ruff checkuv run ty checkuv run mkdocs build --strict