[Proposed Feature] Demonstrate Icechunk virtualization of operational, non-cloud-native data sources

[tylere]

Summary

Source Cooperative is well-positioned to demonstrate a powerful emerging pattern: using
Icechunk to provide cloud-native Zarr access to operational data sources that are
constantly updating and published in non-cloud-native formats — without duplicating the
underlying data.

NOAA's RRFS (Rapid Refresh Forecast System), published as GRIB2 to s3://noaa-rrfs-pds
via the NODD program, is an ideal candidate for a reference implementation of this pattern.

The Pattern

Many high-value scientific and operational datasets are:

• Published in legacy formats (GRIB2, NetCDF, HDF5)
• Constantly updating (hourly, daily forecast cycles)
• Already on S3 or other object storage, but not cloud-native

Icechunk virtual stores allow Source Cooperative to publish a reference metadata layer
that maps Zarr chunks to byte ranges in the source files — giving users a native
xarray.open_zarr() interface with no data duplication, while the source bytes stay where
they are.

Each new data cycle becomes a new Icechunk commit, enabling:

• Transactional updates: new forecast cycles appended atomically
• Versioning: users can pin to or replay any historical snapshot
• Zero storage overhead: only chunk manifests are stored on Source Coop

Why This Matters

This positions Source Cooperative not just as a data host, but as an access layer
for the broader scientific data ecosystem — adding cloud-native interoperability to data
that already exists publicly, but isn't practically usable in modern workflows.

This pattern could apply broadly:

• NOAA NWP model output (RRFS, GFS, HRRR, NAM)
• NASA/ESA satellite L1/L2 products in HDF5
• Copernicus Climate Data Store outputs
• Any operational dataset on S3 in a non-Zarr format

Reference Implementation: NOAA RRFS

As a concrete demonstration, we propose building a VirtualiZarr/Icechunk store over
RRFS GRIB2 output, scoped to:

• CONUS domain
• Key surface variables (2m temp, 10m wind, precip, CAPE, reflectivity)
• 0–24h forecast horizon
• Rolling append as new forecast cycles are published

This gives us a working, publicly visible example to document, present at CNG, and use
as a template for onboarding other datasets.

Open Questions

• Who runs the manifest generation pipeline? (Source Coop infrastructure, NOAA
  partnership, or community contributor?)
• Update cadence — RRFS runs hourly; is near-real-time append feasible or do we
  start with a daily batch?
• Does NODD egress coverage extend to chunk reads by end users, or only to the
  manifest download?
• What's the right interface for contributors to register new virtual datasets of
  this type on Source Coop?

Alternatives Considered

Kerchunk JSON/Parquet reference files: mature tooling, simpler to generate, but no
transactional update model — each new forecast cycle produces a new file rather than an
append. No versioning or snapshot pinning. Icechunk supersedes this approach for
operational, continuously updating data.

Related Work

• Icechunk
• VirtualiZarr
• herbie — HRRR/NWP access with kerchunk support
• Pangeo / Kerchunk
• NODD program: s3://noaa-rrfs-pds

Who Would Benefit

• Weather/climate researchers using Python/Xarray
• ML practitioners

[jedsundwall]

I like this a lot. My inclination is to avoid having Source generate manifests / run pipelines. The Source way of looking at this is to see the manifest as part of a data product and to be agnostic about how it's created before being published to Source. I imagine we could work with dynamical.org to show how this would work.

[jedsundwall]

This also brings up another case of how we should let data publishers tell us more about their data products. E.g. how do they tell us that they have a STAC catalog or an Icechunk manifest? This is probably some kind of config to be set in the product settings.

[tylere]

I agree that this type of work (generating data for Source) should be implemented outside of this repo. I just didn't think of a better place to store the idea.

[tylere]

This also brings up another case of how we should let data publishers tell us more about their data products. E.g. how do they tell us that they have a STAC catalog or an Icechunk manifest? This is probably some kind of config to be set in the product settings.

I have been thinking that the Source product metadata should contain explicit fields for:

• a link to external product webpage (e.g. https://dynamical.org/catalog/noaa-mrms-conus-analysis-hourly/)
• a link to a data license file (i.e. encouraging users to explicitly declare a license, preferably a standard, well known license)
• a link for dataset discussion (e.g. mailto:isaac.corley@taylorgeospatial.org or https://github.com/dynamical-org/reformatters/issues)

Some users are already putting this type of information in the product README, but often it is missing.

Requiring the entries to be a valid URLs is less flexible, but would encourage users to plan out these features, and it would be easier for us to work with programmatically.