Virtualising an AODN/IMOS satellite SST archive (follow-up from #218)

[utas-amberbian]

Following up from #218: details of a good, clean example of one of our satellite SST archives, and two questions.

The data

• ~12,000 NetCDF files on public S3 (anonymous access), ~26 MB each, ~300 GB total, one file per day 1992–2025
• s3://imos-data/IMOS/SRS/SST/ghrsst/L3S-1d/dn/{YYYY}/...
• Fixed 4500 × 6000 grid (Australian region), ~19 variables, gzip+shuffle, chunks (1, 1500, 2000) / (1, 2250, 3000)

What we found

• Structure is stable. Sampled 1995 / 2005 / 2015 / 2024: grid, chunks, dtypes and compression are identical. open_virtual_dataset (v2.7.1, HDFParser) and concat along time work fine.
• Blocker: per-file scale/offset packing. Each file has its own conversion formula (value = int × scale + offset). Over three consecutive days, SST's add_offset moves 292.17 → 288.44 → 288.75 K; some variables change scale_factor too. 11 of 14 gridded variables are affected, so concatenated data is silently wrong by up to ~3.7 K. We know this is xr.concat of virtual datasets silently drops mismatched CF encoding (scale_factor, add_offset, _FillValue) #1004 — mentioning it only because this archive is a clean public reproducer.
• Minor: variable set drifts across eras (19 variables common to all years). We'd virtualise the intersection.

Why we want this

We already keep two copies of this collection: the NetCDF archive (chunked (1, 1500, 2000) — good for daily whole-map reads) and a rechunked, unpacked Zarr copy (s3://aodn-cloud-optimised/satellite_ghrsst_l3s_1day_daynighttime_single_sensor_australia.zarr, chunked (5, 500, 500) — good for time series). A virtual Zarr over the archive would add a native-chunking view without storing a third copy of the bytes.

Questions

1. Given per-file packing, is the realistic path to re-encode the packed variables into a materialised store (e.g. Icechunk) and keep virtual references only for the uniformly-encoded ones — or is there a better pattern?
2. Any gotchas you've seen with this "archive + rechunked copy + virtual native-chunk view" setup?

The archive is public and stable — happy for it to serve as a test case.

cc @TomNicholas (following your suggestion in #218)

[TomNicholas]

Given per-file packing, is the realistic path to re-encode the packed variables into a materialised store (e.g. Icechunk) and keep virtual references only for the uniformly-encoded ones — or is there a better pattern?

Yeah, sorry. Virtualization requires the codecs to be homogenous across arrays, there is no workaround. The closest thing to a workaround is to split the arrays up. tThat's what I had to do with GOES-16 - in that case the scale and offset changed but there were still a few distinct contiguous "eras" of homogenous encoding, so I put each encoding in a separate Zarr group in the same store. A bit gross but the best I could do.

Any gotchas you've seen with this "archive + rechunked copy + virtual native-chunk view" setup?

No that should be fine.

variable set drifts across eras

You could also deal with this with groups, or you could pad the variables with nans for the times they are not available. But you do need a homogenous schema within a group.

[utas-amberbian]

Thanks, Tom — that's clear. Our packing changes per file rather than in eras, so grouping isn't viable here (~12,000 groups). Good to have a definitive answer; happy for this to stay open as a usage example.