From 7e8c63b8b06f8aa23bee37d11dd56ba973d8c3f5 Mon Sep 17 00:00:00 2001 From: TomNicholas Date: Wed, 15 Jul 2026 14:36:04 -0700 Subject: [PATCH 1/3] Update Kerchunk comparison FAQ for sub-chunk isel parity The "Splitting uncompressed data into chunks" row linked PR #199, which was closed without merging. Sub-chunk slicing landed instead via #996: slicing an uncompressed ManifestArray along its largest-stride axis rewrites the reference's byte offset/length. A finer chunk grid along that axis can be built by combining isel with xarray.concat, so no dedicated rechunk method is needed. Also note that chunk selection now covers sub-chunk slicing of uncompressed arrays, not just chunk-aligned selections. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/explanation/faq.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/explanation/faq.md b/docs/explanation/faq.md index 20149eafd..4e1006320 100644 --- a/docs/explanation/faq.md +++ b/docs/explanation/faq.md @@ -231,8 +231,8 @@ Users of Kerchunk may find the following comparison table useful, which shows wh | Renaming variables | ❌ | `xarray.Dataset.rename_vars` | | Renaming dimensions | ❌ | `xarray.Dataset.rename_dims` | | Renaming manifest file paths | `kerchunk.utils.rename_target` | `vds.vz.rename_paths` | -| Splitting uncompressed data into chunks | `kerchunk.utils.subchunk` | `xarray.Dataset.chunk` (❌ Not yet implemented - see [PR #199](https://github.com/zarr-developers/VirtualiZarr/pull/199)) -| Selecting specific chunks | ❌ | `xarray.Dataset.isel` (✅ chunk-aligned selections only) | +| Sub-dividing an uncompressed chunk | `kerchunk.utils.subchunk` | `xarray.Dataset.isel` (✅ for uncompressed arrays — a slice along the largest-stride axis rewrites the reference's byte offset/length, see [#996](https://github.com/zarr-developers/VirtualiZarr/pull/996); a finer chunk grid along that axis can be built by combining `isel` with `xarray.concat`, so no dedicated rechunk method is needed) | +| Selecting specific chunks | ❌ | `xarray.Dataset.isel` (✅ chunk-aligned selections, plus sub-chunk slicing of uncompressed arrays) | **Parallelization** | | | | Parallelized generation of references | Wrapping kerchunk's opener inside `dask.delayed` | Wrapping `open_virtual_dataset` inside `dask.delayed` | Parallelized combining of references (tree-reduce) | `kerchunk.combine.auto_dask` | Wrapping `ManifestArray` objects within `dask.array.Array` objects inside `xarray.Dataset` to use dask's `concatenate` (⚠️ Untested, but also unnecessary) | From 99f7e383a9e699b81a7332799d047e37ed2a552e Mon Sep 17 00:00:00 2001 From: TomNicholas Date: Wed, 15 Jul 2026 14:41:28 -0700 Subject: [PATCH 2/3] Fix stale parallelization rows in Kerchunk comparison FAQ The two Parallelization rows predated open_virtual_mfdataset's `parallel` kwarg: - Generation: point at open_virtual_mfdataset(parallel="dask"|"lithops"| Executor), which parallelizes the per-file open map step, rather than only describing a manual dask.delayed wrap. - Combining (tree-reduce): drop the stale "wrap ManifestArray in dask.array (Untested)" description. VirtualiZarr deliberately combines references serially on the client since they are small, avoiding a distributed tree-reduce (per the Scaling docs). Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/explanation/faq.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/explanation/faq.md b/docs/explanation/faq.md index 4e1006320..693dc06ca 100644 --- a/docs/explanation/faq.md +++ b/docs/explanation/faq.md @@ -234,8 +234,8 @@ Users of Kerchunk may find the following comparison table useful, which shows wh | Sub-dividing an uncompressed chunk | `kerchunk.utils.subchunk` | `xarray.Dataset.isel` (✅ for uncompressed arrays — a slice along the largest-stride axis rewrites the reference's byte offset/length, see [#996](https://github.com/zarr-developers/VirtualiZarr/pull/996); a finer chunk grid along that axis can be built by combining `isel` with `xarray.concat`, so no dedicated rechunk method is needed) | | Selecting specific chunks | ❌ | `xarray.Dataset.isel` (✅ chunk-aligned selections, plus sub-chunk slicing of uncompressed arrays) | **Parallelization** | | | -| Parallelized generation of references | Wrapping kerchunk's opener inside `dask.delayed` | Wrapping `open_virtual_dataset` inside `dask.delayed` -| Parallelized combining of references (tree-reduce) | `kerchunk.combine.auto_dask` | Wrapping `ManifestArray` objects within `dask.array.Array` objects inside `xarray.Dataset` to use dask's `concatenate` (⚠️ Untested, but also unnecessary) | +| Parallelized generation of references | Wrapping kerchunk's opener inside `dask.delayed` | `open_virtual_mfdataset(..., parallel="dask" \| "lithops" \| Executor)`, which parallelizes the per-file `open_virtual_dataset` map step; or manually wrap `open_virtual_dataset` in `dask.delayed` | +| Parallelized combining of references (tree-reduce) | `kerchunk.combine.auto_dask` | Not needed — references are small, so the combine (reduce) step runs serially on the client, deliberately avoiding a distributed tree-reduce (see [Scaling](../how_to/scaling.md)) | | **On-disk serialization (6) and reading (7)** | | | | Kerchunk reference format as JSON | `ujson.dumps(h5chunks.translate())` , then read using an `fsspec.filesystem` mapper | `ds.vz.to_kerchunk('combined.json', format='JSON')` , then read using an `fsspec.filesystem` mapper | | Kerchunk reference format as parquet | `df.refs_to_dataframe(out_dict, "combined.parq")`, then read using an `fsspec` `ReferenceFileSystem` mapper | `ds.vz.to_kerchunk('combined.parq', format=parquet')` , then read using an `fsspec` `ReferenceFileSystem` mapper | From 8e7ba87e075bf2a4589d1cbb7535fc72a7d2a710 Mon Sep 17 00:00:00 2001 From: TomNicholas Date: Wed, 15 Jul 2026 15:07:17 -0700 Subject: [PATCH 3/3] Fix incorrect code snippets in Kerchunk comparison FAQ MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - to_kerchunk format strings: 'JSON' -> 'json' (the format arg only accepts the lowercase literals "dict"/"json"/"parquet"), and fix the missing opening quote in format='parquet'. - DMRPPParser: pass an instance (DMRPPParser()) like every other parser row, not the bare class. - Mark the Kerchunk cell for reading a Zarr v3 store as ❌ (ZarrToZarr is v2-only) instead of leaving it blank. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/explanation/faq.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/explanation/faq.md b/docs/explanation/faq.md index 693dc06ca..0ce322e76 100644 --- a/docs/explanation/faq.md +++ b/docs/explanation/faq.md @@ -210,12 +210,12 @@ Users of Kerchunk may find the following comparison table useful, which shows wh | From a netCDF3 file | `kerchunk.netCDF3.NetCDF3ToZarr` | `open_virtual_dataset(..., parser=NetCDF3Parser())`, via `kerchunk.netCDF3.NetCDF3ToZarr` | | From a COG / tiff file | `kerchunk.tiff.tiff_to_zarr` | `open_virtual_dataset(..., parser=VirtualTIFF())`, via [virtual_tiff](https://github.com/virtual-zarr/virtual-tiff) | | From a Zarr v2 store | `kerchunk.zarr.ZarrToZarr` | `open_virtual_dataset(..., parser=ZarrParser())` | -| From a Zarr v3 store | | `open_virtual_dataset(..., parser=ZarrParser())` | +| From a Zarr v3 store | ❌ | `open_virtual_dataset(..., parser=ZarrParser())` | | From an existing [Icechunk](https://icechunk.io/) repo | ❌ | `open_virtual_dataset(..., parser=IcechunkParser())`, or `IcechunkParser().parse_session(session, registry, native_chunks_prefix=...)` if you already have an open icechunk session | | From a GRIB1/GRIB2 file | `kerchunk.grib2.scan_grib` | `open_virtual_dataset(..., parser=GribberishParser())` (or `open_virtual_datatree(...)`), via [gribberish](https://github.com/mpiannucci/gribberish) | | From a FITS file | `kerchunk.fits.process_file` | `open_virtual_dataset(..., parser=FITSParser())`, via `kerchunk.fits.process_file` | | From a HDF4 file | `kerchunk.hdf4.HDF4ToZarr` | `open_virtual_dataset(..., parser=HDF4Parser())`, via `kerchunk.hdf4.HDF4ToZarr` | -| From a [DMR++](https://opendap.github.io/DMRpp-wiki/DMRpp.html) metadata file | ❌ | `open_virtual_dataset(..., parser=DMRPPParser)` | +| From a [DMR++](https://opendap.github.io/DMRpp-wiki/DMRpp.html) metadata file | ❌ | `open_virtual_dataset(..., parser=DMRPPParser())` | | From existing kerchunk JSON references | `kerchunk.combine.MultiZarrToZarr(append=True)` | `open_virtual_dataset(..., parser=KerchunkJSONParser())` | | From existing kerchunk parquet references | `kerchunk.combine.MultiZarrToZarr(append=True)` | `open_virtual_dataset(..., parser=KerchunkParquetParser())` | | **In-memory representation (2)** | | | @@ -237,8 +237,8 @@ Users of Kerchunk may find the following comparison table useful, which shows wh | Parallelized generation of references | Wrapping kerchunk's opener inside `dask.delayed` | `open_virtual_mfdataset(..., parallel="dask" \| "lithops" \| Executor)`, which parallelizes the per-file `open_virtual_dataset` map step; or manually wrap `open_virtual_dataset` in `dask.delayed` | | Parallelized combining of references (tree-reduce) | `kerchunk.combine.auto_dask` | Not needed — references are small, so the combine (reduce) step runs serially on the client, deliberately avoiding a distributed tree-reduce (see [Scaling](../how_to/scaling.md)) | | **On-disk serialization (6) and reading (7)** | | | -| Kerchunk reference format as JSON | `ujson.dumps(h5chunks.translate())` , then read using an `fsspec.filesystem` mapper | `ds.vz.to_kerchunk('combined.json', format='JSON')` , then read using an `fsspec.filesystem` mapper | -| Kerchunk reference format as parquet | `df.refs_to_dataframe(out_dict, "combined.parq")`, then read using an `fsspec` `ReferenceFileSystem` mapper | `ds.vz.to_kerchunk('combined.parq', format=parquet')` , then read using an `fsspec` `ReferenceFileSystem` mapper | +| Kerchunk reference format as JSON | `ujson.dumps(h5chunks.translate())` , then read using an `fsspec.filesystem` mapper | `ds.vz.to_kerchunk('combined.json', format='json')` , then read using an `fsspec.filesystem` mapper | +| Kerchunk reference format as parquet | `df.refs_to_dataframe(out_dict, "combined.parq")`, then read using an `fsspec` `ReferenceFileSystem` mapper | `ds.vz.to_kerchunk('combined.parq', format='parquet')` , then read using an `fsspec` `ReferenceFileSystem` mapper | | [Icechunk](https://icechunk.io/) store | ❌ | `ds.vz.to_icechunk()`, then read back via xarray (requires zarr-python v3). | ### Which format should I save my virtual references as?