Skip to content

Update stale rows in the Kerchunk comparison FAQ #1045

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
TomNicholas merged 3 commits into from
Jul 15, 2026
Merged

Update stale rows in the Kerchunk comparison FAQ #1045

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7e8c63b
Update Kerchunk comparison FAQ for sub-chunk isel parity
TomNicholas Jul 15, 2026
99f7e38
Fix stale parallelization rows in Kerchunk comparison FAQ
TomNicholas Jul 15, 2026
8e7ba87
Fix incorrect code snippets in Kerchunk comparison FAQ
TomNicholas Jul 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions docs/explanation/faq.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -210,12 +210,12 @@ Users of Kerchunk may find the following comparison table useful, which shows wh
| From a netCDF3 file | `kerchunk.netCDF3.NetCDF3ToZarr` | `open_virtual_dataset(..., parser=NetCDF3Parser())`, via `kerchunk.netCDF3.NetCDF3ToZarr` |
| From a COG / tiff file | `kerchunk.tiff.tiff_to_zarr` | `open_virtual_dataset(..., parser=VirtualTIFF())`, via [virtual_tiff](https://github.com/virtual-zarr/virtual-tiff) |
| From a Zarr v2 store | `kerchunk.zarr.ZarrToZarr` | `open_virtual_dataset(..., parser=ZarrParser())` |
| From a Zarr v3 store | | `open_virtual_dataset(..., parser=ZarrParser())` |
| From a Zarr v3 store | ❌ | `open_virtual_dataset(..., parser=ZarrParser())` |
| From an existing [Icechunk](https://icechunk.io/) repo | ❌ | `open_virtual_dataset(..., parser=IcechunkParser())`, or `IcechunkParser().parse_session(session, registry, native_chunks_prefix=...)` if you already have an open icechunk session |
| From a GRIB1/GRIB2 file | `kerchunk.grib2.scan_grib` | `open_virtual_dataset(..., parser=GribberishParser())` (or `open_virtual_datatree(...)`), via [gribberish](https://github.com/mpiannucci/gribberish) |
| From a FITS file | `kerchunk.fits.process_file` | `open_virtual_dataset(..., parser=FITSParser())`, via `kerchunk.fits.process_file` |
| From a HDF4 file | `kerchunk.hdf4.HDF4ToZarr` | `open_virtual_dataset(..., parser=HDF4Parser())`, via `kerchunk.hdf4.HDF4ToZarr` |
| From a [DMR++](https://opendap.github.io/DMRpp-wiki/DMRpp.html) metadata file | ❌ | `open_virtual_dataset(..., parser=DMRPPParser)` |
| From a [DMR++](https://opendap.github.io/DMRpp-wiki/DMRpp.html) metadata file | ❌ | `open_virtual_dataset(..., parser=DMRPPParser())` |
| From existing kerchunk JSON references | `kerchunk.combine.MultiZarrToZarr(append=True)` | `open_virtual_dataset(..., parser=KerchunkJSONParser())` |
| From existing kerchunk parquet references | `kerchunk.combine.MultiZarrToZarr(append=True)` | `open_virtual_dataset(..., parser=KerchunkParquetParser())` |
| **In-memory representation (2)** | | |
Expand All @@ -231,14 +231,14 @@ Users of Kerchunk may find the following comparison table useful, which shows wh
| Renaming variables | ❌ | `xarray.Dataset.rename_vars` |
| Renaming dimensions | ❌ | `xarray.Dataset.rename_dims` |
| Renaming manifest file paths | `kerchunk.utils.rename_target` | `vds.vz.rename_paths` |
| Splitting uncompressed data into chunks | `kerchunk.utils.subchunk` | `xarray.Dataset.chunk` (❌ Not yet implemented - see [PR #199](https://github.com/zarr-developers/VirtualiZarr/pull/199))
| Selecting specific chunks | ❌ | `xarray.Dataset.isel` (✅ chunk-aligned selections only) |
| Sub-dividing an uncompressed chunk | `kerchunk.utils.subchunk` | `xarray.Dataset.isel` (✅ for uncompressed arrays — a slice along the largest-stride axis rewrites the reference's byte offset/length, see [#996](https://github.com/zarr-developers/VirtualiZarr/pull/996); a finer chunk grid along that axis can be built by combining `isel` with `xarray.concat`, so no dedicated rechunk method is needed) |
| Selecting specific chunks | ❌ | `xarray.Dataset.isel` (✅ chunk-aligned selections, plus sub-chunk slicing of uncompressed arrays) |
**Parallelization** | | |
| Parallelized generation of references | Wrapping kerchunk's opener inside `dask.delayed` | Wrapping `open_virtual_dataset` inside `dask.delayed`
| Parallelized combining of references (tree-reduce) | `kerchunk.combine.auto_dask` | Wrapping `ManifestArray` objects within `dask.array.Array` objects inside `xarray.Dataset` to use dask's `concatenate` (⚠️ Untested, but also unnecessary) |
| Parallelized generation of references | Wrapping kerchunk's opener inside `dask.delayed` | `open_virtual_mfdataset(..., parallel="dask" \| "lithops" \| Executor)`, which parallelizes the per-file `open_virtual_dataset` map step; or manually wrap `open_virtual_dataset` in `dask.delayed` |
| Parallelized combining of references (tree-reduce) | `kerchunk.combine.auto_dask` | Not needed — references are small, so the combine (reduce) step runs serially on the client, deliberately avoiding a distributed tree-reduce (see [Scaling](../how_to/scaling.md)) |
| **On-disk serialization (6) and reading (7)** | | |
| Kerchunk reference format as JSON | `ujson.dumps(h5chunks.translate())` , then read using an `fsspec.filesystem` mapper | `ds.vz.to_kerchunk('combined.json', format='JSON')` , then read using an `fsspec.filesystem` mapper |
| Kerchunk reference format as parquet | `df.refs_to_dataframe(out_dict, "combined.parq")`, then read using an `fsspec` `ReferenceFileSystem` mapper | `ds.vz.to_kerchunk('combined.parq', format=parquet')` , then read using an `fsspec` `ReferenceFileSystem` mapper |
| Kerchunk reference format as JSON | `ujson.dumps(h5chunks.translate())` , then read using an `fsspec.filesystem` mapper | `ds.vz.to_kerchunk('combined.json', format='json')` , then read using an `fsspec.filesystem` mapper |
| Kerchunk reference format as parquet | `df.refs_to_dataframe(out_dict, "combined.parq")`, then read using an `fsspec` `ReferenceFileSystem` mapper | `ds.vz.to_kerchunk('combined.parq', format='parquet')` , then read using an `fsspec` `ReferenceFileSystem` mapper |
| [Icechunk](https://icechunk.io/) store | ❌ | `ds.vz.to_icechunk()`, then read back via xarray (requires zarr-python v3). |

### Which format should I save my virtual references as?
Expand Down
Loading