diff --git a/docs/explanation/faq.md b/docs/explanation/faq.md index 20149eaf..0ce322e7 100644 --- a/docs/explanation/faq.md +++ b/docs/explanation/faq.md @@ -210,12 +210,12 @@ Users of Kerchunk may find the following comparison table useful, which shows wh | From a netCDF3 file | `kerchunk.netCDF3.NetCDF3ToZarr` | `open_virtual_dataset(..., parser=NetCDF3Parser())`, via `kerchunk.netCDF3.NetCDF3ToZarr` | | From a COG / tiff file | `kerchunk.tiff.tiff_to_zarr` | `open_virtual_dataset(..., parser=VirtualTIFF())`, via [virtual_tiff](https://github.com/virtual-zarr/virtual-tiff) | | From a Zarr v2 store | `kerchunk.zarr.ZarrToZarr` | `open_virtual_dataset(..., parser=ZarrParser())` | -| From a Zarr v3 store | | `open_virtual_dataset(..., parser=ZarrParser())` | +| From a Zarr v3 store | ❌ | `open_virtual_dataset(..., parser=ZarrParser())` | | From an existing [Icechunk](https://icechunk.io/) repo | ❌ | `open_virtual_dataset(..., parser=IcechunkParser())`, or `IcechunkParser().parse_session(session, registry, native_chunks_prefix=...)` if you already have an open icechunk session | | From a GRIB1/GRIB2 file | `kerchunk.grib2.scan_grib` | `open_virtual_dataset(..., parser=GribberishParser())` (or `open_virtual_datatree(...)`), via [gribberish](https://github.com/mpiannucci/gribberish) | | From a FITS file | `kerchunk.fits.process_file` | `open_virtual_dataset(..., parser=FITSParser())`, via `kerchunk.fits.process_file` | | From a HDF4 file | `kerchunk.hdf4.HDF4ToZarr` | `open_virtual_dataset(..., parser=HDF4Parser())`, via `kerchunk.hdf4.HDF4ToZarr` | -| From a [DMR++](https://opendap.github.io/DMRpp-wiki/DMRpp.html) metadata file | ❌ | `open_virtual_dataset(..., parser=DMRPPParser)` | +| From a [DMR++](https://opendap.github.io/DMRpp-wiki/DMRpp.html) metadata file | ❌ | `open_virtual_dataset(..., parser=DMRPPParser())` | | From existing kerchunk JSON references | `kerchunk.combine.MultiZarrToZarr(append=True)` | `open_virtual_dataset(..., parser=KerchunkJSONParser())` | | From existing kerchunk parquet references | `kerchunk.combine.MultiZarrToZarr(append=True)` | `open_virtual_dataset(..., parser=KerchunkParquetParser())` | | **In-memory representation (2)** | | | @@ -231,14 +231,14 @@ Users of Kerchunk may find the following comparison table useful, which shows wh | Renaming variables | ❌ | `xarray.Dataset.rename_vars` | | Renaming dimensions | ❌ | `xarray.Dataset.rename_dims` | | Renaming manifest file paths | `kerchunk.utils.rename_target` | `vds.vz.rename_paths` | -| Splitting uncompressed data into chunks | `kerchunk.utils.subchunk` | `xarray.Dataset.chunk` (❌ Not yet implemented - see [PR #199](https://github.com/zarr-developers/VirtualiZarr/pull/199)) -| Selecting specific chunks | ❌ | `xarray.Dataset.isel` (✅ chunk-aligned selections only) | +| Sub-dividing an uncompressed chunk | `kerchunk.utils.subchunk` | `xarray.Dataset.isel` (✅ for uncompressed arrays — a slice along the largest-stride axis rewrites the reference's byte offset/length, see [#996](https://github.com/zarr-developers/VirtualiZarr/pull/996); a finer chunk grid along that axis can be built by combining `isel` with `xarray.concat`, so no dedicated rechunk method is needed) | +| Selecting specific chunks | ❌ | `xarray.Dataset.isel` (✅ chunk-aligned selections, plus sub-chunk slicing of uncompressed arrays) | **Parallelization** | | | -| Parallelized generation of references | Wrapping kerchunk's opener inside `dask.delayed` | Wrapping `open_virtual_dataset` inside `dask.delayed` -| Parallelized combining of references (tree-reduce) | `kerchunk.combine.auto_dask` | Wrapping `ManifestArray` objects within `dask.array.Array` objects inside `xarray.Dataset` to use dask's `concatenate` (⚠️ Untested, but also unnecessary) | +| Parallelized generation of references | Wrapping kerchunk's opener inside `dask.delayed` | `open_virtual_mfdataset(..., parallel="dask" \| "lithops" \| Executor)`, which parallelizes the per-file `open_virtual_dataset` map step; or manually wrap `open_virtual_dataset` in `dask.delayed` | +| Parallelized combining of references (tree-reduce) | `kerchunk.combine.auto_dask` | Not needed — references are small, so the combine (reduce) step runs serially on the client, deliberately avoiding a distributed tree-reduce (see [Scaling](../how_to/scaling.md)) | | **On-disk serialization (6) and reading (7)** | | | -| Kerchunk reference format as JSON | `ujson.dumps(h5chunks.translate())` , then read using an `fsspec.filesystem` mapper | `ds.vz.to_kerchunk('combined.json', format='JSON')` , then read using an `fsspec.filesystem` mapper | -| Kerchunk reference format as parquet | `df.refs_to_dataframe(out_dict, "combined.parq")`, then read using an `fsspec` `ReferenceFileSystem` mapper | `ds.vz.to_kerchunk('combined.parq', format=parquet')` , then read using an `fsspec` `ReferenceFileSystem` mapper | +| Kerchunk reference format as JSON | `ujson.dumps(h5chunks.translate())` , then read using an `fsspec.filesystem` mapper | `ds.vz.to_kerchunk('combined.json', format='json')` , then read using an `fsspec.filesystem` mapper | +| Kerchunk reference format as parquet | `df.refs_to_dataframe(out_dict, "combined.parq")`, then read using an `fsspec` `ReferenceFileSystem` mapper | `ds.vz.to_kerchunk('combined.parq', format='parquet')` , then read using an `fsspec` `ReferenceFileSystem` mapper | | [Icechunk](https://icechunk.io/) store | ❌ | `ds.vz.to_icechunk()`, then read back via xarray (requires zarr-python v3). | ### Which format should I save my virtual references as?