lazy reading of parquet refs

[rsignell]

Does VirtualiZarr support lazy loading of parquet refs?

I can open an xarray virtual dataset from collection of kerchunk-generated parquet refs using kerchunk in less than 1 second:

import xarray as xr
endpoint_url='https://usgs.osn.mghpcc.org'
combined_parquet_aws = 's3://esip/rsignell/necofs.parquet'

ds = xr.open_dataset(f'reference://', engine='zarr', chunks={'time':100}, zarr_format=2, consolidated=False,
                     storage_options=dict(fo=combined_parquet_aws, remote_protocol='s3', lazy=True, asyncronous=True,
                                         target_options={'anon': True, 'endpoint_url':endpoint_url},              
                                         remote_options={'anon': True, 'endpoint_url':endpoint_url, 'asynchronous':True}))

but opening the same dataset with VirtualiZarr takes about 30s (before it errors because of inlined data):

from virtualizarr import open_virtual_dataset
so = dict(anon=True,  endpoint_url='https://usgs.osn.mghpcc.org')
ds = open_virtual_dataset('s3://esip/rsignell/necofs_2017.parquet', filetype='kerchunk', 
                           reader_options=dict(storage_options=so))

[norlandrhagen]

Interesting find @rsignell! I wonder if some of the storage_options are not getting passed though. Might be worth diving into a debugger and investigating if you're interested.