AssertionError in H5adToZarr.dump: not all cells written to Zarr

[ulvelind]

Hi,

Firstly, thank you very much for this package, which allows us to overcome memory limitations on large datasets.

I have around 270 h5ad files to merge and then integrate. I managed to create all the h5ad files in zarr except for one, which gave me an error message.

I ran into an AssertionError when converting an .h5ad file to Zarr using H5adToZarr. The error message explicitly suggests reporting this as a bug.

I have configured everything in an apptainer.

python # version 3.11.14
import scarf # version 0.32.3

print(f" → Création → {zarr_path}")
reader = scarf.H5adReader(
    h5ad_path,
    feature_name_key='name'
)
writer = scarf.H5adToZarr(reader, zarr_loc=zarr_path)
writer.dump()

WARNING: `obsm` slot in H5ad file is empty.
INFO: No value provided for assay names. Will use default value: 'RNA'
WARNING: Could not find cells ids key: _index in `obs`.
Reading attributes from group obs: 100%
WARNING: Reading of obsm failed because it either does not exist or is not in expected format
WARNING: Could not find feature ids key: _index in var.
Reading attributes from group var: 100%
...
----> 7 writer.dump()

File /usr/local/lib/python3.11/site-packages/scarf/writers.py:454, in H5adToZarr.dump(self, batch_size)
    447 total_cells_written = sparse_writer(
    448     store=store,
    449     data_stream=self.h5ad.consume(batch_size),
    450     n_cells=self.h5ad.nCells,
    451     batch_size=batch_size,
    452 )
    453 if total_cells_written != self.h5ad.nCells:
--> 454     raise AssertionError(
    455         "ERROR: This is a bug in H5adToZarr. All cells might not have been successfully "
    456         "written into the zarr file. Please report this issue"
    457     )

Finally, I can still read the Zarr file generated with the correct number of cells before ZarrMerge and without errors.

So I am unsure whether this Zarr file is safe to keep and use in ZarrMerge, or whether there might be subtle corruption

Thank you by advance