Slow input of tree formats with many individual trajectories

[Reshief]

What happened?

Input of alkene datasets of shnitsel data in hierarchical tree format using xr.DataTree for storage are incredibly slow. It even appears to the user that the program is stuck. We are talking about 30 mins for a 50MB dataset, where datasets with 500MB load in less than a second.
The only difference seems to be that the smaller dataset has more trajectories of shorter length.

What did you expect to happen?

Input should be fairly fast. Neither conversion from xr.DataTree to ShnitselDB nor the input of a netcdf file should take this long.

Other things we should know?

This is meant to keep track of the issue of input speed.
It happens for st version 2026.1.1 but we have pinned down the issue to be the xarray.open_datatree() function that seems to slow down to a crawl at a very arbitrary number of datasets.

Is related to xarray Issue Nr. 9511

[Reshief]

As far as I can tell, it is an issue with the hdf5 interface.
h5py takes 11ms to return the name of a variable. Having a name with many datasets with individual variables results in several thousand lookups of variable names.

While in principle this does not seem like an issue, the hdf5 interface seems to do i/o operations (i.e. in-file lookup) for every request of a group of variable name.
This makes the input of a tree not really an issue of the tree structure but of the insane i/o for individual variables and the unfortunate handling of individual arrays.

Maybe we can use a read-only hdf5 backend that is faster with those lookups?

[Reshief]

The issue seems to be the lookup of names of dimension scales of dataset/variable entries.
Each such lookup needs several roundtrips from the python code to the hdf5 backend and the file contents.

Caching the entire structure first and reusing the names of dimension seems to be the most viable solution.
On the other hand, we probably don't want to manage a full custom hdf5 backend.