Optimization related to compression, allowing multithreaded blosc2 to be used as a compression library

[mkuehbach]

• Adding whats required to use blosc2, essentially making explicit what the inclusion of hdf5plugin and pandas already delivered
• Write all non-scalar non-string datasets using chunked storage layout with h5py's autochunker active by default the main motivation behind this move is that it allows everybody to take advantage of Memory optimization NOMAD nexus parser #750 and Memory optimization PYNXTOOLS validate #752 chunked-by-chunk based processing
• deflate, i.e. "gzip" is kept as the standard algo by default.

[sherjeelshabih]

Can you also add a small text in this PR even for now of what changes this introduces to the way the user interacts with this? I believe the writer.py now expects a "filter" key in the Template object. It will be nice to know the "user interface" changes in the PR.

Thanks for introducing this. I hope it makes it easier for the large datasets we run into.

[mkuehbach]

Can you also add a small text in this PR even for now of what changes this introduces to the way the user interacts with this? I believe the writer.py now expects a "filter" key in the Template object. It will be nice to know the "user interface" changes in the PR.

Thanks for introducing this. I hope it makes it easier for the large datasets we run into.

Need to check, thought that "filter" is not required per se, a default kicks in, but yeah i should document this.

[lukaspie]

Looks good, but I am not an expert

[RubelMozumder]

LGTM!
Though I have a general question, how effectively will this multi-threading work in NOMAD? Where NOMAD launches NeXus parser using an asynchronous thread. I do not have any idea about it.