Support “regular axis” metadata (t0/Δt) to avoid storing full axis vectors in NXdata line plots

[NAThompson]

Problem statement

In NeXus NXdata, H5Web can auto-plot a line when the group contains a signal dataset and an axis dataset referenced by @axes.
For a uniformly sampled time series, storing a {times, values} datastructure is unnecessarily expensive because the times array is large and redundant-times[i] = t0 + i Δt.

I’d would like to store a {t0, Δt, values} datastructure and have H5Web still auto-plot vs physical time.

Proposed behavior

Add support for an axis that is implicitly defined as a regular grid. If an axis dataset is missing, but metadata indicates regular sampling, H5Web should synthesize the axis and plot with the correct metadata.

I imagine the h5ls -dlooking something like:

/entry/data (NXdata)
  values         (float[N])   @signal=1
  t0             (float)      @units="s"
  dt             (float)      @units="s"
  @axes = "time"              (virtual axis name)

There is a "checkdown" available: Use a scaling transform in order to map the integers [0, 1, 2, 3, ..., N-1] (which is quite compressible) to [t0, t0 + Δt, ...].

[loichuder]

Hello @NAThompson

Thanks for the suggestion !

Our standard is the NeXus convention and even though I seem to remember something akin in the NeXus doc, I could not find something that generates an axis out of metadata.

However, the "checkdown" you describe can indeed be achieved by attaching FIELDNAME_scaling_factor and FIELDNAME_offset to an axis that contains the integers [0, 1, 2, 3, ..., N-1]. These are not supported in H5Web for now.

If you think storing the axis as integers is an acceptable compromise, we could imagine bringing the support of scaling_factor and offset in H5Web to address your feature request.

[NAThompson]

@loichuder : scaling factor and offset would not just be good for this, but also for storing fixed point numbers!

The one concern I have is that my data vector was so huge that even scrolling around in the values dataset was quite slow. I worry that if {times, values} are synthesized without explicitly exploiting the uniformity of the time axis would become extremely slow. Or is that not how the current integer indexing works during rendering?

[loichuder]

@NAThompson Could you give an estimation of the number of points of your dataset ?

That will be helpful for the tests. If the performance is not satisfactory, we may indeed have to consider something else.

[NAThompson]

@loichuder : ~100 million float32s.

[loichuder]

After discussion with @axelboc and @t20100, we feel like the axes may not be the limiting factor to visualize the data if there is 100 million of points.

Is it a 1D dataset that you are viewing ? With the Line vis ? In which context (myHDF5, VSCode, JupyterLab) ?

Because from quick tests, even without axes, the visualization feels already a bit laggy with 10**6 points in VSCode...

[NAThompson]

@loichuder : It is a 1D dataset, line vis, VSCode extension. You're right: It definitely is a bit laggy with 10**6 points as well.

[axelboc]

I pushed a very small optimisation that at least fixes the lag when the mouse enters and leaves the canvas. So this leaves mainly the zoom lag — I haven't been able to pinpoint any obvious cause, unfortunately.

Given that the FIELDNAME_scaling_factor and FIELDNAME_offset fields are likely to be used as a file size optimisation for very large datasets, and given that H5Web struggles to deal with these very large datasets anyway, I'm not convinced it's worth implementing at this time. I propose to close this issue for the time being, but feel free to re-open if you'd like to keep the discussion going.

FYI, there's apparently a desire to implement function-based dimension scales in HDF5, which could open the door to new possibilities, but this feels very long term.

[NAThompson]

I pushed a very small optimisation that at least fixes the lag when the mouse enters and leaves the canvas. So this leaves mainly the zoom lag — I haven't been able to pinpoint any obvious cause, unfortunately.

This is awesome thanks!

Given that the FIELDNAME_scaling_factor and FIELDNAME_offset fields are likely to be used as a file size optimisation for very large datasets, and given that H5Web struggles to deal with these very large datasets anyway, I'm not convinced it's worth implementing at this time. I propose to close this issue for the time being, but feel free to re-open if you'd like to keep the discussion going.

Well, this issue wasn't really about H5Web. It was "how do I keep my filesize small for archival storage and also have a beautiful native graphics interface for it". So the lag-though important-was a secondary consideration for this issue.

I agree that HDF5 dimension scales do seem to be the right long term solution, but your MR looked good to me!

[axelboc]

I agree that HDF5 dimension scales do seem to be the right long term solution, but your MR looked good to me!

Yeah it's pretty good but it does add a bit of complexity notably because the scaling factor and offset are stored in datasets rather than attributes — nothing crazy, but any code we add is code we need to maintain long term and take into account when making bigger changes (like upgrading React), so if the benefit isn't clear we prefer to be conservative.