Are the write tests here measuring write time correctly?

[charles-turner-1]

Heya,

I've been trying to replicate some of these benchmarks on some real world data we have and I'm finding some pretty different results. I've forked and modified the bench code pretty heavily to reflect our use cases a bit more closely, but when I was doing that I noticed this snippet:

        if len(dimensions) == 1:
            t3 = time.perf_counter()
            dataset[:dimensions[0]] = data
        elif len(dimensions) == 2:
            t3 = time.perf_counter()
            dataset[:dimensions[0], :dimensions[1]] = data
        else:
            t3 = time.perf_counter()
            dataset[:dimensions[0], :dimensions[1], :dimensions[2]] = data
        t4 = time.perf_counter()

        # Add up the times taken to get the total time taken to create and write all datasets
        dataset_creation_time += (t2 - t1)
        dataset_population_time += (t4 - t3)

coming from

File-Format-Testing/datasets_test/write.py

Line 50 in 5d8161f

dataset[:dimensions[0]] = data

This looks to me like you're just measuring the time to write into a buffer, rather than actually write the files to disk? From a usage scenario, I'm pretty sure disk IO rather than filling a buffer will dominate write time, so I don't think this is necessarily benchmarking exactly what you were hoping to?

For example, our loads and writes look something like this:

which is pretty different to the results you guys found - at least at first glance!

(Ignore the high outlier for the HDF5 read, I'm pretty sure thats related to FS block caching for importing the code used to read the files off disk).

Thanks so much for doing this work - it's something I'd never thought about much until I read the paper and it's definitely got me thinking about serialisation more deeply. I'm on my laptop right now, but when I get the chance I'll link my fork of the benchmarks too.

[asriniket]

Thank you for bringing this to our attention!

I plan on fixing the code and re-running the experiments, but first, I wanted your input on the fix. So far, I've modified the write function to flush the data to the OS buffer like so:

        # Flush to ensure data is written to OS buffers before measuring time
        if file_format == 'HDF5':
            file.flush()
        elif file_format == 'netCDF4':
            file.sync()
        elif file_format == 'Zarr':
            pass  # Zarr DirectoryStore writes synchronously

        t4 = time.perf_counter()

        # Add up the times taken to get the total time taken to create and write all datasets
        dataset_creation_time += (t2 - t1)
        dataset_population_time += (t4 - t3)

I could also include an os.fsync() call to ensure data is written to disk and include it in our write time measurement, but as you noted, that could introduce hardware-dependent noise in the measurement.

Would you prefer we measure complete write-to-disk time with fsync(), or does flushing to OS buffers (as shown above) adequately address the timing issue you identified?

[charles-turner-1]

Good question! I'm not 100% sure on the best approach, but I feel like writing to disk is probably the way to go as it's more realistic. Assuming that the benchmarks are all on the same system then hardware dependency should at least be somewhat consistent - I'm happy to run the updated code on a couple systems I have access to too, which should help assess the extent of the noise & it's hardware dependence.