Add option to ignore h5 context manager in preprocessing

[mmccrackan]

This allows preprocessing to bypass the h5 context manager when running off of site.

[tskisner]

@mmccrackan, just checking- have you found any performance issue with the new temp file usage? Or is this bypass just to avoid the copy when you can already guarantee that only a single process is accessing an HDF5 file?

[mmccrackan]

@mmccrackan, just checking- have you found any performance issue with the new temp file usage? Or is this bypass just to avoid the copy when you can already guarantee that only a single process is accessing an HDF5 file?

It seems when running SAT preprocessing, this incurs a big slowdown due to needing to copy both files simultaneously. I need to look into why it didn't seem to affect things when testing on LAT ASO data on Nersc, but I'm guessing it was just due to having less data overall + single node usage. We won't need it for running preprocessing off site so this was just to allow that. Longer term I think we should add an override_copy option to the H5ContextManager to ignore copying as that will be more general for off-site usage.

[tskisner]

Ok, the main use case was long-running rsync at site during simultaneous appending to the files. Maybe there are other cases where h5 files are being appended while another process is reading, but I'm not sure. Agree about having a disable switch in the context manager.