Forcing data size

[TijnBerends]

Hi guys,

I'm just getting started downloading the Antarctica forcing data, using the Globus CLI. However, I'm a bit worried by the absolutely massive volume of data.

I'm just looking at the CESM2-WACCM historical data. The ocean forcing is single-precision, compressed NetCDF, resulting in about 9 Gb of data per variable for the entire 165-year period; quite manageable. The atmospheric forcing data, however, has not been compressed, so each yearly file (with a 2km resolution) is about 440 Mb. With 165 years of data and 14 variables, that adds up to just over 1 Tb of data. And that's just for the historical period; the four different SSP scenarios, which have even more years, would be another 1.8 Tb each, adding up to just over 8 Tb for the entire thing.

I ran a quick test, and turning on NetCDF deflate+shuffle for the atmospheric data reduces the file size by about 75% (lossless compression), which would bring the total volume down to about 2 Tb. That would take me about 60 hours to download, but it might just fit on my HPC disk. 8 Tb would take ten days of non-stop downloading, never mind the fact that I don't have the disk space to store that kind of data.

Would it be possible to apply this compression to the files on the server?

[hgoelzer]

Hi Tijn,

Thanks for reporting back on your experience and for using this discussion forum. You win the price for the first post!
This is something the SMB focus group will have to address (and they have been alerted), I am just adding my thoughts here. I think you are right that compression would be very useful in this case for storage and transfer. It is important to realise though, that depending on the processing that follows, the files may need to be expanded to original size in memory or on disk. That's a factor to plan into the workflow and when estimating the storage requirements on the modellers target machine. Another approach to reduce file and memory size for many modellers would be to also offer files at a lower resolution. The additional work and storage requirements on globus for providing another resolution (say 8 km) may be worth it.

[erwinlambert]

To follow up on this, forcing files at lower resolution (4 or 8 km for AIS) may be very useful to have an alternative.

[drdunmire1417]

Hi all - the atmospheric forcing files are now compressed using Netcdf shuffle+deflate. Hopefully this is OK now. I can also add forcing files at a lower resolution if modeling groups will use them?

[erwinlambert]

Wow that was very fast, and indeed about a factor 4 reduction in file size. I also see that 8km resolution files are online, which may come in very handy. Thanks a lot!

[drdunmire1417]

Yes the 8km files for AIS and 4km for GrIS should also be available now!

[xylar]

Amazing, thanks @drdunmire1417!!