As pointed out by @Katetc - we have a number of places in CMEPS where we use ESMF_VMAllreduce to do global sums. She found this in the course of testing answer changes with changing processor counts for T compsets (CISM forced by DLND), and for that use case, the issue is the global sums in the SMB renormalization (she says that turning off SMB renormalization allows answers to be bit-for-bit when changing processor counts). But, from a quick look through the CMEPS code, it looks like this also comes up in post_rof_mod (dealing with negative runoff) and in prep_atm_mod (enthalpy correction).
I'm not sure we want to go so far as to require that global sums always be bit-for-bit with changing processor count, since that could impact performance. (My impression is that this isn't actually a CESM-wide requirement, but some components have it as a requirement. I'm not sure I'm right about that, though.) But it seems like we should at least have an option for reproducible global sums if a flag is set, and we can then use that flag in our testing. We have a currently-deprecated BFBFLAG, and the easiest course may be to bring that flag back into operation and use it for this purpose.
So I propose:
@briandobbins @mvertens - also looping you in here; I'd welcome your thoughts.
As pointed out by @Katetc - we have a number of places in CMEPS where we use ESMF_VMAllreduce to do global sums. She found this in the course of testing answer changes with changing processor counts for T compsets (CISM forced by DLND), and for that use case, the issue is the global sums in the SMB renormalization (she says that turning off SMB renormalization allows answers to be bit-for-bit when changing processor counts). But, from a quick look through the CMEPS code, it looks like this also comes up in post_rof_mod (dealing with negative runoff) and in prep_atm_mod (enthalpy correction).
I'm not sure we want to go so far as to require that global sums always be bit-for-bit with changing processor count, since that could impact performance. (My impression is that this isn't actually a CESM-wide requirement, but some components have it as a requirement. I'm not sure I'm right about that, though.) But it seems like we should at least have an option for reproducible global sums if a flag is set, and we can then use that flag in our testing. We have a currently-deprecated BFBFLAG, and the easiest course may be to bring that flag back into operation and use it for this purpose.
So I propose:
@briandobbins @mvertens - also looping you in here; I'd welcome your thoughts.