Rare answer changes in Fioi_swpen_vdf and Fioi_swpen_vdr in branch run with CICE in prescribed mode

[billsacks]

This is emerging from ESMCI/cime#4911; see that issue for some context.

The summary is that, with a recent change in the definition of the ERI test, CAM is seeing a failure in the test ERI_D_Ln18.ne16pg3_ne16pg3_mt232.FHIST_C4.derecho_intel.cam-outfrq3s_eri, and it appears that this is due to an answer change within CICE (run in prescribed mode) in the branch run.

An ERI test runs the following cases:

• (1) Startup
• (2) Hybrid from restart files written in (1)
• (3a) Branch from restart files written in (2)
• (3b) Restart from restart files written in (3a)

It then compares:

• (3a) with (2) - to verify that a branch run doesn't change answers
• (3b) with (3a) - to verify that a restart doesn't change answers

The hybrid run changes the start year (to verify that the system works when changing the start year in a hybrid case). Until recently, the start year was changed by 2 years; the recent change to the ERI test is that the start year is now changed by 4 years. With this change, the comparison between (3a) and (2) is failing in this one ERI test, indicating that there is an answer change in the branch case relative to the case it's branching from.

The answer change appears in just two fields: iceImp_Fioi_swpen_vdf and iceImp_Fioi_swpen_vdr. (It looks like these fields are sent from ice to ocn, so it makes sense that, in this F compset, we're not seeing any feedbacks to the rest of the system.) In both cases, the difference is in a single grid cell, index 13627, where the hybrid case (2) has a non-zero value but the branch case (3a) has a zero value.

Before the recent change, case (2) started in year 1981; after the recent change, case (2) starts in year 1983. So it appears that there is a rare problem with non-bit-for-bit behavior of branch runs in CICE run in prescribed mode, which is triggered when branching from a case in early 1983 but not when doing an analogous branch from a case in early 1981.

FYI - @dabail10 @cacraigucar @fischer-ncar

[dabail10]

This should not be surprising because in an F compset, we read in the sea ice concentration and reset the thickness to 2m at every step.

[billsacks]

@dabail10 - can you elaborate? When you say, "this should not be surprising", do you mean that you expect a failure in an ERI test like this? Or is something possibly getting lost in the communication here?

To try to summarize my understanding of the issue: we are seeing isolated differences in an F compset (with CICE%PRES) when comparing a branch run against the run it branched from. I believe that indicates a bug, but let me know if you think differently.

[dabail10]

The F compset is not energy conserving. There is heat to the ocean that is thrown away at every timestep. Do we have restarts for CICE in an F compset? Maybe it should still be reproducible. When did this change?

[billsacks]

My understanding is that a branch run should be bit-for-bit even in a non-fully-coupled compset (just like a restart / continue run). There are restarts for CICE in an F compset. And note that the differences in this case are in a single grid cell, so it seems like things are essentially working, but there must be some rare condition that leads to differences.

There is some background in ESMCI/cime#4911 on what caused this answer change to emerge, but to summarize: the definition of the ERI test recently changed so that the ref2 case now starts in 1983 rather than 1981. Reverting that to start in 1981 brings this test back to passing. Starting in 1983 leads to this difference in a single grid cell. So again, it seems like this is probably a rare bug that has been around for a while but just started appearing by chance due to this minor change in the test definition.

[dabail10]

I am guessing that these fields are not in the coupler restart file. They are not in the CICE restart file. So, perhaps in some cases they are just zero, but in others not.

[billsacks]

These fields are in the cpl restart file:

$ ncdump -h /glade/derecho/scratch/cacraig/archive/ERI_D_Ln18.ne16pg3_ne16pg3_mt232.FHIST_C4.derecho_intel.cam-outfrq3s_eri.20260108_142315_9f7r24.ref2/rest/1983-01-01-14400/ERI_D_Ln18.ne16pg3_ne16pg3_mt232.FHIST_C4.derecho_intel.cam-outfrq3s_eri.20260108_142315_9f7r24.ref2.cpl.r.1983-01-01-14400.nc | grep iceImp_Fioi_swpen
	double iceImp_Fioi_swpen(time, iceImp_ny, iceImp_nx) ;
		iceImp_Fioi_swpen:_FillValue = 1.e+30 ;
		iceImp_Fioi_swpen:units = "W m-2" ;
		iceImp_Fioi_swpen:standard_name = "iceImp_Fioi_swpen" ;
	double iceImp_Fioi_swpen_idf(time, iceImp_ny, iceImp_nx) ;
		iceImp_Fioi_swpen_idf:_FillValue = 1.e+30 ;
		iceImp_Fioi_swpen_idf:units = "W m-2" ;
		iceImp_Fioi_swpen_idf:standard_name = "iceImp_Fioi_swpen_idf" ;
	double iceImp_Fioi_swpen_idr(time, iceImp_ny, iceImp_nx) ;
		iceImp_Fioi_swpen_idr:_FillValue = 1.e+30 ;
		iceImp_Fioi_swpen_idr:units = "W m-2" ;
		iceImp_Fioi_swpen_idr:standard_name = "iceImp_Fioi_swpen_idr" ;
	double iceImp_Fioi_swpen_vdf(time, iceImp_ny, iceImp_nx) ;
		iceImp_Fioi_swpen_vdf:_FillValue = 1.e+30 ;
		iceImp_Fioi_swpen_vdf:units = "W m-2" ;
		iceImp_Fioi_swpen_vdf:standard_name = "iceImp_Fioi_swpen_vdf" ;
	double iceImp_Fioi_swpen_vdr(time, iceImp_ny, iceImp_nx) ;
		iceImp_Fioi_swpen_vdr:_FillValue = 1.e+30 ;
		iceImp_Fioi_swpen_vdr:units = "W m-2" ;
		iceImp_Fioi_swpen_vdr:standard_name = "iceImp_Fioi_swpen_vdr" ;

I'm not certain, but it feels to me like this is likely a problem in CICE, such as missing restart fields there.

[dabail10]

Another important aspect is that these fields with the idf, idr, vdr, vdf are optional and only used in the ocean BGC. So, these should not even be turned on. However, I do see that these are sent all the time. Ok. Talked with Mike Levy about this and I understand a bit more. CMEPS only requests these fields when there is an active ocean. CICE is sending them all the time, so I should change that using the same flag as in the mediator. The coupler restart should not be saving these fields when there is no active ocean as well. I can add this to an upcoming CICE tag.

https://github.com/ESCOMP/CMEPS/blob/8d3ff3aff46a0072679649acd82337a85ccc3b2a/mediator/med_phases_prep_ocn_mod.F90#L471-L484

[billsacks]

Having CICE avoid sending these if they aren't needed feels very reasonable, and should resolve this specific problem in CAM's testing. Thanks!

However, I still wonder if there may be a real issue here that should be resolved. But then again, if this issue is specific to CICE in prescribed mode (which seems possible), and if CICE%PRES only shows up in F compsets (which seems to be the case from a quick check, though I'm not positive), then maybe this is fine.

[dabail10]

So, I have looked into this further and I am confused. There are two sections of code in med_phases_prep_ocn_mod.F90 that set import_swnet_by_bands and export_swnet_by_bands. The export section checks to see if the ocean is calling for Foxx_swnet_vdr and the import section section for the sea ice calling for Fioi_swnet_vdr. So, when I turned off the export of Fioi_swnet_vdr, this should have also removed it from the mediator import state. Yet, I still see it on the restart. It is all zeroes, so I guess it is being initialized somewhere. Maybe because, this also needs to be removed from the ocean import state. However, I don't know why it is there for DOCN-SST mode. Thoughts?

https://github.com/ESCOMP/CMEPS/blob/9991b7bcfd127ca8dabab079cd5593a7782bd965/mediator/med_phases_prep_ocn_mod.F90#L471-L484

https://github.com/ESCOMP/CMEPS/blob/9991b7bcfd127ca8dabab079cd5593a7782bd965/mediator/med_phases_prep_ocn_mod.F90#L530-L532

[dabail10]

I understand this a little better now. There are two flags in med_phases_prep_ocn_mod.F90.

1. export_swnet_by_bands is set when compocn is asking for Foxx_swnet_vdr.

2. import_swnet_by_bands is set when compice is sending Fioi_swnet_vdr.

I did add code to CICE to turn off advertising / exporting Fioi_swnet_vdr. However, this does not get rid of Foxx_swnet_vdr. In the code starting at this line:

https://github.com/ESCOMP/CMEPS/blob/9991b7bcfd127ca8dabab079cd5593a7782bd965/mediator/med_phases_prep_ocn_mod.F90#L486

If export_swnet_by_bands = .true., but import_swnet_by_bands is .false., then it still computes the shortwave by bands in the absence of the fluxes from CICE. What I do not understand is why DOCN is asking for Foxx_swnet_vdr. I see no mention of this field in CDEPS, so it must be set in CMEPS somewhere incorrectly. The only time Foxx_swnet_vdr should be requested is when compocn = 'mom'. So, this is not a CICE problem. This is a CMEPS problem.

[dabail10]

Ok. @billsacks and I met to talk about this and the short term fix is a new CICE tag that turns these fields off in prescribed ice mode. I would like a longer term fix for only sending these fields when compocn = 'mom'.