Enable EAMxx CMORization, E3SM Diags and add ts-level vertical regrid

[chengzhuzhang]

Summary

Objectives:

• Fix the fragile input_files.split(".")[0] gate in e3sm_to_cmip.bash that
  excluded EAMxx runs whose input_files doesn't start with cam/eam/eamxx
  (e.g. input_files = "AVERAGE.nmonths_x1" with input_component = "eamxx").
  Replaced with a prc_typ-based gate that honors input_component. Fix [Bug]: fragile component detection before vertical interpolation #820
• Add EAMxx-specific ncremap flags (--ps_nm=${file}/ps, --vrt_in=<L128>)
  inside the e3sm_to_cmip vertical interpolation block, auto-emitted when
  prc_typ == 'eamxx'. vrt_in_file defaults under
  {diagnostics_base_path}/e3sm_to_cmip_data/grids/vert_L128.nc.
• Add an additive, opt-in ts-level vertical regrid path. New [ts] parameters
  vert_remap_vars, vrt_remap_file, vrt_in_file: when set, the ts task
  also writes vertically remapped files to a sibling directory
  post/{component}/{grid}/ts_vert_remap/{frequency}/{ypf}yr/ alongside ts/,
  so users can use native-level and pressure-level data simultaneously.
• Wire e3sm_to_cmip.bash to auto-detect ts_vert_remap/ files (via
  readlink on the input symlinks) and skip its inline regrid for those vars,
  avoiding redundant work.
• Add examples/post.v3.eamxx.cfg demonstrating the EAMxx pipeline end-to-end
  (climo, ts, e3sm_to_cmip, e3sm_diags). Contributed by @zhangshixuan1987
• Decouple e3sm_diags web publish path from subsection case overrides to allow flexibility for eamxx file names: e.g. 1ma_ne30pg2.AVERAGE.nmonths_x1.1995-01-01-00000.nc. web results stays with the top-level case-named path.

Issue resolution:

• Closes [Bug]: fragile component detection before vertical interpolation #820
• Closes [Feature]: EAMxx cmorization via zppy #817

Select one: This pull request is...

• a bug fix: increment the patch version
• a small improvement: increment the minor version
• a new feature: increment the minor version
• an incompatible (non-backwards compatible) API change: increment the major version

Please fill out either the "Small Change" or "Big Change" section (the latter includes the numbered subsections), and delete the other.

Small Change

• To merge, I will use "Squash and merge". That is, this change should be a single commit.
• Logic: I have visually inspected the entire pull request myself.
• Pre-commit checks: All the pre-commits checks have passed.

Big Change

• To merge, I will use "Create a merge commit". That is, this change is large enough to require multiple units of work (i.e., it should be multiple commits).

1. Does this do what we want it to do?

Required:

• Product Management: I have confirmed with the stakeholders that the objectives above are correct and complete.
• Testing: I have added or modified at least one "min-case" configuration file to test this change. Every objective above is represented in at least one cfg.
• Testing: I have considered likely and/or severe edge cases and have included them in testing.

If applicable:

• Testing: this pull request introduces an important feature or bug fix that we must test often. I have updated the weekly-test configuration files, not just a "min-case" one.
• Testing: this pull request adds at least one new possible parameter to the cfg. I have tested using this parameter with and without any other parameter that may interact with it.

2. Are the implementation details accurate & efficient?

Required:

• Logic: I have visually inspected the entire pull request myself.
• Logic: I have left GitHub comments highlighting important pieces of code logic. I have had these code blocks reviewed by at least one other team member.

If applicable:

• Dependencies: This pull request introduces a new dependency. I have discussed this requirement with at least one other team member. The dependency is noted in zppy/conda, not just an import statement.

3. Is this well documented?

Required:

• Documentation: by looking at the docs, a new user could easily understand the functionality introduced by this pull request.

4. Is this code clean?

Required:

• Readability: The code is as simple as possible and well-commented, such that a new team member could understand what's happening.
• Pre-commit checks: All the pre-commits checks have passed.

If applicable:

• Software architecture: I have discussed relevant trade-offs in design decisions with at least one other team member. It is unlikely that this pull request will increase tech debt.

[Copilot]

Pull request overview

Enables EAMxx CMORization and e3sm_diags pipelines end-to-end, replaces the fragile input_files-prefix gate before vertical interpolation with a prc_typ-based gate, and adds an opt-in [ts] vertical-regrid path that writes pressure-level files to a sibling ts_vert_remap/ directory which e3sm_to_cmip then auto-detects to avoid redundant regridding.

Changes:

• Replace fragile input_files.split(".")[0] gate with prc_typ in ('cam','eam','eamxx'); emit EAMxx-specific ncremap flags (--ps_nm, --vrt_in) when prc_typ=='eamxx'; auto-skip vars already vert-remapped by [ts].
• Add [ts] parameters vert_remap_vars/vrt_remap_file/vrt_in_file with EAMxx defaults under diagnostics_base_path, and [ts] template logic to produce a parallel ts_vert_remap/ output tree.
• Switch e3sm_diags web publish path to ${www}/{{ default_case }}/... (decoupled from subsection case overrides) and add examples/post.v3.eamxx.cfg.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
zppy/templates/e3sm_to_cmip.bash	New prc_typ gate, ts_vert_remap symlink overlay, readlink-based skip, and EAMxx ncremap flags.
zppy/templates/ts.bash	New optional vertical regrid loop producing output_plev/ and ts_vert_remap/ sibling destination.
zppy/templates/e3sm_diags.bash	web_dir now uses {{ default_case }} instead of subsection ${case}.
zppy/ts.py	Defaults for new vrt_remap_file/vrt_in_file when vert_remap_vars is set.
zppy/e3sm_to_cmip.py	EAMxx-only default for vrt_in_file.
zppy/e3sm_diags.py	Propagates config['default']['case'] as default_case for the bash template.
zppy/defaults/default.ini	Declares new [ts] and [e3sm_to_cmip] parameters (and __many__ overrides).
tests/test_sections.py	Adds new params to expected section/subsection dicts.
examples/post.v3.eamxx.cfg	New end-to-end EAMxx example (climo, ts, e3sm_to_cmip, e3sm_diags).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[chengzhuzhang]

@zhangshixuan1987 this PR should enable cmorization and e3sm Diags (except dirunal cycle and TC sets) for eamxx. I needed to make changes to both zppy and your example cfg file to make things work. Could you review: https://github.com/E3SM-Project/zppy/blob/f61b71a1980b8800daa095d0f7764b563083da55/examples/post.v3.eamxx.cfg. and let me know if it looks okay? e3sm_digas results can be find here. There was an issue for start and end year for Wheeler Kiladis, I will have a rerun to create it.

[chengzhuzhang]

@zhangshixuan1987 #830 should already be taken care of by this PR, without needing to add additional parameters.

[zhangshixuan1987]

@zhangshixuan1987 #830 should already be taken care of by this PR, without needing to add additional parameters.

@chengzhuzhang Then we can forget about the #830

[chengzhuzhang]

I realized that more work needed for eamxx daily output AVERAGE.ndays_x1, converting to draft..

[zhangshixuan1987]

@zhangshixuan1987 this PR should enable cmorization and e3sm Diags (except dirunal cycle and TC sets) for eamxx. I needed to make changes to both zppy and your example cfg file to make things work. Could you review: https://github.com/E3SM-Project/zppy/blob/f61b71a1980b8800daa095d0f7764b563083da55/examples/post.v3.eamxx.cfg. and let me know if it looks okay? e3sm_digas results can be find here. There was an issue for start and end year for Wheeler Kiladis, I will have a rerun to create it.

Hi @chengzhuzhang, I think the cfg file looks consistent with what I have on my local use. Below are some of my comments for your reference:

1. I think it may be more convenient to combine atm_2d_monthly_180x360_aave and atm_3d_monthly_180x360_aave into a single workflow, but I originally split them to highlight the different extra_vars requirements. For 2D fields, area,landfrac,ocnfrac are mainly included for downstream analysis convenience, whereas 3D fields also require ps,hyai,hyam,hybi,hybm for vertical interpolation. In particular, ps is only needed for 3D vertical interpolation. If the two sections are combined, all extra_vars may be applied to both 2D and 3D variables, meaning unnecessary variables such as ps could be included in every 2D time-series file. This would introduce redundancy and could substantially increase, potentially even double, the file size for 2D variable outputs. My concern here is mainly about the robustness and efficiency of the processing workflow. I think there are two possible solutions: (1) keep the approach I used, which splits the 2D and 3D field processing into separate workflows; or (2) modify the zppy ts workflow to clean up and remove redundant variables after time-series processing. Overall, I think careful consideration is valuable here in terms of saving space.

2. For the e3sm_to_cmip section, I think there is no problem with omitting cmip_vars. However, the downstream workflow should add a step to check how many variables in the cmip_vars list can actually be processed based on the available model output files. Otherwise, the workflow may generate many error messages that could be misleading, especially for users who are not familiar with this part of the code or with what is happening internally.

All other parts look reasonable to me. Thank you @chengzhuzhang for merging this cfg example.