Skip to content

Add pcmdi#25

Merged
forsyth2 merged 27 commits into
mainfrom
add-pcmdi
Oct 8, 2025
Merged

Add pcmdi#25
forsyth2 merged 27 commits into
mainfrom
add-pcmdi

Conversation

@forsyth2
Copy link
Copy Markdown
Collaborator

@forsyth2 forsyth2 commented May 30, 2025 •
edited
Loading

Summary

Objectives:

Issue resolution:

Select one: This pull request is...

  • a bug fix: increment the patch version
  • a small improvement: increment the minor version
  • a new feature: increment the minor version
  • an incompatible (non-backwards compatible) API change: increment the major version

Please fill out either the "Small Change" or "Big Change" section (the latter includes the numbered subsections), and delete the other.

Big Change

  • To merge, I will use "Create a merge commit". That is, this change is large enough to require multiple units of work (i.e., it should be multiple commits).

1. Does this do what we want it to do?

Required:

  • Product Management: I have confirmed with the stakeholders that the objectives above are correct and complete.
  • Testing: I have considered likely and/or severe edge cases and have included them in testing.

2. Are the implementation details accurate & efficient?

Required:

  • Logic: I have visually inspected the entire pull request myself.
  • Logic: I have left GitHub comments highlighting important pieces of code logic. I have had these code blocks reviewed by at least one other team member.

If applicable:

  • Dependencies: This pull request introduces a new dependency. I have discussed this requirement with at least one other team member. The dependency is noted in zppy-interfaces/conda, not just an import statement.

3. Is this well documented?

Required:

  • Documentation: by looking at the docs, a new user could easily understand the functionality introduced by this pull request.

4. Is this code clean?

Required:

  • Readability: The code is as simple as possible and well-commented, such that a new team member could understand what's happening.
  • Pre-commit checks: All the pre-commits checks have passed.

If applicable:

  • Software architecture: I have discussed relevant trade-offs in design decisions with at least one other team member. It is unlikely that this pull request will increase tech debt.

@forsyth2 forsyth2 self-assigned this May 30, 2025
@forsyth2 forsyth2 mentioned this pull request May 30, 2025
Closed
17 tasks
forsyth2
forsyth2 commented Jun 11, 2025
View reviewed changes
Comment thread zppy_interfaces/pcmdi_diags/pcmdi_setup.py
Comment on lines +155 to +157
if not os.path.exists(catalog_path):
print(f"Warning: Catalogue not found at {catalog_path}")
return
Copy link
Copy Markdown
Collaborator Author

@forsyth2 forsyth2 Jun 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy of https://github.com/E3SM-Project/zppy/pull/719/files#r2141246726:

> cd /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_12/v3.LR.amip_0101/post/scripts
> emacs pcmdi_diags_enso_model_vs_obs_2005-2015.o753616
# Warning: Could not extract dates from *.nc
# Warning: Catalogue not found at pcmdi_diags/ts_enso_catalogue.json
# Warning: Catalogue not found at pcmdi_diags/ts_ref_enso_catalogue.json
# KeyError: "Variable key 'psl' not found in observation dictionary."

> emacs pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o753613 
# Warning: Could not extract dates from *.nc
# Warning: Catalogue not found at pcmdi_diags/climo_mean_climate_catalogue.json
# Warning: Catalogue not found at pcmdi_diags/climo_ref_mean_climate_catalogue.json
# ValueError: not enough values to unpack (expected 4, got 1)

> emacs pcmdi_diags_variability_modes_atm_model_vs_obs_2005-2014.o753615 
# Warning: Could not extract dates from *.nc
# Warning: Catalogue not found at pcmdi_diags/ts_variability_modes_atm_catalogue.json
# Warning: Catalogue not found at pcmdi_diags/ts_ref_variability_modes_atm_catalogue.json
# KeyError: 'psl'

> emacs pcmdi_diags_variability_modes_cpl_model_vs_obs_2005-2014.o753614
# Warning: Could not extract dates from *.nc
# Warning: Catalogue not found at pcmdi_diags/ts_variability_modes_cpl_catalogue.json
# Warning: Catalogue not found at pcmdi_diags/ts_ref_variability_modes_cpl_catalogue.json
# KeyError: 'ts'

forsyth2
forsyth2 commented Jun 11, 2025
View reviewed changes
Comment thread zppy_interfaces/pcmdi_diags/pcmdi_enso.py
Comment on lines +203 to +206
if vkey not in obs_dic:
raise KeyError(
f"Variable key '{vkey}' not found in observation dictionary."
)
Copy link
Copy Markdown
Collaborator Author

@forsyth2 forsyth2 Jun 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where pcmdi_diags_enso_model_vs_obs_2005-2015.o753616 in https://github.com/E3SM-Project/zppy/pull/719/files#r2141246726 fails.

forsyth2
forsyth2 commented Jun 11, 2025
View reviewed changes
Comment thread zppy_interfaces/pcmdi_diags/pcmdi_mean_cimate.py Outdated
regions=mean_climate_parameters.regions,
variables=core_parameters.variables,
fig_format="{{figure_format}}",
model_info=tuple("${model_name}".split(".")), # (mip, exp, model, relm)
Copy link
Copy Markdown
Collaborator Author

@forsyth2 forsyth2 Jun 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o753613 in https://github.com/E3SM-Project/zppy/pull/719/files#r2141246726 fails.

forsyth2
forsyth2 commented Jun 11, 2025
View reviewed changes
Comment thread zppy_interfaces/pcmdi_diags/pcmdi_variability_modes.py
##########################################
# from configuration file
varOBS = variability_modes_parameters.vars
refset = core_output.obs_dic[varOBS]["set"]
Copy link
Copy Markdown
Collaborator Author

@forsyth2 forsyth2 Jun 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where pcmdi_diags_variability_modes_atm_model_vs_obs_2005-2014.o753615 in https://github.com/E3SM-Project/zppy/pull/719/files#r2141246726 fails

Copy link
Copy Markdown
Collaborator Author

@forsyth2 forsyth2 Jun 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where pcmdi_diags_variability_modes_cpl_model_vs_obs_2005-2014.o753614 in https://github.com/E3SM-Project/zppy/pull/719/files#r2141246726 fails.

Copy link
Copy Markdown
Collaborator Author

@forsyth2 forsyth2 Jun 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In E3SM-Project/zppy@ca63656, which is the commit where we pull out pcmdi.py from zppy (to zppy-interfaces), we have:

varOBS  = '{{vars}}'
refset  = obs_dic[varOBS]['set']

In this file zppy_interfaces/pcmdi_diags/pcmdi_variability_modes.py, we have:

    varOBS = variability_modes_parameters.vars
    refset = core_output.obs_dic[varOBS]["set"]

and

        # self.vars is distinct from the list version in CoreParameters
        self.vars: str = args["vars"]

That is,

varOBS is a string, not a list of strings, listing the variables. Yet, the key errors are:

# KeyError: 'psl'
# KeyError: 'ts'

suggesting varOBS is indeed just one variable long.

@forsyth2 forsyth2 force-pushed the add-pcmdi branch 2 times, most recently from 9d96017 to a1a03e1 Compare July 3, 2025 21:42
@forsyth2 forsyth2 mentioned this pull request Jul 4, 2025
Merged
7 tasks
@forsyth2
Copy link
Copy Markdown
Collaborator Author

forsyth2 commented Jul 4, 2025

I was hoping to remove the duplicate logger lines to make debugging easier (#29) but ran into an issue with environments (#30)

forsyth2
forsyth2 commented Jul 4, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator Author

@forsyth2 forsyth2 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still a work-in-progress but I have had some success. Here's my latest status update. (Mostly a reminder note for myself, but @zhangshixuan1987 @chengzhuzhang, you may be interested).

Understanding commits

zppy side:

zppy-interfaces side

  • 0889cad: largely just moves code from zppy to zppy-interfaces, but this required some changes such as adding parameter classes.
  • a1a03e1: My changes

Why are all these changes necessary?

Some of these changes are just to make the code cleaner. Organizing code into different files, adding type annotations, adding log lines, etc. makes the code easier to understand, and by extension, to debug.

The initial pcmdi_diags work was done over a long time period while zppy itself underwent a substantial number of changes. That means there are a lot of places the code could be out of date. The cleaner code helps make clear what errors are occuring.

Results so far

My latest run uses the unique_id_16 tag in the file path. I've been debugging in that file path for a while now. Results:

cd /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_16/v3.LR.amip_0101/post/scripts/
grep "OK" *status
# e3sm_to_cmip_atm_monthly_180x360_aave_2005-2009-0005.status:OK
# e3sm_to_cmip_atm_monthly_180x360_aave_2010-2014-0005.status:OK
# pcmdi_diags_mean_climate_model_vs_obs_2005-2014.status:OK
# pcmdi_diags_variability_modes_atm_model_vs_obs_2005-2014.status:OK
# pcmdi_diags_variability_modes_cpl_model_vs_obs_2005-2014.status:OK
# ts_atm_monthly_180x360_aave_2005-2009-0005.status:OK
# ts_atm_monthly_180x360_aave_2010-2014-0005.status:OK

So, we can see I've now gotten some of the pcmdi_diags functionality to work. For example the web page has:

And we do actually have figures produced, e.g., this one.

Now for remaining debugging:

  1. pcmdi_diags_enso_model_vs_obs_2005-2015.o785853 shows:
input_template=pcmdi_diags/%(output_type)/%(metric_type)/e3sm/amip/v20250702; if the directories based on this template are empty, lstcmd=['enso_driver.py -p parameterfile.py --metricsCollection ENSO_perf --case_id v20250702', 'enso_driver.py -p parameterfile.py --metricsCollection ENSO_proc --case_id v20250702', 'enso_driver.py -p parameterfile.py --metricsCollection ENSO_tel --case_id v20250702'] failed to produce output.

fpaths=[], self.input_dir=pcmdi_diags/%(output_type)/enso_metric/e3sm/amip/v20250702, template=/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_16/v3.LR.amip_0101/post/scripts/tmp.785853.9zmH/pcmdi_diags/graphics/enso_metric/e3sm/amip/v20250702/ENSO_perf/*.png, files in template=[]
  1. pcmdi_diags_synthetic_plots_model_vs_obs.o786426 (note I removed enso form what to include in it) shows:
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-pcmdi-20250529/lib/python3.10/site-packages/zppy_interfaces/pcmdi_diags/synthetic_plots/synthetic_metrics_plotter.py", line 113, in generate
    self._handle_mean_climate(metric)
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-pcmdi-20250529/lib/python3.10/site-packages/zppy_interfaces/pcmdi_diags/synthetic_plots/synthetic_metrics_plotter.py", line 137, in _handle_mean_climate
    merge_lib = collector.collect()
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-pcmdi-20250529/lib/python3.10/site-packages/zppy_interfaces/pcmdi_diags/synthetic_plots/clim_metrics_reader.py", line 140, in collect
    model_lib = self._process_test_model(test_name, model_name)
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-pcmdi-20250529/lib/python3.10/site-packages/zppy_interfaces/pcmdi_diags/synthetic_plots/clim_metrics_reader.py", line 121, in _process_test_model
    model_lib = self._load_clim_metrics_from_files(valid_model_files)
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-pcmdi-20250529/lib/python3.10/site-packages/zppy_interfaces/pcmdi_diags/synthetic_plots/clim_metrics_reader.py", line 50, in _load_clim_metrics_from_files
    lib = Metrics(file_paths)
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-pcmdi-20250529/lib/python3.10/site-packages/pcmdi_metrics/graphics/share/Metrics.py", line 57, in __init__
    ) = read_mean_clim_json_files(files, mip=mip)
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-pcmdi-20250529/lib/python3.10/site-packages/pcmdi_metrics/graphics/share/read_json_mean_clim.py", line 66, in read_mean_clim_json_files
    regions_all.extend(extract_region(var, results_dict[var]))
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-pcmdi-20250529/lib/python3.10/site-packages/pcmdi_metrics/graphics/share/read_json_mean_clim.py", line 124, in extract_region
    region_list, stat_list = extract_region_stat(var, results_dict_var)
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-pcmdi-20250529/lib/python3.10/site-packages/pcmdi_metrics/graphics/share/read_json_mean_clim.py", line 141, in extract_region_stat
    list(results_dict_var["RESULTS"][model_list[0]]["default"][run_list[0]].keys())
IndexError: list index out of range

Failed to handle metric=mean_climate

Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-pcmdi-20250529/lib/python3.10/site-packages/zppy_interfaces/pcmdi_diags/synthetic_plots/synthetic_metrics_plotter.py", line 116, in generate
    self._handle_variability_modes(metric)
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-pcmdi-20250529/lib/python3.10/site-packages/zppy_interfaces/pcmdi_diags/synthetic_plots/synth\
etic_metrics_plotter.py", line 165, in _handle_variability_modes
    merge_lib, mode_season_list = reader.collect_metrics()
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-pcmdi-20250529/lib/python3.10/site-packages/zppy_interfaces/pcmdi_diags/synthetic_plots/movs_metrics_reader.py", line 55, in collect_metrics
    raise FileNotFoundError(
FileNotFoundError: No Synthetic MoVs Metrics Data For v3.LR.amip_0101, Aborting.

Failed to handle metric=variability_modes

RuntimeError: No synthetic metrics plots could be generated.

(Note I've made some code changes since the last run, so line numbers may not match up with the latest code)

How to run

lcrc_conda # My alias to set up conda

cd ~/ez/zppy-interfaces
git branch # Should be `add-pcmdi`
pre-commit run --all-files
git add -A
git status # Should have no un-committed changes
# To create conda env in the first place:
# conda clean --all --y
# conda env create -f conda/dev.yml -n zi-pcmdi-20250529
conda activate zi-pcmdi-20250529 
# Note: this environment is currently hard-coded in zppy/templates/pcmdi_diags.bash
# That's because that file requires us to switch between Unified and an environment with PCMDI.
# This will of course have to be parameterized before merging.
pip install . 

# zppy side
cd ~/ez/zppy
git branch # Should be `zppy_pcmdi_202505`
# Update unique_id in examples/post.v3.LR.amip.0101.cfg
# (It appears in output + www)
# Note: only do this if re-running from scratch
# This is not needed if only making changes to zppy-interfaces and re-testing a job.
pre-commit run --all-files
git add -A
git status # Should have no un-committed changes
# To create conda env in the first place:
# conda clean --all --y
# conda env create -f conda/dev.yml -n zppy-pr719-dev-20250612 
conda activate zppy-pr719-dev-20250612 
pip install . 
zppy -c examples/post.v3.LR.amip.0101.cfg
# WAIT ~5 min
# Update unique_id below, as needed
cd /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_16/v3.LR.amip_0101/post/scripts/ && grep -v "OK" *status
# Visit https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_pr719_output/unique_id_16/v3.LR.amip_0101/pcmdi_diags/

Comment thread zppy_interfaces/pcmdi_diags/pcmdi_enso.py
# now start enso driver
lstcmd = generate_enso_cmds(enso_parameters.enso_groups, core_parameters.case_id)
logger.info(
f"input_template={core_output.input_template}; if the directories based on this template are empty, lstcmd={lstcmd} failed to produce output."
Copy link
Copy Markdown
Collaborator Author

@forsyth2 forsyth2 Jul 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We hit this line in error 1 above

Comment thread zppy_interfaces/pcmdi_diags/pcmdi_enso.py

if not fpaths:
logger.error(
f"fpaths={fpaths}, self.input_dir={self.input_dir}, template={os.path.abspath(template)}, files in template={os.listdir(os.path.join(fdir, group))}"
Copy link
Copy Markdown
Collaborator Author

@forsyth2 forsyth2 Jul 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We hit this line in error 1 above.

Comment thread zppy_interfaces/pcmdi_diags/synthetic_plots/clim_metrics_reader.py
logger.info("file_paths=")
for i, fp in enumerate(file_paths):
logger.info(f"{i}. {fp}")
lib = Metrics(file_paths)
Copy link
Copy Markdown
Collaborator Author

@forsyth2 forsyth2 Jul 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This call to PCMDI doesn't work (see error 2 above), despite file_paths being non-empty.

Comment thread zppy_interfaces/pcmdi_diags/synthetic_plots/movs_metrics_reader.py
Comment on lines +55 to +57
raise FileNotFoundError(
f"No Synthetic MoVs Metrics Data For {model_name}, Aborting."
)
Copy link
Copy Markdown
Collaborator Author

@forsyth2 forsyth2 Jul 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We hit this line in Error 2 above.

Comment thread zppy_interfaces/pcmdi_diags/synthetic_plots/synthetic_metrics_plotter.py Outdated
at_least_one_success = True
except Exception:
traceback.print_exc()
logger.error(f"Failed to handle metric={metric}")
Copy link
Copy Markdown
Collaborator Author

@forsyth2 forsyth2 Jul 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We hit this line twice in Error 2 above.

Comment thread zppy_interfaces/pcmdi_diags/synthetic_plots/synthetic_metrics_plotter.py
traceback.print_exc()
logger.error(f"Failed to handle metric={metric}")
if not at_least_one_success:
raise RuntimeError("No synthetic metrics plots could be generated.")
Copy link
Copy Markdown
Collaborator Author

@forsyth2 forsyth2 Jul 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We hit this line in Error 2 above.

@chengzhuzhang chengzhuzhang self-requested a review July 11, 2025 18:19
forsyth2
forsyth2 commented Jul 25, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator Author

@forsyth2 forsyth2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zhangshixuan1987 I've been trying to debug error 1 of #25 (review), that is, the failing enso run. I've added quite a few checks but I can't seem to get enso_driver.py to produce desired output.

Relevant output from /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_19/v3.LR.amip_0101/post/scripts/pcmdi_diags_enso_model_vs_obs_2005-2015.o818267:

Logs 
2025-07-25 14:14:32,255 [INFO]: pcmdi_setup.py(_save_catalogue:170) >> Saving catalogue pcmdi_diags/ts_enso_catalogue.json, absolute path /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_19/v3.LR.amip_0101/post/scripts/tmp.818267.4D7q/pcmdi_diags/ts_enso_catalogue.json
2025-07-25 14:14:32,257 [INFO]: pcmdi_setup.py(_save_catalogue:170) >> Saving catalogue pcmdi_diags/ts_ref_enso_catalogue.json, absolute path /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_19/v3.LR.amip_0101/post/scripts/tmp.818267.4D7q/pcmdi_diags/ts_ref_enso_catalogue.json
2025-07-25 14:14:34,884 [INFO]: pcmdi_setup.py(set_up:340) >> out_path=model_vs_obs/%(group_type)
2025-07-25 14:14:34,885 [INFO]: pcmdi_enso.py(build_enso_obsvar_catalog:282) >> [INFO] Observation catalogue written to: obs_catalogue.json
2025-07-25 14:14:34,886 [INFO]: pcmdi_enso.py(build_enso_obsvar_landmask:323) >> [INFO] Landmask mapping written to: obs_landmask.json
2025-07-25 14:14:34,887 [INFO]: pcmdi_enso.py(check_enso_input:335) >> Symlinking cmip-standard PRECT to observational variable name pr, if present
2025-07-25 14:14:34,887 [INFO]: pcmdi_enso.py(check_enso_input:335) >> Symlinking cmip-standard ts to observational variable name sst, if present
2025-07-25 14:14:34,888 [INFO]: pcmdi_enso.py(check_enso_input:335) >> Symlinking cmip-standard si10 to observational variable name sfcWind, if present
2025-07-25 14:14:34,888 [INFO]: pcmdi_enso.py(check_enso_input:335) >> Symlinking cmip-standard tauu to observational variable name taux, if present
2025-07-25 14:14:34,889 [INFO]: pcmdi_enso.py(check_enso_input:335) >> Symlinking cmip-standard tauv to observational variable name tauy, if present
2025-07-25 14:14:34,891 [INFO]: pcmdi_enso.py(check_enso_input:335) >> Symlinking cmip-standard toa_cre_lw_mon to observational variable name rltcre, if present
2025-07-25 14:14:34,891 [INFO]: pcmdi_enso.py(check_enso_input:335) >> Symlinking cmip-standard toa_cre_sw_mon to observational variable name rstcre, if present
2025-07-25 14:14:34,891 [INFO]: pcmdi_enso.py(check_enso_input:335) >> Symlinking cmip-standard toa_net_all_mon to observational variable name rtmt, if present
2025-07-25 14:14:34,891 [INFO]: pcmdi_enso.py(generate_enso_cmds:378) >> Commands will be run from current_dir=/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_19/v3.LR.amip_0101/post/scripts/tmp.818267.4D7q
2025-07-25 14:14:34,892 [INFO]: pcmdi_enso.py(main:173) >> input_template=pcmdi_diags/%(output_type)/%(metric_type)/e3sm/amip/v20250725; if the directories based on this template are empty, lstcmd=['enso_driver.py -p parameterfile.py --metricsCollection ENSO_perf --case_id v20250725', 'enso_driver.py -p parameterfile.py --metricsCollection ENSO_proc --case_id v20250725', 'enso_driver.py -p parameterfile.py --metricsCollection ENSO_tel --case_id v20250725'] failed to produce output.
2025-07-25 14:14:34,892 [INFO]: pcmdi_enso.py(main:177) >> Running parallel jobs for ['enso_driver.py -p parameterfile.py --metricsCollection ENSO_perf --case_id v20250725', 'enso_driver.py -p parameterfile.py --metricsCollection ENSO_proc --case_id v20250725', 'enso_driver.py -p parameterfile.py --metricsCollection ENSO_tel --case_id v20250725']
2025-07-25 14:14:34,910 [INFO]: utils.py(run_parallel_jobs:61) >> Running 3 subprocesses...
2025-07-25 14:14:55,267 [INFO]: pcmdi_enso.py(check_enso_output:390) >> Checking ENSO output.
2025-07-25 14:14:55,267 [INFO]: pcmdi_enso.py(check_enso_output:393) >> Command 1 finished:
2025-07-25 14:14:55,267 [INFO]: pcmdi_enso.py(check_enso_output:394) >> STDOUT: mip: e3sm
exp: amip
models: ['v3-LR']
realization:  0101
mc_name: ENSO_perf
outdir: pcmdi_diags/%(output_type)/enso_metric/e3sm/amip/v20250725/ENSO_perf
netcdf_path: pcmdi_diags/diagnostic_results/enso_metric/e3sm/amip/v20250725/ENSO_perf
debug: False
obs_cmor: True
obs_cmor_path: ts_ref
egg_pth: /gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-pcmdi-20250529/share/pmp
output directory for graphics:pcmdi_diags/graphics/enso_metric/e3sm/amip/v20250725/ENSO_perf
output directory for diagnostic_results:pcmdi_diags/diagnostic_results/enso_metric/e3sm/amip/v20250725/ENSO_perf
output directory for metrics_results:pcmdi_diags/metrics_results/enso_metric/e3sm/amip/v20250725/ENSO_perf
list_variables: ['pr', 'sst', 'taux']
list_obs: ['ERA5']
PMPdriver: dict_obs readin end
Process start:Fri Jul 25 14:14:54 2025
models: ['v3-LR']
 ----- model:  v3-LR  ---------------------
PMPdriver: var loop start for model  v3-LR
realization: 0101
run_in_modpath: 3
PMPdriver: model loop end
Process end: Fri Jul 25 14:14:54 2025
2025-07-25 14:14:55,285 [INFO]: pcmdi_enso.py(check_enso_output:395) >> STDERR: 
2025-07-25 14:14:55,285 [INFO]: pcmdi_enso.py(check_enso_output:396) >> Return code: 0
2025-07-25 14:14:55,288 [INFO]: pcmdi_enso.py(check_vars:456) >> All requested variables ['pr', 'sst', 'taux'] found in directory /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_19/v3.LR.amip_0101/post/scripts/tmp.818267.4D7q/ts
2025-07-25 14:14:55,288 [ERROR]: pcmdi_enso.py(check_output_dirs:480) >> graphics output directory is empty: /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_19/v3.LR.amip_0101/post/scripts/tmp.818267.4D7q/pcmdi_diags/graphics/enso_metric/e3sm/amip/v20250725/ENSO_perf
2025-07-25 14:14:55,289 [ERROR]: pcmdi_enso.py(check_output_dirs:480) >> diagnostic_results output directory is empty: /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_19/v3.LR.amip_0101/post/scripts/tmp.818267.4D7q/pcmdi_diags/diagnostic_results/enso_metric/e3sm/amip/v20250725/ENSO_perf
2025-07-25 14:14:55,289 [ERROR]: pcmdi_enso.py(check_output_dirs:480) >> metrics_results output directory is empty: /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_19/v3.LR.amip_0101/post/scripts/tmp.818267.4D7q/pcmdi_diags/metrics_results/enso_metric/e3sm/amip/v20250725/ENSO_perf
2025-07-25 14:14:55,289 [ERROR]: pcmdi_enso.py(check_enso_output:401) >> Command 1 failed to produce expected output directories.
2025-07-25 14:14:55,289 [INFO]: pcmdi_enso.py(check_enso_output:393) >> Command 2 finished:
2025-07-25 14:14:55,289 [INFO]: pcmdi_enso.py(check_enso_output:394) >> STDOUT: mip: e3sm
exp: amip
models: ['v3-LR']
realization:  0101
mc_name: ENSO_proc
outdir: pcmdi_diags/%(output_type)/enso_metric/e3sm/amip/v20250725/ENSO_proc
netcdf_path: pcmdi_diags/diagnostic_results/enso_metric/e3sm/amip/v20250725/ENSO_proc
debug: False
obs_cmor: True
obs_cmor_path: ts_ref
egg_pth: /gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-pcmdi-20250529/share/pmp
output directory for graphics:pcmdi_diags/graphics/enso_metric/e3sm/amip/v20250725/ENSO_proc
output directory for diagnostic_results:pcmdi_diags/diagnostic_results/enso_metric/e3sm/amip/v20250725/ENSO_proc
output directory for metrics_results:pcmdi_diags/metrics_results/enso_metric/e3sm/amip/v20250725/ENSO_proc
list_variables: ['ssh', 'sst', 'taux', 'thf']
list_obs: ['ERA5']
Observation dataset ERA5 is not given for variable thf
PMPdriver: dict_obs readin end
Process start:Fri Jul 25 14:14:54 2025
models: ['v3-LR']
 ----- model:  v3-LR  ---------------------
PMPdriver: var loop start for model  v3-LR
realization: 0101
run_in_modpath: 3
PMPdriver: model loop end
Process end: Fri Jul 25 14:14:54 2025
2025-07-25 14:14:55,289 [INFO]: pcmdi_enso.py(check_enso_output:395) >> STDERR: 
2025-07-25 14:14:55,289 [INFO]: pcmdi_enso.py(check_enso_output:396) >> Return code: 0
2025-07-25 14:14:55,290 [ERROR]: pcmdi_enso.py(check_vars:450) >> Variables missing data: ['ssh', 'thf'] in directory /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_19/v3.LR.amip_0101/post/scripts/tmp.818267.4D7q/ts
2025-07-25 14:14:55,290 [ERROR]: pcmdi_enso.py(check_vars:453) >> Full contents of /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_19/v3.LR.amip_0101/post/scripts/tmp.818267.4D7q/ts: ['e3sm.amip.v3-LR.0101.Amon.rsds.200501-201512.nc', 'ts_files.txt', 'tauu_files.txt', 'rsds_files.txt', 'rlut_files.txt', 'e3sm.amip.v3-LR.0101.Amon.rlut.200501-201512.nc', 'tas_files.txt', 'e3sm.amip.v3-LR.0101.Amon.prsn.200501-201512.nc', 'tauv_files.txt', 'psl_files.txt', 'e3sm.amip.v3-LR.0101.Amon.hfls.200501-201512.nc', 'hfls_files.txt', 'e3sm.amip.v3-LR.0101.Amon.psl.200501-201512.nc', 'e3sm.amip.v3-LR.0101.Amon.rlus.200501-201512.nc', 'e3sm.amip.v3-LR.0101.Amon.rsdt.200501-201512.nc', 'e3sm.amip.v3-LR.0101.Amon.ts.200501-201512.nc', 'e3sm.amip.v3-LR.0101.Amon.hfss.200501-201512.nc', 'e3sm.amip.v3-LR.0101.Amon.taux.200501-201512.nc', 'sst_files.txt', 'rsdt_files.txt', 'e3sm.amip.v3-LR.0101.Amon.sst.200501-201512.nc', 'e3sm.amip.v3-LR.0101.Amon.tauu.200501-201512.nc', 'hfss_files.txt', 'taux_files.txt', 'prsn_files.txt', 'e3sm.amip.v3-LR.0101.Amon.pr.200501-201512.nc', 'e3sm.amip.v3-LR.0101.Amon.tas.200501-201512.nc', 'e3sm.amip.v3-LR.0101.Amon.rlds.200501-201512.nc', 'rlus_files.txt', 'e3sm.amip.v3-LR.0101.Amon.tauy.200501-201512.nc', 'pr_files.txt', 'rlds_files.txt', 'tauy_files.txt', 'e3sm.amip.v3-LR.0101.Amon.tauv.200501-201512.nc']
2025-07-25 14:14:55,293 [ERROR]: pcmdi_enso.py(check_enso_output:398) >> Command 2 failed to produce expected variables.
2025-07-25 14:14:55,293 [ERROR]: pcmdi_enso.py(check_output_dirs:480) >> graphics output directory is empty: /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_19/v3.LR.amip_0101/post/scripts/tmp.818267.4D7q/pcmdi_diags/graphics/enso_metric/e3sm/amip/v20250725/ENSO_proc
2025-07-25 14:14:55,293 [ERROR]: pcmdi_enso.py(check_output_dirs:480) >> diagnostic_results output directory is empty: /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_19/v3.LR.amip_0101/post/scripts/tmp.818267.4D7q/pcmdi_diags/diagnostic_results/enso_metric/e3sm/amip/v20250725/ENSO_proc
2025-07-25 14:14:55,294 [ERROR]: pcmdi_enso.py(check_output_dirs:480) >> metrics_results output directory is empty: /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_19/v3.LR.amip_0101/post/scripts/tmp.818267.4D7q/pcmdi_diags/metrics_results/enso_metric/e3sm/amip/v20250725/ENSO_proc
2025-07-25 14:14:55,294 [ERROR]: pcmdi_enso.py(check_enso_output:401) >> Command 2 failed to produce expected output directories.
2025-07-25 14:14:55,294 [INFO]: pcmdi_enso.py(check_enso_output:393) >> Command 3 finished:
2025-07-25 14:14:55,294 [INFO]: pcmdi_enso.py(check_enso_output:394) >> STDOUT: mip: e3sm
exp: amip
models: ['v3-LR']
realization:  0101
mc_name: ENSO_tel
outdir: pcmdi_diags/%(output_type)/enso_metric/e3sm/amip/v20250725/ENSO_tel
netcdf_path: pcmdi_diags/diagnostic_results/enso_metric/e3sm/amip/v20250725/ENSO_tel
debug: False
obs_cmor: True
obs_cmor_path: ts_ref
egg_pth: /gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-pcmdi-20250529/share/pmp
output directory for graphics:pcmdi_diags/graphics/enso_metric/e3sm/amip/v20250725/ENSO_tel
output directory for diagnostic_results:pcmdi_diags/diagnostic_results/enso_metric/e3sm/amip/v20250725/ENSO_tel
output directory for metrics_results:pcmdi_diags/metrics_results/enso_metric/e3sm/amip/v20250725/ENSO_tel
list_variables: ['pr', 'sst']
list_obs: ['ERA5']
PMPdriver: dict_obs readin end
Process start:Fri Jul 25 14:14:54 2025
models: ['v3-LR']
 ----- model:  v3-LR  ---------------------
PMPdriver: var loop start for model  v3-LR
realization: 0101
run_in_modpath: 3
PMPdriver: model loop end
Process end: Fri Jul 25 14:14:54 2025
2025-07-25 14:14:55,294 [INFO]: pcmdi_enso.py(check_enso_output:395) >> STDERR: 
2025-07-25 14:14:55,294 [INFO]: pcmdi_enso.py(check_enso_output:396) >> Return code: 0
2025-07-25 14:14:55,294 [INFO]: pcmdi_enso.py(check_vars:456) >> All requested variables ['pr', 'sst'] found in directory /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_19/v3.LR.amip_0101/post/scripts/tmp.818267.4D7q/ts
2025-07-25 14:14:55,294 [ERROR]: pcmdi_enso.py(check_output_dirs:480) >> graphics output directory is empty: /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_19/v3.LR.amip_0101/post/scripts/tmp.818267.4D7q/pcmdi_diags/graphics/enso_metric/e3sm/amip/v20250725/ENSO_tel
2025-07-25 14:14:55,302 [ERROR]: pcmdi_enso.py(check_output_dirs:480) >> diagnostic_results output directory is empty: /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_19/v3.LR.amip_0101/post/scripts/tmp.818267.4D7q/pcmdi_diags/diagnostic_results/enso_metric/e3sm/amip/v20250725/ENSO_tel
2025-07-25 14:14:55,302 [ERROR]: pcmdi_enso.py(check_output_dirs:480) >> metrics_results output directory is empty: /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_19/v3.LR.amip_0101/post/scripts/tmp.818267.4D7q/pcmdi_diags/metrics_results/enso_metric/e3sm/amip/v20250725/ENSO_tel
2025-07-25 14:14:55,302 [ERROR]: pcmdi_enso.py(check_enso_output:401) >> Command 3 failed to produce expected output directories.
2025-07-25 14:14:55,302 [ERROR]: pcmdi_enso.py(main:182) >> Execution failed: ENSO output check failed.
Land mask estimated using regionmask method.
Land mask estimated using regionmask method.
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-pcmdi-20250529/bin/zi-pcmdi-enso", line 8, in <module>
    sys.exit(main())
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-pcmdi-20250529/lib/python3.10/site-packages/zppy_interfaces/pcmdi_diags/pcmdi_enso.py", line 183, in main
    raise e
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-pcmdi-20250529/lib/python3.10/site-packages/zppy_interfaces/pcmdi_diags/pcmdi_enso.py", line 180, in main
    check_enso_output(results)
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-pcmdi-20250529/lib/python3.10/site-packages/zppy_interfaces/pcmdi_diags/pcmdi_enso.py", line 406, in check_enso_output
    raise RuntimeError("ENSO output check failed.")
RuntimeError: ENSO output check failed.

Parameters from /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_19/v3.LR.amip_0101/post/scripts/tmp.818267.4D7q/parameterfile.py:

Parameter file 
import os
import sys
import json
#####################
# Basic Information
#####################

start_yr = int('2005')
end_yr = int('2015')
num_years = end_yr - start_yr + 1
period = f"{start_yr:04d}01-{end_yr:04d}12"

model_parts = 'e3sm.amip.v3-LR.0101'.split('.')
mip, exp, product, realm = model_parts[:4]

##############################################
# Configuration Shared with PCMDI Diagnostics
##############################################

# Whether to generate NetCDF outputs for observations and model results
nc_out_obs = True
nc_out_model = True

# Output file extension: use .nc if either output is enabled,
# otherwise default to .xml
ext = ".nc" if nc_out_model or nc_out_obs else ".xml"

# User annotation and debug flag
user_notes = 'Provenance and results'
debug = False

# Enable plot generation for model and observation
plot_model = True
plot_obs = True  # optional

# Execution mode and output format
run_type = 'model_vs_obs'
figure_format = 'png'

# Save interpolated model climatologies?
save_test_clims = True

# Save all metrics results in a single file?
# Set to 'n' as metrics are computed per variable
metrics_in_single_file = 'n'

# Custom values for land/sea masking
regions_values = {
    "land": 100.0,
    "ocean": 0.0
}

# Template path for land/sea mask file (fixed input)
modpath_lf = os.path.join(
    'fixed',
    'sftlf.put_model_here.nc'
)






# Parameter Setup for ENSO Metrics
modnames = [product]
realization = realm

modpath = os.path.join(
    'ts',
    '{}.{}.put_model_here.%(realization).{}.%(variable).{}.nc'.format(
        mip, exp, 'Amon', period
    )
)

# Observation/Reference settings
obs_cmor = True
obs_cmor_path = 'ts_ref'
obs_catalogue = 'obs_catalogue.json'

# Land/Sea mask for reference data
reference_data_lf_path = json.load(open('obs_landmask.json'))

# Metrics collection type (e.g., ENSO_perf, ENSO_tel, ENSO_proc)
# Defined externally via metricsCollection

# Output directory structure
results_dir = os.path.join(
    'pcmdi_diags',
    '%(output_type)',
    'enso_metric',
    '%(mip)',
    '%(exp)',
    'v20250725',
    '%(metricsCollection)',
)

# Output filenames for JSON and NetCDF
json_name = "%(mip)_%(exp)_%(metricsCollection)_v20250725_put_model_here_%(realization)"

netcdf_name = json_name

Since you have more familiarity with using PCMDI, are you able to offer any insights into what could be going wrong? I'm fairly confident I'm feeding the correct inputs to enso_driver.py but it's possible something might be incorrect before that call.

@forsyth2
Copy link
Copy Markdown
Collaborator Author

forsyth2 commented Jul 25, 2025

Parameters from /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_19/v3.LR.amip_0101/post/scripts/tmp.818267.4D7q/parameterfile.py:

@zhangshixuan1987 I notice a lot of parameters here are meant to be expanded/substituted. Does PCMDI do that or is there a mistake in the logic that writes the parameterfile.py? E.g, should all substitutions be realized in the parameter file?

@forsyth2
Copy link
Copy Markdown
Collaborator Author

forsyth2 commented Jul 25, 2025

should all substitutions be realized in the parameter file?

The answer to this is apparently "no" since parameterfile.py versions from working jobs (like variability_modes) also have substitution strings.

forsyth2
forsyth2 commented Jul 25, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator Author

@forsyth2 forsyth2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zhangshixuan1987 I also tried to take a look at the second error in #25 (review) -- the synthetic plots error

I've found that it fails for both mean_climate and variability_modes, meaning that even though those jobs completed successfully, they appear to have not produced sufficient output.

Comment thread zppy_interfaces/pcmdi_diags/synthetic_plots/clim_metrics_reader.py
Comment on lines +50 to +67
"""
FAILURE:

list(results_dict_var["RESULTS"][model_list[0]]["default"][run_list[0]].keys())
IndexError: list index out of range

cat /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_pr719_output/unique_id_21/v3.LR.amip_0101/pcmdi_diags/model_vs_obs/metrics_data/mean_climate/rlus.2.5x2.5.e3sm.amip.v3-LR_0101.v20250725.json

"RESULTS": {
"v3-LR": {
"default": {
"source": "ceres_ebaf_v4_1"
}
}
},

SYNTHETIC PLOTS ERROR #1: "source" is supposed to be a dictionary itself, even though mean_climate job completed successfully!
"""
Copy link
Copy Markdown
Collaborator Author

@forsyth2 forsyth2 Jul 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mean_climate error

Comment thread zppy_interfaces/pcmdi_diags/synthetic_plots/utils.py
Comment on lines +36 to +46
FAILURE

No files found in /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_pr719_output/unique_id_21/v3.LR.amip_0101/pcmdi_diags/model_vs_obs/metrics_data/variability_modes/*/* that match pattern: var_mode_*.json

ls /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_pr_719_output/unique_id_21/v3.LR.amip_0101/pcmdi_diags/model_vs_obs/metrics_data/variability_modes/
AMO NAM NAO NPGO NPO PDO PNA PSA1 PSA2 SAM

ls /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_pr719_output/unique_id_21/v3.LR.amip_0101/pcmdi_diags/model_vs_obs/metrics_data/variability_modes/AMO/HadISST2/
AMO_ts_EOF1_monthly_obs_1869-2014.nc AMO_ts_EOF1_yearly_obs_1869-2014.nc

SYNTHETIC PLOTS ERROR #2: No json files produced by variability modes, even though those jobs completed successfully!
Copy link
Copy Markdown
Collaborator Author

@forsyth2 forsyth2 Jul 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

variability_modes error

@forsyth2
Copy link
Copy Markdown
Collaborator Author

forsyth2 commented Jul 25, 2025

Updated action item list from E3SM-Project/zppy#719 (comment)

  • Get the example cfg to actually work (on all 4 subtasks), with shorter test year periods
    • mean_climate and variability_modes produce plots and finish successfully, but synthetic_plots appears to fail because of insufficient input from these
    • enso doesn't appear to be producing any output.
    • @zhangshixuan1987 Perhaps it would be useful to have a meeting to go over these issues and/or do pair programming to resolve them. Thanks!
  • Run example cfg on original full year set
  • Run integration tests to make sure other tasks don't break
  • Clean up parameters (e.g., make sure we're only introducing parameters we really need)
  • Add unit/integration tests for this PR

@forsyth2
Copy link
Copy Markdown
Collaborator Author

forsyth2 commented Jul 30, 2025

@zhangshixuan1987 Here's everything relevant to trying to reproduce/fix my errors. Thanks! And let me know if you run into issues at any of the steps.

Let's see what's missing or going wrong in the first place

Example cfg is shared in E3SM-Project/zppy#656 (comment)

Note that this example ran on more years of output. I've been testing on a smaller year set, to allow for faster turnaround times.

# File path to that output
cd /lcrc/group/e3sm/public_html/diagnostic_output/ac.szhang/e3sm-pcmdi/v3.LR.amip_0101/pcmdi_diags/viewer/
ls
# diag_data.html  e3sm_pmp_logo.png  index.html  methodology.html
cd ..
ls
# model_vs_obs  viewer
cd model_vs_obs/
ls
# CLIM_patttern  ENSO_metric  ERROR_metric  metrics_data  MOV_metric

# Let's compare that to my most recent output
cd /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_pr719_output/unique_id_21/v3.LR.amip_0101/pcmdi_diags/
ls 
# model_vs_obs
# NOTICE: No viewer; I presume that's because the synthetic plots task is failing
cd model_vs_obs/
ls
# metrics_data  MOV_metric
# NOTICE: Missing CLIM_pattern, ENSO_metric, ERROR_metric
cd MOV_metric
ls
# MOV_eoftest  MOV_pattern  MOV_telecon
ls /lcrc/group/e3sm/public_html/diagnostic_output/ac.szhang/e3sm-pcmdi/v3.LR.amip_0101/pcmdi_diags/model_vs_obs/MOV_metric
# MOV_compose  MOV_eoftest  MOV_pattern  MOV_telecon
# NOTICE: Shixuan's results have one more directory: MOV_compose

The errors I'm encountering are the ones described at:

  • The PCMDI enso call doesn't seem to be producing output
  • Synthetic plots appear to be missing input data from both mean_climate and MOV, despite those subtasks completing "successfully".

Instructions to run as I've been running

# If you don't have a zppy-interfaces repo, do this:
git clone git@github.com:E3SM-Project/zppy-interfaces.git

cd zppy-interfaces
git remote -v
# `origin` should point to `git@github.com:E3SM-Project/zppy-interfaces.git`
git fetch origin add-pcmdi
git checkout -b add-pcmdi origin/add-pcmdi
git log
# You should see these 6 commits:
# Debugging synthetic plots
# Further ENSO updates
# Fix loggers
# ENSO updates
# Refactor and fixes
# Move PCMDI code from zppy
conda clean --all --y
conda env create -f conda/dev.yml -n zi-pcmdi-diags-20250730
# Note the line above is how pcmdi_metrics gets installed:
grep pcmdi conda/dev.yml         
#  - pcmdi_metrics
conda activate zi-pcmdi-diags-20250730
python -m pip install . # Installs zppy-interfaces into the zi-pcmdi-diags-20250730 environment
cd ..

# If you don't have a zppy repo, do this:
git clone git@github.com:E3SM-Project/zppy.git

cd zppy
git remote -v
# `origin` should point to `git@github.com:E3SM-Project/zppy.git`
git fetch origin zppy_pcmdi_202505
git checkout -b zppy_pcmdi_202505 origin/zppy_pcmdi_202505
git log
# You should see these 3 commits:
# Minor updates
# Refactor and fixes
# Add PCMDI Diags to zppy 
conda clean --all --y
conda env create -f conda/dev.yml -n zppy-pcmdi-diags-20250730
conda activate zppy-pcmdi-diags-20250730
grep -n "conda activate zi-pcmdi-20250529" zppy/templates/pcmdi_diags.bash
# 492:source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi-pcmdi-20250529
# 835:source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi-pcmdi-20250529
emacs zppy/templates/pcmdi_diags.bash
# You're going to have change both appearances of "zi-pcmdi-20250529" to your environment name, which will be "zi-pcmdi-diags-20250730" if you've been following these instructions exactly.
# AND you're going to have change both appearances of the conda filepath with the path to your own conda. Try looking in your `~/.bashrc` for any conda setup that could give you a clue what that path is.
# This code editing is obviously not the intended end-state for users. I was just using a hard-coded environment for testing, but I think we're ultimately going to have to add a new parameter.
# We already have the `environment_commands` parameter, but the problem here is we need to to use *two* different environments: Unified to run NCO, and then the `zppy-interfaces` environment to run `PCMDI`. We have to switch between them, as can be shown with:
grep -n "source " zppy/templates/pcmdi_diags.bash
# 492:source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi-pcmdi-20250529
# 494:source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# 835:source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi-pcmdi-20250529
# 844:source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
emacs examples/post.v3.LR.amip.0101.cfg 
# We're going to have make some edits here too.
# Change these 2 paths to be something of your choosing:
# output = /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_21/v3.LR.amip_0101
# www = /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_pr719_output/unique_id_21
# NOTE: this cfg has some key differences to the example script linked at https://github.com/E3SM-Project/zppy/pull/656#issuecomment-2832487957:
# 1) The year sets are shrunk to a shorter time range
# 2) I removed unnecessary tasks/subtasks for our testing purposes
# 3) I removed enso_metric from the synthetic plots (since enso wasn't producing anything to include)
# 4) `debug = True`, so temporary workdirs that zppy constructs will not get deleted
git diff
# You should only see the changes you just made to
# 1) zppy/templates/pcmdi_diags.bash
# 2) examples/post.v3.LR.amip.0101.cfg 
python -m pip install . # Installs zppy into the zppy-pcmdi-diags-20250730 environment
zppy -c examples/post.v3.LR.amip.0101.cfg # This will launch all the jobs
# Web output will be at https://web.lcrc.anl.gov/public/e3sm/ + whatever you put for `www` (minus the `/lcrc/group/e3sm/public_html/` prefix)
# Output will be at whatever you put for `output` + `/post/scripts`. Often it will be helpful to inspect the contents of the temporary workdirs created by the jobs. Those are the tmp.{job_id}.{random-string} subdirectories.

Target deadlines

Aiming to merge by October 1, if want to include this addition in testing for the next E3SM-Unified release.

Remaining aciton items after we get this example cfg working:

  • Run the original, complete example cfg from Zppy pcmdi new zppy#656 (comment)
  • Run integration tests to make sure other tasks don't break
  • Clean up parameters (e.g,, make sure we're only introducing parameters we really need, include an extra_environment_commands parameter)
  • Add unit/integration tests for this PR

@forsyth2 forsyth2 added the priority: high High priority task (for next release) label Aug 9, 2025
@forsyth2
Copy link
Copy Markdown
Collaborator Author

forsyth2 commented Sep 17, 2025 •
edited
Loading

@zhangshixuan1987 Thanks for the additional commits. I've done an initial review & test run, described below. So far, everything is looking good! I have additional action items (noted at the end) for me to check on, but it looks like the core functionality is working 🎉

The PRs

Initial commits:

Initial testing instructions for Shixuan: comment

New commits from Shixuan:

Reviewing those changes

On GitHub Files Changed > hold + click to select a range of commits

The 7 new commits in zppy-interfaces: https://github.com/E3SM-Project/zppy-interfaces/pull/25/files/4c6d96f017dd4c9a4487aef5c0d901ebb534d156..2267b361ab1b02ca583cdc1852436d235e3707f1

The 6 new commits in zppy: https://github.com/E3SM-Project/zppy/pull/732/files/64d99d9b7bccc3f0a81fff5d7d933a02115708a5..46ecd165af67460efcfdc2ffdbd1b4e28a960c75

Testing those changes

lcrc_conda

cd ~/ez/zppy-interfaces
git status
# Check for uncommitted changes
git checkout add-pcmdi
git log
# Good, has my 6 commits from https://github.com/E3SM-Project/zppy-interfaces/pull/25/commits
git checkout -b add-pcmdi-backup-20250917-initial-6-commits
git checkout add-pcmdi
git fetch upstream add-pcmdi
git reset --hard upstream/add-pcmdi
# Good, has all 7+6 commits from https://github.com/E3SM-Project/zppy-interfaces/pull/25/commits

cd ~/ez/zppy
git status
# Check for uncommitted changes
git checkout zppy_pcmdi_202505
git log
# Good, has my 3 commits from https://github.com/E3SM-Project/zppy/pull/719/commits
git checkout -b zppy_pcmdi_202505-backup-20250917-initial-3-commits
git checkout zppy_pcmdi_202505
git fetch zhangshixuan1987 zppy_pcmdi_202505_rev
git checkout -b zppy_pcmdi_202505_rev zhangshixuan1987/zppy_pcmdi_202505_rev
git log
# Good, has all 6+3 commits from https://github.com/E3SM-Project/zppy/pull/732/commits

# Let's review the zppy changes again
# https://github.com/E3SM-Project/zppy/pull/732/files/64d99d9b7bccc3f0a81fff5d7d933a02115708a5..46ecd165af67460efcfdc2ffdbd1b4e28a960c75
diff /lcrc/group/e3sm/ac.szhang/acme_scratch/e3sm_project/zppy/zppy/templates/inclusions/e3sm_to_cmip/vrt_remap_plev19.nc /home/ac.forsyth2/ez/zppy/zppy/templates/inclusions/e3sm_to_cmip/vrt_remap_plev19.nc
# No diffs; it's the same content

# Make some changes for me to run
git status
# 3 changed files

# Let's review the zi changes again
# https://github.com/E3SM-Project/zppy-interfaces/pull/25/files/4c6d96f017dd4c9a4487aef5c0d901ebb534d156..2267b361ab1b02ca583cdc1852436d235e3707f1

# No changes for me to make

cd ~/ez/zppy-interfaces
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zi-pcmdi-diags-20250917
conda activate zi-pcmdi-diags-20250917
pre-commit run --all-files
python -m pip install .

cd ~/ez/zppy
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zppy-pcmdi-diags-20250917
conda activate zppy-pcmdi-diags-20250917
# Update unique id and zi environment in examples/post.v3.LR.amip.0101.cfg
pre-commit run --all-files
git add -A
python -m pip install .
zppy -c examples/post.v3.LR.amip.0101.cfg
# RUNNING, as of 2025-09-17 12:10pm
cd /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_22/v3.LR.amip_0101/post/scripts
grep -v "OK" *status
# Good, no errors!

# Review results:
# For full directory tree: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_pr719_output/unique_id_22/v3.LR.amip_0101/pcmdi_diags/model_vs_obs/
# For viewer: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_pr719_output/unique_id_22/v3.LR.amip_0101/pcmdi_diags/model_vs_obs/viewer/ 
# There's way too many plots to check manually, so it's unclear if all look good.
# Several plots are missing, perhaps because they weren't requested or the data is unavailable.

# Push my changes
cd ~/ez/zppy
git commit -m "Testing changes"
git push zhangshixuan1987 zppy_pcmdi_202505_rev

Remaining action items for me:

  • Clean up parameters (e.g,, make sure we're only introducing parameters we really need)
  • Disable ENSO for now, given the comment about compatibility with E3SM Unified. (Mean climate metrics and modes of variability can be enabled, buit ENSO metrics still relies on CDAT, and so will need to be enabled later.)
  • Review entire set of changes in zppy & zi cohesively (rather than just the recent changes, as above)
  • Run existing integration tests to make sure other tasks don't break
  • Add unit/integration tests for this PR

@forsyth2
Copy link
Copy Markdown
Collaborator Author

forsyth2 commented Sep 18, 2025

Status update, end-of-day 2025-09-17:

  • Clean up parameters (e.g,, make sure we're only introducing parameters we really need)
Details

Checking additions to zppy/defaults/default.ini:

# Parameters added to [e3sm_to_cmip] task #####################################

# Used in zppy/templates/e3sm_to_cmip.bash:
cmip_plevdata = string(default="inclusions/e3sm_to_cmip/vrt_remap_plev19.nc")
interp_vars = string(default="U,V,T,Q,RELHUM,OMEGA,Z3")

# Parameters added as part of new [pcmdi_diags] task ##########################

# Used in zppy/pcmdi_diags.py:
reference_data_path = string(default="")
ref_years = string_list(default=list(""))
ts_num_years_ref = integer(default=5) # TODO: Remove? Appears to be getting checked for no reason
ts_years = string_list(default=list(""))
e3sm_to_cmip_atm_subsection = string(default="")

# Used in zppy/pcmdi_diags.py & in zppy/templates/pcmdi_diags.bash:
obs_ts = string(default="")
model_name_ref = string(default="ERA5")
model_tableID_ref = string(default="Amon")
reference_data_path_ts = string(default="")
ref_final_yr = string(default="")
ref_start_yr = string(default="")
ts_num_years = integer(default=5)
run_type = string(default="model_vs_obs")
grid = string(default="180x360_aave") # Also appears in zppy/templates/e3sm_to_cmip.bash, but that's under the e3sm_to_cmip task
cmip_enso_dir = string(default="")
cmip_clim_dir = string(default="")
cmip_movs_dir = string(default="")

# Used in zppy/pcmdi_diags.py & in zppy/utils.py:
sets = string_list(default=list("mean_climate","variability_modes_atm","variability_modes_cpl","enso","synthetic_plots"))

# Used in zppy/templates/pcmdi_diags.bash:
environment_commands_secondary = string(default="") 
obs_sets = string(default="default")
model_name = string(default="e3sm.historical.v3-LR.0051")
model_tableID = string(default="Amon")
generate_sftlf = string(default=True)
cmip_vars = string(default="pr,prw,psl,rlds,rldscs,rlut,rlutcs,rsut,rsutcs,rsds,rsdscs,rsdt,rsus,rsuscs,rlus,rtmt,sfcWind,tas,tauu,tauv,ts,ta,ua,va,zg") # Also appears in zppy/templates/e3sm_to_cmip.bash, but that's under the e3sm_to_cmip task
vars = string(default="pr,prw,psl,rlds,rldscs,rltcre,rstcre,rlut,rlutcs,rsds,rsdscs,rsdt,rsus,rsuscs,rlus,rsut,rtmt,sfcWind,tas,tauu,tauv,ts,ta-200,ta-850,ua-200,ua-850,va-200,va-850,zg-500")
ref_name = string(default="") # TODO: Remove? Appears to be set and then not used
multiprocessing = boolean(default=True)
num_workers = integer(default=24)
figure_format = string(default="png")
atm_modes = string(default="NAM,NAO,PNA,NPO,SAM,PSA1,PSA2")
cpl_modes = string(default="PDO,NPGO,AMO")
enso_groups = string(default="ENSO_perf,ENSO_proc,ENSO_tel")
regions = string(default="global,ocean,land,NHEX,SHEX,TROPICS")

# Used in zppy/templates/pcmdi_diags.bash, specifically copying files from `pcmdi_external_prefix` into the current workdir:
# TODO: We might want to see if these parameters, in particular, can be removed by including the files rather than using an external prefix.
regions_specs = string(default="pcmdi_data/region/regions_specs.json") # File used in the parameterfile.py
reference_alias = string(default="pcmdi_data/reference/reference_alias.json") # File not actually used, but appears useful as a reference. IS used in zi as `obs_alias_file` in link_observation.py.
pcmdi_viewer_util = string(default="pcmdi_data/utility/pcmdi_viewer_util.py") # File not actually used
pcmdi_zppy_util = string(default="pcmdi_data/utility/pcmdi_zppy_util.py") # File not actually used
synthetic_metrics = string(default="pcmdi_data/synthetic_metrics/synthetic_metrics_list.json") # File not actually used. IS used in zi in pcmdi_synthetic_plots.py

# Used in zppy/templates/pcmdi_diags.bash, but only set as part of parameterfile.py:
# L524-L787 of this bash file are actually part of this parameter python file!
# cat > parameterfile.py << EOF
# EOF
# This wasn't pulled out to zppy-interfaces because it must be *passed* to PMP commands with `-p`.
# In theory, it would be possible to pull out this Python code so that it could actually be checked with pre-commit, but we can leave that for a future PR.
ModUnitsAdjust = string(default="")
ObsUnitsAdjust = string(default="")
frequency = string(default="mo") # Also appears in zppy/templates/e3sm_to_cmip.bash, but that's under the e3sm_to_cmip task
seasons = string(default="monthly")
landmask = string(default=False)
RmDomainMean = string(default=True)
EofScaling = string(default=False)
ConvEOF = string(default=True)
CBF = string(default=True)
eofn_mod_max = integer(default=3)
mov_plot_obs = string(default=True)
mov_plot_model = string(default=True)
mov_nc_out_obs = string(default=True)
mov_nc_out_model = string(default=True)
save_test_clims = string(default=True)
target_grid = string(default="1x1")
target_grid_string = string(default="1px1p")
regrid_tool = string(default="esmf") # TODO: Looks like we need to ADD regrid_tool_ocn...
regrid_method = string(default="regrid2")
regrid_method_ocn = string(default="conservative")
pcmdi_debug = string(default=False)

# Used in `zppy/templates/pcmdi_diags.bash`, specifically only in `zi-pcmdi-synthetic-plots`:
pcmdi_webtitle = string(default="E3SM-PMP-Diagnostics")
pcmdi_version = string(default="v3.8.2")
pcmdi_viewer_template = string(default="pcmdi_data/viewer")
synthetic_sets = string(default="portrait,parcoord")
cmip_clim_set = string(default="cmip6.historical.v20250707")
cmip_movs_set = string(default="cmip6.historical.v20220825")
cmip_enso_set = string(default="cmip6.historical.v20210620")
git add -A
pre-commit run --all-files
git commit -m "Update parameters"
git push zhangshixuan1987 zppy_pcmdi_202505_rev

Commit: 4dabd25d, "Update parameters"

Remaining action items:

  • Disable ENSO for now, given the comment about compatibility with E3SM Unified. (Mean climate metrics and modes of variability can be enabled, but ENSO metrics still relies on CDAT, and so will need to be enabled later.)
  • Check for changes to e3sm_to_cmip that may break other tasks
  • Look for TODOs in code
  • Run and fix unit tests
  • Review entire set of changes in zppy & zi cohesively (rather than just the recent changes, as above)
  • Run existing integration tests to make sure other tasks don't break
  • Add unit/integration tests for this PR

@forsyth2
Copy link
Copy Markdown
Collaborator Author

forsyth2 commented Sep 18, 2025

Status update

  • Check for changes to e3sm_to_cmip that may break other tasks
Details 
cd ~/ez/zppy
lcrc_conda
conda activate zppy-pcmdi-diags-20250917
# Make fixes
pre-commit run --all-files
git add -A
git commit -m "e3sm_to_cmip fixes"

Commit: 26a99311, "e3sm_to_cmip fixes"

  • Look for TODOs in code
Details 
cd ~/ez/zppy-interfaces
git grep -n TODO
# Nothing relevant to pcmdi diags
cd ~/ez/zppy
git grep -n TODO
# Address relevant TODO items
pre-commit run --all-files
git add -A
git commit -m "Add json to inclusions"

Commit: 8e8c1962, "Add json to inclusions"

  • Run and fix unit tests
Details 
cd ~/ez/zppy-interfaces
lcrc_conda
conda activate zi-pcmdi-diags-20250917
pre-commit run --all-files
python -m pip install .
pytest tests/unit/global_time_series/test_*.py
# 10 passed in 29.22s

cd ~/ez/zppy
lcrc_conda
conda activate zppy-pcmdi-diags-20250917
pre-commit run --all-files
python -m pip install .
pytest tests/test_*.py
# 2 failed, 23 passed, 1 warning in 0.70s
# (warning is ok, and has appeared on `main` anyway)

# Make fixes
pytest tests/test_*.py
# 25 passed in 0.62s

pre-commit run --all-files
git add -A
git commit -m "Fix unit tests"
git push zhangshixuan1987 zppy_pcmdi_202505_rev
# GitHub Actions now pass on https://github.com/E3SM-Project/zppy/pull/732

Commit: 42dcbe72, "Fix unit tests"

Remaining action items:

  • Run PCMDI example cfg to make sure nothing there broke
  • Run existing integration tests to make sure other tasks don't break
  • Review entire set of changes in zppy & zi cohesively (rather than just the recent changes, as above)
  • Add unit/integration tests for this PR
  • Disable ENSO for now, given the comment about compatibility with E3SM Unified. (Mean climate metrics and modes of variability can be enabled, buit ENSO metrics still relies on CDAT, and so will need to be enabled later.)
  • See Discussion post: Optional conda dependencies #34

@forsyth2
Copy link
Copy Markdown
Collaborator Author

forsyth2 commented Sep 19, 2025

Status update

  • Run PCMDI example cfg to make sure nothing there broke
Details 
lcrc_conda
cd ~/ez/zppy-interfaces
git status
# Check for uncommitted changes
git log
# Good, has all 7+6 commits from https://github.com/E3SM-Project/zppy-interfaces/pull/25/commits
conda activate zi-pcmdi-diags-20250917
python -m pip install .

lcrc_conda
cd ~/ez/zppy
git status
# Check for uncommitted changes
git log
# Good, has all 5+6+3 commits from https://github.com/E3SM-Project/zppy/pull/732/commits
conda activate zppy-pcmdi-diags-20250917
python -m pip install .

# Update unique id in examples/post.v3.LR.amip.0101.cfg (22->23)
# Iterate: 24, 25, 26, 27, 28
pre-commit run --all-files
git add -A
python -m pip install .
zppy -c examples/post.v3.LR.amip.0101.cfg
cd /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_28/v3.LR.amip_0101/post/scripts
grep -v "OK" *status

# Commence binary search testing to determine which commit introduced errors
# My 5 commits: https://github.com/E3SM-Project/zppy/pull/732/files/46ecd165af67460efcfdc2ffdbd1b4e28a960c75..42dcbe727ce26c7abb92d51d58b6cfac727aebaf#diff-4454bf29794ce5324453a1de06d91223b238348b52b7c89e00e57127bfba12ec
# 1. Testing changes -- we know this worked, unique_id22 tested it
# 2. Update parameters
# 3. e3sm_to_cmip fixes
# 4. Add json to inclusions
# 5. Fix unit tests -- we know this is failing, no matter what fixes I do
git restore --staged examples/post.v3.LR.amip.0101.cfg zppy/defaults/default.ini zppy/templates/e3sm_to_cmip.bash zppy/templates/pcmdi_diags.bash
git restore examples/post.v3.LR.amip.0101.cfg zppy/defaults/default.ini zppy/templates/e3sm_to_cmip.bash zppy/templates/pcmdi_diags.bash
git status
# No uncommitted changes
git checkout -b binary-search-test-try1 26a99311d1c211b42e9bf2c682d83babba830c8e
git log
# Good, only has the first 3 of the 5 commits
# Update unique id in examples/post.v3.LR.amip.0101.cfg (29)
pre-commit run --all-files
git add -A
python -m pip install .
zppy -c examples/post.v3.LR.amip.0101.cfg
cd /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_29/v3.LR.amip_0101/post/scripts
grep -v "OK" *status
# Still fails, so it has to be an earlier commit

cd ~/ez/zppy
git restore --staged examples/post.v3.LR.amip.0101.cfg
git restore examples/post.v3.LR.amip.0101.cfg
git status
# No uncommitted changes
git checkout -b binary-search-test-try2 4dabd25d86ee4f5ee1ce83bed472276697319894
git log
# Good, only has the first 2 of the 5 commits
# Update unique id in examples/post.v3.LR.amip.0101.cfg (30)
pre-commit run --all-files
python -m pip install .
zppy -c examples/post.v3.LR.amip.0101.cfg
cd /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_30/v3.LR.amip_0101/post/scripts
grep -v "OK" *status
# No errors!
# It appears commit 3 (e3sm_to_cmip fixes) introduced the errors.
git grep grid zppy/templates/e3sm_to_cmip.bash
# zppy/templates/e3sm_to_cmip.bash:dest={{ output }}/post/{{ component }}/{{ ts_grid }}/ts/{{ frequency }}/{{ '%dyr' % (ypf) }}
# zppy/templates/e3sm_to_cmip.bash:  dest_cmip={{ output }}/post/{{ component }}/{{ grid }}/cmip_ts/{{ frequency }}
#
# So... just need to convert that one grid to ts_grid too?
# Maybe also put `{{ cmip_vars }} \` in quotes
cd ~/ez/zppy
git restore examples/post.v3.LR.amip.0101.cfg
git checkout zppy_pcmdi_202505_rev
# Good, has all 5 of the new commits
# Add the potential fixes of commit 3
# Update unique id in examples/post.v3.LR.amip.0101.cfg (31)
pre-commit run --all-files
python -m pip install .
zppy -c examples/post.v3.LR.amip.0101.cfg
cd /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_31/v3.LR.amip_0101/post/scripts
grep -v "OK" *status
# Failures still, those fixes didn't work.

# The one that doesn't work:
cd /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_31/v3.LR.amip_0101/post/scripts
emacs e3sm_to_cmip_atm_monthly_180x360_aave_2010-2014-0005.bash
#   --var-list \
#   'ua, va, ta, wa, zg, hur, tas, ts, psl, ps, sfcWind, huss, pr, prc, prsn, evspsbl, tauu, tauv, hfls, clt, rlds, rlus, rsds, rsus, hfss, clivi, clwvi, prw, rldscs, rlut, rlutcs, rsdt, rsuscs, rsut, rsutcs, rtmt, abs550aer, od550aer, rsdscs, tasmax, tasmin' \
#   --realm \
#   atm \
#
# The one that works:
cd /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_30/v3.LR.amip_0101/post/scripts
emacs e3sm_to_cmip_atm_monthly_180x360_aave_2010-2014-0005.bash
#   --var-list \
#   'ua, va, ta, wa, zg, hur, pr, prc, prsn, ts, tas, prw, psl, sfcWind, tasmax, tasmin, tauu, tauv, rtmt, rsdt, rsds, rsdscs, rlds, rldscs, rsus, rsuscs, rsut, rsutcs, rlus, rlut, rlutcs, clivi, clwvi, clt, evspsbl, hfls, hfss, huss' \
#   --realm \
#   atm \


# Let's go commit-by-commit...
cd ~/ez/zppy
git restore examples/post.v3.LR.amip.0101.cfg zppy/templates/e3sm_to_cmip.bash
git checkout -b zppy_pcmdi_202505_rev-all-5-new-commits-20250918
git log
# Good, all 5 new commits present
git checkout zppy_pcmdi_202505_rev
git rebase -i 46ecd165af67460efcfdc2ffdbd1b4e28a960c75
# Keep first 3 new commits, drop last 2
git log
# Good, only the first 3 new commits present
# Add the fixes to the 3rd commit
# Update unique id in examples/post.v3.LR.amip.0101.cfg (32)
pre-commit run --all-files
python -m pip install .
zppy -c examples/post.v3.LR.amip.0101.cfg
cd /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_32/v3.LR.amip_0101/post/scripts
grep -v "OK" *status
# No errors, success!
cd ~/ez/zppy
git add -A
git commit -m "e3sm_to_cmip fixes follow-up"

# Now, add back the 4th commit, "Add json to inclusions"
git cherry-pick 8e8c19620079d77fd8a8d070aca0707ba128df34
git log
# Good, we have the 3 commits, the follow-up commit, and the "Add json to inclusions" commit
# Make some initial fixes before testing again
# Update unique id in examples/post.v3.LR.amip.0101.cfg (33)
pre-commit run --all-files
python -m pip install .
zppy -c examples/post.v3.LR.amip.0101.cfg
cd /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_33/v3.LR.amip_0101/post/scripts
grep -v "OK" *status
# pcmdi_diags_synthetic_plots_model_vs_obs.status:ERROR (11)
# FileNotFoundError: [Errno 2] No such file or directory: 'synthetic_metrics_list.json'

# The json file needed to be synthetic_metrics_list.json, not synthetic_metrics.json...
# Fixing.
cd ~/ez/zppy
# Update unique id in examples/post.v3.LR.amip.0101.cfg (34)
pre-commit run --all-files
python -m pip install .
zppy -c examples/post.v3.LR.amip.0101.cfg # ~45 minutes to run to completion
cd /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_34/v3.LR.amip_0101/post/scripts
grep -v "OK" *status
# No errors, success!
cd ~/ez/zppy
git add -A
git commit -m "Add json to inclusions follow-up"

# For full directory tree: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_pr719_output/unique_id_34/v3.LR.amip_0101/pcmdi_diags/model_vs_obs/
# For viewer: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_pr719_output/unique_id_34/v3.LR.amip_0101/pcmdi_diags/model_vs_obs/viewer/ 

# Now, add back the 5th commit, "Fix unit tests"
git cherry-pick 42dcbe727ce26c7abb92d51d58b6cfac727aebaf
git log
# Good, we have the 5 commits + both follow-up commits
# This commit only changed a unit test, so no need to re-run the example cfg.

# Push my changes
git push -f zhangshixuan1987 zppy_pcmdi_202505_rev
# See changes from all 7 commits: https://github.com/E3SM-Project/zppy/pull/732/files/46ecd165af67460efcfdc2ffdbd1b4e28a960c75..f9944a94837bdd7ba1d5d55ca7b106f8eac92439

Remaining action items:

  • Run existing integration tests to make sure other tasks don't break
  • Review entire set of changes in zppy & zi cohesively (rather than just the recent changes, as above)
  • Add unit/integration tests for this PR
  • Disable ENSO for now, given the comment about compatibility with E3SM Unified. (Mean climate metrics and modes of variability can be enabled, buit ENSO metrics still relies on CDAT, and so will need to be enabled later.)
  • See Discussion post: Optional conda dependencies #34

@forsyth2
Copy link
Copy Markdown
Collaborator Author

forsyth2 commented Sep 19, 2025

Status update:

  • Run existing integration tests to make sure other tasks don't break
Details 
lcrc_conda
cd ~/ez/zppy-interfaces
git status
# Check for uncommitted changes
git checkout -b add-pcmdi-backup-20250918 # Non-rebased backup
git checkout add-pcmdi
git fetch upstream main
git rebase upstream/main # Good, no conflicts
rm -rf build
conda clean --all --y
time conda env create -f conda/dev.yml -n zi-pcmdi-diags-20250918 # 4m15s
conda activate zi-pcmdi-diags-20250918
pre-commit run --all-files
python -m pip install .
pytest tests/unit/global_time_series/test_*.py
# 10 passed in 23.92s

cd ~/ez/zppy
git status
# Check for uncommitted changes
git checkout -b zppy_pcmdi_202505_rev-backup-20250918 # Non-rebased backup
git checkout zppy_pcmdi_202505_rev
git fetch upstream main
git rebase upstream/main # Good, no conflicts
rm -rf build
conda clean --all --y
time conda env create -f conda/dev.yml -n zppy-pcmdi-diags-20250918 # 3m40s
conda activate zppy-pcmdi-diags-20250918
pre-commit run --all-files
python -m pip install .
pytest tests/test_*.py
# 25 passed, 1 warning in 0.30s
# (warning is ok, and has appeared on `main` anyway)

# We'll also need to set up e3sm_diags for the full integration test suite
cd ~/ez/e3sm_diags
git status
# Check for uncommitted changes
git fetch upstream main
git checkout main
git reset --hard upstream/main
rm -rf build
conda clean --all --y
time conda env create -f conda-env/dev.yml -n e3sm-diags-main-20250918 # 2m58s
conda activate e3sm-diags-main-20250918
python -m pip install .

cd ~/ez/zppy
conda activate zppy-pcmdi-diags-20250918
python -m pip install .
# Edit tests/integration/utils.py:
# TEST_SPECIFICS: Dict[str, Any] = {
#     "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm-diags-main-20250918",
#     "global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi-pcmdi-diags-20250918",
#     "cfgs_to_run": [
#         "weekly_bundles",
#         "weekly_comprehensive_v2",
#         "weekly_comprehensive_v3",
#         "weekly_legacy_3.0.0_bundles",
#         "weekly_legacy_3.0.0_comprehensive_v2",
#         "weekly_legacy_3.0.0_comprehensive_v3",
#     ],
#     "tasks_to_run": ["e3sm_diags", "mpas_analysis", "global_time_series", "ilamb"],
#     "unique_id": "integration_testing_zppy_pmp_20250918",
# }
python tests/integration/utils.py
git status
# This will include the generated cfgs we need to run now.
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
# ilamb_1980-1981
# Traceback (most recent call last):
#   File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zppy-pcmdi-diags-20250918/bin/zppy", line 8, in <module>
#     sys.exit(main())
#   File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zppy-pcmdi-diags-20250918/lib/python3.9/site-packages/zppy/__main__.py", line 82, in main
#     _launch_scripts(config, script_dir, job_ids_file, plugins)
#   File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zppy-pcmdi-diags-20250918/lib/python3.9/site-packages/zppy/__main__.py", line 269, in _launch_scripts
#     existing_bundles = ilamb(config, script_dir, existing_bundles, job_ids_file)
#   File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zppy-pcmdi-diags-20250918/lib/python3.9/site-packages/zppy/ilamb.py", line 80, in ilamb
#     submit_script(
#   File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zppy-pcmdi-diags-20250918/lib/python3.9/site-packages/zppy/utils.py", line 413, in submit_script
#     if tmp[0] in ("OK"):
# IndexError: list index out of range
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg

# Picking back up 2025-09-19 morning
lcrc_conda
conda activate zppy-pcmdi-diags-20250918

# Check on bundles status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/integration_testing_zppy_pmp_20250918/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/integration_testing_zppy_pmp_20250918/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors

# Now, run bundles part 2
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg

# Review finished runs
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/integration_testing_zppy_pmp_20250918/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# global_time_series_1980-1990.status:ERROR (8)
# (Plus the ILAMB error on run, mentioned above)
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/integration_testing_zppy_pmp_20250918/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/integration_testing_zppy_pmp_20250918/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# global_time_series_1980-1990.status:ERROR (8)
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/integration_testing_zppy_pmp_20250918/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors

# So, something's going on with the v2 runs...
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/integration_testing_zppy_pmp_20250918/v2.LR.historical_0201/post/scripts
emacs global_time_series_1980-1990.o903905
# KeyError: 'rgn'
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/integration_testing_zppy_pmp_20250918/v2.LR.historical_0201/post/scripts
emacs global_time_series_1980-1990.o903844
# KeyError: 'rgn'
# `rgn` doesn't show up in cmd+f for https://github.com/E3SM-Project/zppy/pull/732/files or for https://github.com/E3SM-Project/zppy-interfaces/pull/25/files, even when making sure all diffs are loaded.
# Is it possible this is an issue on `main`??

# Let's proceed with the test suite to see if any more errors exist.
ls tests/integration/test_*.py
# tests/integration/test_bash_generation.py  tests/integration/test_defaults.py
# tests/integration/test_bundles.py          tests/integration/test_images.py
# tests/integration/test_campaign.py         tests/integration/test_last_year.py
pytest tests/integration/test_bash_generation.py
# 1 failed in 1.95s, diffs appear to be expected based on recent changes on `main` + this PR
pytest tests/integration/test_campaign.py
# 6 failed in 2.25s, diffs appear to be expected based on recent changes on `main` + this PR
pytest tests/integration/test_defaults.py
# 1 failed in 0.44s, diffs appear to be expected based on recent changes on `main` + this PR
pytest tests/integration/test_last_year.py
# 1 failed in 0.37s, diff is because the prov file is always unique
pytest tests/integration/test_bundles.py
# 2 passed in 0.26s
pytest tests/integration/test_images.py
# Copy the output of early_test_images_summary.md to a Pull Request comment
# ================================== short test summary info ===================================
# FAILED tests/integration/test_images.py::test_images - ValueError: list.remove(x): x not in list
# =============================== 1 failed in 2412.87s (0:40:12) ===============================
cat early_test_images_summary.md
# It's an early summary, so we know an exception was encountered
# But that appears to be just because of the ValueError above, which seems to happen because of:
#  tasks_to_run = tasks_to_run.copy()
#  tasks_to_run.remove("mpas_analysis")
# It looks like https://github.com/E3SM-Project/zppy/pull/733 accidentally gave the copy the same name as the original...
# In any case, not a problem with the PMP PR

gives:

Test name Total images Correct images Missing images Mismatched images Diff subdir
comprehensive_v2_e3sm_diags 3031 2966 0 65 /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v2_www/integration_testing_zppy_pmp_20250918/v2.LR.historical_0201/image_check_failures_comprehensive_v2/e3sm_diags
comprehensive_v2_mpas_analysis 854 854 0 0 /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v2_www/integration_testing_zppy_pmp_20250918/v2.LR.historical_0201/image_check_failures_comprehensive_v2/mpas_analysis
comprehensive_v2_global_time_series 6 0 6 0 /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v2_www/integration_testing_zppy_pmp_20250918/v2.LR.historical_0201/image_check_failures_comprehensive_v2/global_time_series
comprehensive_v2_ilamb 776 0 776 0 /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v2_www/integration_testing_zppy_pmp_20250918/v2.LR.historical_0201/image_check_failures_comprehensive_v2/ilamb
comprehensive_v3_e3sm_diags 3860 3793 0 67 /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/integration_testing_zppy_pmp_20250918/v3.LR.historical_0051/image_check_failures_comprehensive_v3/e3sm_diags
comprehensive_v3_mpas_analysis 854 854 0 0 /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/integration_testing_zppy_pmp_20250918/v3.LR.historical_0051/image_check_failures_comprehensive_v3/mpas_analysis
comprehensive_v3_global_time_series 15 15 0 0 /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/integration_testing_zppy_pmp_20250918/v3.LR.historical_0051/image_check_failures_comprehensive_v3/global_time_series
comprehensive_v3_ilamb 774 774 0 0 /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/integration_testing_zppy_pmp_20250918/v3.LR.historical_0051/image_check_failures_comprehensive_v3/ilamb
bundles_e3sm_diags 1692 1622 0 70 /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_bundles_www/integration_testing_zppy_pmp_20250918/v3.LR.historical_0051/image_check_failures_bundles/e3sm_diags
bundles_global_time_series 3 3 0 0 /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_bundles_www/integration_testing_zppy_pmp_20250918/v3.LR.historical_0051/image_check_failures_bundles/global_time_series
bundles_ilamb 388 388 0 0 /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_bundles_www/integration_testing_zppy_pmp_20250918/v3.LR.historical_0051/image_check_failures_bundles/ilamb
legacy_3.0.0_comprehensive_v2_e3sm_diags 3031 2962 2 67 /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_www/integration_testing_zppy_pmp_20250918/v2.LR.historical_0201/image_check_failures_legacy_3.0.0_comprehensive_v2/e3sm_diags
legacy_3.0.0_comprehensive_v2_global_time_series 6 0 6 0 /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_www/integration_testing_zppy_pmp_20250918/v2.LR.historical_0201/image_check_failures_legacy_3.0.0_comprehensive_v2/global_time_series
legacy_3.0.0_comprehensive_v2_ilamb 776 776 0 0 /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_www/integration_testing_zppy_pmp_20250918/v2.LR.historical_0201/image_check_failures_legacy_3.0.0_comprehensive_v2/ilamb
legacy_3.0.0_comprehensive_v3_e3sm_diags 3860 3793 0 67 /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_www/integration_testing_zppy_pmp_20250918/v3.LR.historical_0051/image_check_failures_legacy_3.0.0_comprehensive_v3/e3sm_diags
legacy_3.0.0_comprehensive_v3_global_time_series 15 15 0 0 /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_www/integration_testing_zppy_pmp_20250918/v3.LR.historical_0051/image_check_failures_legacy_3.0.0_comprehensive_v3/global_time_series
legacy_3.0.0_comprehensive_v3_ilamb 774 774 0 0 /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_www/integration_testing_zppy_pmp_20250918/v3.LR.historical_0051/image_check_failures_legacy_3.0.0_comprehensive_v3/ilamb

There are some test failures, but it's unclear how much is from this PR directly. Will reconcile results with results from testing main branch today.

Remaining action items:

  • Review entire set of changes in zppy & zi cohesively (rather than just the recent changes, as above)
  • Add unit/integration tests for this PR
  • Disable ENSO for now, given the comment about compatibility with E3SM Unified. (Mean climate metrics and modes of variability can be enabled, buit ENSO metrics still relies on CDAT, and so will need to be enabled later.)
  • See Discussion post: Optional conda dependencies #34

@forsyth2
Copy link
Copy Markdown
Collaborator Author

forsyth2 commented Sep 19, 2025

Will reconcile results with results from testing main branch today.

See E3SM-Project/zppy#634 (comment)

It does appear that this PR causes some ilamb/global_time_series issues. Strangely, this PR's test run didn't run into the same mpas_analysis issues that the main branch test did...

@forsyth2
Copy link
Copy Markdown
Collaborator Author

forsyth2 commented Sep 23, 2025 •
edited
Loading

  • Run existing integration tests to make sure other tasks don't break
Details

Let's review the results we have. As for the global time series issues:

cd /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v2_www/integration_testing_zppy_pmp_20250918/v2.LR.historical_0201/image_check_failures_comprehensive_v2/global_time_series
cat missing_images.txt 
# global_time_series/global_time_series_1980-1990_results/v2.LR.historical_0201_glb_lnd.png
# global_time_series/global_time_series_1980-1990_results/v2.LR.historical_0201_glb_original.png
# global_time_series/global_time_series_1980-1990_results/v2.LR.historical_0201_n_lnd.png
# global_time_series/global_time_series_1980-1990_results/v2.LR.historical_0201_n_original.png
# global_time_series/global_time_series_1980-1990_results/v2.LR.historical_0201_s_lnd.png
# global_time_series/global_time_series_1980-1990_results/v2.LR.historical_0201_s_original.png
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/integration_testing_zppy_pmp_20250918/v2.LR.historical_0201/post/scripts
emacs global_time_series_1980-1990.o903844 
# KeyError: 'rgn'

Claude suggests "The land variables might not be regionally aggregated in your E3SM output, or they might be in different files (land history files rather than ocean files)." That makes me think it might be an issue stemming from E3SM-Project/zppy#717, and simply wansn't revealed in the main testing (E3SM-Project/zppy#634 (comment)) because of the blocking MPAS-Analysis failures.

And as for the ILAMB job submission error, Claude suggests "This type of error typically indicates a problem with the HPC job submission system rather than the scientific computation itself."

While I can't say for certain until we're testing against expected results that pass on main it seems likely the PMP PRs aren't introducing major errors.

  • Disable ENSO for now, given the comment about compatibility with E3SM Unified. (Mean climate metrics and modes of variability can be enabled, buit ENSO metrics still relies on CDAT, and so will need to be enabled later.)
  • See Discussion post: Optional conda dependencies #34
Details 
cd ~/ez/zppy
git status
# Check for uncommitted changes
git checkout zppy_pcmdi_202505_rev
git log
# Has commits from https://github.com/E3SM-Project/zppy/pull/732/commits + 2 testing commits
# Let's drop those 2 testing commits
git rebase -i 25e622376a9767706e9f1b65c2e4cc6f1425cd28
# pick d7ac246e Fix unit tests
# d f422a7c5 Completed testing
# d db14ce2e Completed testing 2
git log
# Good, back to the original commits

# Ok, let's disable ENSO now
# Add changes to disable ENSO on both the zppy & zi sides.
# We only have to worry about the CDAT dependency in the zi conda environment though.

cd ~/ez/zppy-interfaces
git status
# Check for uncommitted changes
git checkout add-pcmdi
git log
# Good, has all 7+6 commits from https://github.com/E3SM-Project/zppy-interfaces/pull/25/commits
rm -rf build
lcrc_conda
conda clean --all --y # This has been hanging recently, so let's just skip
conda env create -f conda/dev.yml -n zi-pcmdi-diags-20250922
conda activate zi-pcmdi-diags-20250922
pre-commit run --all-files # Iterate until passing
git add -A
python -m pip install .
conda list -n zi-pcmdi-diags-20250922 > zi-pcmdi-diags-20250922_packages.txt
# Good, these don't have any matches:
grep -in cdat_info zi-pcmdi-diags-20250922_packages.txt
grep -in cdms2 zi-pcmdi-diags-20250922_packages.txt
grep -in cdtime zi-pcmdi-diags-20250922_packages.txt
grep -in cdutil zi-pcmdi-diags-20250922_packages.txt
grep -in enso_metrics zi-pcmdi-diags-20250922_packages.txt
grep -in libcdms zi-pcmdi-diags-20250922_packages.txt
grep -in pcmdi_metrics zi-pcmdi-diags-20250922_packages.txt
# 308:pcmdi_metrics             3.9.3              pyhd8ed1ab_0    conda-forge
# Good, we still have pcmdi metrics

# Let's commit the changes.
git add -A
git commit -m "Disable ENSO"
git push -f upstream add-pcmdi # Force-push because we did a rebase during testing
cd ~/ez/zppy
git add -A
git commit -m "Disable ENSO"
git push -f zhangshixuan1987 zppy_pcmdi_202505_rev # Force-push because we did a rebase during testing

It appears the enso_metrics dependency (and by the transitive property, the CDAT dependency) has been removed.

Commit (zppy-interfaces): 7eaab5d, "Disable ENSO"
Commit (zppy): 3c39f692, "Disable ENSO"

Remaining action items:

  • Add unit/integration tests for this PR
  • Determine if the sets parameter and the subsection name actually fill different roles in terms of deciding dependencies, etc.
  • Review entire set of changes in zppy & zi cohesively (rather than just the recent changes, as above)

@forsyth2
Copy link
Copy Markdown
Collaborator Author

forsyth2 commented Sep 24, 2025

  • Determine if the sets parameter and the subsection name actually fill different roles in terms of deciding dependencies, etc.
Details 
cd ~/ez/zppy
# Make changes in zppy
git add -A
lcrc_conda
rm -rf build
conda env create -f conda/dev.yml -n zppy-pcmdi-diags-20250923
conda activate zppy-pcmdi-diags-20250923
pre-commit run --all-files # Iterate until passing
python -m pip install .
git add -A
git diff --staged # Review changes
git commit -m "Replace sets and subsection with current_set"

Commit: 535c1c36, "Replace sets and subsection with current_set"

  • Make sure all the parameters added in default.ini appear in main list + many list. Also, alphabetize parameters within sections.
Details 
cd ~/ez/zppy
# Make changes in zppy
git add -A
pre-commit run --all-files # Iterate until passing
git add -A
git status
git commit -m "Organize parameters"

Commit: 4325ca0a, "Organize parameters"

  • Run example cfg using latest code changes
Details 
cd ~/ez/zppy-interfaces
git status
# Check for uncommitted changes 
# branch add-pcmdi
git log
# Last commit: Disable ENSO
rm -rf build
lcrc_conda
conda env create -f conda/dev.yml -n zi-pcmdi-diags-20250923
conda activate zi-pcmdi-diags-20250923
pre-commit run --all-files
python -m pip install .
pytest tests/unit/global_time_series/test_*.py
# 10 passed in 29.49s

cd ~/ez/zppy
git status
# Check for uncommitted changes 
# branch zppy_pcmdi_202505_rev
git log
# Last commits: Organize parameters, Replace sets and subsection with current_set, Disable ENSO
conda activate zppy-pcmdi-diags-20250923
pre-commit run --all-files
python -m pip install .
pytest tests/test_*.py
# 25 passed, 1 warning in 0.61s
# (warning is ok, and has appeared on `main` anyway)
# Edit examples/post.v3.LR.amip.0101.cfg
# environment_commands_secondary = "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi-pcmdi-diags-20250923"
zppy -c examples/post.v3.LR.amip.0101.cfg

# Need to make fixes, iterate: unique_id 35-46
# Edit examples/post.v3.LR.amip.0101.cfg
# output = /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_N/v3.LR.amip_0101
# www = /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_pr719_output/unique_id_N
python -m pip install . && zppy -c examples/post.v3.LR.amip.0101.cfg

# Good, all jobs are launching successfully.
# Now, let's check that the dependencies are correct.
cd /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts
emacs pcmdi_diags_mean_climate_model_vs_obs_2005-2014.settings # e3sm_to_cmip jobs appearing twice
#   'dependencies': [ '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/ts_atm_monthly_180x360_aave_2005-2009-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2005-2009-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2010-2014-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/ts_atm_monthly_180x360_aave_2010-2014-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2005-2009-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2010-2014-0005.status'],
emacs pcmdi_diags_variability_modes_cpl_model_vs_obs_2005-2014.settings # e3sm_to_cmip jobs appearing twice
#   'dependencies': [ '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/ts_atm_monthly_180x360_aave_2005-2009-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2005-2009-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2010-2014-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/ts_atm_monthly_180x360_aave_2010-2014-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2005-2009-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2010-2014-0005.status'],
emacs pcmdi_diags_variability_modes_atm_model_vs_obs_2005-2014.settings # e3sm_to_cmip jobs appearing twice
#   'dependencies': [ '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/ts_atm_monthly_180x360_aave_2005-2009-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2005-2009-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2010-2014-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/ts_atm_monthly_180x360_aave_2010-2014-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2005-2009-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2010-2014-0005.status'],
emacs pcmdi_diags_synthetic_plots_model_vs_obs.settings
#   'dependencies': [ '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2005-2009-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2010-2014-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2005-2009-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2010-2014-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/pcmdi_diags_mean_climate_model_vs_obs_2005-2014.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/pcmdi_diags_variability_modes_cpl_model_vs_obs_2005-2014.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/pcmdi_diags_variability_modes_atm_model_vs_obs_2005-2014.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2005-2009-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2010-2014-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2005-2009-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2010-2014-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/pcmdi_diags_mean_climate_model_vs_obs_2005-2014.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/pcmdi_diags_variability_modes_cpl_model_vs_obs_2005-2014.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/pcmdi_diags_variability_modes_atm_model_vs_obs_2005-2014.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2005-2009-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2010-2014-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2005-2009-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2010-2014-0005.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/pcmdi_diags_mean_climate_model_vs_obs_2005-2014.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/pcmdi_diags_variability_modes_cpl_model_vs_obs_2005-2014.status',
#                     '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/pcmdi_diags_variability_modes_atm_model_vs_obs_2005-2014.status'],
python
>>> from collections import Counter
>>> dependencies = ... 
>>> Counter(dependencies) # Let's count the repeats
# '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2005-2009-0005.status': 6
# '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/e3sm_to_cmip_atm_monthly_180x360_aave_2010-2014-0005.status': 6
# 
# '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/pcmdi_diags_mean_climate_model_vs_obs_2005-2014.status': 3
# '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/pcmdi_diags_variability_modes_cpl_model_vs_obs_2005-2014.status': 3
# '/lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_46/v3.LR.amip_0101/post/scripts/pcmdi_diags_variability_modes_atm_model_vs_obs_2005-2014.status': 3

cd /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_47/v3.LR.amip_0101/post/scripts
emacs pcmdi_diags_mean_climate_model_vs_obs_2005-2014.settings
emacs pcmdi_diags_variability_modes_cpl_model_vs_obs_2005-2014.settings
emacs pcmdi_diags_variability_modes_atm_model_vs_obs_2005-2014.settings
emacs pcmdi_diags_synthetic_plots_model_vs_obs.settings
# All dependencies look much better now

# Real run
cd ~/ez/zppy
pre-commit run --all-files # Iterate until passing
python -m pip install .
git add -A
git status
# Edit examples/post.v3.LR.amip.0101.cfg
# output = /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_48/v3.LR.amip_0101
# www = /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_pr719_output/unique_id_48
zppy -c examples/post.v3.LR.amip.0101.cfg

cd /lcrc/group/e3sm/ac.forsyth2/zppy_pr719_output/unique_id_48/v3.LR.amip_0101/post/scripts
grep -v "OK" *status
# No errors

git add -A
git commit -m "Fixes to get example cfg working"
git log
# Commits look correct
git push zhangshixuan1987 zppy_pcmdi_202505_rev

Web results:

Commit: 774d131a, "Fixes to get example cfg working"

Remaining action items:

  • Add unit/integration tests for this PR
  • Review entire set of changes in zppy & zi cohesively (rather than just the recent changes, as above)

forsyth2
forsyth2 commented Oct 7, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator Author

@forsyth2 forsyth2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tomvothecoder @andrewdnolan @xylar I've now made eight attempts to get conda working on this PR rebased off main (post-merging of #33), and really not getting any closer. Do any of you have an idea of how I can get this working? Thanks!

(@chengzhuzhang FYI ^)

The latest from Claude:

If you absolutely must use pcmdi_metrics==3.9.3 specifically, then Python 3.13 support is not possible with your current dependency stack, because that exact version has incompatible constraints. In that case, you'd need to either:

Wait for a backport/patch of pcmdi_metrics 3.9.3 with relaxed constraints, or
Drop Python 3.13 support temporarily

But I'd strongly recommend unpinning pcmdi_metrics first to see if a newer version works for your needs.

Don't we need pcmdi_metrics v3.9.3 per the discussion at #34? (And in any case I tried with it unpinned and conda still wouldn't work).

(And of course we do need Python 3.13 support per discussions re: E3SM-Project/zstash#384)

@forsyth2
Copy link
Copy Markdown
Collaborator Author

forsyth2 commented Oct 7, 2025

get conda working on this PR

By that I mean, have the GitHub Actions passing for python 3.11, 3.12, 3.13.

@forsyth2
Copy link
Copy Markdown
Collaborator Author

forsyth2 commented Oct 7, 2025

Conda issues now resolved. GitHubActions is passing: https://github.com/E3SM-Project/zppy-interfaces/actions/runs/18321556698?pr=25. Thanks @andrewdnolan!

Once Chrysalis is back up, I will collapse the conda commits into one, and run the example/tests cfgs. If those pass, we can merge the PRs and do continued testing during the E3SM Unified testing period.

@forsyth2
Copy link
Copy Markdown
Collaborator Author

forsyth2 commented Oct 7, 2025

I'm running the two example cfgs + the test cfg now. I do have some further code changes to commit.

Re: cmip_plevdata -- I believe the best solution is to keep this in diagnostics_base_path and access it as we do other data in say reference_data_path (as opposed to using inclusions/). Jinja2 apparently doesn't work well with binary files as templates. It's also not very streamlined to have a binary file in our git repo.

@forsyth2
Copy link
Copy Markdown
Collaborator Author

forsyth2 commented Oct 7, 2025

I'm running the two example cfgs + the test cfg now.

@chengzhuzhang @zhangshixuan1987 Unfortunately, I encountered a number of errors in the tests, that I don't have time to immediately look into. My testing/latest changes commit is on the zppy side: E3SM-Project/zppy@9127193.

cd /lcrc/group/e3sm/ac.forsyth2/zppy_pmp_hist/unique_id_20251007_3/v3.LR.amip_0101/post/scripts
grep -v "OK" *status
# pcmdi_diags_mean_climate_model_vs_obs_2005-2014.status:ERROR (11)
# pcmdi_diags_synthetic_plots_model_vs_obs.status:WAITING 954696
# pcmdi_diags_variability_modes_atm_model_vs_obs_2005-2014.status:ERROR (11)
# pcmdi_diags_variability_modes_cpl_model_vs_obs_2005-2014.status:ERROR (11)
cd /lcrc/group/e3sm/ac.forsyth2/zppy_pmp_amip/unique_id_20251007_3/v3.LR.amip_0101/post/scripts
grep -v "OK" *status
# pcmdi_diags_mean_climate_model_vs_obs_2005-2014.status:ERROR (11)
# pcmdi_diags_synthetic_plots_model_vs_obs.status:WAITING 954688
# pcmdi_diags_variability_modes_atm_model_vs_obs_2005-2014.status:ERROR (11)
# pcmdi_diags_variability_modes_cpl_model_vs_obs_2005-2014.status:ERROR (11)
cd /lcrc/group/e3sm/ac.forsyth2/zppy_pmp_hist/unique_id_20251007_3/v3.LR.amip_0101/post/scripts
grep -v "OK" *status
# pcmdi_diags_mean_climate_model_vs_obs_2005-2014.status:ERROR (11)
# pcmdi_diags_synthetic_plots_model_vs_obs.status:WAITING 954696
# pcmdi_diags_variability_modes_atm_model_vs_obs_2005-2014.status:ERROR (11)
# pcmdi_diags_variability_modes_cpl_model_vs_obs_2005-2014.status:ERROR (11)

If I can't resolve these errors quickly tomorrow, we'll have to consider merging an earlier working version of the branch.

Also, just to note from a testing perspective, E3SM-Project/zppy#732 introduces 74 new parameters to handle pcmdi_diags, which is more than double e3sm_diags's 34. That means many degrees of freedom in testing. Already I've noticed some string that should probably be boolean, but I'm concerned about changing them now given our time constraints in testing.

@zhangshixuan1987
Copy link
Copy Markdown
Contributor

zhangshixuan1987 commented Oct 7, 2025

@forsyth2 : Hi Ryan, the issues all pointed to the same errors to me:
File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/zi-pcmdi-diags-20251007-test1/lib/python3.13/site-packages/xarray/backends/scipy_.py", line 289, in prepare_variable raise ValueError( f"unexpected encoding for scipy backend: {list(variable.encoding)}" ) ValueError: unexpected encoding for scipy backend: ['zlib', 'complevel']

Seems that these are due to the update of the library. The lines that caused the errors seem to be lines 260 in "zppy_interfaces/pcmdi_diags/pcmdi_setup.py":

  • comp = dict(_FillValue=1.0e20, zlib=True, complevel=5)

I think that you may simply remove "zlib=True, complevel=5" and see if errors are gone. These setups are just trying to compress NC file.

@forsyth2
Copy link
Copy Markdown
Collaborator Author

forsyth2 commented Oct 7, 2025

Thanks @zhangshixuan1987 I will try testing that tonight or tomorrow.

@zhangshixuan1987
Copy link
Copy Markdown
Contributor

zhangshixuan1987 commented Oct 7, 2025

Thanks @zhangshixuan1987 I will try testing that tonight or tomorrow.

Let me know if you need help from me to run some tests for you. I can do it if you provide a preferred way for me to follow.

@forsyth2
Copy link
Copy Markdown
Collaborator Author

forsyth2 commented Oct 8, 2025

Thanks, I'm now running with that change, so we'll see what happens. I'll let you know if I need extra testing (if so, the example cfgs will be easier for you to run, as the test cfg has a complicated setup).

@forsyth2
Copy link
Copy Markdown
Collaborator Author

forsyth2 commented Oct 8, 2025 •
edited
Loading

First, let's look at the viewers

Second, let's look at the output directories

cd /lcrc/group/e3sm/ac.forsyth2/zppy_pmp_amip/unique_id_20251007_4/v3.LR.amip_0101/post/scripts
grep -v "OK" *status
# Success
cd /lcrc/group/e3sm/ac.forsyth2/zppy_pmp_hist/unique_id_20251007_4/v3.LR.amip_0101/post/scripts
grep -v "OK" *status
# Success
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/unique_id_test_20251007_4/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# pcmdi_diags_synthetic_plots_model_vs_obs.status:ERROR (11)

Third, let's check for errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_pmp_amip/unique_id_20251007_4/v3.LR.amip_0101/post/scripts
grep -in -A 1 error *.o*

gives:

grep output 
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:7991:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-7992-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:8160:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-8161-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:8329:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-8330-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:8498:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-8499-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:8667:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-8668-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:8836:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-8837-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:9005:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-9006-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:9174:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-9175-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:9343:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-9344-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:9512:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-9513-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:9681:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-9682-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:9850:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-9851-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:10019:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-10020-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:10130:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-10131-UnsupportedOperationException: getX called on empty Point
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:10236:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-10237-UnsupportedOperationException: getX called on empty Point
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:10400:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-10401-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:10511:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-10512-UnsupportedOperationException: getX called on empty Point
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:10675:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-10676-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:10844:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-10845-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:11013:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-11014-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:11182:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-11183-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:11351:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-11352-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:11520:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-11521-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:11689:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-11690-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:11858:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-11859-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:12027:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-12028-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:12196:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-12197-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:12365:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-12366-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:12534:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-12535-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183:12703:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955183-12704-'Tropics'

Looking at the amip example, the OCEAN & LAND plots are grayed out, and Extra-Tropics has no Portrait or ParCoord plots.

cd /lcrc/group/e3sm/ac.forsyth2/zppy_pmp_hist/unique_id_20251007_4/v3.LR.amip_0101/post/scripts
grep -in -A 1 error *.o*

gives:

grep output 
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:7991:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-7992-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:8160:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-8161-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:8329:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-8330-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:8498:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-8499-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:8667:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-8668-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:8836:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-8837-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:9005:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-9006-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:9174:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-9175-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:9343:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-9344-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:9512:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-9513-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:9681:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-9682-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:9850:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-9851-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:10019:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-10020-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:10130:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-10131-UnsupportedOperationException: getX called on empty Point
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:10236:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-10237-UnsupportedOperationException: getX called on empty Point
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:10400:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-10401-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:10511:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-10512-UnsupportedOperationException: getX called on empty Point
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:10675:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-10676-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:10844:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-10845-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:11013:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-11014-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:11182:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-11183-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:11351:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-11352-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:11520:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-11521-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:11689:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-11690-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:11858:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-11859-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:12027:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-12028-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:12196:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-12197-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:12365:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-12366-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:12534:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-12535-'Tropics'
--
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191:12703:error occured for  v3-LR 0101
pcmdi_diags_mean_climate_model_vs_obs_2005-2014.o955191-12704-'Tropics'
--
pcmdi_diags_synthetic_plots_model_vs_obs.o955194:176:2025-10-07 20:15:06,827 [ERROR]: synthetic_metrics_plotter.py(generate:170) >> Failed to handle metric=mean_climate: index 0 is out of bounds for axis 1 with size 0
pcmdi_diags_synthetic_plots_model_vs_obs.o955194-177-Traceback (most recent call last):
--
pcmdi_diags_synthetic_plots_model_vs_obs.o955194:224:IndexError: index 0 is out of bounds for axis 1 with size 0
pcmdi_diags_synthetic_plots_model_vs_obs.o955194-225-2025-10-07 20:15:06,838 [INFO]: synthetic_metrics_plotter.py(generate:165) >> Processing metric: variability_modes

Unlike the amip example, the synthetic plots report failing to handle mean_climate.

Looking at the historical example, the OCEAN & LAND plots have grayed out RMSE ParCoord plots, and Extra-Tropics has no Portrait or ParCoord plots.

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/unique_id_test_20251007_4/v3.LR.historical_0051/post/scripts
grep -in -A 1 error *.o*
# Too much output to copy here
grep -in error *.o* | wc -l
# 403
# (That's 403 matches)
grep -in "failed to handle" *.o*
# pcmdi_diags_synthetic_plots_model_vs_obs.o955178:51:2025-10-07 19:34:33,812 [ERROR]: synthetic_metrics_plotter.py(generate:170) >> Failed to handle metric=mean_climate: No synthetic mean climate metrics found for model: v3.LR.historical_0051
# pcmdi_diags_synthetic_plots_model_vs_obs.o955178:71:2025-10-07 19:34:34,840 [ERROR]: synthetic_metrics_plotter.py(generate:170) >> Failed to handle metric=variability_modes: No Synthetic MoVs Metrics Data For v3.LR.historical_0051, Aborting.

Looking at the test case, no viewer is created at all.

Fourth, debugging

@zhangshixuan1987 Could you take a look at these logs? I'm not seeing obvious answers here. The output directories are noted above. The provenance cfgs can be found with ls *.cfg in each .../post/scripts directory. Thanks!

(Also, I've pushed my latest changes: c430c3f & E3SM-Project/zppy@43eb226)

@zhangshixuan1987
Copy link
Copy Markdown
Contributor

zhangshixuan1987 commented Oct 8, 2025 •
edited
Loading

@forsyth2 :

Looking at the [amip example](https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_pmp_amip/unique_id_20251007_4/v3.LR.amip_0101/pcmdi_diags/model_vs_obs/viewer/), the OCEAN & LAND plots are grayed out, and Extra-Tropics has no Portrait or ParCoord plots.

cd /lcrc/group/e3sm/ac.forsyth2/zppy_pmp_hist/unique_id_20251007_4/v3.LR.amip_0101/post/scripts
grep -in -A 1 error *.o*

This is actually not an error. There’s no CMIP6 data available for the OCEAN and LAND regional metrics in the mean-climate category, so those figures are expected to be missing.

I didn’t remove the OCEAN & LAND region plots because these metrics will appear for historical simulations. This example was mainly to illustrate what happens when the configuration includes:

[[ synthetic_plots ]]
clim_regions = "global,ocean,land"

but we do not have the metrics data for specific regions such as ocean and land. If users prefer not to show the gray placeholder figures on the viewer page, they can simply set:

clim_regions = "global"

@forsyth2
Copy link
Copy Markdown
Collaborator Author

forsyth2 commented Oct 8, 2025

@chengzhuzhang and I have decided that in the interest of time re: the E3SM Unified release, we are going to merge the two PMP PRs (this one & E3SM-Project/zppy#732), with the understanding there will be follow-up PRs made during the E3SM Unified Testing Period to add finishing touches and/or fix any bugs that are discovered.

@forsyth2
Copy link
Copy Markdown
Collaborator Author

forsyth2 commented Oct 8, 2025

I am specifically merging this PR before addressing #37 for the purpose of making sure conda's package versions work together properly.

@forsyth2 forsyth2 merged commit 539d3db into main Oct 8, 2025
5 checks passed
@zhangshixuan1987
Copy link
Copy Markdown
Contributor

zhangshixuan1987 commented Oct 8, 2025

@forsyth2 : I rebased my code to your latest changes:c430c& E3SM-Project/zppy@43eb226)). I rerun the diagnostics on my side with the examples/post.v3.LR.historical.0101.cfg. The zppy-pcmdi seems to be run without any issues, as the viewer page below was generated:

In your historical example There are some noticeable features in the summary table section:

Screenshot 2025-10-08 at 9 27 18 AM

The greyed ParCoord plot for RMSE in "Ocean" and "Land" rows, but this is not an issue to me, as you click the Portrait plot, you will see that there is no data for the CMIP6 reference models. The parallel coordinate plots were likely not able to be constructed with only one model point. Therefore, I do not think this is a code bug, and the user can easily remove the OCEAN and LAND region from the list of the "clim_regions" in ".cfg" files to not show these figures.

In addition, there is a small bug in the final viewer page, i.e. the section EMoVs is shifted to the right compared to the Mean Climate:

Screenshot 2025-10-08 at 9 45 17 AM

I’ve added a modification in viewer.py to address this issue and pushed the update. In addition, I included safeguard checks and warning messages in synthetic_metrics_plotter.py to help catch potential issues and failure in a normal situation (e.g., metrics data is missing or is NAN etc. while the whole processing is complete).

@forsyth2
Copy link
Copy Markdown
Collaborator Author

forsyth2 commented Oct 8, 2025

Ok, thanks @zhangshixuan1987, we will address further fixes in a follow-up PR.

@forsyth2 forsyth2 deleted the add-pcmdi branch October 8, 2025 18:06
@zhangshixuan1987
Copy link
Copy Markdown
Contributor

zhangshixuan1987 commented Oct 8, 2025 •
edited
Loading

Ok, thanks @zhangshixuan1987, we will address further fixes in a follow-up PR.

@forsyth2 : Sure. The only change is in zppy-interface, which is related to the module for pcmdi diagnostics. The revised feature was in commit 92108680 . Just listed here as a record for the next stage's work. A clean run with the changed code and viewer page can be seen here:

Hopefully all the things are now good enough.

This was referenced Oct 9, 2025
Merged
Merged
xylar
xylar reviewed Oct 10, 2025
View reviewed changes
Comment thread conda/dev.yml
- lxml
- matplotlib-base >=3.8.2,<3.10
- mpas_tools >=0.21.0
- mpas_tools
Copy link
Copy Markdown
Contributor

@xylar xylar Oct 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@forsyth2, @zhangshixuan1987, @chengzhuzhang, this change should not have been made. Do you know why it happened? It's not really harmful but it isn't related to pcmdi at all.

Copy link
Copy Markdown
Contributor

@zhangshixuan1987 zhangshixuan1987 Oct 10, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xylar: I think @forsyth2 would know more about the conda environment upgrade. To my understanding, the conda environment here is shared across the entire zppy-interface rather than being specific to just the pcmdi component. @forsyth2 tried to rebase this branch to the latest master branch to keep consistency.

Copy link
Copy Markdown
Contributor

@xylar xylar Oct 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this PR should ideally only have changed dependencies that are related to PCMDI, which mpas_tools is not. I do not think the bounds should have been dropped at all on mpas_tools but certainly not in this PR. The same with the changes to mypy below.

It's not a huge deal but these should be fixed in a follow-up PR.

Copy link
Copy Markdown
Collaborator Author

@forsyth2 forsyth2 Oct 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xylar This change and the other one you mentioned (#25 (comment)) were introduced in db29aab.

That commit was made after rebasing off the latest main (i.e., the one that had the conda changes of #33). I therefore assumed any new conda issues were because of changes in this PR.

That commit was the result of my iterating with Claude + some input from @andrewdnolan. I suppose the conclusion here is that Claude suggested unnecessary or unrelated changes.

Copy link
Copy Markdown
Contributor

@xylar xylar Oct 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I don't think those changes were correct. Let's make a PR to fix them.

Copy link
Copy Markdown
Collaborator Author

@forsyth2 forsyth2 Oct 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do.

Copy link
Copy Markdown
Contributor

@xylar xylar Oct 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went ahead and fix this in #42

xylar
xylar reviewed Oct 10, 2025
View reviewed changes
Comment thread conda/dev.yml
Comment on lines +51 to +53
# pip dependencies
- pip:
- mypy==1.18.2
Copy link
Copy Markdown
Contributor

@xylar xylar Oct 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should also not have been part of this PR.

@forsyth2 forsyth2 mentioned this pull request Oct 10, 2025
Merged
7 tasks
@xylar xylar mentioned this pull request Oct 10, 2025
Merged
7 tasks
This was referenced Oct 13, 2025
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xylar xylar xylar left review comments

@zhangshixuan1987 zhangshixuan1987 zhangshixuan1987 left review comments

@chengzhuzhang chengzhuzhang Awaiting requested review from chengzhuzhang

Assignees

@forsyth2 forsyth2

Labels

priority: high High priority task (for next release)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@forsyth2 @zhangshixuan1987 @chengzhuzhang @xylar