Skip to content

Zppy pcmdi new #656

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zhangshixuan1987 wants to merge 23 commits into from
Closed

Zppy pcmdi new #656

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
43550e2
Reformat by pre-commit run
zhangshixuan1987 Dec 2, 2024
a9e8708
Small bug fix in the first line to avoid failure
zhangshixuan1987 Dec 26, 2024
561c82c
update the workflow to be modulized
zhangshixuan1987 Dec 30, 2024
b6c638e
move the needed modules to inclusions
zhangshixuan1987 Dec 30, 2024
23b6fa2
clean up the reduandent files for zppy-pcmdi
zhangshixuan1987 Dec 30, 2024
f7fcaaa
remove rltcre and rstcre from the default list of cmip_vars as these
zhangshixuan1987 Dec 31, 2024
7a52e32
adjustment on the workflow
zhangshixuan1987 Jan 3, 2025
2d38e9b
Add revisions on the sythentic plot
zhangshixuan1987 Jan 6, 2025
67ca00d
bug fix
zhangshixuan1987 Jan 7, 2025
205a479
reorgnize the including files
zhangshixuan1987 Jan 8, 2025
2a811c8
Revise the workflow related to zppy-pcmdi
zhangshixuan1987 Jan 10, 2025
0d3c387
Revise mpi to ensure the writing process finished
zhangshixuan1987 Jan 10, 2025
9ff8c18
Bug fix
zhangshixuan1987 Jan 14, 2025
b2b156d
Bug fix
zhangshixuan1987 Jan 15, 2025
ae24608
bug fix
zhangshixuan1987 Jan 15, 2025
902b2c5
Rename fixed directory to be clearer
zhangshixuan1987 Jan 19, 2025
c281986
Change the default set of cmip6 data for synthetic plots
zhangshixuan1987 Jan 19, 2025
43dafdb
Refine the workflow
zhangshixuan1987 Feb 4, 2025
d30ade2
Refine workflow
zhangshixuan1987 Feb 5, 2025
bdf224f
Add Enso metrics summary plot
zhangshixuan1987 Feb 18, 2025
0c16188
Refinement on the code and add viewer page
zhangshixuan1987 Apr 26, 2025
ecaa510
bug fix
zhangshixuan1987 Apr 26, 2025
ae3e19f
Refine the code to move "viewer" into the results directory
zhangshixuan1987 Apr 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
201 changes: 107 additions & 94 deletions zppy/defaults/default.ini
View file
Open in desktop
Comment thread
zhangshixuan1987 marked this conversation as resolved.
Copy link
Copy Markdown
Collaborator Author

@zhangshixuan1987 zhangshixuan1987 Apr 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zhangshixuan1987: This file has been recoded and refined, and is ready for review.

Original file line number Diff line number Diff line change
Expand Up @@ -138,20 +138,22 @@ scratch = string(default="")
[pcmdi_diags]
backend = string(default="mpl")
cfg = string(default="")
# File of cmip variable lists (cmip6 convention)
cmip_variables = string(default="pcmdi_diags/cmip_variables.json")
# Title of zppy-pcmdi diagnostics
pcmdi_webtitle = string(default="E3SM-PMP Diagnostics")
# Version of zppy-pcmdi code
pcmdi_version = string(default="v3.8.2")
# File of specified regions for mean climate calculation
regions_specs = string(default="pcmdi_diags/regions_specs.json")
# File of derived variables
derived_variable = string(default="pcmdi_diags/derived_variable.json")
regions_specs = string(default="pcmdi_data/region/regions_specs.json")
# File of observation data name for mean climate calculation
reference_alias = string(default="pcmdi_diags/reference_alias.json")
# File of fuction to generate land/sea mask
process_sftlf = string(default="pcmdi_diags/process_sftlf.py")
# File of fuction to generate mean climate metrics figure
clim_plot_parser = string(default="pcmdi_diags/mean_climate_plot_parser.py")
# File of module to plot mean climate metrics figure
clim_plot_driver = string(default="pcmdi_diags/mean_climate_plot_driver.py")
reference_alias = string(default="pcmdi_data/reference/reference_alias.json")
# Utility file with functions for zppy-pcmdi data processing
pcmdi_zppy_util = string(default="pcmdi_data/utility/pcmdi_zppy_util.py")
# Utility file with functions for zppy-pcmdi viewer page processing
pcmdi_viewer_util = string(default="pcmdi_data/utility/pcmdi_viewer_util.py")
# Template files for zppy-pcmdi viewer page processing
pcmdi_viewer_template = string(default="pcmdi_data/viewer")
# File of variable list to generate synthetic metrics plot
synthetic_metrics = string(default="pcmdi_data/synthetic_metrics/synthetic_metrics_list.json")
# Path to observation time-series data
# Required for "mean_climate","variability_mode","enso"
obs_ts = string(default="")
Expand All @@ -160,69 +162,71 @@ obs_ts = string(default="")
obs_sets = string(default="default")
# options specific for constructing pcmdi preferred file name conventions
# required for "model_vs_obs" comparison
cmip_name = string(default="e3sm.historical.v3-LR.0051")
model_name = string(default="e3sm.historical.v3-LR.0051")
model_tableID = string(default="Amon")
# required for "model_vs_model" comparison
cmip_name_ref = string(default="e3sm.historical.v3-LR.0051")
# options shared by pcmdi
pmp_debug = string(default=False)
model_name_ref = string(default="ERA5")
model_tableID_ref = string(default="Amon")
# variables in the cmip6 table that can be potentially used by pcmdi
# this list depends on the definition of cmip variable
# required for "mean climate" diagnostics
cmip_vars = string(default="pr,prw,psl,rlds,rldscs,rlut,rlutcs,rsut,rsutcs,rsds,rsdscs,rsdt,rsus,rsuscs,rlus,rtmt,sfcWind,tas,tauu,tauv,ts,ta,ua,va,zg")
# flag to process the land/sea mask within pcmdi
generate_sftlf = string(default=True)
# variables to be used by the pcmdi diagnostics
# needs to setup for each subsections
vars = string(default="")
# needs to setup for each subsections, defalut setup is the mean climate metrics
vars = string(default="pr,prw,psl,rlds,rldscs,rltcre,rstcre,rlut,rlutcs,rsds,rsdscs,rsdt,rsus,rsuscs,rlus,rsut,rtmt,sfcWind,tas,tauu,tauv,ts,ta-200,ta-850,ua-200,ua-850,va-200,va-850,zg-500")
# sets of diagnostics from pcmdi package
sets = string_list(default=list("mean_climate","variability_mode_atm","variability_mode_cpl","enso"))
sets = string_list(default=list("mean_climate","variability_modes_atm","variability_modes_cpl","enso","synthetic_plots"))
# options to identify subset of pcmdi drivers ("mean_climate","variability_mode","enso")
subset = string(default="")
#options for cmip model metrics data from pcmdi
#group of pcmdi generated cmip metrics data (mip.exp.version)
pcmdi_data_set=string(default="cmip6.historical.v20220928")
# path to pcmdi generated cmip metrics data
pcmdi_data_path=string(default="")
##########################################################################################
# below followed the setup in e3sm_diag but used for PCMDI workflow
##########################################################################################
# See url<need to work on document later>
multiprocessing = boolean(default=True)
# See url<need to work on document later>
num_workers = integer(default=24)
# See url<need to work on document later>
figure_format = string(default="png")
# comparision type (same as e3sm_diag)
run_type = string(default="model_vs_obs")
# Used to label the results directory
# Options are "model_vs_obs" and "model_vs_model"
tag = string(default="model_vs_obs")
subsets = string_list(default=list("mean_climate","variability_modes","enso"))
###########################################################################################
# Required for run_type="model_vs_model" runs, different from e3sm_diag,
# model_vs_model in pcmdi referred to the comparision of two model simulations
# with observations and cmip models.
###########################################################################################
#path for reference model data (time series)
reference_data_path = string(default="")
reference_data_path_ts = string(default="")
# pcmdi_diags.py will set to match `years` if not specified
ref_years = string_list(default=list(""))
# End year (i.e., the last year to use) for the reference data
ref_end_yr = string(default="")
# The years increment for reference data
ts_num_years_ref = integer(default=5)
# Final year (i.e., the last available year) for the reference data
ref_final_yr = string(default="")
# Start year for the reference data
ref_start_yr = string(default="")
# reference model name
ref_name = string(default="")
# The years increment for reference data
ts_num_years_ref = integer(default=5)
# The years range for test model data
ts_years = string_list(default=list(""))
# The years increment for test model data
ts_num_years = integer(default=5)
# Set to true to swap test and ref when run_type="model_vs_model"
swap_test_ref = boolean(default=False)
##########################################################################################
# below followed the setup in e3sm_diag but used for PCMDI workflow
##########################################################################################
# See url<need to work on document later>
multiprocessing = boolean(default=True)
# See url<need to work on document later>
num_workers = integer(default=24)
# See url<need to work on document later>
figure_format = string(default="png")
# comparision type (same as e3sm_diag)
run_type = string(default="model_vs_obs")
# Used to label the results directory
# Options are "model_vs_obs" and "model_vs_model"
tag = string(default="model_vs_obs")
##########################################################################################
# options for pcmdi mode varibility diagnostics
# vars = "psl" for atm_modes
# vars = "ts" for cpl_modes
#########################################################################################
#name of atmospheric modes varibility
atm_modes = string_list(default=list("NAM","NAO","PNA","NPO","SAM","PSA1","PSA2"))
atm_modes = string(default="NAM,NAO,PNA,NPO,SAM,PSA1,PSA2")
#name of coupled modes varibility
cpl_modes = string_list(default=list("PDO","NPGO","AMO"))
cpl_modes = string(default="PDO,NPGO,AMO")
#keywards for unit conversion in pcmdi (model)
ModUnitsAdjust = string(default="")
#keywards for unit conversion in pcmdi (observation)
Expand All @@ -236,17 +240,15 @@ RmDomainMean = string(default=True)
EofScaling = string(default=False)
ConvEOF = string(default=True)
CBF = string(default=True)
cmec = string(default=True)
update_json = string(default=False)
plot_obs = string(default=True)
plot = string(default=True)
nc_out_obs = string(default=True)
nc_out = string(default=True)
mov_plot_obs = string(default=True)
mov_plot_model = string(default=True)
mov_nc_out_obs = string(default=True)
mov_nc_out_model = string(default=True)
##########################################################################################
# options for pcmdi enso diagnostics
# vars = "psl,pr,prsn,ts,tas,tauu,tauv,hflx,hfss,rlds,rsds,rlus,rlut,rsdt"
##########################################################################################
groups = string_list(default=list("ENSO_perf","ENSO_proc","ENSO_tel"))
enso_groups = string(default="ENSO_perf,ENSO_proc,ENSO_tel")
##########################################################################################
# optional for mean climate diagnostics
# vars = "pr,prw,psl,rlds,rldscs,rltcre,rstcre,rlut,rlutcs,rsds,rsdscs,rsdt,rsus,rsuscs,
Expand All @@ -255,95 +257,106 @@ groups = string_list(default=list("ENSO_perf","ENSO_proc","ENSO_tel"))
##########################################################################################
# model data grid after remapping
grid = string(default="180x360_aave")
#flag to turn on regional mean climate metrics
regional = string(default="y")
#default regions for mean climate metrics data
regions = string(default="global,ocean,land,NHEX,SHEX,TROPICS,NHEX_ocean,SHEX_ocean,NHEX_land,SHEX_land,ocean_50S50N")
#more options can be found at "regions_specs"
#regions = string(default="global,ocean,land,NHEX,SHEX,TROPICS,NHEX_ocean,SHEX_ocean,NHEX_land,SHEX_land,ocean_50S50N")
regions = string(default="global,ocean,land,NHEX,SHEX,TROPICS")
# save derived climatology data
save_test_clims = string(default=True)
# method to determine the way to process mean climate data
# default used nco instead of pcmdi built-in function
climatology_process_method = string(default="nco")
# Regridding by pcmdi (default is to regrid data to 2.5x2.5 grid for diagnostic metrics)
# Required for mean climate
# OPTIONS: '2.5x2.5' or an actual cdms2 grid object
target_grid = string(default="2.5x2.5")
# OPTIONS: String for description on the selected grid
target_grid_string = string(default="2p5x2p5")
# Regridding by pcmdi (required for mean climate)
# OPTIONS: '1x1' for e3sm, '2.5x2.5' for cmip, or an actual cdms2 grid object,
target_grid = string(default="1x1")
# OPTIONS: "1px1p" for e3sm, "2p5x2p5" for cmip, description on the selected grid
target_grid_string = string(default="1px1p")
# OPTIONS: 'regrid2','esmf'
regrid_tool = string(default="esmf")
# OPTIONS: 'linear','conservative', only if tool is esmf
regrid_method = string(default="regrid2")
# OPTIONS: 'linear','conservative', only if tool is esmf
regrid_method_ocn = string(default="conservative")
# setup for parallel coordinate plots (hide makers for sigle model)
parcord_show_markers = string(default=False)
# setup for portrait plots (add vertical line to separate test and reference models)
portrait_vertical_line = string(default=True)
###################################################################
#options for synthetic plots with exisiting pcmdi cmip model metrics
###################################################################
synthetic_sets = string(default="portrait,parcoord")
# path to pcmdi generated cmip metrics data
cmip_enso_dir = string(default="")
cmip_clim_dir = string(default="")
cmip_movs_dir = string(default="")
#group of pcmdi generated cmip metrics data (mip.exp.version)
cmip_clim_set = string(default="cmip6.historical.v20250108")
cmip_movs_set = string(default="cmip6.historical.v20220825")
cmip_enso_set = string(default="cmip6.historical.v20210620")
# options shared by pcmdi
pcmdi_debug = string(default=False)

[[__many__]]
backend = string(default=None)
cfg = string(default=None)
vars = string(default=None)
grid = string(default=None)
cmip_metadata = string(default=None)
cmip_variables = string(default=None)
pcmdi_data_set = string(default=None)
pcmdi_data_path = string(default=None)
derived_variable = string(default=None)
cmip_enso_set = string(default=None)
cmip_clim__set = string(default=None)
cmip_movs_set = string(default=None)
cmip_enso_dir = string(default=None)
cmip_clim_dir = string(default=None)
cmip_movs_dir = string(default=None)
synthetic_sets = string(default=None)
reference_alias = string(default=None)
regions_specs = string(default=None)
process_sftlf = string(default=None)
pcmdi_webtitle = string(default=None)
pcmdi_version = string(default=None)
pcmdi_zppy_util = string(default=None)
pcmdi_viewer_util = string(default=None)
pcmdi_viewer_template = string(default=None)
synthetic_metrics = string(default=None)
multiprocessing = boolean(default=None)
num_workers = integer(default=None)
obs_ts = string(default=None)
figure_format = string(default=None)
ref_end_yr = string(default=None)
ref_final_yr = string(default=None)
ref_name = string(default=None)
ref_start_yr = string(default=None)
ref_years = string_list(default=None)
ref_start_yr = string(default=None)
ref_final_yr = string(default=None)
reference_data_path = string(default=None)
reference_data_path_ts = string(default=None)
run_type = string(default=None)
sets = string_list(default=None)
swap_test_ref = boolean(default=None)
tag = string(default=None)
ts_years = string_list(default=None)
ts_num_years = integer(default=None)
ts_num_years_ref = integer(default=None)
climatology_process_method = string(default=None)
target_grid = string(default=None)
target_grid_string = string(default=None)
regrid_tool = string(default=None)
regrid_method = string(default=None)
regrid_method_ocn = string(default=None)
obs_sets = string(default=None)
regions = string(default=None)
regional = string(default=None)
save_test_clims = string(default=None)
seasons = string(default=None)
RmDomainMean = string(default=None)
EofScaling = string(default=None)
ConvEOF = string(default=None)
CBF = string(default=None)
cmec = string(default=None)
update_json = string(default=None)
subset = string(default=None)
subsets = string_list(default=None)
landmask = string(default=None)
frequency = string(default=None)
generate_sftlf = string(default=None)
atm_modes = string_list(default=None)
cpl_modes = string_list(default=None)
groups = string_list(default=None)
atm_modes = string(default=None)
cpl_modes = string(default=None)
enso_groups = string(default=None)
ModUnitsAdjust = string(default=None)
ObsUnitsAdjust = string(default=None)
cmip_name = string(default=None)
cmip_name_ref = string(default=None)
pmp_debug = string(default=None)
nc_out_obs = string(default=None)
nc_out = string(default=None)
plot_obs = string(default=None)
plot = string(default=None)
parcord_show_markers = string(default=None)
portrait_vertical_line = string(default=None)
model_name = string(default=None)
model_name_ref = string(default=None)
model_tableID = string(default=None)
model_vars = string(default=None)
mov_nc_out_obs = string(default=None)
mov_nc_out_model = string(default=None)
mov_plot_obs = string(default=None)
mov_plot_model = string(default=None)
pcmdi_debug = string(default=None)

[e3sm_diags]
# See https://e3sm-project.github.io/e3sm_diags/_build/html/master/available-parameters.html
Expand Down
Loading