Update references to Hera /scratch1 and /scratch2

[RussTreadon-NOAA]

Hera /scratch1 and /scratch2 are scheduled to be decommissioned 8/5/2025. GDASApp contains several references to /scratch and /scratch2

/scratch1

• ./ush/ufoeval/config_jedi.yaml: HOMEgfs: /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test
• ./ush/ufoeval/config_jedi.yaml: STAGEDIR: /scratch1/NCEPDEV/da/Russ.Treadon/STAGEDIR
• ./ush/ufoeval/config_jedi.yaml: RUNDIR: /scratch1/NCEPDEV/stmp2/Russ.Treadon/RUNDIRS
• ./ush/ufoeval/config_jedi.yaml: APPEXE: /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/exec/gdas.x
• ./ush/ufoeval/config_jedi.yaml: INCEXE: /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/exec/fv3jedi_fv3inc.x
• ./ush/ufoeval/config_gsi.yaml: HOMEgfs: /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test
• ./ush/ufoeval/config_gsi.yaml: STAGEDIR: /scratch1/NCEPDEV/da/Russ.Treadon/STAGEDIR
• ./ush/ufoeval/config_gsi.yaml: RUNDIR: /scratch1/NCEPDEV/stmp2/Russ.Treadon/RUNDIRS
• ./ush/ufoeval/config_gsi.yaml: APPEXE: /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/exec/gsi.x
• ./ush/ufoeval/README_phase3: Hera: /scratch1/NCEPDEV/da/Russ.Treadon/STAGEDIR
• ./ush/ufoeval/run_ufo_hofx_test.sh: workdir=/scratch1/NCEPDEV/stmp2/$LOGNAME/ufoeval/$cycle/${obtype}_noqc
• ./ush/ufoeval/run_ufo_hofx_test.sh: workdir=/scratch1/NCEPDEV/stmp2/$LOGNAME/ufoeval/$cycle/${obtype}
• ./ush/ufoeval/run_ufo_hofx_test.sh: GDASApp=${GDASApp:-/scratch1/NCEPDEV/da/$LOGNAME/git/GDASApp/} # Change this to your own branch
• ./ush/ufoeval/run_ufo_hofx_test.sh: JCBinstall=${JCBinstall:-/scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/opt}
• ./ush/ufoeval/run_ufo_hofx_test.sh: export Datapath='/scratch1/NCEPDEV/da/Emily.Liu/UFO_eval/data/gsi_geovals_l127/nofgat_aug2021/'$dataprocdate
• ./ush/ufoeval/run_ufo_hofx_test.sh: FixDir=/scratch1/NCEPDEV/da/Cory.R.Martin/GDASApp/fix
• ./ush/ufsda/yamltools.py: 'datapath': '/scratch1/NCEPDEV/global/glopara/fix/orog/20220805/C'+str(case)+'/',
• ./test/marine/CMakeLists.txt: "/scratch1/NCEPDEV/da/common/ci/bufr"
• ./prototypes/gen_prototype.sh: HOMEDIR: "/scratch1/NCEPDEV/da/${USER}"
• ./prototypes/configs/cp0.sh:PROTOROOT=/scratch1/NCEPDEV/da/Cory.R.Martin/GDAS_prototypes
• ./prototypes/configs/cp0.sh:DUMPDIR=/scratch1/NCEPDEV/da/role.jedipara/dump/
• ./prototypes/configs/cp0.sh:ICSDIR=/scratch1/NCEPDEV/stmp4/Cory.R.Martin/tmp_ics
• ./modulefiles/EVA/hera.lua:local pyenvpath = "/scratch1/NCEPDEV/da/python/envs/"
• ./modulefiles/GDAS/hera.intel.lua:setenv("CRTM_FIX","/scratch1/NCEPDEV/da/role.jedipara/GDASApp/fix/crtm/2.4.0")
• ./modulefiles/GDAS/hera.intel.lua:setenv("GDASAPP_TESTDATA","/scratch1/NCEPDEV/da/role.jedipara/GDASApp/testdata")
• ./modulefiles/GDAS/hera.intel.lua:setenv("GDASAPP_UNIT_TEST_DATA_PATH", "/scratch1/NCEPDEV/da/role.jedipara/GDASApp/unittestdata")
• ./utils/test/testdata/icec_amsr2_south_2.cdl: :history = "Mon Sep 25 15:42:01 2023: ncks -d Number_of_X_Dimension,100,800,100 -d Number_of_Y_Dimension,100,800,100 /scratch1/NCEPDEV/stmp4/Shastri.Paturi/forAndrew/gdas.20210701/00/icec/AMSR2-SEAICE-SH_v2r2_GW1_s202107010120180_e202107010259170_c202107010340150.nc icec_amsr2_south_2.nc\nCreated by GAASP Version 2.0, Release 2.0" ;
• ./utils/test/testdata/icec_amsr2_south_1.cdl: :history = "Mon Sep 25 15:26:46 2023: ncks -d Number_of_X_Dimension,100,800,100 -d Number_of_Y_Dimension,100,800,100 /scratch1/NCEPDEV/stmp4/Shastri.Paturi/forAndrew/gdas.20210701/00/icec/AMSR2-SEAICE-SH_v2r2_GW1_s202106302341190_e202107010120160_c202107010202390.nc icec_amsr2_south.nc\nCreated by GAASP Version 2.0, Release 2.0" ;
• ./utils/test/testdata/sss_smos_1.cdl: :history = "Mon Oct 2 16:01:51 2023: ncks -d n_grid_points,100,92700,1800 -v SSS_corr,Sigma_SSS_corr,Latitude,Longitude,Mean_acq_time,Dg_quality_SSS_corr /scratch1/NCEPDEV/stmp4/Shastri.Paturi/forAndrew/gdas.20210701/00/SSS/SM_OPER_MIR_OSUDP2_20210630T210913_20210630T220228_700_001_1.nc sss_smos_1_sub.nc" ;
• ./utils/test/testdata/sss_smos_2.cdl: :history = "Mon Oct 2 16:02:44 2023: ncks -d n_grid_points,100,92700,1800 -v SSS_corr,Sigma_SSS_corr,Latitude,Longitude,Mean_acq_time,Dg_quality_SSS_corr /scratch1/NCEPDEV/stmp4/Shastri.Paturi/forAndrew/gdas.20210701/00/SSS/SM_OPER_MIR_OSUDP2_20210630T215911_20210630T225230_700_001_1.nc sss_smos_2_sub.nc" ;
• ./utils/test/testdata/icec_amsr2_north_2.cdl: :history = "Mon Sep 25 15:41:32 2023: ncks -d Number_of_X_Dimension,400,600,100 -d Number_of_Y_Dimension,400,700,100 /scratch1/NCEPDEV/stmp4/Shastri.Paturi/forAndrew/gdas.20210701/00/icec/AMSR2-SEAICE-NH_v2r2_GW1_s202107010120180_e202107010259170_c202107010340150.nc icec_amsr2_north_2.nc\nCreated by GAASP Version 2.0, Release 2.0" ;
• ./utils/test/testdata/icec_amsr2_north_1.cdl: :history = "Mon Sep 25 15:10:33 2023: ncks -d Number_of_X_Dimension,400,600,100 -d Number_of_Y_Dimension,400,700,100 /scratch1/NCEPDEV/stmp4/Shastri.Paturi/forAndrew/gdas.20210701/00/icec/AMSR2-SEAICE-NH_v2r2_GW1_s202106302341190_e202107010120160_c202107010202390.nc output.nc\nCreated by GAASP Version 2.0, Release 2.0" ;
• ./utils/test/testdata/sss_smap_2.cdl: :history = "Mon Oct 2 18:36:57 2023: ncks -d phony_dim_0,20,70,25 -d phony_dim_1,30,800,40 -d phony_dim_2,2 -v lat,lon,smap_sss,smap_sss_uncertainty,quality_flag,row_time /scratch1/NCEPDEV/stmp4/Shastri.Paturi/forAndrew/gdas.20210701/00/SSS/SMAP_L2B_SSS_NRT_34258_A_20210630T231436.h5 sss_smap_2_sub.nc" ;
• ./utils/test/testdata/sss_smap_1.cdl: :history = "Mon Oct 2 18:36:14 2023: ncks -d phony_dim_0,20,70,25 -d phony_dim_1,30,800,40 -d phony_dim_2,2 -v lat,lon,smap_sss,smap_sss_uncertainty,quality_flag,row_time /scratch1/NCEPDEV/stmp4/Shastri.Paturi/forAndrew/gdas.20210701/00/SSS/SMAP_L2B_SSS_NRT_34257_A_20210630T213609.h5 sss_smap_1_sub.nc" ;
• ./utils/test/testdata/ghrsst_sst_ma_202103241540.cdl: :history = "Thu Mar 28 17:03:57 2024: ncks -d lon,11826,11835 -d lat,602,606 /scratch1/NCEPDEV/global/glopara/data/experimental_obs/gdas.20210324/18/ocean/sst/20210324154000-OSPO-L3U_GHRSST-SSTsubskin-AVHRRF_MA-ACSPO_V2.70-v02.0-fv01.0.nc ghrsst_sst_ma_202103241540.nc\nCreated by the L2P-to-L3U conversion tool, which was developed and provided by NOAA/NESDIS/STAR and CCNY. The version is 4.1.5" ;
• ./utils/test/testdata/ghrsst_sst_ma_202103241550.cdl: :history = "Thu Mar 28 17:06:08 2024: ncks -d lon,11826,11835 -d lat,602,606 /scratch1/NCEPDEV/global/glopara/data/experimental_obs/gdas.20210324/18/ocean/sst/20210324155000-OSPO-L3U_GHRSST-SSTsubskin-AVHRRF_MA-ACSPO_V2.70-v02.0-fv01.0.nc ghrsst_sst_ma_202103241550.nc\nCreated by the L2P-to-L3U conversion tool, which was developed and provided by NOAA/NESDIS/STAR and CCNY. The version is 4.1.5" ;
• ./utils/test/CMakeLists.txt: "/scratch1/NCEPDEV/da/common/ci/obs"

/scratch2

• ./test/aero/global-workflow/jjob_var_final.sh: export UTILROOT="/scratch2/NCEPDEV/ensemble/save/Walter.Kolczynski/hpc-stack/intel-18.0.5.274/prod_util/1.2.2"
• ./test/aero/global-workflow/jjob_var_init.sh: export UTILROOT="/scratch2/NCEPDEV/ensemble/save/Walter.Kolczynski/hpc-stack/intel-18.0.5.274/prod_util/1.2.2"
• ./test/aero/global-workflow/jjob_var_run.sh: export UTILROOT="/scratch2/NCEPDEV/ensemble/save/Walter.Kolczynski/hpc-stack/intel-18.0.5.274/prod_util/1.2.2"
• ./test/marine/CMakeLists.txt: "/scratch2/NCEPDEV/ocean/Guillaume.Vernieres/data/static/common/RECCAP2_region_masks_all_v20221025.nc"

GDASApp PR #1830 updates paths for ush/ufoeval/run_ufo_hofx_test.sh

This issue is opened to examine other instances of /scratch1 and /scratch2 and update as appropriate.

Notes:

• single strikethrough indicates no change needed to given file
• double strikethrough indicates file or directory removed
• italicized file is modified in via PR #1830

[RussTreadon-NOAA]

Thank you @ADCollard for opening PR #1830. This is a timely reminder that we need to check and update other /scratch1 and /scratch2 paths in GDASApp.

[RussTreadon-NOAA]

Work for this issue will be done in bugfix/hera_scratch

[RussTreadon-NOAA]

I am not familar with all the files with /scratch1 and /scratch2 references.

Perhaps we no longer need some of the listed files. If this is the case, let me know and I'll remove unnecessary files.

Other files we need but I'm not sure what the correct /scratch3 or /scratch4 path is. These files are listed below with the developer(s) I think can help answer the What do we do? question. If you're not the right person to contact for given file, who should I contact?

Thank you for your help in figuring out the correct way to update the paths in these files.

@CoryMartin-NOAA

./prototypes/gen_prototype.sh:  HOMEDIR: "/scratch1/NCEPDEV/da/${USER}"
./prototypes/configs/cp0.sh:PROTOROOT=/scratch1/NCEPDEV/da/Cory.R.Martin/GDAS_prototypes
./prototypes/configs/cp0.sh:DUMPDIR=/scratch1/NCEPDEV/da/role.jedipara/dump/
./prototypes/configs/cp0.sh:ICSDIR=/scratch1/NCEPDEV/stmp4/Cory.R.Martin/tmp_ics

@AndrewEichmann-NOAA , @guillaumevernieres

./ush/ufsda/yamltools.py:                'datapath': '/scratch1/NCEPDEV/global/glopara/fix/orog/20220805/C'+str(case)+'/',
./utils/test/testdata/icec_amsr2_south_2.cdl:           :history = "Mon Sep 25 15:42:01 2023: ncks -d Number_of_X_Dimension,100,800,100 -d Number_of_Y_Dimension,100,800,100 /scratch1/NCEPDEV/stmp4/Shastri.Paturi/forAndrew/gdas.20210701/00/icec/AMSR2-SEAICE-SH_v2r2_GW1_s202107010120180_e202107010259170_c202107010340150.nc icec_amsr2_south_2.nc\nCreated by GAASP Version 2.0, Release 2.0" ;
./utils/test/testdata/icec_amsr2_south_1.cdl:           :history = "Mon Sep 25 15:26:46 2023: ncks -d Number_of_X_Dimension,100,800,100 -d Number_of_Y_Dimension,100,800,100 /scratch1/NCEPDEV/stmp4/Shastri.Paturi/forAndrew/gdas.20210701/00/icec/AMSR2-SEAICE-SH_v2r2_GW1_s202106302341190_e202107010120160_c202107010202390.nc icec_amsr2_south.nc\nCreated by GAASP Version 2.0, Release 2.0" ;
./utils/test/testdata/sss_smos_1.cdl:           :history = "Mon Oct  2 16:01:51 2023: ncks -d n_grid_points,100,92700,1800 -v SSS_corr,Sigma_SSS_corr,Latitude,Longitude,Mean_acq_time,Dg_quality_SSS_corr /scratch1/NCEPDEV/stmp4/Shastri.Paturi/forAndrew/gdas.20210701/00/SSS/SM_OPER_MIR_OSUDP2_20210630T210913_20210630T220228_700_001_1.nc sss_smos_1_sub.nc" ;
./utils/test/testdata/sss_smos_2.cdl:           :history = "Mon Oct  2 16:02:44 2023: ncks -d n_grid_points,100,92700,1800 -v SSS_corr,Sigma_SSS_corr,Latitude,Longitude,Mean_acq_time,Dg_quality_SSS_corr /scratch1/NCEPDEV/stmp4/Shastri.Paturi/forAndrew/gdas.20210701/00/SSS/SM_OPER_MIR_OSUDP2_20210630T215911_20210630T225230_700_001_1.nc sss_smos_2_sub.nc" ;
./utils/test/testdata/icec_amsr2_north_2.cdl:           :history = "Mon Sep 25 15:41:32 2023: ncks -d Number_of_X_Dimension,400,600,100 -d Number_of_Y_Dimension,400,700,100 /scratch1/NCEPDEV/stmp4/Shastri.Paturi/forAndrew/gdas.20210701/00/icec/AMSR2-SEAICE-NH_v2r2_GW1_s202107010120180_e202107010259170_c202107010340150.nc icec_amsr2_north_2.nc\nCreated by GAASP Version 2.0, Release 2.0" ;
./utils/test/testdata/icec_amsr2_north_1.cdl:           :history = "Mon Sep 25 15:10:33 2023: ncks -d Number_of_X_Dimension,400,600,100 -d Number_of_Y_Dimension,400,700,100 /scratch1/NCEPDEV/stmp4/Shastri.Paturi/forAndrew/gdas.20210701/00/icec/AMSR2-SEAICE-NH_v2r2_GW1_s202106302341190_e202107010120160_c202107010202390.nc output.nc\nCreated by GAASP Version 2.0, Release 2.0" ;
./utils/test/testdata/sss_smap_2.cdl:           :history = "Mon Oct  2 18:36:57 2023: ncks -d phony_dim_0,20,70,25 -d phony_dim_1,30,800,40 -d phony_dim_2,2 -v lat,lon,smap_sss,smap_sss_uncertainty,quality_flag,row_time /scratch1/NCEPDEV/stmp4/Shastri.Paturi/forAndrew/gdas.20210701/00/SSS/SMAP_L2B_SSS_NRT_34258_A_20210630T231436.h5 sss_smap_2_sub.nc" ;
./utils/test/testdata/sss_smap_1.cdl:           :history = "Mon Oct  2 18:36:14 2023: ncks -d phony_dim_0,20,70,25 -d phony_dim_1,30,800,40 -d phony_dim_2,2 -v lat,lon,smap_sss,smap_sss_uncertainty,quality_flag,row_time /scratch1/NCEPDEV/stmp4/Shastri.Paturi/forAndrew/gdas.20210701/00/SSS/SMAP_L2B_SSS_NRT_34257_A_20210630T213609.h5 sss_smap_1_sub.nc" ;
./utils/test/testdata/ghrsst_sst_ma_202103241540.cdl:           :history = "Thu Mar 28 17:03:57 2024: ncks -d lon,11826,11835 -d lat,602,606 /scratch1/NCEPDEV/global/glopara/data/experimental_obs/gdas.20210324/18/ocean/sst/20210324154000-OSPO-L3U_GHRSST-SSTsubskin-AVHRRF_MA-ACSPO_V2.70-v02.0-fv01.0.nc ghrsst_sst_ma_202103241540.nc\nCreated by the L2P-to-L3U conversion tool,  which was developed and provided by NOAA/NESDIS/STAR and CCNY. The version is 4.1.5" ;
./utils/test/testdata/ghrsst_sst_ma_202103241550.cdl:           :history = "Thu Mar 28 17:06:08 2024: ncks -d lon,11826,11835 -d lat,602,606 /scratch1/NCEPDEV/global/glopara/data/experimental_obs/gdas.20210324/18/ocean/sst/20210324155000-OSPO-L3U_GHRSST-SSTsubskin-AVHRRF_MA-ACSPO_V2.70-v02.0-fv01.0.nc ghrsst_sst_ma_202103241550.nc\nCreated by the L2P-to-L3U conversion tool,  which was developed and provided by NOAA/NESDIS/STAR and CCNY. The version is 4.1.5" ;

Note: We may not need to update the path in the cdl files. What do you think?

@givelberg , @guillaumevernieres

./test/marine/CMakeLists.txt:   "/scratch1/NCEPDEV/da/common/ci/bufr"
./test/marine/CMakeLists.txt:           "/scratch2/NCEPDEV/ocean/Guillaume.Vernieres/data/static/common/RECCAP2_region_masks_all_v20221025.nc"

@kevindougherty-noaa , @CoryMartin-NOAA

./modulefiles/EVA/hera.lua:local pyenvpath = "/scratch1/NCEPDEV/da/python/envs/"

@apchoiCMD , @guillaumevernieres

./utils/test/CMakeLists.txt:            "/scratch1/NCEPDEV/da/common/ci/obs"

@andytangborn , @CoryMartin-NOAA

./test/aero/global-workflow/jjob_var_final.sh:    export UTILROOT="/scratch2/NCEPDEV/ensemble/save/Walter.Kolczynski/hpc-stack/intel-18.0.5.274/prod_util/1.2.2"
./test/aero/global-workflow/jjob_var_init.sh:    export UTILROOT="/scratch2/NCEPDEV/ensemble/save/Walter.Kolczynski/hpc-stack/intel-18.0.5.274/prod_util/1.2.2"
./test/aero/global-workflow/jjob_var_run.sh:    export UTILROOT="/scratch2/NCEPDEV/ensemble/save/Walter.Kolczynski/hpc-stack/intel-18.0.5.274/prod_util/1.2.2"

[CoryMartin-NOAA]

Thanks @RussTreadon-NOAA for the UTILROOT, I'm betting this can be removed now that we are all using spack-stack, but we need to test. For the ./prototypes directory, we should probably just delete that whole directory. @kevindougherty-noaa you have a working environment on Hera and Ursa, right?

[apchoiCMD]

Thanks @RussTreadon-NOAA for caring of these files and directories of utils.
The cdl file should remain as is and CMakeLists.txt will be needed to change to /scratch3/NCEPDEV/da/common/ci/obs

[RussTreadon-NOAA]

Thank you @CoryMartin-NOAA and @apchoiCMD for your replies.

The following change have been committed to bugfix/hera_scratch

• remove directory prototypes
• update aero jjob drivers.
• update path in utils/test/CMakeLists.txt has been updated

@CoryMartin-NOAA , in updating the aero jjob drivers, I noticed that these tests are turned off in test/aero/CMakeLists.txt

# tests that require the full build
if(BUILD_GDASBUNDLE)
    # Test exgdas scripts from the global-worflow
    #if (WORKFLOW_TESTS)
    #    add_subdirectory(global-workflow)
    #endif()
endif(BUILD_GDASBUNDLE)

Should we activate these tests or keep 'em off. If off, should we remove the tests from CMakeLists.txt and delete test/aero/global-workflow?

[CoryMartin-NOAA]

I vote keep them off for now, we should enable them again later, but I'm not sure if they will work out of the box and we shouldn't prioritize fixing them right now

[RussTreadon-NOAA]

@jiaruidong2017

./ush/ufsda/yamltools.py contains 'datapath': '/scratch1/NCEPDEV/global/glopara/fix/orog/20220805/C'+str(case)+'/',. The /scratch1 fileset is scheduled to be decommissioned 8/5/2025. We need to update paths.

I checked the commit history of this file. You added the /scratch1 path via GDASApp PR #145. This PR goes back to October 2022. Not surprising, directory /scratch1/NCEPDEV/global/glopara/fix/orog/20220805 no longer exists.

I see several YYYYMMDD directories in /scratch3/NCEPDEV/global/role.glopara/fix/orog. The most recent is /scratch3/NCEPDEV/global/role.glopara/fix/orog/20240917/. While this directory contain many case C* directories, the naming convention for the oro_data.tile*.nc files differs from filename_orog in yamltools.py.

What's the best way to resolve this? Seems we need to do more than just update the path.

[RussTreadon-NOAA]

@givelberg and @guillaumevernieres

File ./test/marine/CMakeLists.txt refers to /scratch2/NCEPDEV/ocean/Guillaume.Vernieres/data/static/common/RECCAP2_region_masks_all_v20221025.nc

This line was added in September 2024 via GDASApp PR #1275.

Do we have this file in a /scratch3 or /scratch4 location?

[CoryMartin-NOAA]

@RussTreadon-NOAA I'm 99% sure that
./ush/ufsda/yamltools.py has a lot of deprecated code in it, like that line you mentioned @jiaruidong2017 about, that hasn't been used since the switch to JCB I don't think