CMOR files ending up in wrong directory

[jhkennedy]

When I've been running my add-ilamb tests, I noticed a supremely weird behavior with the CMOR job.

I am running on acme1 with:

• Model ouput: /p/user_pub/e3sm/kennedy52/20181015_BCRC_1850SPINUP_OIBGC.ne30_oECv3.edison.0105--0106
• processflow project_path = /p/user_pub/e3sm/kennedy52/test_processflow
• using /export/kennedy52/E3SM/processflow/ADDILAMB.cfg for processflow
• full run command: ((ii++)); processflow -c ADDILAMB.cfg -l add_ilamb_${ii}.log --debug > >(tee -a add_ilamb_${ii}.out) 2> >(tee -a add_ilamb_${ii}.err >&2)

I have it setup to run two cmor jobs for 010501--010512 and 010601--010612, which produce CMORized output files:

output/pp/cmor/spinup/cmor/
├── 0105_0105
│   ├── CMIP6
│   │   └── CMIP
│   │       └── E3SM-Project
│   │           └── E3SM-1-0
│   │               └── piControl
│   │                   └── r1i1p1f1
│   │                       ├── Amon
│   │                       │   ├── pr
│   │                       │   │   └── gr
│   │                       │   │       └── v20190312
│   │                       │   │           └── pr_Amon_E3SM-1-0_piControl_r1i1p1f1_gr_010501-010512.nc
│   │                       │   ├── prc
│   │                       │   │   └── gr
│   │                       │   │       └── v20190312
│   │                       │   │           └── prc_Amon_E3SM-1-0_piControl_r1i1p1f1_gr_010601-010612.nc
│   │                       │   ├── rlds
│   │                       │   │   └── gr
│   │                       │   │       └── v20190312
│   │                       │   │           └── rlds_Amon_E3SM-1-0_piControl_r1i1p1f1_gr_010601-010612.nc
│   │                       │   ├── rlus
│   │                       │   │   └── gr
│   │                       │   │       └── v20190312
│   │                       │   │           └── rlus_Amon_E3SM-1-0_piControl_r1i1p1f1_gr_010601-010612.nc
│   │                       │   ├── rsds
│   │                       │   │   └── gr
│   │                       │   │       └── v20190312
│   │                       │   │           └── rsds_Amon_E3SM-1-0_piControl_r1i1p1f1_gr_010501-010512.nc
│   │                       │   ├── rsus
│   │                       │   │   └── gr
│   │                       │   │       └── v20190312
│   │                       │   │           └── rsus_Amon_E3SM-1-0_piControl_r1i1p1f1_gr_010501-010512.nc
│   │                       │   └── tas
│   │                       │       └── gr
│   │                       │           └── v20190312
│   │                       │               └── tas_Amon_E3SM-1-0_piControl_r1i1p1f1_gr_010601-010612.nc
│   │                       └── Lmon
│   │                           ├── lai
│   │                           │   └── gr
│   │                           │       └── v20190312
│   │                           │           └── lai_Lmon_E3SM-1-0_piControl_r1i1p1f1_gr_010601-010612.nc
│   │                           └── tsl
│   │                               └── gr
│   │                                   └── v20190312
│   │                                       └── tsl_Lmon_E3SM-1-0_piControl_r1i1p1f1_gr_010501-010512.nc
│   ├── converter.log
│   └── user_input.json
└── 0106_0106
    ├── CMIP6
    │   └── CMIP
    │       └── E3SM-Project
    │           └── E3SM-1-0
    │               └── piControl
    │                   └── r1i1p1f1
    │                       ├── Amon
    │                       │   ├── pr
    │                       │   │   └── gr
    │                       │   │       └── v20190312
    │                       │   │           └── pr_Amon_E3SM-1-0_piControl_r1i1p1f1_gr_010501-010512.nc
    │                       │   ├── prc
    │                       │   │   └── gr
    │                       │   │       └── v20190312
    │                       │   │           └── prc_Amon_E3SM-1-0_piControl_r1i1p1f1_gr_010601-010612.nc
    │                       │   ├── rlds
    │                       │   │   └── gr
    │                       │   │       └── v20190312
    │                       │   │           └── rlds_Amon_E3SM-1-0_piControl_r1i1p1f1_gr_010601-010612.nc
    │                       │   ├── rlus
    │                       │   │   └── gr
    │                       │   │       └── v20190312
    │                       │   │           └── rlus_Amon_E3SM-1-0_piControl_r1i1p1f1_gr_010601-010612.nc
    │                       │   ├── rsds
    │                       │   │   └── gr
    │                       │   │       └── v20190312
    │                       │   │           └── rsds_Amon_E3SM-1-0_piControl_r1i1p1f1_gr_010501-010512.nc
    │                       │   ├── rsus
    │                       │   │   └── gr
    │                       │   │       └── v20190312
    │                       │   │           └── rsus_Amon_E3SM-1-0_piControl_r1i1p1f1_gr_010501-010512.nc
    │                       │   └── tas
    │                       │       └── gr
    │                       │           └── v20190312
    │                       │               └── tas_Amon_E3SM-1-0_piControl_r1i1p1f1_gr_010601-010612.nc
    │                       └── Lmon
    │                           ├── lai
    │                           │   └── gr
    │                           │       └── v20190312
    │                           │           └── lai_Lmon_E3SM-1-0_piControl_r1i1p1f1_gr_010601-010612.nc
    │                           └── tsl
    │                               └── gr
    │                                   └── v20190312
    │                                       └── tsl_Lmon_E3SM-1-0_piControl_r1i1p1f1_gr_010501-010512.nc
    ├── converter.log
    └── user_input.json

Notice above:

• in the 0105_0105 directory, lai, prc, rlds, tas, rlus are all _010601_010612.nc files!
• in the 0106_0106 directory, tsl, rsds, pr, rsus are all _010501_010512.nc files!
  but, in both cases, the rest of the variables are correct (meaning it looks like these two sets of variables swapped places).

So somehow the files are being placed in the wrong directory but I don't see anything in the jobs/cmor.py that would/should/could do this.

[jhkennedy]

Note: this happens every time for me, and the two sets of variables are always the same, so I don't think it's transient behavior.

[sterlingbaldwin]

This is uhhh, non-optimal. I'll look into the CMOR job and see if I can find whats causing the issue.