Rerun subset of periodic surging experiments that did not make it 9ka

[andrewdnolan]

About a third of the periodic surging experiment did not make it the end of the 9ka simulation. The purpose of this PR is to rerun those simulations that failed, with a bit more attention and babysitting this time around.

[andrewdnolan]

Identifying incomplete runs

Assuming you have a PeriodicSurges_Timeseries_9ka.zarrr file as generated by expr/03_PeriodicSurge/run/crmpt12/postprocess_timeseries.py, we can identify the runs that did not make it past 8ka with:

import numpy as np
import xarray as xr

zarr_path = 'data/PeriodicSurges_Timeseries_9ka.zarr'

src = xr.open_dataset(zarr_path, engine='zarr')

nt = src.dims["t"]

betas = []
periods = []
for SP in src.SP:
    for beta in src.beta:
        da = src.sel(SP=SP, beta=beta).percent_temperate

        # number of nans
        nn = np.count_nonzero(da.isnull())
        # final valid timestep
        idx, = np.where(da.isnull())
        t_f = da.t.isel(t=idx[0]-1)

        if (nn > 1) & (t_f < 8e3):
            betas.append(float(beta))
            periods.append(float(SP))

np.savetxt("incomplete_runs.dat",  np.array([periods, betas]).T)

The incomplete_runs.dat file is then used by the expr/03_PeriodicSurge/make_experiment.py script in order to generate the run commands and job submission script for the simulations that need to be rerun.

[andrewdnolan]

After the first 3ka chunk eight simulations failed, due to a divergent velocity, and need to removed from the suite of simulations to rerun. In order to identify the simulations that complete run:

touch run/crmpt12/crmpt12.commands.p2

for i in $(seq 34); do  
     if tail "logs/crmpt12/crmpt12_46759282_${i}.out" | grep -q "ELMER SOLVER FINISHED AT:"; then
          sed -n "${i}p" ./run/crmpt12/crmpt12.commands >> run/crmpt12/crmpt12.commands.p2
     fi
done