Bring in climo file hanging during creation fix

[justin-richling]

The current code can lend itself to climo files hanging during creation. The main improvement here is that the new implementation isolates worker processes much more cleanly and prevents nested parallelism / shared-memory issues that can cause hangs or deadlocks.

Major Changes:

---

1. Switched from default multiprocessing context to 'spawn' which creates completely fresh Python interpreters for each worker instead of cloning the parent memory state.

That avoids inherited corrupted state and is much safer for scientific I/O workflows.

2. Reduced what gets passed into workers

The original code passed the entire ADF object into every worker process.

That can be problematic because:

• large Python objects must be pickled
• custom objects sometimes fail serialization
• the object may contain:
  • open file handles
  • locks
  • loggers
  • multiprocessing-unsafe state
  • dask objects

This can create hangs during worker initialization.

The new version passes only a simple string for adf_user

3. Imports moved inside worker function

With multiprocessing, especially 'spawn', importing libraries inside workers can help avoid:

• inherited thread state
• inherited dask schedulers
• inherited HDF5 state

4. Explicitly disabling dask multithreading

This created a nested parallelism which can lead to:

• oversubscription
• thread contention
• filesystem stalls
• hangs

Now each worker process computes serially internally.

5. Switched fully to open_mfdataset(..., chunks=...)

The new version forces dask-backed lazy arrays.

6. Proper dataset cleanup using context manager

This guarantees:

• file handles close correctly
• NetCDF locks are released
• HDF5 resources are freed

Without explicit closure, workers can accumulate open file descriptors and eventually hang.

7. Added explicit garbage collection

Scientific Python stacks sometimes retain:

• large arrays
• dask graphs
• HDF5 references

Explicit garbage collection helps workers release memory sooner between tasks.

8. Added exception handling inside workers

Previously a worker crash could:

• silently terminate
• freeze starmap
• leave pool state inconsistent

Now failures are caught and reported cleanly.

@brianpm I tried to tackle the core concepts of the changes and where they are helping fix the code, please let me know if I'm off or something I messed!

EDIT: These changes were supplied by @brianpm!

[brianpm]

This looks like a clean implementation of what I was doing!

[justin-richling]

@brianpm These were your changes! I forgot to tag you for credit.