High metadata IOPS from Python MPI processes causing Lustre MDS overload

[nvasquez-plac]

Describe the bug

Bug Report: Python MPI startup causes Lustre Metadata Server overload

Problem

When running ensemble jobs, each MPI process independently loads the full Python environment on startup. This generates up to 30,000 metadata operations/sec per job. This was identified by Dmitri while monitoring SYMFLUENCE on ARC (U. of Calgary) during the calibration of SUMMA + mizuRoute using the ASYNC-DDS optimization algorithm (1 basin, 50 cores, 100 batches).

Profiled Execution Stages

Three distinct patterns were observed:

Stage	Metadata IOPS	Notes
mizuRoute.exe	~0 on shared NFS	Clean — no shared filesystem traffic (modified mizuRoute)
Python startup (launching summa_sundials.exe)	up to 30,000 ops/sec	Problematic — each MPI process searches for modules/libraries simultaneously
summa_sundials.exe	~1,000 ops/sec	Moderate — dynamic library loads, clears quickly

The Python stage is the critical bottleneck. Python installations contain many files, and finding/loading a library requires searching through many directories. Each MPI process does this independently, and they all start at approximately the same time.

Proposed Solutions

• Short-term (quick fix): Copy the Python venv to /tmp (local node storage) before job startup, as is already done for file I/O operations.
• Long-term (preferred): Persist MPI workers and start them in batches to avoid simultaneous environment loading across processes.

Additional Notes

• Only strictly needed Python modules should be loaded — modules loaded for convenience but never used contribute to the problem.
• This was diagnosed by Dimitri using filesystem profiling tools.

References

Internal email thread between Darri, Martyn, Nicolás, and Dimitri (March 2026). Suggestions made by Dmitri.

SYMFLUENCE version

No response

Steps to reproduce

On Arc (U. of Calgary):
cd /work/comphyd_lab/users/SYMFLUENCE
sbatch run_confluence_calib.sh

The configuration (yaml) file for the basin can be found here:
/work/comphyd_lab/users/SYMFLUENCE/0_config_files/century_basins_exp7_ASYNC-DDS/config_CAN_01BJ010_meso.yaml

Relevant logs or screenshots

[DarriEy]

Hi Nico,

PR #22 addresses this. It adds a persistent MPI worker pool that launches Python processes once and keeps them alive across all calibration batches, communicating via stdin/stdout pipes instead of the filesystem. This eliminates the repeated Python-import metadata storms entirely.

To test on ARC

cd /work/comphyd_lab/users/SYMFLUENCE
git fetch origin
git checkout fix/persistent-mpi-workers
pip install -e .   # if using editable install, otherwise skip
sbatch run_confluence_calib.sh

No config changes needed — execute_batch() will automatically use the persistent workers. You should see log lines like:

Starting persistent MPI workers (50 ranks) via mpirun
Persistent MPI workers started successfully

If anything goes wrong, it falls back to the old spawn-per-batch behavior automatically and logs a warning, so there's no risk of breaking the run.

It would be great if Dmitri could profile this run to confirm the IOPS reduction. Looking forward to the results!

Cheers,
Darri

[DarriEy]

Fix pushed! The worker script was using a Python 3.12+ logging feature and ARC has only Python 3.11. Can you pull and retry:
git pull origin fix/persistent-mpi-workers
Then resubmit the same job. The worker logs are now more verbose too, so if anything else comes up we'll see it.

[nvasquez-plac]

Hey Darri! Thanks for fixing it — the Python 3.11 issue is now resolved, and the job starts correctly :)

However, I’ve noticed a performance difference with the new persistent MPI workers that I wanted to mention.

Before the fix:

• Each batch of 50 tasks finished in about 190 seconds
• summa=55s, mizuroute=100s, metrics=0.6s per run
• A full 100-batch run completed successfully in roughly 5.5 hours
• Each rank set up its own scratch directory in /tmp before running

After this fix:

• The initial pool evaluation (1 task) took 5 minutes 44 seconds instead of around 2 minutes
• Once persistent workers started, the job got completely stuck — no batch completions at all :/
• mizuRoute processes each running 7-8 minutes with no output
• More than 50 processes in D state (I/O wait)

Some ideas:
In the old logs, each rank explicitly creates its own scratch directory in /tmp before starting. In the new logs, that step appears to be missing — so the persistent workers might be reading and writing directly to /work for every SUMMA and mizuRoute run. Could that explain the performance difference before and after the fix?

I monitored the job for about 10-12 min, but the first batch never finished. Attached is the log file in case you might look at it. You can also find the file here: /work/comphyd_lab/users/SYMFLUENCE/logs/SYMFLUENCE_calibration_40607760.log

SYMFLUENCE_calibration_40607760.log

[DarriEy]

That’s peculiar! It should be routing this to the correct /tmp path. I think the most likely reason that could cause this is that the stdin/stdout pipe relay through mpirun is deadlocking. The coordinator's blocking write could stall if the task payload exceeds the OS pipe buffer (~64KB), and mpirun's stdin relay to rank 0 adds another layer of unreliability. This could be why the first batch never completed.

I've rewritten the communication layer to use file-based signaling through $SLURM_TMPDIR to confirm if this is the case. The coordinator writes tasks.pkl file + a tasks.ready signal, the broker polls for the signal, distributes via MPI, and writes back results.pkl + results.ready. 

 Can you pull and retry:
 git pull origin fix/persistent-mpi-workers

The task/result files now go to /tmp/symfluence_mpi_comm_*/, which is the same local storage you were already using.

[nvasquez-plac]

Hey Darri! I just resubmitted the job, and the problem persists. From monitoring the job, it seems the slowdown is specific to the routing phase. SUMMA runs normally (I checked the cores using "htop" and also the worker's log), but when the persistent MPI workers reach mizuRoute, 49 concurrent mizuRoute.exe processes only sustain ~65–79% CPU each while a Python coordinator remains near 100% CPU, with negligible I/O wait.

From the workers' log files, it seems that mizuRoute never starts during the MPI parallelization phase. Attached are two examples, one before and another after the fix (after letting the job run for about 15 minutes). I also attached the job log file. Is there anything particular around mizuRoute that might be causing this issue?

You can find the log file here: /work/comphyd_lab/users/SYMFLUENCE/logs/SYMFLUENCE_calibration_40609368.log

The SUMMA log files here (example for one core): /work/comphyd_lab/users/SYMFLUENCE_data/century_basins_exp7_ASYNC-DDS/domain_CAN_01BJ010_meso/simulations/run_async-dds/process_10/simulations/run_1/SUMMA/logs

The mizuRoute log files here (example for one core):
/work/comphyd_lab/users/SYMFLUENCE_data/century_basins_exp7_ASYNC-DDS/domain_CAN_01BJ010_meso/simulations/run_async-dds/process_10/simulations/run_1/mizuRoute/logs

mizuroute_worker_1061987_iter00000_ind010.log
mizuroute_worker_3137599_iter00000_ind010.log
SYMFLUENCE_calibration_40609368.log

[DarriEy]

Hi Nico, thanks for the detailed logs — they were very helpful!

Looking at the "after" mizuRoute log: only the 4-line header is written (by Python before launching mizuRoute.exe), but mizuRoute itself never produced output. Combined with your observation that 49 mizuRoute processes were at 65-79% CPU while a Python process was at 100%, I found the root cause:

Rank 0 was deadlocking the MPI gather. Rank 0 was both running its own tasks AND collecting results from other workers. While rank 0 was busy running its own SUMMA+mizuRoute (~155s), all 49 other workers that finished earlier were blocking on comm.send(results) waiting for rank 0 to post the matching comm.recv. This created a cascading stall.

Fix pushed: Rank 0 is now a dedicated broker — it only distributes tasks and gathers results, never runs tasks itself. We launch N+1 MPI ranks (51 for your 50-core setup), so rank 0 is free to immediately receive results as workers complete. I also added per-task diagnostic logging so we can see the exact paths and progress.

git pull origin fix/persistent-mpi-workers

The worker logs should now show lines like:

[Rank 1] Starting task 0 (ind=0, proc=0, summa_dir=/path/to/...)
[Rank 1] Completed task 0 (ind=0, score=-0.089)

This will confirm whether paths are hitting /tmp or /work, and let us track exactly where things get stuck if anything else comes up.

Cheers,
Darri

[nvasquez-plac]

Hi Darri! Thanks for reviewing the bug!

MizuRoute still gets stuck, but now I've run into another problem. When SYMFLUENCE tries to set up the 50 cores, I get this message:

There are not enough slots available in the system to satisfy the 51 (see log file attached).

Also, it seems SUMMA and mizuRoute are running in the /work folder instead of the /tmp folder. I checked the manager files for processor 1, as well as the output files:

/work/comphyd_lab/users/SYMFLUENCE_data/century_basins_exp7_ASYNC-DDS/domain_CAN_01BJ010_meso/simulations/run_async-dds/process_1/settings

Please let me know if you would like me to check anything else.

Cheers,
Nico

SYMFLUENCE_calibration_40610421.log