Pardiso for clusters (cpardiso)

[ma-sadeghi]

Hi,
Have you looked into supporting Pardiso for clusters (so it can be run on machines with distributed memory)?
Thanks!

[haasad]

Hey, no, I haven't looked into that. I used to have access to a server with 256GB memory, which was always enough for my usage. Is this something you plan to use?

[ma-sadeghi]

I'd love to use it. The issue is that at some point (depends on the memory bandwidth of the CPU), memory operations become the bottleneck. In my experience, this critical point tends to be around 10ish cores (again, it depends on the hardware, but I tested on Compute Canada resources, which are fairly modern). So, a CPU with 32 or 64 cores, for instance, give you almost the same speedup as one with 8 cores.

With clusters, on the other hand, each node gets its own memory bandwidth, and so much higher speed ups become possible. I'd love to contribute, but I have no experience with interfacing Python with C/Fortran code. MKL seems to already have a cluster version of pypardiso (see here).

If you could give me some pointers on how to start, I can take a crack at it, otherwise, I'd really appreciate it if it's something that interests you.

[kurtenkera]

Hi @haasad, I am trying to run PyPARDISO on a single node of a HPC which has 64 cores and 256GB memory. When I run PyPARDISO on my PC with 32 cores and 60GB memory, I can execute pypardiso.spsolve() in roughly 3 minutes for my problem, where my matrix A is roughly a ~1Mill x 1Mill square matrix. However, for some reason I get a significant speed reduction on the HPC despite the increase in cores and memory. I monitored my job performance, and it appears the HPC was only using 1 core out of the allocated 64 cores, so this explains the speed reduction.

My question: Is this just a feature of pypardiso? That it can only take advantage of multiple cores on a PC, but not a HPC? Or do I need to change my Slurm commands (see my Slurm commands below for running PyPARDISO on the HPC):

#!/bin/bash

#SBATCH --job-name=SDTO_box
#SBATCH --time=00:20:00
#SBATCH --mem=256G
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=64
#SBATCH --account=OD-229549
#SBATCH --output=SDTO_box.txt

python simulations/sg_3DTO/SDTO.py

[urob]

I monitored my job performance, and it appears the HPC was only using 1 core out of the allocated 64 cores

Try exporting OMP_NUM_THREADS. If you only want MKL to be multithreaded for some reason, you could also export
MKL_NUM_THREADS.

[kurtenkera]

Hi @urob,

Do you mean doing something like:

#!/bin/bash

#SBATCH --job-name=SDTO_box
#SBATCH --time=00:20:00
#SBATCH --mem=256G
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=64
#SBATCH --account=OD-229549
#SBATCH --output=SDTO_box.txt

export OMP_NUM_THREADS=64
python simulations/sg_3DTO/SDTO.py