Skip to content

MPI failure with num_ranks = 3 #2

@gzagaris

Description

@gzagaris

The code fails when the number of ranks is 3.

15:34 $ mpiexec -np 3 ./fvsand.exe
Warning: PAMI CUDA HOOK disabled
Warning: PAMI CUDA HOOK disabled
Warning: PAMI CUDA HOOK disabled
rank 0, cnt 1
rank 0, dev 0, prop Tesla V100-SXM2-32GB, pci 4, 0, 4
rank 2, cnt 1
rank 2, dev 0, prop Tesla V100-SXM2-32GB, pci 6, 0, 4
rank 1, cnt 1
rank 1, dev 0, prop Tesla V100-SXM2-32GB, pci 5, 0, 4
nfaces=0
[tra015:2355786] *** An error occurred in MPI_Isend
[tra015:2355786] *** reported by process [823525377,2]
[tra015:2355786] *** on communicator MPI_COMM_WORLD
[tra015:2355786] *** MPI_ERR_RANK: invalid rank
[tra015:2355786] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[tra015:2355786] ***    and potentially your MPI job)
[tra015:2355760] PMIX ERROR: UNREACHABLE in file server/pmix_server.c at line 2193
[tra015:2355760] 1 more process has sent help message help-mpi-errors.txt / mpi_errors_are_fatal
[tra015:2355760] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

Notice nfaces=0 above. Seems that this may be a bug with the partitioner?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions