Hello,
I just install nvhpc 23.11 on a RH 8.6 system following the provided instructions.
A first test code compilation went fine but the run with slurm (v22.09) of a 2 mpi ranks job fails with this error :
opal_init:startup:internal-failure
any gotchas ?
Kind regards,
Lucian Anton.
Hi Lucian Anton,
I’ve not encountered this myself but in doing a web search, it seems that this error can occur when the OpenMPI runtime libraries fail to load. Since it only fails with Slurm, my best guess is that the Slurm environment variable LD_LIBRARY_PATH isn’t configured to include the path to these libraries.
Because we need to build OpenMPI4 and HPCX using specific CUDA versions, the MPI tools under the “hpcx” and “openmpi4” are actually trampolines to the specific install.
While convenient for the user, the LD_LIBARY_PATH needs to point to the specific install based on the CUDA version in use.
For example, if you’re using HPCX with CUDA 12.3 on an x86_64 system, you’d set LD_LIBRARY_PATH to:
[base_dir]/Linux_x86_64/23.11/comm_libs/12.3/hpcx/hpcx-2.16/ompi/lib/
Note that with 23.11 we ship installs for CUDA 11.0, 11.8, and 12.3, but these can be used with other CUDA versions as well, just use the one that closest to your CUDA install.
OpenMPI 3.1.5, doesn’t have a trampoline, so you’d set this to the base path.
If this doesn’t help, I’ll ask the folks here who manage our MPI builds for ideas.
-Mat
Hi Mat,
I have added OpenMPI4 path the LD_LIBRARY_PATH and I get a new error
“PML ob1 cannot be selected”
However if I use the module nvhpc-openmpi3 and I add the corresponding openmpi3 lib path to LD_LIB… it works but with several warning which I put below
libibverbs: Warning: couldn't open config directory '/usr/local/etc/libibverbs.d'.
libibverbs: Warning: couldn't open config directory '/usr/local/etc/libibverbs.d'.
--------------------------------------------------------------------------
[[3802,32579],1]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:
....
Almost there but not quite :)
Lucian
What happens when you run ompi_info in one of the CUDA-versioned MPI directories with LD_LIBRARY_PATH set?
It looks as follow. I see that the paths for the ucx and cuda are hardcoded. Can they be overridden?
nvhpc_2023_2311_Linux_x86_64_cuda_multi/Linux_x86_64/23.11/comm_libs/12.3/openmpi4/openmpi-4.1.5/bin/ompi_info
Package: Open MPI qa@sky2 Distribution
Open MPI: 4.1.5
Open MPI repo revision: v4.1.5
Open MPI release date: Feb 23, 2023
Open RTE: 4.1.5
Open RTE repo revision: v4.1.5
Open RTE release date: Feb 23, 2023
OPAL: 4.1.5
OPAL repo revision: v4.1.5
OPAL release date: Feb 23, 2023
MPI API: 3.1.0
Ident string: 4.1.5
Prefix: /lus/h2resw01/scratch/syla/nvhpc_2023_2311_Linux_x86_64_cuda_multi/Linux_x86_64/23.11/comm_libs/12.3/openmpi4/openmpi-4.1.5
Configured architecture: x86_64-pc-linux-gnu
Configure host: sky2
Configured by: qa
Configured on: Sat Oct 14 04:04:36 UTC 2023
Configure host: sky2
Configure command line: '--enable-shared' '--enable-static' '--without-tm'
'--disable-mpi-cxx' '--disable-wrapper-runpath'
'--without-libnl' '--without-verbs'
'--without-xpmem' '--without-cray-xpmem'
'--enable-mpirun-prefix-by-default'
'--with-libevent=internal' '--with-slurm'
'--with-ucx=/proj/nv/libraries/Linux_x86_64/23.11/openmpi4-12/238661-rel-4/comm_libs/12.2/openmpi4/openmpi-4.1.5'
'--with-cuda=/proj/cuda/12.2/Linux_x86_64'
'--enable-mpi1-compatibility'
'--enable-mca-no-build=btl-uct'
'--prefix=/proj/nv/libraries/Linux_x86_64/23.11/openmpi4-12/238661-rel-4/comm_libs/12.2/openmpi4/openmpi-4.1.5'
'--with-wrapper-ldflags=-Wl,-rpath,$ORIGIN:$ORIGIN/../../lib:$ORIGIN/../../../../../compilers/lib:$ORIGIN/../../../../../../compilers/lib'
Built by: qa
Built on: Sat Oct 14 04:17:24 UTC 2023
Built host: sky2
C bindings: yes
C++ bindings: no
Fort mpif.h: yes (all)
Fort use mpi: yes (full: ignore TKR)
Fort use mpi size: deprecated-ompi-info-value
Fort use mpi_f08: yes
Fort mpi_f08 compliance: The mpi_f08 module is available, but due to
limitations in the
/proj/nv/Linux_x86_64/238661-rel/compilers/bin/nvfortran
compiler and/or Open MPI, does not support the
following: array subsections, direct passthru
(where possible) to underlying Open MPI's C
functionality
Fort mpi_f08 subarrays: no
Java bindings: no
Wrapper compiler rpath: rpath
C compiler: /proj/nv/Linux_x86_64/238661-rel/compilers/bin/nvc
C compiler absolute:
C compiler family name: PGI
C compiler version: 23.11-0
C++ compiler: /proj/nv/Linux_x86_64/238661-rel/compilers/bin/nvc++
C++ compiler absolute: none
Fort compiler: /proj/nv/Linux_x86_64/238661-rel/compilers/bin/nvfortran
Fort compiler abs:
Fort ignore TKR: yes (!DIR$ IGNORE_TKR)
Fort 08 assumed shape: yes
Fort optional args: yes
Fort INTERFACE: yes
Fort ISO_FORTRAN_ENV: yes
Fort STORAGE_SIZE: yes
Fort BIND(C) (all): yes
Fort ISO_C_BINDING: yes
Fort SUBROUTINE BIND(C): yes
Fort TYPE,BIND(C): yes
Fort T,BIND(C,name="a"): yes
Fort PRIVATE: yes
Fort PROTECTED: yes
Fort ABSTRACT: yes
Fort ASYNCHRONOUS: yes
Fort PROCEDURE: yes
Fort USE...ONLY: yes
Fort C_FUNLOC: yes
Fort f08 using wrappers: yes
Fort MPI_SIZEOF: yes
C profiling: yes
C++ profiling: no
Fort mpif.h profiling: yes
Fort use mpi profiling: yes
Fort use mpi_f08 prof: yes
C++ exceptions: no
Thread support: posix (MPI_THREAD_MULTIPLE: yes, OPAL support: yes,
OMPI progress: no, ORTE progress: yes, Event lib:
yes)
Sparse Groups: no
Internal debug support: no
MPI interface warnings: yes
MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
dl support: yes
Heterogeneous support: no
mpirun default --prefix: yes
MPI_WTIME support: native
Symbol vis. support: yes
Host topology support: yes
IPv6 support: no
MPI1 compatibility: yes
MPI extensions: affinity, cuda, pcollreq
FT Checkpoint support: no (checkpoint thread: no)
C/R Enabled Debugging: no
MPI_MAX_PROCESSOR_NAME: 256
MPI_MAX_ERROR_STRING: 256
MPI_MAX_OBJECT_NAME: 64
MPI_MAX_INFO_KEY: 36
MPI_MAX_INFO_VAL: 256
MPI_MAX_PORT_NAME: 1024
MPI_MAX_DATAREP_STRING: 128
MCA allocator: basic (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA allocator: bucket (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA btl: self (MCA v2.1.0, API v3.1.0, Component v4.1.5)
MCA btl: smcuda (MCA v2.1.0, API v3.1.0, Component v4.1.5)
MCA btl: tcp (MCA v2.1.0, API v3.1.0, Component v4.1.5)
MCA btl: vader (MCA v2.1.0, API v3.1.0, Component v4.1.5)
MCA compress: bzip (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA compress: gzip (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA crs: none (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA dl: dlopen (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA event: libevent2022 (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA hwloc: hwloc201 (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA if: linux_ipv6 (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA if: posix_ipv4 (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA installdirs: env (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA installdirs: config (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA memory: patcher (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA mpool: hugepage (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA patcher: overwrite (MCA v2.1.0, API v1.0.0, Component
v4.1.5)
MCA pmix: isolated (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA pmix: pmix3x (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA pstat: linux (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA rcache: grdma (MCA v2.1.0, API v3.3.0, Component v4.1.5)
MCA rcache: gpusm (MCA v2.1.0, API v3.3.0, Component v4.1.5)
MCA rcache: rgpusm (MCA v2.1.0, API v3.3.0, Component v4.1.5)
MCA reachable: weighted (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA shmem: mmap (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA shmem: posix (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA shmem: sysv (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA timer: linux (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA errmgr: default_app (MCA v2.1.0, API v3.0.0, Component
v4.1.5)
MCA errmgr: default_hnp (MCA v2.1.0, API v3.0.0, Component
v4.1.5)
MCA errmgr: default_orted (MCA v2.1.0, API v3.0.0, Component
v4.1.5)
MCA errmgr: default_tool (MCA v2.1.0, API v3.0.0, Component
v4.1.5)
MCA ess: env (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA ess: hnp (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA ess: pmi (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA ess: singleton (MCA v2.1.0, API v3.0.0, Component
v4.1.5)
MCA ess: tool (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA ess: slurm (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA filem: raw (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA grpcomm: direct (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA iof: hnp (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA iof: orted (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA iof: tool (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA odls: default (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA odls: pspawn (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA oob: tcp (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA plm: isolated (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA plm: rsh (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA plm: slurm (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA ras: simulator (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA ras: slurm (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA regx: fwd (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA regx: naive (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA regx: reverse (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA rmaps: mindist (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA rmaps: ppr (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA rmaps: rank_file (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA rmaps: resilient (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA rmaps: round_robin (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA rmaps: seq (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA rml: oob (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA routed: binomial (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA routed: direct (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA routed: radix (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA rtc: hwloc (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA schizo: flux (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA schizo: ompi (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA schizo: orte (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA schizo: jsm (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA schizo: slurm (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA state: app (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA state: hnp (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA state: novm (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA state: orted (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA state: tool (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA bml: r2 (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA coll: adapt (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA coll: basic (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA coll: han (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA coll: inter (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA coll: libnbc (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA coll: self (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA coll: sm (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA coll: sync (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA coll: tuned (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA coll: cuda (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA fcoll: dynamic (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA fcoll: dynamic_gen2 (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA fcoll: individual (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA fcoll: two_phase (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA fcoll: vulcan (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA fs: ufs (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA io: ompio (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA io: romio321 (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA op: avx (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA osc: pt2pt (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA pml: cm (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA pml: ob1 (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA pml: ucx (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA pml: v (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA rte: orte (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA sharedfp: individual (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA sharedfp: lockedfile (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA sharedfp: sm (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA topo: basic (MCA v2.1.0, API v2.2.0, Component v4.1.5)
MCA topo: treematch (MCA v2.1.0, API v2.2.0, Component
v4.1.5)
MCA vprotocol: pessimist (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
Hi,
Have you received my output from ompi_info? I don’t see in the forum. It might have hit some size limit.
Looks like the post got flagged as spam. Not sure why. Though here’s what you posted:
It looks as follow. I see that the paths for the ucx and cuda are hardcoded. Can they be overridden?
nvhpc_2023_2311_Linux_x86_64_cuda_multi/Linux_x86_64/23.11/comm_libs/12.3/openmpi4/openmpi-4.1.5/bin/ompi_info
Package: Open MPI qa@sky2 Distribution
Open MPI: 4.1.5
Open MPI repo revision: v4.1.5
Open MPI release date: Feb 23, 2023
Open RTE: 4.1.5
Open RTE repo revision: v4.1.5
Open RTE release date: Feb 23, 2023
OPAL: 4.1.5
OPAL repo revision: v4.1.5
OPAL release date: Feb 23, 2023
MPI API: 3.1.0
Ident string: 4.1.5
Prefix: /lus/h2resw01/scratch/syla/nvhpc_2023_2311_Linux_x86_64_cuda_multi/Linux_x86_64/23.11/comm_libs/12.3/openmpi4/openmpi-4.1.5
Configured architecture: x86_64-pc-linux-gnu
Configure host: sky2
Configured by: qa
Configured on: Sat Oct 14 04:04:36 UTC 2023
Configure host: sky2
Configure command line: '--enable-shared' '--enable-static' '--without-tm'
'--disable-mpi-cxx' '--disable-wrapper-runpath'
'--without-libnl' '--without-verbs'
'--without-xpmem' '--without-cray-xpmem'
'--enable-mpirun-prefix-by-default'
'--with-libevent=internal' '--with-slurm'
'--with-ucx=/proj/nv/libraries/Linux_x86_64/23.11/openmpi4-12/238661-rel-4/comm_libs/12.2/openmpi4/openmpi-4.1.5'
'--with-cuda=/proj/cuda/12.2/Linux_x86_64'
'--enable-mpi1-compatibility'
'--enable-mca-no-build=btl-uct'
'--prefix=/proj/nv/libraries/Linux_x86_64/23.11/openmpi4-12/238661-rel-4/comm_libs/12.2/openmpi4/openmpi-4.1.5'
'--with-wrapper-ldflags=-Wl,-rpath,$ORIGIN:$ORIGIN/../../lib:$ORIGIN/../../../../../compilers/lib:$ORIGIN/../../../../../../compilers/lib'
Built by: qa
Built on: Sat Oct 14 04:17:24 UTC 2023
Built host: sky2
C bindings: yes
C++ bindings: no
Fort mpif.h: yes (all)
Fort use mpi: yes (full: ignore TKR)
Fort use mpi size: deprecated-ompi-info-value
Fort use mpi_f08: yes
Fort mpi_f08 compliance: The mpi_f08 module is available, but due to
limitations in the
/proj/nv/Linux_x86_64/238661-rel/compilers/bin/nvfortran
compiler and/or Open MPI, does not support the
following: array subsections, direct passthru
(where possible) to underlying Open MPI's C
functionality
Fort mpi_f08 subarrays: no
Java bindings: no
Wrapper compiler rpath: rpath
C compiler: /proj/nv/Linux_x86_64/238661-rel/compilers/bin/nvc
C compiler absolute:
C compiler family name: PGI
C compiler version: 23.11-0
C++ compiler: /proj/nv/Linux_x86_64/238661-rel/compilers/bin/nvc++
C++ compiler absolute: none
Fort compiler: /proj/nv/Linux_x86_64/238661-rel/compilers/bin/nvfortran
Fort compiler abs:
Fort ignore TKR: yes (!DIR$ IGNORE_TKR)
Fort 08 assumed shape: yes
Fort optional args: yes
Fort INTERFACE: yes
Fort ISO_FORTRAN_ENV: yes
Fort STORAGE_SIZE: yes
Fort BIND(C) (all): yes
Fort ISO_C_BINDING: yes
Fort SUBROUTINE BIND(C): yes
Fort TYPE,BIND(C): yes
Fort T,BIND(C,name="a"): yes
Fort PRIVATE: yes
Fort PROTECTED: yes
Fort ABSTRACT: yes
Fort ASYNCHRONOUS: yes
Fort PROCEDURE: yes
Fort USE...ONLY: yes
Fort C_FUNLOC: yes
Fort f08 using wrappers: yes
Fort MPI_SIZEOF: yes
C profiling: yes
C++ profiling: no
Fort mpif.h profiling: yes
Fort use mpi profiling: yes
Fort use mpi_f08 prof: yes
C++ exceptions: no
Thread support: posix (MPI_THREAD_MULTIPLE: yes, OPAL support: yes,
OMPI progress: no, ORTE progress: yes, Event lib:
yes)
Sparse Groups: no
Internal debug support: no
MPI interface warnings: yes
MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
dl support: yes
Heterogeneous support: no
mpirun default --prefix: yes
MPI_WTIME support: native
Symbol vis. support: yes
Host topology support: yes
IPv6 support: no
MPI1 compatibility: yes
MPI extensions: affinity, cuda, pcollreq
FT Checkpoint support: no (checkpoint thread: no)
C/R Enabled Debugging: no
MPI_MAX_PROCESSOR_NAME: 256
MPI_MAX_ERROR_STRING: 256
MPI_MAX_OBJECT_NAME: 64
MPI_MAX_INFO_KEY: 36
MPI_MAX_INFO_VAL: 256
MPI_MAX_PORT_NAME: 1024
MPI_MAX_DATAREP_STRING: 128
MCA allocator: basic (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA allocator: bucket (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA btl: self (MCA v2.1.0, API v3.1.0, Component v4.1.5)
MCA btl: smcuda (MCA v2.1.0, API v3.1.0, Component v4.1.5)
MCA btl: tcp (MCA v2.1.0, API v3.1.0, Component v4.1.5)
MCA btl: vader (MCA v2.1.0, API v3.1.0, Component v4.1.5)
MCA compress: bzip (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA compress: gzip (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA crs: none (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA dl: dlopen (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA event: libevent2022 (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA hwloc: hwloc201 (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA if: linux_ipv6 (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA if: posix_ipv4 (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA installdirs: env (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA installdirs: config (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA memory: patcher (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA mpool: hugepage (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA patcher: overwrite (MCA v2.1.0, API v1.0.0, Component
v4.1.5)
MCA pmix: isolated (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA pmix: pmix3x (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA pstat: linux (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA rcache: grdma (MCA v2.1.0, API v3.3.0, Component v4.1.5)
MCA rcache: gpusm (MCA v2.1.0, API v3.3.0, Component v4.1.5)
MCA rcache: rgpusm (MCA v2.1.0, API v3.3.0, Component v4.1.5)
MCA reachable: weighted (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA shmem: mmap (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA shmem: posix (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA shmem: sysv (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA timer: linux (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA errmgr: default_app (MCA v2.1.0, API v3.0.0, Component
v4.1.5)
MCA errmgr: default_hnp (MCA v2.1.0, API v3.0.0, Component
v4.1.5)
MCA errmgr: default_orted (MCA v2.1.0, API v3.0.0, Component
v4.1.5)
MCA errmgr: default_tool (MCA v2.1.0, API v3.0.0, Component
v4.1.5)
MCA ess: env (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA ess: hnp (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA ess: pmi (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA ess: singleton (MCA v2.1.0, API v3.0.0, Component
v4.1.5)
MCA ess: tool (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA ess: slurm (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA filem: raw (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA grpcomm: direct (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA iof: hnp (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA iof: orted (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA iof: tool (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA odls: default (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA odls: pspawn (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA oob: tcp (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA plm: isolated (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA plm: rsh (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA plm: slurm (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA ras: simulator (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA ras: slurm (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA regx: fwd (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA regx: naive (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA regx: reverse (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA rmaps: mindist (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA rmaps: ppr (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA rmaps: rank_file (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA rmaps: resilient (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA rmaps: round_robin (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA rmaps: seq (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA rml: oob (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA routed: binomial (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA routed: direct (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA routed: radix (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA rtc: hwloc (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA schizo: flux (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA schizo: ompi (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA schizo: orte (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA schizo: jsm (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA schizo: slurm (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA state: app (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA state: hnp (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA state: novm (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA state: orted (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA state: tool (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA bml: r2 (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA coll: adapt (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA coll: basic (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA coll: han (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA coll: inter (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA coll: libnbc (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA coll: self (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA coll: sm (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA coll: sync (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA coll: tuned (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA coll: cuda (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA fcoll: dynamic (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA fcoll: dynamic_gen2 (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA fcoll: individual (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA fcoll: two_phase (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA fcoll: vulcan (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA fs: ufs (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA io: ompio (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA io: romio321 (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA op: avx (MCA v2.1.0, API v1.0.0, Component v4.1.5)
MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA osc: pt2pt (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component v4.1.5)
MCA pml: cm (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA pml: ob1 (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA pml: ucx (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA pml: v (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA rte: orte (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA sharedfp: individual (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA sharedfp: lockedfile (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
MCA sharedfp: sm (MCA v2.1.0, API v2.0.0, Component v4.1.5)
MCA topo: basic (MCA v2.1.0, API v2.2.0, Component v4.1.5)
MCA topo: treematch (MCA v2.1.0, API v2.2.0, Component
v4.1.5)
MCA vprotocol: pessimist (MCA v2.1.0, API v2.0.0, Component
v4.1.5)
Ok, that’s what I sent.
So the output from ompi_info shows me that the system should be able to resolve the runtime dependencies on the Open MPI and UCX libraries.
In particular, ompi_info above lists the ob1 pml component as being available:
MCA pml: ob1 (MCA v2.1.0, API v2.0.0, Component v4.1.5)
What did you set your $LD_LIBRARY_PATH set to when you see the error “PML ob1 cannot be selected”? I’m not sure why Open MPI is not resolving this for you.
Hi,
For openmpi I have added this path to LD_LIBRARY_PATH
/nvhpc_2023_2311_Linux_x86_64_cuda_multi/Linux_x86_64/23.11/comm_libs/12.3/openmpi4/openmpi-4.1.5/libLucian
A follow up to this issue:
I found out that nvhpc works with srun if I use cuda11 and nvhpc-hpcx-cuda11 module.
I have to use cuda11 because my cuda kernel driver is not compatible with cuda 12.3.
Note: the install script does not ask which cuda version to use, therefore one has select cuda 11 with the environment variables
NVHPC_DEFAULT_CUDA=11.8
NVHPC_STDPAR_CUDACC=80
Inspecting the output from ompi_info -a for the cuda11 and cuda12 open mpi installations I see that openmpi for cuda11 is not using ‘–with-wrapper-ldflags’ in configure.