Error using mpiexex (or mpirun)

Accelerated Computing HPC Compilers nvc, nvc++ and nvfortran
1

Hi all,

I am getting a run-time error using the openMPI library included in the 26.3 NVIDIA HPC SDK.

My system reports the following versions on the compiler and library:

$ nvc --version

nvc 26.3-0 64-bit target on x86-64 Linux -tp haswell
NVIDIA Compilers and Tools
Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

$ mpicc --version

nvc 26.3-0 64-bit target on x86-64 Linux -tp haswell
NVIDIA Compilers and Tools
Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

$ mpiexec --version
mpiexec (OpenRTE) 4.1.9a1

Below is a copy of the error I am getting:

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

mpiexec -n 4 parallelSpmv
parallelSpmv: symbol lookup error: /opt/nvidia.26_3/hpc_sdk/Linux_x86_64/26.3/comm_libs/13.1/hpcx/hpcx-2.25.1/ucx/mt/lib/ucx/libuct_cuda.so.0: undefined symbol: cuGetProcAddress_v2
parallelSpmv: symbol lookup error: /opt/nvidia.26_3/hpc_sdk/Linux_x86_64/26.3/comm_libs/13.1/hpcx/hpcx-2.25.1/ucx/mt/lib/ucx/libuct_cuda.so.0: undefined symbol: cuGetProcAddress_v2
parallelSpmv: symbol lookup error: /opt/nvidia.26_3/hpc_sdk/Linux_x86_64/26.3/comm_libs/13.1/hpcx/hpcx-2.25.1/ucx/mt/lib/ucx/libuct_cuda.so.0: undefined symbol: cuGetProcAddress_v2
parallelSpmv: symbol lookup error: /opt/nvidia.26_3/hpc_sdk/Linux_x86_64/26.3/comm_libs/13.1/hpcx/hpcx-2.25.1/ucx/mt/lib/ucx/libuct_cuda.so.0: undefined symbol: cuGetProcAddress_v2

Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.

mpiexec detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

Process name: [[59678,1],1]
Exit code: 127

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Before installing the 26.3 NVIDIA HPC SDK I was using the 25.7 NVIDIA HPC SDK without any problem.

Any idea?

I really appreciate any help you can provide.

Thanks.

2

Hi Edgar,

The symbol “cuGetProcAddress_v2” should be part of the CUDA Driver (libcuda.so).

Do you have the CUDA driver installed on this system?

If so, what version is installed (i.e. what’s the output from “nvidia-smi”)?

I’m wondering if there’s a driver mismatch in that you’re using the HPC-X MPI configured for the 13.1 CUDA Driver and if you have an earlier driver installed, it might be causing issues.

This is just a guess, but we can start with this.

-Mat

3

Hi Mat. Thanks for answering.

This is the output of the nvidia-smi command:

±----------------------------------------------------------------------------+
| NVIDIA-SMI 470.256.02 Driver Version: 470.256.02 CUDA Version: 11.4 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce … Off | 00000000:01:00.0 N/A | N/A |
| N/A 51C P0 N/A / N/A | 420MiB / 1998MiB | N/A Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
| 1 NVIDIA GeForce … Off | 00000000:07:00.0 N/A | N/A |
| N/A 29C P8 N/A / N/A | 6MiB / 2000MiB | N/A Default |
| | | N/A |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
±----------------------------------------------------------------------------+

4

Yes, the CUDA driver is a few generations old. Are you able to update?

If not, then you might try using the HPC-X built for CUDA 12.9 under “Linux_x86_64/26.3/comm_libs/12.9/hpcx/hpcx-2.25.1/ompi/bin/” (assuming you downloaded the NVHPC SDK that includes both 12.9 and 13.1).

Otherwise, you may need to stick with 25.7 which included builds for CUDA 11.

5

Hi Mat.

I am not able to update the CUDA driver in this machine.

Yes I got the NVHPC SDK that includes both 12.9 and 13.1.

I tried to use 12.9 without luck. However, I did not expended enough time on that. I will try again, and post here the results.

Thanks again.

6

Hi Mat,

In order to use 12.9 instead of 13.1 I made a change to the file “/opt/nvidia/hpc_sdk/Linux_x86_64/26.3/cmake/NVHPCConfig.cmake”.

At the very top, I changed:

set(DEFAULT_CUDA_VERSION 13.1)
set(CUDA_VERSIONS 13.1 12.9)
set(PROF_SUFFIX x64)

for

set(DEFAULT_CUDA_VERSION 12.9)
set(CUDA_VERSIONS 13.1 12.9)
set(PROF_SUFFIX x64)

However, when using cmake to create my project, cmake is still getting 13.1, as you can see here (please, see the bolded line):

-- The C compiler identification is NVHPC 26.3.0
– Detecting C compiler ABI info
– Detecting C compiler ABI info - done
– Check for working C compiler: /opt/nvidia/hpc_sdk/Linux_x86_64/latest/compilers/bin/nvc - skipped
– Detecting C compile features
– Detecting C compile features - done
– CMAKE_C_COMPILER_ID: NVHPC
– Found MPI_C: /opt/nvidia.26_3/hpc_sdk/Linux_x86_64/26.3/comm_libs/13.1/hpcx/hpcx-2.25.1/ompi/lib/libmpi.so (found version “3.1”)
– Found MPI: TRUE (found version “3.1”)
– Configuring done (0.8s)
– Generating done (0.0s)
– Build files have been written to: /home/efblack/Development/MatrixGetterAndPlotter/parallelSpmv/buildNvidia

I am probably missing something.

Thank you very much for your help,

Edgar

7

Hi Edgar,

The MPI version being use would be controlled by either explicitly setting your PATH to the version built with 12.9 or by setting the flag “-gpu=cuda12.9”.

The “mpiexec” in “/opt/nvidia.26_3/hpc_sdk/Linux_x86_64/26.3/comm_libs/bin” is really a wrapper script with the CUDA version controlled by the “-gpu=cuda” flag.

You can also set your PATH to the 12.9 version:

export PATH=opt/nvidia.26_3/hpc_sdk/Linux_x86_64/26.3/comm_libs/12.9/hpcx/hpcx-2.25.1/ompi/bin/:$PATH

-Mat

8

Hi Mat.

Thank you again.

Your last suggestion fixed the cmake problem.

efblack@blackPanther:~/Development/MatrixGetterAndPlotter/parallelSpmv/buildNvidia$ cmake ..
– The C compiler identification is NVHPC 26.3.0
– Detecting C compiler ABI info
– Detecting C compiler ABI info - done
– Check for working C compiler: /opt/nvidia/hpc_sdk/Linux_x86_64/latest/compilers/bin/nvc - skipped
– Detecting C compile features
– Detecting C compile features - done
– CMAKE_C_COMPILER_ID: NVHPC
– Found MPI_C: /opt/nvidia.26_3/hpc_sdk/Linux_x86_64/26.3/comm_libs/12.9/hpcx/hpcx-2.25.1/ompi/lib/libmpi.so (found version “3.1”)
– Found MPI: TRUE (found version “3.1”)
– Configuring done (1.1s)
– Generating done (0.0s)
– Build files have been written to: /home/efblack/Development/MatrixGetterAndPlotter/parallelSpmv/buildNvidia

However, the runtime problem persists.:

mpiexec --bind-to core --report-bindings -n 4 parallelSpmv ../matrices/dc1.mm_bin ../matrices/dc1.in_bin ../matrices/dc1.out_bin

[blackPanther:973671] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: [BB/../../..]
[blackPanther:973671] MCW rank 1 bound to socket 0[core 1[hwt 0-1]]: [../BB/../..]
[blackPanther:973671] MCW rank 2 bound to socket 0[core 2[hwt 0-1]]: [../../BB/..]
[blackPanther:973671] MCW rank 3 bound to socket 0[core 3[hwt 0-1]]: [../../../BB]
parallelSpmv: symbol lookup error: /opt/nvidia.26_3/hpc_sdk/Linux_x86_64/26.3/comm_libs/12.9/hpcx/hpcx-2.25.1/ucx/mt/lib/ucx/libuct_cuda.so.0: undefined symbol: cuGetProcAddress_v2
parallelSpmv: symbol lookup error: /opt/nvidia.26_3/hpc_sdk/Linux_x86_64/26.3/comm_libs/12.9/**hpcx/hpcx-2.25.1/ucx/mt/lib/ucx/libuct_cuda.so.0: undefined symbol: cuGetProcAddress_v2
parallelSpmv: symbol lookup error: /opt/nvidia.26_3/hpc_sdk/Linux_x86_64/26.3/comm_libs/12.9/hpcx/hpcx-2.25.1/ucx/mt/lib/ucx/libuct_cuda.so.0: undefined symbol: cuGetProcAddress_v2
parallelSpmv: symbol lookup error: /opt/nvidia.26_3/hpc_sdk/Linux_x86_64/26.3/comm_libs/12.9/hpcx/hpcx-2.25.1/ucx/mt/lib/ucx/libuct_cuda.so.0: undefined symbol: cuGetProcAddress_v2

Primary job terminated normally, but 1 process returned

a non-zero exit code. Per user-direction, the job has been aborted.

mpiexec detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

Process name: [[48137,1],3]

Exit code: 127

Notice that 12.9 is being used.

Thanks again,

Edgar

9

That’s what I wasn’t sure about. Looks like the “v2” versions, including cuGetProcAddress_v2, of the CUDA API was added in 12.0 so the older CUDA 11.4 driver doesn’t recognize these routines.

The recommendation would be to update your CUDA driver, but sans that, you’ll need to revert back to the older compilers which have a MPI build for CUDA 11.

Building you own OpenMPI, might be another option, but I don’t know what other issues you might run into.

10

Hi all,

I was able to make it run.

Inside the directory:

/opt/nvidia.26_3/hpc_sdk/Linux_x86_64/26.3/comm_libs/12.9/hpcx/

there are two directories:

drwxr-xr-x 10 efblack efblack 4096 Mar 20 14:24 hpcx-2.20
drwxr-xr-x 11 efblack efblack 4096 Mar 20 14:30 hpcx-2.25.1

with a link “latest” pointing to hpcx-2.25.1.

I needed to re-link “latest” to point to hpcx-2.20.

Of course, I also needed to set my PATH to the 12.9 and the hpcx-2.20 directories:

export PATH=opt/nvidia.26_3/hpc_sdk/Linux_x86_64/26.3/comm_libs/12.9/hpcx/hpcx-2.20/ompi/bin/:$PATH

My conclusion: hpcx-2.20, which seems to be (OpenRTE) 4.1.7a1, still works for me, while hpcx-2.25.1, which seems to be (OpenRTE) 4.1.9a1, does not work in my system.

Thanks,

Edgar