Hi, I have been testing the new nvhpc 24.1 on our multigpu MPI+OpenACC test code, availble here: [image] GitHub - predsci/multigpu-test-code: This code mimics the basic MPI+GPU tasks of PSI's... This code mimics the basic MPI+GPU tasks of PSI's codes, for use with testing multi-G…

Hi, Here are some updates on this issue with the new 24.5. I am activating the compiler environment with: #!/bin/bash version=24.5 NVARCH=`uname -s`_`uname -m`; export NVARCH NVCOMPILERS=/opt/nvidia/hpc_sdk; export NVCOMPILERS MANPATH=$MANPATH:$NVCOMPILERS/$NVARCH/$version/compilers/man; export M…

Hi, Nope - still seg faults. – Ron

[image] caplanr: Maybe there is some kind of incompatibility of the HPCX MPI with Linux kernel 6? I doubt this would be a kernel issue and when I test your code on an Ubuntu 22.04 system it works fine. Though I’m using A100s in this case so it could be a combination of RTX on newer kernels. …

Hi, I am not familiar in running a debugger. What commands would I do to run it that way? I installed the CUDA toolkit but those samples don’t seem to be there. I am only doing these runs on a single RTX GPU. However, I DO have an MPi call that sends and receives to the same rank (rank 0) so m…

I found the samples github and ran P2P: RLAP4-NV2401: ~/Desktop/cuda_samples/cuda-samples/Samples/0_Introduction/simpleP2P $ ./simpleP2P [./simpleP2P] - Starting... Checking for multiple GPUs... CUDA-capable device count: 1 Two or more GPUs with Peer-to-Peer access capability are required for ./si…

Ok, I assumed you were doing multi-GPU so the output is expected for a single device, though P2P shouldn’t matter. I did run on a single RTX 4080 here (1, 2, and 4 ranks) but didn’t see any issue. To debug, you run “mpirun -np 1 gdb ./psi_multigpu_test_code”, then type “run” from the command promp…

Hi, I am not using that launch script as it is only needed for the “stdpar” version. I am running the original version (psi_multigpu_test_code.f) with 1 GPU. I get the seg fault on both a system where a single GPU is used for graphics and compute, as well as on a system where CUDA_VISIBLE_DEVICES…

Hi, Sorry - I just read the posts above and found the map stuff. I found the map file: maps.txt (94.1 KB) – Ron

Thanks Ron. “map” shows that the multi-threaded UCX libraries are getting pulled in, so that’s not the issue. Can you rerun the code in the debugger and then from the prompt type “where”? Then post the output from the back trace so I can see where the segv is coming from.

Hi, It’s coming from the “Waitall” after the non-blocking iSend and iRecv: (gdb) where #0 __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:317 #1 0x00007ffff7af9a76 in ucs_memcpy_relaxed (len=498000, src=0x7fff815c8580, dst=<optimized out>) at /build-resu…

NV 24.1 Default MPI seg faulting on derived type host_data MPI calls - sometimes

Accelerated Computing HPC Compilers nvc, nvc++ and nvfortran

mfatica February 2, 2024, 12:21am 2

Could you try to set up HPCX in this way?

$ source /opt/nvidia/hpc_sdk/Linux_x86_64/24.1/comm_libs/12.3/hpcx/hpcx-2.17.1/hpcx-mt-init-ompi.sh
$ hpcx_load

Running HPCX-OpenMPI included in NVIDIA HPC SDK 24.1 causes unusual segfault

Topic		Replies	Views
Segmentation fault with NVHPC 25.1 nvc, nvc++ and nvfortran	17	805	June 3, 2025
Running HPCX-OpenMPI included in NVIDIA HPC SDK 24.1 causes unusual segfault nvc, nvc++ and nvfortran networking-ucx , openmpi , hpc-x	3	1170	February 29, 2024
Windows 11 + WSL + CUDA-aware MPI + GeForce 40 series = seg fault, but with GeForce 30 series = OK nvc, nvc++ and nvfortran	3	639	May 15, 2024
MPI send + OpenACC + acc_malloc fail with NVFortran, but work with C nvc, nvc++ and nvfortran	10	349	September 6, 2024
NVHPC Code with Multiple GPUs inside Singularity Container gives UCX Error GPU-Accelerated Libraries	0	301	May 6, 2024
Running nvidia Fortran on multiple GPUs with MPI nvc, nvc++ and nvfortran	36	1315	December 14, 2024
Openacc with cuda nvc, nvc++ and nvfortran cuda	4	539	April 22, 2023
Request support/help for PBS with OpenMPI Legacy PGI Compilers (archived)	21	15429	August 9, 2022
17.10 seg faulting a code that worked in 17.9 Legacy PGI Compilers (archived)	20	15954	April 5, 2019
Suggestions needed to debug a weird segfault Legacy PGI Compilers (archived)	8	1227	July 12, 2021

NV 24.1 Default MPI seg faulting on derived type host_data MPI calls - sometimes

Related topics