Segfault in __pgi_uacc_register_static (libacchost.so) at OpenACC pre-main init

Accelerated Computing HPC Compilers nvc, nvc++ and nvfortran
1

Hi all,

We’re seeing a reproducible segfault in an OpenACC (nvfortran) application
that only occurs on an 8x A100-SXM4 NVSwitch system (AWS p4d.24xlarge),
while the exact same build runs fine on single-GPU T4 and L4 instances.
Posting here in case anyone has seen something similar, or can point us
toward what to check next.

== Environment (crashing system) ==

- AWS EC2 p4d.24xlarge (Tokyo region)
- OS: Red Hat Enterprise Linux 9.7, kernel 5.14.0-687.10.1.el9_8.x86_64
- GPU: 8x NVIDIA A100-SXM4-40GB
- GPU Addressing Mode: HMM (per `nvidia-smi -q`)
- NVIDIA Driver: 610.43.02, CUDA 12.4 (UMD 13.3)
- NVIDIA HPC SDK: nvfortran 24.9, OpenMPI 4.1.5 (bundled with NVHPC 24.9)
- MIG: disabled via `sudo nvidia-smi -mig 0`
- Relevant build flags: -acc -gpu=cc75,cc80,cc90,mem:separate -cuda -O3 -traceback -tp=host

== Symptom ==

Even with `mpiexec -np 1`, the process crashes immediately at startup,
before main()/MPI_Init:

Caught signal 11 (Segmentation fault: address not mapped to object at address 0x13)
libacchost.so(__pgi_uacc_register_static+0x3c)

The exact same build works correctly on:
- g4dn.4xlarge (Tesla T4)
- g6.4xlarge (NVIDIA L4)

== What we’ve ruled out ==

Can’t reproduce on our own single-GPU dev box (A30, no NVSwitch) even
with every variable aligned to the customer’s environment (exact driver
version, NVHPC version, HMM addressing mode, driver flavor). Individually
tested and eliminated: NVHPC/driver version mismatches, mpiexec/libmpi
version mismatches, MPI transport (ucx vs ob1), multi-rank+MPS sharing a
GPU, CUDA_VISIBLE_DEVICES edge cases, -gpu=managed vs mem:separate (no
effect either way), two independently-built official release binaries.
All crash-free on single GPU. Leading hypothesis is GPU count/NVSwitch
topology, but that’s process-of-elimination, not a confirmed mechanism.

== What we found from static analysis ==

The faulting instruction (`__pgi_uacc_register_static+0x3c` in
libacchost.so) is `mov 0x10(%r14),%rbp`. %r14 holds the function’s 5th
argument per the x86-64 calling convention. 0x3 + 0x10 = 0x13 matches
the reported fault address exactly, so the 5th arg is effectively `0x3`
at crash time — a small invalid integer, not a dangling pointer.

Traced the actual caller (a `.init_array` entry, confirmed via gdb `bt`
with no intermediate frame): it loads the 5th arg with
`mov $0x1a85b80,%r8d` — a **hardcoded absolute immediate baked into the
instruction bytes**, pointing at a fixed global (`__PGI_CUDA_LOC`, a
40-byte `void*[5]` table whose first entry points into `.nv_fatbin`).
Confirmed the binary is `ET_EXEC` (not PIE) and `readelf -r` shows zero
relocations for this symbol — so this isn’t a relocation/ASLR failure
either. On paper this instruction can’t produce anything but 0x1a85b80.

We also confirmed the real CUDA device-enumeration path
(`__pgi_uacc_cuda_init_framework`) hasn’t run yet at the point of the
crash, and checked libcuda.so/libmpi.so/libcudart.so for any early
constructor-time logic that could touch shared state before this point
— nothing found in any of them.

== Question ==

Has anyone seen a case where a fixed immediate-load instruction like
this ends up with a wrong value in the destination register at the
point of use, specifically on multi-GPU NVSwitch systems that doesn’t
reproduce on single-GPU boxes? We’re struggling to find a mechanism
that would touch a callee-saved register or a process’s own .text page
this early (before main(), before any CUDA device enumeration) in a way
that’s specific to GPU count/topology.

Also since the crash appears to be in generic compiler-generated
registration infrastructure rather than anything specific to our
application logic, we’d expect even a minimal OpenACC “hello world”
(a single trivial `!$acc kernels` region) built with the same NVHPC
version to crash the same way on this hardware. Has anyone tried
something that minimal on a multi-GPU NVSwitch system? We plan to test
this ourselves but don’t yet have access to matching hardware.

Happy to provide `nm`/`objdump`/gdb output, full build flags, or
anything else that would help narrow this down.

Thanks!

2

Hi chen,weilun,

It’s difficult to tell what’s wrong but my initial guess would be for a version mismatch between the libacchost.so the binary was built with and the runtime on the AWS system.

“__PGI_CUDA_LOC” (since renamed to “__NV_CUDA_LOC”), is the entry point into the device code and “register_static” is the routine that registers the CUDA kernels with the driver. If there were mismatched libacchost.so, that could be jumping to the wrong spot and cause a segv.

Are you building the binary on one system and then running it on AWS? If so, can you do a “ldd myexe” so we can see which libacchost.so it’s pointing at?

Another thing to try is to build the binary statically, i.e. add “-Bstatic_nvidia”, so the shared libraries aren’t used.

The other idea is if there’s something wrong with the multi-GPU targets, specifically with cc90 (A100). If so, try compiling with just “-gpu=cc90” to see if that’s the problem.

If those ideas don’t work out, I think of other possible ways this could happen.

You’re compiler version is a bit old, but I looked through our issue reports and don’t see anything similar, so doubt that the compiler version matters. But you might give our latest release a try just in case.