Debugging Segmentation Fault on a CUDA Application with Newer CUDA Version

Accelerated Computing CUDA CUDA NVCC Compiler
1

Hello,
I’m encountering a segmentation fault when loading an application and need some guidance on how to approach debugging this issue. I am building the application from a repo so I expect the error is with my build configuration or tool chain

Problem:
• The application crashes (segfaults) as soon as it’s loaded.
• The repository I’m working with is based on a stable version that many others are successfully running.
System Details:
• OS: Ubuntu 24.04 (fresh install)
• CUDA Version: cuda_12.0.r12.0/compiler.32267302_0
• NVCC Version: Cuda compilation tools, release 12.0, V12.0.140
• Driver Version: NVIDIA GeForce RTX 3090 CUDA Driver Version / Runtime Version 12.7 / 12.6
Build Configuration:
• The project is currently set to build with the flag gencode=arch=compute_37,code=sm_37, but this is not supported in the new version of nvcc.
• I attempted several alternative flags, such as gencode=arch=compute_86,code=sm_86, but the issue persists.
Background:
• I suspect the project is typically built and run on older versions of Linux and CUDA, and I assumed that the newer CUDA version I’m using would be backward-compatible.
• However, it seems that there may be incompatibilities or issues I’m missing.
Questions:
1. Has anyone encountered similar issues with backward compatibility when upgrading to newer CUDA versions (e.g., nvcc v12.x)?
2. Is there a recommended strategy for updating or modifying build flags when migrating a project to a newer CUDA version?
3. Any advice on how to debug or trace the segmentation fault effectively in this context?
Any insights or suggestions would be greatly appreciated!
Justin Madison

Error:
Thread 1 “ggraphitti” received signal SIGSEGV, Segmentation fault.
0x0000555555772936 in __cudaRegisterLinkedBinary (prelinked_fatbinc=0x555555b45ab8 <__fatbinwrap_cc7064e5_28_AllVerticesDeviceFuncs_d_cpp_83fdfb72>, callback_fp=0x7ffff7dad8aa <__nv_cudaEntityRegisterCallback(void**)>) at /usr/lib/nvidia-cuda-toolkit/bin/crt/link.stub:145

145 __cudaPrelinkedFatbins[__i] = (void*)prelinked_fatbinc->data;

2

Justin - not sure that’s the same problem I was having (your ‘arch=37’ makes me really nervous), but I just stumbled over the identical same error message (which is how i found your post - i was looking for a solution :-) ), and after i fixed mine i thought i’d share in case your problem is the same as mine.

In my case, i had a library libinner.so (built with cuda) and another libouter.so (that also used cuda and linked to libinner.so). The app could use and run with libinner just fine, but the moment it tried to make the app use libouter the app died on startup with the above error message.

After some more searching I came across this here:
CUDA separable compilation + shared libraries -> "Invalid function" error - #6 by tomilovanatoliy , and the solution at the very bottom did indeed work for me: If i set both my libraries’ default visibility to hidden, and only explicitly exported the few symbols that the needed, the error went away.

What I presume is that each library has some automatically generated cuda initialization/setup code, and if the libraries’ default visibility is ‘visible’ then these will get into each others’ way. If so then I assume the same error could happen if you have only one shared lib, and a cuda-main app using it, but I haven’t tried this.

Hope this helps.