Running:
NVIDIA-SMI 495.29.05 Driver Version: 495.29.05 CUDA Version: 11.5
on a GV100.
Sometimes, attempting to create a cuda context would result in a segfault.
Here is an example of a sample crash from the bandwidthTest in the demo suite:
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff66a254b in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
(gdb) bt
#0 0x00007ffff66a254b in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#1 0x00007ffff66623d1 in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#2 0x00007ffff64c5be0 in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#3 0x000000000042727b in ?? ()
#4 0x0000000000427308 in ?? ()
#5 0x00007ffff7fa347f in __pthread_once_slow (once_control=0x6a82d0, init_routine=0x4272c0)
at pthread_once.c:116
#6 0x0000000000474959 in ?? ()
#7 0x000000000041e0b7 in ?? ()
#8 0x0000000000446ae6 in ?? ()
#9 0x0000000000408865 in ?? ()
#10 0x0000000000403c2c in ?? ()
#11 0x00007ffff7a720b3 in __libc_start_main (main=0x403bf0, argc=1, argv=0x7fffffffe528,
init=, fini=, rtld_fini=,
stack_end=0x7fffffffe518) at …/csu/libc-start.c:308
#12 0x0000000000403dfe in ?? ()
Once this problem triggers, it seems to be sticky until MPS server is restarted.
Any help would be appreciated.