Program hangs in the ptxjitcompiler.so

I have a program built with Cuda 11.6 and Cuda 11.6 version built TensorFlow.
The driver version and CUDA version from nvidia-smi is as following:

Driver Version: 510.47.03 CUDA Version: 11.6

The program hangs in an unknown call stack.
The call stack is as following, any clues about it? Thank you very much.

Thread 16 (Thread 0x7f65ec7fb700 (LWP 82142)):
#0 0x00007f5ed049616d in ?? () from /lib64/libnvidia-ptxjitcompiler.so.1
#1 0x00007f5ed0825e04 in ?? () from /lib64/libnvidia-ptxjitcompiler.so.1
#2 0x00007f5ed082b81e in ?? () from /lib64/libnvidia-ptxjitcompiler.so.1
#3 0x00007f5ed082b927 in ?? () from /lib64/libnvidia-ptxjitcompiler.so.1
#4 0x00007f5ed04b97d4 in ?? () from /lib64/libnvidia-ptxjitcompiler.so.1
#5 0x00007f5ed04b993f in ?? () from /lib64/libnvidia-ptxjitcompiler.so.1
#6 0x00007f5ed0491a4b in ?? () from /lib64/libnvidia-ptxjitcompiler.so.1
#7 0x00007f5ed0492cfa in ?? () from /lib64/libnvidia-ptxjitcompiler.so.1
#8 0x00007f5ed0673993 in ?? () from /lib64/libnvidia-ptxjitcompiler.so.1
#9 0x00007f5ed0673a37 in ?? () from /lib64/libnvidia-ptxjitcompiler.so.1
#10 0x00007f5ed035b624 in ?? () from /lib64/libnvidia-ptxjitcompiler.so.1
#11 0x00007f5ed0364068 in ?? () from /lib64/libnvidia-ptxjitcompiler.so.1
#12 0x00007f5ed0368362 in ?? () from /lib64/libnvidia-ptxjitcompiler.so.1
#13 0x00007f5ed0369747 in ?? () from /lib64/libnvidia-ptxjitcompiler.so.1
#14 0x00007f5ed035cddc in __cuda_CallJitEntryPoint () from /lib64/libnvidia-ptxjitcompiler.so.1
#15 0x00007f6608795a69 in ?? () from /lib64/libcuda.so.1
#16 0x00007f66087a7767 in ?? () from /lib64/libcuda.so.1
#17 0x00007f66085c9b42 in ?? () from /lib64/libcuda.so.1
#18 0x00007f6608668efa in ?? () from /lib64/libcuda.so.1
#19 0x00007f660866960b in ?? () from /lib64/libcuda.so.1
#20 0x00007f660a48f02c in ?? () from /gpu_test/cuda-11.6/lib64/libcudart.so
#21 0x00007f660a47e55e in ?? () from /gpu_test/cuda-11.6/lib64/libcudart.so
#22 0x00007f660a495594 in ?? () from /gpu_test/cuda-11.6/lib64/libcudart.so
#23 0x00007f660a4971c2 in ?? () from /gpu_test/cuda-11.6/lib64/libcudart.so
#24 0x00007f660a48d05e in ?? () from /gpu_test/cuda-11.6/lib64/libcudart.so
#25 0x00007f660a46e4f3 in ?? () from /gpu_test/cuda-11.6/lib64/libcudart.so
#26 0x00007f660a4a67ab in cudaEventRecord_ptsz () from /gpu_test/cuda-11.6/lib64/libcudart.so
#27 0x00007f6205a65cc3 in accel::cuda::CudaStream::CudaStream() () from /gpu_test/am/tf212_am_abi0_cuda116/test.so