cudaFree() Indefinite Hang on RTX PRO 6000

Graphics / Linux Linux
,
1

I’m experiencing under sustained multi-stream GPU workloads involving repeated tensor allocations of varying sizes, random crashes. The problem appeared first in IsaacLab (#4668) but then i was able to reproduce similar behaviour within different versions (dev/newton branch as well) and other simulation environments (maniskill and mjlab). Doing some more digging apparently i’m able to get same error even with the attached simple python script, that is allocating and deallocating memory.

cuda_test.txt (480 Bytes)

in ~1-2 hours i get this error:

Traceback (most recent call last):

  File "/home/gminelli-iit.local/cuda_test.py", line 16, in <module>

    torch.cuda.synchronize(dev)

  File "/home/gminelli-iit.local/.conda/envs/rob311/lib/python3.11/site-packages/torch/cuda/__init__.py", line 1040, in synchronize

    return torch._C._cuda_synchronize()

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

RuntimeError: CUDA error: unspecified launch failure

CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.

For debugging consider passing CUDA_LAUNCH_BLOCKING=1

Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

running the script with CUDA_LAUNCH_BLOCKING=1 it gets stuck at a random point of the execution and exploring it with gdb i found this:

Captured with `sudo gdb -p <PID> -batch -ex "thread 1" -ex "bt full"`:

#10  cudaFree()                                         ← hangs here
#11  c10::cuda::CUDACachingAllocator::Native::uncached_delete()
#12  c10::StorageImpl::~StorageImpl()
#13  c10::TensorImpl::~TensorImpl()
#14  THPVariable_subclass_clear()
#15  THPVariable_subclass_dealloc()                     ← Python GC freeing tensor
#16  _PyEval_EvalFrameDefault


**Thread 2 (cuda00001400006 — CUDA IPC channel) at hang:**

#0  __GI___poll(fds=..., nfds=3, timeout=-1)           ← infinite wait, never wakes
#1-3  ?? () from /lib/x86_64-linux-gnu/libcuda.so.1

in my understanding the main thread is blocked inside libcuda.so.1 waiting for a completion signal from the IPC thread. The IPC thread is in poll(timeout=-1) and never receives a wakeup.

full gdb output here:
gdb_output.txt (24.7 KB)

Additional observations:

  • Not working even with OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1
  • Reproduced on both driver 570.195.03 and 590.48.01
  • No Xid errors of any kind in dmesg during hang or after kill
  • kill -9 <PID> is able to kill the process
  • GPU utilization drops to 0% at moment of hang (confirmed via nvidia-smi)
  • PCIe ASPM is hardware-disabled on this system (confirmed via lspci -vv)
  • Disabling Linux runtime PM (echo on > /sys/bus/pci/devices/0000:81:00.0/power/control) seems to reduce problem occurrence but does not eliminate it (or maybe this is a spurious correlation since the error occurs randomly)
  • Same script runs fine on an RTX3090 GPU and a RTX 5090

Env:

  • GPU: NVIDIA RTX PRO 6000 (GB202 / Blackwell, sm_120)
  • Drivers tested: 570.195.03
  • Kernel module: NVIDIA UNIX Open Kernel Module (nvidia-open)
  • CUDA Driver API: 13.1 (forward compat)
  • OS: Ubuntu 22.04 LTS
  • PyTorch: 2.7 nightly, CUDA 12.8 build, Python 3.11
  • PCIe: Gen 5 x16, ASPM hardware-disabled