CUDA Setup and Installation

Accelerated Computing CUDA CUDA Setup and Installation
, ,
1

Environment:

OS:
Windows + WSL2

WSL Kernel:
6.6.123.2-microsoft-standard-WSL2+

GPU:
Dual NVIDIA GPUs

CUDA:
12.6

PyTorch:
2.11.0+cu126

Problem:

When running long-time dual GPU CUDA workloads,
the process randomly crashes after several hours.

Errors:

cudaErrorUnknown

AcceleratorError:
CUDA error: unknown error

Symptoms:

  1. GPU utilization drops to 0%
  2. Process remains alive
  3. Sometimes one GPU crashes and the other enters busy loop
  4. Watchdog restart required

Windows Event Viewer:

nvlddmkm Event ID 153 appeared before CUDA crash.

Example:

20:42:25
nvlddmkm Event 153

20:44
CUDA crash

22:07:25
nvlddmkm Event 153

22:07
dual GPU failure

Previous Driver:

NVIDIA 576.80

Actions Tried:

  • CUDA test passed
  • PyTorch matrix multiplication test passed
  • Reduced GPU power limit
  • Checked WSL dmesg
  • Checked dxg logs

Solution:

Downgraded NVIDIA driver:

576.80 → 572.83

After downgrade:

  • No nvlddmkm Event 153
  • No cudaErrorUnknown
  • Long-time dual GPU workload stable

Question:

Is there any known stability issue with driver 576.80 under WSL2 + heavy CUDA workload?

2

Is the issue that if one GPU crashes the other is also affected some minutes after?