Hi everyone,
I got my new RTX PRO 6000 Blackwell GPU a few days ago and set it up on my workstation. However, I am frequently running into crashes and want to find out if this is just a software issue that can be fixed or a hardware issue and I should RMA.
System
- Fedora 43 + KDE Plasma (Wayland)
- NVIDIA RTX PRO 6000 Blackwell
- 3x 4k monitors over displayport attached directly to the GPU (at some point also another 1080p monitor)
- 1000 watt PSU
- Intel i9 10850k
- 64GB DDR4
- m.2 SSD main, SATA SSD secondary
- open nvidia akmod drivers via rpmfusion 580.142
$ modinfo nvidia | grep ^version
version: 580.142
$ rpm -qa | grep akmod-nvidia
akmod-nvidia-580.142-2.fc43.x86_64
$ cat /etc/modprobe.d/nvidia.conf
options nvidia NVreg_EnableGpuFirmware=1
options nvidia-drm modeset=1 fbdev=1
$ cat /etc/modprobe.d/blacklist.conf
blacklist nouveau
blacklist nova_core
Symptoms
- regular black screen. Sometimes the kernel is fine (just the graphics dies; music still plays and I can still interact with things, although not very well since I cannot see anything), sometimes the kernel crashes/freezes as well
- happened on YouTube video scrubbing, but sometimes playback is just fine (other 30 minute video ran fine, but the scrubbing crashed after just seconds)
- happens frequently when running vLLM (latest master as of right now; did a completely fresh install) with Qwen-3.6-27b with 80% memory target (but possibly also other models. Ran also with Qwen-3.5-9b, which seemed to run slightly better)
- ComfyUI with stable diffusion and qwen image worked just fine
I tried running gpu-burn but could not get it to work:
git clone https://github.com/wilicc/gpu-burn
cd gpu-burn
docker build -t gpu_burn .
docker run --rm --gpus all gpu_burn
>> docker: Error response from daemon: failed to discover GPU vendor from CDI: no known GPU vendor found
And so I just used a custom pytorch stress test:
import torch
import time
print(f"GPU: {torch.cuda.get_device_name(0)}")
print(f"VRAM: {torch.cuda.get_device_properties(0).total_memory / 1024**3:.1f} GB")
size = 16384*5
a = torch.randn(size, size, device='cuda', dtype=torch.float16)
b = torch.randn(size, size, device='cuda', dtype=torch.float16)
for i in range(10000):
c = torch.matmul(a, b)
torch.cuda.synchronize()
PyTorch version I used: 2.11.0+cu130
This stress test did bring the GPU to a 600W power draw and heated it up quite a bit, but nothing crashed. I let the test run for 5-10 minutes, still fine.
At the end I started vLLM again with the Qwen-3.6-27B and this time it already crashed before vLLM was finished starting up. I attached the bug reports from before I ran the python stress test (everything healthy) and after I ran the vLLM (just after the screens turned black).
nvidia-bug-report_before.log.gz (1.1 MB)
nvidia-bug-report_after.log.gz (924.5 KB)
Obvious error messages:
Apr 25 18:37:46 fedora.fritz.box kernel: NVRM: Xid (PCI:0000:01:00): 79, pid=9112, name=nvidia-smi, GPU has fallen off the bus.
Apr 25 18:37:46 fedora.fritz.box kernel: NVRM: Xid (PCI:0000:01:00): 154, GPU recovery action changed from 0x0 (None) to 0x2 (Node Reboot Required)
If more information is needed, I am happy to provide it or run additional tests.
Edit: I just removed the GPU from my PC and put in my previous RTX 3090. I didn’t change anything, but it runs perfectly fine, so I would rule out any system configuration (I also had this GPU before the upgrade, but technically I had different drivers using the runfile installer).