Hi,
I’m posting this in case others find it useful. I purchased
a RTX Pro 6000 Blackwell Workstation Edition in May of 2025
and installed it into an ASRock Z890 Taichi motherboard with
an Intel Core Ultra 9 285k. The PC dual boots between Linux
and Windows 11 Pro, with the vast majority of my time spent
in Linux. The PC’s display is driven via HDMI through the
Intel iGPU.
From May of 2025 through August 2026, I’ve been experiencing
frequent segmentation violations and the occasional complete
system lockup. This, when running significant PyTorch loads.
I’ve tried various ASRock/AMI BIOS versions, different NVIDIA
driver versions, even different DDR5 RAM modules at various
XMP and JEDEC profiles and lower timings. With FrankQuadro’s
unwavering support, I’ve even tried a VBIOS update. The VBIOS
update was largely motivated by a desire to play with MIG but
I had also wondered if it might help stabilize the platform.
A big thank you to Frank for all of his considerable help.
When working with Google’s Gemini in an effort to diagnose
the problem, Gemini had written for me a benchmark Python
script. That script reliably locked up my PC within seconds
to minutes. It was never able to get past the 4 minute
duration without a complete system lockup.
The system journal was not always able to record the error
before the lockup. But when it did, it was complaining
of a segmentation violation in libcuda.so.
Gemini was quick to point out that the problem was not
necessarily in libcuda.so or the NVIDIA GPU driver in
general. Instead, Gemini theorized that there was some
problem on the PCIE bus or its related components.
Gemini had me try limiting the PCIE bus to Gen4 speeds.
Then went on to take a look at ASPM. Finally, C-States.
Bingo!
The ASRock/AMI BIOS has C-States enabled by default.
Disabling the C-States was completely transformative.
The PyTorch benchmark script that Gemini wrote for me
was now able to run for any duration without error or
lockup. Indeed, I have been challenging the system for
nearly a month now and it has been absolutely stable.
As an experiment, I tried enabling C-States but restricting
it to transitions between only C0 and C1. The lockups
came back. So C-States disabled is how I run now.
I also notice that with C-States disabled, when loaded
with a PyTorch job, my CPU runs about 8 degrees Celsius
cooler. A nice benefit that I find ironic given that
C-States is a feature that is intended to save power.
I suppose it does when the system is at or near idle.
Anyhow, perhaps my circumstance is unique. I’m happy to
report that the platform is super solid now that I’ve
disabled C-States. If this post helps even just one
other person, I’ll feel that it was worthwhile.
Thanks
Feathers