Disabling C-States

Hi,

I’m posting this in case others find it useful. I purchased
a RTX Pro 6000 Blackwell Workstation Edition in May of 2025
and installed it into an ASRock Z890 Taichi motherboard with
an Intel Core Ultra 9 285k. The PC dual boots between Linux
and Windows 11 Pro, with the vast majority of my time spent
in Linux. The PC’s display is driven via HDMI through the
Intel iGPU.

From May of 2025 through August 2026, I’ve been experiencing
frequent segmentation violations and the occasional complete
system lockup. This, when running significant PyTorch loads.

I’ve tried various ASRock/AMI BIOS versions, different NVIDIA
driver versions, even different DDR5 RAM modules at various
XMP and JEDEC profiles and lower timings. With FrankQuadro’s
unwavering support, I’ve even tried a VBIOS update. The VBIOS
update was largely motivated by a desire to play with MIG but
I had also wondered if it might help stabilize the platform.
A big thank you to Frank for all of his considerable help.

When working with Google’s Gemini in an effort to diagnose
the problem, Gemini had written for me a benchmark Python
script. That script reliably locked up my PC within seconds
to minutes. It was never able to get past the 4 minute
duration without a complete system lockup.

The system journal was not always able to record the error
before the lockup. But when it did, it was complaining
of a segmentation violation in libcuda.so.

Gemini was quick to point out that the problem was not
necessarily in libcuda.so or the NVIDIA GPU driver in
general. Instead, Gemini theorized that there was some
problem on the PCIE bus or its related components.

Gemini had me try limiting the PCIE bus to Gen4 speeds.
Then went on to take a look at ASPM. Finally, C-States.
Bingo!

The ASRock/AMI BIOS has C-States enabled by default.
Disabling the C-States was completely transformative.
The PyTorch benchmark script that Gemini wrote for me
was now able to run for any duration without error or
lockup. Indeed, I have been challenging the system for
nearly a month now and it has been absolutely stable.

As an experiment, I tried enabling C-States but restricting
it to transitions between only C0 and C1. The lockups
came back. So C-States disabled is how I run now.

I also notice that with C-States disabled, when loaded
with a PyTorch job, my CPU runs about 8 degrees Celsius
cooler. A nice benefit that I find ironic given that
C-States is a feature that is intended to save power.
I suppose it does when the system is at or near idle.

Anyhow, perhaps my circumstance is unique. I’m happy to
report that the platform is super solid now that I’ve
disabled C-States. If this post helps even just one
other person, I’ll feel that it was worthwhile.

Thanks
Feathers

Interesting findings!
There have been a lot of reports of instabilities of PRO-6000, although most of them were manifesting as XID 79 “GPU fallen off the bus”.

Please consider editing the title to " Disabling C-States prevents CUDA segmentation faults on RTX PRO 6000" for better visibility and state the following:

  • distro and version
  • kernel version
  • NV driver version
  • PyTorch version
  • exactly how and what you disabled:
    • UEFI settings
    • kernel command line params
    • module params
    • others?

Thanks!

Hi,

To be clear.
I’m not recommending that NVIDIA GPU owners blindly disable C-States.
I’m not saying that my circumstance is anything other than unique to myself.
I’m not saying that C-States in of itself is the root cause of my problem.

For me, disabling C-States makes a profound difference. However, I’m quick
to concede that doing so may be masking or mitigating some other root cause.
I do suspect that I would have experienced these problems regardless of GPU
model and perhaps regardless of GPU brand. I simply do not know as I have
a sample size of one.

I am saying, if chasing a gremlin, toggling off C-States may deserve to
be among the list of things to try for diagnostic purposes.

I suspect my system has a defect in the PCIE implementation. A defect
whose presence is triggered when a device that is capable of placing
sizable I/O demands upon the PCIE bus places that load. The alleged
defect may be in the motherboard, the BIOS, or perhaps the CPU.

As per your request, my PC’s specifics over the 14 month duration of
stability problems.

Distro and Version:
Debian 13 currently
Fedora 42 previously

Kernel Version:
Debian 13: 6.12.x, 6.17.x, 7.0.x, 7.1.x, perhaps others in between.
Fedora 42: Not sure, was in the 6.x range, various different kernels.

NV Driver Version:
Always the Open driver.
Fedora: 580 and 590 series
Debian: 590, 595, 610, and as of 2026/09/10 615, most recently at 595.91.07.

PyTorch Version:
Various versions 2.8.0 through 2.14.0.
CUDA versions: 2.8.x, 2.9.x, 3.0.x, 3.1.x, 3.2.x
Python versions: 3.10.x, 3.11.x, 3.12.x, 3.13.x
Every PyTorch application either runs as a container on podman or gets its own
dedicated Python virtual environment. Had been using conda/pip, but now uv.

UEFI Settings:
All defaults except the following.
C-States disabled, enabled is the default.
RGB lighting turned off.
Chassis fans set to monitor CPU and switched to Performance profile.
CoolerControl takes over once Linux boots. Custom sensor, max of CPU and GPU.

Other:
GPU (<77 C), CPU (<81), DRAM (<55), and NVME (<65) temps always monitored and
never allowed to go high.
GPU clock limited to 18000 MHz but have tried without limits.
Four Corsair RS140 Max chassis fans, two in and two out.
Noctua NH-D15 G2 on the CPU, was NZXT Kraken Elite 2024 280 AIO liquid cooler
until recently.
Thermal Grizzly WireView Pro II Wired, a recent addition.
NZXT C1500 power supply.
APC 1500 UPS
Six WD SN850x M.2 NVME SSD, 1x4TB (Windows & Linux), 5x8TB (BTRFS).
TPT1 at 65C, TPT2 at 75C.
Six 128 GiB swap partitions, one per NVME drive, equal priority.
DDR5 is 2x64 Corsair Vengeance CMK128GX5M2B6400C42W ver 3.63.01, and is
on ASRock’s QVL.
DDR5 was 2x48 Corsair Dominator CMP96GX5M2B6600C32W until about February,
not on ASRock’s QVL.
Both sets of DDR5 DIMMs passed numerous memory tests with various testing utilities.

Kernel Command Line Params:
GRUB_CMDLINE_LINUX_DEFAULT=‘quiet resume=UUID=d9769811-187b-48f3-a9a2-65ee4e18ec0b \
libata.allow_tpm=1’

Module Params:
admin@Taichi:~$ /usr/sbin/dkms status
nct6687d/1, 6.12.57+deb13-amd64, x86_64: installed
nct6687d/1, 6.17.13+deb13-amd64, x86_64: installed
nct6687d/1, 7.1.7+deb13-amd64, x86_64: installed
nct6687d/1, 7.1.8+deb13-amd64, x86_64: installed
nvidia/615.71.09, 6.12.57+deb13-amd64, x86_64: installed
nvidia/615.71.09, 6.17.13+deb13-amd64, x86_64: installed
nvidia/615.71.09, 7.1.7+deb13-amd64, x86_64: installed
nvidia/615.71.09, 7.1.8+deb13-amd64, x86_64: installed
wireview-hwmon/1.5.1, 6.12.57+deb13-amd64, x86_64: installed
wireview-hwmon/1.5.1, 6.17.13+deb13-amd64, x86_64: installed
wireview-hwmon/1.5.1, 7.1.7+deb13-amd64, x86_64: installed
wireview-hwmon/1.5.1, 7.1.8+deb13-amd64, x86_64: installed

Thanks
Feathers