RTX 4090: silent whole-host reset triggered by NVENC on Linux (580-595-610)

Graphics / Linux Linux
1

I have been dealing with this issue since March thinking it was something on my system.

It is now clear this is an Nvidia driver issue and has been for a while.

Summary

On an RTX 4090 under Linux, exercising NVENC silently hard-resets the entire host — instant
power-cycle-like reset, no kernel panic, no oops, no Xid, no MCE, nothing in pstore. The GPU is cold
(46 °C) and drawing only ~92 W of its 450 W budget when it happens.

The fault is codec-dependent, which we think is the most useful clue:

Encoder Result Timing
av1_nvenc Reproducible crash ~15 s into the encode, steady state
h264_nvenc Intermittent crash (2 of 10 runs) ~11–40 s after the encode ends, at idle, during/after the clock ramp-down
hevc_nvenc No crash observed (2 of 2 runs) — (incl. a deliberate 120 s post-encode idle watch)

This is not an application bug: it reproduces with bare ffmpeg, no VR runtime, no compositor
involvement beyond the normal desktop, and no other GPU load.

Minimal reproducer

ffmpeg -hide_banner -y -f lavfi -i testsrc2=duration=300:size=1280x720:rate=90 \
       -c:v av1_nvenc -qp 18 out.mp4

Host resets silently ~15 seconds in. No output is flushed; the file is truncated or absent.

Substituting h264_nvenc reproduces intermittently and with different timing (dies after the
encode completes, during the return to idle). hevc_nvenc has not reproduced.

Telemetry at the moment of death

Sampled at 1 Hz with nvidia-smi, sync-flushed each sample so the log survives the reset
(av1_nvenc run, 2026-08-06):

time          gpuC  gpuW   util%  enc%  memMiB  pstate  memclk  smclk   cpu_pkg
14:35:23.855   43   31.78    1      0    1167    P8       405    210     44.6    <- idle
14:35:24.983   45   54.67    9     45    1645    P2     10251   2790     45.4    <- encoder engages
14:35:26.102   45   91.58    9     50    1645    P2     10251   2790     45.6
   ... steady 91-92 W, 45-46 C, enc 50% for 15 seconds ...
14:35:38.583   46   91.93    9     50    1645    P2     10251   2790     58.9
14:35:39.699   46   92.12    9     50    1645    P2     10251   2790     59.4    <- last sample
                                                                                 <- SILENT RESET

Nothing anomalous precedes it. Temperature, power, clocks and encoder utilisation are all flat and
unremarkable. Note GPU utilisation is only 9 % — the graphics engine is nearly idle; it is the encoder engine that is active.

Full 1 Hz captures available if reqeusted.

Failure signature

  • Instant whole-host reset. Not a hang, not a GPU fallback, not a driver restart.
  • /sys/fs/pstore/ is empty every time, with kernel.panic_on_oops=1 and
    kernel.softlockup_panic=1 armed. They have never once tripped.
  • No Xid, no NVRM error, no MCE, no PCIe AER, no oops in journalctl -k for the crashed boot.
  • The systemd journal simply stops mid-stream; journald reports the file “corrupted or uncleanly shut down” on the next boot.
  • No core dump for any process.

The kernel never gets a chance to react, which suggests the failure is below the kernel’s handler
level.

Environment
GPU NVIDIA GeForce RTX 4090
CPU AMD Ryzen 9 7950X
RAM 64 GiB DDR5, JEDEC 4800 MT/s, 1.1 V — EXPO/XMP disabled (no memory overclock)
OS Fedora 44
Kernel 7.0.9-205.fc44.x86_64
Driver 610.43.03, open kernel module (RPM Fusion akmod-nvidia) (Also 580 and 595)
Desktop KDE Plasma 6.6.5, Wayland
ffmpeg 8.1.1 (Fedora)

What we have already ruled out

Testing has been extensive (see “History” below). The following are excluded:

  • Driver branch — reproduces on 580.159.03, 595.71.05, and 610.43.03.
  • Kernel module variant — reproduces on both the open and proprietary builds of 595.71.05.
  • Distro / kernel — reproduces on Fedora 43 (kernels 6.18.10, 7.0.8) and Fedora 44 (7.0.9).
  • Thermals — dies at 45–46 °C GPU.
  • Power capacity — dies at ~92 W on a card rated to 450 W. Gross PSU capacity is not
    plausibly the constraint.
  • Graphics load — GPU utilisation is 9 % at the time of death; only the encoder is busy.
  • Memory overclock — EXPO is off; RAM runs at JEDEC 4800.
  • Application software — reproduces with bare ffmpeg and nothing else running.

Partial mitigation (a clue, not a fix)

Locking the VRAM clock low with nvidia-smi -pm 1 && nvidia-smi -lmc 5001 measurably reduces crash frequency but does not eliminate it — we have crashed with the lock held. Crashes have been observed at the full P0 VRAM clock (10501 MHz), at the P2 clock (10251 MHz) and at the P8 idle floor (405 MHz), so clock state alone does not explain it either.

Regression

This hardware ran sustained NVENC workloads (VR streaming, many hours) without incident in
January–February 2026, on a driver branch predating 580. The machine was then not used for that workload until mid-May 2026, at which point the fault was present and has persisted through every driver, kernel and distro combination since. We were unable to recover the exact January driver version to bisect — it is no longer packaged for this distro release and will not build against
current kernels.

History

The fault was originally investigated as a bug in WiVRn (an OpenXR streaming server) at
https://github.com/WiVRn/WiVRn/issues/981, because that was the workload exercising NVENC. It has since been shown to reproduce with no VR software present at all, which is why it is being
reported here. A second contributor with an RTX 4090 has reproduced a matching whole-system crash independently.

What would help

  1. Confirmation of whether the codec asymmetry (av1 reproduces readily, h264 intermittently,
    hevc not at all) points at a specific engine or firmware path.
  2. Guidance on capturing anything at all from a failure this abrupt — every standard mechanism
    (pstore, panic-on-oops, Xid, MCE) has produced nothing across dozens of occurrences.
  3. Whether GSP firmware is implicated — we have not been able to test NVreg_EnableGpuFirmware=0
    because the open module requires GSP.

Correction to my original post — hevc_nvenc is not immune.

In the original report I listed hevc_nvenc as “no crash observed (2 of 2 runs)” and presented the
codec asymmetry as the key clue. That was based on short encodes: with the default input pacing,
ffmpeg encodes the 300-second test clip in about 25 seconds of wall time, so those runs only
exercised the encoder briefly.

I have since re-run hevc_nvenc as a sustained encode, pacing the input at native frame rate
(-re) so the encoder runs continuously, at settings closer to a real VR streaming workload
(1920x960, 90 fps):

ffmpeg -hide_banner -y -re -f lavfi -i testsrc2=duration=300:size=1920x960:rate=90 \
       -c:v hevc_nvenc -qp 18 out.mp4

It crashed the host after 3 minutes 41 seconds of continuous encoding — same signature as
before: instant silent reset, empty pstore, no Xid, no MCE, no oops, journal terminating mid-line.

State at the last 1 Hz sample before the reset:

53 C · 76 W · GPU util 3% · encoder util 8-11% · 2160 MiB · P2 · mem 10251 MHz · SM 2565 MHz
CPU package 46 C

Worth noting the encoder duty cycle was only ~8-11% here — real-time pacing leaves the encoder
idle between frames, a much lighter load than the ~50% of the earlier burst runs. It still failed.

Revised interpretation: the codec ordering reflects time-to-failure, not immunity. Ranked by
how quickly each reproduces:

Encoder Time to host reset
av1_nvenc ~15 s, during the encode
h264_nvenc after a ~25 s encode, during/after the return to idle (2 of 10 runs)
hevc_nvenc ~3 min 41 s of sustained encoding

So all three NVENC codecs reproduce it; they differ in how long they take. Duration of encoder
activity appears to matter at least as much as which codec is in use.

The av1_nvenc reproducer in the original post is unaffected and remains the fastest way to
trigger it (~15 seconds, one command, no other software involved).

Apologies for the initial framing — I had over-read a difference that turned out to be an artifact
of how long each test ran.