RTX PRO 5000 Blackwell (Windows): displays freeze for the whole duration of GPU-saturating work - only after ~24h+ uptime, fresh boot immune, only reb

Gaming and Visualization Technologies General Topics & Other SDKs Drivers - Linux, Windows, MacOS
1

System: RTX PRO 5000 Blackwell 48 GB (desktop workstation) | Windows 11 (26200) | i7-13700K, 128 GB | 3 monitors (one 240 Hz, two lower refresh) | LLM inference via WSL2 (vLLM 0.23 and llama.cpp) | reproduced on drivers 596.59, 596.72 (R595 Production) and 610.47 (R610 NFB)

Symptom: once the machine has been up roughly a day, any GPU-saturating workload freezes ALL displays for the entire duration of the workload - the desktop renders zero frames until the GPU work ends, then recovers. A ~150K-token LLM prefill (100% util at the 300 W cap, ~70 s) reproduces it deterministically as a ~70 s display freeze; a long-running agent job means the screen is frozen for minutes at a time. During the freeze: compute continues at full speed, background apps keep running, but NVML queries (nvidia-smi) hang. On R595 the display always comes back when the workload finishes. On 610.47 one such multi-minute frozen stretch ended in a spontaneous dirty reboot (Kernel-Power 41, no bugcheck, no minidump). No Xid, no TDR, no nvlddmkm event is ever logged (TdrDelay is 10 s and never fires).

Key finding - uptime dependence: the identical workload on a fresh boot runs perfectly smooth. The failure appears only after ~24 h+ of system uptime, and severity is progressive: at ~33 h only saturating prefill triggers it; by ~38 h even model loads and light decode bursts do. Only a full reboot (or driver reinstall, i.e. GPU/GSP reinit) resets the behavior.

Eliminated by direct experiment:

• inference engine (vLLM and llama.cpp both reproduce; llama.cpp prefill also rules out CUDA graphs)

• prefill chunk size (max-num-batched-tokens 2048 and 512: no effect)

• engine process age (an 8-minute-old process reproduces on an aged host; engine restarts do not reset it)

• WSL guest (wsl --shutdown + full VM rebuild does not clear it)

• display driver user-mode state (Win+Ctrl+Shift+B restart does not clear it)

• Windows kernel leaks as cause (CPUs ~idle during the freeze; pool/handle growth is an uptime bystander)

• hardware (0 ECC / remapped rows, 0 PCIe replays, happens at low temperature too)

• P-state / power-state cycling: core+memory clocks locked (-lgc/-lmc) from GPU init, card telemetry-verified pinned at P1 for 37 h with zero P-state transitions - it still degraded and froze on schedule

• commit pressure, VRAM oversubscription, DWM leak, screensaver/display-sleep

ETW evidence: matched WPR (GPU+Video) traces of the identical 150K-token prefill - fresh boot (smooth) vs ~34 h uptime (frozen). During the freeze, DPC totals over the ~70 s window are only dxgkrnl ~2.2 s and nvlddmkm ~1.5 s (CPUs essentially idle - this is not CPU-side spinning). Display/present work simply never gets serviced while the compute queue is saturated. A full nvidia-smi -q diff fresh-vs-aged shows no accumulating counter. Happy to share both .etl files plus 15-second telemetry CSVs covering multiple full aging cycles.

Working hypothesis: state internal to the GSP firmware / hardware scheduler accumulates over wall-clock uptime and progressively degrades compute preemption in favor of display work. On an aged system, display work appears to get NO engine time at all while compute is resident - the freeze lasts exactly as long as the workload. Nothing host-visible tracks it; only GPU reinitialization resets it.

Ask: is this a known issue on Blackwell workstation parts? I can provide traces and telemetry, and I am happy to run debug builds or additional instrumentation. Possibly related community reports: silent Blackwell hangs under sustained inference (open-gpu-kernel-modules issue 1111) and several unexplained RTX PRO 4000/5000/6000 freeze threads - but I have not seen the uptime dependence isolated anywhere else.

2

Hi there @krstevskif, welcome to the NVIDIA developer forums!

I checked our internal issue database but could not find anything that fits your problem description.

I opened a new bug for this, but it usually takes quite some time until there might be feedback.

Thank you for bringing this to our attention!

4

Hi @MarkusHoHo ,

Thanks for the update.

Looking forward to a resolution as this is driving me crazy. Let me know if you need any additional info from me.

Cheers

5

Same bug here on an RTX 5080 — root cause found and fixed on our side (me + Opus haha) : repeated nvidia-smi launches inside WSL2 leak the host driver’s NVRM kernel pool, and nvlddmkm DPCs grow with it.

✅ Quick fix (step by step)

  1. Find who launches nvidia-smi inside WSL2. Run this in WSL for ~30 s and look at the parents:
    for i in $(seq 120); do pgrep -a nvidia-smi; sleep 0.25; done | sort -u
    Typical culprits: resource monitors, dashboards, container stats, statusline scripts, ComfyUI-style “system monitor” nodes. A single long-running nvidia-smi -l N is fine in our tests; a new process every few seconds is not.
  2. Measure the leak on the Windows host (no admin needed): watch the non-paged pool tag NVRM (poolmon from the WDK, or NtQuerySystemInformation(SystemPoolTagInformation)). Sample it twice, 60 s apart, with the GPU idle. A steady climb (we had +20 allocations/s ) means you are affected.
  3. Stop launching nvidia-smi periodically from WSL2. Query the GPU through one persistent NVML session instead (e.g. pynvml: nvmlInit() once, then call the getters in your loop), or collect GPU metrics on the Windows side. Native Windows nvidia-smi did not leak in our tests.
  4. Check the fix: the NVRM slope should drop to ~0 (we went from 20.8/s to 0.03/s).
  5. Purge what already leaked: only a host reboot (or a driver restart) frees it. wsl --shutdown and Win+Ctrl+Shift+B do not. If you try pnputil /restart-device on the GPU that drives your displays, stop the WSL service first (Stop-Service WslService): any WSL client relaunching the VM mid-restart left all our screens black, twice.
  6. Verify: an xperf -on PROC_THREAD+LOADER+DPC+INTERRUPT trace analyzed with xperf -a dpcisr (or LatencyMon) should show no long DPCs in nvlddmkm.sys.

Setup

RTX 5080 16 GB (Blackwell), driver 616.92, Windows 11 Pro 26200.9457, WSL 2.9.12 / Ubuntu 24.04, Ryzen 5 5600G (PCIe 3.0 x16), LLM inference with ollama in WSL2 (qwen3:14b). The same pattern (app hangs, WATCHDOG 0x141, only after 37–75 h of uptime) already existed with our previous RTX 3070 Ti, so it is not Blackwell-only.

What we found

  • What accumulates: the host non-paged pool tag NVRM. On the degraded box it had reached 2.14 M live allocations / 625 MB; after a reboot, 52 k / 67 MB.
  • What feeds it: every launch of nvidia-smi inside WSLhind (same finding as ComfyUI-DaSiWa-Nodes#53, samedriver). Our resource monitor launched nvidia-smi --query-gpu … and --query-compute-apps … every 2 s: ~20 leaked allocations/s, ~1.8 M/day. Stopping only that monitor took the slope from 20.8/s to 0.00/s. Unloading the LLM, stopping netdata’s single nvidia-smi -l 5 loop, or releasing the GPU changed nothing.
  • How it becomes freezes: an xperf DPC/ISR trace on the degraded box (33 s) showed 1,114 DPCs of 8–32 ms, all in nvlddmkm.sys; every other driver stayed under 0.5 ms. After the purge, the same trace showed no DPC above 512 µs. The long DPCs also explain what we saw at idle: robotic USB audio and input lag. During sustained CUDA work, the whole desktop froze (up to 116 s).
  • Markers: dxgvmb_send_sync_msg: wait_for_completion failed in the guest dmesg, and nvidia-smi / CUDA calls becoming 2–3× slower, appear before the big freezes.

Tests performed (26–27 Sep, ~40 h of investigation)

  • Continuous instrumentation: desktop responsiveness e-priority scheduling-gap detector, GPUpower/utilization every 0.5 s, and a 30 s NVRM sampler. 57 desktop freezes recorded (35 ≥ 1 s, 12 ≥ 10 s, max 116.6 s) and 190+ scheduling gaps logged.
  • 30 days of event logs correlated with uptime (26 boo happened between 37 and 75 h of uptime, never before30 h.
  • A/B tests, one variable at a time: NVML pollers off (freezes unchanged); native Windows ollama vs WSL2 with the same model and load (native froze too: 10 s vs 39 s); fresh boot with the same burst (0 freeze vs 39 s); fresh boot with the normal desktop load and with the full LLM workload (0 freeze); HAGS off (degradation at the same dose, so not HAGS); wsl --shutdown (freeze persists); 13 standardized 8-request LLM bursts across these conditions.
  • Accelerated reproduction: 2–4 shell loops relaunching nvidia-smi -q -x inside WSL2 plus a CUDA load → NVRM +500–1,000 allocations/s, first sub-second stalls after ~55 min, the wait_for_completion marker after ~80 min, a 47.6 s freeze after ~2 h, captured in a WPR trace.
  • Source isolation: stopped each GPU client one by one, LLM runner, GPU release, then our monitor); only themonitor mattered.
  • After the fix: full LLM workload plus normal desktopeeze.

For NVIDIA @MarkusHoHo

  • The GPU-PV path should free the resources of a client process when nvidia-smi (or any short-lived NVML client) exits inside WSL2.
  • nvlddmkm DPC time should not scale with the number of stale objects.

Happy to share the NVRM sampler (PowerShell, no admin), the xperf reports, the WPR trace and the full timeline.

Guillaume

6

Full write-up with charts, the step-by-step fix and all the measurements (NVRM pool growth, DPC traces, 4 months of uptime vs hangs):

⧉ https://claude.ai/artifact/QU1ujwqdGpC8Akyz7PuyiU