[Bug report] RTX 5090 GPU lost, 0x116 TDR, and failed warm reboot after PCIe SRAM/ECC events (Windows 11, 610.62)

Gaming and Visualization Technologies General Topics & Other SDKs Drivers - Linux, Windows, MacOS
1

1. Summary

I am reporting two NVIDIA GPU-loss incidents on the same RTX 5090 workstation on August 1, 2026 (KST). The first incident emitted explicit NVIDIA Event ID 14 messages for uncorrectable SRAM/ECC errors in the GPU PCIE P2PREQ and PCIE REORDER units, followed by a TDR reset storm and loss of the GPU. A later incident occurred after a successful CUDA training job had already completed: the GPU became lost while nearly idle, Windows bugchecked with 0x116 VIDEO_TDR_FAILURE in nvlddmkm.sys, and the immediate warm boot then bugchecked with 0x7F, parameter 1 0x8 (double fault). The second dump was not written.

Confirmed: The direct crash occurred in the Windows DirectX/NVIDIA TDR reset path. The unresolved fault domain includes the NVIDIA driver, GSP/VBIOS firmware, GPU hardware, PCIe link or power-state transitions, and potentially WSL2 GPU-PV as a trigger or state contributor.

I am not claiming that the workload scheduler, WSL2, the PSU, the model, or a hardware defect is the root cause. NVIDIA Bug 6546168 already contains a private evidence bundle. A related WSL report is open as microsoft/WSL#41224.

2. System configuration

  • GPU: MSI GeForce RTX 5090 SUPRIM, 32 GB (GB202)
  • CPU: AMD Ryzen 9 9950X3D
  • Motherboard: ASRock X870E Nova WiFi, UEFI/BIOS 4.43
  • RAM: 128 GB
  • PSU: Super Flower Leadex VII Pro Platinum 1200 W, ATX 3.1
  • GPU power cable: PSU-supplied native 12V-2x6 cable
  • OS: Windows 11 Pro 64-bit, version 25H2, build 26200.8875
  • NVIDIA driver: 610.62 WHQL (32.0.16.1062, dated 2026-06-11)
  • HAGS: enabled; unchanged during containment
  • WSL: 2.7.10.0, kernel 6.18.33.2-2, Ubuntu under WSL2
  • Workloads: PyTorch/CUDA through AI Toolkit and ComfyUI; a host-side scheduler mediates admission for GPU model-loading jobs
  • TDR registry overrides: none (TdrDelay and TdrDdiDelay unchanged)

3. Incident 1 — PCIe SRAM/ECC events and GPU loss

Observed (2026-08-01 16:46:24–16:48:58 KST):

  • 769 nvlddmkm events over 153.851 seconds: 60 Event ID 14 and 709 Event ID 153.
  • Four occurrences each of:
    • PCIE P2PREQ, Uncorrectable SRAM Error
    • An uncorrectable ECC error has been detected on GPU in the PCIE P2PREQ unit
    • PCIE REORDER, Uncorrectable SRAM Error
    • An uncorrectable ECC error has been detected on GPU in the PCIE REORDER unit
  • Event ID 153 included 708 GpuRcReset TDR occurred on GPUID:100 messages and one UCodeReset TDR occurred on GPUID:100.
  • nvidia-smi exited with code 6 and reported the GPU lost; a reboot was required.
  • The last available host-side telemetry during the incident reported approximately 9.3 GiB of GPU memory use, well below the 32 GB capacity. Because the driver subsequently became unavailable, this value should not be interpreted as an authoritative measurement at the exact failure instant. No OOM was logged.
  • No new Windows kernel dump was retained for this incident.

Observed precursor, not a claimed direct cause: WSL logged dxgkio_make_resident: Ioctl failed: -12 at 14:10:22 and 15:02:02 KST, about 1 hour 44 minutes before the host driver event burst. These events are included in the WSL report because they may represent earlier resource or GPU-PV state, but the timing does not prove causation.

Not yet isolated: The event payload narrows the failing area but does not distinguish GPU silicon damage from driver/GSP firmware, PCIe signal integrity, or a link/power-state transition.

4. Incident 2 — GPU lost near idle, 0x116, then 0x7F after warm boot

Observed:

  • 18:33:18 KST: a FLUX.2 Klein 9B LoRA training job completed successfully. Peak host-reported total GPU memory use was 20,447 MB.
  • 18:36:09 KST: last healthy sample was 2% utilization, 2,733 MB used, 41 C, 38.12 W, PCIe Gen1 x16.
  • 18:36:19 KST: the scheduler’s health check received nvidia-smi exit code 6 (GPU is lost) and blocked the next GPU job.
  • 18:36:43 KST: Windows bugchecked with 0x116 VIDEO_TDR_FAILURE.
  • 18:37:28 KST: the immediate next boot bugchecked with 0x7F UNEXPECTED_KERNEL_MODE_TRAP, parameter 1 0x8 (double fault). volmgr Event 161 recorded that dump creation failed, so no second stack is available.
  • The intended next generation output directory was absent after reboot, and the ComfyUI log contains no post-startup prompt or model-load record. This does not support a claim that a VAE/model compatibility error directly crashed Windows.

Observed scheduler scope: The scheduler detected GPU loss before admitting the next job. It had polled nvidia-smi during the cooldown/health window. There is no evidence that it caused the initial fault. However, whether repeated NVML/nvidia-smi calls are safe or can worsen an already wedged NV_ERR_GPU_IS_LOST state remains a question for NVIDIA. Similar failures in the related forum reports occurred without this scheduler.

5. WinDbg findings for the retained 0x116 dump

Confirmed:

VIDEO_TDR_FAILURE (116)
Arg2: nvlddmkm+0x1958210
Arg3: 0xffffffffc000009a (STATUS_INSUFFICIENT_RESOURCES)
Arg4: 0x4
IMAGE_NAME: nvlddmkm.sys
FAILURE_BUCKET_ID: 0x116_IMAGE_nvlddmkm.sys
FAILURE_ID_HASH: {c89bfe8c-ed39-f658-ef27-f2898997fdbd}

nt!KeBugCheckEx
dxgkrnl!TdrBugcheckOnTimeout+0x101
dxgkrnl!ADAPTER_RENDER::Reset+0x220
dxgkrnl!DXGADAPTER::Reset+0x58a
dxgkrnl!TdrResetFromTimeout+0x15
dxgkrnl!TdrResetFromTimeoutWorkItem+0x22

NVIDIA private symbols were not available, so the dump identifies the module and reset path but not an internal NVIDIA function. Microsoft documents 0x116 as a failed attempt to reset the display driver after a timeout, and documents 0x7F/0x8 as a double fault; neither code by itself proves the physical root cause.

6. What was not happening

  • The retained 0x116 did not occur at full GPU utilization or full VRAM.
  • The preceding training job had finished successfully.
  • There was no recorded CUDA OOM for either failure.
  • The second job was blocked before admission; there is no logged ComfyUI prompt/model load after startup and no intended output directory.
  • No TDR registry extension was active.
  • No WHEA hardware error was retained in the relevant Windows event window.
  • There is no evidence sufficient to identify GPUQ, WSL2, the PSU, a VAE/model incompatibility, or defective GPU hardware as the sole cause.

7. Recovery behavior

The first incident required a reboot. During the second sequence, Windows completed the initial 0x116 bugcheck but the immediate warm boot failed with 0x7F/0x8, and no second dump was written. A subsequent cold power cycle restored GPU visibility.

This resembles other reports in which the GPU remains in a contaminated/lost state across a warm reboot, but that is an inference, not proven from the missing second dump.

8. Reversible containment now applied

These are containment measures only, not claimed fixes:

  • PCIe Link State Power Management disabled for AC and DC.
  • Scheduled GPU workload task disabled.
  • Scheduler daemon/wrapper stopped.
  • ComfyUI listener stopped.
  • Automatic GPU work stopped pending vendor/physical clearance.
  • TdrDelay/TdrDdiDelay unchanged.
  • HAGS unchanged.
  • NVIDIA driver unchanged.
  • No deliberate stress reproduction because the failure removes the physical display and may leave the device unrecoverable without a cold power cycle.

9. Related public cases (verified 2026-08-01)

These are comparison points, not proof of a shared root cause:

  1. RTX 5090 (GB202) + Ryzen 9 9950X3D: Xid 79 at idle, opened July 4, 2026. The thread includes a Windows RTX 5090/9950X3D report with display loss, Event 153 GpuRcReset, a retained 0x116, Arg3 0xC000009A, and cold-power-cycle recovery. Later posts describe NV_ERR_GPU_IS_LOST, failed GSP RPCs, a warm-reboot recurrence, and failures without an external nvidia-smi monitor.
  2. NVIDIA Forum topic 367016, opened April 19, 2026, reports multiple Blackwell/Windows failure modes including 0x116 with Arg3 0xC000009A; one July 30 update says the class persisted on driver 610.88. The thread authors’ D3cold/power-transition interpretation is their hypothesis, not a confirmed match to this case.
  3. WSL issue #41224, opened August 1, 2026. This is the companion report for the earlier WSL dxgkio_make_resident: -12 observations and later host NVIDIA SRAM/ECC/TDR failure. The issue explicitly labels the earlier WSL messages as possible precursor state rather than a simultaneous/direct trigger.
  4. Microsoft debugger references: Bug Check 0x116 VIDEO_TDR_FAILURE and Bug Check 0x7F UNEXPECTED_KERNEL_MODE_TRAP.

10. Vendor tracking

  • NVIDIA support/engineering bug: 6546168
  • Microsoft WSL issue: microsoft/WSL#41224
  • A private evidence archive containing original diagnostic artifacts was previously supplied to the vendors. Only sanitized text extracts are attached publicly.

Could NVIDIA please link this forum report to Bug 6546168 if appropriate?

11. Questions for NVIDIA

  1. Are the PCIE P2PREQ and PCIE REORDER uncorrectable SRAM/ECC Event 14 payloads known GB202/GSP diagnostic signatures?
  2. Can those payloads be emitted by recoverable driver/GSP state corruption, or do they normally indicate permanent SRAM/GPU damage?
  3. Is there a known driver or VBIOS fix for this 0x116/GPU-loss behavior near idle or for failed GPU/GSP initialization after a warm reboot?
  4. Do these repeated uncorrectable payloads and the failed warm boot meet NVIDIA’s threshold for RMA, even though the card becomes visible after a cold power cycle?
  5. What additional sanitized logs would be most useful? I can provide targeted event extracts, a private minidump, dxdiag, and timestamps through the existing bug channel.
  6. Can this be correlated with NVIDIA Bug 6546168?
  7. After NV_ERR_GPU_IS_LOST/nvidia-smi exit 6, is repeated NVML or nvidia-smi polling supported, or should monitoring stop immediately to avoid additional calls into a wedged driver/GSP state?

12. Sanitized evidence

Attached text files:

  • SANITIZED-WINDBG.txt — retained 0x116 analysis excerpt, with host/user paths and dump pointers removed where not diagnostically needed.
  • SANITIZED-NVIDIA-EVENTS.txt — aggregate event counts, exact diagnostic payload strings, and WSL precursor timestamps; computer name, device path, event-record IDs, and raw XML/binary removed.

Original dumps, EVTX/XML, machine identifiers, email/message identifiers, local paths, and executable content are not public. SHA-256 hashes and a redaction checklist are retained locally for chain-of-custody.

SANITIZED-NVIDIA-EVENTS.txt (1.6 KB)

SANITIZED-WINDBG.txt (1.4 KB)

2

Update: third recurrence narrows the trigger to repeated FLUX.2 Klein 9B qfloat8 block transfers

I have now recorded another GPU-loss event on 2026-08-02 KST. I am not intentionally reproducing it again because the earlier sequence caused physical display loss, 0x116, and a failed warm reboot.

Direct recurrence

The first active-load failure and this new failure occurred in the same AI Toolkit model-loading loop and surfaced from the same source statement:

toolkit/util/quantize.py:414
block.to(base_model.device_torch, dtype=base_model.torch_dtype, non_blocking=True)
torch.AcceleratorError: CUDA error: unknown error

This loop processes 32 FLUX.2 Klein Base 9B transformer blocks as:

CPU BF16 block -> CUDA -> qfloat8 quantization -> CPU
  • Earlier active-load incident: progress stopped after 25/32 blocks; the next transfer was the 26th block (single_blocks[17]).
  • New incident: progress stopped after 23/32 blocks; the next transfer was the 24th block (single_blocks[15]).
  • The first NVIDIA driver event followed the visible progress stall by about 12.8 seconds and 11 seconds respectively.
  • Approximately 13.4 GiB and 12.6 GiB of block data had been cycled before the two stalls.

The new training configuration used a different dataset schedule and 3000 rather than 2500 target steps, but dataset loading, latent caching, LoRA creation, and forward/training had not started. AI Toolkit PR #837 only clamps BF16 residual values during forward and therefore does not affect this loader/quantization failure.

Not an OOM, thermal event, or full-power event

The new failed model-load interval reached only:

  • 6,171 MiB observed GPU memory
  • 93.5 W peak in the incident interval
  • 44 C peak
  • last relevant normal sample near the first driver event: 44.14 W, 38 C, 3% utilization, PCIe Gen4 x16
  • configured power limit: 450 W

No CUDA OOM was logged.

Driver event fingerprint is effectively identical

The unique (EventID, payload) sets in the two active-load failures are identical: 28/28 payload types match. Twenty-seven payload counts are also identical. Only the number of GpuRcReset TDR records differs (708 in the earlier incident, 516 in the new incident).

Both incidents contain:

  • Event 14 total: 60
  • PCIE P2PREQ, Uncorrectable SRAM Error: 4
  • matching P2PREQ uncorrectable ECC message: 4
  • PCIE REORDER, Uncorrectable SRAM Error: 4
  • matching REORDER uncorrectable ECC message: 4
  • UCodeReset TDR: 1
  • the same CMDre register payload family

In the new incident, the first Event 14 (CMDre...) occurred at 00:08:26.633 KST and the Event 153 reset storm began 0.214 seconds later. The first explicit REORDER SRAM payload was 18.3 seconds after that first Event 14; the first P2PREQ payload was 27.4 seconds after it.

This ordering matters: the SRAM/ECC payload is real, but public event records alone do not show whether it is the primary syndrome or a condition exposed during repeated failed reset attempts. NVIDIA private register/syndrome decoding is needed.

Current root-cause assessment

  • The qfloat8 block H2D/quantize/D2H path is now a strong, repeatable trigger locus.
  • A user-space AI Toolkit/PyTorch failure should not cause physical display loss, hundreds of host reset attempts, or a state that survives warm reboot. The most likely lower-level domain is NVIDIA host KMD/GSP/WDDM interacting with WSL GPU-PV lifecycle/residency state.
  • A transient GPU/PCIe link/card/VBIOS/power-delivery hardware problem remains possible and is not being dismissed.
  • The low incident power, temperature, and VRAM do not support a simple 1200 W PSU-capacity, overheating, or VRAM-exhaustion explanation.
  • The post-job nvidia-smi polling in the earlier near-idle incident may have amplified an already wedged state, but it cannot explain the new failure: the AI Toolkit block transfer stalled first while the managed process was still running.

Local containment and software workaround

I have stopped GPU work pending a cold reboot and vendor guidance. Local containment now:

  • treats this Klein 9B qfloat8 workload family as one heavy WSL lifecycle per Windows boot;
  • persists that boundary before launch so a scheduler restart cannot bypass it;
  • checks process exit before any NVIDIA telemetry query;
  • treats the first NVIDIA query timeout/GPU-lost response during a protected workload as reboot-required and performs no second query;
  • keeps Windows ComfyUI/Ollama CUDA contexts stopped during WSL training.

AI Toolkit has also been locally modified with an opt-in CPU qfloat8 conversion path. This removes the 32 repeated BF16 H2D/D2H quantization round trips, uses strict per-submodule error propagation, records the exact block/phase, and drops the loaded state-dict mapping before conversion. This is an avoidance patch, not a claim that the underlying NVIDIA/GPU fault is fixed.

Requests for NVIDIA

Please link this recurrence to Bug 6546168 and advise on the following:

  1. Can NVIDIA decode the retained CMDre/P2PREQ/REORDER payload sequence and determine whether the uncorrectable SRAM/ECC messages are a primary GB202 syndrome or a reset-aftereffect?
  2. Could a WSL GPU-PV guest context/residency/transfer failure put the host KMD/GSP into a state that loses the physical display and repeats GpuRcReset hundreds of times?
  3. Is driver 610.62 fix [5449920] relevant, and does 610.88 contain any non-public related KMD/GSP reset fix?
  4. Can the KMD/GSP reset path be hardened so a guest CUDA allocation/transfer failure is contained without host display loss or a failed warm reboot?
  5. Does this repeated exact P2PREQ/REORDER signature meet NVIDIA’s threshold for board/RMA diagnostics even though a cold power cycle restores visibility?
  6. Which bounded trace should be captured before any NVIDIA-approved reproduction? I can prepare a complete kernel dump and an incident-time WSL GPU-PV ETL if NVIDIA/Microsoft provide the appropriate profile.

Companion Microsoft report and detailed recurrence comment:

The private vendor evidence remains associated with NVIDIA Bug 6546168; raw dumps and machine-specific artifacts are not being posted publicly.

3

Hi there @kutaelee0, welcome to the NVIDIA Developer Forums.

Your bug report is in our system and I saw reference to this forum thread in those bug details. You should expect all further communication to go through that bug directly.

Thanks!

5

Update: 610.74 controlled reproduction did not recur (2026-08-14 KST)

Per NVIDIA Bug 6546168’s request, I updated the host driver from 610.62 to 610.74 and ran a controlled reproduction of the narrowed v277/v288 second-load trigger.

Test path

  • WSL2 / AI Toolkit at the original commit e00f3791e221eb65d87351224a7788906b361b89
  • Original GPU-side qfloat8 transformer path retained, including toolkit/util/quantize.py:414 with non_blocking=True
  • FLUX.2 Klein Base 9B transformer and Qwen3-8B text encoder quantized to qfloat8
  • Two separate Python processes in the same Windows boot, with a five-second gap
  • Each cycle completed 32/32 transformer blocks and generated/VAE-decoded one 512x512 PNG
  • The prior v277/v288 failures stopped during the second load at 25/32 and 23/32 blocks respectively

Result

  • Both cycles exited 0
  • Peak total GPU memory observed by the host-side scheduler: 20,424 MiB
  • No GPU loss, TDR, physical display loss, or warm-reboot problem
  • Zero new nvlddmkm, Display, WHEA, BugCheck, volmgr, or Kernel-Power target events
  • No incident-time WSL dxg GPU error
  • Post-test GPU health returned to baseline and remained healthy

The one-step images were intentionally low quality; the test criterion was completion of the full model-load, quantization, generation, and VAE-decode lifecycle, not visual quality.

As a separate check on 610.74, a WSL BF16 power-transition run also completed the sequence 41.82 W idle → 575.21 W at 100% utilization → 41.83 W → 575.60 W at 100% utilization without GPU loss or new target System events. This does not exclude PSU/cable transient behavior.

Conclusion: 610.74 did not reproduce the direct second-load failure in this single controlled two-cycle run. This is evidence against deterministic recurrence, not proof that an intermittent driver, firmware, PCIe, board, or power fault is fixed. It also does not yet cover long training/cooldown or an AI Toolkit-to-ComfyUI cross-runtime handoff.

No new dump was produced because no failure occurred. I am sending the same result through NVIDIA Bug 6546168 and will use that private bug for further engineering communication, as requested.## Update: 610.74 controlled reproduction did not recur (2026-08-14 KST)

Per NVIDIA Bug 6546168’s request, I updated the host driver from 610.62 to 610.74 and ran a controlled reproduction of the narrowed v277/v288 second-load trigger.

Test path

  • WSL2 / AI Toolkit at the original commit e00f3791e221eb65d87351224a7788906b361b89
  • Original GPU-side qfloat8 transformer path retained, including toolkit/util/quantize.py:414 with non_blocking=True
  • FLUX.2 Klein Base 9B transformer and Qwen3-8B text encoder quantized to qfloat8
  • Two separate Python processes in the same Windows boot, with a five-second gap
  • Each cycle completed 32/32 transformer blocks and generated/VAE-decoded one 512x512 PNG
  • The prior v277/v288 failures stopped during the second load at 25/32 and 23/32 blocks respectively

Result

  • Both cycles exited 0
  • Peak total GPU memory observed by the host-side scheduler: 20,424 MiB
  • No GPU loss, TDR, physical display loss, or warm-reboot problem
  • Zero new nvlddmkm, Display, WHEA, BugCheck, volmgr, or Kernel-Power target events
  • No incident-time WSL dxg GPU error
  • Post-test GPU health returned to baseline and remained healthy

The one-step images were intentionally low quality; the test criterion was completion of the full model-load, quantization, generation, and VAE-decode lifecycle, not visual quality.

As a separate check on 610.74, a WSL BF16 power-transition run also completed the sequence 41.82 W idle → 575.21 W at 100% utilization → 41.83 W → 575.60 W at 100% utilization without GPU loss or new target System events. This does not exclude PSU/cable transient behavior.

Conclusion: 610.74 did not reproduce the direct second-load failure in this single controlled two-cycle run. This is evidence against deterministic recurrence, not proof that an intermittent driver, firmware, PCIe, board, or power fault is fixed. It also does not yet cover long training/cooldown or an AI Toolkit-to-ComfyUI cross-runtime handoff.

No new dump was produced because no failure occurred. I am sending the same result through NVIDIA Bug 6546168 and will use that private bug for further engineering communication, as requested.

6

I am in awe of your diligent investigation and reporting skillz @kutaelee ! 👏

Unfortunately, I’ve also been experiencing this issue repeatedly with ComfyUI workflows running Minimax H3 (and Krea 2 occasionally) on an HP Omen 45L (Intel Core Ultra9 285K + 64GB DDR5 RAM + NVIDIA GeForce RTX 5090 32GB DDR7 VRAM). Running Windows 11 Pro. I’ve tested it on a new ComfyUI v0.34.6 install with PyTorch v2.12.1+cu130.

I’ve managed to stop Windows rebooting when it happens by turning off HAGS. But ComfyUI loses the GPU. I have to quit Comfy, then disable the 5090 in Device Manager and re-enable it again, and then run ComfyUI again, for it to detect the GPU again. Yes, I guess just restarting Windows could be faster :)

I’m actually on the latest Nvidia Studio Driver, v.616.56, installed as a “clean installation” after using DDU to remove any previous drivers for the 5090 in Windows safe mode. So, the fix that was deployed in 610.74 should be in 616.56, right?

The PSU in the Omen 45L is an “1200 W 80 Plus Gold certified ATX power supply”.

Is my only option here to upgrade my PSU? (The PC is still under its first year warranty, but I suspect I’ll have a hard time justifying to HP that this behavior is a “hardware failure” under their warranty, given the highly specific scenario in which it occurs.)

I’m open to any ideas you have for working around this issue. Thanks!

7

For anyone else coming across this, switching from the latest Studio Driver (616.56) to the Game Ready Driver (616.64) resolved these GPU loss issues for me. Seems like Nvidia only pushed the “fix” that went into 610.74 to the Game Ready driver channel, up until and including the 616.64 Game Ready release anyway. I mistakenly thought the Studio driver was supposed to be more stable and reliable for Creators.

I see that 616.92 has released today (Studio and Game Ready channels), but I have not updated to that yet.