"RTX 5090 + 5070 Ti Multi-GPU Training: CUDA Driver Crash During Backward Pass (sm_120, PyTorch, gradient_checkpointing)"

AI & Data Science NVIDIA Nemotron
, ,
1

Summary

Training a large language model (Nemotron-3-Nano-30B-A3B) across two Blackwell GPUs (RTX 5090 + RTX 5070 Ti) causes unrecoverable CUDA driver crashes during the backward pass. Two distinct failure modes depending on configuration:

  1. With gradient_checkpointing=True: “expected device meta but got cuda:0” error during gradient computation

  2. Without gradient checkpointing: CUDA driver crashes entirely with “device not ready” errors, requiring full system reboot

Single-GPU inference on the same hardware works perfectly. The issue is specific to multi-GPU training with backward pass computation across both devices.

Hardware

Component Details
GPU 0 PNY GeForce RTX 5090 EPIC-X ARGB OC — 32GB GDDR7
GPU 1 MSI GeForce RTX 5070 Ti Ventus 3X OC — 16GB GDDR7
Combined VRAM 48GB
CPU AMD Ryzen 9 9950X3D
RAM 64GB DDR5-6000
Motherboard MSI MEG X870E Godlike
PSU 1250W
PCIe Layout 5090 in top x16 slot (full bandwidth), 5070 Ti in second slot (x4 chipset)

Thermal Validation

Both GPUs were stress-tested with OCCT 3D Adaptive Steady, Heavy load, 15 minutes prior to any ML work:

  • 5090: 69°C core / 72°C memory junction @ 338W

  • 5070 Ti: 63°C core / 62°C memory junction @ 191W

  • Combined draw: ~530W

  • Zero PCIe lane errors, zero WHEA errors

Hardware is stable under sustained compute load. This is not a thermal or power issue.

Software Environment

Component Version
OS Windows — WSL2 with Docker
Docker Image nvidia/cuda:13.0.0-devel-ubuntu24.04 (custom)
NVIDIA Driver Latest as of March 2026 (supports sm_120)
CUDA Toolkit (container) 13.0
PyTorch 2.9.0a0+git0fabc3b (compiled from source with sm_120 support)
mamba-ssm 2.3.1 (compiled from source against CUDA 13.0)
causal-conv1d 1.6.1 (compiled from source)
transformers Latest
PEFT Latest
bitsandbytes Latest
Python 3.12

Model

  • Nemotron-3-Nano-30B-A3B (NVIDIA’s hybrid Mamba-Transformer architecture)

  • Using the FP8 weights (nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8)

  • Loaded with device_map="auto" and max_memory={0: "28GiB", 1: "10GiB"} to split across both GPUs

  • LoRA fine-tuning via PEFT (r=8, target_modules=[“q_proj”, “v_proj”])

Reproduction Steps

Configuration that triggers the crash:

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model, TaskType
from transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling
from datasets import load_dataset
import torch

tokenizer = AutoTokenizer.from_pretrained(
    "/models/nemotron-fp8", trust_remote_code=True
)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    "/models/nemotron-fp8",
    device_map="auto",
    max_memory={0: "28GiB", 1: "10GiB"},
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
)

model.enable_input_require_grads()
lora_config = LoraConfig(
    r=8, lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05, bias="none",
    task_type=TaskType.CAUSAL_LM,
)
model = get_peft_model(model, lora_config)

# ... load and tokenize dataset ...

training_args = TrainingArguments(
    output_dir="/models/sable-lora-v1",
    num_train_epochs=1,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=8,
    learning_rate=2e-5,
    bf16=True,
    gradient_checkpointing=True,  # Toggle this for different failure modes
    optim="adamw_torch",
)

trainer = Trainer(
    model=model, args=training_args,
    train_dataset=tokenized, data_collator=data_collator,
)
trainer.train()  # Crashes here

Failure Mode 1: gradient_checkpointing=True

Error: expected device meta but got cuda:0

This occurs during the backward pass when gradient checkpointing attempts to recompute activations across the device boundary. The model layers are split across cuda:0 (5090) and cuda:1 (5070 Ti) via device_map="auto". During recomputation, the checkpoint mechanism appears to lose track of which device the tensors should be on.

Failure Mode 2: gradient_checkpointing=False

Error: Unrecoverable CUDA driver crash — “device not ready”

Without gradient checkpointing, the full activation tensors must be held in memory. The backward pass begins but the CUDA driver crashes entirely within seconds. The Docker container becomes unresponsive. nvidia-smi returns “device not ready” for both GPUs. A full system reboot is required to recover. The driver does not recover on its own and the GPUs do not reset.

This is the more concerning failure mode — the driver should handle an out-of-memory condition gracefully rather than crashing.

Additional Context

What works on this exact hardware:

  • Single-GPU inference: Nemotron runs at 95+ tokens/sec on the 5090 alone via llama.cpp. Rock solid, no issues.

  • OCCT stress test: 15 minutes of sustained heavy GPU compute on both cards simultaneously. Zero errors.

  • Multi-GPU model loading: The model loads correctly across both GPUs with device_map="auto". Forward pass for inference works. The crash is specific to the backward pass during training.

  • Basic PyTorch tensor ops on both GPUs: Creating tensors, moving between devices, matrix multiplies — all work fine.

What I’ve ruled out:

  • Thermal throttling: GPUs stay well within limits during the brief period before crash

  • Power delivery: 1250W PSU, combined draw ~530W under stress, plenty of headroom

  • PCIe errors: Zero WHEA errors in Event Viewer, zero PCIe lane errors in stress testing

  • Memory corruption: memtest clean, no other stability issues

Likely contributing factors:

  • The FP8 model’s custom modeling_nemotron_h.py uses a naive Python Mamba forward implementation (torch_forward) instead of the fast mamba-ssm CUDA kernels. This creates very large intermediate tensors (1-8GB per Mamba layer) during forward/backward. On a single GPU this would OOM normally, but across two GPUs the cross-device tensor management appears to trigger the driver crash rather than a clean OOM error.

Expected Behavior

  • With gradient checkpointing: Training should work across both GPUs, with activations recomputed correctly on the appropriate device.

  • Without gradient checkpointing: If VRAM is insufficient, PyTorch should raise a clean torch.cuda.OutOfMemoryError rather than crashing the CUDA driver.

  • The driver should never enter an unrecoverable state that requires a system reboot.

Questions for NVIDIA

  1. Is multi-GPU training across two sm_120 devices with different VRAM sizes (32GB + 16GB) a supported configuration for PyTorch device_map="auto"?

  2. Is there a known issue with the CUDA driver crashing (rather than returning OOM) when backward pass memory allocation fails on sm_120?

  3. Are there recommended driver versions or settings for multi-GPU training on RTX 5090 + 5070 Ti?

Thank you for any guidance. Happy to provide additional logs, nvidia-bug-report outputs, or run specific diagnostic steps.

2

I may have a useful data point for this class of RTX 5090 multi-GPU crashes.

My setup is not identical, but somewhat similar:

  • RTX 5090 AORUS MASTER ICE on the CPU PEG slot, PCIe Gen5 x16

  • RTX 4060 Ti on a chipset/DMI slot, PCIe Gen4 x4

  • Gigabyte Z790 AORUS MASTER X, BIOS F19a

  • Intel i9-13900KS

  • Windows 11

  • NVIDIA Studio driver 596.36

  • CUDA/AI training workloads, ComfyUI/ai-toolkit

At first the issue looked like a PSU, GSP firmware, or defective GPU problem. I was seeing nvlddmkm Event ID 14/153, including:

  • PCIE P2PREQ, Uncorrectable SRAM Error

  • PCIE REORDER, Uncorrectable SRAM Error

  • followed by GpuRcReset TDR

  • application side: CUDA error: unknown / GPU lost

Notably, there was never a full system power-off, no BSOD, and no Kernel-Power 41 event. It was an internal GPU reset/TDR, which argues against a simple PSU OCP trip.

After controlled A/B testing, the 5090 itself appears stable. I was able to run several long single-GPU training runs on the RTX 5090 at PCIe Gen5 x16, 600 W power limit, full CPU load, with no ECC/TDR events and no power instability.

The crashes only became reproducible when a real cross-GPU/offload path was involved. In my case, ComfyUI MultiGPU / DisTorch2 offload to the second GPU reproduced the same “CUDA unknown error → GPU lost” behavior. Disabling the second GPU in Windows Device Manager made the 5090 training stable again.

So at least in my setup, the root cause was not PSU transients or bad 5090 silicon, but a fragile cross-GPU PCIe/P2P/topology path: 5090 on CPU PEG ↔ second GPU behind chipset/DMI.

One important detail: CUDA_VISIBLE_DEVICES=0 is not equivalent to disabling the second GPU. It hides the second GPU from the CUDA application, but the GPU is still enumerated by the OS/NVIDIA driver and the multi-GPU PCIe topology still exists. In my case, disabling the second GPU at the device level was much more effective than only using CUDA_VISIBLE_DEVICES.

Also, even if direct CUDA peer access is unavailable, disabled, or falls back to host-staged transfers on a given GeForce setup, the critical point is that the workload still creates a cross-GPU/offload path through the platform PCIe topology. In my case the failure still surfaced in the PCIE P2PREQ / PCIE REORDER units.

This may be worth checking in similar reports where:

  • each GPU works fine individually

  • synthetic load tests pass

  • single-GPU training is stable

  • crashes happen only with tensor parallelism, offload, custom all-reduce, NCCL/P2P, or multi-GPU training

  • the second GPU is connected through a chipset/DMI slot rather than CPU lanes

Possible diagnostic steps:

  1. Test the RTX 5090 alone with the second GPU disabled in Device Manager or physically removed.

  2. Compare that against CUDA_VISIBLE_DEVICES=0 only.

  3. Check whether the second GPU is behind the chipset/DMI rather than CPU PEG lanes.

  4. Disable custom all-reduce / GPU offload / P2P paths where possible.

  5. Test with ReBAR, ASPM, and IOMMU/VT-d settings if available.

  6. Check whether the crash signature changes when the second GPU is disabled at the device level.

I am not claiming this is the cause for every RTX 5090 crash. PSU, driver/GSP, BIOS, and signal-integrity issues can obviously exist too. But in my case the decisive variable was the active cross-GPU PCIe/P2P/topology path, not raw power draw.