8× RTX PRO 6000 Blackwell: cuDNN autotuning triggers host hang requiring AC power cycle

AI & Data Science Deep Learning (Training & Inference) cuDNN
, , ,
1

We’re reporting a reproducible system hang involving eight NVIDIA RTX PRO 6000 Blackwell GPUs, triggered while cuDNN autotuning is enabled.

History

This server ran for approximately 10 months without this issue. The problem began after reinstalling the operating system. No hardware changes were made when the problem started.

We subsequently replaced storage drives and the power-distribution board as troubleshooting steps, but neither resolved the failure.

System

  • 8× NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GB
  • Dual AMD EPYC 9654
  • ASRock Rack GENOA2D24G-2L+, BIOS 2.06
  • Tested Ubuntu 22.04 and 24.04
  • Multiple NVIDIA driver versions tested
  • Latest captured reproduction: NVIDIA 570.195.03, open kernel module; PyTorch 2.10.0+cu128; cuDNN 9.10.2
  • Latest kernel comparison: 6.8.0-142-lowlatency and 6.8.0-138-generic

Failure pattern

With torch.backends.cudnn.benchmark=True:

  • Individual-GPU tests pass.
  • Tests using groups of two or four GPUs pass.
  • The failure occurs when all eight GPUs participate together.
  • The standalone convolution reproducer completes one or two iterations before the host becomes unreachable.
  • Recovery repeatedly requires an AC power cycle. Remote BMC resets do not reliably restore operation.

With autotuning disabled:

  • The isolated eight-GPU convolution passes.
  • An extended ResNet18 run completed 205 iterations.
  • Additional GPU memory, NCCL, and simultaneous CPU/GPU stress tests passed.

Heavy diagnostic logging sometimes allows an otherwise failing run to complete, suggesting timing or algorithm selection influences reproduction.

Troubleshooting completed

The failure persisted across:

  • Ubuntu 22.04 and 24.04.
  • Different kernels, including generic and low-latency variants.
  • Multiple NVIDIA driver versions.
  • Replacement storage drives and migration of Docker storage to the replacement RAID1 array.
  • Replacement of the shared power-distribution board.

Other GPU burn, communication, and tensor-parallel inference workloads have passed. The clearest trigger identified so far is all eight GPUs running together with cuDNN autotuning enabled.

Minimal reproducer

import time
import torch
from torch import nn

torch.backends.cudnn.benchmark = True
torch.backends.cudnn.allow_tf32 = True

n = torch.cuda.device_count()
print("GPUs:", n, "cuDNN:", torch.backends.cudnn.version(), flush=True)

model = nn.Conv2d(
    3, 64, kernel_size=7, stride=2, padding=3, bias=False
).eval().cuda()

if n > 1:
    model = nn.DataParallel(model)

x = torch.randn(n * 256, 3, 224, 224).cuda()

for step in range(3):
    torch.cuda.synchronize()
    print("BEGIN", step, time.time(), flush=True)
    model(x)
    torch.cuda.synchronize()
    print("PASS", step, time.time(), flush=True)

This reproducer can leave our server requiring physical power recovery. Changing benchmark to False provides the passing comparison.

Captured diagnostics

cuDNN frontend logs show execution-plan profiling during the failing iteration. We have not identified the exact failing plan.

During the generic-kernel test, these NVIDIA driver assertions appeared after the passing baseline container exited, before the subsequent failing run:

Assertion failed: pIOVAS != NULL @ io_vaspace.c:592
Assertion failed: pIOVAS != NULL @ io_vaspace.c:601
Assertion failed: Sysmemdesc outlived its attached pGpu @ mem_desc.c:1510

Their relationship to the final hang is not yet established.

Attached are the passing/failing logs, kernel logs, and an older NVIDIA diagnostic bundle from September 30. The latest detailed logs cover Ubuntu 22.04; the diagnostic bundle predates the latest isolated tests.

is this a known issue? Thank you.

full-workaround.log (40.7 KB)

nvidia-bug-report-20260930-older.log.gz (380.8 KB)

generic-kernel-live.log (334.1 KB)

generic-conv-on.log (181.3 KB)