Repeated Xid 31 MMU faults during AMP-enabled YOLOv8s training on DGX Spark (GB10)

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
,
1

I am seeing repeated Xid 31 MMU faults on one NVIDIA DGX Spark while running AMP-enabled YOLOv8s training.

Environment

  • DGX Spark / NVIDIA GB10
  • DGX OS 7.5.0, Ubuntu 24.04.4, kernel 6.17.0-1026-nvidia
  • Host driver 580.159.03
  • NGC PyTorch 26.06-py3
  • Container log: Python 3.12.3, PyTorch 2.13.0a0+8145d630e8.nv26.06, Ultralytics 8.4.110
  • CUDA forward compatibility enabled: CUDA 13.3 user driver 610.43.02 over kernel driver 580.159.03

Observations

  • Five AMP-enabled YOLOv8s runs reached the workload; all five produced Xid 31.
  • Tested AMP-enabled settings included batch 16 / image size 960, batch 12 / 960, and batch 8 / 640.
  • All five kernel events reported ENGINE GRAPHICS GPC2 and FAULT_PDE ACCESS_TYPE_VIRT_READ, from different Python PIDs.
  • With CUDA_LAUNCH_BLOCKING=1, the synchronous application error was:
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

at torch.nn.functional.conv2d.

  • A standalone FP16 torch.matmul loop completed 3,000 seconds / 208,000 iterations without a new Xid.
  • One AMP-disabled training run was manually recorded as running 3.5 hours / 7 epochs without a new Xid before intentional termination. The raw stdout for that run was not retained, so I am treating it as limited evidence.
  • An NGC PyTorch 24.08 comparison was invalid: that image reported GB10 as unsupported and then failed on a NumPy/OpenCV ABI error before GPU work.
  • One-minute telemetry over the captured window reached at most 71 C and 44.03 W. This does not exclude short transients.

Interpretation

The results are strongly associated with the path exercised by AMP-enabled YOLOv8s convolution in this environment. I am not claiming that cuDNN is proven to be the root cause. The alternatives still include an application-level illegal access, a cuDNN/driver/GSP/address-translation interaction, or a localized hardware/firmware issue. The fact that every event reports GPC2 is one reason I do not want to rule out hardware.

Questions

  1. Is this GPC2 FAULT_PDE / VIRT_READ signature known on DGX Spark/GB10 with driver 580.159.03 and NGC PyTorch 26.06?
  2. Is there a currently supported NGC/driver combination that is best for an A/B test on GB10?
  3. Should repeated Xid 31 events that all report GPC2 trigger DGX Spark Field Diagnostics or a hardware support case?
  4. Which additional trace is most useful: Compute Sanitizer, cuDNN logging, a minimal Conv2d loop, Nsight Systems, or another NVIDIA diagnostic?

I can provide the full logs and nvidia-bug-report.sh output through a private support channel. I am not posting the raw archive publicly because it contains system and network identifiers.

P.S. For additional context, the DGX Spark is not our primary system for YOLO training. Our RTX 6000 workstation is currently occupied with large-scale workloads, so we used the available DGX Spark as a secondary compute resource for small-scale YOLOv8s training and AMP compatibility testing.

2

Hi @dall9, please DM me your full logs and nvidia-bug-report, and I will have them reviewed

3

Actually, I was able to reproduce this issue with Pytorch 26.06, but not in Pytorch 26.05. Can you try the 26.05 container and see if it still crashes?