TL;DR: Batch OCR inference (ONNX Runtime CUDA EP, self-built wheel for sm_121/aarch64/CUDA 13) crashed deterministically on GB10 units running older driver/kernel stacks (580.126.09 / kernel 6.17.0-1008, and 580.95.05 / kernel 6.14.0-1015). The identical workload (same container image, same model, same input batch, same client) ran 100% clean on a GB10 unit with driver 580.142 / kernel 6.17.0-1018. After upgrading the failing units to DGX OS 7.5.0 (driver 580.173.02 / kernel 6.17.0-1031), the same batch passes with zero errors. Posting the full evidence chain in case it helps others hitting the same signatures, and to ask NVIDIA whether this matches a known fix.
Setup (A/B comparison)
| Failing units | Passing unit | |
|---|---|---|
| Hardware | 2x GB10 units (two different OEMs) | GB10 unit (third OEM) |
| DGX OS | 7.4.0 / 7.3.1 | 7.5.0 |
| Kernel | 6.17.0-1008 / 6.14.0-1015 | 6.17.0-1018 |
| Driver | 580.126.09 / 580.95.05 | 580.142 |
| CUDA | 13.0 | 13.0 |
Workload: OCR model inference via ONNX Runtime GPU (self-compiled wheel for CUDA 13 + sm_121 + aarch64; no official prebuilt exists for this combo), 2 worker processes. Another GPU workload shares the same unified memory pool. Input: batches of images with paired concurrent requests.
Failure behavior (100% reproducible on the old stack)
Every batch run crashed, with one request failing mid-batch each time. Two distinct user-space error signatures across runs, both “internal error” class from different CUDA libraries:
Signature 1: cuDNN frontend, Conv node
onnxruntime...Fail: Non-zero status code returned while running Conv node.
Name:'Conv.9' Status Message: CUDNN_FE failure 11: CUDNN_BACKEND_API_FAILED ;
GPU=0 ; file=.../onnxruntime/core/providers/cuda/nn/conv.cc ; line=496 ;
expr=s_.cudnn_fe_graph->execute(cudnn_handle, s_.variant_pack, ws.get());
Signature 2: cuBLAS, FusedMatMul node
onnxruntime...Fail: Non-zero status code returned while running FusedMatMul node.
Name:'MatMul.12/MatmulTransposeFusion/' Status Message: CUBLAS failure 14:
CUBLAS_STATUS_INTERNAL_ERROR ; GPU=0 ;
file=.../onnxruntime/core/providers/cuda/math/matmul.cc ; line=341
Secondary failure, worth highlighting: after the crash, the worker process’s CUDA context is poisoned. The respawned worker fails device detection entirely:
CUDA failure 100: no CUDA-capable device is detected ; GPU=-1 ;
expr=cudaGetDeviceCount(&num_devices);
...
Falling back to ['CPUExecutionProvider'] and retrying.
The service then silently continues on CPU: no crash, no error, just multi-x slower inference until the container is restarted. Only a full container restart restores GPU access. This silent degradation is arguably worse than the crash itself.
Key diagnostic observations
- No Xid errors in dmesg across all crashes. The kernel/driver layer logged nothing. The failure is entirely in user-space CUDA libraries.
- Not a simple OOM: tens of GB of unified memory were still available at the moment of crash (sampled at 10s intervals).
- Concurrency alone is not sufficient: the same paired-concurrent pattern ran clean on the newer stack, and sustained serial load also ran clean on the old stack in earlier testing.
- Hardware exonerated: three GB10 units from three different OEMs showed the same pattern. Failures tracked the software stack version, not the hardware.
- GPU utilization was low (<40%) at crash time; This is a single-GPU inference workload.
Fix confirmation
Upgraded both failing units via standard OTA (apt full-upgrade): DGX OS 7.5.0, kernel 6.17.0-1031, driver 580.173.02. Re-ran the exact same input batch with the same client: entire batch passed, zero errors. Also re-ran a synthetic reproduction (repeated same-second concurrent request pairs): all passed.
Questions for NVIDIA
- Does this match a known fix in the 580.14x–580.17x driver range (or the corresponding cuDNN/cuBLAS updates bundled with DGX OS 7.5.0) for unified-memory allocation under concurrent inference load?
- Is the post-crash
cudaGetDeviceCountfailure (CUDA failure 100 in a respawned process while the host GPU remains healthy) a known consequence of these library-level faults, and is there a recommended in-process recovery path short of a full container restart? - Recommendation for GB10 fleet operators: should driver/OS currency be treated as a hard baseline requirement for sustained batch inference workloads? Our experience suggests yes. Happy to share more details if useful.
Hope this saves someone the debugging time it cost us. The deterministic repro + A/B across stacks made this tractable; if you’re hitting CUDNN_FE failure 11 or CUBLAS_STATUS_INTERNAL_ERROR on GB10 under batch load, check your driver version first.