CUDA error 802 (system not yet initialized) on single-GPU GH200, driver 595.71.05 — nvidia-smi works, cuInit() fails

Accelerated Computing CUDA CUDA Setup and Installation
, , ,
1

System

  • Hardware: NVIDIA GH200 (single GPU, PCI ID 10de:2342), 72-core Grace CPU, aarch64
  • OS: Ubuntu 22.04 (jammy), kernel 5.15.0-181-generic
  • Driver: 595.71.05 (server-open flavor — nvidia-driver-595-server-open, nvidia-dkms-595-server-open)
  • CUDA toolkit (client side): 13.0, PyTorch 2.11.0+cu130
  • No NVSwitch, no InfiniBand fabric — this is a standalone single-GPU node, not part of an HGX/NVL cluster

Problem

nvidia-smi works correctly and shows the GPU with full memory free:

NVIDIA-SMI 595.71.05   Driver Version: 595.71.05   CUDA Version: 13.2
GPU 0  NVIDIA GH200 480GB   00000009:01:00.0   2MiB / 97871MiB used   0% util

But any CUDA client fails to initialize, including a minimal raw driver-API test with no PyTorch involved:

import ctypes
cuda = ctypes.CDLL('libcuda.so.1')
print(cuda.cuInit(0))        # -> 802
count = ctypes.c_int()
print(cuda.cuDeviceGetCount(ctypes.byref(count)))   # -> 3 (NOT_INITIALIZED), count=0

PyTorch reports the same:

UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount().
Error 802: system not yet initialized

What I’ve ruled out

  • NUMA: numactl --hardware shows the GPU’s HBM (node 1, ~97GB) correctly online and free
  • Permissions: /dev/nvidia* and /dev/nvidia-uvm* exist with crw-rw-rw-, all kernel modules (nvidia, nvidia_uvm, nvidia_drm, nvidia_modeset) loaded
  • Fabric Manager: not applicable — nvidia-fabricmanager.service correctly exits with “No CX Bridge devices detected! Detected Pre-NVL5 system… NV_WARN_NOTHING_TO_DO”, since this is a single-GPU board with no NVSwitch
  • Persistence daemon: nvidia-persistenced active and running
  • Confidential Computing: Compute Mode is Default, not a restricted/protected mode
  • SMMU/IOMMU: CONFIG_ARM_SMMU_V3_SVA=y is set, SMMU-v3 instances initialize cleanly in dmesg with no errors
  • KASLR: confirmed disabled at boot already (KASLR disabled due to lack of seed), so not a factor
  • GPU hardware health: no Xid errors, VBIOS (96.00.8D.00.03) and GSP firmware (595.71.05) versions look consistent
  • Driver/CUDA version compatibility: driver supports up to CUDA 13.2, client is 13.0 — no mismatch
  • Architecture: confirmed aarch64 throughout, no x86/arm mismatch in the torch build

What I changed along the way (for context)

The system originally had nvidia-driver-595-open (desktop/workstation flavor) installed, which already showed this same 802 error before any changes were made. While investigating, I found this GPU specifically requires the open kernel module:

NVRM: The NVIDIA GPU 0009:01:00.0 (PCI ID: 10de:2342) installed in this system
NVRM: requires use of the NVIDIA open kernel modules.

I migrated the package family from desktop (-open) to server (-server-open) to add nvidia-fabricmanager-595 cleanly without apt removing the driver stack (an earlier, separate issue where the closed-module -server metapackage conflicted with the open-module requirement above). The package swap itself went cleanly (verified via apt-get install --dry-run before applying), and the open kernel module now loads correctly post-swap. However, the core 802 error was present before this change and is still present after it — so the package family was a real but unrelated fix.

Relevant dmesg lines (post current driver state)

NVRM: GPU 0009:01:00.0: Enabling SMMU SVA feature failed due to lack of necessary kernel configs.
NVRM: loading NVIDIA UNIX Open Kernel Module for aarch64  595.71.05
NVRM: numa memblock size of zero found during device start

I haven’t been able to determine whether the “SMMU SVA feature failed” or “numa memblock size of zero” lines are cosmetic/benign (since NUMA topology and SMMU both otherwise look healthy via the checks above) or are pointing at the actual root cause.

Question

Has anyone seen CUDA error 802 specifically on a single-GPU GH200 (no NVSwitch, no Fabric Manager dependency) that wasn’t resolved by Fabric Manager, InfiniBand modules, or KASLR? Given everything above checks out, I suspect this may be either:

  1. A platform-specific NVLink-C2C initialization step that the generic Ubuntu driver packages aren’t completing correctly on this node, or
  2. A BIOS/firmware-level setting specific to GH200 boards that I’m missing

Any pointers toward GH200-specific init requirements, recommended diagnostic steps, or whether this warrants an NVIDIA Enterprise Support case would be much appreciated. Happy to provide any additional logs.