ERROR: The NVIDIA Driver is present, but CUDA failed to initialize. [[ System not yet initialized (error 802) ]]

Infrastructure & Networking Software And Drivers Mellanox OFED
1

I have NVidia GB200 x4 GPU and I am not able to run any torch based applications or cuda examples like devicequery.

nvidia-smi works. I have fabric manager installed.

Driver Version: 580.105.08 CUDA Version: 13.0

sudo systemctl status nvidia-fabricmanager
× nvidia-fabricmanager.service - NVIDIA fabric manager service
     Loaded: loaded (/usr/lib/systemd/system/nvidia-fabricmanager.service; disabled; preset: enabled)
     Active: failed (Result: exit-code) since Sat 2025-11-15 05:31:49 UTC; 20min ago
    Process: 408857 ExecStartPre=/usr/bin/nvidia-fabricmanager-start.sh --mode precheck (code=exited, status=0/SUCCESS)
    Process: 408919 ExecStart=/usr/bin/nvidia-fabricmanager-start.sh --mode start (code=exited, status=1/FAILURE)
        CPU: 314ms

Nov 15 05:31:49 gb2002159d3014 systemd[1]: Starting nvidia-fabricmanager.service - NVIDIA fabric manager service...
Nov 15 05:31:49 gb2002159d3014 nvidia-fabricmanager-start.sh[408919]: Detected Pre-NVL5 system
Nov 15 05:31:49 gb2002159d3014 nvidia-fabricmanager-start.sh[408926]: request to query NVSwitch device information from NVSwitch driver failed with error:WARNING Nothing to do [NV_WARN_NOTHING_TO_DO]
Nov 15 05:31:49 gb2002159d3014 nvidia-fabricmanager-start.sh[408919]: "/usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg" failed! Exit code: 1
Nov 15 05:31:49 gb2002159d3014 systemd[1]: nvidia-fabricmanager.service: Control process exited, code=exited, status=1/FAILURE
Nov 15 05:31:49 gb2002159d3014 systemd[1]: nvidia-fabricmanager.service: Failed with result 'exit-code'.
Nov 15 05:31:49 gb2002159d3014 systemd[1]: Failed to start nvidia-fabricmanager.service - NVIDIA fabric manager service.

Fabric manager exits with this status. I followed DGX OS 7 guide to install the drivers.

2

Good day!

May I know do you have other peripheral devices of the GB200 server? GB200 system is a rack-level system. Except GB200 compute tray, the switch tray is used to enable the NVLink. Do you have the switch tray?