Failed to start nvidia-fabricmanager.service on centos8

Accelerated Computing DGX Systems (Data Center)
, ,
1

The NVLink detection results are as follows

image
image1165×453 83.7 KB

the error log content upon startup is as follows

Jul 30 23:17:45 GQ203-JTGPUA800-01 nv-fabricmanager[5694]: NVLink initialization failed for NodeId:0 GPU PCI bus id:00000000:04:00.0 enumIndex:0 NVLinkIndex 1
Jul 30 23:17:45 GQ203-JTGPUA800-01 nv-fabricmanager[5694]: NVLink initialization failed for NodeId:0 GPU PCI bus id:00000000:04:00.0 enumIndex:0 NVLinkIndex 4
Jul 30 23:17:45 GQ203-JTGPUA800-01 nv-fabricmanager[5694]: NVLink initialization failed for NodeId:0 GPU PCI bus id:00000000:04:00.0 enumIndex:0 NVLinkIndex 5
Jul 30 23:17:45 GQ203-JTGPUA800-01 nv-fabricmanager[5694]: NVLink initialization failed for NodeId:0 GPU PCI bus id:00000000:04:00.0 enumIndex:0 NVLinkIndex 6
Jul 30 23:17:45 GQ203-JTGPUA800-01 nv-fabricmanager[5694]: NVLink initialization failed for NodeId:0 GPU PCI bus id:00000000:04:00.0 enumIndex:0 NVLinkIndex 7
Jul 30 23:17:45 GQ203-JTGPUA800-01 nv-fabricmanager[5694]: NVLink initialization failed for NodeId:0 GPU PCI bus id:00000000:04:00.0 enumIndex:0 NVLinkIndex 9
Jul 30 23:17:45 GQ203-JTGPUA800-01 nv-fabricmanager[5694]: NVLink initialization failed for NodeId:0 GPU PCI bus id:00000000:04:00.0 enumIndex:0 NVLinkIndex 11
Jul 30 23:17:45 GQ203-JTGPUA800-01 nv-fabricmanager[5694]: NVLink initialization failed for NodeId:0 GPU PCI bus id:00000000:14:00.0 enumIndex:1 NVLinkIndex 1
Jul 30 23:17:45 GQ203-JTGPUA800-01 nv-fabricmanager[5694]: NVLink initialization failed for NodeId:0 GPU PCI bus id:00000000:14:00.0 enumIndex:1 NVLinkIndex 4
Jul 30 23:17:45 GQ203-JTGPUA800-01 nv-fabricmanager[5694]: NVLink initialization failed for NodeId:0 GPU PCI bus id:00000000:14:00.0 enumIndex:1 NVLinkIndex 5
Jul 30 23:17:45 GQ203-JTGPUA800-01 nv-fabricmanager[5694]: NVLink initialization failed for NodeId:0 GPU PCI bus id:00000000:14:00.0 enumIndex:1 NVLinkIndex 6
Jul 30 23:17:45 GQ203-JTGPUA800-01 nv-fabricmanager[5694]: NVLink initialization failed for NodeId:0 GPU PCI bus id:00000000:14:00.0 enumIndex:1 NVLinkIndex 7
Jul 30 23:17:45 GQ203-JTGPUA800-01 nv-fabricmanager[5694]: NVLink initialization failed for NodeId:0 GPU PCI bus id:00000000:14:00.0 enumIndex:1 NVLinkIndex 9
Jul 30 23:17:45 GQ203-JTGPUA800-01 nv-fabricmanager[5694]: NVLink initialization failed for NodeId:0 GPU PCI bus id:00000000:14:00.0 enumIndex:1 NVLinkIndex 11
Jul 30 23:17:45 GQ203-JTGPUA800-01 nv-fabricmanager[5694]: NVSwitch failure detected and degraded mode configuration set to abort Fabric Manager
Jul 30 23:17:45 GQ203-JTGPUA800-01 nv-fabricmanager[5692]: GPU or NVSwitch failure occurred and degraded mode configuration evaluates to abort Fabric Manager
Blockquote