Hello,
We have a H100 SMX5 server which is virtualized.
On the bare metal it had NVLink and Fabric Manager running.
We need to have the fabric manager running also in the VM on top.
The VM is setup with fresh system:
Ubuntu 22.04
Cuda 12.2 (535.161.7)
Fabric manager with the same version as a driver.
When we restarted and checked for Fabric Manager Status, it failed to start with an error:
sudo apt install cuda-drivers-fabricmanager-535
sudo systemctl status nvidia-fabricmanager
× nvidia-fabricmanager.service - NVIDIA fabric manager service
Loaded: loaded (/lib/systemd/system/nvidia-fabricmanager.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2024-04-11 23:28:39 UTC; 8h ago
Apr 11 23:28:39 002 systemd[1]: Starting NVIDIA fabric manager service…
Apr 11 23:28:39 002 nv-fabricmanager[2732]: request to query NVSwitch device information from NVSwitch driver failed with error:WARNING Nothing to do [NV_WARN_NOTHING_TO_DO]
Apr 11 23:28:39 002 systemd[1]: nvidia-fabricmanager.service: Control process exited, code=exited, status=1/FAILURE
Apr 11 23:28:39 002 systemd[1]: nvidia-fabricmanager.service: Failed with result ‘exit-code’.
Apr 11 23:28:39 002 systemd[1]: Failed to start NVIDIA fabric manager service.
Topology is correct:
We cannot find anything about this error fabric documentation.
Please assist