I’m having the same error. I tried disabling KASLR. Was there something else that was done as well?
bbaez@b200-temp:/persist/bbaez$ sudo modprobe ib_umad
modprobe: ERROR: could not insert 'ib_umad': Invalid argument
modprobe: ERROR: ../libkmod/libkmod-module.c:1084 command_do() Error running install command '/sbin/modprobe --ignore-install ib_umad && (if [ -x /sbin/mlnx_bf_configure ]; then /sbin/mlnx_bf_configure; fi)' for module ib_umad: retcode 1
modprobe: ERROR: could not insert 'ib_umad': Invalid argument
Error when starting fabric manager:
Jul 24 20:28:03 b200-temp nvidia-fabricmanager-start.sh[102206]: Checking mlx5_0...
Jul 24 20:28:03 b200-temp nvidia-fabricmanager-start.sh[102206]: Using device mlx5_0, port 0x5000e603003ae5a0
Jul 24 20:28:03 b200-temp nvidia-fabricmanager-start.sh[102206]: Detected NVL5+ system
Jul 24 20:28:03 b200-temp nvidia-fabricmanager-start.sh[102206]: Kernel module "ib_umad" has not been loaded, fabric manager cannot be started
Jul 24 20:28:03 b200-temp nvidia-fabricmanager-start.sh[102206]: Please run "modprobe ib_umad" before starting fabric manager
Jul 24 20:28:03 b200-temp systemd[1]: nvidia-fabricmanager.service: Control process exited, code=exited, status=1/FAILURE
packages installed and nvidia-smi
bbaez@b200-temp:/persist/bbaez$ dpkg -l | grep nvidia
ii libnvidia-cfg1-575-server:amd64 575.57.08-0ubuntu0.24.04.2 amd64 NVIDIA binary OpenGL/GLX configuration library
ii libnvidia-common-575-server 575.57.08-0ubuntu0.24.04.2 all Shared files used by the NVIDIA libraries
ii libnvidia-compute-575-server:amd64 575.57.08-0ubuntu0.24.04.2 amd64 NVIDIA libcompute package
ii libnvidia-container-tools 1.17.8-1 amd64 NVIDIA container runtime library (command-line tools)
ii libnvidia-container1:amd64 1.17.8-1 amd64 NVIDIA container runtime library
ii libnvidia-decode-575-server:amd64 575.57.08-0ubuntu0.24.04.2 amd64 NVIDIA Video Decoding runtime libraries
ii libnvidia-egl-wayland1:amd64 1:1.1.13-1build1 amd64 Wayland EGL External Platform library -- shared library
ii libnvidia-encode-575-server:amd64 575.57.08-0ubuntu0.24.04.2 amd64 NVENC Video Encoding runtime library
ii libnvidia-extra-575-server:amd64 575.57.08-0ubuntu0.24.04.2 amd64 Extra libraries for the NVIDIA Server Driver
ii libnvidia-fbc1-575-server:amd64 575.57.08-0ubuntu0.24.04.2 amd64 NVIDIA OpenGL-based Framebuffer Capture runtime library
ii libnvidia-gl-575-server:amd64 575.57.08-0ubuntu0.24.04.2 amd64 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii nvidia-compute-utils-575-server 575.57.08-0ubuntu0.24.04.2 amd64 NVIDIA compute utilities
ii nvidia-container-toolkit 1.17.8-1 amd64 NVIDIA Container toolkit
ii nvidia-container-toolkit-base 1.17.8-1 amd64 NVIDIA Container Toolkit Base
ii nvidia-dkms-575-server-open 575.57.08-0ubuntu0.24.04.2 amd64 NVIDIA DKMS package (open kernel module)
ii nvidia-driver-575-server-open 575.57.08-0ubuntu0.24.04.2 amd64 NVIDIA driver (open kernel) metapackage
ii nvidia-fabricmanager-575 575.57.08-0ubuntu0.24.04.1 amd64 Fabric Manager for NVSwitch based systems.
ii nvidia-firmware-575-server-575.57.08 575.57.08-0ubuntu0.24.04.2 amd64 Firmware files used by the kernel module
ii nvidia-headless-no-dkms-575-server-open 575.57.08-0ubuntu0.24.04.2 amd64 NVIDIA headless metapackage - no DKMS (open kernel module)
ii nvidia-kernel-common-575-server 575.57.08-0ubuntu0.24.04.2 amd64 Shared files used with the kernel module
ii nvidia-kernel-source-575-server-open 575.57.08-0ubuntu0.24.04.2 amd64 NVIDIA kernel source package
ii nvidia-utils-575-server 575.57.08-0ubuntu0.24.04.2 amd64 NVIDIA Server Driver support binaries
ii xserver-xorg-video-nvidia-575-server 575.57.08-0ubuntu0.24.04.2 amd64 NVIDIA binary Xorg driver
bbaez@b200-temp:/persist/bbaez$ systemctl | grep nvidia
sys-bus-pci-drivers-nvidia.device loaded active plugged /sys/bus/pci/drivers/nvidia
● nvidia-fabricmanager.service loaded failed failed NVIDIA fabric manager service
nvidia-persistenced.service loaded active running NVIDIA Persistence Daemon
bbaez@b200-temp:/persist/bbaez$ nvidia-smi
Thu Jul 24 20:20:00 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 575.57.08 Driver Version: 575.57.08 CUDA Version: 12.9 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA B200 Off | 00000000:1A:00.0 Off | 0 |
| N/A 29C P0 196W / 1000W | 0MiB / 183359MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA B200 Off | 00000000:3B:00.0 Off | 0 |
| N/A 36C P0 202W / 1000W | 0MiB / 183359MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 2 NVIDIA B200 Off | 00000000:4C:00.0 Off | 0 |
| N/A 35C P0 198W / 1000W | 0MiB / 183359MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 3 NVIDIA B200 Off | 00000000:5D:00.0 Off | 0 |
| N/A 28C P0 197W / 1000W | 0MiB / 183359MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 4 NVIDIA B200 Off | 00000000:9B:00.0 Off | 0 |
| N/A 26C P0 192W / 1000W | 0MiB / 183359MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 5 NVIDIA B200 Off | 00000000:BB:00.0 Off | 0 |
| N/A 32C P0 196W / 1000W | 0MiB / 183359MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 6 NVIDIA B200 Off | 00000000:CA:00.0 Off | 0 |
| N/A 33C P0 196W / 1000W | 0MiB / 183359MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 7 NVIDIA B200 Off | 00000000:DC:00.0 Off | 0 |
| N/A 26C P0 197W / 1000W | 0MiB / 183359MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+