Hi Team,
We are trying to deploy vGPU backed kubevirt VM on baremetal Kubernetes cluster with below software and hardware combinations but its not working after multiple attempts.
OS: Red Hat Enterprise Linux 9.6
kernel: 5.14.0-570.12.1.el9_6.x86_64
Kubernetes : 1.33.5 ( CRIO runtime )
Hardware : Asus server with H100 GPU ( 8 GPU’s )
Nivida Operator : 26.3.1 / 26.3.2 ( Tried both )
vGPU driver used to build the vgpu-manager image : NVIDIA-Linux-x86_64-595.71.03-KVM ( AIE version due to H100 GPU). Tried also with x86_64-595, x86_64-580, x86_64-535 images.
Grub settings :
nouveau.modeset=0 intel_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 nohpet
We are bale to deploy the operator on the cluster and all pods are in running state along with vGPU profile also detected properly by the kubelet.
NAME READY STATUS RESTARTS AGE
gpu-operator-d7b57c8d7-fg5tr 1/1 Running 0 18m
gpu-operator-node-feature-discovery-gc-775976dc9d-6lrxt 1/1 Running 0 18m
gpu-operator-node-feature-discovery-master-c784d775d-hdb65 1/1 Running 0 18m
gpu-operator-node-feature-discovery-worker-79s9h 1/1 Running 0 18m
nvidia-vgpu-device-manager-wclvg 1/1 Running 0 3m41s
nvidia-vgpu-manager-daemonset-pwxvv 1/1 Running 0 4m13s
Kubectl describe node :
Capacity:
cpu: 224
devices.kubevirt.io/kvm: 1k
devices.kubevirt.io/tun: 1k
devices.kubevirt.io/vhost-net: 1k
nvidia.com/NVIDIA_H100XM-40C: 16
nvidia.com/gpu: 0
pods: 110
Allocatable:
cpu: 223
devices.kubevirt.io/kvm: 1k
devices.kubevirt.io/tun: 1k
devices.kubevirt.io/vhost-net: 1k
nvidia.com/NVIDIA_H100XM-40C: 16
nvidia.com/gpu: 0
pods: 110
vGPU manager is able to create the mdev device which can be seen on the host.
grep -H "NVIDIA H100XM-40C" /sys/class/mdev_bus/*/mdev_supported_types/nvidia-*/name
/sys/class/mdev_bus/0000:0f:00.2/mdev_supported_types/nvidia-1143/name:NVIDIA H100XM-40C
/sys/class/mdev_bus/0000:0f:00.3/mdev_supported_types/nvidia-1143/name:NVIDIA H100XM-40C
/sys/class/mdev_bus/0000:0f:00.4/mdev_supported_types/nvidia-1143/name:NVIDIA H100XM-40C
/sys/class/mdev_bus/0000:0f:00.5/mdev_supported_types/nvidia-1143/name:NVIDIA H100XM-40C
We have patched the kubevirt CR with the mdev disable configuration.
spec:
certificateRotateStrategy: {}
configuration:
developerConfiguration:
featureGates:
- GPU
- disableMDevConfiguration
When we deploy the kubevirt VM on the cluster it always fails with error : failed to setup container for group 465: Failed to set iommu for container: Invalid argument"
{"component":"virt-launcher","level":"error","msg":"Unable to read from monitor: Connection reset by peer","pos":"qemuMonitorIORead:396","subcomponent":"libvirt","thread":"166","timestamp":"2026-06-25T11:52:28.869000Z"}
{"component":"virt-launcher","level":"error","msg":"internal error: QEMU unexpectedly closed the monitor (vm='default_gpu-vm'): 2026-06-25T11:52:28.868625Z qemu-kvm: -device {\"driver\":\"vfio-pci\",\"id\":\"ua-gpu-gpu1\",\"sysfsdev\":\"/sys/bus/mdev/devices/521347d3-13c6-4224-9de9-a4201f408b4b\",\"display\":\"off\",\"bus\":\"pci.9\",\"addr\":\"0x0\"}: vfio 521347d3-13c6-4224-9de9-a4201f408b4b: failed to setup container for group 465: Failed to set iommu for container: Invalid argument","pos":"qemuProcessReportLogError:2061","subcomponent":"libvirt","thread":"166","timestamp":"2026-06-25T11:52:28.869000Z"}
We have tried multiple combinations and its not working. There is not much we are able to find about this error or if there is any configuration or license we are missing. Can you help us to fix this VM bring up issue or let us know if we need to do any other configuration changes ( Kernel, OS, Nvidia Operator, BIOS ) to make vGPU work on the Baremetal K8s setup I have mentioned.
Thank you for your help !