Although the vGPU document for Ubuntu KVM explains that multiple vGPUs are supported (within a limited situation), starting a VM with multiple vGPUs (have tried two and three vGPUs) returns an error.
Here is the link of the document: https://docs.nvidia.com/grid/latest/grid-vgpu-release-notes-ubuntu/index.html#multiple-vgpu-support
I have succeeded to start the VM with a single vGPU, and the nvidia-smi command in the VM shows the vGPU fine.
However, both time-shared vGPUs and MIG-enabled vGPUs fail to start the VM with multiple vGPUs.
I am using Ubuntu 18.04.5 with kernel version 5.4.0-90-generic, and using the 13.x vGPU software.
nvidia-smi shows that the driver version is 470.82, so I think it is vGPU software v13.1.
I am using the Ubuntu KVM + QEMU v4.0 for the hypervisor, and also using Ubuntu 18.04 VM images.
Also, I am using the A100-pcie-40GB GPU.
Here are some error messages when I run # virsh create vm.xml
error: Failed to create domain from vm.xml
error: internal error: qemu unexpectedly closed the monitor: 2021-12-15T02:29:57.628152Z qemu-system-x86_64: -device vfio-pci,id=hostdev0,sysfsdev=/sys/bus/mdev/devices/10106e78-7703-4099-ac5d-ac49ebdd2cbc,bus=pci.0,addr=0x9: warning: vfio 10106e78-7703-4099-ac5d-ac49ebdd2cbc: Could not enable error recovery for the device
2021-12-15T02:29:57.661640Z qemu-system-x86_64: -device vfio-pci,id=hostdev1,sysfsdev=/sys/bus/mdev/devices/f3751074-bca9-4602-b621-7285cf5d5b2f,bus=pci.0,addr=0xa: vfio f3751074-bca9-4602-b621-7285cf5d5b2f: error getting device from group 182: Input/output error
Verify all devices in group 182 are bound to vfio-<bus> or pci-stub and not already in use
Also, my journalctl log shows that the first vGPU was initialized, but the second vGPU incurs an error,
which says that: multiple vGPUs in a VM not supported.
Dec 15 11:30:56 mango1 nvidia-vgpu-mgr[7293]: error: vmiop_log: (0x1): init_device_instance failed for inst 1 with error 1 (multiple vGPUs in a VM not supported)
Dec 15 11:30:56 mango1 nvidia-vgpu-mgr[7293]: error: vmiop_log: (0x1): Initialization: init_device_instance failed error 1
Dec 15 11:30:56 mango1 nvidia-vgpu-mgr[7293]: error: vmiop_log: display_init failed for inst: 1
Dec 15 11:30:56 mango1 nvidia-vgpu-mgr[7293]: error: vmiop_env_log: (0x1): vmiope_process_configuration failed with 0x1f
Dec 15 11:30:56 mango1 nvidia-vgpu-mgr[7293]: error: vmiop_env_log: (0x1): plugin_initialize failed with error:0x1f
Dec 15 11:31:01 mango1 nvidia-vgpu-mgr[7293]: notice: vmiop_log: (0x0): Srubbing completed but notification missed
The vGPU documentation also provides the steps to utilize multiple vGPUs in a single KVM-backed VM.
(Link: https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#adding-vgpu-to-red-hat-el-kvm-vm)
How can I use multiple vGPUs in a single VM backed with Ubuntu KVM, as the document suggests?
Any help would be very grateful, and thanks in advance.