Multiple GPU Assignments to a Single Hyper-V VM with DDA

Virtual GPU Forums More vGPU Forums
,
1

I recently configured Discrete Device Assignment (DDA) on my Windows Server with Hyper-V and successfully assigned a GPU to a virtual machine using the steps outlined in this reference manual.

My Setup:

  • Windows Server with Hyper-V
  • Multiple GPUs available (Example: NVIDIA RTX A400)

What I’ve Done:

  • Successfully assigned one GPU to a VM using DDA
    • Obtain the location path of the GPU that I want to assign to a VM: "PCIROOT(36)#PCI(0000)#PCI(0000)"
    • Dismount the device: Dismount-VMHostAssignableDevice -LocationPath "PCIROOT(36)#PCI(0000)#PCI(0000)" -Force
    • Assign the device to the VM: Add-VMAssignableDevice -LocationPath "PCIROOT(36)#PCI(0000)#PCI(0000)" -VMName Debian12_Dev
    • Power on the VM, and the guest OS (Debian) is able to use the GPU.

Now, I want to add multiple GPUs to a single VM using Hyper-V DDA. I tried the following:

  • Obtain the location path of GPU1 & GPU2 that I want to assign to a VM:
    • GPU1 device location path: PCIROOT(80)#PCI(0200)#PCI(0000)#PCI(1000)#PCI(0000)
    • GPU2 device location path: PCIROOT(36)#PCI(0000)#PCI(0000)
  • Dismount the devices:
    • Dismount-VMHostAssignableDevice -LocationPath "PCIROOT(80)#PCI(0200)#PCI(0000)#PCI(1000)#PCI(0000)" -Force
    • Dismount-VMHostAssignableDevice -LocationPath "PCIROOT(36)#PCI(0000)#PCI(0000)" -Force
  • Assign the devices to the VM:
    • Add-VMAssignableDevice -LocationPath "PCIROOT(36)#PCI(0000)#PCI(0000)" -VMName Debian12_Dev
    • Add-VMAssignableDevice -LocationPath "PCIROOT(80)#PCI(0200)#PCI(0000)#PCI(1000)#PCI(0000)" -VMName Debian12_Dev
  • Power on the VM, but the guest OS (Debian) identifies only one GPU.

Question:
Has anyone tried adding multiple GPUs to a single VM using Hyper-V DDA? If so, what steps did you follow, and did you encounter any challenges?

I’m seeking to optimize GPU resources for specific workloads within a single VM and would appreciate any insights, experiences, or tips from the community.

Thanks in advance!

2

The aforementioned issues were successfully resolved by configuring MMIO space, as outlined in the official Microsoft document: Microsoft Official Document.

GPUs, particularly, require additional MMIO space for the VM to access the memory of that device. While each VM starts with 128 MB of low MMIO space and 512 MB of high MMIO space by default, certain devices or multiple devices may require more space, potentially exceeding these values.

Subsequently, I reconfigured the VM following the instructions in the Microsoft Official Document: VM Preparation for Graphics Devices.

  • Enabled Write-Combining on the CPU using the cmdlet:
    Set-VM -GuestControlledCacheTypes $true -VMName VMName

  • Configured the 32-bit MMIO space with the cmdlet:
    Set-VM -LowMemoryMappedIoSpace 3Gb -VMName VMName

  • Configured greater than 32-bit MMIO space with the cmdlet:
    Set-VM -HighMemoryMappedIoSpace 33280Mb -VMName VMName

To dismount the GPU devices from the host:

  • Located the device’s location path
  • Copied the device’s location path
  • Disabled the GPU in Device Manager

Dismounted the GPU devices from the host partition using the cmdlet:
Dismount-VMHostAssignableDevice -Force -LocationPath $locationPath

Assigned the GPU devices to the VM using the cmdlet:
Add-VMAssignableDevice -LocationPath $locationPath -VMName VMName

The configuration of the VM for DDA has been successfully completed.

Both GPUs are now recognized in my Linux Hyper-V VM:

root@DEB-HYPERV-6fabb3a422fb6e499b57dd2e11a7aa59:~# lspci
0000:00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (AGP disabled) (rev 03)
0000:00:07.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 01)
0000:00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)
0000:00:07.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 02)
0000:00:08.0 VGA compatible controller: Microsoft Corporation Hyper-V virtual VGA
0000:00:0a.0 Ethernet controller: Digital Equipment Corporation DECchip 21140 [FasterNet] (rev 20)
076a:00:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)
e95f:00:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)
root@DEB-HYPERV-6fabb3a422fb6e499b57dd2e11a7aa59:~#