NVIDIA L4 GPU Not Detected in Linux Hyper-V VM After DDA

[sfuke]

Environment Details:

• Host Machine:

  • OS: Windows Server 2022 (Hyper-V)
  • GPU: NVIDIA L4 x 3
  • CPU: Intel Xeon Gold 6542Y (with VT-d enabled)
  • IOMMU and SR-IOV: Enabled in BIOS

• Virtual Machine (VM):

  • OS: Linux ( Debian 12)
  • Hyper-V DDA Configurations completing without errors.

Steps Performed:

1. Check the Current State of the VM

   PS C:\Users\Administrator> Get-VM
   

   Output:

   Name                State CPUUsage(%) MemoryAssigned(M) Uptime   Status     Version
   ----                ----- ----------- ----------------- ------   ------     -------
   VM_4.5.12_Build Off   0           0                 00:00:00 正常稼働中 10.0
   

2. Set the Automatic Stop Action

   PS C:\Users\Administrator> Set-VM -Name VM_4.5.12_Build -AutomaticStopAction TurnOff
   

3. Enable Guest Controlled Cache Types

   PS C:\Users\Administrator> Set-VM -GuestControlledCacheTypes $true -VMName VM_4.5.12_Build
   

4. Configure Memory-Mapped I/O Space

   • Allocated low memory-mapped I/O space:

     PS C:\Users\Administrator> Set-VM -LowMemoryMappedIoSpace 3Gb -VMName VM_4.5.12_Build
     

   • Allocated high memory-mapped I/O space:

     PS C:\Users\Administrator> Set-VM -HighMemoryMappedIoSpace 33280Mb -VMName VM_4.5.12_Build
     

5. Dismount the GPU from the Host
   The GPU was removed from the host using the following command:

   PS C:\Users\Administrator> Dismount-VMHostAssignableDevice -LocationPath "PCIROOT(9E)#PCI(0100)#PCI(0000)" -Force
   

6. Assign the GPU to the VM
   The GPU was assigned to the VM using:

   PS C:\Users\Administrator> Add-VMAssignableDevice -LocationPath "PCIROOT(9E)#PCI(0100)#PCI(0000)" -VMName VM_4.5.12_Build
   

7. Start the VM
   After completing the DDA setup, the VM was started:

   PS C:\Users\Administrator> Start-VM -Name VM_4.5.12_Build
   

8. Verify the GPU in the Guest OS
   Inside the guest OS, the following command was used to check if the GPU is visible:

   lspci | grep -i nvidia
   

9. Started the VM and checked for the GPU in the guest OS using lspci.
   

   

Observed Issue:

• The GPU does not appear in the output of lspci inside the Linux VM, despite all configurations completing without errors.
• No errors or warnings were observed during the DDA setup process.

Additional Notes:

• NVIDIA drivers are installed on Windows server host.
• Data Center L4 Driver for Windows 560.94 | Windows Server 2022
  Driver Details | NVIDIA
  
  

[sfuke]

Hi All,

I wanted to provide an update regarding the issue I faced with NVIDIA L4 GPUs on Windows Server Hyper-V VMs using Discrete Device Assignment (DDA). I was able to resolve this issue successfully by following the guidance provided in the NVIDIA documentation:

Bug #2812853: Microsoft DDA not working with some GPUs.

The problem occurred because GPUs with more than 16 GB of memory require additional MMIO (Memory-Mapped Input/Output) space for proper mapping in the guest VM. Without this configuration, the GPU wouldn’t be detected properly in the VM.

---

Solution:

The workaround involves allocating sufficient HighMemoryMappedIoSpace for the VM based on the GPU’s BAR1 memory size and the number of GPUs assigned to the VM. Here’s the step-by-step process:

1. Use the following formula to calculate the required MMIO space:

   MMIO space=2×gpu-bar1-memory×assigned-gpus

Where:

• gpu-bar1-memory: The amount of BAR1 memory for one GPU (equal to total GPU memory if not specified).
• assigned-gpus: The number of GPUs assigned to the VM.

4. Assign the calculated MMIO space to the VM using the Set-VM PowerShell command on the Hyper-V host.

---

Example: NVIDIA L4 GPU with 23 GB Memory

For a VM with 1 GPU assigned, where each GPU has 23 GB of BAR1 memory:

MMIO space=2×23 GB×1=46 GB

PowerShell Command:

Run the following on the Hyper-V host to set the required MMIO space:

Set-VM –HighMemoryMappedIoSpace 46GB –VMName <VM_Name>

---

Example: Multiple GPUs Assigned

For 3 NVIDIA L4 GPUs, each with 23 GB of memory assigned to a single VM:

MMIO space=2×23 GB×3=138 GB

PowerShell Command:

Run the following to set the MMIO space for the VM:

Set-VM –HighMemoryMappedIoSpace 138GB –VMName <VM_Name>

---

Verification:

Once the MMIO space is configured, reboot the VM and check that the GPU is recognized correctly in the guest VM using tools like nvidia-smi or lspci (for Linux VMs).

I hope this helps anyone facing similar issues. If you have additional questions, feel free to ask!