Nvidia Tesla T4, Hyper-V DDA, not detected after VM Reboot

Gaming and Visualization Technologies General Topics and Other SDKs GPU - Hardware
1

I am experiencing challenging issue with DDA passthrough on Hyper-V. I was successful with an initial passthrough of our Tesla T4 GPU into Ubuntu 22 VM using DDA assignment on Hyper-V. On first boot the, GPU is detected by Ubuntu and nvidia-smi successfully displays GPU information. nvidia-smi is enabled by the nvidia-driver-570 and cuda-12.8, dkms. Once the VM is rebooted, the nvidia-driver fails to load the gpu until the host is rebooted.

However, once the Ubuntu VM is rebooted, the device can still be seen with lspci, but the driver fails to load.

image
image1633×650 54.8 KB

DDA Setup information:

### ---- Disable the GPU on the Host ---- ###

## 1. Before assigning the GPU to the VM, disable it on the host:
Disable-PnpDevice -InstanceId "<GPU_INSTANCE_ID>" -Confirm:$false

	# Example Disable GPU by InstanceId
	Disable-PnpDevice -InstanceId "PCI\VEN_10DE&DEV_1EB8&SUBSYS_12A210DE&REV_A1\4&269F7882&0&0000" -Confirm:$false

### ---- Dismount the GPU from the Host ---- ###

## 1. Dismount the GPU from the Host:
Dismount-VMHostAssignableDevice -force -LocationPath "<Device_LocationPath>"

	# Example, Dismount the GPU
	Dismount-VMHostAssignableDevice -Force -LocationPath "PCIROOT(D7)#PCI(0000)#PCI(0000)"

## 2. Verify that the GPU is available for passthrough:
Get-VMHostAssignableDevice

	# Example results showing the GPU is ready for passthrough:
	InstanceID   : PCIP\VEN_10DE&DEV_1EB8&SUBSYS_12A210DE&REV_A1\4&269F7882&0&0000
	LocationPath : PCIROOT(D7)#PCI(0000)#PCI(0000)
	CimSession   : CimSession: .
	ComputerName : WIN-ESLFJ6F5RHO
	IsDeleted    : False

### ---- VM Configuration for DDA ---- ###

## 1. Get the target VM
$NAME = "<VMNAME>"
$VM = Get-VM -Name $NAME 

## 2. Configure the VM to use static memory
$VM | Set-VMMemory -DynamicMemoryEnabled $false

## 3. Configure the VM to shutdown instead of saving state:
$VM | Set-VM -AutomaticStopAction ShutDown
$VM | Set-VM -CheckpointType Disabled

## 4. Enable Write-Combining on the CPU for improved performance:
$VM | Set-VM -GuestControlledCacheTypes $true

## 5. Configure Memory-Mapped I/O (MMIO) space:
# Tesla T4 
# mmio-space = 2 ˟ gpu-bar1-memory ˟ assigned-gpus
# gpu-bar1-memory = 16G
# assigned-gpus = 1
# LowMemoryMappedIoSpace = 3GB
# HighMemoryMappedIoSpace = 32GB

$VM | Set-VM -LowMemoryMappedIoSpace  3GB
$VM | Set-VM -HighMemoryMappedIoSpace 32GB

## 6. Disable Secure Boot in Hyper-V firmware:
$VM | Set-VMFirmware -EnableSecureBoot Off

## 7. Processor Optimizations:
$VM | Set-VMProcessor -ApicMode x2Apic
$VM | Set-VMProcessor -CompatibilityForMigrationEnabled $false
$VM | Set-VMProcessor -CompatibilityForOlderOperatingSystemsEnabled $false 
$VM | Set-VMProcessor -EnableHostResourceProtection $false 

## 8. Memory Optimizations:
$VM | Set-VMMemory -AlignProperties
$VM | Set-VMMemory -HugePagesEnabled $true
$VM | Set-VMMemory -MemoryEncryptionPolicy Disabled

## 9. Network Optimizations:
$VM | Set-VMNetworkAdapter -VrssEnabled $true
$VM | Set-VMNetworkAdapter -VmmqEnabled $true

## 10. Disable Hyper-V Synthetic GPU and nouveau
$VM | Set-VM $VMName -EnhancedSessionTransportType VMBus
echo "blacklist hyperv_fb" | sudo tee -a /etc/modprobe.d/blacklist.conf

## 11. Disable nouveau
echo "blacklist nouveau" | sudo tee -a /etc/modprobe.d/blacklist.conf

### ---- Assign the GPU to the VM ---- ###

## 1. Assign the device to the VM:
$VM | Add-VMAssignableDevice -LocationPath "<Device_LocationPath>"

	# Example, assigning the device:
	$VM | Add-VMAssignableDevice -LocationPath "PCIROOT(D7)#PCI(0000)#PCI(0000)"