Only 4/8 GPUs can be initialized with AMD SEV-SNP and NVIDIA PPCIe mode in CVM

costinchen · January 5, 2026, 8:11am

Hello, I am trying to launch a guest OS with H20 GPU * 8 + NVSwitch * 4 passed through into it, but only 4 of 8 GPUs can be successfully initialized.

Environment:

CPU: AMD EPYC 9K84 96-Core
GPU: Nvidia H20 GPU * 8
Driver: 580.95.05 open
Cuda: 13.0.2

qemu command:

QEMU_CMDLINE=${QEMU_REAL_PATH}
QEMU_CMDLINE+=" -enable-kvm -cpu EPYC-v4,host-phys-bits=on -machine q35"
QEMU_CMDLINE+=" -smp $CORES,maxcpus=255"
QEMU_CMDLINE+=" -m ${MEM}G,slots=2,maxmem=128G"
QEMU_CMDLINE+=" -no-reboot"
QEMU_CMDLINE+=" -bios $OVMF_REAL_PATH"
QEMU_CMDLINE+=" -drive file=$VM_IMAGE_REAL_PATH,if=none,id=disk0,format=qcow2"
QEMU_CMDLINE+=" -device virtio-scsi-pci,id=scsi0,disable-legacy=on,iommu_platform=true,romfile= "
QEMU_CMDLINE+=" -device scsi-hd,drive=disk0"
QEMU_CMDLINE+=" -netdev user,id=vmnic,hostfwd=tcp::2222-:22"
QEMU_CMDLINE+=" -device virtio-net-pci,disable-legacy=on,iommu_platform=true,netdev=vmnic,romfile= "
if [ -n "${SEV_SNP}" ]; then
        QEMU_CMDLINE+=" -machine confidential-guest-support=sev0,vmport=off"
        QEMU_CMDLINE+=" -object sev-snp-guest,id=sev0,policy=0x30002,cbitpos=51,reduced-phys-bits=1"
                if [ -n "${KERNEL_HASH}" ]; then
                        QEMU_CMDLINE+=",kernel-hashes=on"
                        QEMU_CMDLINE+=" -kernel ${KERNEL_REAL_PATH}"
                        QEMU_CMDLINE+=" -initrd ${INITRD_REAL_PATH}"
                        QEMU_CMDLINE+=" -append \"root=/dev/sda2 console=ttyS0 pci=realloc,nocrs pci=assign-busses\""
                fi
        QEMU_CMDLINE+=" -object memory-backend-memfd,id=ram1,size=${MEM}G,share=true,prealloc=true"
        QEMU_CMDLINE+=" -machine memory-backend=ram1"
fi
QEMU_CMDLINE+=" -nographic -vga none"
QEMU_CMDLINE+=" -monitor pty -monitor unix:monitor,server,nowait"
if [ -n "${MULTI_GPU}" ]; then
        GPU_PCI_ID=("" "03:00.0" "16:00.0" "1c:00.0" "2e:00.0" "47:00.0" "48:00.0" "49:00.0" "4a:00.0" "84:00.0" "9c:00.0" "b6:00.0" "bb:00.0")
        for i in {1..12}
        do
                QEMU_CMDLINE+=" -device pcie-root-port,id=pci.$i,bus=pcie.0,chassis=$i"
                QEMU_CMDLINE+=" -device vfio-pci,host=${GPU_PCI_ID[$i]},bus=pci.$i"
        done
else
        QEMU_CMDLINE+=" -device pcie-root-port,id=pci.1,bus=pcie.0,chassis=1"
        QEMU_CMDLINE+=" -device vfio-pci,host=03:00.0,bus=pci.1"
fi
echo "qemu cmd: $QEMU_CMDLINE"
sh -c "${QEMU_CMDLINE}" 2>&1 | tee -a ${LOG_DIR}/qemu.log

dmesg output:

[   31.356037] nvidia 0000:01:00.0: enabling device (0000 -> 0002)
[   33.833812] nvidia 0000:02:00.0: enabling device (0000 -> 0002)
[   36.217807] nvidia 0000:03:00.0: enabling device (0000 -> 0002)
[   38.818882] nvidia 0000:04:00.0: enabling device (0000 -> 0002)
[   41.161805] nvidia 0000:09:00.0: enabling device (0000 -> 0002)
[   43.481820] nvidia 0000:0a:00.0: enabling device (0000 -> 0002)
[   46.065811] nvidia 0000:0b:00.0: enabling device (0000 -> 0002)
[   48.618831] nvidia 0000:0c:00.0: enabling device (0000 -> 0002)
[   51.127814] NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64  580.95.05  Release Build  (dvs-builder@U22-I3-B17-02-5)  Tue Sep 23 09:55:41 UTC 2025
[   51.132332] nvidia-nvswitch0: open (major=240)
[   51.132586] nvidia-nvswitch1: open (major=240)
[   51.132861] nvidia-nvswitch2: open (major=240)
[   51.133045] nvidia-nvswitch3: open (major=240)
[   51.138009] nvidia-nvlink: nvlink driver open
[   51.138014] nvidia-nvlink: nvlink driver close
[   51.138017] nvidia-nvlink: nvlink driver open
[   51.138185] nvidia-nvswitch0: open (major=240)
[   51.138190] nvidia-nvswitch0: open (major=240)
[   51.138192] nvidia-nvswitch0: open (major=240)
[   51.138194] nvidia-nvswitch0: open (major=240)
[   51.138196] nvidia-nvswitch0: open (major=240)
[   51.138199] nvidia-nvswitch1: open (major=240)
[   51.138201] nvidia-nvswitch1: open (major=240)
[   51.138203] nvidia-nvswitch1: open (major=240)
[   51.138205] nvidia-nvswitch1: open (major=240)
[   51.138208] nvidia-nvswitch1: open (major=240)
[   51.138211] nvidia-nvswitch2: open (major=240)
[   51.138213] nvidia-nvswitch2: open (major=240)
[   51.138215] nvidia-nvswitch2: open (major=240)
[   51.138235] nvidia-nvswitch2: open (major=240)
[   51.138237] nvidia-nvswitch2: open (major=240)
[   51.138239] nvidia-nvswitch3: open (major=240)
[   51.138241] nvidia-nvswitch3: open (major=240)
[   51.138243] nvidia-nvswitch3: open (major=240)
[   51.138246] nvidia-nvswitch3: open (major=240)
[   51.138248] nvidia-nvswitch3: open (major=240)
[   51.180751] nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64  580.95.05  Release Build  (dvs-builder@U22-I3-B17-02-5)  Tue Sep 23 09:42:01 UTC 2025
[   51.240262] ACPI Warning: \_SB.PCI0.S18.S00._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20230628/nsarguments-61)
[   53.994835] ACPI Warning: \_SB.PCI0.S20.S00._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20230628/nsarguments-61)
[   56.677962] [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
[   56.776636] ACPI Warning: \_SB.PCI0.S28.S00._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20230628/nsarguments-61)
[   59.573637] ACPI Warning: \_SB.PCI0.S30.S00._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20230628/nsarguments-61)
[   62.307918] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:01:00.0 on minor 0
[   62.307937] nvidia 0000:01:00.0: [drm] No compatible format found
[   62.307940] nvidia 0000:01:00.0: [drm] Cannot find any crtc or sizes
[   62.307977] [drm] [nvidia-drm] [GPU ID 0x00000200] Loading driver
[   62.383695] ACPI Warning: \_SB.PCI0.S58.S00._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20230628/nsarguments-61)
[   65.188756] ACPI Warning: \_SB.PCI0.S60.S00._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20230628/nsarguments-61)
[   69.587765] NVRM: gpuClearFbhubPoisonIntrForBug2924523_GA100: FBHUB Interrupt detected. Clearing it.
[   69.588332] ACPI Warning: \_SB.PCI0.S68.S00._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20230628/nsarguments-61)
[   72.368332] NVRM: gpuClearFbhubPoisonIntrForBug2924523_GA100: FBHUB Interrupt detected. Clearing it.
[   72.368913] ACPI Warning: \_SB.PCI0.S70.S00._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20230628/nsarguments-61)
[   75.588764] NVRM: kgspCheckGspRmCcCleanup_GH100: CC secret cleanup successful!
[   76.006763] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:02:00.0 on minor 1
[   76.006776] nvidia 0000:02:00.0: [drm] No compatible format found
[   76.006780] nvidia 0000:02:00.0: [drm] Cannot find any crtc or sizes
[   76.006831] [drm] [nvidia-drm] [GPU ID 0x00000300] Loading driver
[   76.382932] NVRM: kgspCheckGspRmCcCleanup_GH100: CC secret cleanup successful!
[   77.173214] NVRM: kgspCheckGspRmCcCleanup_GH100: CC secret cleanup successful!
[   77.653649] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:03:00.0 on minor 2
[   77.653676] nvidia 0000:03:00.0: [drm] No compatible format found
[   77.653680] nvidia 0000:03:00.0: [drm] Cannot find any crtc or sizes
[   77.653721] [drm] [nvidia-drm] [GPU ID 0x00000400] Loading driver
[   78.057580] NVRM: kgspCheckGspRmCcCleanup_GH100: CC secret cleanup successful!
[   78.507070] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:04:00.0 on minor 3
[   78.507084] nvidia 0000:04:00.0: [drm] No compatible format found
[   78.507087] nvidia 0000:04:00.0: [drm] Cannot find any crtc or sizes
[   78.507121] [drm] [nvidia-drm] [GPU ID 0x00000900] Loading driver
[   78.566046] NVRM: gpumgrCheckRmFirmwarePolicy: Disabling GSP offload -- GPU not supported
[   78.566071] NVRM: nvAssertFailedNoLog: Assertion failed: pVGpu != NULL @ objvgpu.c:148
[   78.566099] NVRM: osInitNvMapping: *** Cannot attach gpu
[   78.566118] NVRM: RmInitAdapter: osInitNvMapping failed, bailing out of RmInitAdapter
[   78.566147] NVRM: GPU 0000:09:00.0: RmInitAdapter failed! (0x22:0x56:763)
[   78.567796] NVRM: GPU 0000:09:00.0: rm_init_adapter failed, device minor number 4
[   78.607338] [drm:nv_drm_dev_load [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000900] Failed to allocate NvKmsKapiDevice
[   78.608259] [drm] [nvidia-drm] [GPU ID 0x00000a00] Loading driver
[   78.677499] NVRM: gpumgrCheckRmFirmwarePolicy: Disabling GSP offload -- GPU not supported
[   78.677526] NVRM: nvAssertFailedNoLog: Assertion failed: pVGpu != NULL @ objvgpu.c:148
[   78.677552] NVRM: osInitNvMapping: *** Cannot attach gpu
[   78.677571] NVRM: RmInitAdapter: osInitNvMapping failed, bailing out of RmInitAdapter
[   78.677599] NVRM: GPU 0000:0a:00.0: RmInitAdapter failed! (0x22:0x56:763)
[   78.679126] NVRM: GPU 0000:0a:00.0: rm_init_adapter failed, device minor number 5
[   78.704316] [drm:nv_drm_dev_load [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000a00] Failed to allocate NvKmsKapiDevice
[   78.705255] [drm] [nvidia-drm] [GPU ID 0x00000b00] Loading driver
[   78.778935] NVRM: gpumgrCheckRmFirmwarePolicy: Disabling GSP offload -- GPU not supported
[   78.778960] NVRM: nvAssertFailedNoLog: Assertion failed: pVGpu != NULL @ objvgpu.c:148
[   78.778989] NVRM: osInitNvMapping: *** Cannot attach gpu
[   78.779008] NVRM: RmInitAdapter: osInitNvMapping failed, bailing out of RmInitAdapter
[   78.779067] NVRM: GPU 0000:0b:00.0: RmInitAdapter failed! (0x22:0x56:763)
[   78.780591] NVRM: GPU 0000:0b:00.0: rm_init_adapter failed, device minor number 6
[   78.812416] [drm:nv_drm_dev_load [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000b00] Failed to allocate NvKmsKapiDevice
[   78.813349] [drm] [nvidia-drm] [GPU ID 0x00000c00] Loading driver
[   78.872103] NVRM: gpumgrCheckRmFirmwarePolicy: Disabling GSP offload -- GPU not supported
[   78.872130] NVRM: nvAssertFailedNoLog: Assertion failed: pVGpu != NULL @ objvgpu.c:148
[   78.872157] NVRM: osInitNvMapping: *** Cannot attach gpu
[   78.872175] NVRM: RmInitAdapter: osInitNvMapping failed, bailing out of RmInitAdapter
[   78.872210] NVRM: GPU 0000:0c:00.0: RmInitAdapter failed! (0x22:0x56:763)
[   78.873734] NVRM: GPU 0000:0c:00.0: rm_init_adapter failed, device minor number 7
[   78.912314] [drm:nv_drm_dev_load [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000c00] Failed to allocate NvKmsKapiDevice
[   79.054523] NVRM: gpumgrCheckRmFirmwarePolicy: Disabling GSP offload -- GPU not supported
[   79.054548] NVRM: nvAssertFailedNoLog: Assertion failed: pVGpu != NULL @ objvgpu.c:148
[   79.054574] NVRM: osInitNvMapping: *** Cannot attach gpu

nvidia-smi and nvidia-smi conf-compute -q output:

Mon Jan  5 15:29:21 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H20                     On  |   00000000:01:00.0 Off |                    0 |
| N/A   35C    P0             74W /  500W |       0MiB /  97871MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA H20                     On  |   00000000:02:00.0 Off |                    0 |
| N/A   34C    P0             75W /  500W |       0MiB /  97871MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA H20                     On  |   00000000:03:00.0 Off |                    0 |
| N/A   30C    P0             73W /  500W |       0MiB /  97871MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA H20                     On  |   00000000:04:00.0 Off |                    0 |
| N/A   30C    P0             71W /  500W |       0MiB /  97871MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

==============NVSMI CONF-COMPUTE LOG==============

    CC State                   : OFF
    Multi-GPU Mode             : Protected PCIe
    CPU CC Capabilities        : AMD SEV-SNP
    GPU CC Capabilities        : CC Capable
    CC GPUs Ready State        : Ready

5.lspci -nnk -v -d 10de: output:

01:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2329] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:198b]
        Physical Slot: 0
        Flags: bus master, fast devsel, latency 0, IRQ 23
        Memory at 1002000000 (64-bit, prefetchable) [size=16M]
        Memory at 2000000000 (64-bit, prefetchable) [size=128G]
        Memory at 1000000000 (64-bit, prefetchable) [size=32M]
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable- Count=1/16 Maskable+ 64bit+
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [9c] Vendor Specific Information: Len=14 <?>
        Capabilities: [b0] MSI-X: Enable+ Count=9 Masked-
        Capabilities: [100] Null
        Capabilities: [12c] Latency Tolerance Reporting
        Capabilities: [134] Physical Resizable BAR
        Capabilities: [1b8] Advanced Error Reporting
        Capabilities: [2a4] Vendor Specific Information: ID=0001 Rev=1 Len=014 <?>
        Capabilities: [2b8] Power Budgeting <?>
        Capabilities: [2f0] Device Serial Number 2e-ff-31-94-70-2d-b0-48
        Kernel driver in use: nvidia
        Kernel modules: nvidia_drm, nvidia

02:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2329] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:198b]
        Physical Slot: 0-2
        Flags: bus master, fast devsel, latency 0, IRQ 20
        Memory at 6002000000 (64-bit, prefetchable) [size=16M]
        Memory at 4000000000 (64-bit, prefetchable) [size=128G]
        Memory at 6000000000 (64-bit, prefetchable) [size=32M]
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable- Count=1/16 Maskable+ 64bit+
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [9c] Vendor Specific Information: Len=14 <?>
        Capabilities: [b0] MSI-X: Enable+ Count=9 Masked-
        Capabilities: [100] Null
        Capabilities: [12c] Latency Tolerance Reporting
        Capabilities: [134] Physical Resizable BAR
        Capabilities: [1b8] Advanced Error Reporting
        Capabilities: [2a4] Vendor Specific Information: ID=0001 Rev=1 Len=014 <?>
        Capabilities: [2b8] Power Budgeting <?>
        Capabilities: [2f0] Device Serial Number b6-3b-47-a2-c4-2d-b0-48
        Kernel driver in use: nvidia
        Kernel modules: nvidia_drm, nvidia

03:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2329] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:198b]
        Physical Slot: 0-3
        Flags: bus master, fast devsel, latency 0, IRQ 21
        Memory at 7002000000 (64-bit, prefetchable) [size=16M]
        Memory at 8000000000 (64-bit, prefetchable) [size=128G]
        Memory at 7000000000 (64-bit, prefetchable) [size=32M]
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable- Count=1/16 Maskable+ 64bit+
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [9c] Vendor Specific Information: Len=14 <?>
        Capabilities: [b0] MSI-X: Enable+ Count=9 Masked-
        Capabilities: [100] Null
        Capabilities: [12c] Latency Tolerance Reporting
        Capabilities: [134] Physical Resizable BAR
        Capabilities: [1b8] Advanced Error Reporting
        Capabilities: [2a4] Vendor Specific Information: ID=0001 Rev=1 Len=014 <?>
        Capabilities: [2b8] Power Budgeting <?>
        Capabilities: [2f0] Device Serial Number 72-4b-35-2a-02-2d-b0-48
        Kernel driver in use: nvidia
        Kernel modules: nvidia_drm, nvidia

04:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2329] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:198b]
        Physical Slot: 0-4
        Flags: bus master, fast devsel, latency 0, IRQ 22
        Memory at 12002000000 (64-bit, prefetchable) [size=16M]
        Memory at 10000000000 (64-bit, prefetchable) [size=128G]
        Memory at 12000000000 (64-bit, prefetchable) [size=32M]
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable- Count=1/16 Maskable+ 64bit+
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [9c] Vendor Specific Information: Len=14 <?>
        Capabilities: [b0] MSI-X: Enable+ Count=9 Masked-
        Capabilities: [100] Null
        Capabilities: [12c] Latency Tolerance Reporting
        Capabilities: [134] Physical Resizable BAR
        Capabilities: [1b8] Advanced Error Reporting
        Capabilities: [2a4] Vendor Specific Information: ID=0001 Rev=1 Len=014 <?>
        Capabilities: [2b8] Power Budgeting <?>
        Capabilities: [2f0] Device Serial Number 8d-90-d6-b4-22-2d-b0-48
        Kernel driver in use: nvidia
        Kernel modules: nvidia_drm, nvidia

05:00.0 Bridge [0680]: NVIDIA Corporation Device [10de:22a3] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:1796]
        Physical Slot: 0-5
        Flags: bus master, fast devsel, latency 0, IRQ 59
        Memory at 86000000 (64-bit, non-prefetchable) [size=32M]
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable+ Count=1/1 Maskable+ 64bit+
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [9c] Vendor Specific Information: Len=14 <?>
        Capabilities: [100] Null
        Capabilities: [1b8] Advanced Error Reporting
        Capabilities: [2a4] Vendor Specific Information: ID=0001 Rev=1 Len=014 <?>
        Capabilities: [2f0] Device Serial Number 2e-9f-9d-4e-b0-2d-b0-48
        Kernel driver in use: nvidia-nvswitch
        Kernel modules: nvidia_drm, nvidia

06:00.0 Bridge [0680]: NVIDIA Corporation Device [10de:22a3] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:1796]
        Physical Slot: 0-6
        Flags: bus master, fast devsel, latency 0, IRQ 60
        Memory at 84000000 (64-bit, non-prefetchable) [size=32M]
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable+ Count=1/1 Maskable+ 64bit+
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [9c] Vendor Specific Information: Len=14 <?>
        Capabilities: [100] Null
        Capabilities: [1b8] Advanced Error Reporting
        Capabilities: [2a4] Vendor Specific Information: ID=0001 Rev=1 Len=014 <?>
        Capabilities: [2f0] Device Serial Number a4-5c-9a-21-10-2d-b0-48
        Kernel driver in use: nvidia-nvswitch
        Kernel modules: nvidia_drm, nvidia

07:00.0 Bridge [0680]: NVIDIA Corporation Device [10de:22a3] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:1796]
        Physical Slot: 0-7
        Flags: bus master, fast devsel, latency 0, IRQ 61
        Memory at 82000000 (64-bit, non-prefetchable) [size=32M]
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable+ Count=1/1 Maskable+ 64bit+
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [9c] Vendor Specific Information: Len=14 <?>
        Capabilities: [100] Null
        Capabilities: [1b8] Advanced Error Reporting
        Capabilities: [2a4] Vendor Specific Information: ID=0001 Rev=1 Len=014 <?>
        Capabilities: [2f0] Device Serial Number 46-3e-ae-50-80-2d-b0-48
        Kernel driver in use: nvidia-nvswitch
        Kernel modules: nvidia_drm, nvidia

08:00.0 Bridge [0680]: NVIDIA Corporation Device [10de:22a3] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:1796]
        Physical Slot: 0-8
        Flags: bus master, fast devsel, latency 0, IRQ 62
        Memory at 80000000 (64-bit, non-prefetchable) [size=32M]
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable+ Count=1/1 Maskable+ 64bit+
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [9c] Vendor Specific Information: Len=14 <?>
        Capabilities: [100] Null
        Capabilities: [1b8] Advanced Error Reporting
        Capabilities: [2a4] Vendor Specific Information: ID=0001 Rev=1 Len=014 <?>
        Capabilities: [2f0] Device Serial Number 5c-f8-18-14-d4-2d-b0-48
        Kernel driver in use: nvidia-nvswitch
        Kernel modules: nvidia_drm, nvidia

09:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2329] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:198b]
        Physical Slot: 0-9
        Flags: bus master, fast devsel, latency 0, IRQ 23
        Memory at 13002000000 (64-bit, prefetchable) [size=16M]
        Memory at 14000000000 (64-bit, prefetchable) [size=128G]
        Memory at 13000000000 (64-bit, prefetchable) [size=32M]
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable- Count=1/16 Maskable+ 64bit+
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [9c] Vendor Specific Information: Len=14 <?>
        Capabilities: [b0] MSI-X: Enable- Count=9 Masked-
        Capabilities: [100] Null
        Capabilities: [12c] Latency Tolerance Reporting
        Capabilities: [134] Physical Resizable BAR
        Capabilities: [1b8] Advanced Error Reporting
        Capabilities: [2a4] Vendor Specific Information: ID=0001 Rev=1 Len=014 <?>
        Capabilities: [2b8] Power Budgeting <?>
        Capabilities: [2f0] Device Serial Number 9f-21-0f-69-a4-2d-b0-48
        Kernel driver in use: nvidia
        Kernel modules: nvidia_drm, nvidia

0a:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2329] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:198b]
        Physical Slot: 0-10
        Flags: bus master, fast devsel, latency 0, IRQ 20
        Memory at 18002000000 (64-bit, prefetchable) [size=16M]
        Memory at 16000000000 (64-bit, prefetchable) [size=128G]
        Memory at 18000000000 (64-bit, prefetchable) [size=32M]
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable- Count=1/16 Maskable+ 64bit+
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [9c] Vendor Specific Information: Len=14 <?>
        Capabilities: [b0] MSI-X: Enable- Count=9 Masked-
        Capabilities: [100] Null
        Capabilities: [12c] Latency Tolerance Reporting
        Capabilities: [134] Physical Resizable BAR
        Capabilities: [1b8] Advanced Error Reporting
        Capabilities: [2a4] Vendor Specific Information: ID=0001 Rev=1 Len=014 <?>
        Capabilities: [2b8] Power Budgeting <?>
        Capabilities: [2f0] Device Serial Number e3-56-92-5a-7d-2d-b0-48
        Kernel driver in use: nvidia
        Kernel modules: nvidia_drm, nvidia

0b:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2329] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:198b]
        Physical Slot: 0-11
        Flags: bus master, fast devsel, latency 0, IRQ 21
        Memory at 19002000000 (64-bit, prefetchable) [size=16M]
        Memory at 1a000000000 (64-bit, prefetchable) [size=128G]
        Memory at 19000000000 (64-bit, prefetchable) [size=32M]
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable- Count=1/16 Maskable+ 64bit+
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [9c] Vendor Specific Information: Len=14 <?>
        Capabilities: [b0] MSI-X: Enable- Count=9 Masked-
        Capabilities: [100] Null
        Capabilities: [12c] Latency Tolerance Reporting
        Capabilities: [134] Physical Resizable BAR
        Capabilities: [1b8] Advanced Error Reporting
        Capabilities: [2a4] Vendor Specific Information: ID=0001 Rev=1 Len=014 <?>
        Capabilities: [2b8] Power Budgeting <?>
        Capabilities: [2f0] Device Serial Number 3b-27-8d-fa-bc-2d-b0-48
        Kernel driver in use: nvidia
        Kernel modules: nvidia_drm, nvidia

0c:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2329] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:198b]
        Physical Slot: 0-12
        Flags: bus master, fast devsel, latency 0, IRQ 22
        Memory at 1e002000000 (64-bit, prefetchable) [size=16M]
        Memory at 1c000000000 (64-bit, prefetchable) [size=128G]
        Memory at 1e000000000 (64-bit, prefetchable) [size=32M]
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable- Count=1/16 Maskable+ 64bit+
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [9c] Vendor Specific Information: Len=14 <?>
        Capabilities: [b0] MSI-X: Enable- Count=9 Masked-
        Capabilities: [100] Null
        Capabilities: [12c] Latency Tolerance Reporting
        Capabilities: [134] Physical Resizable BAR
        Capabilities: [1b8] Advanced Error Reporting
        Capabilities: [2a4] Vendor Specific Information: ID=0001 Rev=1 Len=014 <?>
        Capabilities: [2b8] Power Budgeting <?>
        Capabilities: [2f0] Device Serial Number 73-4d-fa-97-9a-2d-b0-48
        Kernel driver in use: nvidia
        Kernel modules: nvidia_drm, nvidia

sbellock · January 5, 2026, 11:13pm

Please confirm, from the host with GitHub - NVIDIA/gpu-admin-tools: GPU Admin Tools. Includes Confidential Computing controls for H100, and other functionality, that all GPUs are in PPCIe mode.

costinchen · January 6, 2026, 3:58am

Yes, all GPUs are in PPCIe mode

[root@TENCENT64 gpu-admin-tools]# sudo python3 ./nvidia_gpu_tools.py --devices gpus --query-ppcie-mode
NVIDIA GPU Tools version v2025.11.21o
Command line arguments: ['./nvidia_gpu_tools.py', '--devices', 'gpus', '--query-ppcie-mode']
2026-01-06,11:57:29.117 WARNING  GPU 0000:03:00.0 ? 0x2329 BAR0 0x0 was in D0/control:auto, forced power control to on. New state D0
2026-01-06,11:57:29.123 WARNING  GPU 0000:16:00.0 ? 0x2329 BAR0 0x0 was in D0/control:auto, forced power control to on. New state D0
2026-01-06,11:57:29.126 WARNING  GPU 0000:1c:00.0 ? 0x2329 BAR0 0x0 was in D0/control:auto, forced power control to on. New state D0
2026-01-06,11:57:29.129 WARNING  GPU 0000:2e:00.0 ? 0x2329 BAR0 0x0 was in D0/control:auto, forced power control to on. New state D0
2026-01-06,11:57:29.133 WARNING  GPU 0000:84:00.0 ? 0x2329 BAR0 0x0 was in D0/control:auto, forced power control to on. New state D0
2026-01-06,11:57:29.137 WARNING  GPU 0000:9c:00.0 ? 0x2329 BAR0 0x0 was in D0/control:auto, forced power control to on. New state D0
2026-01-06,11:57:29.140 WARNING  GPU 0000:b6:00.0 ? 0x2329 BAR0 0x0 was in D0/control:auto, forced power control to on. New state D0
2026-01-06,11:57:29.145 WARNING  GPU 0000:bb:00.0 ? 0x2329 BAR0 0x0 was in D0/control:auto, forced power control to on. New state D0
Topo:
  PciBridge 0000:00:01.1 14ab:1022
   PciBridge 0000:01:00.0 c030:1000
    PciBridge 0000:02:00.0 c030:1000
     GPU 0000:03:00.0 H20 0x2329 BAR0 0x174042000000
  PciBridge 0000:12:01.1 14ab:1022
   PciBridge 0000:13:00.0 c030:1000
    PciBridge 0000:14:01.0 c030:1000
     GPU 0000:16:00.0 H20 0x2329 BAR0 0x1de042000000
  PciBridge 0000:12:03.1 14a5:1022
   PciBridge 0000:19:00.0 c030:1000
    PciBridge 0000:1a:01.0 c030:1000
     GPU 0000:1c:00.0 H20 0x2329 BAR0 0x1e4042000000
  PciBridge 0000:2b:01.1 14ab:1022
   PciBridge 0000:2c:00.0 c030:1000
    PciBridge 0000:2d:00.0 c030:1000
     GPU 0000:2e:00.0 H20 0x2329 BAR0 0x104042000000
  PciBridge 0000:80:01.1 14ab:1022
   PciBridge 0000:81:00.0 c030:1000
    PciBridge 0000:82:01.0 c030:1000
     GPU 0000:84:00.0 H20 0x2329 BAR0 0x334042000000
  PciBridge 0000:99:01.1 14ab:1022
   PciBridge 0000:9a:00.0 c030:1000
    PciBridge 0000:9b:00.0 c030:1000
     GPU 0000:9c:00.0 H20 0x2329 BAR0 0x3a4042000000
  PciBridge 0000:b2:01.1 14ab:1022
   PciBridge 0000:b3:00.0 c030:1000
    PciBridge 0000:b4:01.0 c030:1000
     GPU 0000:b6:00.0 H20 0x2329 BAR0 0x2c4042000000
  PciBridge 0000:b2:03.1 14a5:1022
   PciBridge 0000:b9:00.0 c030:1000
    PciBridge 0000:ba:00.0 c030:1000
     GPU 0000:bb:00.0 H20 0x2329 BAR0 0x2be042000000
2026-01-06,11:57:29.146 INFO     Selected GPU 0000:03:00.0 H20 0x2329 BAR0 0x174042000000
2026-01-06,11:57:29.146 INFO     Selected GPU 0000:16:00.0 H20 0x2329 BAR0 0x1de042000000
2026-01-06,11:57:29.146 INFO     Selected GPU 0000:1c:00.0 H20 0x2329 BAR0 0x1e4042000000
2026-01-06,11:57:29.146 INFO     Selected GPU 0000:2e:00.0 H20 0x2329 BAR0 0x104042000000
2026-01-06,11:57:29.146 INFO     Selected GPU 0000:84:00.0 H20 0x2329 BAR0 0x334042000000
2026-01-06,11:57:29.146 INFO     Selected GPU 0000:9c:00.0 H20 0x2329 BAR0 0x3a4042000000
2026-01-06,11:57:29.146 INFO     Selected GPU 0000:b6:00.0 H20 0x2329 BAR0 0x2c4042000000
2026-01-06,11:57:29.146 INFO     Selected GPU 0000:bb:00.0 H20 0x2329 BAR0 0x2be042000000
2026-01-06,11:57:29.146 WARNING  GPU 0000:03:00.0 H20 0x2329 BAR0 0x174042000000 has PPCIe mode on, some functionality may not work
2026-01-06,11:57:29.146 INFO     GPU 0000:03:00.0 H20 0x2329 BAR0 0x174042000000 PPCIe mode is on
2026-01-06,11:57:29.146 WARNING  GPU 0000:16:00.0 H20 0x2329 BAR0 0x1de042000000 has PPCIe mode on, some functionality may not work
2026-01-06,11:57:29.146 INFO     GPU 0000:16:00.0 H20 0x2329 BAR0 0x1de042000000 PPCIe mode is on
2026-01-06,11:57:29.146 WARNING  GPU 0000:1c:00.0 H20 0x2329 BAR0 0x1e4042000000 has PPCIe mode on, some functionality may not work
2026-01-06,11:57:29.146 INFO     GPU 0000:1c:00.0 H20 0x2329 BAR0 0x1e4042000000 PPCIe mode is on
2026-01-06,11:57:29.146 WARNING  GPU 0000:2e:00.0 H20 0x2329 BAR0 0x104042000000 has PPCIe mode on, some functionality may not work
2026-01-06,11:57:29.146 INFO     GPU 0000:2e:00.0 H20 0x2329 BAR0 0x104042000000 PPCIe mode is on
2026-01-06,11:57:29.146 WARNING  GPU 0000:84:00.0 H20 0x2329 BAR0 0x334042000000 has PPCIe mode on, some functionality may not work
2026-01-06,11:57:29.146 INFO     GPU 0000:84:00.0 H20 0x2329 BAR0 0x334042000000 PPCIe mode is on
2026-01-06,11:57:29.146 WARNING  GPU 0000:9c:00.0 H20 0x2329 BAR0 0x3a4042000000 has PPCIe mode on, some functionality may not work
2026-01-06,11:57:29.147 INFO     GPU 0000:9c:00.0 H20 0x2329 BAR0 0x3a4042000000 PPCIe mode is on
2026-01-06,11:57:29.147 WARNING  GPU 0000:b6:00.0 H20 0x2329 BAR0 0x2c4042000000 has PPCIe mode on, some functionality may not work
2026-01-06,11:57:29.147 INFO     GPU 0000:b6:00.0 H20 0x2329 BAR0 0x2c4042000000 PPCIe mode is on
2026-01-06,11:57:29.147 WARNING  GPU 0000:bb:00.0 H20 0x2329 BAR0 0x2be042000000 has PPCIe mode on, some functionality may not work
2026-01-06,11:57:29.147 INFO     GPU 0000:bb:00.0 H20 0x2329 BAR0 0x2be042000000 PPCIe mode is on
2026-01-06,11:57:29.147 WARNING  GPU 0000:bb:00.0 H20 0x2329 BAR0 0x2be042000000 restoring power control to auto
2026-01-06,11:57:29.147 WARNING  GPU 0000:b6:00.0 H20 0x2329 BAR0 0x2c4042000000 restoring power control to auto
2026-01-06,11:57:29.147 WARNING  GPU 0000:9c:00.0 H20 0x2329 BAR0 0x3a4042000000 restoring power control to auto
2026-01-06,11:57:29.147 WARNING  GPU 0000:84:00.0 H20 0x2329 BAR0 0x334042000000 restoring power control to auto
2026-01-06,11:57:29.147 WARNING  GPU 0000:2e:00.0 H20 0x2329 BAR0 0x104042000000 restoring power control to auto
2026-01-06,11:57:29.147 WARNING  GPU 0000:1c:00.0 H20 0x2329 BAR0 0x1e4042000000 restoring power control to auto
2026-01-06,11:57:29.147 WARNING  GPU 0000:16:00.0 H20 0x2329 BAR0 0x1de042000000 restoring power control to auto
2026-01-06,11:57:29.147 WARNING  GPU 0000:03:00.0 H20 0x2329 BAR0 0x174042000000 restoring power control to auto

sbellock · January 7, 2026, 12:40am

@costinchen is nvidia-persistenced being run immediately after guest VM initialization?

costinchen · January 7, 2026, 2:45am

Yes, nvidia-persistenced.servicestarted immediately after VM initialization before GPUs initialization.
When I removed SEV-SNP in qemu arguments and set the PPCIe mode to off, all GPUs are successfully initialized in VM.

Topic		Replies	Views
Broken GPU state query failure in AMD + H100 Confidential Computing	9	1474	February 1, 2024
Nvidia-smi reads only 3 GPUs from 4 Linux	7	3064	June 3, 2017
Tesla P4 PCI pass through from RHOSP 13 to RHEL 7.6/Windows VMs issues Linux	15	1997	April 30, 2021
Openstack KVM Ubuntu 22.04 NVIDIA A100 80GB PCIe mdev_supported_types More vGPU Forums	1	568	April 3, 2025
Nvidia driver on Ubuntu 24.04.04 LTS with AMD threadripper 3995WX + Nvidia RTX 3080 Ti fails and gdm-x-session deletes GPU-0 Linux	5	413	March 21, 2026
SOLVED - RmInitAdapter failed! to load 530.41.03 (or any nvidia modules other than 450.236.01) Linux via ESXi 7.0u3 Passthrough PCI GTX 1650 Linux ubuntu , linux , esxi	3	25006	May 24, 2023
Could not initialize plugin 'libnvidia-vgx.so' for vGPU 'nvidia_a16-4q' XenDesktop	16	6813	April 21, 2025
ENOMEM when running CUDA sample on host GPU where another GPU is passed through via IOMMU/vfio-pci Linux	1	894	May 19, 2019
vGPU install stuck General Discussion	2	578	July 16, 2024
Only 7 of 8 GPUs are loaded: Dual GRID M40's on HP Proliant DL580 G7 Ubuntu 16.04.5 Server Linux	2	1191	July 18, 2020

Only 4/8 GPUs can be initialized with AMD SEV-SNP and NVIDIA PPCIe mode in CVM

Related topics