Nvidia 2x H100s passthrough failed with insufficient memory

Hello,

Hi, I am having the issue with 2xH100 VM setup. I am trying to pass through 2x H100s. One H100 is loaded and is working fine, the second H100 fails with the
nvidia-bug-report.log.gz (276.2 KB)
following error messages.

[ 0.902604] pci 0000:02:00.0: [10de:2331] type 00 class 0x030200
[ 0.903470] pci 0000:02:00.0: reg 0x10: [mem 0xffffffffff000000-0xffffffffffffffff 64bit pref]
[ 0.904112] pci 0000:02:00.0: reg 0x18: [mem 0xffffffe000000000-0xffffffffffffffff 64bit pref]
[ 0.905217] pci 0000:02:00.0: reg 0x20: [mem 0xfffffffffe000000-0xffffffffffffffff 64bit pref]
[ 0.906302] pci 0000:02:00.0: Max Payload Size set to 128 (was 256, max 256)
[ 0.908133] pci 0000:02:00.0: Enabling HDA controller
[ 0.909281] pci 0000:02:00.0: 252.048 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x16 link at 0000:00:05.0 (capable of 504.112 Gb/s with 32.0 GT/s PCIe x16 link)
[ 0.912200] pci 0000:00:05.0: PCI bridge to [bus 02]
[ 1.042145] pci 0000:02:00.0: can’t claim BAR 0 [mem 0xffffffffff000000-0xffffffffffffffff 64bit pref]: no compatible bridge window
[ 1.043585] pci 0000:02:00.0: can’t claim BAR 2 [mem 0xffffffe000000000-0xffffffffffffffff 64bit pref]: no compatible bridge window
[ 1.044062] pci 0000:02:00.0: can’t claim BAR 4 [mem 0xfffffffffe000000-0xffffffffffffffff 64bit pref]: no compatible bridge window

[ 1.131465] pci 0000:00:05.0: BAR 15: no space for [mem size 0x3000000000 64bit pref]
[ 1.132385] pci 0000:00:05.0: BAR 15: failed to assign [mem size 0x3000000000 64bit pref]
[ 1.133335] pci 0000:00:04.0: PCI bridge to [bus 01]
[ 3.571329] pci 0000:00:04.0: bridge window [mem 0xc000000000-0xe002ffffff 64bit pref]
[ 6.729684] pci 0000:02:00.0: BAR 2: no space for [mem size 0x2000000000 64bit pref]
[ 6.731789] pci 0000:02:00.0: BAR 2: failed to assign [mem 0xffffffe000000000-0xffffffffffffffff 64bit pref]
[ 6.734253] pci 0000:02:00.0: BAR 4: no space for [mem size 0x02000000 64bit pref]
[ 6.736138] pci 0000:02:00.0: BAR 4: failed to assign [mem 0xfffffffffe000000-0xffffffffffffffff 64bit pref]
[ 6.738573] pci 0000:02:00.0: BAR 0: no space for [mem size 0x01000000 64bit pref]
[ 6.740466] pci 0000:02:00.0: BAR 0: failed to assign [mem 0xffffffffff000000-0xffffffffffffffff 64bit pref]
[ 6.742904] pci 0000:00:05.0: PCI bridge to [bus 02]
[ 6.748646] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]
[ 6.751044] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window]
[ 6.752599] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[ 6.754296] pci_bus 0000:00: resource 7 [mem 0x80000000-0xafffffff window]
[ 6.756001] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window]
[ 6.757720] pci_bus 0000:00: resource 9 [mem 0xc000000000-0xe003007fff window]
[ 6.759518] pci_bus 0000:01: resource 2 [mem 0xc000000000-0xe002ffffff 64bit pref]

attached is the bug report