Pass-through GPU-CC with pcie-root-port

Accelerated Computing Confidential Computing
1

Based on the cc deployment guide, when we launch a CVM, if we pass through a GPU connected to a pcie-root-port, everything works well.

-device pcie-root-port,id=pci.1,bus=pcie.0
-device vfio-pci,host=$NVIDIA_GPU,bus=pci.1

However, if we just pass through a single GPU

-device vfio-pci,host=$NVIDIA_GPU

In the dmesg log, it reports that

NVRM: getPCIELinkRateMBps: Unknown PCIe speed

It seems the rmapi control command NV2080_CTRL_CMD_BUS_GET_INFO does not populate the busInfo. I have also tested in non-CC mode and this configuration works fine. Could you explain the reason for passing through the GPU with a pcie-root-port? Thanks.

2

If -device pcie-root-port,id=pci.1,bus=pcie.0 is not added, could you run cuda applications smoothly? Or is there any performance loss?

By the way, could you share the Nvidia driver version you are using?

3

The card is available in the nvidia-smi. But I cannot run the CUDA application. It will report NVRM: getPCIELinkRateMBps: Unknown PCIe speed
The driver version is 550.54.15

4

Thank you for pointing out. I did not notice that before.

Before experts from Nvidia reply to this question, I suggest hacking into the function getPCIELinkRateMBps that reports this error. For example, remove the error in this function and hard-code linkRate.

5

I am wondering if this is designed due to CC requirement. The link rate is read through the rmapi interface.