Updated DOCA 2.7 to 3.0 now I have Broadcom / LSI Virtual PCIe Placeholder Endpoint

Infrastructure & Networking BlueField
1

Note. Created this issue here because the NVIDIA forum does not allow me to create Topic in other forum threads!!! This should really be under DOCA or Network Adapter threads.

I have a server that runs RHEL8.8 on two numa nodes, with no OS virtualization / Virtual Machines / No docker containers used in my system.
When I run the following command:

sudo lspci -t -vvv | grep -EB1 "NVIDIA|Mellanox|Switch"

 |                                           +-01.0-[c0]----00.0  Broadcom / LSI Virtual PCIe Placeholder Endpoint
 |                                           +-02.0-[c1]--+-00.0  Mellanox Technologies MT2910 Family [ConnectX-7]
 |                                           |            \-00.1  Mellanox Technologies MT2910 Family [ConnectX-7]
 |                                           +-03.0-[c2]----00.0  NVIDIA Corporation Device 26b9
 |                                           +-04.0-[c3]----00.0  NVIDIA Corporation Device 26b9
 |                                           \-1f.0-[c4]----00.0  Broadcom / LSI PCIe Switch management endpoint
--
 |                                           +-02.0-[af]----00.0  Broadcom / LSI Virtual PCIe Placeholder Endpoint
 |                                           +-03.0-[b0]----00.0  NVIDIA Corporation Device 26b9
 |                                           \-04.0-[b1]----00.0  NVIDIA Corporation Device 26b9
--
 |           +-00.4  Intel Corporation Device 0b23
 |           \-01.0-[99]--+-00.0  Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
 |                        \-00.1  Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
--
 |           +-00.4  Intel Corporation Device 0b23
 |           \-01.0-[3d-44]----00.0-[3e-44]--+-00.0-[3f]----00.0  NVIDIA Corporation Device 26b9
 |                                           +-01.0-[40]----00.0  NVIDIA Corporation Device 26b9
 |                                           +-02.0-[41]----00.0  NVIDIA Corporation Device 26b9
--
 |                                           +-04.0-[43]----00.0  Broadcom / LSI Virtual PCIe Placeholder Endpoint
 |                                           \-1f.0-[44]----00.0  Broadcom / LSI PCIe Switch management endpoint
--
 |                                           +-01.0-[2d]----00.0  Broadcom / LSI Virtual PCIe Placeholder Endpoint
 |                                           +-02.0-[2e]--+-00.0  Mellanox Technologies MT2910 Family [ConnectX-7]
 |                                           |            \-00.1  Mellanox Technologies MT2910 Family [ConnectX-7]
 |                                           +-03.0-[2f]----00.0  Broadcom / LSI Virtual PCIe Placeholder Endpoint
 |                                           \-04.0-[30]----00.0  NVIDIA Corporation Device 26b9

The server previously had DOCA 2.7 installed and after updating to DOCA 3.0, It we see the “Broadcom / LSI Virtual PCIe Placeholder Endpoint”

cat /proc/cmdline contains iommu=off intel_iommu=off

SR-IOV should be disabled on the system.
The above server has major performance issues when crossing numa node
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
D\D 0 1 2 3 4 5 6 7
0 652.13 26.40 26.40 26.40 3.73 3.55 3.59 3.46 <<<<<<<<<<<<<<<<<<<
1 26.40 676.11 26.40 26.40 22.82 22.82 22.81 22.82
2 26.40 26.40 674.07 26.40 22.82 22.82 22.81 22.81
3 26.40 26.40 26.40 677.58 22.82 22.81 22.81 22.81
4 22.81 22.82 22.81 22.82 675.24 26.40 26.40 26.40
5 22.81 22.81 22.82 22.82 26.40 676.70 26.40 26.40
6 22.81 22.81 22.81 22.82 26.40 26.40 677.87 26.40
7 22.82 22.82 22.81 22.81 26.40 26.40 26.40 677.58

Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
D\D 0 1 2 3 4 5 6 7
0 652.26 52.18 52.17 52.18 7.16 7.59 7.00 7.44 <<<<<<<<<<<<<<<<<<<
1 52.18 647.92 52.20 52.20 43.92 43.92 43.92 43.92
2 52.19 52.18 651.18 52.19 43.91 43.91 43.91 43.92
3 52.19 52.19 52.19 653.63 43.93 43.93 43.91 43.93
4 7.58 43.92 43.91 43.93 650.09 52.19 52.19 52.19
5 7.37 43.92 43.91 43.93 52.18 651.45 52.19 52.19
6 7.31 43.92 43.92 43.92 52.19 52.18 650.77 52.20
7 6.85 43.92 43.91 43.93 52.20 52.20 52.18 650.91

Comparing the above server to another server, we don’t see this issue!
I have another server with the same hardware, RHEl8.8 & DOCA 3.0 software configuration ( at least we believe it’s the same software config).

sudo lspci -t -vvv | grep -EB1 "NVIDIA|Mellanox|Switch"
 |           +-00.4  Intel Corporation Device 0b23
 |           \-01.0-[bd-c4]----00.0-[be-c4]--+-00.0-[bf]--+-00.0  Mellanox Technologies MT2910 Family [ConnectX-7]
 |                                           |            \-00.1  Mellanox Technologies MT2910 Family [ConnectX-7]
 |                                           +-01.0-[c0]----00.0  NVIDIA Corporation Device 26b9
 |                                           +-02.0-[c1]----00.0  NVIDIA Corporation Device 26b9
--
 |                                           +-04.0-[c3]--
 |                                           \-1f.0-[c4]----00.0  Broadcom / LSI PCIe Switch management endpoint
--
 |           +-00.4  Intel Corporation Device 0b23
 |           \-01.0-[ab-b1]----00.0-[ac-b1]--+-00.0-[ad]----00.0  NVIDIA Corporation Device 26b9
 |                                           +-01.0-[ae]----00.0  NVIDIA Corporation Device 26b9
--
 |           +-00.4  Intel Corporation Device 0b23
 |           \-01.0-[99]--+-00.0  Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
 |                        \-00.1  Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
--
 |           +-00.4  Intel Corporation Device 0b23
 |           \-01.0-[43-4a]----00.0-[44-4a]--+-00.0-[45]----00.0  NVIDIA Corporation Device 26b9
 |                                           +-01.0-[46]----00.0  NVIDIA Corporation Device 26b9
--
 |                                           +-04.0-[49]--
 |                                           \-1f.0-[4a]----00.0  Broadcom / LSI PCIe Switch management endpoint
--
 |           +-00.4  Intel Corporation Device 0b23
 |           \-01.0-[1e-24]----00.0-[1f-24]--+-00.0-[20]----00.0  NVIDIA Corporation Device 26b9
 |                                           +-01.0-[21]----00.0  NVIDIA Corporation Device 26b9
 |                                           +-02.0-[22]--+-00.0  Mellanox Technologies MT2910 Family [ConnectX-7]
 |                                           |            \-00.1  Mellanox Technologies MT2910 Family [ConnectX-7]

My questions are as follows:
Q1. Using DOCA 3.0 with ConnectX-7, and DOCA 3.0 is Single Root IO Virtualization (SR-IOV) required for ALL use cases (doca profile -all), even when the host OS does has no virtualization/ no VMs ?

Q2. Similar.. When using DOCA 3.0 with ConnectX-7, without the use of virtualization/VMs should the mlxconfig -d <device_name> set SRIOV_EN=0 NUM_OF_VFS=0 ?

Q3. For system with No virtualization/No VMs , what performance impact, if any, will occur when SR-IOV is disabled ?

Q4. If SR-IOV is not required, then how do I make sure “Broadcom / LSI Virtual PCIe Placeholder Endpoint” is disabled and does not exist?

Q5. Can you please provide a document with steps to cleanly uninstall any previous versions of DOCA (Specifically in my case I’ve transitioned from DOCA2.7 to 3.0)

2

Hi @dan.hartman ,

For Q1: It depends on the specific applicaiton.

For Q2/Q3: I don’t think it’s necessary. As with Q1, it depends on which functions or features you are using. However if you don’t need SR-IOV, disabling it would be a good choice.

For Q4:

  • First of all, make sure you have the same pci-ids version on the two machines, then the PCI device comparation with command “lspci” would be more reliable.
  • If you want to disable a pci device, you can try this command: “echo 1 > /sys/bus/pci/devices/xxxx:xx:xx.x/remove” or use udev rules.

For Q5: Please refer to DOCA-Host Installation and Upgrade - NVIDIA Docs .

3

Thank you for you replies.

Q1: follow-up. My specific application involves using ConnectX-7 with DPDK & DOCA to forward receive UDP packets direct to NVIDIA L40S GPU. The application is similar to the GPUPacketPRocessing sample application. So given this application, would you expect SR-IOV to be disabled?

Q2/Q3: Sounds good.

Q4. The server4 with the “Broadcom / LSI Virtual PCIe Placeholder Endpoint” is currently down for maintenance. I’m trying to get a copy of the /usr/share/hwdata/pci.ids from server 4 to properly respond to this NVIDIA forum post.

Once I have the file I can compare against the other machines and share the information..

Q5. Thank you for providing the link to the documentation. Even though the uninstall instructions remove previous versions of software, I’m concerned the uninstall does not undo or restore all settings such as the state of configuration files, driver s software persistent settings (kernel boot cmdline params such as hugepages), driver settings (NVIDIA smem) , hardware persistent settings (ie mlxconfig settings), etc. Will the installation of DOCA3.0 ensure these settings are configured correctly ?

4

Q4 Update. The pci.ids is identical between the servers. I’ve attached a zip file of the pci.ids from server4.

pci.zip (305.8 KB)

Q5 (received following email reply from NVIDIA with following): Updating to DOCA 3.0 does not necessarily restore settings or uninstalling 2.7 but you can use instructions on this page to reset all settings after uninstalling/installing.

Q6 (New) What specific system configuration item(s) (hardware, software, configuration setting, etc) causes the “Broadcom / LSI Virtual PCIe Placeholder Endpoint” to appear in lspci as shown in the tree ?

Q7 (New) Do you think server4 p2p slow performance as indicated by the <<<<<<<<< comments above, are caused by the nodes “Broadcom / LSI Virtual PCIe Placeholder Endpoint” nodes ? Any other tips for diagnosing the slowdown?

5

Hi @dan.hartman ,

Hope you well.

For Q1: If you don’t need SR-IOV, disabling it would be a good choice.

For Q5: I think the best way for updating a module/system is: (1) backup your important datas and configurations before update; (2) check your datas and configurations one by one to confirm the update. No mather whether the programs can restore the configurations or not.

For Q6/Q7: It’s about the baseboard design or layout of the machines. Simply disabling the device via command may not necessarily improve performance. I think a more efficient way is to do more tests to find the key bottlenecks of the machine/system, or to contact the manufacturer of the machine to find out the differences of the machines.

Best Regards.