Max number of queue pairs on Connectx-{5,6,7,8}

Infrastructure & Networking Adapters and Cables Ethernet Adapter Cards
1

Hello,

I failed to find a document stating the maximum number of RC queue pairs on ConnectX cards.

Does someone either know of such document?

I was trying to do little scaling tests and both Cx5 and Cx7 couldn’t complete test when I hit 8192 RC queue pairs per process, 8191 was fine.

Thanks,

–
Vesa

2

Hello,

A minor update / additional information:

After some more poking around it would seem that on both CX-5 and CX-7 cards I can get away with 103 ib_send_bw (v 6.17) processes, each with 1266 queue pairs, or 211 processes with 618 queue pairs. In both cases the sum of QPs is 130398.

Aiming 130400 for fails as it seems to go above the limit and 130399 cannot be reasonably tested due to the unknown limitation of 8191 per process and 130399 being a prime. ib_send_bw claims it would accept upto 16384 queue pairs per process.

Would it then be reasonable to assume the max number of queue pairs on this config is indeed 130398.

CX7 cards are running FW 16.31.1014, mlx5_core 23.10-3.2.2 on AlmaLinux 9.4

Any ideas and help is greatly appreciated.

Thanks,

–
Vesa

3

Hi Vesa,
There is no limitation in the NIC itself. The limitation comes from the driver.
You can review on your system what it is with the following command. On my setup it is 128K. Keep in mind some QPs are used internally by the HW so what you observe makes sense.
ibv_devinfo -d mlx5_1 -v | grep qp
max_qp: 131072
…
You can review the driver limitation in the following path:

Regards,
Yaniv

4

Thank you. This is very useful information indeed.

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.