How to setup ConnectX5 and ConnectX6DX for mixed(TCP + UDP) traffic and PCP mode performance?

Infrastructure & Networking Software And Drivers Mellanox OFED
1

Hi Experts and NVIDIA technical supporter,

I would like to contact you with the following two inquiries regarding performance tuning settings for ConnectX5 and ConnectX6.

  1. Inquiry regarding CQE_COMPRESSION
  • Recently, there was a problem with the ConnectX5 card that when receiving TCP and UDP short bytes mixed traffic, performance was lower than when receiving only TCP or UDP single traffic.
  • NVIDIA advised as follows :

For mixed traffic, there can be an impact with how HW completions are managed as CQE zipped sessions will be broken before all possible CQEs are merged into a single block,
The reason is the title CQE (think of it as a group unique attribute, like only IPV4 + UDP) is changed,
for example, mixing IPV4 UDP + IPV4 TCP traffic together on the same queue will break the CQE zip session.
setting :
mcra /dev/mst/mt4119_pciconf0 0x815e0.0 0x00000000
mcra /dev/mst/mt4119_pciconf0 0x81600.0 0x00000000

Question1 : Can these settings also apply to the ConnectX6 card? so please confirm whether the same settings can be applied as shown below.
In addition, I changed it by getting information(“0x815e8.31 0” and “0x81608.31 0”) be found on the following site(How to find the maximum number of RX Queues for a NIC (ConnectX-5)? - #4 by wichtounet), but I don’t know for the registers how to affects performance .
Please explain what the register settings are changing.

#mcra setting (CQE_COMPRESSION ratio off)
mst start
#CNX5
mcra /dev/mst/mt4119_pciconf0 0x815e0.0 0x00000000
mcra /dev/mst/mt4119_pciconf0 0x81600.0 0x00000000
mcra /dev/mst/mt4119_pciconf0 0x815e8.31 0
mcra /dev/mst/mt4119_pciconf0 0x81608.31 0

#CNX6
mcra /dev/mst/mt4125_pciconf0 0x815e0.0 0x00000000
mcra /dev/mst/mt4125_pciconf0 0x81600.0 0x00000000
mcra /dev/mst/mt4125_pciconf0 0x815e8.31 0
mcra /dev/mst/mt4125_pciconf0 0x81608.31 0

  1. Inquiry regarding PCP → DSCP mode change
  • For the ConnectX5 card, we previously recommended the following settings due to performance degradation issues when mixed traffic inflows with different L2 VLAN header Priority field values.

Question2 : Please confirm whether this also applies to the ConnectX6 card.

mlnx_qos -i eth2 --trust=dscp
for dscp in {0…63}; do mlnx_qos -i eth2 --dscp2prio set,$dscp,0; usleep 100; done

2

Hi ,

Please open a support ticket to enterprisesupport@nvidia.com
Once you open the ticket , make sure to provide sysinfo-snapshot from the server and ib_write_bw / ib_read_bw test command line outputs with doca_dma_copy test results .

Thanks
Samer