【ConnectX-7】ib_write_bw bandwidth decrease periodly

Infrastructure & Networking Adapters and Cables Ethernet Adapter Cards
1

Hello ,

When I do ib_write_bw test with ConnectX-7, found the bandwidth decrease periodly.

ib_write_bw
ib_write_bw481×295 57.6 KB

After debug, I found that ConnectX-7 [RoCE v2] serdes frequency is greater than Ethernet Switch frequency。

Problem occurrence process：

  1. The sending speed of the Packet from the sender is too fast
  2. As time accumulates, the buffer of the ethernet switch becomes full, and congestion occurs on the sending side of the switch, then ethernet switch marking the packet ECN bit to 1.
  3. After receiving the Packet marked with ECN, the reciever connectx-7 rdma nic sends the CNP Packet.
  4. After the sender rdma nic receives the CNP Packet, the DCQCN algorithm is enabled to reduce the rate.
  5. After the rate is reduced, the congestion on the switch is relieved. The Packet is no longer marked with ECN, and the receiving end no longer sends CNP
  6. For the DCQCN algorithm of the network card at the sender, if no CNP is received within a certain period of time, speed recovery is carried out to increase the rate.

From Ethernet Switch I can found RXPPM to prove that ConnectX-7 freqence is greater than Ethernet Switch.

image
image865×843 145 KB

image
image833×130 90.2 KB

Besides, I found that if set limit rate to 399Gb/s by mlnx_qos, this issue won’t happen .

$ sudo mlnx_qos -i ens21np0 -r 0,0,0,0,0,399,0,0
DCBX mode: OS controlled
Priority trust state: dscp
dscp2prio mapping:
        prio:0 dscp:07,06,05,04,03,02,01,00,
        prio:1 dscp:15,14,13,12,11,10,09,08,
        prio:2 dscp:23,22,21,20,19,18,17,16,
        prio:3 dscp:31,30,29,28,27,26,25,24,
        prio:4 dscp:39,38,37,36,35,34,33,32,
        prio:5 dscp:47,46,45,44,43,42,41,40,
        prio:6 dscp:55,54,53,52,51,50,49,48,
        prio:7 dscp:63,62,61,60,59,58,57,56,
Receive buffer size (bytes): 19872,1046016,0,0,0,0,0,0,max_buffer_size=4148640
Cable len: 7
PFC configuration:
        priority    0   1   2   3   4   5   6   7
        enabled     0   0   0   0   0   1   0   0
        buffer      0   0   0   0   0   1   0   0
tc: 0 ratelimit: unlimited, tsa: vendor
         priority:  1
tc: 1 ratelimit: unlimited, tsa: vendor
         priority:  0
tc: 2 ratelimit: unlimited, tsa: vendor
         priority:  2
tc: 3 ratelimit: unlimited, tsa: vendor
         priority:  3
tc: 4 ratelimit: unlimited, tsa: vendor
         priority:  4
tc: 5 ratelimit: 399.0 Gbps, tsa: vendor
         priority:  5
tc: 6 ratelimit: unlimited, tsa: vendor
         priority:  6
tc: 7 ratelimit: unlimited, tsa: vendor
         priority:  7

So , the root cause is ConnectX-7 frequence is too high, how I can configure the ConnectX-7 to fix the issue?

2

Hi,

DCQCN (via CNP) is a second level recovery due to inability of the switch to cope with the traffic. The actual problem is on the receiving port of the switch. As such it is an incoming switch buffer overflow and not outgoing switch buffer overflow. To address that you might want to consider enabling PFC on the switch.

As you mentioned, another way it to rate limit the traffic from the ConnectX-7.

Regards,

Yaniv