We have a very unique use-case where we transmit data from a Linux server (Ubuntu) via ConnectX-5
to an FPGA. The FPGA cannot handle 100G of traffic and controls the incomming RoCE traffic using Ethernet PFC Pause packets.
The FPGA debug scenario enables us to accurately determine the latency between the start of a pause request that is sent to the ConnectX-5 and the moment where the ConnecX-5 stops sending data.
However, within the debug time window of approximately 40 us we do not see that the ConnectX-5 reacts to the pause request, suggesting that it takes more than 40 us to react to a pause request.
Therefore, I have several questions:
How long does it take for the ConnectX-5 (and other cards) to react to a received pause request?
How accurately is the pause quanta time implemented?
Is this delay a constant time? or by how much does it vary in the worst case?
Do newer adapter cards (ConnectX-6 …) offer quicker response times?
And same for the release of the pause request:
How long does it take for the ConnectX-5 to start transmitting again after the pause is released (no more pause packets, and quanta time expired)?
First, you want to make sure PFC is enabled on the NIC. Sometimes, it is enabled on priority X and traffic is running on priority Y, thus it will not be working at all.
Assuming you have configured things correctly and running your traffic on the proper priority …
Our NICs are spec compliant (IEEE 802.3) so you can refer to “31B.3.7 Timing considerations for PAUSE operation” for more details.
The general idea is dependent on the link speed. E.g., for a 100Gb/s the NIC will not begin to transmit a new frame more than 394 pause quanta after the reception of a valid pause frame that contains a non-zero value of pause time. 394 * 512(bits) == 25,216 Bytes.
The pause quanta is a 16-bit value. We usually populate it with 0xFFFF when we send pause frames from our NICs. Meaning, 65536 * 512(bits) == 4MB (for ConnectX-5, ConnectX-6 Dx). Assuming you send such pause the NIC will stop transmission for 4MB. In case the NIC receives a new pause during this time we will restart the timer (4MB).
[You may want to verify what is the value you are sending from the FPGA]