Bluefield3 NIC generating PFC frames when running NCCL all2all

Infrastructure & Networking BlueField
1

Hi,

I have 4 servers with each 8x H200 nvidia GPUs with Bluefield3 NICs. When running all2all NCCL test I see the NICs generating PFC frames to trottle down the bandwidth, hence I’m not reaching optimal values for this NCCL run.

The network and the NICs have been configured with the following DCQCN QoS settings using mlnx_qos and cma_roce_tos

  • RoCEv2: DSCP26 –> priority 3 (PFC enabled). Has 90% bandwidth
  • Controll (CNP): DSCP 48 –> priority6. Has strict priority
  • All other: DSCP0 —> priority 0. Has 10% bandwidth.

My current QoS settings are the following:

DCBX mode: OS controlled
Priority trust state: dscp
dscp2prio mapping:
        prio:0 dscp:07,06,05,04,03,02,01,00,
        prio:1 dscp:15,14,13,12,11,10,09,08,
        prio:2 dscp:23,22,21,20,19,18,17,16,
        prio:3 dscp:26,31,30,29,28,27,25,24,
        prio:4 dscp:39,38,37,36,35,34,33,32,
        prio:5 dscp:47,46,45,44,43,42,41,40,
        prio:6 dscp:48,55,54,53,52,51,50,49,
        prio:7 dscp:63,62,61,60,59,58,57,56,
Receive buffer size (bytes): 19872,588672,0,0,0,0,0,0,max_buffer_size=4151520
Cable len: 5
PFC configuration:
        priority    0   1   2   3   4   5   6   7
        enabled     0   0   0   1   0   0   0   0   
        buffer      0   0   0   1   0   0   0   0   
tc: 0 ratelimit: unlimited, tsa: ets, bw: 10%
         priority:  0
         priority:  1
         priority:  2
         priority:  4
         priority:  5
         priority:  7
tc: 3 ratelimit: unlimited, tsa: ets, bw: 90%
         priority:  3
tc: 6 ratelimit: unlimited, tsa: strict
         priority:  6

Looking to the interface statistics I can many PFC frames send out on prio3. Interface statistics confirm the same (no shown here).

ethtool -S ens1f0np0 | grep prio3
     rx_prio3_bytes: 19777577035128
     rx_prio3_packets: 7511525940
     rx_prio3_discards: 0
     tx_prio3_bytes: 17883206394538
     tx_prio3_packets: 4817097965
     rx_prio3_pause: 0
     rx_prio3_pause_duration: 0
     tx_prio3_pause: 679979
     tx_prio3_pause_duration: 978992
     rx_prio3_pause_transition: 0
     rx_prio3_buf_discard: 0
     rx_prio3_cong_discard: 0
     rx_prio3_marked: 0

When looking to the output of mlnx_qos shown above I can see the receive buffer has 0 buffer room assigned to prio3. Only 19872 and 588672 bytes assigned to prio0 and prio1 respectively. Receive buffer size (bytes): 19872,588672,0,0,0,0,0,0,max_buffer_size=4151520 Is this how I should interprete this?

I tried to put more buffer room on prio3 with the following command but it fails with the following

sudo mlnx_qos -i mlx5_0 --buffer_size=19872,588672,0,10000,0,0,10000,0
Priority trust state is not supported on your system
Buffers commands are not supported on your system

Can someone provide any guidance how to reduce the amount of PFC frames generated from the Bluefield3 NICs? I suspect the buffer size is causing the issue, but I might be overlooking something else. Any help would be much appreciated.

Version:

  • cuda 13.1
  • nvidia-smi 590.48.01
  • bluefield3 FW 32.43.2026
2

Disabling ACS and setting the mode to Performance in BIOS did the trick

3

To clarify why that helped:

ACS is required for virtualized environments. I’m suspecting you are in bare-metal environment.

When you had ACS enabled that extended the latency between the NIC and the GPU (transactions went the long route through the CPU) resulting in back pressure into the network. The back pressure was visible as PFC being sent to the network to stop the transmitter from sending more data (temporarily). The sender NIC does NOT trottle down the traffic but rather STOP it. Congestion Control (not discussed in this thread) is trottling down the sending traffic.

Regards,

Yaniv

4

Thanks for clarifying!

Setup is indeed on bare-metal environment