Read port priority counters is 0 .

iriaforever · October 22, 2015, 12:11pm

Hi.

I had a problem port priority 3 counters is not.

Reference is https://community.mellanox.com/s/article/howto-run-roce-over-l2-enabled-with-pfc

What should we check

Thank you in advance!

pc1 - Switch - pc2

Switch firmware 3.4.2008

pc1 and 2 is Ububtu 14.04

Fimware 2.35.5100

Driver 3.1.-1.0.3

pc1

root@Clotho:~# ib_write_bw -R --report_gbits --port=12500 -D 10

Waiting for client to connect… *

RDMA_Write BW Test

Dual-port : OFF Device : mlx4_0

Number of qps : 1 Transport type : IB

Connection type : RC Using SRQ : OFF

CQ Moderation : 100

Mtu : 1024[B]

Link type : Ethernet

Gid index : 0

Max inline data : 0[B]

rdma_cm QPs : ON

Data ex. method : rdma_cm

Waiting for client rdma_cm QP to connect

Please run the same command with the IB/RoCE interface IP

local address: LID 0000 QPN 0x025a PSN 0x86a3f8

GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:10

remote address: LID 0000 QPN 0x025a PSN 0x35cd7d

GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:20

#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]

65536 417600 0.00 36.47 0.069568

root@Clotho:~# ethtool -S p1p1 | grep prio_3

rx_pause_prio_3: 0

rx_pause_duration_prio_3: 0

rx_pause_transition_prio_3: 0

tx_pause_prio_3: 0

tx_pause_duration_prio_3: 0

tx_pause_transition_prio_3: 0

rx_prio_3_packets: 0

rx_prio_3_bytes: 0

tx_prio_3_packets: 0

tx_prio_3_bytes: 0

root@Clotho:~# ./roce-check.sh

============================

pfcrx / pfctx parameters

============================

INTERFACE: mlx4_en

RX: 0x08

TX: 0x08

============================

VLAN parameters

============================

VLAN: p1p1.100

EGRESS priority mappings: 0:3 1:3 2:3 3:3 4:3 5:3 6:3 7:3

pc2.

ib_write_bw -R --report_gbits 192.168.10.10 --port=12500 -D 10

RDMA_Write BW Test

Dual-port : OFF Device : mlx4_0

Number of qps : 1 Transport type : IB

Connection type : RC Using SRQ : OFF

TX depth : 128

CQ Moderation : 100

Mtu : 1024[B]

Link type : Ethernet

Gid index : 0

Max inline data : 0[B]

rdma_cm QPs : ON

Data ex. method : rdma_cm

local address: LID 0000 QPN 0x025a PSN 0x35cd7d

GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:20

remote address: LID 0000 QPN 0x025a PSN 0x86a3f8

GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:10

#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]

Conflicting CPU frequency values detected: 2799.937000 != 1220.531000. CPU Frequency is not max.

65536 417600 0.00 36.47 0.069568

root@Aprotos:~# ethtool -S p1p1 | grep prio_3

rx_pause_prio_3: 0

rx_pause_duration_prio_3: 0

rx_pause_transition_prio_3: 0

tx_pause_prio_3: 0

tx_pause_duration_prio_3: 0

tx_pause_transition_prio_3: 0

rx_prio_3_packets: 0

rx_prio_3_bytes: 0

tx_prio_3_packets: 0

tx_prio_3_bytes: 0

root@Aprotos:~# ./roce-check.sh

============================

pfcrx / pfctx parameters

============================

INTERFACE: mlx4_en

RX: 0x08

TX: 0x08

============================

VLAN parameters

============================

VLAN: p1p1.100

EGRESS priority mappings: 0:3 1:3 2:3 3:3 4:3 5:3 6:3 7:3

iriaforever · October 30, 2015, 4:54pm

Thanks, Sophie Naudin

Swich configuration is Hybrid mode and enable pfc setting seems there is no problem.

switch-5e13ce [standalone: master] (config) # show dcb priority-flow-control

PFC enabled

Priority Enabled List :3

Priority Disabled List :0 1 2 4 5 6 7

TC Lossless

0 N

1 Y

2 Y

3 N

Interface PFC admin PFC oper

Eth1/1 On Enabled

Eth1/2 Disabled Disabled

Eth1/3 Disabled Disabled

Eth1/4 Disabled Disabled

Eth1/5 On Enabled

Eth1/6 Disabled Disabled

Eth1/7 Disabled Disabled

Eth1/8 Disabled Disabled

Eth1/9 Disabled Disabled

Eth1/10 Disabled Disabled

Eth1/11 Disabled Disabled

Eth1/12 Disabled Disabled

Eth1/13 Disabled Disabled

Eth1/14 Disabled Disabled

Eth1/15 Disabled Disabled

Eth1/16 Disabled Disabled

Eth1/17 Disabled Disabled

Eth1/18 Disabled Disabled

Eth1/19 Disabled Disabled

Eth1/20 Disabled Disabled

Eth1/21 Disabled Disabled

Eth1/22 Disabled Disabled

Eth1/23 Disabled Disabled

Eth1/24 Disabled Disabled

Eth1/25 Disabled Disabled

Eth1/26 Disabled Disabled

Eth1/27 Disabled Disabled

Eth1/28 Disabled Disabled

Eth1/29 Disabled Disabled

Eth1/30 Disabled Disabled

Eth1/31 Disabled Disabled

Eth1/32 Disabled Disabled

Eth1/33 Disabled Disabled

Eth1/34 Disabled Disabled

Eth1/35 Disabled Disabled

Eth1/36 Disabled Disabled

According to your instruction Check it all.

mlx4_en.conf

openibd restart

/sys/module/mlx4_en/parameters/pfcrx & pfctx

pc1

root@Atropos:~# cat /proc/net/vlan/p1p1.100

p1p1.100 VID: 100 REORDER_HDR: 1 dev->priv_flags: 1

total frames received 0

total bytes received 0

Broadcast/Multicast Rcvd 0

total frames transmitted 8

total bytes transmitted 648

Device: p1p1

INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0

EGRESS priority mappings: 0:3 1:3 2:3 3:3 4:3 5:3 6:3 7:3

root@Atropos:~# ib_write_bw -R --report_gbits --port=12500 -D 10

Waiting for client to connect… *

RDMA_Write BW Test

Dual-port : OFF Device : mlx4_0

Number of qps : 1 Transport type : IB

Connection type : RC Using SRQ : OFF

CQ Moderation : 100

Mtu : 2048[B]

Link type : Ethernet

Gid index : 0

Max inline data : 0[B]

rdma_cm QPs : ON

Data ex. method : rdma_cm

Waiting for client rdma_cm QP to connect

Please run the same command with the IB/RoCE interface IP

local address: LID 0000 QPN 0x0267 PSN 0xfa896c

GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:20

remote address: LID 0000 QPN 0x026e PSN 0x5ce508

GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:30

#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]

65536 447200 0.00 39.07 0.074512

pc2

root@ubuntu:/etc/modprobe.d# cat /proc/net/vlan/p3p1.100

p3p1.100 VID: 100 REORDER_HDR: 1 dev->priv_flags: 1

total frames received 0

total bytes received 0

Broadcast/Multicast Rcvd 0

total frames transmitted 8

total bytes transmitted 648

Device: p3p1

INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0

EGRESS priority mappings: 0:3 1:3 2:3 3:3 4:3 5:3 6:3 7:3

root@ubuntu:/etc/modprobe.d# ib_write_bw -R --report_gbits 192.168.10.20 --port=12500 -D 10

RDMA_Write BW Test

Dual-port : OFF Device : mlx4_1

Number of qps : 1 Transport type : IB

Connection type : RC Using SRQ : OFF

TX depth : 128

CQ Moderation : 100

Mtu : 2048[B]

Link type : Ethernet

Gid index : 0

Max inline data : 0[B]

rdma_cm QPs : ON

Data ex. method : rdma_cm

local address: LID 0000 QPN 0x026e PSN 0x5ce508

GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:30

remote address: LID 0000 QPN 0x0267 PSN 0xfa896c

GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:20

#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]

Conflicting CPU frequency values detected: 1199.980000 != 3799.960000. CPU Frequency is not max.

65536 447200 0.00 39.07 0.074512

switch-5e13ce [standalone: master] (config) # show interfaces ethernet 1/1 counters priority 3

Rx

0 packets

0 unicast packets

0 multicast packets

0 broadcast packets

0 bytes

0 pause packets

0 pause duration milliseconds

Tx

0 packets

0 unicast packets

0 multicast packets

0 broadcast packets

0 bytes

0 pause packets

switch-5e13ce [standalone: master] (config) # show interfaces ethernet 1/1 counters priority 0

Rx

176634481 packets

176634354 unicast packets

34 multicast packets

93 broadcast packets

697235467340 bytes

0 pause packets

0 pause duration milliseconds

Tx

164914925 packets

164907341 unicast packets

7583 multicast packets

1 broadcast packets

648406932028 bytes

0 pause packets

switch-5e13ce [standalone: master] (config) # show interfaces ethernet 1/5 counters priority 3

Rx

0 packets

0 unicast packets

0 multicast packets

0 broadcast packets

0 bytes

0 pause packets

0 pause duration milliseconds

Tx

0 packets

0 unicast packets

0 multicast packets

0 broadcast packets

0 bytes

0 pause packets

switch-5e13ce [standalone: master] (config) # show interfaces ethernet 1/5 counters priority 0

Rx

164907378 packets

164907341 unicast packets

36 multicast packets

1 broadcast packets

645108150520 bytes

0 pause packets

0 pause duration milliseconds

Tx

176641795 packets

176634354 unicast packets

7416 multicast packets

25 broadcast packets

700768765837 bytes

0 pause packets

switch-5e13ce [standalone: master] (config) #

Still now it is not solve. Advise, Thanks!!

iriaforever · October 30, 2015, 11:33am

Thanks, Ophir Maor

Switch Port 1 and 5 not packets counters

switch-5e13ce [standalone: master] (config) # show interfaces ethernet 1/1 counters priority 3

Rx

0 packets

0 unicast packets

0 multicast packets

0 broadcast packets

0 bytes

0 pause packets

0 pause duration milliseconds

Tx

0 packets

0 unicast packets

0 multicast packets

0 broadcast packets

0 bytes

0 pause packets

switch-5e13ce [standalone: master] (config) # show interfaces ethernet 1/1 counters priority 5

Rx

0 packets

0 unicast packets

0 multicast packets

0 broadcast packets

0 bytes

0 pause packets

0 pause duration milliseconds

Tx

0 packets

0 unicast packets

0 multicast packets

0 broadcast packets

0 bytes

0 pause packets

yes, Switch PFC is enabled

---- switch running config –

Running database “initial”

Generated at 2015/10/30 11:19:07 +0000

Hostname: switch-5e13ce

Running-config temporary prefix mode setting

no cli default prefix-modes enable

License keys

license install …

DCBX PFC configuration

dcb priority-flow-control enable force

dcb priority-flow-control priority 3 enable

interface ethernet 1/1 dcb priority-flow-control mode on force

interface ethernet 1/5 dcb priority-flow-control mode on force

Interface Ethernet configuration

interface ethernet 1/1 mtu 4200 force

interface ethernet 1/5 mtu 4200 force

interface ethernet 1/1 switchport mode hybrid

interface ethernet 1/5 switchport mode hybrid

VLAN configuration

vlan 100

interface ethernet 1/1 switchport hybrid allowed-vlan all

interface ethernet 1/5 switchport hybrid allowed-vlan all

DCBX ETS configuration

no dcb ets enable

dcb ets tc bandwidth 20 20 30 30

Network interface configuration

no interface mgmt0 dhcp

interface mgmt0 ip address xxx.xxx.xx.xxx /25

Network interface IPv6 configuration

no interface mgmt0 ipv6 address autoconfig default

no interface mgmt0 ipv6 enable

Other IP configuration

ip route 0.0.0.0 0.0.0.0 xxx.xxx.xxx.xxx

AAA remote server configuration

ldap bind-password ********

radius-server key ********

tacacs-server key ********

Network management configuration

web proxy auth basic password ********

X.509 certificates configuration

Certificate name system-self-signed, ID 0e5aa8139ab66157a7a051bcf66540771fb3cab2

(public-cert config omitted since private-key config is hidden)

Persistent prefix mode setting

cli default prefix-modes enable

My FDR HCA is MCX353A-FCBT (both)
RX=cat /sys/module/mlx4_en/parameters/pfcrx;printf “0x%x\n” $RX

TX=cat /sys/module/mlx4_en/parameters/pfctx;printf “0x%x\n” $TX

Result is

pc1.

root@Clotho:/etc/modprobe.d# TX=cat /sys/module/mlx4_en/parameters/pfctx;printf “0x%x\n” $TX

0x8

root@Clotho:/etc/modprobe.d# RX=cat /sys/module/mlx4_en/parameters/pfcrx;printf “0x%x\n” $RX

0x8

root@Clotho:/etc/modprobe.d#

pc2.

root@Atropos:/etc/modprobe.d# TX=cat /sys/module/mlx4_en/parameters/pfctx;printf “0x%x\n” $TX

0x8

root@Atropos:/etc/modprobe.d# RX=cat /sys/module/mlx4_en/parameters/pfcrx;printf “0x%x\n” $RX

0x8

root@Atropos:/etc/modprobe.d#

And See most of the reference to the document, but The Result is still the same.

Any help Thanks!!

ophirm · October 27, 2015, 6:37pm

Hi,

1.Can you send the output on the switch for the port priority counters?

Can you send the running config of the switch, to make sure PFC is enabled.
Which adapter is used?
can you send the output for this

RX=cat /sys/module/mlx4_en/parameters/pfcrx;printf “0x%x\n” $RX

and

TX=cat /sys/module/mlx4_en/parameters/pfctx;printf “0x%x\n” $TX

spruitt · October 29, 2015, 7:15pm

Hi Seol,

MLNX-OS Switch Configuration:

a. Have you created a VLAN and set the switchports to hybrid (or trunk) mode?

b. Have you Enable PFC on the switch and on the interfaces?

c. Does the “show dcb priority-flow-control” reports the accurate configuration?

Server configuration:

a. Do you have the /etc/modprobe.d/mlx4_en.conf file with the following options:

options mlx4_en pfctx=0x08 pfcrx=0x08

b. Did you restart openibd service ? (#/etc/init.d/openidb restart)

Please also verify under /sys/module/mlx4_en/parameters/pfcrx & pfctx files that the priority is set accordingly.

c. When you run the “ib_write_bw” from the client (Aprotos), does the IP 192.168.10.10 correspond to the VLAN: p1p1.100 on server Clotho?

Port priority counters from the Mellanox switch (via MLNX-OS CLI):

a. Does the interface(s) on the switch report any pause packets? (check for both interfaces from the servers).

ie: switch (config) # show interfaces ethernet 1/1 counters priority 3

Thank you,

Sophie.

spruitt · November 2, 2015, 8:31pm

Hi Seol,

Originally you used PC1 “Clotho” & PC2 “Aprotos”, on my previous statement I asked:

When you run the “ib_write_bw” from the client (Aprotos), does the IP 192.168.10.10 correspond to the VLAN: p1p1.100 on server Clotho?
If you are testing with client (Clotho) to server (Aprotos), do the counters report the same?
Are you now introducing and testing with a different host? (Ubuntu)
What are the switch & HCA model/partnumber/PSID?

Thank you,

Sophie.

Topic		Replies	Views
Setting service level in ib_send_bw doesn't increase expected priority counters in ethtool Ethernet Adapter Cards	1	1135	August 25, 2022
mlnx_qos cannot assign priority values to TCs after 8 SR-IOV devices Virtualization For Infiniband And Ethernet	25	1258	December 6, 2018
PFC with ConnectX-5 Ethernet Adapter Cards	2	667	November 30, 2017
Switch boot problem InfiniBand/VPI Switch Systems switches , 1-dev	15	2213	December 10, 2018
mlx5_0 port is Down ! InfiniBand/VPI Adapter Cards	9	3520	April 12, 2024
Ib_send_bw --ipv6 Errors out Mellanox OFED	1	1353	September 10, 2023
Poor bandwidth performance when running with large block size Ethernet Adapter Cards iterations , bytes	9	1539	May 28, 2018
PCI-e Quad Port NIC problem, same but different Jetson TX1	11	1551	October 18, 2021
TK1 mini-PCIe stuck at 2.5GT/s Jetson TK1	14	6116	November 26, 2014
PCIE(C0) - is not working for Jetson AGX orin Jetson AGX Orin pcie	15	1068	October 10, 2023

Read port priority counters is 0 .

Running database “initial”

Generated at 2015/10/30 11:19:07 +0000

Hostname: switch-5e13ce

Running-config temporary prefix mode setting

License keys

DCBX PFC configuration

Interface Ethernet configuration

VLAN configuration

DCBX ETS configuration

Network interface configuration

Network interface IPv6 configuration

Other IP configuration

AAA remote server configuration

ldap bind-password ********

radius-server key ********

tacacs-server key ********

Network management configuration

web proxy auth basic password ********

X.509 certificates configuration

Certificate name system-self-signed, ID 0e5aa8139ab66157a7a051bcf66540771fb3cab2

(public-cert config omitted since private-key config is hidden)

Persistent prefix mode setting

Related topics