Mellanox Technologies MT28800 Family [ConnectX-5 Ex] and ConnectX-5, Full Mode ，Latency increase a lot (759us → 1116us)

[user68489]

1, Ethernet controller infomation

​

lspci | grep -i mellanox

3f:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]

3f:00.1 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]

b3:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]

b3:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]

​

ibv_devinfo -d mlx5_0

hca_id: mlx5_0

transport: InfiniBand (0)

fw_ver: 16.29.1016

node_guid: 0c42:a103:0060:0610

sys_image_guid: 0c42:a103:0060:0610

vendor_id: 0x02c9

vendor_part_id: 4121

hw_ver: 0x0

board_id: MT_0000000013

phys_port_cnt: 1

port: 1

state: PORT_ACTIVE (4)

max_mtu: 4096 (5)

active_mtu: 1024 (3)

sm_lid: 0

port_lid: 0

port_lmc: 0x00

link_layer: Ethernet

​

---

ib_write_lat -d mlx5_0 -p 2500 -R -F -x 3 -a --report_gbits

ib_write_lat -d mlx5_0 -p 2500 -R -F -a --report_gbits ${IP} -x 3

---

​

​

[user68489]

ibstat

…​

CA ‘mlx5_0’

CA type: MT4121

Number of ports: 1

Firmware version: 16.29.1016

Hardware version: 0

Node GUID: 0x0c42a10300600610

System image GUID: 0x0c42a10300600610

Port 1:

State: Active

Physical state: LinkUp

Rate: 100

Base lid: 0

LMC: 0

SM lid: 0

Capability mask: 0x00010000

Port GUID: 0x0e42a1fffe600610

Link layer: Ethernet

---

ethtool ens12f0

Settings for ens12f0:

Supported ports: [ FIBRE ]

Supported link modes: 1000baseKX/Full

10000baseKR/Full

40000baseKR4/Full

40000baseCR4/Full

40000baseSR4/Full

40000baseLR4/Full

25000baseCR/Full

25000baseKR/Full

25000baseSR/Full

50000baseCR2/Full

50000baseKR2/Full

100000baseKR4/Full

100000baseSR4/Full

100000baseCR4/Full

100000baseLR4_ER4/Full

Supported pause frame use: Symmetric

Supports auto-negotiation: Yes

Supported FEC modes: None RS

Advertised link modes: 1000baseKX/Full

10000baseKR/Full

40000baseKR4/Full

40000baseCR4/Full

40000baseSR4/Full

40000baseLR4/Full

25000baseCR/Full

25000baseKR/Full

25000baseSR/Full

50000baseCR2/Full

50000baseKR2/Full

100000baseKR4/Full

100000baseSR4/Full

100000baseCR4/Full

100000baseLR4_ER4/Full

Advertised pause frame use: No

Advertised auto-negotiation: Yes

Advertised FEC modes: RS

Link partner advertised link modes: Not reported

Link partner advertised pause frame use: No

Link partner advertised auto-negotiation: Yes

Link partner advertised FEC modes: Not reported

Speed: 100000Mb/s

Duplex: Full

Port: FIBRE

PHYAD: 0

Transceiver: internal

Auto-negotiation: on

Supports Wake-on: d

Wake-on: d

Current message level: 0x00000004 (4)

link

Link detected: yes

​

[user68489]

mlnx_qos -i ens12f0

DCBX mode: OS controlled

Priority trust state: dscp

dscp2prio mapping:

prio:0 dscp:07,06,05,04,03,02,01,00,

prio:1 dscp:15,14,13,12,11,10,09,08,

prio:2 dscp:23,22,21,20,19,18,17,16,

prio:3 dscp:31,30,29,28,27,26,25,24,

prio:4 dscp:39,38,37,36,35,34,33,32,

prio:5 dscp:47,46,45,44,43,42,41,40,

prio:6 dscp:55,54,53,52,51,50,49,48,

prio:7 dscp:63,62,61,60,59,58,57,56,

Receive buffer size (bytes): 130944,130944,0,0,0,0,0,0,

Cable len: 7

PFC configuration:

priority 0 1 2 3 4 5 6 7

enabled 0 0 0 1 0 0 0 0

buffer 0 0 0 1 0 0 0 0

tc: 0 ratelimit: unlimited, tsa: vendor

priority: 1

tc: 1 ratelimit: unlimited, tsa: vendor

priority: 0

tc: 2 ratelimit: unlimited, tsa: vendor

priority: 2

tc: 3 ratelimit: unlimited, tsa: vendor

priority: 3

tc: 4 ratelimit: unlimited, tsa: vendor

priority: 4

tc: 5 ratelimit: unlimited, tsa: vendor

priority: 5

tc: 6 ratelimit: unlimited, tsa: vendor

priority: 6

tc: 7 ratelimit: unlimited, tsa: vendor

priority: 7

[abirman]

Hello,

I this case, in order to achieve a better adapter performance, I can recommend to follow the performance tuning guide.

Regards,

Anatoly