ib_write_bw does not go beyond 20G/s when using average packet size 512

Infrastructure & Networking Adapters and Cables
,
1

Dear community,

We tried also some home made tools, with multithreading, and multiple work requests in parallel, and the maximum we could reach is 60G/s.

Is this the maximum speed the adapter can reach for this size of packet ?

numactl --cpubind=1 --membind=1 ib_write_bw -d mlx5_0 -s 512 -n 10000000 -F --report_gbit 192.168.70.160

#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]

512 10000000 0.00 20.58 5.023455

2

Which card are you use ?

3

we have 100G cards.

Please find below more details:

lspci | grep Mellanox

e2:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]

e2:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]

ibv_devinfo

hca_id: mlx5_0

transport: InfiniBand (0)

fw_ver: 16.31.1014

node_guid: b859:9f03:00c6:c552

sys_image_guid: b859:9f03:00c6:c552

vendor_id: 0x02c9

vendor_part_id: 4119

hw_ver: 0x0

board_id: MT_0000000012

phys_port_cnt: 1

port: 1

state: PORT_ACTIVE (4)

max_mtu: 4096 (5)

active_mtu: 1024 (3)

sm_lid: 0

port_lid: 0

port_lmc: 0x00

link_layer: Ethernet

hca_id: mlx5_1

transport: InfiniBand (0)

fw_ver: 16.31.1014

node_guid: b859:9f03:00c6:c553

sys_image_guid: b859:9f03:00c6:c552

vendor_id: 0x02c9

vendor_part_id: 4119

hw_ver: 0x0

board_id: MT_0000000012

phys_port_cnt: 1

port: 1

state: PORT_ACTIVE (4)

max_mtu: 4096 (5)

active_mtu: 1024 (3)

sm_lid: 0

port_lid: 0

port_lmc: 0x00

link_layer: Ethernet

4

Hi ,

Please refer to the Performance tuning guide .

https://community.mellanox.com/s/article/performance-tuning-for-mellanox-adapters

Thanks,

Samer