I used ‘-d’ to indicate the devices to send RDMA traffic to server, but ib_send_bw either ignores this option to use another device:
for example, here I want to use dev mlx5_0, but ib_send_bw choose mlx5_1.
is it an issue of ib_send_bw or my configuration problem?
btw, I am using the latest OFED linux driver.
client side output:
ib_send_bw --report_gbit -R -d mlx5_0 -D60 -p 18516 -T 96 100.3.20.11
Send BW Test
Dual-port : OFF Device : mlx5_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 128
CQ Moderation : 100
Mtu : 1024[B]
Link type : Ethernet
GID index : 3
Max inline data : 0[B]
rdma_cm QPs : ON
Data ex. method : rdma_cm TOS : 96
local address: LID 0000 QPN 0x0953 PSN 0x68693d
GID: 00:00:00:00:00:00:00:00:00:00:255:255:100:01:10:11
remote address: LID 0000 QPN 0x01b2 PSN 0xe4cb2
GID: 00:00:00:00:00:00:00:00:00:00:255:255:100:03:20:11
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
Conflicting CPU frequency values detected: 1203.267000 != 2330.035000. CPU Frequency is not max.
65536 4766200 0.00 83.30 0.158875
:~$ show_gids
DEV PORT INDEX GID IPv4 VER DEV
mlx5_0 1 0 fe80:0000:0000:0000:9a03:9bff:fe67:cb82 v1 ens2f0
mlx5_0 1 1 fe80:0000:0000:0000:9a03:9bff:fe67:cb82 v2 ens2f0
mlx5_0 1 2 0000:0000:0000:0000:0000:ffff:6401:140b 100.1.20.11 v1 ens2f0
mlx5_0 1 3 0000:0000:0000:0000:0000:ffff:6401:140b 100.1.20.11 v2 ens2f0
mlx5_1 1 0 fe80:0000:0000:0000:9a03:9bff:fe67:cb83 v1 ens2f1
mlx5_1 1 1 fe80:0000:0000:0000:9a03:9bff:fe67:cb83 v2 ens2f1
mlx5_1 1 2 0000:0000:0000:0000:0000:ffff:6401:0a0b 100.1.10.11 v1 ens2f1
mlx5_1 1 3 0000:0000:0000:0000:0000:ffff:6401:0a0b 100.1.10.11 v2 ens2f1
n_gids_found=8
:~$ ibdev2netdev
mlx5_0 port 1 ==> ens2f0 (Up)
mlx5_1 port 1 ==> ens2f1 (Up)
server side:
ib_send_bw --report_gbit -D60 -R -p 18516 -d mlx5_0
- Waiting for client to connect… *
Send BW Test
Dual-port : OFF Device : mlx5_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
RX depth : 512
CQ Moderation : 100
Mtu : 1024[B]
Link type : Ethernet
GID index : 3
Max inline data : 0[B]
rdma_cm QPs : ON
Data ex. method : rdma_cm
Waiting for client rdma_cm QP to connect
Please run the same command with the IB/RoCE interface IP
local address: LID 0000 QPN 0x01b2 PSN 0xe4cb2
GID: 00:00:00:00:00:00:00:00:00:00:255:255:100:03:20:11
remote address: LID 0000 QPN 0x0953 PSN 0x68693d
GID: 00:00:00:00:00:00:00:00:00:00:255:255:100:01:10:11
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
Conflicting CPU frequency values detected: 1202.680000 != 2641.012000. CPU Frequency is not max.
65536 4766161 0.00 83.29 0.158872