ib_send_bw failing on CX5 port.

Infrastructure & Networking Software And Drivers
,
1

I am trying to data transfer between two CX5 ports usinf ib_write_bw. but it is failing.

[root@ctech123 64]# ibdev2netdev

mlx5_0 port 1 ==> enp94s0f0 (Up)

mlx5_1 port 1 ==> enp94s0f1 (Up)

[root@ctech123 64]#

mlx5_0 is live port .

mlx5_0 and mlx5_1 are connected by cable.

Server

[root@ctech123 64]# ib_send_bw -F -d mlx5_0 -D 66 -p 8005 --report_gbits -b

  • Waiting for client to connect… *

Send Bidirectional BW Test

Dual-port : OFF Device : mlx5_0

Number of qps : 1 Transport type : IB

Connection type : RC Using SRQ : OFF

TX depth : 128

RX depth : 512

CQ Moderation : 100

Mtu : 1024[B]

Link type : Ethernet

GID index : 4

Max inline data : 0[B]

rdma_cm QPs : OFF

Data ex. method : Ethernet

local address: LID 0000 QPN 0x0115 PSN 0x99e04e

GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:134:145:61

remote address: LID 0000 QPN 0x091c PSN 0x39922e

GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:134:01:01

ethernet_read_keys: Couldn’t read remote address

Unable to read to socket/rdam_cm

Failed to exchange data between server and clients

[root@ctech123 64]#

Client Side

[root@ctech123 64]# ib_send_bw -F -d mlx5_1 -D 66 --report_gbits -p 8005 10.134.1.1 -b -m 4096

Requested mtu is higher than active mtu

Changing to active mtu - 3

Send Bidirectional BW Test

Dual-port : OFF Device : mlx5_1

Number of qps : 1 Transport type : IB

Connection type : RC Using SRQ : OFF

TX depth : 128

RX depth : 512

CQ Moderation : 100

Mtu : 1024[B]

Link type : Ethernet

GID index : 2

Max inline data : 0[B]

rdma_cm QPs : OFF

Data ex. method : Ethernet

local address: LID 0000 QPN 0x091c PSN 0x39922e

GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:134:01:01

remote address: LID 0000 QPN 0x0115 PSN 0x99e04e

GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:134:145:61

Failed to modify QP 2332 to RTR

Unable to Connect the HCA’s through the link

[root@ctech123 64]#

2

Hello Sanchi,

From your output, I see that you are attempting to perform the test over different subnets:

[Server]

GID index : 4

[Client]

GID index : 2

Dependent on your network configuration, these may not be compatible with each other.

To view a list of active GIDs for your adapters, please run:

show_gids

To run the test using a specified GID index, please run the ib_send_bw command with the ‘-x’ parameter.

Best regards,

Mellanox Technical Support

3

Thanks Sam for your reply.

I ran the show_gid and below is the output. How to know if the GIDs are not compatible with each other.?

[root@ctech123 64]# ./show_gid

DEV PORT INDEX GID IPv4 VER DEV

mlx5_0 1 0 fe80:0000:0000:0000:526b:4bff:fe16:6ac4 v1 enp94s0f0

mlx5_0 1 1 fe80:0000:0000:0000:526b:4bff:fe16:6ac4 v2 enp94s0f0

mlx5_0 1 2 fe80:0000:0000:0000:c2dc:0375:03ec:c8f8 v1 enp94s0f0

mlx5_0 1 3 fe80:0000:0000:0000:c2dc:0375:03ec:c8f8 v2 enp94s0f0

mlx5_0 1 4 0000:0000:0000:0000:0000:ffff:0a86:913d 10.134.145.61 v1 enp94s0f0

mlx5_0 1 5 0000:0000:0000:0000:0000:ffff:0a86:913d 10.134.145.61 v2 enp94s0f0

mlx5_0 1 6 2606:b400:0418:2770:6b88:f869:24bd:ee86 v1 enp94s0f0

mlx5_0 1 7 2606:b400:0418:2770:6b88:f869:24bd:ee86 v2 enp94s0f0

mlx5_1 1 0 fe80:0000:0000:0000:526b:4bff:fe16:6ac5 v1 enp94s0f1

mlx5_1 1 1 fe80:0000:0000:0000:526b:4bff:fe16:6ac5 v2 enp94s0f1

mlx5_1 1 2 0000:0000:0000:0000:0000:ffff:0a86:0101 10.134.1.1 v1 enp94s0f1

mlx5_1 1 3 0000:0000:0000:0000:0000:ffff:0a86:0101 10.134.1.1 v2 enp94s0f1

n_gids_found=12

[root@ctech123 64]#