Can SOFTROCE on a client send message to ROCE on a server(not mlx5 card, it's a roce card from another company) by using ib_send_bw?

Infrastructure & Networking Software And Drivers
, ,
1

We try to let SOFTROCE on a client send message to ROCE on a server(not mlx5 card, it’s a roce card from another company) by using ib_send_bw. But it fails, the error message is:

Send BW Test

Dual-port : OFF Device : hrn0_1

Number of qps : 1 Transport type : IB

Connection type : RC Using SRQ : OFF

TX depth : 128

CQ Moderation : 100

Mtu : 1024[B]

Link type : Ethernet

GID index : 2

Max inline data : 0[B]

rdma_cm QPs : OFF

Data ex. method : Ethernet

local address: LID 0000 QPN 0x0008 PSN 0xb34c0a

GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:151:154:85

remote address: LID 0000 QPN 0x0012 PSN 0xeae1e

GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:151:154:82

#bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps]

Completion with error at client

Failed status 12: wr_id 0 syndrom 0x0

scnt=128, ccnt=0

Does anyone knows how to fix the problem?

2

Hello Qi,

Many thanks for your inquiry on the Mellanox Community.

Based on the information provided, this question should be redirected to the SoftRoCE community as Mellanox only handles SoftRoCE related issues regarding their adapters.

The link to the SoftROCE Community is → https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home

But for now, I will assist you as much as I can.

In my lab, I had no issue configuring SoftRoCE on our Mellanox adapters. We disabled h/w RoCE for this test.

In your output, it looks like you did not created the SoftROCE device for your NIC, as the SoftROCE device should have the naming, rxe

Example output:

rxe_cfg -l

Name Link Driver Speed NMTU IPv4_addr RDEV RMTU

eno1 yes igb

ens1 yes mlx5_core rxe0 4096 (5)

SoftROCE device in our setup is rxe1 (Intel) and rxe0 (Intel)

Make sure you create your SoftROCE device first. Example output:

rxe_cfg add eno1

rxe_cfg -l

Name Link Driver Speed NMTU IPv4_addr RDEV RMTU

eno1 yes igb rxe1 1024 (3)

ens1 yes mlx5_core rxe0 4096 (5)

Then you can do the ib_send_bw test on the SoftROCE device. Example output:

Send BW Test

Dual-port : OFF Device : rxe0

Number of qps : 1 Transport type : IB

Connection type : RC Using SRQ : OFF

RX depth : 512

CQ Moderation : 100

Mtu : 4096[B]

Link type : Ethernet

GID index : 1

Max inline data : 0[B]

rdma_cm QPs : OFF

Data ex. method : Ethernet

local address: LID 0000 QPN 0x0011 PSN 0x33a36b

GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:10

remote address: LID 0000 QPN 0x0011 PSN 0x8d44b1

GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:25

#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]

Conflicting CPU frequency values detected: 2401.069000 != 1535.646000. CPU Frequency is not max.

2 1000 0.000000 0.002896 0.180997

Conflicting CPU frequency values detected: 2400.899000 != 2318.806000. CPU Frequency is not max.

4 1000 0.000000 0.006505 0.203270

Conflicting CPU frequency values detected: 2400.977000 != 2331.040000. CPU Frequency is not max.

Many thanks,

~Mellanox Technical Support