Verifying RoCE on ConnectX-5 EN and connectivity issues


I have two debian stretch hosts cabled up to eachother and running ConnectX-5 mcx516a cards. I have installed OFED and all the kernel drivers, and have followed the steps laid out in to the T, all without issue.

However, all of the given ibv_… tools can’t seem to establish a connection, and all error out with a host unreachable error. I’ve confirmed that the two hosts can talk to one another over the connected cable, they have IPs and can ARP one another, I even went so far as to manually add ARP entries for both of them, and they can ping one another over the connection.

What am I missing? Is this a idiosyncrasy of the EN series? If so, how can I test and confirm RoCE functionality on these cards and expose it to user-level programs and utilities?

Thanks for your time

Hello Christian,

Hope you are doing well.

There are a couple things you might try that could help

1 Double check that you are using the proper syntax. The ibv commands can be very picky about this. The structure of the server command should be:

ibv_rc_pingpong -d -i <IB_port> -g

the structure of the client command should be:

ibv_rc_pingpong -d <device -i -g

Your commands might look like the following for example:

Server: ibv_rc_pingpong -d mlx5_0 -i 2 -g 0

Client: ibv_rc_pingpong -d mlx5_0 -i 2 -g 0 server-1

You can see the GID index of each port by using the command: show_gids

2 You could also try running an iperf test. You can install using any package manager such as :

yum install iperf

From there a basic test would look like:

Server: iperf -s

Client: iperf -c

3 finally one other option is that you could try using the VMA library if you have it installed. A command you could try running might look like:

Server: VMA_SPEC=latency sockperf server -i

Client: VMA_SPEC=latency sockperf ping-pong -i

For more information on these commands and VMA you can refer to the manual at

Hope one of these options works for you. If you are still running into issues you can open a case with as you have a valid support contract or I can open a case for you if you ask here.