Verifying RoCE on ConnectX-5 EN and connectivity issues

Hello,

I have two debian stretch hosts cabled up to eachother and running ConnectX-5 mcx516a cards. I have installed OFED and all the kernel drivers, and have followed the steps laid out in https://community.mellanox.com/s/article/recommended-network-configuration-examples-for-roce-deployment to the T, all without issue.

However, all of the given ibv_… tools can’t seem to establish a connection, and all error out with a host unreachable error. I’ve confirmed that the two hosts can talk to one another over the connected cable, they have IPs and can ARP one another, I even went so far as to manually add ARP entries for both of them, and they can ping one another over the connection.

What am I missing? Is this a idiosyncrasy of the EN series? If so, how can I test and confirm RoCE functionality on these cards and expose it to user-level programs and utilities?

Thanks for your time

Hello Christian,

Hope you are doing well.

There are a couple things you might try that could help

1 Double check that you are using the proper syntax. The ibv commands can be very picky about this. The structure of the server command should be:

ibv_rc_pingpong -d -i <IB_port> -g

the structure of the client command should be:

ibv_rc_pingpong -d <device -i -g

Your commands might look like the following for example:

Server: ibv_rc_pingpong -d mlx5_0 -i 2 -g 0

Client: ibv_rc_pingpong -d mlx5_0 -i 2 -g 0 server-1

You can see the GID index of each port by using the command: show_gids

2 You could also try running an iperf test. You can install using any package manager such as :

yum install iperf

From there a basic test would look like:

Server: iperf -s

Client: iperf -c

3 finally one other option is that you could try using the VMA library if you have it installed. A command you could try running might look like:

Server: LD_PRELOAD=libvma.so VMA_SPEC=latency sockperf server -i

Client: LD_PRELOAD=libvma.so VMA_SPEC=latency sockperf ping-pong -i

For more information on these commands and VMA you can refer to the manual at https://docs.mellanox.com/display/VMAv894/Messaging+Accelerator+%28VMA%29+Documentation+Rev+8.9.4

Hope one of these options works for you. If you are still running into issues you can open a case with support@mellanox.com as you have a valid support contract or I can open a case for you if you ask here.

Thanks,

Abigail.