Hello,
I have two debian stretch hosts cabled up to eachother and running ConnectX-5 mcx516a cards. I have installed OFED and all the kernel drivers, and have followed the steps laid out in https://community.mellanox.com/s/article/recommended-network-configuration-examples-for-roce-deployment to the T, all without issue.
However, all of the given ibv_… tools can’t seem to establish a connection, and all error out with a host unreachable error. I’ve confirmed that the two hosts can talk to one another over the connected cable, they have IPs and can ARP one another, I even went so far as to manually add ARP entries for both of them, and they can ping one another over the connection.
What am I missing? Is this a idiosyncrasy of the EN series? If so, how can I test and confirm RoCE functionality on these cards and expose it to user-level programs and utilities?
Thanks for your time
Hello Christian,
Hope you are doing well.
There are a couple things you might try that could help
1 Double check that you are using the proper syntax. The ibv commands can be very picky about this. The structure of the server command should be:
ibv_rc_pingpong -d -i <IB_port> -g
the structure of the client command should be:
ibv_rc_pingpong -d <device -i -g
Your commands might look like the following for example:
Server: ibv_rc_pingpong -d mlx5_0 -i 2 -g 0
Client: ibv_rc_pingpong -d mlx5_0 -i 2 -g 0 server-1
You can see the GID index of each port by using the command: show_gids
2 You could also try running an iperf test. You can install using any package manager such as :
yum install iperf
From there a basic test would look like:
Server: iperf -s
Client: iperf -c
3 finally one other option is that you could try using the VMA library if you have it installed. A command you could try running might look like:
Server: LD_PRELOAD=libvma.so VMA_SPEC=latency sockperf server -i
Client: LD_PRELOAD=libvma.so VMA_SPEC=latency sockperf ping-pong -i
For more information on these commands and VMA you can refer to the manual at https://docs.mellanox.com/display/VMAv894/Messaging+Accelerator+%28VMA%29+Documentation+Rev+8.9.4
Hope one of these options works for you. If you are still running into issues you can open a case with support@mellanox.com as you have a valid support contract or I can open a case for you if you ask here.
Thanks,
Abigail.