I have a Mellanox ConnectX-3 dual port adapter. Output of lspci:
07:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
I am on Centos 7.6.1810. The Mellanox OFED driver version is 4.0-188.8.131.52.
Output of ibstat:
CA type: MT4099
Number of ports: 2
Firmware version: 2.42.5030
Hardware version: 1
Node GUID: 0x0002c903002f3380
System image GUID: 0x0002c903002f3383
Physical state: LinkUp
Base lid: 80
SM lid: 2
Capability mask: 0x02594868
Port GUID: 0x0002c903002f3381
Link layer: InfiniBand
Physical state: Disabled
Base lid: 0
SM lid: 0
Capability mask: 0x04010000
Port GUID: 0x0202c9fffe2f3388
Link layer: Ethernet
I have tried with ibdump, but I have the following error:
-E- Failed to set sniffer mode. ibv_create_flow failed: No such file or directory
This problem might be because Flow Steering is not enabled, to enable it:
1. Add the following to /etc/modprobe.d/mlnx.conf file:
options mlx4_core log_num_mgm_entry_size=-1
2. Restart the drivers.
I tried what the error suggest, but still the same. If it is helpful, I don’t have the command ibv_create_flow failed, which the error mentions, and I couldn’t find a way to install it.
I read that it is possible to capture the RDMA packets with ethtool and tcpdump, but that is only since ConnectX-4 adapters.
My final purpose is to test if the MPI traffic is going over InfiniBand stack or if it is going all the way down/up the TCP/IP stack. So, if you know some other method to test it, it will be well received.
Thanks in advance.