How to capture RDMA not RoCE packets on a ConnectX-3 adapter?. Is MPI traffic going over InfiniBand or IPoIB?.

Infrastructure & Networking Adapters and Cables InfiniBand/VPI Adapter Cards
1

I have a Mellanox ConnectX-3 dual port adapter. Output of lspci:

07:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]

I am on Centos 7.6.1810. The Mellanox OFED driver version is 4.0-1.0.1.0.

Output of ibstat:

CA ‘mlx4_0’

CA type: MT4099

Number of ports: 2

Firmware version: 2.42.5030

Hardware version: 1

Node GUID: 0x0002c903002f3380

System image GUID: 0x0002c903002f3383

Port 1:

State: Active

Physical state: LinkUp

Rate: 56

Base lid: 80

LMC: 1

SM lid: 2

Capability mask: 0x02594868

Port GUID: 0x0002c903002f3381

Link layer: InfiniBand

Port 2:

State: Down

Physical state: Disabled

Rate: 10

Base lid: 0

LMC: 0

SM lid: 0

Capability mask: 0x04010000

Port GUID: 0x0202c9fffe2f3388

Link layer: Ethernet

I have tried with ibdump, but I have the following error:

-E- Failed to set sniffer mode. ibv_create_flow failed: No such file or directory

This problem might be because Flow Steering is not enabled, to enable it:

1. Add the following to /etc/modprobe.d/mlnx.conf file:

options mlx4_core log_num_mgm_entry_size=-1

2. Restart the drivers.

I tried what the error suggest, but still the same. If it is helpful, I don’t have the command ibv_create_flow failed, which the error mentions, and I couldn’t find a way to install it.

I read that it is possible to capture the RDMA packets with ethtool and tcpdump, but that is only since ConnectX-4 adapters.

My final purpose is to test if the MPI traffic is going over InfiniBand stack or if it is going all the way down/up the TCP/IP stack. So, if you know some other method to test it, it will be well received.

Thanks in advance.

2

Hi Hamilton,

For the CX-3 cards, ibdump is the tool to be used as you correctly mentioned that ethtool and tcpdump is for CX-4 and above only. To answer your question about MPI traffic going over IB or IPoIB, then the answer is infiniband. Some collective libraries like HCOLL in HPC-X depends on IPoIB. HCOLL uses rdmacm API to create multicast groups. RDMACM API require IPoIB setup. This is only to create the mcast groups. the actual multicast still uses IB .

You may refer our HPC-X User Manual for further information—> https://docs.mellanox.com/pages/viewpage.action?pageId=12006256

Thanks,

Namrata.

3

Also, based on the output provided, Firmware version: 2.42.5030, seems to be like an OEM card. To confirm, you can run “ibv_devinfo |grep board_id” . If the output shows value starting other than “MT_”, then its an OEM card. Also, our latest driver version is 4.6 and you are running a fairly old one and I see the following known issue in 4.0

Description: ibdump does not work when IPoIB device managed Flow Steering is OFF and at least one of the ports is configured as InfiniBand.

WA: Enable IPoIB Flow Steering and restart the driver. For further information, please refer to MLNX_OFED User Manual section Enable/Disable Flow Steering.

4

What is the error that you receive when running ibdump? Did you restart the driver after enabling Flow Steering?