ibdump unable to capture MPI traffic on MLNX ConnectX-3 FDR HCAs

Infrastructure & Networking Software And Drivers Mellanox OFED
1

Hi all,

I have been trying to to capture MPI traffic using ibdump for one way latency measurements (at the physical layer).

But all ibdump captures are IPoIB packets interleaved with Subnet monitoring packets such as perfqueries from the UFM/SM sweeps.

Specs:

ibdump, 2.0.0-5, built on Aug 21 2013, 13:09:58. GIT Version: 41a4dd2

I have also tried: ibdump, 2.0.0-8, built on Apr 28 2014, 17:06:51. GIT Version: f8e989d

I have also used --mem-mode and --max-burst to prevent packet dropping but with no luck.

Ofed_info: MLNX_OFED_LINUX-2.0-3.0.0.3 (OFED-2.0-3.0.0)

bash-4.1$ ibv_devinfo

hca_id: mlx4_0

transport: InfiniBand (0)

fw_ver: 2.30.3200

node_guid: 0002:c903:0044:e220

sys_image_guid: 0002:c903:0044:e223

vendor_id: 0x02c9

vendor_part_id: 4099

hw_ver: 0x1

board_id: IBM1100110019

phys_port_cnt: 1

port: 1

state: PORT_ACTIVE (4)

max_mtu: 4096 (5)

active_mtu: 4096 (5)

sm_lid: 1

port_lid: 15

port_lmc: 0x00

link_layer: InfiniBand

Using tshark -r sniffer.pcap| grep -i infiniband to parse the .pcap files, I typically get this:

49 6.469888 LID: 35 → LID: 1 InfiniBand 290 PERF (PortCounters)50 6.469951 LID: 35 → LID: 1 InfiniBand 290 PERF (PortCountersExtended)239 36.492380 LID: 1 → LID: 35 InfiniBand 290 PERF (PortCounters)240 36.492388 LID: 1 → LID: 35 InfiniBand 290 PERF (PortCountersExtended)241 36.492477 LID: 35 → LID: 1 InfiniBand 290 PERF (PortCounters)

cat /etc/modprobe.d/mlnx.conf:

blacklist mlx4_core

blacklist mlx4_en

blacklist mlx5_core

blacklist mlx5_ib

I would highly appreciate anyone who can help understand this?

Thanks,

Fabrice