Unable to capture outgoing packets with ibdump

Infrastructure & Networking Software And Drivers Mellanox OFED
1

Hi all,

Appreciate any help with this. I am trying to capture network packets flowing to and from a machine with ConnectX-3 10GbE card using the ibdump utility. Somehow I am only able to see incoming packets in the captured file and not any outgoing packets at all.

I can reproduce the issue by running sockperf ping-pong on the system and capturing with ibdump . (I have tried using sockperf with and without the --load-vma option which doesn’t change anything). All I see are incoming packets.

I use the VMA library so I have to install and use Mellanox OFED, so ibdump remains the only way I can capture packets in this setup.

Useful info:

OS: RHEL 6.4

NIC: ConnectX-3 EN 10GbE

Captured using: ibdump (no options since there is only one device and only one port is up)

Analysed using: Wireshark

$ ibdump -v:

ibdump, 2.0.0-5, built on Aug 21 2013, 13:09:58. GIT Version: 41a4dd2

$ ofed_info

MLNX_OFED_LINUX-2.0-3.0.0 (OFED-2.0-3.0.0)

$ ethtool -i eth2:

driver: mlx4_en

version: 2.1.6 (Aug 27 2013)

firmware-version: 2.30.3110

bus-info: 0000:05:00.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: no

supports-register-dump: no

supports-priv-flags: no

Any inputs are appreciated.

Thanks,

Sid

Request to mods: Please post this question in the right forum if not already so. Thanks.

2

The modprobe line in the answer is missing the “options” keyword. Should be -

/etc/modprobe.d/mlnx.conf:

options mlx4_core log_num_mgm_entry_size=-1

3

Hi,

To use ibdump on ROCE, flow steering must be enabled :

in /etc/modprobe.d/mlnx.conf :

mlx4_core log_num_mgm_entry_size=-1

restart driver

After you set that I recommend you upgrade your driver to the latest 2.3-2. if that doesn’t help, please try the version I attached.

I think tcpdump is also a good option, depends on what you’re trying to filter.

4

Erez,

Is it possible to use ibdump inside a VM with a ConnectX-3 SR-IOV VF based device passed through? Doesn’t seem to be working for us:

[root@hc13 ~]# ethtool -i mlx0

driver: mlx4_en

version: 3.4-1.0.0 (25 Sep 2016)

firmware-version: 2.40.7000

bus-info: 0000:00:06.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: no

supports-register-dump: no

supports-priv-flags: yes

[root@hc13 ~]# ibv_devinfo

hca_id: mlx4_0

transport: InfiniBand (0)

fw_ver: 2.40.7000

node_guid: 0014:0500:8bcf:a171

sys_image_guid: f452:1403:0094:7c93

vendor_id: 0x02c9

vendor_part_id: 4100

hw_ver: 0x1

board_id: MT_1100120019

phys_port_cnt: 1

Device ports:

port: 1

state: PORT_ACTIVE (4)

max_mtu: 4096 (5)

active_mtu: 4096 (5)

sm_lid: 0

port_lid: 0

port_lmc: 0x00

link_layer: Ethernet

[root@hc13 ~]# show_gids

DEV PORT INDEX GID IPv4 VER DEV

mlx4_0 1 0 fe80:0000:0000:0000:3c7a:e8ff:fe84:ac60 V1 mlx0

mlx4_0 1 1 fdfd:eb1a:eb10:e400:3c7a:e8ff:fe84:ac60 V1 mlx0

mlx4_0 1 2 0000:0000:0000:0000:0000:ffff:ac10:e52f 172.16.229.47 V1 mlx0

[root@hc13 ~]# ibdump -d mlx0

Initiating resources …

searching for IB devices in host

-E- IB device mlx0 wasn’t found

-E- failed to create resources

[root@hc13 ~]# ibdump -d mlx4_0

Initiating resources …

searching for IB devices in host

-E- Failed to open mst device mlx4_0: Operation not supported

-E- failed to create resources