No sniffer flag using ethtool --show-priv-flags

Infrastructure & Networking Adapters and Cables Firmware Tools
1

I have a Mellanox X-5 NIC. I was trying to use tcpdump to capture the traffic on mlx5_0 port.

But I don’t see the sniffer flag when using “ethtool --show-priv-flag mlx5_0”, so I can’t turn on the sniffer to capture the traffic.

Here is few more info about the system we have

$ ethtool --version

ethtool version 4.8

$ tcpdump --version

tcpdump version 4.9.2

libpcap version 1.5.3

OpenSSL 1.0.2k-fips 26 Jan 2017

$ ethtool --show-priv-flags mlx0

Private flags for mlx0:

rx_cqe_moder : on

tx_cqe_moder : off

rx_cqe_compress : off

tx_cqe_compress : off

rx_striding_rq : on

rx_no_csum_complete: off

xdp_tx_mpwqe : on

dropless_rq : off

per_channel_stats : on

hw_lro : off

tx_xdp_hw_checksum : off

skb_xmit_more : off

$uname

Linux ares 3.10.0-1127.13.1.el7.x86_64 #1 SMP Tue Jun 23 15:46:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

OFED

MLNX_OFED_LINUX-5.1-0.6.6.0-rhel7.8-x86_64

Thanks.

2

Hi ,

Please note that we removed support for Offloaded Traffic Sniffer feature, but we can suggest using the docker-container solution to use RDMA devices, in order to capture and analyze RDMA packets using tcpdump.

Requirements

  • CentOS/RHEL 7.x / 8.x

  • Upstream Kernel must be higher than 4.9

​- ConnectX-3/4/5

Installation instructions:

  1. Install OS that is compatible with kernel 4.9 and above

  2. Install Upstream kernel starting from version 4.9 support sniffing RDMA(RoCE) traffic

  3. Yum install docker

  4. Docker pull mellanox/tcpdump-rdma

  5. Service docker start

  6. Docker run -it -v /dev/infiniband:/dev/infiniband -v /tmp/traces:/tmp/traces --net=host --privileged mellanox/tcpdump-rdma bash

  7. Install MFT 4.9

  8. Install perftest package from MLNX_OFED RPMS directory

  9. Capture RoCE packets with the following:

tcpdump -i mlx5_0 -s 0 -w /tmp/traces/capture1.pcap

or

tcpdump -i mlx4_0 -s 0 -w /tmp/traces/capture1.pcap​

  1. Run ib_write_bw test , as below:

Server : ib_write_bw -d mlx5_0 -a -F

Client: ib_write_bw -a -F <Server_ip>

  1. Open the pcap through wireshark to verify

Thanks,

Samer

1 Like
3

I got this errs when capture RoCE packets by docker:
root@test:/# tcpdump --version
tcpdump version 4.99.1
libpcap version 1.10.1 (with TPACKET_V3)
OpenSSL 3.0.2 15 Mar 2022
root@test:/# tcpdump -i mlx5_0
tcpdump: mlx5_0: No such device exists
(SIOCGIFHWADDR: No such device)
How can i solove this problem？

4

Depending on the type of RDMA card you have installed mlx5_0 may or may not be the correct device. Running “tcpdump -D” will show you the list of available interfaces to use with tcpdump. Look for those listed as “RDMA Sniffer”

For example:
tcpdump -D |grep -i rdma
24.rocep4s0 (RDMA sniffer)
25.rocep7s0f0 (RDMA sniffer)
26.rocep7s0f1 (RDMA sniffer)
27.rocep10s0f0 (RDMA sniffer)
28.rocep10s0f1 (RDMA sniffer)
29.rocep33s0f0 (RDMA sniffer)
30.rocep33s0f1 (RDMA sniffer)
31.rocep36s0f0 (RDMA sniffer)
32.ibp36s0f1 (RDMA sniffer)

If there are no interfaces with RDMA support and you do have an RDMA enabled card installed, then most likely the libpcap version you are using was not built with RDMA support.

For example libpcap version 1.10.1 installed with Ubuntu 22.04 does not enable RDMA support by default. If this is the case then recompiling libpcap using the --enable-rdma flag or finding a newer version that was compiled correctly should resolve the issue.

thanks,
-matt