how to capture Roce with tcpdump and ethtool in connectX4 and centOS 7.4 and OFED 5.1?

I have run ethtool --set-priv-flags enp3s0f1 sniffer on but it shows

ethtool: bad command line argument(s)

For more information run ethtool -h

then I run

ethtool --show-priv-flags enp3s0f1

it shows

Private flags for enp3s0f1:

rx_cqe_moder : on

tx_cqe_moder : on

rx_cqe_compress : on

tx_cqe_compress : on

rx_striding_rq : on

rx_no_csum_complete: on

dropless_rq : on

per_channel_stats : on

hw_lro : on

tx_xdp_hw_checksum : on

skb_xmit_more : on

There is no sniffer at all !!! And would you like to tell me how to solve this problem?

Hi Luo,

We removed support for Offloaded Traffic Sniffer feature, but we can suggest using the docker-container solution to use RDMA devices, in order to capture and analyze RDMA packets using tcpdump.

Requirements

  • CentOS/RHEL 7.x / 8.x

  • Upstream Kernel must be higher than 4.9

​- ConnectX-3/4/5

Installation instructions:

  1. Install OS that is compatible with kernel 4.9 and above

  2. Install Upstream kernel starting from version 4.9 support sniffing RDMA(RoCE) traffic

  3. Yum install docker

  4. Docker pull mellanox/tcpdump-rdma

  5. Service docker start

  6. Docker run -it -v /dev/infiniband:/dev/infiniband -v /tmp/traces:/tmp/traces --net=host --privileged mellanox/tcpdump-rdma bash

  7. Install MFT 4.9

  8. Install perftest package from MLNX_OFED RPMS directory

  9. Capture RoCE packets with the following:

tcpdump -i mlx5_0 -s 0 -w /tmp/traces/capture1.pcap

or

tcpdump -i mlx4_0 -s 0 -w /tmp/traces/capture1.pcap​

  1. Run ib_write_bw test , as below:

Server : ib_write_bw -d mlx5_0 -a -F

Client: ib_write_bw -a -F <Server_ip>

  1. Open the pcap through wireshark to verify

Regards,

Chen

Where can one find the upstream kernel necessary for this Docker container approach?