not able to capure packets with ibdump

Infrastructure & Networking Adapters and Cables Ethernet Adapter Cards
1

Hi,

I am using 10G Mellanox card. I have generated traffic with krping. krping is success. But ibdump packets counts are not incrementing. Am I missing some thing?

Do I need to use any specific MLNX OFED stack for 10G card?

Currently I am using MNPA19-XTR. In the web I got wireshark to capture RDMA packets for windows.

For Linux any specific wireshark package available?

Thanks

Rama

2

Thanks Talat, Your inputs are use full us to proceed further

3

Hi Rama,

could you please tell me the device type (ConnectX3 or ConnectX4) and the Link type( Ethernet or Infiniband )

I guess that you use Ethernet and you want to capture RoCE packets.

you can see HowTo Enable, Verify and Troubleshoot RDMA https://community.mellanox.com/s/article/howto-enable--verify-and-troubleshoot-rdma

To be able to use ibdump, you need to enable flow steering.

a. To enable flow-steering:

  • add/create /etc/modprobe.d/mlx4.conf file and add this line:

options mlx4_core log_num_mgm_entry_size=-1

  • restart the driver

#/etc/init.d/openibd restart

(Make sure the you still have IP configured on the interface)

b. Run some RDMA traffic (e.g. ib_send_bw or similar above)

c. run ibdump to create *.pcap file.

# ibdump

Initiating resources …

searching for IB devices in host

Port active_mtu=1024

MR was registered with addr=0x61b8f0, lkey=0x10010d00, rkey=0x10010d00, flags=0x1

------------------------------------------------

Device : “mlx4_0”

Physical port : 1

Link layer : Ethernet

Dump file : sniffer.pcap

Sniffer WQEs (max burst size) : 4096

------------------------------------------------

Ready to capture (Press ^c to stop):

Captured: 82133 packets, 88626986 bytes

Interrupted (signal 2) - exiting …

Captured: 82133 packets, 88626986 bytes

# ls

sniffer.pcap

#

d. Open the pcap file using wireshark (or similar program)

In this case RoCE V1 was used (ethertype 0x8915)

Thanks,

Talat

4

Hi Talat,

My device is Connect X2, 10G Ethernet. It is Ethernet type. I can able to capture packets now with wireshark. But the packets are ROCV1 packets. I have generated traffic udaddy, krping. From web I found to get traffic ROCEV2 I have to insert driver mlx4_core with parameter roce_mode=2. But to get this driver option I need to install MLNX_OFED stack.

Any tools are available to get ROCEv2 packets

Thanks

Rama

5

Hi,

As we see in the Mellanox Webpage, ConnectX2 devices doesn’t support RoCE V2, see http://www.mellanox.com/page/products_dyn?product_family=79&mtag=roce http://www.mellanox.com/page/products_dyn?product_family=79&mtag=roce

"

The Mellanox NICs with its hardware offload support takes advantage of this efficient RDMA transport (InfiniBand) services over Ethernet to deliver ultra-low latency for performance-critical and transaction intensive applications such as financial, data base, storage, and content delivery networks.

Mellanox adapter cards (ConnectX-3 Pro and onward) support RoCE which allows RDMA to be used across both L2 and L3 networks thanks to IP and UDP headers.

"