Mellanox connect-x5 + RDMA debugging

Infrastructure & Networking Software And Drivers Mellanox OFED
1

Hi,

We are working on RDMA data transfer from custom ZynqMP-SOC platform to server PC(OS-CantOS-8 stream) with mellanox connect-x5 NIC card.
We are performing RDMA write operation from ZynqMP-SOC platform to server PC.

We would like to know on debugging techniques available using Mellanox OFED-stack/ethtool applications on the server PC. I have the following questions on the same.

  1. Is it possible to reset hw_counter,ethtool statistics without disconnecting ethernet link. If “YES”, can you please share procedure for the same?
  2. How to find total number of ROCE packets recieved on the Mellanox side?
  3. How to find NACK packets sent from Mellanox?
  4. How to find number of error packets received on Mellanox side?

Could you please help us to understand above questions?

Thanks and Regards,
Abhishek

2

hi

1 NO, it’s impossible
2 In gerneral, ROCE will use different priority, you can check that priorit counter by ethtool -S
3 check hw-counter in /sys/class/infiniband/mlx5_0/ports/1/hw_counters/, if there’s no such counter, it means we do not have such one
4 same as 3

1 Like
3

Hi shim,

Thanks for your input.
How ROCE priority is dependent on number of ROCE packets received? Is there any way on mellanox side to calculate total data transmitted using RDMA write operation?

Best Regards,
Abhishek

4

hi

you can use mlnx_qos set a single priority for roce,
https://docs.nvidia.com/networking/pages/viewpage.action?pageId=89154269

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.