ConnectX-4 Lx: How to configure and verify LRO on Ubuntu 20.04

Infrastructure & Networking DOCA Enterprise Networking
1

I read similar and related topics, but there seems contradiction.

  1. https://forums.developer.nvidia.com/t/cant-seem-to-set-lro-to-on-using-connectx-4-lx-and-ubuntu-20- 04/206013
  2. Ethtool - NVIDIA Docs
  3. ESPCommunity

My config are

Ubuntu20.04

$ uname -a
Linux TK5-3WP091210 5.15.0-71-generic #78~20.04.1-Ubuntu SMP Wed Apr 19 11:26:48 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

$ sudo lshw | grep Mellan -B3 -A11 *-network description: Ethernet interface
product: MT27710 Family [ConnectX-4 Lx]
vendor: Mellanox Technologies
physical id: 0
bus info: pci@0000:3b:00.0
logical name: ens1np0
version: 00
serial: 1c:34:da:55:64:24
capacity: 40Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=5.15.0-71-generic duplex=full firmware=14.29.1016 (MT_2430110032) ip=192.168.0.5 latency=0 link=yes multicast=yes
resources: irq:256 memory:ac000000-adffffff memory:ab000000-ab0fffff memory:ae000000-ae7fffff

$ ethtool -i ens
driver: mlx5_core
version: 5.15.0-71-generic
firmware-version: 14.29.1016 (MT_2430110032)
expansion-rom-version:
bus-info: 0000:3b:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

Experiment scenario
client-server are under same network. Trying to measure TCP throughput using ConnectX-4 Lx interface

$ ethtool -k ens1np0 | grep large
large-receive-offload: off [requested on]

I faced same issue as the first link (large-receive-offload: off [requested on] state. The answer says that LRO is now part of GRO. I thought LRO is used implicitly based on second link.
However, stats says LRO seems not working.

$ ethtool -S ens1np0 | grep lro
rx_lro_packets: 0
rx_lro_bytes: 0
rx0_lro_packets: 0
rx0_lro_bytes: 0
rx1_lro_packets: 0
rx1_lro_bytes: 0
…
rx62_lro_packets: 0
rx62_lro_bytes: 0

For reference, tso is below
test@TK5-3WP091210:~$ ethtool -S ens1np0 | grep tso tx_tso_packets: 24018965307
tx_tso_bytes: 490596498918613
tx_tso_inner_packets: 0
tx_tso_inner_bytes: 0
tx0_tso_packets: 861677417
tx0_tso_bytes: 20428438406096
tx0_tso_inner_packets: 0
tx0_tso_inner_bytes: 0
…

The third link says that ethtool -K to be able to set lro and see the lro stats to verify, but I could not see lro-ed packet.

2

No change after updating driver and firmware. It is now

$ ethtool -i ens1np0
driver: mlx5_core
version: 5.8-3.0.7
firmware-version: 14.32.1010 (MT_2430110032)
expansion-rom-version:
bus-info: 0000:3b:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

3

I found the solution. Setting private flag solves as below.
$ sudo ethtool --set-priv-flags ${iface} rx_striding_rq on

I found dmesg says Disabling LRO, not supported in legacy RQ and the related error and flag in mlx5 driver code.

I believe this is also helpful for the question.
Can’t seem to set LRO to “on” using ConnectX-4 Lx and Ubuntu 20.04 - Infrastructure & Networking / Software And Drivers - NVIDIA Developer Forums

Last question is, what is side effect of enabling rx_striding_rq?