Hi,
We run a mix of ConnectX-5, ConnectX-6 VPI, and ConnectX-6 DX network cards. Recently, we moved to upgrade the DOCA OFED and firmware versions of these devices. After doing so, it was observed that in the case of IPv6 tunnel packets (eg. IPv4 inside of IPv6), RSS is being performed using the outer IP header fields only. I’ve managed to isolate it to specific firmware versions. More specifically, we tested the following:
| - 20.38.1002 |
|---|
| - 20.38.1900 |
| - 20.39.1002 |
| - 20.39.2048 |
| - 20.39.4082 |
| - 20.43.2026 |
| - 20.43.2566 |
And found that anything later than 20.38.1900 was exhibiting the problematic behaviour. The OFED version remained the same during these tests. On any version where RSS is not working on the inner IP header fields, ethtool -k $IFACE shows that the following offloads are suddenly disabled and cannot be re-enabled:
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
We’ve reproduced the issue by generating IPv4-in-IPv6 packets from another device (where the inner header fields varied but the outer header remained the same) and using a combination of ethtool statistics and per-queue packet captures (using rxtxcpu) to verify that all traffic was landing on a single receive queue. As soon as we revert to 20.38.1900 or earlier, tx-ipxip{4|6}-segmentationoffload automatically re-enables itself and inner RSS functions properly. This issue does not seem to affect inner RSS for GRE tunnel packets.
To rule it out having anything to do with our configuration, I’ve provisioned a fresh installation of Ubuntu 25.04 Server running doca-ofed from DOCA 3.1.0 and the latest available firmware version.
# cat /proc/cpuinfo | grep 'model name' | head -n 1
model name : AMD EPYC 7713 64-Core Processor
# uname -a
Linux test 6.14.0-29-generic #29-Ubuntu SMP PREEMPT_DYNAMIC Thu Aug 7 18:32:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
mlxfwmanager --online
Querying Mellanox devices firmware ...
Device #1:
----------
Device Type: ConnectX6
Part Number: MCX653106A-ECA_Ax
Description: ConnectX-6 VPI adapter card; H100Gb/s (HDR100; EDR IB and 100GbE); dual-port QSFP56; PCIe3.0 x16; tall bracket; ROHS R6
PSID: MT_0000000224
PCI Device Name: 0000:c1:00.0
Base MAC: 0c42a153a6a2
Base GUID: 0c42a1030053a6a2
Versions: Current Available
FW 20.43.2566 20.43.1014
PXE 3.7.0500 3.7.0500
UEFI 14.37.0013 14.36.0016
Status: Up to date
# ethtool -i enp193s0f0np0
driver: mlx5_core
version: 25.07-0.9.7
firmware-version: 20.43.2566 (MT_0000000224)
expansion-rom-version:
bus-info: 0000:c1:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
# ethtool -i enp193s0f0np0
driver: mlx5_core
version: 25.07-0.9.7
firmware-version: 20.43.2566 (MT_0000000224)
expansion-rom-version:
bus-info: 0000:c1:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
root@test:/home/ubuntu# ethtool -k enp193s0f0np0
Features for enp193s0f0np0:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: on
tx-gso-list: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: on [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: on
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]
# ethtool --show-priv-flags enp193s0f0np0
Private flags for enp193s0f0np0:
rx_cqe_moder : off
tx_cqe_moder : off
rx_cqe_compress : off
tx_cqe_compress : off
rx_striding_rq : on
rx_no_csum_complete: off
xdp_tx_mpwqe : on
skb_tx_mpwqe : on
tx_port_ts : off
dropless_rq : off
per_channel_stats : on
tx_xdp_hw_checksum : off
# ethtool -x enp193s0f0np0
RX flow hash indirection table for enp193s0f0np0 with 63 RX ring(s):
0: 0 1 2 3 4 5 6 7
8: 8 9 10 11 12 13 14 15
16: 16 17 18 19 20 21 22 23
24: 24 25 26 27 28 29 30 31
32: 32 33 34 35 36 37 38 39
40: 40 41 42 43 44 45 46 47
48: 48 49 50 51 52 53 54 55
56: 56 57 58 59 60 61 62 0
64: 1 2 3 4 5 6 7 8
72: 9 10 11 12 13 14 15 16
80: 17 18 19 20 21 22 23 24
88: 25 26 27 28 29 30 31 32
96: 33 34 35 36 37 38 39 40
104: 41 42 43 44 45 46 47 48
112: 49 50 51 52 53 54 55 56
120: 57 58 59 60 61 62 0 1
128: 2 3 4 5 6 7 8 9
136: 10 11 12 13 14 15 16 17
144: 18 19 20 21 22 23 24 25
152: 26 27 28 29 30 31 32 33
160: 34 35 36 37 38 39 40 41
168: 42 43 44 45 46 47 48 49
176: 50 51 52 53 54 55 56 57
184: 58 59 60 61 62 0 1 2
192: 3 4 5 6 7 8 9 10
200: 11 12 13 14 15 16 17 18
208: 19 20 21 22 23 24 25 26
216: 27 28 29 30 31 32 33 34
224: 35 36 37 38 39 40 41 42
232: 43 44 45 46 47 48 49 50
240: 51 52 53 54 55 56 57 58
248: 59 60 61 62 0 1 2 3
RSS hash key:
cc:ef:b1:6d:e3:2e:ef:f0:f8:fe:90:f4:bd:58:46:c8:6b:57:9b:50:01:4b:29:b8:ca:e4:af:15:17:08:95:60:c8:39:ff:58:47:6e:e6:90
RSS hash function:
toeplitz: on
xor: off
crc32: off
RSS input transformation:
symmetric-xor: off
# dmesg -T | grep mlx
[Tue Sep 9 13:31:47 2025] mlx_compat: loading out-of-tree module taints kernel.
[Tue Sep 9 13:31:47 2025] mlx_compat: module verification failed: signature and/or required key missing - tainting kernel
[Tue Sep 9 13:31:47 2025] mlx5_core 0000:c1:00.0: firmware version: 20.43.2566
[Tue Sep 9 13:31:47 2025] mlx5_core 0000:c1:00.0: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link)
[Tue Sep 9 13:31:48 2025] mlx5_core 0000:c1:00.0: E-Switch: Total vports 2, per vport: max uc(128) max mc(2048)
[Tue Sep 9 13:31:48 2025] mlx5_core 0000:c1:00.0: Flow counters bulk query buffer size increased, bulk_query_len(8)
[Tue Sep 9 13:31:48 2025] mlx5_core 0000:c1:00.0: Port module event: module 0, Cable plugged
[Tue Sep 9 13:31:48 2025] mlx5_core 0000:c1:00.0: mlx5_pcie_event:325:(pid 496): PCIe slot power capability was not advertised.
[Tue Sep 9 13:31:48 2025] mlx5_core 0000:c1:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)
[Tue Sep 9 13:31:48 2025] mlx5_core 0000:c1:00.1: firmware version: 20.43.2566
[Tue Sep 9 13:31:48 2025] mlx5_core 0000:c1:00.1: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link)
[Tue Sep 9 13:31:48 2025] mlx5_core 0000:c1:00.1: Flow counters bulk query buffer size increased, bulk_query_len(8)
[Tue Sep 9 13:31:48 2025] mlx5_core 0000:c1:00.1: Port module event: module 1, Cable unplugged
[Tue Sep 9 13:31:48 2025] mlx5_core 0000:c1:00.1: mlx5_pcie_event:325:(pid 496): PCIe slot power capability was not advertised.
[Tue Sep 9 13:31:48 2025] mlx5_core 0000:c1:00.0 enp193s0f0np0: renamed from eth0
[Tue Sep 9 13:32:03 2025] mlx5_core 0000:c1:00.0 enp193s0f0np0: Link up
[Tue Sep 9 13:32:03 2025] mlx5_core 0000:c1:00.0 mlx5_0: Port: 1 Link ACTIVE
[Tue Sep 9 13:32:03 2025] mlx5_core 0000:c1:00.1 ibp193s0f1: renamed from ib0
# ethtool -n enp193s0f0np0 rx-flow-hash tcp4;
TCP over IPV4 flows use these fields for computing Hash flow key:
IP SA
IP DA
L4 bytes 0 & 1 [TCP/UDP src port]
L4 bytes 2 & 3 [TCP/UDP dst port]
# ethtool -n enp193s0f0np0 rx-flow-hash udp4;
UDP over IPV4 flows use these fields for computing Hash flow key:
IP SA
IP DA
L4 bytes 0 & 1 [TCP/UDP src port]
L4 bytes 2 & 3 [TCP/UDP dst port]
# ethtool -n enp193s0f0np0 rx-flow-hash tcp6;
TCP over IPV6 flows use these fields for computing Hash flow key:
IP SA
IP DA
L4 bytes 0 & 1 [TCP/UDP src port]
L4 bytes 2 & 3 [TCP/UDP dst port]
# ethtool -n enp193s0f0np0 rx-flow-hash udp6;
UDP over IPV6 flows use these fields for computing Hash flow key:
IP SA
IP DA
L4 bytes 0 & 1 [TCP/UDP src port]
L4 bytes 2 & 3 [TCP/UDP dst port]
I believe that this is a possible regression and would greatly appreciate assistance with the matter.