ConnectX-5 RSS maps same hash to different cores

[yanniszark]

Hi all! I am trying to setup a Mellanox card’s RSS and get seemingly illogical results.

TLDR: I want to configure a NIC’s RSS so that it sends all UDP packets to a single CPU core. (needed for some research benchmarks / experiments I am running)

Problem: Although I set the RSS hash function to use only IP destination address, packets from different client machines go to different cores. Packets from the same machine go to the same core.

Steps:

1. I set RSS hashing to only consider IP address destination for input to the hashing function: sudo ethtool -N ens1f0np0 rx-flow-hash udp4 d
2. Validate the setting with sudo ethtool -n ens1f0np0 rx-flow-hash udp4

   $ sudo ethtool -n ens1f0np0 rx-flow-hash udp4
   UDP over IPV4 flows use these fields for computing Hash flow key:
   IP DA
   

3. Send multiple packets from 1 machine. Check the eBPF (XDP) trace logs to validate they use the same core.
4. Send packets from another client machine. Check the eBPF (XDP) trace logs to validate they use another core.

Any idea what might be going on?
Video for reference: Mellanox ConnectX-5 RSS not working as expected - YouTube

[xiaofengl]

The rx-flow-hash also calculate on L4 port,

L4 bytes 0 & 1 [TCP/UDP src port]
L4 bytes 2 & 3 [TCP/UDP dst port]

Also you can try set “n”,

ethtool -N ens1f0np0 rx-flow-hash udp4 d n

       rx-flow-hash tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6
       m|v|t|s|d|f|n|r...
              Configures the hash options for the specified flow
              type.

              m   Hash on the Layer 2 destination address of the rx packet.
              v   Hash on the VLAN tag of the rx packet.
              t   Hash on the Layer 3 protocol field of the rx packet.
              s   Hash on the IP source address of the rx packet.
              d   Hash on the IP destination address of the rx packet.
              f   Hash on bytes 0 and 1 of the Layer 4 header of the rx packet.
              n   Hash on bytes 2 and 3 of the Layer 4 header of the rx packet.
              r   Discard all packets of this flow type. When this option is
                  set, all other options are ignored.

[xiaofengl]

What OS/ethtool version?
Chcek the ring and hash rss enable by,

ethtool -l eth_dev,
ethtool -x eth_dev

And could you check set rules like,

ethtool -N eth_dev flow-type udp4 src-ip s-ip dst-ip d-ip action 1

man page of ethtool

https://man7.org/linux/man-pages/man8/ethtool.8.html

action N will map to RXQ then RXQ map to CPU core

       action N
              Specifies the Rx queue to send packets to, or some
              other action.

              -1            Drop the matched flow
              -2            Use the matched flow as a Wake-on-LAN filter
              0 or higher   **Rx queue** to route the flow