Best way to split single UDP stream across NIC queues in DPDK

Infrastructure & Networking Software And Drivers Application Accelerator Software
1

Hello,

Software in use:
MLNX_OFED 24.10-0.7.0
DPDK 24.07

I’m using a ConnectX-6 VPI card and am wanting to spread a UDP stream’s packets across multiple NIC queues using DPDK’s rte_flow mechanism. Due to the ConnectX-6 limitations my current solution to do this is to setup a new flow rule redirecting the stream to a different NIC queue in a round-robin fashion on a given interval with an increasing priority and periodically destroying all of these flows and starting over again.

For example, for 4 NIC queues:
Interval 1: Create flow Eth / IPv4 / UDP:port → Queue 1, priority 2000
Interval 2: Create flow Eth / IPv4 / UDP:port → Queue 2, priority 1999
…
Interval n: Destroy all prior flows, Create flow Eth / IPv4 / UDP:port → Queue 1, priority 2000

This scheme appears to work but when the flows are destroyed there’s a ‘hiccup’ where there’s an outsized (compared to prior operations) delay when creating new flow rules after the destruction. For example, if the typical flow creation takes 20us then the destruction of 2000 flows would take ~9ms and the next few creations take 750+us.

Because of that, I’d like to see if there’s a way to split up a UDP stream across multiple NIC queues without having to dynamically create and destroy flows.

Some alternatives I’ve considered:

  • Using a RAW item to look at a piece of the UDP payload I know changes and directing the packet to a queue. This failed because the RAW flow item isn’t supported by the mlx5 PMD.
  • Using a RANDOM item to spread the packets out. This failed because the ConnectX-6 doesn’t support HWS as far as I can tell.
  • Updating an existing flow with the new queue flow action. This isn’t possible because the mlx5 PMD doesn’t support the ‘rte_flow_actions_update()’ API.

I cannot split the stream up at the source, and the MAC-IP-UDP tuple is unchanging, making most forms of RSS inapplicable.

In summary I’d like some guidance on how to either manage the cost of destroying old flows or an entirely alternative flow scheme to split this stream up. If the answer is ‘move to ConnectX-6 Dx or newer’ then so be it, but I’d like to make it work on a ConnectX-6 if possible.

Thank you for your time!

2

Since I posted this, I’ve come up with potentially filtering on the IPv4 packet ID field (I don’t care about uniformity, uniqueness, etc just that it changes often), but the flow rules don’t seem to actually filter, the packets get duplicated and sent to both queues.

testpmd equivalent flows used for this scheme (4 RX queues in use):

  • flow create 0 group 0 priority 0 ingress pattern eth / ipv4 packet_id spec 0 packet_id mask 3 / udp dst is 9999 / end actions queue index 0 / end
  • flow create 0 group 0 priority 0 ingress pattern eth / ipv4 packet_id spec 1 packet_id mask 3 / udp dst is 9999 / end actions queue index 1 / end
  • flow create 0 group 0 priority 0 ingress pattern eth / ipv4 packet_id spec 2 packet_id mask 3 / udp dst is 9999 / end actions queue index 2 / end
  • flow create 0 group 0 priority 0 ingress pattern eth / ipv4 packet_id spec 3 packet_id mask 3 / udp dst is 9999 / end actions queue index 3 / end
3

Hi

Too many limitation in your scenario, the only way I can get is try with LPM:
flow create 0 group 1 ingress pattern eth / ipv4 dst is 2.2.2.10 dst mask 255.255.0.0 / end actions queue index 0 / end

But I didn’t know your card’s OPN, so I can not sure this is supported or not on your card.
If not, I think need to move to CX6DX Above.

Thank you
Meng, Shi