Receive Side Scaling

Infrastructure & Networking Adapters and Cables Ethernet Adapter Cards
1

Hello,

I’m running RHEL 7 acting as a router and performing throughput tests.

My machine is a 6 physical cores Xeon with Connect-X 3 EN dual-port.

The Spirent generator sends packets from many different flows to one interface and the Linux kernel forwards them to the other interface (unidirectional test).

By default there are 4 RX queues created by the driver for each interafce (largest power of 2 not greater then cpu count), so I changed it to 6 to get one queue per CPU.

I disabled the irqbalance daemon and set my custom IRQ mapping to get the interrupts evenly distributed.

However the load is still distributed among 4 softirqs with 100% load and the remaining 2 are sleeping.

The distribution is perfect according to the /proc/interrupts file.

When I turn hyper-threading ON, to have 12 CPUs and 8 RX queues by default, there are still only 4 softirqs using 100%, which seems strange to me.

The /proc/interrupts file shows even distribution among all cores.

The behaviour is the same with RHEL 7 kernel and upstream vanilla 3.18 kernel.

Any advice how to make more softirqs do the work?

2

I am routing, so there is lots of IP/Port pairs. In fact I can see that all queues are used from /proc/interrupts.

The top utility is simply buggy in this case, however perf works correctly and shows correct CPU utilisation.

3

RSS based on IP/PORT pair. How many open sockets exist on the Linux host?

Enabling hyperthreading will add logical CPU, but this logical CPU will share interrupts with base CPU.

4

Seems a bit weird. Have you looked through the Red Hat Knowledge base for relevant info?

5

Hello Justin,

thank you for your answer.

The only useful documentation that I found on the Red Hat sites seems to be 2.4. Interrupt and process binding. https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_MRG/2/html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-General_System_Tuning-Interrupt_and_Process_Binding.html

Otherwise it remains still unanswered.