The RPS/RSS seems not take effect in platform Xavier/Orin

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
1

I tried to dispatch udp packet to multi-CPU over RPS/RSS, but it seems not available.
The steps are as follow:

  1. list all net rx queues:
    root@nvidia-orin:/# ll /sys/class/net/eth*/queues/rx-*/rps_cpus
    -rw-r–r-- 1 root root 4096 6月 23 20:47 /sys/class/net/eth0/queues/rx-0/rps_cpus
    -rw-r–r-- 1 root root 4096 6月 23 20:47 /sys/class/net/eth0/queues/rx-1/rps_cpus
    -rw-r–r-- 1 root root 4096 6月 23 20:47 /sys/class/net/eth0/queues/rx-2/rps_cpus
    -rw-r–r-- 1 root root 4096 6月 23 20:47 /sys/class/net/eth0/queues/rx-3/rps_cpus
    -rw-r–r-- 1 root root 4096 6月 23 20:47 /sys/class/net/eth0/queues/rx-4/rps_cpus
    -rw-r–r-- 1 root root 4096 6月 23 20:47 /sys/class/net/eth0/queues/rx-5/rps_cpus
    -rw-r–r-- 1 root root 4096 6月 23 20:47 /sys/class/net/eth0/queues/rx-6/rps_cpus
    -rw-r–r-- 1 root root 4096 6月 23 20:47 /sys/class/net/eth0/queues/rx-7/rps_cpus
    -rw-r–r-- 1 root root 4096 6月 23 20:47 /sys/class/net/eth0/queues/rx-8/rps_cpus
    -rw-r–r-- 1 root root 4096 6月 23 20:47 /sys/class/net/eth0/queues/rx-9/rps_cpus

  2. enable mult-queue (is seems the following command runs failed):
    root@nvidia-orin:/# ethtool -L eth0 rx 10
    Cannot get device channel parameters: Operation not supported

  3. run following shell script “rps_cpus.sh” as root user:

#!/bin/bash

Enable RPS (Receive Packet Steering)

rfc=4096
cc=$(grep -c processor /proc/cpuinfo)
rsfe=$(echo $cc*$rfc | bc)
echo “rsfe: $rsfe”
sysctl -w net.core.rps_sock_flow_entries=$rsfe
for fileRps in $(ls /sys/class/net/eth*/queues/rx-/rps_cpus)
do
echo fff > $fileRps
done
for fileRfc in $(ls /sys/class/net/eth/queues/rx-/rps_flow_cnt)
do
echo $rfc > $fileRfc
done
tail /sys/class/net/eth/queues/rx-*/{rps_cpus,rps_flow_cnt}

  1. detect the network rx irq by following command:

root@nvidia-orin:/# cat /proc/interrupts | grep eth0
56: 0 0 0 0 4 0 0 0 0 0 0 0 GICv3 416 Level eth0.common_irq
57: 7353 0 0 0 0 0 0 0 0 0 0 0 GICv3 417 Level eth0.vm0
58: 0 0 0 0 0 0 0 0 0 0 0 0 GICv3 418 Level eth0.vm1
59: 0 0 0 0 0 0 0 0 0 0 0 0 GICv3 419 Level eth0.vm2
60: 7 0 0 0 0 0 0 0 0 0 0 0 GICv3 420 Level eth0.vm3
61: 0 0 0 0 0 0 0 0 0 0 0 0 GICv3 421 Level eth0.vm4

  1. It seems all rx irq still processed by CPU0. Is there any good idea in this case? Thanks in advance.
3

I don’t have answer for this RPS/RSS issue, may other developers help to share experiences.

4

Thanks anyway.

5

@kame_lqm

Could you confirm whether you kernel has enabled the CONFIG_RPS.

CONFIG_NET_NCSI is not set

CONFIG_RPS=y
CONFIG_RFS_ACCEL=y

6

Thanks for your replay. And it is already enabled RPS in Kconfig file.

image
image950×632 13.9 KB

7

image
image735×40 2.14 KB

image

8

Hi,
I have the same problem on xavier (5.10.104-tegra #1 SMP PREEMPT Wed Aug 10 20:17:07 PDT 2022 aarch64 aarch64 aarch64 GNU/Linux) as @kame_lqm.

I’m trying to get RPS & RFS to work so that interrupts for any of the network interfaces (ethX, can or the wifi) are handled by multiple cores, not just solitary CPU0.

Running watch -n1 -d cat /proc/interrupts and updating the parameters thus:

echo "ff" | sudo tee /sys/class/net/<device>/queues/rx-0/rps_cpus
echo "2048" | sudo tee /sys/class/net/<device>/queues/rx-0/rps_flow_cnt
echo "32768" | sudo tee /proc/sys/net/core/rps_sock_flow_entries

shows no affect - solitary CPU is processing the packets while the rest of the 7 cores are sitting on zero.

Seems kernel has this enabled:

/usr/src/linux-headers-5.10.104-tegra-ubuntu20.04_aarch64/kernel-5.10/net/Kconfig

config RPS
        bool
        depends on SMP && SYSFS
        default y

config RFS_ACCEL
        bool
        depends on RPS
        select CPU_RMAP
        default y

config XPS
        bool
        depends on SMP
        default y

and:

kernel-5.10/.config:
# CONFIG_NET_NCSI is not set
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_XPS=y

Could someone please help with getting this to work?
We have a high volume of UDP traffic that we believe is being delivered fine by our network, but the bottleneck seems to be on the processing of the packets by the OS.