I am using Jetson AGX Orin 32GB on my custom device, and encountering extremely high CPU0 load during UDP sending and receiving via the eth0 interface.
1.I will use iperf3 for demonstration below. My custom device is equipped with two Orin units. Device A acts as the receiver, and Device B serves as the sender.
Enter the following command on Device A:iperf3 -s -B 0.0.0.0 -p 5001
Run the following command on Device B:iperf3 -c 192.168.1.102 -u -b 5000M -P 4 -t 10 -p 5001 -l 8000
At this point, I found that the soft interrupt (si) on CPU0 of both Device A and Device B reaches as high as 99.7%, almost fully occupied by soft interrupts. This typically indicates an extremely heavy load from network packet processing.
This issue occurs frequently when my LiDAR device sends UDP packets. It causes severe CPU0 overload and triggers various system problems. Meanwhile, the UDP transmit and receive throughput is also severely limited. Could you please advise on any feasible solutions to resolve this issue?
Then that is different problem which has nothing to do with above patch.
Please check with irqtop tool and see if handling IRQ is causing CPU0 loading high.
Subject: CPU0 high load during UDP traffic – eth0 IRQ stuck on CPU0
When running UDP sending and receiving tasks, CPU0 exhibits abnormally high load.Verification via irqtop -i and cat /proc/interrupts confirms that all receive interrupts of eth0 are strictly pinned to CPU0.
Attempts to adjust IRQ affinity through smp_affinity return an Invalid argument error.
Attached evidence files:
-
irqtop.txt: Real-time interrupt throughput, showing eth0 interrupts are processed exclusively by CPU0.
-
interrupts.txt: Cumulative interrupt statistics since system boot, also showing all interrupts concentrated on CPU0.
Please help confirm whether this is a known IRQ pinning policy, and if any patch is available to distribute the load across CPUs.
irqtop.txt (3.5 KB)
interrupts.txt (22.8 KB)
Summary of findings
On my custom Jetson AGX Orin 32GB board, running UDP send/receive causes CPU0 to be fully loaded.
Using irqtop -i -d 1 and cat /proc/interrupts, I’ve confirmed that all eth0 hardware interrupts (vm0~vm4) are exclusively routed to CPU0, and the resulting softirq processing (%sirq) keeps CPU0 at ~100% utilization while other cores remain nearly idle.
The /proc/interrupts snapshot shows tens of thousands of eth0 interrupts accumulated on CPU0 only.
Attempts to manually change smp_affinity for these IRQs fail with Invalid argument.
Please advise if this is a known IRQ binding policy in current L4T/BSP, and whether there is a patch or recommended method to spread network interrupt handling across multiple cores.
echo 1 > /sys/devices/platform/bus@0/6800000.ethernet/net/eno1/threaded
Could you try this before doing test?
Hello Wayne,
Thank you for your previous suggestion. I tried the following steps and would like to share the results and seek further advice.
System: Custom Jetson AGX Orin 32GB, L4T [35.5.0]
What I did:
-
I could not find the threaded sysfs file under /sys/devices/platform/6810000.ethernet/net/eth0/ (the correct path for my eth0, as indicated by readlink /sys/class/net/eth0). The file simply does not exist.
-
As a workaround, I enabled the global threadirqs kernel parameter by adding it to the APPEND line in /boot/extlinux/extlinux.conf.
-
After reboot, I ran an iperf3 UDP test (client on device A, server on device B) and captured irqtop output on both ends.
Results:
-
Receiver side (irqtop-b.txt): CPU0 softirq dropped from ~100% to ~30%. Rescheduling IPIs spread to other cores (CPU2, etc.), indicating that threadirqs helped distribute the softirq processing.
-
Sender side (irqtop-a.txt): CPU0 is still heavily loaded. eth0.vm0 hardware interrupt rate is ~9700/sec, all on CPU0, driving its %sirq to 65%+. Other CPUs remain nearly idle for this interrupt.
-
smp_affinity modification for these IRQs still returns Invalid argument.
It appears that threadirqs only threads the handler but does not change the hardware IRQ affinity. The root cause is that the eth0 queue interrupts (vm0~vm4) are still hard-wired to CPU0.
My questions:
-
Is there a way to distribute the eth0 queue interrupts across multiple CPUs on Jetson AGX Orin? (e.g., via device tree property, kernel command line, or a per‑queue sysfs interface like threaded or rx_queue_affinity)
-
If not, would enabling RPS (Receive Packet Steering) be a recommended interim workaround for the sender side?
-
Is NVIDIA planning to add a per‑device threaded sysfs node or fix the IRQ affinity lock in a future BSP release?
I have attached the irqtop outputs from both the sender and receiver for your reference.
Regards,
jingyi.LI
You may need to upgrade to Jetpack 6.
The BSP you are using is too old to have any new fix. For example, above path is from Jetpack6 which has threaded node.
nvidia工程师，您好
我使用的是R36.4.3，jetpack6.2 ，Agx Orin
也会有上面相同的问题，UDP占用CPU0负载太高了，我查看中断也是网络软中断基本占满了CPU0。导致使用UDP无法跑满带宽。
我们用的网络节点是MGBE：ethernet@6800000
echo 1 > /sys/devices/platform/bus@0/6800000.ethernet/net/eno1/threaded
这一行命令无效，执行完之后，现象和上面相同。
Could you share the perf number you saw on your UDP test?
测试的命令和上面的问题基本相同，包括调整中断等，通过指定iperf3 -l调整包大小，也只是有一些改善，但是网络软中断占用cpu0过高的问题依然存在。
一次UDP测试，一次TCP测试结果对比：
UDP通信时CPU0负载率：
mpstat输出：
Could you try to adjust the MTU side to 9000 and test?
UDP through put is low and expected. Our QA test also revealed that.
UL and DL also show different result
MTU增大到9000速率也只是有略微提升到2/3G，但是还是达不到实际的5G速率
核心问题还是在软中断的CPU分配。有办法可以把网络TX/RX软中断分配到多个CPU上执行吗？
Hi,
We checked the irqtop but looks like not full IRQ is on CPU0 after we enabled the threaded.
Hello WayneWWW
抱歉回复晚了
MTU=9000这种巨帧传输方式实用性太低了，我们最终还是要用MTU正常的模式。
从你的图片上来看NET_RX/TX都没有中断。
测试观察grep eno1 /proc/interrupts
nvidia@EAORA07BXFI:~$ sudo grep eno1 /proc/interrupts
162: 0 0 0 0 9 0 0 0 0 0 0 0 GICv3 416 Level eno1.common_irq
163: 358012 0 0 0 0 0 0 0 0 0 0 0 GICv3 417 Level eno1.vm0
164: 16787 0 0 0 0 0 0 0 0 0 0 0 GICv3 418 Level eno1.vm1
165: 72 0 0 0 0 0 0 0 0 0 0 0 GICv3 419 Level eno1.vm2
166: 80 0 0 0 0 0 0 0 0 0 0 0 GICv3 420 Level eno1.vm3
167: 35 0 0 0 0 0 0 0 0 0 0 0 GICv3 421 Level eno1.vm4
NET_RX/NET_TX都发生在CPU0上。
我通过一些文件手动指定中断都不起作用：比如：
/proc/irq/
/sys/class/net/eno1/queues/rx-0/rps_cpus等等…
Hi,
I tried to change the smp_affinity of these irq and I can see cpu0 not get interrupt allocated.
You could try it as well.
我的配置不起作用，中断计数仍然是在CPU0上。
您是怎么配置的，可以详细描述一下吗？
commands be like below:
echo 0xffc | sudo tee /proc/irq/163/smp_affinity
这样配置只是会屏蔽掉中断在CPU0和CPU1上执行，所有的中断又会跑到CPU2（ffc最低位）上执行。所有的中断仍是在一个CPU2上执行的
我们最终还是要解决这个UDP速率受限的问题，目前查的结果UDP速率受限的问题，还是指向了网络软中断无法自动分配到不同的CPU上导致的。
Hi,
It is not possible to achieve what you want because the ARM GIC design would only make the IRQ into just one CPU at a time. It is hardware limitation.
