[Jetson AGX Orin] Intel Network Card RX Interrupts Locked to CPU0 Despite Affinity Settings

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
1

Dear NVIDIA Support Team,

I am encountering issues with interrupt distribution on my Jetson AGX Orin when using an Intel 82599ES 10-Gigabit Network Card. All RX interrupts are consistently routed to CPU0, even after attempting various affinity configuration methods.

System Configuration

  • Hardware:
    • Jetson AGX Orin Dev Kit (JetPack 6.0 rev2 / L4T R36.2.1)
    • PCIe Card: Intel 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
    • PCIe Slot: x16 Gen3
  • Software:
    • Kernel: 5.15.136-tegra (PREEMPT RT)
    • Driver: ixgbe v5.15.136-tegra
    • Firmware: 0x8000091d, 1.1825.0

Problem Symptoms

  1. Interrupt Concentration:
    All RX queues (eth1-TxRx-0 to eth1-TxRx-11) are handled by CPU0, as shown in /proc/interrupts:
302:      89874          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI 671612928 Edge      eth1-TxRx-0
303:   18512388          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI 671612929 Edge      eth1-TxRx-1
304:    2211057          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI 671612930 Edge      eth1-TxRx-2
305:   17802941          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI 671612931 Edge      eth1-TxRx-3
306:     308635          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI 671612932 Edge      eth1-TxRx-4
307:     660103          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI 671612933 Edge      eth1-TxRx-5
308:     371502          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI 671612934 Edge      eth1-TxRx-6
309:      95708          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI 671612935 Edge      eth1-TxRx-7
310:     251272          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI 671612936 Edge      eth1-TxRx-8
311:     829679          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI 671612937 Edge      eth1-TxRx-9
312:     201855          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI 671612938 Edge      eth1-TxRx-10
313:     156559          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI 671612939 Edge      eth1-TxRx-11
314:          3          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI 671612940 Edge      eth1
  1. Failed Affinity Configuration:
    Attempts to set CPU affinity via /proc/irq or /sys/class/net have failed:
$ echo 1 | sudo tee /proc/irq/302/smp_affinity_list
tee: /proc/irq/302/smp_affinity_list: Invalid argument

$ echo 1 | sudo tee /proc/irq/302/smp_affinity
tee: /proc/irq/302/smp_affinity: Invalid argument
  1. RPS Configuration Ineffective:
    Configuring RPS (Receive Packet Steering) did not redistribute interrupts:
$ cat /sys/class/net/eth1/queues/rx-*/rps_cpus
001
008
016
002
004
008
016
032
064
128
256
512

Despite configuration, interrupts remain on CPU0.

Troubleshooting Steps

  1. PCIe Device Verification:
$ lspci -s 0005:01:00.0
0005:01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
  1. Multi-Queue Support:
$ ethtool -l eth1
  Channel parameters for eth1:
  Pre-set maximums:
  RX:             n/a
  TX:             n/a
  Other:          1
  Combined:       12
  Current hardware settings:
  RX:             n/a
  TX:             n/a
  Other:          1
  Combined:       12
  1. Interrupt Affinity Settings:
    All IRQs show affinity mask 0-11 (all CPUs), but interrupts still go to CPU0:
$ cat /proc/irq/{302..314}/smp_affinity_list
0-11
0-11
0-11
0-11
0-11
0-11
0-11
0-11
0-11
0-11
0-11
0-11
0-11

Questions

  1. Why are interrupts locked to CPU0 even when:
  • smp_affinity_list is set to 0-11?
  • RPS is configured to distribute to other CPUs?
  1. How can I force interrupts to be distributed across CPU1-CPU11?
  • Are there Tegra-specific kernel parameters or driver flags for this?
  1. Is there a hardware/software compatibility issue between the pcie network card and Jetson AGX Orin?

Thank you for your assistance!

2

yes, it is a known issue and will not fix.

You could apply the GIC MSI patch here to unblock the GIC MSI and your interrupt could be routed to other CPU cores.

3

Here’s my operation process on the Jetson device:



sudo apt update
sudo apt dist-upgrade
sudo apt install nvidia-l4t-openwfd nvidia-l4t-vulkan-sc-dev nvidia-l4t-vulkan-sc-samples nvidia-l4t-vulkan-sc-sdk nvidia-l4t-vulkan-sc nvidia-l4t-jetsonpower-gui-tools

mkdir -p /nvme/ff/mnt/r363/Linux_for_Tegra/rootfs/boot
export KERNEL_HEADERS=/nvme/ff/mnt/r363/Linux_for_Tegra/source/kernel/kernel-jammy-src
export CROSS_COMPILE=/usr/bin/aarch64-linux-gnu-
export INSTALL_MOD_PATH=/nvme/ff/mnt/r363/Linux_for_Tegra/rootfs

tar -xjf public_sources.tbz2 
cd /nvme/ff/mnt/r363/Linux_for_Tegra/source
tar xf kernel_src.tbz2
tar xf kernel_oot_modules_src.tbz2
tar xf nvidia_kernel_display_driver_source.tbz2

zcat /proc/config.gz > /nvme/ff/mnt/r363/Linux_for_Tegra/source/kernel/kernel-jammy-src/.config

cd /nvme/ff/mnt/r363/Linux_for_Tegra/source/kernel/kernel-jammy-src
patch -p1 < /home/ff/nv_src/msi-patch-kernel.patch

cd /nvme/ff/mnt/r363/Linux_for_Tegra/source/hardware/nvidia/t23x/nv-public
patch -p1 < /home/ff/nv_src/msi-patch-dt.patch


cd /nvme/ff/mnt/r363/Linux_for_Tegra/source
make -C kernel 
sudo -E make install -C kernel

make modules
sudo -E make modules_install

make dtbs

cp ./nvidia-oot/device-tree/platform/generic-dts/dtbs/tegra234-p3737-0000+p3701-0005-nv.dtb /boot/dtb/ff_tegra234-p3737-0000+p3701-0005-nv.dtb
cp ./kernel/kernel-jammy-src/arch/arm64/boot/Image /boot/Image
cp -r /nvme/ff/mnt/r363/Linux_for_Tegra/rootfs/lib/modules/5.15.136-tegra  /lib/modules/


ff@ubuntu:~$ cat /boot/extlinux/extlinux.conf
TIMEOUT 30
DEFAULT primary

MENU TITLE L4T boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      FDT /boot/dtb/ff_tegra234-p3737-0000+p3701-0005-nv.dtb
      INITRD /boot/initrd
      APPEND ${cbootargs} root=PARTUUID=414fd07d-bcf2-47ca-bd64-6f3e420fabe3 rw rootwait rootfstype=ext4 mminit_loglevel=4 console=ttyTCU0,115200 console=ttyAMA0,115200 firmware_class.path=/etc/firmware fbcon=map:0 net.ifnames=0 nospectre_bhb video=efifb:off console=tty0 nv-auto-config

# When testing a custom kernel, it is recommended that you create a backup of
# the original kernel and add a new entry to this file so that the device can
# fallback to the original kernel. To do this:
#
# 1, Make a backup of the original kernel
#      sudo cp /boot/Image /boot/Image.backup
#
# 2, Copy your custom kernel into /boot/Image
#
# 3, Uncomment below menu setting lines for the original kernel
#
# 4, Reboot

LABEL backup
      MENU LABEL backup kernel
      LINUX /boot/Image_bak
      INITRD /boot/initrd
      APPEND ${cbootargs} root=PARTUUID=414fd07d-bcf2-47ca-bd64-6f3e420fabe3 rw rootwait rootfstype=ext4 mminit_loglevel=4 console=ttyTCU0,115200 console=ttyAMA0,115200 firmware_class.path=/etc/firmware fbcon=map:0 net.ifnames=0 nospectre_bhb video=efifb:off console=tty0 nv-auto-config


 When starting the Jetson with the compiled product image, the device fails to boot normally, and the screen outputs the error messages shown in the attached picture.

111
1111602×894 194 KB