Adding processor affinity to GPIO interrupt (using GIC)

Robotics & Edge Computing Jetson Systems Jetson AGX Xavier
1

gpio_irq.txt (3.4 KB)
Continuing the discussion from Adjusting GPIO IRQ affinity in GIC:

Device: Jetson Xavier 32Gb
JetPack: R32 (release), REVISION: 7.1, GCID: 29818004, BOARD: t186ref, EABI: aarch64, DATE: Sat Feb 19 17:07:00 UTC 2022

I am using GPIO417 as interrupt source. My goal is to assign GPIO417 to Core-0 (or any other core) using affinity.

Details:
I have written a Linux Kernel Module (which is attached) which processes the interrupt and signals user space application.

When I checked, my IRQ number is 330. Also you can see that I am able to register the interrupt (116 times in below snapshot)

nvidia@ubuntu:~$ cat /proc/interrupts | grep my_gpio
 330:        116          0          0          0          0          0          0          0 ff% tegra-gpio-129 Edge      my_gpio_irq_signal

If I try to assign affinity to interrupt, I get Input/output error

nvidia@ubuntu:~$ echo 0 | sudo tee /proc/irq/330/smp_affinity
0
tee: /proc/irq/330/smp_affinity: Input/output error

I understand that, we need to configure the interrupt at GIC level. I am okay with all the interrupts handled by GIC to be routed to core-0 only.

But How do I assign processor affinity using GIC? I am unable to identify GIC and its interrupt number.

I am giving some more information which may help to answer my question.

nvidia@ubuntu:~$ dmesg  | grep "registered GPIOs"
[    0.373824] gpiochip_setup_dev: registered GPIOs 288 to 511 on device: gpiochip0 (tegra-gpio)
[    0.379486] gpiochip_setup_dev: registered GPIOs 248 to 287 on device: gpiochip1 (tegra-gpio-aon)
[    0.500734] gpiochip_setup_dev: registered GPIOs 240 to 247 on device: gpiochip2 (max77620-gpio)

My interrupt is 417 so, it must be from gpiochip0

and my /proc/interrupts dump for reference

nvidia@ubuntu:~$ cat /proc/interrupts 
            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       
   1:          0          0          0          0          0          0          0          0 ff%    GICv2-25 Level     vgic
   2:          0          0          0          0          0          0          0          0 ff%    GICv2-29 Level     trusty
   3:     226038     221027     574594     177397     110381     114915     135411     127406 ff%    GICv2-30 Level     arch_timer
   4:          0          0          0          0          0          0          0          0 ff%    GICv2-27 Level     kvm guest timer
   6:      16518          0          0          0          0          0          0          0 ff    GICv2-208 Level     hsp
   7:          0          0          0          0          0          0          0          0 ff    GICv2-202 Level     arm-smmu global fault
   8:          0          0          0          0          0          0          0          0 ff    GICv2-203 Level     arm-smmu global fault
   9:          0          0          0          0          0          0          0          0 ff    GICv2-264 Level     arm-smmu global fault
  10:          0          0          0          0          0          0          0          0 ff    GICv2-265 Level     arm-smmu global fault
  11:          0          0          0          0          0          0          0          0 ff    GICv2-272 Level     arm-smmu global fault
  12:          0          0          0          0          0          0          0          0 ff    GICv2-273 Level     arm-smmu global fault
  13:          0          0          0          0          0          0          0          0 ff    GICv2-368 Level     tegra-p2u-intr
  14:          0          0          0          0          0          0          0          0 ff    GICv2-369 Level     tegra-p2u-intr
  15:          0          0          0          0          0          0          0          0 ff    GICv2-370 Level     tegra-p2u-intr
  16:          0          0          0          0          0          0          0          0 ff    GICv2-371 Level     tegra-p2u-intr
  17:          0          0          0          0          0          0          0          0 ff    GICv2-372 Level     tegra-p2u-intr
  18:          0          0          0          0          0          0          0          0 ff    GICv2-373 Level     tegra-p2u-intr
  19:          0          0          0          0          0          0          0          0 ff    GICv2-374 Level     tegra-p2u-intr
  20:          0          0          0          0          0          0          0          0 ff    GICv2-375 Level     tegra-p2u-intr
  21:          0          0          0          0          0          0          0          0 ff    GICv2-376 Level     tegra-p2u-intr
  22:          0          0          0          0          0          0          0          0 ff    GICv2-377 Level     tegra-p2u-intr
  23:          0          0          0          0          0          0          0          0 ff    GICv2-253 Level     tegra-p2u-intr
  24:          0          0          0          0          0          0          0          0 ff    GICv2-254 Level     tegra-p2u-intr
  25:          0          0          0          0          0          0          0          0 ff    GICv2-378 Level     tegra-p2u-intr
  26:          0          0          0          0          0          0          0          0 ff    GICv2-379 Level     tegra-p2u-intr
  27:          0          0          0          0          0          0          0          0 ff    GICv2-380 Level     tegra-p2u-intr
  28:          0          0          0          0          0          0          0          0 ff    GICv2-381 Level     tegra-p2u-intr
  29:          0          0          0          0          0          0          0          0 ff    GICv2-382 Level     tegra-p2u-intr
  30:          0          0          0          0          0          0          0          0 ff    GICv2-383 Level     tegra-p2u-intr
  31:          0          0          0          0          0          0          0          0 ff    GICv2-235 Level     tegra-p2u-intr
  32:          0          0          0          0          0          0          0          0 ff    GICv2-252 Level     tegra-p2u-intr
  33:          1          0          0          0          0          0          0          0 ff    GICv2-104 Level     tegra-pcie-intr, PCIe PME, aerdrv
  34:          0          0          0          0          0          0          0          0 ff    GICv2-105 Level     tegra-pcie-msi
  35:          1          0          0          0          0          0          0          0 ff    GICv2-77 Level     tegra-pcie-intr, PCIe PME, aerdrv
  36:          0          0          0          0          0          0          0          0 ff    GICv2-78 Level     tegra-pcie-msi
  37:          1          0          0          0          0          0          0          0 ff    GICv2-81 Level     tegra-pcie-intr, PCIe PME, aerdrv
  38:          0          0          0          0          0          0          0          0 ff    GICv2-82 Level     tegra-pcie-msi
  39:          1          0          0          0          0          0          0          0 ff    GICv2-85 Level     tegra-pcie-intr, PCIe PME, aerdrv
  40:        470          0          0          0          0          0          0          0 ff    GICv2-86 Level     tegra-pcie-msi
  41:          0          0          0          0          0          0          0          0 ff    GICv2-226 Level     ether_qos.common_irq
  43:          0          0          0          0          0          0          0          0 ff    GICv2-222 Level     2490000.ether_qos.rx0
  44:          0          0          0          0          0          0          0          0 ff    GICv2-218 Level     2490000.ether_qos.tx0
  51:      10760          0          0          0          0          0          0          0 ff    GICv2-144 Level   
  54:          0          0          0          0          0          0          0          0 ff    GICv2-152 Level     combined_uart rx
  55:      55736          0          0          0          0          0          0          0 ff    GICv2-97 Level     mmc0
  56:       1083          0          0          0          0          0          0          0 ff    GICv2-94 Level     mmc1
  57:          0          0          0          0          0          0          0          0 ff    GICv2-76 Level     ufshcd
  58:          0          0          0          0          0          0          0          0 ff    GICv2-68 Level     3210000.spi
  59:          0          0          0          0          0          0          0          0 ff    GICv2-69 Level     c260000.spi
  60:         34          0          0          0          0          0          0          0 ff    GICv2-57 Level     3160000.i2c
  61:         76          0          0          0          0          0          0          0 ff    GICv2-58 Level     c240000.i2c
  62:          0          0          0          0          0          0          0          0 ff    GICv2-59 Level     3180000.i2c
  63:          0          0          0          0          0          0          0          0 ff    GICv2-60 Level     3190000.i2c
  64:          0          0          0          0          0          0          0          0 ff    GICv2-62 Level     31b0000.i2c
  65:          3          0          0          0          0          0          0          0 ff    GICv2-63 Level     31c0000.i2c
  66:          0          0          0          0          0          0          0          0 ff    GICv2-64 Level     c250000.i2c
  67:          7          0          0          0          0          0          0          0 ff    GICv2-65 Level     31e0000.i2c
  68:          0          0          0          0          0          0          0          0 ff    GICv2-72 Level     can0
  69:          0          0          0          0          0          0          0          0 ff    GICv2-74 Level     can1
  70:        378          0          0          0          0          0          0          0 ff    GICv2-193 Level     snd_hda_tegra
  71:          0          0          0          0          0          0          0          0 ff    GICv2-51 Level     bc00000.rtcpu
  72:        338          0          0          0          0          0          0          0 ff    GICv2-242 Level     d230000.actmon
  73:      80366          0          0          0          0          0          0          0 ff    GICv2-297 Level     host_syncpt
  74:          2          0          0          0          0          0          0          0 ff    GICv2-295 Level     host_status
  75:          0          0          0          0          0          0          0          0 ff    GICv2-238 Level     vic
  76:          0          0          0          0          0          0          0          0 ff    GICv2-268 Level     nvdla0
  77:          0          0          0          0          0          0          0          0 ff    GICv2-269 Level     nvdla1
  78:      64881          0          0          0          0          0          0          0 ff    GICv2-185 Level     15200000.nvdisplay
  79:          0          0          0          0          0          0          0          0 ff    GICv2-186 Level     15210000.nvdisplay
  80:          0          0          0          0          0          0          0          0 ff    GICv2-187 Level     15220000.nvdisplay
  81:          0          0          0          0          0          0          0          0 ff    GICv2-191 Level     tegra_dp
  82:          0          0          0          0          0          0          0          0 ff    GICv2-192 Level     tegra_dp
  85:          0          0          0          0          0          0          0          0 ff    GICv2-194 Level     cec_irq
  87:          0          0          0          0          0          0          0          0 ff    GICv2-266 Level     pva-isr
  88:          0          0          0          0          0          0          0          0 ff    GICv2-267 Level     pva-isr
  97:          0          0          0          0          0          0          0          0 ff%    GICv2-397 Level     carmel-pmu
  98:          0          0          0          0          0          0          0          0 ff    GICv2-169 Level     tegra-oc-event
  99:          0          0          0          0          0          0          0          0 ff    GICv2-270 Level     noc_nonsecure_irq
 100:          0          0          0          0          0          0          0          0 ff    GICv2-271 Level     noc_secure_irq
 101:          0          0          0          0          0          0          0          0 ff       PM-42 Level     tegra_rtc
 102:          0          0          0          0          0          0          0          0 ff    GICv2-255 Level     mc_status
 104:          0          0          0          0          0          0          0          0 ff    GICv2-165 Level     c150000.tegra-hsp
 115:         29          0          0          0          0          0          0          0 ff    GICv2-214 Level     b950000.tegra-hsp, b950000.tegra-hsp, b950000.tegra-hsp
 119:          0          0          0          0          0          0          0          0 ff    GICv2-315 Level     3ad0000.se_elp
 121:          0          0          0          0          0          0          0          0 ff    GICv2-108 Level     gpcdma.0
 122:          0          0          0          0          0          0          0          0 ff    GICv2-109 Level     gpcdma.1
 123:          0          0          0          0          0          0          0          0 ff    GICv2-110 Level     gpcdma.2
 124:          0          0          0          0          0          0          0          0 ff    GICv2-111 Level     gpcdma.3
 125:          3          0          0          0          0          0          0          0 ff    GICv2-112 Level     gpcdma.4
 126:          3          0          0          0          0          0          0          0 ff    GICv2-113 Level     gpcdma.5
 127:          0          0          0          0          0          0          0          0 ff    GICv2-114 Level     gpcdma.6
 128:          0          0          0          0          0          0          0          0 ff    GICv2-115 Level     gpcdma.7
 129:          0          0          0          0          0          0          0          0 ff    GICv2-116 Level     gpcdma.8
 130:          0          0          0          0          0          0          0          0 ff    GICv2-117 Level     gpcdma.9
 131:          0          0          0          0          0          0          0          0 ff    GICv2-118 Level     gpcdma.10
 132:          0          0          0          0          0          0          0          0 ff    GICv2-119 Level     gpcdma.11
 133:          0          0          0          0          0          0          0          0 ff    GICv2-120 Level     gpcdma.12
 134:          0          0          0          0          0          0          0          0 ff    GICv2-121 Level     gpcdma.13
 135:          0          0          0          0          0          0          0          0 ff    GICv2-122 Level     gpcdma.14
 136:          0          0          0          0          0          0          0          0 ff    GICv2-123 Level     gpcdma.15
 137:          0          0          0          0          0          0          0          0 ff    GICv2-124 Level     gpcdma.16
 138:          0          0          0          0          0          0          0          0 ff    GICv2-125 Level     gpcdma.17
 139:          0          0          0          0          0          0          0          0 ff    GICv2-126 Level     gpcdma.18
 140:          0          0          0          0          0          0          0          0 ff    GICv2-127 Level     gpcdma.19
 141:          0          0          0          0          0          0          0          0 ff    GICv2-128 Level     gpcdma.20
 142:          0          0          0          0          0          0          0          0 ff    GICv2-129 Level     gpcdma.21
 143:          0          0          0          0          0          0          0          0 ff    GICv2-130 Level     gpcdma.22
 144:          0          0          0          0          0          0          0          0 ff    GICv2-131 Level     gpcdma.23
 145:          0          0          0          0          0          0          0          0 ff    GICv2-132 Level     gpcdma.24
 146:          0          0          0          0          0          0          0          0 ff    GICv2-133 Level     gpcdma.25
 147:          0          0          0          0          0          0          0          0 ff    GICv2-134 Level     gpcdma.26
 148:          0          0          0          0          0          0          0          0 ff    GICv2-135 Level     gpcdma.27
 149:          0          0          0          0          0          0          0          0 ff    GICv2-136 Level     gpcdma.28
 150:          0          0          0          0          0          0          0          0 ff    GICv2-137 Level     gpcdma.29
 151:          0          0          0          0          0          0          0          0 ff    GICv2-138 Level     gpcdma.30
 249:          0          0          0          0          0          0          0          0 ff% tegra-gpio-48 Edge      force-recovery
 253:          0          0          0          0          0          0          0          0 ff% tegra-gpio-52 Level     phy_interrupt
 256:          0          0          0          0          0          0          0          0 ff% tegra-gpio-55 Edge      3400000.sdhci cd
 259:          0          0          0          0          0          0          0          0 ff% tegra-gpio-58 Level     tmp451
 299:          0          0          0          0          0          0          0          0 ff% tegra-gpio-98 Edge      15200000.nvdisplay
 330:       6704          0          0          0          0          0          0          0 ff% tegra-gpio-129 Edge      my_gpio_irq_signal
 393:          0          0          0          0          0          0          0          0 ff% tegra-gpio-192 Edge      bluetooth hostwake
 461:          0          0          0          0          0          0          0          0 ff% tegra-gpio-aon-32 Edge      ufs_cd_gpio
 465:          0          0          0          0          0          0          0          0 ff% tegra-gpio-aon-36 Edge      power-key
 474:      90712          0          0          0          0          0          0          0 ff       PM-195 Level     xhci-hcd:usb1
 475:          3          0          0          0          0          0          0          0 ff       PM-196 Level     3610000.xhci
 476:          0          0          0          0          0          0          0          0 ff       PM-199 Level     3610000.xhci
 477:      30290          0          0          0          0          0          0          0 ff    GICv2-102 Level     gk20a_stall
 478:          0          0          0          0          0          0          0          0 ff    GICv2-103 Level     gk20a_nonstall
 479:          0          0          0          0          0          0          0          0 01*    GICv2-424 Level     ras-fhi
 480:          0          0          0          0          0          0          0          0 02*    GICv2-425 Level     ras-fhi
 481:          0          0          0          0          0          0          0          0 04*    GICv2-426 Level     ras-fhi
 482:          0          0          0          0          0          0          0          0 08*    GICv2-427 Level     ras-fhi
 483:          0          0          0          0          0          0          0          0 10*    GICv2-428 Level     ras-fhi
 484:          0          0          0          0          0          0          0          0 20*    GICv2-429 Level     ras-fhi
 485:          0          0          0          0          0          0          0          0 40*    GICv2-430 Level     ras-fhi
 486:          0          0          0          0          0          0          0          0 80*    GICv2-431 Level     ras-fhi
 487:          0          0          0          0          0          0          0          0 ff    GICv2-262 Level     noc_nonsecure_irq
 488:          0          0          0          0          0          0          0          0 ff    GICv2-263 Level     noc_secure_irq
 489:          0          0          0          0          0          0          0          0 ff    GICv2-292 Level     noc_nonsecure_irq
 490:          0          0          0          0          0          0          0          0 ff    GICv2-204 Level     noc_secure_irq
 491:          0          0          0          0          0          0          0          0 ff    GICv2-294 Level     noc_nonsecure_irq
 492:          0          0          0          0          0          0          0          0 ff    GICv2-206 Level     noc_secure_irq
 493:          0          0          0          0          0          0          0          0 ff    GICv2-291 Level     noc_nonsecure_irq
 494:          0          0          0          0          0          0          0          0 ff    GICv2-207 Level     noc_secure_irq
 495:          0          0          0          0          0          0          0          0 ff    GICv2-293 Level     noc_nonsecure_irq
 496:          0          0          0          0          0          0          0          0 ff    GICv2-205 Level     noc_secure_irq
 498:          1          0          0          0          0          0          0          0 ff       PM-241 Edge      max77620-top
 502:          0          0          0          0          0          0          0          0 ff% max77620-top-3 Edge      max77620-gpio
 503:          1          0          0          0          0          0          0          0 ff% max77620-top-4 Edge      max77686-rtc
 507:          0          0          0          0          0          0          0          0 ff% max77620-top-8 Edge      max77620-thermal
 508:          0          0          0          0          0          0          0          0 ff% max77620-top-9 Edge      max77620-thermal
 563:          1          0          0          0          0          0          0          0 ff% max77686-rtc-1 Edge      rtc-alarm1
 565:          0          0          0          0          0          0          0          0 ff%  PCI-MSI-0 Edge      predator_driver
1333:        470          0          0          0          0          0          0          0 ff%  PCI-MSI-0 Edge      eth1-TxRx-0
1334:        470          0          0          0          0          0          0          0 ff%  PCI-MSI-1 Edge      eth1-TxRx-1
1335:        470          0          0          0          0          0          0          0 ff%  PCI-MSI-2 Edge      eth1-TxRx-2
1336:        470          0          0          0          0          0          0          0 ff%  PCI-MSI-3 Edge      eth1-TxRx-3
1337:        470          0          0          0          0          0          0          0 ff%  PCI-MSI-4 Edge      eth1-TxRx-4
1338:        470          0          0          0          0          0          0          0 ff%  PCI-MSI-5 Edge      eth1-TxRx-5
1339:        470          0          0          0          0          0          0          0 ff%  PCI-MSI-6 Edge      eth1-TxRx-6
1340:        470          0          0          0          0          0          0          0 ff%  PCI-MSI-7 Edge      eth1-TxRx-7
1341:          0          0          0          0          0          0          0          0 ff%  PCI-MSI-8 Edge      eth1
1589:          0          0          0          0          0          0          0          0 ff       PM-211 Level     sw-wake
IPI0:     129004     226703     139154     143108      74636      76299     107080     100708       Rescheduling interrupts
IPI1:        719        679      10335       5655       5596       6567       5320       9164       Function call interrupts
IPI2:          0          0          0          0          0          0          0          0       CPU stop interrupts
IPI3:          0          0          0          0          0          0          0          0       Timer broadcast interrupts
IPI4:          1          0          0          0          0          0          0          0       IRQ work interrupts
IPI5:          0          0          0          0          0          0          0          0       CPU wake-up interrupts
 Err:          0
 RTE:          0          0          0          0          0          0          0          0       RT throttling events
3

Hi aravind.chakravarti,

Are you using the devkit or custom board for AGX Xavier?

What’s the exact pin for GPIO417 in your case?

Have you tried enteringsudo first for this operation?

$ sudo su
# echo 0 | tee /proc/irq/330/smp_affinity

Please share the full dmesg for further check.

4

It is a custom hardware. I am not sure about physical GPIO pin used. But, I am sure the pin is correct. Because I am getting interrupt to my Linux Kernel Module and as you can see, /proc/interrupts/ shows 116 interrupts being hit to the GPIO

nvidia@ubuntu:~$ cat /proc/interrupts | grep my_gpio
 330:        116          0          0          0          0          0          0          0 ff% tegra-gpio-129 Edge      my_gpio_irq_signal

Yes. I tried. It doesn’t work either. Here is log + dmesg. kernel didn’t through any error.

root@ubuntu:/home/nvidia# echo 0 | tee /proc/irq/330/smp_affinity
0
tee: /proc/irq/330/smp_affinity: Input/output error
root@ubuntu:/home/nvidia# sudo dmesg | tail
[ 5553.720152] usb 1-3.2: SerialNumber: 84209609DF0E
[ 5553.724343] Chip Version Info: CHIP_8188E_Normal_Chip_TSMC_D_CUT_1T1R_RomVer(0)
[ 5553.748961] tegra-xudc-new 3550000.xudc: PMQOS CPU boost enabled
[ 5553.749065] Could not get extcon-dev /xudc@3550000:vbus(0)
[ 5553.841781] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[ 5554.349671] MAC Address = x̶x̶:̶y̶y̶:̶z̶z̶:̶x̶x̶:̶y̶y̶:̶z̶z̶
[ 5554.353114] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[ 5554.405181] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[ 5557.390866] R8188EU: assoc success
[ 5557.429388] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready

I think it is more to do with GIC (Global Interrupt Controller). Please check earlier forum which I had referenced (attaching again here: Adjusting GPIO IRQ affinity in GIC - Jetson & Embedded Systems / Jetson Orin NX - NVIDIA Developer Forums). Can you please help me configuring interrupt affinity using GIC?

5

@KevinFFF @linuxdev

Can you please help me? I want to know how to configure the processor affinity at GIC level for GPIO.

6

I’m just now looking at this and kind of commenting as I read, so I have not read to the end yet. FYI, this is already at core 0:

The default for most hardware interrupts will be core 0. An entire set of GPIOs are on a single hardware IRQ (a GIC handles “clumps” of GPIO). I don’t know if it is an error to assign an IRQ already on core 0 to core 0, but if it succeeded, then nothing would have changed unless previously something was causing the IRQ to change affinity (in which case the affinity would stop changing; this would imply there was no previous affinity).

I’ve not done this, but it looks like platform_get_irq could be used in your driver to find out what its IRQ is; then a kprint could be used to tell you in a log (just be careful to not spam this). See:
https://docs.kernel.org/driver-api/infrastructure.html

About this:

There might be more than one IRQ involved. How is this custom hardware attached? Is it on PCIe? Is it wired to the bus at a physical address (and thus also a device tree involved)? For what specific purpose do you want to tie this to CPU0 for?

This again says it is using CPU0 for hardware IRQ 330:

If you are at CPU0 already, what would the purpose be of this:

It comes back to the question of what it is you want to accomplish. Does it really matter if changing from CPU0 to CPU0 does not work since it isn’t actually changing anything? Most hardware IRQs on CPU0 don’t migrate anyway.

7

How do you know it is GPIO417? Is it the result from /sys/kernel/debug/gpio?
What the physical connection of your setup? Could you share the block diagram of your connection?

Could you share the source of your custom kernel module to handle this IRQ?

8

Thanks you for your responses @linuxdev

Let me go step by step:

Here is the block-diagram

image
image2124×724 52.4 KB

Our Nvidia Jetson which is a custom hardware is running RedHawk OS. It is a RealTime OS. We receive an interrupt every 10ms once which is extremely precise as it is coming from a external device running plain bare-metal. We need to responds quickly by reading & processing data using UART interface within few microseconds.

RedHawk OS provides processor shield facility, where I can shield processor for interrupts, process and local timers

  • Step 1:
    Shield the processor Core-0 for external interrupts, process, local timers
  • Step 2:
    Now assign GPIO interrupt to core-0. Without this step, GPIO interrupt would move to any other core, as its affinity is 0xFF. Let me demonstrate this also step by step.

Without processor affinity set, GPIO interrupt is assigned to core-0 (as you also explained). Below, we can see that, we received all 859 interrupts to core-0 only.

nvidia@ubuntu:~$ shield

     CPUID      irqs     ltmrs     procs
--------------------------------------------------
         0        no        no        no
         1        no        no        no
         2        no        no        no
         3        no        no        no
         4        no        no        no
         5        no        no        no
         6        no        no        no
         7        no        no        no
nvidia@ubuntu:~$ cat /proc/interrupts | grep 330
 330:        859          0          0          0          0          0          0          0 ff% tegra-gpio-129 Edge      my_gpio_irq_signal
nvidia@ubuntu:~$

Now, let me mask core-0 to irq, proceses and local timers. Now as you can see, Core-0 handled the first 859 interrupts, but after shielding, the next 12 interrupts are now handled by Core-1. This happens because the interrupt affinity is set to 0xFF, and Linux chooses the next available core (Core-1) to handle it.

nvidia@ubuntu:~$ sudo shield --irq=0 --proc=0 --loc=0
nvidia@ubuntu:~$ shield

     CPUID      irqs     ltmrs     procs
--------------------------------------------------
         0       yes       yes       yes
         1        no        no        no
         2        no        no        no
         3        no        no        no
         4        no        no        no
         5        no        no        no
         6        no        no        no
         7        no        no        no
nvidia@ubuntu:~$ cat /proc/interrupts | grep 330
 330:      859        12          0          0          0          0          0          0 ff% tegra-gpio-129 Edge      my_gpio_irq_signal
nvidia@ubuntu:~$

The only way, I can retain IRQ to core-0 is by setting its affinity.

We had discussed this with RedHawk team, they confirmed that, this is something to do with Linux and RedHawk OS has not altered processor affinity part.

9

Thank you for responding @KevinFFF

I am sure it is GPIO417. The reason is, I can see interrupt logs in

nvidia@ubuntu:~$ cat /proc/interrupts | grep my_gpio
 330:        116          0          0          0          0          0          0          0 ff% tegra-gpio-129 Edge      my_gpio_irq_signal

If I disconnect my interrupt physically, then I do not see number of interrupts (116 in this case) being incremented in my my_gpio_irq_signal interrupt line.

Also, from manufacturer I got information that GPIO 417 is physically connected to PIN T7

Sure. I just responded my block diagram to @linuxdev’s comment above.

Sure. Actually, I had attached this in my original question (as a text file as forum is not allowing me to attach .c files). I am attaching again for your reference. Let me know if there is any difficulty in viewing the file. I can upload to google drive and share the link.
gpio_irq.txt (3.4 KB)

Let me know if you need any more info.

10

I do not have experience with Redhawk, which probably changes many things. If this were just the default L4T, then it would be easier to answer, but if we talk about just the Linux kernel there are some takeaways which probably still matter. Take this with a proverbial “grain of salt” though since Redhawk might have some scheme to change things. Do verify though if the installed software on the Jetson uses a modified kernel.

First, if we are just talking about software interrupts, then those would indeed be able to migrate anywhere. The stock Linux though will tend to stick to one core because of the scheduler understanding cache. Cache is probably one of the biggest differences to consider when working with a hard RT system instead of soft RT since cache changes memory operations to no longer be deterministic. Because of the stock Linux kernel understanding how to get better performance through cache it is often better to keep a number of related processes/threads on a single core. If something originates on CPU0 (core 0), then it is unusual for it to migrate to a different core without something nudging the scheduler (such as pressure to run when something else is higher priority).

Hardware IRQs have the additional requirement that there must be actual wiring between a core and the IRQ. On Intel desktop CPU architecture you will find a programmable IO-APIC (I/O Asynchronous Programmable Interrupt Controller; AMD has a different way of working with this, but has more or less the same idea). This APIC allows reprogramming of an IRQ to wire it to any core. Without this many hardware IRQs could be serviced only on one core.

So far as I know Jetsons have not evolved with any equivalent of that IO-APIC. Certainly all of the original and derivative Jetson designs for years have many many parts of the hardware only able to reach CPU0. Remember that this is in hardware. One could use affinity to try to force any hardware IRQ to any core, but when the scheduler goes to run that driver the interrupt is rescheduled on the core which has access to the hardware. You would see a hardware IRQ to a core unable to process this, and then you would see a reschedule which results in CPU0 being the core which finally runs the driver upon IRQ. It is more or less a waste of time, but the kernel (I don’t know if Redhawk uses its own entirely custom kernel or if it starts with Linux, though I think it probably does start with that) won’t actually run the driver on another core even if affinity tells it to. The wiring forces this.

Most good quality hardware drivers will first do the very minimal work which talks directly to the hardware, and then take any further processing (e.g., a checksum would be an example) and issue a software interrupt which releases CPU0 (or whatever CPU used if not a Jetson) and then the software interrupt could choose to migrate or yield to another process. Most of the time the soft IRQ would not migrate simply because the CPU’s cache would become a cache miss if migrating, whereas remaining on one core has a good chance of a cache hit. Hard RT tends to not use cache even if it is available since it means non-deterministic memory timing.

When you look at “/proc/interrupts” you see only the hardware IRQs. Timers are an example of something which each core has access to, but you will notice on a Jetson that there is an overwhelming number of hardware IRQs which run only on CPU0; when another core gets a hardware IRQ it might be an illusion that the core is actually running anything since you will also tend to see a reschedule (which puts the processing back onto CPU0).

Btw, I’m not sure what you mean by “shielding”. I suspect you mean masking (it is a bitwise mask when setting up affinity).

I have no idea if the Redhawk version of kernel has knowledge of which hardware IRQ can or cannot be scheduled on a non-CPU0 core. However, being an RT kernel, the goal would be to take the highest priority content and preempt anything interfering with that thread/process/driver and put it on hold. Should the process on hold also have a high priority, then it might try to migrate to a new core. There is where the problem occurs should the Redhat kernel not know that the migration won’t work due to having only a CPU0 wiring to the hardware. I just don’t know.

When you see something which is a hardware IRQ try to run on a core other than CPU0, are you certain that the non-CPU0 core is actually what runs this without rescheduling back to CPU0?

If there is an attempt to reschedule to a non-CPU0 core due to some sort of scheduling pressure, then it would be reasonable for a non-aware kernel scheduler to try to shift to a new core. It does happen that entire groups of GPIO can migrate to a new core on a Jetson without needing to migrate back to CPU0 (I think the wiring exists for this as a group; an IO-APIC would more or less make it possible to migrate individual GPIO, but no such IO-APIC exists in this case). The default on an L4T or mainline kernel would be that GPIO would not migrate away from CPU0. I do not know how the Redhawk kernel works with affinity, but I’m guessing it isn’t any different than a regular Linux kernel (the difference would be in scheduling).

Do you have any way to verify if your process, when migrating to non-CPU0, is actually running on a different core and not rescheduling and migrating back to CPU0 for the actual operation? I would think this needs to be answered first or you might be asking the wrong question.

11

Thank you very much for the detailed answer @linuxdev. — I really appreciate your time!

I completely agree with your explanation here

I completely agree with your assessment. My original question was not about relocating the interrupt to a different core, but rather about pinning it specifically to core-0.

To summarize, my objective is to change the GPIO interrupt affinity from 0xff (all cores) to 0x01 (core-0). However, this change does not take effect because the affinity needs to be set at the GIC (Generic Interrupt Controller) level, rather than at the individual GPIO level.

I hope this explanation is clear and not causing any confusion.

To illustrate my point, let me demonstrate this behavior using a UART interrupt:

With the default processor affinity (0xff), the UART interrupt is directed to core-0 by default, as shown below:

nvidia@ubuntu:~$ cat /proc/interrupts | grep serial
  51:        131          0          0          0          0          0          0          0 ff    GICv2-144 Level     3100000.serial

Now, I update the affinity to 0x08 to target core-3:

nvidia@ubuntu:~$ echo 8 | sudo tee /proc/irq/51/smp_affinity
[sudo] password for nvidia: 
8
nvidia@ubuntu:~$ cat /proc/interrupts | grep serial
  51:        131          0          0         16          0          0          0          0 08    GICv2-144 Level     3100000.serial

As shown, the new interrupts are now being handled by core-3.

Next, I switch the affinity back to core-0 using 0x01:

echo 1 | sudo tee /proc/irq/51/smp_affinity
1
nvidia@ubuntu:~$ cat /proc/interrupts | grep serial
  51:        143          0          0       16          0          0          0          0 01    GICv2-144 Level     3100000.serial

To reiterate:
All I’m trying to achieve is the same behavior for the GPIO interrupt — that is, to route it specifically to core-0. Since setting affinity at the individual GPIO level doesn’t seem to work, we need to configure it at the GIC level (which is what previous many forum posts are suggesting).

Even if the GPIO interrupt is hardwired to core-0, it should not be an issue in my use case. My only goal is to set the affinity to 0x01 (core-0) instead of the default 0xff I do not want to move GPIO interrupt to any other core.

12

Any RT patch, which probably includes both FreeRTOS and Redhawk (but I don’t really know details because I have not looked at either), would likely drastically patch or change the default Linux scheduler. Scheduling and preempting is by far the modification needed for real-time. You are probably right that you could set affinity to CPU0 and avoid an attempt to migrate to other cores (which would eventually have to migrate back to CPU0), but someone knowing Redhawk would have to answer that. The standard Linux method on a Jetson is just setting to CPU0 for that hardware IRQ on the GIC as a whole. However, this wouldn’t be needed on a default Linux kernel unless something really unusual occurred.

I do not know what specific IRQ is for the GIC, but yes, you could attempt to bind that to CPU0. What a Redhawk modified kernel will do with that I don’t know; hard real-time would attempt to make migration possible if it is required to meet scheduling under the current load. Does the scheduler in Redhawk know that individual GPIOs cannot migrate, and that the GIC will migrate more than one GPIO? I don’t know.

I will ask this: What is the issue you are running into? Is the GPIO triggering as an IRQ too late or with bad timing?

Incidentally, you could perform a test where your kernel module doesn’t do anything other than return after doing something (anything, e.g., compare 0 to 0) and then return. Then spam the GPIO for that particular IRQ at a high enough rate to saturate it, and simultaneously watch “/proc/interrupts” for that IRQ and the GIC and see how its IRQ and reschedules go. Do this once with defaults, and then use ordinary Linux affinity and perform the experiment again. See if migration and reschedule goes away or not. You might also have to experiment with higher priority to the GIC, but I have no idea how to do that in Redhawk.

13

Yes @linuxdev. I am expecting a “interrupt dispatch latency” of few microseconds (less than 100us) but I am getting latency of +/-1ms. Which will not serve purpose for my application.

Sorry to ask again, but how do I know the GIC for my GPIO interrupt?

Thank you for replying my questions with lots of patience!

14

I do not know the specific information for the GIC interrupt as a whole which works with that specific GPIO. Maybe try flooding the pin with IRQs and watch which one goes up in “/proc/interrupts” (it is not an actual file and it will update as you watch if you refresh your file viewer). That is something @KevinFFF could probably answer.

15

It seems you can assign the interrupt handled by core-0 here.

Could you try running the following command instead?

#echo 1 > /proc/irq/330/smp_affinity

It seems your interrupt has been default handled by core-0.

Please also check if the information from GPIO Interrupt Mapping Configuration can help for your case.

16

Hi, i am working with Aravind on this issue.
while going through the GPIO interrupt mapping configuration file , we found that all GPIO pins are assigned to INT0.
Does it mean that GPIO interrupts are wired to CPU0?
what will happen if we change it to INT1 for that particular pin( GPIO 417 is Pin T7 in our case)?

17

I think a group of pins are handled by the same wire, and thus you can move only entire groups. That’s the GIC. You have to move all or none in that group so far as I know, but @KevinFFF may have more information on that.

Generally speaking, there is a lot of hardware which has wiring only for CPU0 IRQ handling. If you tell the scheduler to use another CPU core via affinity for one of those which cannot reach a different core, then the IRQ will try to run on the alternate core, and then reschedule for CPU0. So far as I know GIC can actually migrate as a whole (not individual GPIO pins) to a new core. In the case that you want the IRQ to stick to CPU0, then affinity would probably work to prevent an attempt at migrating to a new core (a waste of time) and then migrating back to CPU0.

There is a huge caveat here though: Are you using the stock kernel, or is it the Redhawk RT kernel? If it is Redhawk, then it is possible the Redhawk kernel is not aware of which hardware is only capable of going to CPU0 and might have different rules for scheduling.

18

Can you please elaborate how to change irq for group of GPIO pins at GIC in linux for tegra?
We are contacting concurrent about redhawk?

19

MB1 configures the interrupt mapping for all GPIO pins. Each GPIO controller has 8 interrupt outputs to send the interrupts to different CPUs/Co-processors. The interrupt routing register configuration determines which pin interrupt is sent to which CPU/Processor. Interrupt 0 is routed to CCPLEX/BPMP and the other interrupts are routed to APE, SPE, SCE and RCE. If the interrupts are not bound to CCPLEX, then the /proc/interrupts will not be detecting the interrupts.

I think you can just keep default settings and your interrupt would be handled by CPU0.

20

CCPLEX here means all 7 cores of CPU?
Does it mean INT0 is wired to CPU block and INT1 …to INT7 are wired other specific processing blocks?

21

Correct!

Is there any issue if you just keep default settting?