Running a TX2i on devkit carrier. I captured some log output over the debug uart (pin 8,10 on 40 pin header of the devkit carrier) that shows a watchdog and restart. I’m trying to diagnose what caused this.
52.181513] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [reaper-1:5746]
[ 52.189125] Kernel panic - not syncing: softlockup: hung tasks
[ 52.194947] CPU: 0 PID: 5746 Comm: reaper-1 Tainted: G L 4.9.337-tegra #1
[ 52.202844] Hardware name: storm (DT)
[ 52.206494] Call trace:
[ 52.208936] [<0000000098320390>] dump_backtrace+0x0/0x198
[ 52.214323] [<00000000343b18a2>] show_stack+0x24/0x30
[ 52.219364] [<00000000374ab194>] dump_stack+0xa0/0xc4
[ 52.224403] [<00000000bb3845d3>] panic+0x128/0x2a4
[ 52.229185] [<0000000080aa35c8>] watchdog_unpark_threads+0x0/0x98
[ 52.235266] [<00000000d9f3a79e>] __hrtimer_run_queues+0xd8/0x360
[ 52.241259] [<00000000f434bb69>] hrtimer_interrupt+0xa8/0x1e0
[ 52.246993] [<000000008a6f970c>] tegra186_timer_isr+0x34/0x48
[ 52.252725] [<00000000858aec17>] __handle_irq_event_percpu+0x68/0x288
[ 52.259150] [<00000000230e0b05>] handle_irq_event_percpu+0x28/0x60
[ 52.265314] [<0000000064eedebc>] handle_irq_event+0x50/0x80
[ 52.270872] [<0000000066336f9e>] handle_fasteoi_irq+0xd4/0x1c0
[ 52.276692] [<00000000178b1c25>] generic_handle_irq+0x34/0x50
[ 52.282424] [<0000000002c03407>] __handle_domain_irq+0x68/0xc0
[ 52.288241] [<00000000293ad936>] gic_handle_irq+0x5c/0xb0
[ 52.293625] [<00000000eda57863>] el1_irq+0xe8/0x194
[ 52.298492] [<000000000d525847>] irq_exit+0xd0/0x118
[ 52.303443] [<00000000d33caf34>] __handle_domain_irq+0x6c/0xc0
[ 52.309260] [<00000000293ad936>] gic_handle_irq+0x5c/0xb0
[ 52.314646] [<00000000f38c72c2>] el0_irq_naked+0x54/0x60
[ 52.319947] SMP: stopping secondary CPUs
[ 52.323862] Kernel Offset: disabled
[ 52.327340] Memory Limit: none
[ 52.330385] trusty-log panic notifier - trusty version Built: 23:39:30 Nov 4 2024 [ 52.343508] Rebooting in 5 seconds..
The stack trace seems to show that this is a problem in system code during the handling of an interrupt. After reset the I looked at the processes (sudo ps aux) and found that the closest PID to the watchdog triggering PID is indeed an interrupt, so it seems likely that the watchdog occured with one of the interrupts I registered.
root 5747 0.0 0.0 0 0 ? S 09:10 0:00 [irq/243-SampleE]
root 5748 0.0 0.0 0 0 ? S 09:10 0:00 [irq/240-SampleE]
root 5749 0.0 0.0 0 0 ? S 09:10 0:00 [irq/241-SampleE]
I have a systemd service running that opens three gpio pins as interrupts using libgpiod v1. The “SampleE” is an abbreviated version of my process name. I service the interrupt events from libgpiod using gpiod_line_event_wait() and gpiod_line_event_read().
Why would the interrupt handling process stall and watchdog? If it occurred in my code I’d expect to see it in the stack trace. The interrupt gpio pin was floating when this happened, so it seems that it may have picked up noise and toggles rapidly. That is not good, but I would not expect a watchdog panic to occur. In fact if there is a hardware fault and the line starts to misbehave I absolutely do not want my whole system to start resetting randomly!