sevm89
March 6, 2020, 7:26am
1
Hi NVidia Team
For testings we run Jetson AGX Xavier Modules with our own carrier board on a minimal rootfs with busybox. We build it with the information of the following page:https://elinux.org/Jetson/Busybox_RootFS
We now have some systems that sometimes show the following error message in dmesg:
[ 39.080659] INFO: rcu_preempt self-detected stall on CPU
[ 39.080835] 0-...: (1 GPs behind) idle=4df/140000000000002/0 softirq=358/358 fqs=2001
[ 39.080993] (t=5250 jiffies g=-106 c=-107 q=242)
[ 39.081088] Task dump for CPU 0:
[ 39.081164] swapper/0 R running task 0 0 0 0x00000002
[ 39.081300] Call trace:
[ 39.081363] [<ffffff800808bdb8>] dump_backtrace+0x0/0x198
[ 39.081458] [<ffffff800808c37c>] show_stack+0x24/0x30
[ 39.081557] [<ffffff80080ed4d0>] sched_show_task+0xf8/0x148
[ 39.081657] [<ffffff80080f01b0>] dump_cpu_task+0x48/0x58
[ 39.081758] [<ffffff80081c282c>] rcu_dump_cpu_stacks+0xb8/0xec
[ 39.081860] [<ffffff80081331a8>] rcu_check_callbacks+0x728/0xa48
[ 39.081962] [<ffffff8008139a04>] update_process_times+0x34/0x60
[ 39.082121] [<ffffff800814af68>] tick_sched_handle.isra.5+0x38/0x70
[ 39.082587] [<ffffff800814afec>] tick_sched_timer+0x4c/0x90
[ 39.083038] [<ffffff800813a738>] __hrtimer_run_queues+0xd8/0x360
[ 39.083457] [<ffffff800813b088>] hrtimer_interrupt+0xa8/0x1e0
[ 39.083903] [<ffffff8008be9e00>] arch_timer_handler_phys+0x38/0x58
[ 39.088652] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 39.088661] 0-...: (2 GPs behind) idle=4df/140000000000002/0 softirq=356/358 fqs=2260
[ 39.088669] (detected by 1, t=5252 jiffies, g=-291, c=-292, q=18)
[ 39.088670] Task dump for CPU 0:
[ 39.088677] swapper/0 R running task 0 0 0 0x00000002
[ 39.088678] Call trace:
[ 39.088699] [<ffffff80080863bc>] __switch_to+0x9c/0xc0
[ 39.088702] [<0000000000000002>] 0x2
[ 39.130985] [<ffffff8008127c68>] handle_percpu_devid_irq+0x90/0x2b0
[ 39.137266] [<ffffff800812224c>] generic_handle_irq+0x34/0x50
[ 39.142866] [<ffffff8008122930>] __handle_domain_irq+0x68/0xc0
[ 39.148904] [<ffffff8008080d44>] gic_handle_irq+0x5c/0xb0
[ 39.154416] [<ffffff8008082be8>] el1_irq+0xe8/0x18c
[ 39.159317] [<ffffff80081383f8>] call_timer_fn+0x38/0x1e0
[ 39.164829] [<ffffff8008138714>] expire_timers+0x144/0x188
[ 39.170251] [<ffffff800813889c>] run_timer_softirq+0x144/0x178
[ 39.176215] [<ffffff8008081054>] __do_softirq+0x13c/0x3b0
[ 39.181545] [<ffffff80080bb298>] irq_exit+0xd0/0x118
[ 39.186526] [<ffffff8008122934>] __handle_domain_irq+0x6c/0xc0
[ 39.192391] [<ffffff8008080d44>] gic_handle_irq+0x5c/0xb0
[ 39.197725] [<ffffff8008082be8>] el1_irq+0xe8/0x18c
[ 39.202725] [<ffffff800814b320>] tick_nohz_idle_exit+0xe8/0x118
[ 39.208754] [<ffffff8008112cd0>] cpu_startup_entry+0xe8/0x200
[ 39.214533] [<ffffff8008f45d8c>] rest_init+0x84/0x90
[ 39.219346] [<ffffff80095e0b64>] start_kernel+0x370/0x384
[ 39.224776] [<ffffff80095e0204>] __primary_switched+0x80/0x94
[ 39.230454] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 39.235916] 0-...: (1 GPs behind) idle=4df/140000000000002/0 softirq=358/358 fqs=2002
[ 39.244102] (detected by 7, t=5252 jiffies, g=-106, c=-107, q=242)
[ 39.250576] Task dump for CPU 0:
[ 39.253990] swapper/0 R running task 0 0 0 0x00000002
[ 39.260903] Call trace:
[ 39.263534] [<ffffff80080863bc>] __switch_to+0x9c/0xc0
[ 39.268865] [<0000000000000002>] 0x2
[ 42.145760] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[ 42.166095] usb 1-4.1: new high-speed USB device number 5 using tegra-xusb
[ 42.187059] usb 1-4.1: config 1 has an invalid interface number: 8 but max is 4
[ 42.187273] usb 1-4.1: config 1 has an invalid interface number: 10 but max is 4
[ 42.187430] usb 1-4.1: config 1 has no interface number 1
[ 42.187534] usb 1-4.1: config 1 has no interface number 4
[ 42.188597] usb 1-4.1: New USB device found, idVendor=1199, idProduct=9071
[ 42.188863] usb 1-4.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 42.189016] usb 1-4.1: Product: Sierra Wireless MC7455 Qualcomm® Snapdragonâ
¢ X7 LTE-A
[ 42.189180] usb 1-4.1: Manufacturer: Sierra Wireless, Incorporated
[ 42.189345] usb 1-4.1: SerialNumber: LQ94110315031028
[ 42.439491] mmc1: Enabling vmmc regulator
[ 42.642570] mmc1: Disabling vmmc regulator
[ 43.058357] mmc1: Enabling vmmc regulator
[ 43.266395] mmc1: Disabling vmmc regulator
[ 45.315458] igb 0004:01:00.0 eth1: igb: eth1 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX
[ 45.316110] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[ 45.473147] mmc1: Enabling vmmc regulator
[ 51.696562] mmc1: Enabling vmmc regulator
[ 51.898650] mmc1: Disabling vmmc regulator
[ 55.446304] mmc1: Enabling vmmc regulator
[ 55.654330] mmc1: Disabling vmmc regulator
[ 57.292723] mmc1: Enabling vmmc regulator
[ 57.498670] mmc1: Disabling vmmc regulator
[ 57.906422] mmc1: Enabling vmmc regulator
[ 58.110297] mmc1: Disabling vmmc regulator
[ 60.154323] mmc1: Enabling vmmc regulator
[ 60.358306] mmc1: Disabling vmmc regulator
[ 64.734335] mmc1: Enabling vmmc regulator
[ 65.358345] mmc1: Disabling vmmc regulator
[ 66.190512] mmc1: Enabling vmmc regulator
[ 66.394690] mmc1: Disabling vmmc regulator
[ 69.322394] mmc1: Enabling vmmc regulator
[ 69.526405] mmc1: Disabling vmmc regulator
[ 78.102333] mmc1: Enabling vmmc regulator
[ 78.306389] mmc1: Disabling vmmc regulator
[ 80.754440] mmc1: Enabling vmmc regulator
[ 80.958389] mmc1: Disabling vmmc regulator
[ 85.242473] mmc1: Enabling vmmc regulator
[ 85.446336] mmc1: Disabling vmmc regulator
[ 87.282478] mmc1: Enabling vmmc regulator
[ 87.486520] mmc1: Disabling vmmc regulator
[ 98.510555] mmc1: Enabling vmmc regulator
[ 98.714462] mmc1: Disabling vmmc regulator
[ 104.426351] mmc1: Enabling vmmc regulator
[ 104.630411] mmc1: Disabling vmmc regulator
[ 104.834572] mmc1: Enabling vmmc regulator
[ 105.038331] mmc1: Disabling vmmc regulator
[ 108.726747] mmc1: Enabling vmmc regulator
[ 108.930351] mmc1: Disabling vmmc regulator
[ 110.568492] mmc1: Enabling vmmc regulator
[ 110.770725] mmc1: Disabling vmmc regulator
[ 113.020346] mmc1: Enabling vmmc regulator
[ 113.222566] mmc1: Disabling vmmc regulator
[ 150.367067] mmc1: Enabling vmmc regulator
[ 150.570801] mmc1: Disabling vmmc regulator
[ 165.258476] mmc1: Enabling vmmc regulator
[ 165.462926] mmc1: Disabling vmmc regulator
Is there a problem with our rootfs or what could lead to these messages?
You will probably need to provide a serial console log. Even if the Linux stage does not have serial console enabled (but hopefully it does) a log can show setup until the switch to the Linux kernel.
sevm89
July 28, 2020, 2:03pm
3
We have a new production and we do temperature cycling tests with our custom carrier board and the jetson agx xavier module. Now we see the strange behavior, that we get cpu stalls at negative temperatures (around -25°C). The dmesg of such errors are below:
[ 84.552730] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0]#1
[ 178.234739] random: crng init done#1
We have systems with the exact same configuration and minimal filesystem, that go through these temperature cycling without any errors, but like 30-40% of all devices show cpu stalls, only at negative temperatures. Do you have any idea what could stall the cpu? Any help for debugging this topic is appreciated.
I doubt I can help, but wanted to point out this:[ 624.583025] Workqueue: events igb_watchdog_task
“IGB” should be the Intel network driver. Whatever you have attached to the board at the time of failure would be important, and if the failure is always from the IGB driver, then it implies the Intel gigabit device probably does not work at this temperature. Is it always IGB failing, or does the failure change? I assume IGB is something you’ve added since the default module does not use an Intel gigabit chip (correct me if I am wrong).
sevm89
July 29, 2020, 7:12am
5
Hi linuxdev
Thank you for your answer.
[ 56.557212] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0]#1
[ 288.576221] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0]#1 #1 #1 #1 #1 #1
Can it be that the CPU stall results in this igb errors as the workqueue of the device can not be handled? We have many other systems (also x86) with working IGB I210 devices, where we never saw a problem at low temperatures.
It is possible that other errors cause something else to fail to respond, but from the error messages it is hard to say what the original cause is.
This one does look interesting:
Do you have the Jetson working for a bit of time prior to failures? If you have some time, then perhaps on serial console, or through some other method which leaves the last response, run “watch -n 1 ifconfig ...that ethernet device name...”
For example, if the device name of your ethernet is “eth1”, then:watch -n 1 ifconfig eth1
There are certain network misconfiguration issues which could cause an excess of work, and that might find a weakness resulting in something similar to what you are seeing even if it isn’t an outright bug or hardware failure. The mention of SYN flooding implies this is possibly an ethernet data driven issue. It would be nice to see what kind of network statistics show up prior to failure if normal operation works for a short time prior to failure.
Hi,
I don’t think you need to follow the method from that page since it is for tk1. So that page is not being verified for maybe 4 years.
What is your requirement for “minimal rootfs”? I would still suggest you could use original rootfs.
sevm89
August 6, 2020, 8:43am
8
Hi WayneWWW
We did now the same test with the JetPack 4.2.2 and also here we see errors at -25°C. In the dmesg the following appears:
[ 34.805127] INFO: rcu_preempt self-detected stall on CPU
Other error messages:
[ 5.764532] mmc1: Enabling vmmc regulator
[ 56.571065] CPU: 0 PID: 12 Comm: watchdog/0 Not tainted 4.9.140-tegra #1
[ 77.990033] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 71s!
So it seems a general problem and not a problem of the minimal rootfs. When we changed the Jetson AGX Xavier Module of systems that showed this behavior with one that did not, the error went with the module. This makes us believe that it is not a problem of our carrier board but of the Jetson AGX Xavier Module itself. Any suggestions what we could try? What could lead to this stalls at negative temperature?
sevm89
August 10, 2020, 1:37pm
9
Any news on this?
Hi,
Could you check it by directly putting pure jetpack release + nv devkit to -25 degree environment?
FYI, ifconfig information would still be useful since this shows various classifications of network errors. If the network itself has issues, then it would make sense to see the error regardless of most changes.
sevm89
August 14, 2020, 8:37am
12
We have tested the behavior with the Jetson AGX Xavier DevKit and a module that showed the errors. We did not see any CPU stalls.
Hi,
Could you disable 3400000.sdhci in your device tree?
That really seems to improve the behavior at negative temperatures. We need more testing to see if it totally solves it. Could you explain why enabling the SD card leads to this issue? As we have a SD card holder on our design, it is not really an option for us to permanently disable the interface. Could another specific configuration of the sdhci lead to this?
Hi sevm89,
That was not a real debug or solution. I just saw his sdmmc spew lots of errors so asking him to disable it first.
Could you explain why enabling the SD card leads to this issue?
And I cannot explain it to you either. If Devkit + module are working fine under -25 degree environment, it means this issue only happens to your board.
In that case, please paste your carrier board schematic here for us to review. Also, if you confirm this issue is 100% resolved after disabling sdmmc1, you could only share the schematic of sdmmc design part.