Jetson Xavier AGX nvgpu_timeout_expired

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

Hi,

Recently I have noticed our Jetson Xavier AGX for some unknown reason reboots many times, we checked all the system logs, especially the temperature, input voltage, CPU usage, available memory and we don’t even see any Kernel panic message in system logs.
We notices also two red lines in dmesg after boot that we don’t know if that is causing the issue or not.
I’m attaching the dmesg for your review.
I also upgraded the kernel using apt-get dist-upgrade, but still the same issue.
# R32 (release), REVISION: 4.4, GCID: 23942405, BOARD: t186ref, EABI: aarch64, DATE: Fri Oct 16 19:37:08 UTC 2020

Does anyone any clue about what are those failures, and what they are happening? and potential reasons for reboots.

And more specifically red lines that I see in dmesg:

[ 5.938705] cgroup: cgroup2: unknown option "nsdelegate"

and

`

[ 63.972816] nvgpu: 17000000.gv11b __nvgpu_timeout_expired_msg_cpu:94 [ERR] Timeout detected @ nvgpu_vm_unmap+0x114/0x198 [nvgpu] sync-unmap failed on 0x1efcb00000

`

Thanks.

` 5.868475] usb 1-2: Manufacturer: Quectel
[    5.868478] usb 1-2: SerialNumber: df25af41
[    5.935261] ip_tables: (C) 2000-2006 Netfilter Core Team
[    5.938705] cgroup: cgroup2: unknown option "nsdelegate"
[    5.955394] systemd[1]: systemd 237 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
[    5.955932] systemd[1]: Detected architecture arm64.
[    5.963788] systemd[1]: Set hostname to <ubuntu-desktop>.
[    6.057138] systemd[1]: File /lib/systemd/system/systemd-journald.service:36 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
[    6.057149] systemd[1]: Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
[    6.193911] random: systemd: uninitialized urandom read (16 bytes read)
[    6.196425] systemd[1]: Created slice User and Session Slice.
[    6.196605] random: systemd: uninitialized urandom read (16 bytes read)
[    6.196626] systemd[1]: Reached target User and Group Name Lookups.
[    6.196686] random: systemd: uninitialized urandom read (16 bytes read)
[    6.196705] systemd[1]: Reached target Swap.
[    6.197615] systemd[1]: Created slice System Slice.
[    6.197975] systemd[1]: Listening on RPCbind Server Activation Socket.
[    6.227149] gpio tegra-gpio wake44 for gpio=192(Y:0)
[    6.238665] EXT4-fs (mmcblk0p1): re-mounted. Opts: (null)
[    6.305327] random: crng init done
[    6.305335] random: 7 urandom warning(s) missed due to ratelimiting
[    6.403949] nvgpu: 17000000.gv11b          nvgpu_nvhost_syncpt_init:291  [INFO]  syncpt_unit_base 60000000 syncpt_unit_size 400000 size 1000

[    6.872686] systemd-journald[2464]: Received request to flush runtime journal from PID 1
[    7.041252] Intel(R) Wireless WiFi driver for Linux
[    7.041259] Copyright(c) 2003- 2015 Intel Corporation
[    7.041519] iwlwifi 0003:01:00.0: enabling device (0000 -> 0002)
[    7.044079] iwlwifi 0003:01:00.0: Direct firmware load for iwlwifi-8265-26.ucode failed with error -2
[    7.044279] iwlwifi 0003:01:00.0: Falling back to user helper
[    7.376114] using random self ethernet address
[    7.376231] using random host ethernet address
[    7.575988] FAT-fs (loop0): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[    8.049719] Mass Storage Function, version: 2009/09/11
[    8.049728] LUN: removable file: (no medium)
[    8.061761] using random self ethernet address
[    8.061878] using random host ethernet address
[    8.112704] rndis0: HOST MAC ee:41:43:4b:97:70
[    8.112907] rndis0: MAC ee:41:43:4b:97:71
[    8.114146] usb0: HOST MAC ee:41:43:4b:97:72
[    8.114177] usb0: MAC ee:41:43:4b:97:73
[    8.114195] tegra-xudc-new 3550000.xudc: exiting ELPG
[    8.115001] tegra-xudc-new 3550000.xudc: exiting ELPG done
[    8.115016] tegra-xudc-new 3550000.xudc: ep 0 (type: 0, dir: out) enabled
[    8.115835] tegra-xudc-new 3550000.xudc: entering ELPG
[    8.116412] tegra-xudc-new 3550000.xudc: entering ELPG done
[    8.116455] tegra-xudc-new 3550000.xudc: exiting ELPG
[    8.117937] tegra-xudc-new 3550000.xudc: exiting ELPG done
[    8.117960] tegra-xudc-new 3550000.xudc: entering ELPG
[    8.118161] tegra-xudc-new 3550000.xudc: entering ELPG done
[    8.123648] l4tbr0: port 1(rndis0) entered blocking state
[    8.123655] l4tbr0: port 1(rndis0) entered disabled state
[    8.125062] device rndis0 entered promiscuous mode
[    8.131831] IPv6: ADDRCONF(NETDEV_UP): rndis0: link is not ready
[    8.142550] l4tbr0: port 2(usb0) entered blocking state
[    8.142560] l4tbr0: port 2(usb0) entered disabled state
[    8.157085] device usb0 entered promiscuous mode
[    8.164714] IPv6: ADDRCONF(NETDEV_UP): usb0: link is not ready
[    8.369637] iwlwifi 0003:01:00.0: Direct firmware load for iwlwifi-8265-25.ucode failed with error -2
[    8.369830] iwlwifi 0003:01:00.0: Falling back to user helper
[    8.694325] ras_fhi_disable: FHI 482 disabled
[    8.695372] CPU4: shutdown
[    8.695468] psci: CPU4 killed.
[    8.729351] iwlwifi 0003:01:00.0: Direct firmware load for iwlwifi-8265-24.ucode failed with error -2
[    8.729546] iwlwifi 0003:01:00.0: Falling back to user helper
[    8.738786] ras_fhi_disable: FHI 483 disabled
[    8.747032] CPU5: shutdown
[    8.764868] psci: Retrying again to check for CPU kill
[    8.765350] psci: CPU5 killed.
[    8.790093] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    8.819280] ras_fhi_disable: FHI 484 disabled
[    8.825280] CPU6: shutdown
[    8.825379] psci: CPU6 killed.
[    8.869555] gpio tegra-gpio wake20 for gpio=52(G:4)
[    8.877356] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    8.882309] ras_fhi_disable: FHI 485 disabled
[    8.890891] CPU7: shutdown
[    8.891036] psci: CPU7 killed.
[    8.933199] nvgpu: 17000000.gv11b                 tpc_pg_mask_store:843  [INFO]  no value change, same mask already set
[    8.955637] iwlwifi 0003:01:00.0: Direct firmware load for iwlwifi-8265-23.ucode failed with error -2
[    8.955852] iwlwifi 0003:01:00.0: Falling back to user helper
[    8.967546] iwlwifi 0003:01:00.0: loaded firmware version 22.391740.0 op_mode iwlmvm
[    9.039677] iwlwifi 0003:01:00.0: Detected Intel(R) Dual Band Wireless AC 8265, REV=0x230
[    9.041776] iwlwifi 0003:01:00.0: L1 Disabled - LTR Enabled
[    9.042071] iwlwifi 0003:01:00.0: L1 Disabled - LTR Enabled
[    9.312877] ieee80211 phy0: Selected rate control algorithm 'iwl-mvm-rs'
[    9.313610] thermal thermal_zone8: failed to read out thermal zone (-5)
[    9.313775] thermal thermal_zone8: Registering thermal zone thermal_zone8 for type iwlwifi
[    9.378795] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[    9.822348] Wake76 for irq=199
[    9.822358] Wake77 for irq=199
[    9.822362] Wake78 for irq=199
[    9.822366] Wake79 for irq=199
[    9.822369] Wake80 for irq=199
[    9.822373] Wake81 for irq=199
[    9.822376] Wake82 for irq=199
[    9.836641] tegra-xusb 3610000.xhci: Upgrade port 0 to USB3.0
[    9.836655] tegra-xusb 3610000.xhci: Upgrade port 1 to USB3.0
[   10.701325] eqos 2490000.ether_qos eth0: Link is Up - 100Mbps/Full - flow control off
[   10.706158] usb usb2: usb_suspend_both: status 0
[   10.706390] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   18.939744] bpmp: mrq 22 took 3996000 us
[   19.469593] zram: Added device: zram0
[   19.479494] zram: Added device: zram1
[   19.482354] zram: Added device: zram2
[   19.483591] zram: Added device: zram3
[   19.504956] zram0: detected capacity change from 0 to 4184698880
[   21.302608] bpmp: mrq 22 took 1904000 us
[   23.368510] bpmp: mrq 22 took 1468000 us
[   26.909160] bpmp: mrq 22 took 1596000 us
[   28.941989] bpmp: mrq 22 took 1256000 us
[   30.316494] Adding 4086616k swap on /dev/zram0.  Priority:5 extents:1 across:4086616k SS
[   30.326645] zram1: detected capacity change from 0 to 4184698880
[   30.353678] usbcore: registered new interface driver cdc_wdm
[   30.358740] Adding 4086616k swap on /dev/zram1.  Priority:5 extents:1 across:4086616k SS
[   30.368182] zram2: detected capacity change from 0 to 4184698880
[   30.408522] Adding 4086616k swap on /dev/zram2.  Priority:5 extents:1 across:4086616k SS
[   30.415456] zram3: detected capacity change from 0 to 4184698880
[   30.456302] Adding 4086616k swap on /dev/zram3.  Priority:5 extents:1 across:4086616k SS
[   30.934236] qmi_wwan_q 1-2:1.4: cdc-wdm0: USB WDM device
[   30.935135] qmi_wwan_q 1-2:1.4: Quectel RM500Q-GL work on RawIP mode
[   30.935768] qmi_wwan_q 1-2:1.4: rx_urb_size = 31744
[   30.939091] qmi_wwan_q 1-2:1.4 rmnet_usb0: register 'qmi_wwan_q' at usb-3610000.xhci-2, RMNET/USB device, 5a:4f:77:cd:bc:be
[   30.941536] net rmnet_usb0: qmap_register_device(rmnet_usb0.1)=0
[   30.941564] net rmnet_usb0: qmap_register_device rmnet_usb0.1
[   30.941837] usbcore: registered new interface driver qmi_wwan_q
[   31.047419] usbcore: registered new interface driver qmi_wwan
[   35.423347] bpmp: mrq 22 took 2500000 us
[   39.871211] net rmnet_usb0: ul_data_aggregation_max_datagrams=11, ul_data_aggregation_max_size=4096, dl_minimum_padding=0
[   40.317809] net rmnet_usb0: link_state 0x0 -> 0x1
[   40.668664] tegradc 15200000.nvdisplay: blank - powerdown
[   40.668684] tegradc 15210000.nvdisplay: blank - powerdown
[   40.668693] tegradc 15220000.nvdisplay: blank - powerdown
[   53.794136] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[   53.794162] Bluetooth: BNEP socket layer initialized
[   54.228849] fuse init (API version 7.26)
**[   63.972816] nvgpu: 17000000.gv11b   __nvgpu_timeout_expired_msg_cpu:94   [ERR]  Timeout detected @ nvgpu_vm_unmap+0x114/0x198 [nvgpu] sync-unmap failed on 0x1efcb00000**
**[   63.973151] nvgpu: 17000000.gv11b                    nvgpu_vm_unmap:1229 [WRN]  2 references remaining on 0x1efcb00000**
**[   66.095468] bpmp: mrq 22 took 1844000 us**
**[   68.460945] nvgpu: 17000000.gv11b   __nvgpu_timeout_expired_msg_cpu:94   [ERR]  Timeout detected @ nvgpu_vm_unmap+0x114/0x198 [nvgpu] sync-unmap failed on 0x1eff000000**
**[   68.461255] nvgpu: 17000000.gv11b                    nvgpu_vm_unmap:1229 [WRN]  3 references remaining on 0x1eff000000**
**[   70.930928] bpmp: mrq 22 took 2148000 us`**
3

Did you dump the log from uart or you always have to type some commands (e.g. dmesg/cat syslog) to check the log?

4

I have to use dmesg/syslog to get those because the boards are in the field and we don’t have access to them other than remote SSH.

5

We need your help to access the board through uart. Only the log from that can tell you what is going on when device reboots. Other log like dmesg or syslog may not tell the exact result.

If you cannot access the board, maybe you can tell us what kind of application that is running on the board so that we can try to reproduce your issue with our device.

6

The board is connected to the internet with a Quectel 5G module that is connected to the board over USB C and an external power supply. The application is a python code that reads UDP packets from Ethernet port and writes it to a Azure instance using a socket connection over 5G.
5G module is using QMI_WWAN Driver using the adjusted image using source code provided by Nvidia and Jetpack.

7

Sounds like not related to GPU usage. If that is your case, then we cannot reproduce your issue because we don’t have 5G module.

Please try to get the log if possible or we cannot help.

8

Thanks. I will see what we can do.

9

I connected one Xavier board that we have here to see if we can reproduce the same issue.
I plug my laptop into the Micro-USB port on Xavier and connecting to serial using minicom on ttyUSB3 port. I see it asks me to log with use and password, but it doesn’t automatically show the dmesg or other logs, for example when I plug and unplug a USB mouse, it typically shows the messages on dmesg, but it doesn’t show on debugging serial port, should do anything or if anything happens for kernel it will automatically push to the serial port?

10

Hi,

It should output some logs when you press reset button. These logs are not dmesg. Please try that first to confirm that you are really dumping logs from uart.

11

I tried to see if this happens with the system that we have in the house.
I’ve got the same reboot and this is the last massages from the system USB debug port.
debug_log.txt (83.7 KB)

" [2020-12-06 00:00:37] ubuntu-desktop login: 
[2020-12-06 22:36:54] Ubuntu 18.04.5 LTS ubuntu-desktop ttyTCU0
[2020-12-06 22:36:54] 
[2020-12-06 22:36:54] ubuntu-desktop login: [244275.081218] INFO: rcu_sched detected stalls on CPUs/tasks:
[2020-12-07 10:02:24] [244275.081441] 	0-...: (920 ticks this GP) idle=e73/140000000000002/0 softirq=48861044/48861044 fqs=2352 
[2020-12-07 10:02:24] [244275.081613] 	(detected by 5, t=5252 jiffies, g=4877268, c=4877267, q=2)
[2020-12-07 10:02:25] [244276.249165] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ksoftirqd/0:3]
[2020-12-07 10:02:25] [244276.249755] Kernel panic - not syncing: softlockup: hung tasks
[2020-12-07 10:02:25] [244276.249884] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G             L  4.9.140-tegra #1
[2020-12-07 10:02:25] [244276.250036] Hardware name: Jetson-AGX (DT)
[2020-12-07 10:02:25] [244276.250129] Call trace:
[2020-12-07 10:02:25] [244276.250207] [<ffffff800808bdb8>] dump_backtrace+0x0/0x198
[2020-12-07 10:02:25] [244276.250321] [<ffffff800808c37c>] show_stack+0x24/0x30
[2020-12-07 10:02:25] [244276.250429] [<ffffff800845c7a0>] dump_stack+0x98/0xc0
[2020-12-07 10:02:25] [244276.250548] [<ffffff80081c1438>] panic+0x11c/0x298
[2020-12-07 10:02:25] [244276.250657] [<ffffff8008181760>] watchdog_unpark_threads+0x0/0x98
[2020-12-07 10:02:25] [244276.250788] [<ffffff80081399e0>] __hrtimer_run_queues+0xd8/0x360
[2020-12-07 10:02:25] [244276.250912] [<ffffff800813a330>] hrtimer_interrupt+0xa8/0x1e0
[2020-12-07 10:02:25] [244276.251046] [<ffffff8008bffe98>] arch_timer_handler_phys+0x38/0x58
[2020-12-07 10:02:25] [244276.251181] [<ffffff8008126f10>] handle_percpu_devid_irq+0x90/0x2b0
[2020-12-07 10:02:25] [244276.251309] [<ffffff80081214f4>] generic_handle_irq+0x34/0x50
[2020-12-07 10:02:25] [244276.251728] [<ffffff8008121bd8>] __handle_domain_irq+0x68/0xc0
[2020-12-07 10:02:25] [244276.252208] [<ffffff8008080d44>] gic_handle_irq+0x5c/0xb0
[2020-12-07 10:02:25] [244276.255282] [<ffffff8008082c28>] el1_irq+0xe8/0x194
[2020-12-07 10:02:25] [244276.260537] [<ffffff8008d94314>] csum_partial_ext+0xc/0x18
[2020-12-07 10:02:25] [244276.266219] [<ffffff8008d96a70>] __skb_checksum+0x70/0x358
[2020-12-07 10:02:25] [244276.271901] [<ffffff8008d96dac>] skb_checksum+0x54/0x68
[2020-12-07 10:02:25] [244276.277071] [<ffffff8008d9e178>] __skb_checksum_complete+0x30/0xc8
[2020-12-07 10:02:25] [244276.283544] [<ffffff8008e24aa0>] tcp_v4_rcv+0x590/0xc00
[2020-12-07 10:02:25] [244276.288621] [<ffffff8008dfb4b0>] ip_local_deliver_finish+0x80/0x278
[2020-12-07 10:02:25] [244276.295006] [<ffffff8008dfbbfc>] ip_local_deliver+0x54/0xf0
[2020-12-07 10:02:25] [244276.300781] [<ffffff8008dfb880>] ip_rcv_finish+0x1d8/0x3a0
[2020-12-07 10:02:25] [244276.306379] [<ffffff8008dfbf08>] ip_rcv+0x270/0x3a8
[2020-12-07 10:02:25] [244276.311203] [<ffffff8008da9c20>] __netif_receive_skb_core+0x3b8/0xad8
[2020-12-07 10:02:25] [244276.317925] [<ffffff8008dad010>] __netif_receive_skb+0x28/0x78
[2020-12-07 10:02:25] [244276.323964] [<ffffff8008daf5dc>] process_backlog+0x94/0x140
[2020-12-07 10:02:25] [244276.329740] [<ffffff8008daf2e4>] net_rx_action+0xf4/0x358
[2020-12-07 10:02:25] [244276.335075] [<ffffff8008081054>] __do_softirq+0x13c/0x3b0
[2020-12-07 10:02:25] [244276.340162] [<ffffff80080bb218>] irq_exit+0xd0/0x118
[2020-12-07 10:02:25] [244276.345487] [<ffffff8008121bdc>] __handle_domain_irq+0x6c/0xc0
[2020-12-07 10:02:25] [244276.351524] [<ffffff8008080d44>] gic_handle_irq+0x5c/0xb0
[2020-12-07 10:02:25] [244276.356861] [<ffffff8008082c28>] el1_irq+0xe8/0x194
[2020-12-07 10:02:25] [244276.361416] [<ffffff80080baf3c>] run_ksoftirqd+0x4c/0x58
[2020-12-07 10:02:25] [244276.367108] [<ffffff80080e07c8>] smpboot_thread_fn+0x160/0x248
[2020-12-07 10:02:25] [244276.372705] [<ffffff80080dbe64>] kthread+0xec/0xf0
[2020-12-07 10:02:25] [244276.377690] [<ffffff80080838a0>] ret_from_fork+0x10/0x30
[2020-12-07 10:02:25] [244276.383475] SMP: stopping secondary CPUs
[2020-12-07 10:02:25] [244276.387701] Kernel Offset: disabled
[2020-12-07 10:02:25] [244276.390990] Memory Limit: none
[2020-12-07 10:02:25] [244276.394142] trusty-log panic notifier - trusty version Built: 20:53:35 Oct 27 2020 [244276.416301] Rebooting in 5 seconds..
[2020-12-07 10:02:30] ÿäÿâShutdown state requested 1
[2020-12-07 10:02:30] Rebooting system ...
[2020-12-07 10:02:30] ÿâ
[2020-12-07 10:02:30] [0000.055] W> RATCHET: MB1 binary ratchet value 4 is too large than ratchet level 2 from HW fuses.
[2020-12-07 10:02:30] [0000.063] I> MB1 (prd-version: 1.5.1.3-t194-41334769-d2a21c57)
[2020-12-07 10:02:30] [0000.069] I> Boot-mode: Coldboot
[2020-12-07 10:02:30] [0000.071] I> Chip revision : A02P
[2020-12-07 10:02:30] [0000.075] I> Bootrom patch version : 15 (correctly patched)
[2020-12-07 10:02:30] [0000.080] I> ATE fuse revision : 0x200
[2020-12-07 10:02:30] [0000.083] I> Ram repair fuse : 0x0
[2020-12-07 10:02:30] [0000.086] I> Ram Code : 0x2
[2020-12-07 10:02:30] [0000.089] I> rst_source : 0xb
[2020-12-07 10:02:30] [0000.091] I> rst_level : 0x1
[2020-12-07 10:02:30] [0000.095] I> Boot-device: eMMC
[2020-12-07 10:02:30] [0000.110] I> sdmmc DDR50 mode
[2020-12-07 10:02:30] [0000.114] I> Active Boot chain : 1
[2020-12-07 10:02:30] [0000.117] I> Boot-device: eMMC
[2020-12-07 10:02:30] [0000.121] W> MB1_PLATFORM_CONFIG: device prod data is empty in MB1 BCT.
[2020-12-07 10:02:30] [0000.127] I> Temperature = 43000
[2020-12-07 10:02:30] [0000.130] W> Skipping boost for clk: BPMP_CPU_NIC
[2020-12-07 10:02:30] [0000.135] W> Skipping boost for clk: BPMP_APB
[2020-12-07 10:02:30] [0000.139] W> Skipping boost for clk: AXI_CBB
[2020-12-07 10:02:30] [0000.143] W> Skipping boost for clk: AON_CPU_NIC
[2020-12-07 10:02:30] [0000.147] W> Skipping boost for clk: CAN1
[2020-12-07 10:02:30] [0000.151] W> Skipping boost for clk: CAN2
[2020-12-07 10:02:30] [0000.155] I> Boot-device: eMMC
[2020-12-07 10:02:30] [0000.158] I> Boot-device: eMMC
[2020-12-07 10:02:30] [0000.167] I> Sdmmc: HS400 mode enabled
[2020-12-07 10:02:30] [0000.172] I> ECC region[0]: Start:0x0, End:0x0
[2020-12-07 10:02:30] [0000.176] I> ECC region[1]: Start:0x0, End:0x0
[2020-12-07 10:02:30] [0000.180] I> ECC region[2]: Start:0x0, End:0x0
[2020-12-07 10:02:30] [0000.184] I> ECC region[3]: Start:0x0, End:0x0
[2020-12-07 10:02:30] [0000.188] I> ECC region[4]: Start:0x0, End:0x0
[2020-12-07 10:02:30] [0000.192] I> Non-ECC region[0]: Start:0x80000000, End:0x100000000
[2020-12-07 10:02:30] [0000.198] I> Non-ECC region[1]: Start:0x0, End:0x0
[2020-12-07 10:02:30] [0000.202] I> Non-ECC region[2]: Start:0x0, End:0x0
[2020-12-07 10:02:30] [0000.207] I> Non-ECC region[3]: Start:0x0, End:0x0
[2020-12-07 10:02:30] [0000.211] I> Non-ECC region[4]: Start:0x0, End:0x0
[2020-12-07 10:02:30] [0000.217] E> FAILED: Thermal config
[2020-12-07 10:02:30] [0000.224] E> FAILED: MEMIO rail config
[2020-12-07 10:02:30] [0000.243] I> Boot-device: eMMC
[2020-12-07 10:02:30] [0000.252] I> sdmmc bdev is already initialized
[2020-12-07 10:02:30] [0000.324] I> MB1 done
[2020-12-07 10:02:30] 
[2020-12-07 10:02:30] ÿýÿàmain enter
[2020-12-07 10:02:30] SPE VERSION #: R01.00.14 Created: Sep 19 2018 @ 11:03:21
[2020-12-07 10:02:30] HW Function test
[2020-12-07 10:02:30] Start Scheduler.
[2020-12-07 10:02:30] in late init
[2020-12-07 10:02:30] ÿâ
[2020-12-07 10:02:30] [0000.332] I> Welcome to MB2(TBoot-BPMP) (version: 00.00.2018.32-mobile-389501e9)
[2020-12-07 10:02:30] [0000.333] I> DMA Heap @ [0x526fa000 - 0x52ffa000]
[2020-12-07 10:02:30] [0000.334] I> Default Heap @ [0xd486400 - 0xd48a400]
[2020-12-07 10:02:30] [0000.334] E> DEVICE_PROD: Invalid value data = 70020000, size = 0.
[2020-12-07 10:02:30] [0000.340] W> device prod register failed
[2020-12-07 10:02:30] [0000.344] I> Boot-device: eMMC
[2020-12-07 10:02:30] [0000.347] I> Boot_device: SDMMC_BOOT instance: 3
[2020-12-07 10:02:30] [0000.353] I> sdmmc-3 params source = boot args
[2020-12-07 10:02:30] [0000.356] I> sdmmc bdev is already initialized
[2020-12-07 10:02:30] [0000.360] I> sdmmc-3 params source = boot args
[2020-12-07 10:02:30] [0000.368] I> Found 17 partitions in SDMMC_BOOT (instance 3)
[2020-12-07 10:02:30] [0000.375] I> Found 42 partitions in SDMMC_USER (instance 3)
[2020-12-07 10:02:30] [0000.376] I> Active Boot chain : 1
[2020-12-07 10:02:30] [0000.379] I> parsing oem signed section of bpmp-fw header done
[2020-12-07 10:02:30] [0000.384] I> bpmp-fw binary init read from storage
[2020-12-07 10:02:30] [0000.389] I> oem authentication of bpmp-fw header done
[2020-12-07 10:02:30] [0000.395] I> bpmp-fw binary done read from storage
[2020-12-07 10:02:30] [0000.398] I> bpmp-fw: Authentication init Done
[2020-12-07 10:02:30] [0000.403] I> parsing oem signed section of cpubl header done
[2020-12-07 10:02:30] [0000.408] I> cpubl binary init read from storage
[2020-12-07 10:02:30] [0000.412] I> bpmp-fw: Authentication Finalize Done
[2020-12-07 10:02:30] [0000.417] I> oem authentication of cpubl header done
[2020-12-07 10:02:30] [0000.422] I> cpubl binary done read from storage
[2020-12-07 10:02:30] [0000.426] I> cpubl: Authentication init Done
[2020-12-07 10:02:30] [0000.431] I> parsing oem signed section of rce header done
[2020-12-07 10:02:30] [0000.436] I> rce binary init read from storage
[2020-12-07 10:02:30] [0000.440] I> Relocating BR-BCT
[2020-12-07 10:02:30] [0000.443] I> cpubl: Authentication Finalize Done
[2020-12-07 10:02:30] [0000.448] I> oem authentication of rce header done
[2020-12-07 10:02:30] [0000.452] I> rce binary done read from storage
[2020-12-07 10:02:30] [0000.456] I> rce: Authentication init Done
[2020-12-07 10:02:30] [0000.461] I> parsing oem signed section of ape header done
[2020-12-07 10:02:30] [0000.466] I> ape binary init read from storage
[2020-12-07 10:02:30] [0000.470] I> rce: Authentication Finalize Done
[2020-12-07 10:02:30] [0000.474] I> oem authentication of ape header done
[2020-12-07 10:02:30] [0000.479] I> ape binary done read from storage
[2020-12-07 10:02:30] [0000.483] I> ape: Authentication init Done
[2020-12-07 10:02:30] [0000.488] I> parsing oem signed section of tos header done
[2020-12-07 10:02:30] [0000.492] I> tos binary init read from storage
[2020-12-07 10:02:30] [0000.497] I> ape: Authentication Finalize Done
[2020-12-07 10:02:30] [0000.502] I> oem authentication of tos header done
[2020-12-07 10:02:30] [0000.506] I> tos binary done read from storage
[2020-12-07 10:02:30] [0000.510] I> tos: Authentication init Done
[2020-12-07 10:02:30] [0000.514] I> parsing oem signed section of bpmp-fw-dtb header done
[2020-12-07 10:02:31] [0000.520] I> bpmp-fw-dtb binary init read from storage
[2020-12-07 10:02:31] [0000.525] I> tos: Authentication Finalize Done
[2020-12-07 10:02:31] [0000.531] I> oem authentication of bpmp-fw-dtb header done
[2020-12-07 10:02:31] [0000.537] I> bpmp-fw-dtb binary done read from storage
[2020-12-07 10:02:31] [0000.540] I> bpmp-fw-dtb: Authentication init Done
[2020-12-07 10:02:31] [0000.544] I> parsing oem signed section of cpubl-dtb header done
[2020-12-07 10:02:31] [0000.550] I> cpubl-dtb binary init read from storage
[2020-12-07 10:02:31] [0000.555] I> bpmp-fw-dtb: Authentication Finalize Done
[2020-12-07 10:02:31] [0000.592] I> oem authentication of cpubl-dtb header done
[2020-12-07 10:02:31] [0000.593] I> cpubl-dtb binary done read from storage
[2020-12-07 10:02:31] [0000.593] I> cpubl-dtb: Authentication init Done
[2020-12-07 10:02:31] [0000.595] I> parsing oem signed section of eks header done
[2020-12-07 10:02:31] [0000.596] I> eks binary init read from storage
[2020-12-07 10:02:31] [0000.597] I> cpubl-dtb: Authentication Finalize Done
[2020-12-07 10:02:31] [0000.597] I> oem authentication of eks header done
[2020-12-07 10:02:31] [0000.601] I> eks binary done read from storage
[2020-12-07 10:02:31] [0000.605] I> eks: Authentication init Done
[2020-12-07 10:02:31] [0000.609] I> eks: Authentication Finalize Done
[2020-12-07 10:02:31] [0000.613] I> EKB detected (length: 0x410) @ VA:0x5270a400
[2020-12-07 10:02:31] ÿäNOTICE:  BL31: v1.3(release):de895fd9e
[2020-12-07 10:02:31] NOTICE:  BL31: Built : 20:51:20, Oct 27 2020
[2020-12-07 10:02:31] ipc-unittest-main: 1519: Welcome to IPC unittest!!!
[2020-12-07 10:02:31] ipc-unittest-main: 1531: waiting forever
[2020-12-07 10:02:31] ipc-unittest-srv: 329: Init unittest services!!!
[2020-12-07 10:02:31] hwkey-agent: 40: hwkey-agent is running!!
[2020-12-07 10:02:31] hwkey-agent: 182: key_mgnt_processing .......
[2020-12-07 10:02:31] hwkey-agent: 157: Init hweky-agent services!!
[2020-12-07 10:02:31] platform_bootstrap_epilog: trusty bootstrap complete
[2020-12-07 10:02:31] ÿâ
[2020-12-07 10:02:31] 
[2020-12-07 10:02:31] welcome to lk
[2020-12-07 10:02:31] calling constructors
[2020-12-07 10:02:31] initializing heap
[2020-12-07 10:02:31] creating bootstrap completion thread
[2020-12-07 10:02:31] top of bootstrap2()
[2020-12-07 10:02:31] initializing platform
[2020-12-07 10:02:31] bpmp: platform_init
[2020-12-07 10:02:31] tag is 57f8a77779f848bf2ecf21dabee5645f
[2020-12-07 10:02:31] tag_show initialized
[2020-12-07 10:02:31] dt initialized
[2020-12-07 10:02:31] mail initialized
[2020-12-07 10:02:31] chipid initialized
[2020-12-07 10:02:31] fuse initialized
[2020-12-07 10:02:31] sku initialized
[2020-12-07 10:02:31] speedo initialized
[2020-12-07 10:02:31] ec_get_ec_list: found 45 ecs
[2020-12-07 10:02:31] ec initialized
[2020-12-07 10:02:31] ec_mrq initialized
[2020-12-07 10:02:31] vmon_populate_monitors: found 3 monitors
[2020-12-07 10:02:31] vmon initialized
[2020-12-07 10:02:31] adc initialized
[2020-12-07 10:02:31] fmon_populate_monitors: found 73 monitors
[2020-12-07 10:02:31] fmon initialized
[2020-12-07 10:02:31] fmon_mrq initialized
[2020-12-07 10:02:31] reset initialized
[2020-12-07 10:02:31] nvhs initialized
[2020-12-07 10:02:31] 392 clocks registered
[2020-12-07 10:02:31] WARNING: pll_c4 has no dyn ramp
[2020-12-07 10:02:31] clk_mrq_init: mrq handler registered
[2020-12-07 10:02:31] clk initialized
[2020-12-07 10:02:31] nvlink initialized
[2020-12-07 10:02:31] io_dpd initialized
[2020-12-07 10:02:31] io_dpd initialized
[2020-12-07 10:02:31] thermal initialized
[2020-12-07 10:02:31] i2c5 controller initialized
[2020-12-07 10:02:31] initialized i2c mrq handling
[2020-12-07 10:02:31] i2c initialized
[2020-12-07 10:02:31] regulator initialized
[2020-12-07 10:02:31] avfs_clk_platform initialized
[2020-12-07 10:02:31] soctherm initialized
[2020-12-07 10:02:31] aotag initialized
[2020-12-07 10:02:31] powergate initialized
[2020-12-07 10:02:31] dvs initialized
[2020-12-07 10:02:31] pm initialized
[2020-12-07 10:02:31] pg_late initialized
[2020-12-07 10:02:31] strap initialized
[2020-12-07 10:02:31] tag initialized
[2020-12-07 10:02:31] emc initialized
[2020-12-07 10:02:31] clk_dt initialized
[2020-12-07 10:02:31] avfs_ccplex_platform initialized
[2020-12-07 10:02:31] tj_max: dt node not found
[2020-12-07 10:02:31] tj_init initialized
[2020-12-07 10:02:31] uphy_mrq_init: mrq handler registered
[2020-12-07 10:02:31] uphy_dt initialized
[2020-12-07 10:02:31] uphy initialized
[2020-12-07 10:02:31] safereg_init: period 80 ms
[2020-12-07 10:02:31] ec_late initialized
[2020-12-07 10:02:31] mrq initialized
[2020-12-07 10:02:31] ÿá
[2020-12-07 10:02:31] [0001.117] I> Welcome to Cboot
[2020-12-07 10:02:31] ÿâfmon_post initialized
[2020-12-07 10:02:31] ÿá[0001.117] I> Cboot Version: t194-aad9d75e
[2020-12-07 10:02:31] [0001.117] I> CPU-BL Params @ 0xf2820000
[2020-12-07 10:02:31] [0001.118] I>  0) Base:0x00000000 Size:0x00000000
[2020-12-07 10:02:31] [0001.121] I>  1) Base:0xf1100000 Size:0x00100000
[2020-12-07 10:02:31] [0001.126] I>  2) Base:0xf2000000 Size:0x00200000
[2020-12-07 10:02:31] [0001.130] I>  3) Base:0xf1200000 Size:0x00200000
[2020-12-07 10:02:31] ÿâclk_set_parent failed for clk can1, parent pll_aon (-22)
[2020-12-07 10:02:31] clk_set_parent failed for clk can2, parent pll_aon (-22)
[2020-12-07 10:02:31] clk_set_parent failed for clk dmic5, parent pll_aon (-22)
[2020-12-07 10:02:31] clk_set_parent failed for clk i2c2, parent pll_aon (-22)
[2020-12-07 10:02:31] clk_set_parent failed for clk i2c8, parent pll_aon (-22)
[2020-12-07 10:02:31] clk_set_parent failed for clk spi2, parent pll_aon (-22)
[2020-12-07 10:02:31] clk_set_parent failed for clk pwm4, parent pll_aon (-22)
[2020-12-07 10:02:31] clk_dt_late initialized
[2020-12-07 10:02:31] machine_check initialized
[2020-12-07 10:02:31] pm_post initialized
[2020-12-07 10:02:31] dbells initialized
[2020-12-07 10:02:31] avfs_clk_platform_post initialized
[2020-12-07 10:02:31] dmce initialized
[2020-12-07 10:02:31] cvc initialized
[2020-12-07 10:02:31] ccplex_avfs_hw_init: nafll_cluster0: not monitored
[2020-12-07 10:02:31] ccplex_avfs_hw_init: nafll_cluster1: not monitored
[2020-12-07 10:02:31] ccplex_avfs_hw_init: nafll_cluster2: not monitored
[2020-12-07 10:02:31] ccplex_avfs_hw_init: nafll_cluster3: not monitored
[2020-12-07 10:02:31] avfs_clk_mach_post initialized
[2020-12-07 10:02:31] regulator_post initialized
[2020-12-07 10:02:31] rm initialized
[2020-12-07 10:02:31] sc7_diag initialized
[2020-12-07 10:02:31] thermal_test initialized
[2020-12-07 10:02:31] serial_late initialized
[2020-12-07 10:02:31] clk_post initialized
[2020-12-07 10:02:31] clk_dt_post initialized
[2020-12-07 10:02:31] mc_reg initialized
[2020-12-07 10:02:31] pg_post initialized
[2020-12-07 10:02:31] dyn_modules initialized
[2020-12-07 10:02:31] sku_debugfs initialized
[2020-12-07 10:02:31] speedo_debugfs initialized
[2020-12-07 10:02:31] adc_debugfs initialized
[2020-12-07 10:02:31] clk_debugfs initialized
[2020-12-07 10:02:31] ÿá[0001.134] I>  4) Base:0xf1000000 Size:0x00100000
[2020-12-07 10:02:31] [0001.239] I>  5) Base:0xf0f00000 Size:0x00100000
[2020-12-07 10:02:31] [0001.244] I>  6) Base:0xf3800000 Size:0x00400000
[2020-12-07 10:02:31] [0001.248] I>  7) Base:0xf1c00000 Size:0x00400000
[2020-12-07 10:02:31] [0001.253] I>  8) Base:0xf0e00000 Size:0x00100000
[2020-12-07 10:02:31] [0001.257] I>  9) Base:0xf0d00000 Size:0x00100000
[2020-12-07 10:02:31] [0001.262] I> 10) Base:0xf3000000 Size:0x00800000
[2020-12-07 10:02:31] [0001.266] I> 11) Base:0x40000000 Size:0x00040000
[2020-12-07 10:02:31] [0001.271] I> 12) Base:0xf0c00000 Size:0x00100000
[2020-12-07 10:02:31] [0001.275] I> 13) Base:0x40046000 Size:0x00002000
[2020-12-07 10:02:31] ÿâemc_debugfs initialized
[2020-12-07 10:02:31] dvs_debugfs initialized
[2020-12-07 10:02:31] fmon_debugfs initialized
[2020-12-07 10:02:31] vmon_debugfs initialized
[2020-12-07 10:02:31] pg_debugfs initialized
[2020-12-07 10:02:31] profile_fs initialized
[2020-12-07 10:02:31] debugfs_cons initialized
[2020-12-07 10:02:31] mail_fs initialized
[2020-12-07 10:02:31] profile initialized
[2020-12-07 10:02:31] cvc_debugfs initialized
[2020-12-07 10:02:31] dmce_debugfs initialized
[2020-12-07 10:02:31] ec_debugfs initialized
[2020-12-07 10:02:31] rm_debugfs initialized
[2020-12-07 10:02:31] soctherm_debug initialized
[2020-12-07 10:02:31] gr_reader initialized
[2020-12-07 10:02:31] mods initialized
[2020-12-07 10:02:31] dt_fs initialized
[2020-12-07 10:02:31] debugfs_mrq initialized
[2020-12-07 10:02:31] debug_mrq initialized
[2020-12-07 10:02:31] debug_safereg initialized
[2020-12-07 10:02:31] initializing target
[2020-12-07 10:02:31] calling apps_init()
[2020-12-07 10:02:31] starting app shell
[2020-12-07 10:02:31] entering main console loop
[2020-12-07 10:02:31] ] ÿá[0001.280] I> 14) Base:0x40048000 Size:0x00002000
[2020-12-07 10:02:31] [0001.334] I> 15) Base:0xac000000 Size:0x00004000
[2020-12-07 10:02:31] [0001.339] I> 16) Base:0x4004a000 Size:0x00002000
[2020-12-07 10:02:31] [0001.343] I> 17) Base:0xf0b00000 Size:0x00100000
[2020-12-07 10:02:31] [0001.348] I> 18) Base:0x4004c000 Size:0x00002000
[2020-12-07 10:02:31] [0001.352] I> 19) Base:0xf2200000 Size:0x00600000
[2020-12-07 10:02:31] [0001.357] I> 20) Base:0x4004e000 Size:0x00002000
[2020-12-07 10:02:31] [0001.361] I> 21) Base:0xf09d0000 Size:0x0000c000
[2020-12-07 10:02:31] [0001.366] I> 22) Base:0x00000000 Size:0x00000000
[2020-12-07 10:02:31] [0001.370] I> 23) Base:0xf09e0000 Size:0x00020000
[2020-12-07 10:02:31] [0001.375] I> 24) Base:0xf6000000 Size:0x02000000
[2020-12-07 10:02:31] [0001.379] I> 25) Base:0x40050000 Size:0x00002000
[2020-12-07 10:02:31] [0001.383] I> 26) Base:0x40040000 Size:0x00006000
[2020-12-07 10:02:31] [0001.388] I> 27) Base:0xf1800000 Size:0x00400000
[2020-12-07 10:02:31] [0001.392] I> 28) Base:0xf4c00000 Size:0x01400000
[2020-12-07 10:02:31] [0001.397] I> 29) Base:0xf1400000 Size:0x00400000
[2020-12-07 10:02:31] [0001.401] I> 30) Base:0xf0a00000 Size:0x00100000
[2020-12-07 10:02:31] [0001.406] I> 31) Base:0x00000000 Size:0x00000000
[2020-12-07 10:02:31] [0001.410] I> 32) Base:0xf8000000 Size:0x08000000
[2020-12-07 10:02:31] [0001.415] I> 33) Base:0x00000000 Size:0x00000000
[2020-12-07 10:02:31] [0001.419] I> 34) Base:0xf3c00000 Size:0x01000000
[2020-12-07 10:02:31] [0001.424] I> 35) Base:0xab000000 Size:0x01000000
[2020-12-07 10:02:31] [0001.428] I> 36) Base:0xa0000000 Size:0x0b000000
[2020-12-07 10:02:31] [0001.433] I> 37) Base:0xf2800000 Size:0x00800000
[2020-12-07 10:02:31] [0001.437] I> 38) Base:0x80000000 Size:0x20000000
[2020-12-07 10:02:31] [0001.441] I> 39) Base:0xb0000000 Size:0x08000000
[2020-12-07 10:02:31] [0001.446] I> 40) Base:0x00000000 Size:0x00000000
[2020-12-07 10:02:31] [0001.450] I> 41) Base:0x00000000 Size:0x00000000
[2020-12-07 10:02:31] [0001.455] I> 42) Base:0x00000000 Size:0x00000000
[2020-12-07 10:02:31] [0001.459] I> 43) Base:0x00000000 Size:0x00000000
[2020-12-07 10:02:31] [0001.464] I> 44) Base:0x00000000 Size:0x00000000
[2020-12-07 10:02:31] [0001.468] I> 45) Base:0x00000000 Size:0x00000000
[2020-12-07 10:02:31] [0001.473] GIC-SPI Target CPU: 0
[2020-12-07 10:02:31] [0001.476] Interrupts Init done
[2020-12-07 10:02:31] [0001.479] calling constructors
[2020-12-07 10:02:31] [0001.481] initializing heap
[2020-12-07 10:02:31] [0001.484] I> Heap: [0xa0691568 ... 0xab000000]
[2020-12-07 10:02:31] [0001.488] initializing threads
[2020-12-07 10:02:31] [0001.491] initializing timers
[2020-12-07 10:02:31] [0001.494] creating bootstrap completion thread
[2020-12-07 10:02:31] [0001.498] top of bootstrap2()
[2020-12-07 10:02:31] [0001.501] CPU: MIDR: 0x4E0F0040, MPIDR: 0x80000000
[2020-12-07 10:02:31] [0001.506] initializing platform
[2020-12-07 10:02:31] [0001.509] E> DEVICE_PROD: Invalid value data = 0, size = 0.
[2020-12-07 10:02:31] [0001.514] W> device prod register failed
[2020-12-07 10:02:31] [0001.518] I> Bl_dtb @0xaaf00000
[2020-12-07 10:02:31] [0001.524] W> "plugin-manager" doesn't exist, creating
[2020-12-07 10:02:32] [0001.526] W> "ids" doesn't exist, creating
[2020-12-07 10:02:32] [0001.530] W> "connection" doesn't exist, creating
[2020-12-07 10:02:32] [0001.534] W> "configs" doesn't exist, creating
[2020-12-07 10:02:32] [0001.545] I> Find /i2c@3160000's alias i2c0
[2020-12-07 10:02:32] [0001.546] I> Reading eeprom i2c=0 address=0x50
[2020-12-07 10:02:32] [0001.572] I> Device at /i2c@3160000:0x50
[2020-12-07 10:02:32] [0001.572] I> Reading eeprom i2c=0 address=0x56
[2020-12-07 10:02:32] [0001.597] I> Device at /i2c@3160000:0x56
[2020-12-07 10:02:32] [0001.598] I> Find /i2c@3180000's alias i2c2
[2020-12-07 10:02:32] [0001.599] I> Reading eeprom i2c=2 address=0x54
[2020-12-07 10:02:32] [0001.600] E> I2C: slave not found in slaves.
[2020-12-07 10:02:32] [0001.601] E> I2C: Could not write 0 bytes to slave: 0x00a8 with repeat start true.
[2020-12-07 10:02:32] [0001.602] E> I2C_DEV: Failed to send register address 0x00000000.
[2020-12-07 10:02:32] [0001.602] E> I2C_DEV: Could not read 256 registers of size 1 from slave 0xa8 at 0x00000000 via instance 2.
[2020-12-07 10:02:32] [0001.611] E> eeprom: Failed to read I2C slave device
[2020-12-07 10:02:32] [0001.616] I> Eeprom read failed 0x3526070d
[2020-12-07 10:02:32] [0001.620] I> Reading eeprom i2c=2 address=0x57
[2020-12-07 10:02:32] [0001.624] E> I2C: slave not found in slaves.
[2020-12-07 10:02:32] [0001.628] E> I2C: Could not write 0 bytes to slave: 0x00ae with repeat start true.
[2020-12-07 10:02:32] [0001.636] E> I2C_DEV: Failed to send register address 0x00000000.
[2020-12-07 10:02:32] [0001.642] E> I2C_DEV: Could not read 256 registers of size 1 from slave 0xae at 0x00000000 via instance 2.
[2020-12-07 10:02:32] [0001.651] E> eeprom: Failed to read I2C slave device
[2020-12-07 10:02:32] [0001.656] I> Eeprom read failed 0x3526070d
[2020-12-07 10:02:32] [0001.660] I> Reading eeprom i2c=2 address=0x52
[2020-12-07 10:02:32] [0001.664] E> I2C: slave not found in slaves.
[2020-12-07 10:02:32] [0001.668] E> I2C: Could not write 0 bytes to slave: 0x00a4 with repeat start true.
[2020-12-07 10:02:32] [0001.676] E> I2C_DEV: Failed to send register address 0x00000000.
[2020-12-07 10:02:32] [0001.682] E> I2C_DEV: Could not read 256 registers of size 1 from slave 0xa4 at 0x00000000 via instance 2.
[2020-12-07 10:02:32] [0001.691] E> eeprom: Failed to read I2C slave device
[2020-12-07 10:02:32] [0001.696] I> Eeprom read failed 0x3526070d
[2020-12-07 10:02:32] [0001.701] I> Find /i2c@c240000's alias i2c1
[2020-12-07 10:02:32] [0001.704] I> Reading eeprom i2c=1 address=0x52
[2020-12-07 10:02:32] [0001.710] E> I2C: slave not found in slaves.
[2020-12-07 10:02:32] [0001.712] E> I2C: Could not write 0 bytes to slave: 0x00a4 with repeat start true.
[2020-12-07 10:02:32] [0001.720] E> I2C_DEV: Failed to send register address 0x00000000.
[2020-12-07 10:02:32] [0001.726] E> I2C_DEV: Could not read 256 registers of size 1 from slave 0xa4 at 0x00000000 via instance 1.
[2020-12-07 10:02:32] [0001.735] E> eeprom: Retry to read I2C slave device.
[2020-12-07 10:02:32] [0001.740] E> I2C: slave not found in slaves.
[2020-12-07 10:02:32] [0001.744] E> I2C: Could not write 0 bytes to slave: 0x00a4 with repeat start true.
[2020-12-07 10:02:32] [0001.752] E> I2C_DEV: Failed to send register address 0x00000000.
[2020-12-07 10:02:32] [0001.758] E> I2C_DEV: Could not read 256 registers of size 1 from slave 0xa4 at 0x00000000 via instance 1.
[2020-12-07 10:02:32] [0001.767] E> eeprom: Failed to read I2C slave device
[2020-12-07 10:02:32] [0001.772] I> Eeprom read failed 0x3526070d
[2020-12-07 10:02:32] [0001.776] I> Reading eeprom i2c=1 address=0x50
[2020-12-07 10:02:32] [0001.780] E> I2C: slave not found in slaves.
[2020-12-07 10:02:32] [0001.784] E> I2C: Could not write 0 bytes to slave: 0x00a0 with repeat start true.
[2020-12-07 10:02:32] [0001.792] E> I2C_DEV: Failed to send register address 0x00000000.
[2020-12-07 10:02:32] [0001.798] E> I2C_DEV: Could not read 256 registers of size 1 from slave 0xa0 at 0x00000000 via instance 1.
[2020-12-07 10:02:32] [0001.807] E> eeprom: Retry to read I2C slave device.
[2020-12-07 10:02:32] [0001.812] E> I2C: slave not found in slaves.
[2020-12-07 10:02:32] [0001.816] E> I2C: Could not write 0 bytes to slave: 0x00a0 with repeat start true.
[2020-12-07 10:02:32] [0001.824] E> I2C_DEV: Failed to send register address 0x00000000.
[2020-12-07 10:02:32] [0001.829] E> I2C_DEV: Could not read 256 registers of size 1 from slave 0xa0 at 0x00000000 via instance 1.
[2020-12-07 10:02:32] [0001.839] E> eeprom: Failed to read I2C slave device
[2020-12-07 10:02:32] [0001.844] I> Eeprom read failed 0x3526070d
[2020-12-07 10:02:32] [0001.848] I> create_pm_ids: id: 2888-0004-400-L, len: 15
[2020-12-07 10:02:32] [0001.853] I> config: mem-type:00,power-config:00,misc-config:00,modem-config:00,touch-config:00,display-config:00,, len: 93
[2020-12-07 10:02:32] [0001.864] I> create_pm_ids: id: 2822-0000-700-K, len: 15
[2020-12-07 10:02:32] [0001.869] I> config: mem-type:00,power-config:00,misc-config:00,modem-config:00,touch-config:00,display-config:00,, len: 93
[2020-12-07 10:02:32] [0001.880] I> Adding plugin-manager/ids/2888-0004-400=/i2c@3160000:module@0x50
[2020-12-07 10:02:32] [0001.888] W> "i2c@3160000" doesn't exist, creating
[2020-12-07 10:02:32] [0001.892] W> "module@0x50" doesn't exist, creating
[2020-12-07 10:02:32] [0001.897] I> Adding plugin-manager/ids/2822-0000-700=/i2c@3160000:module@0x56
[2020-12-07 10:02:32] [0001.904] W> "module@0x56" doesn't exist, creating
[2020-12-07 10:02:32] [0001.911] I> Adding plugin-manager/cvm
[2020-12-07 10:02:32] [0001.912] W> "chip-id" doesn't exist, creating
[2020-12-07 10:02:32] [0001.916] I> Adding plugin-manager/chip-id/A02P
[2020-12-07 10:02:32] [0001.920] I> Plugin-manager override starting
[2020-12-07 10:02:32] [0001.926] I> node /plugin-manager/fragement-tegra-wdt-en matches
[2020-12-07 10:02:32] [0001.936] I> node /plugin-manager/fragement-soft-wdt matches
[2020-12-07 10:02:32] [0001.944] I> node /plugin-manager/fragment-pcie-c5-rp matches
[2020-12-07 10:02:32] [0001.949] I> node /plugin-manager/fragment-tegra-ufs-lane10 matches
[2020-12-07 10:02:32] [0001.959] I> Disable plugin-manager status in FDT
[2020-12-07 10:02:32] [0001.960] I> Plugin-manager override finished successfully
[2020-12-07 10:02:32] [0001.960] I> gpio framework initialized
[2020-12-07 10:02:32] [0001.963] I> tegrabl_gpio_driver_register: register 'nvidia,tegra194-gpio' driver
[2020-12-07 10:02:32] [0001.970] I> tegrabl_gpio_driver_register: register 'nvidia,tegra194-gpio-aon' driver
[2020-12-07 10:02:32] [0001.976] I> tegrabl_tca9539_init: i2c bus: 1, slave addr: 0x46
[2020-12-07 10:02:32] [0001.984] E> fetch_driver_phandle_from_dt: failed to get node with compatible ti,tca9539
[2020-12-07 10:02:32] [0001.992] E> fetch_driver_phandle_from_dt: failed to get node with compatible nxp,tca9539
[2020-12-07 10:02:32] [0001.998] W> tegrabl_tca9539_init: failed to fetch phandle from dt
[2020-12-07 10:02:32] [0002.004] I> tegrabl_tca9539_init: i2c bus: 1, slave addr: 0x44
[2020-12-07 10:02:32] [0002.012] E> fetch_driver_phandle_from_dt: failed to get node with compatible ti,tca9539
[2020-12-07 10:02:32] [0002.020] E> fetch_driver_phandle_from_dt: failed to get node with compatible nxp,tca9539
[2020-12-07 10:02:32] [0002.026] W> tegrabl_tca9539_init: failed to fetch phandle from dt
[2020-12-07 10:02:32] [0002.034] I> fixed regulator driver initialized
[2020-12-07 10:02:32] [0002.046] I> register 'maxim' power off handle
[2020-12-07 10:02:32] [0002.047] I> virtual i2c enabled
[2020-12-07 10:02:32] [0002.047] I> registered 'maxim,max20024' pmic
[2020-12-07 10:02:32] [0002.048] I> tegrabl_gpio_driver_register: register 'max20024-gpio' driver
[2020-12-07 10:02:32] [0002.055] I> Boot-device: eMMC
[2020-12-07 10:02:32] [0002.057] I> Boot_device: SDMMC_BOOT instance: 3
[2020-12-07 10:02:32] [0002.066] I> sdmmc-3 params source = boot args
[2020-12-07 10:02:32] [0002.066] I> create_pm_ids: id: 2888-0004-400-L, len: 15
[2020-12-07 10:02:32] [0002.071] I> config: mem-type:00,power-config:00,misc-config:00,modem-config:00,touch-config:00,display-config:00,, len: 93
[2020-12-07 10:02:32] [0002.082] I> create_pm_ids: id: 2822-0000-700-K, len: 15
[2020-12-07 10:02:32] [0002.087] I> config: mem-type:00,power-config:00,misc-config:00,modem-config:00,touch-config:00,display-config:00,, len: 93
[2020-12-07 10:02:32] [0002.099] I> sdmmc bdev is already initialized
[2020-12-07 10:02:32] [0002.103] I> sdmmc-3 params source = boot args
[2020-12-07 10:02:32] [0002.161] I> Found 17 partitions in SDMMC_BOOT (instance 3)
[2020-12-07 10:02:32] [0002.173] I> Found 42 partitions in SDMMC_USER (instance 3)
[2020-12-07 10:02:32] [0002.185] I> enabling 'vdd-hdmi-5v0' regulator
[2020-12-07 10:02:32] [0002.192] I> regulator 'vdd-hdmi-5v0' already enabled
[2020-12-07 10:02:32] [0002.192] E> tegrabl_display_init_regulator: hdmi cable is not connected
[2020-12-07 10:02:32] [0002.193] E> tegrabl_display_get_pdata, failed to parse dtb settings
[2020-12-07 10:02:32] [0002.197] E> invalid display type
[2020-12-07 10:02:32] [0002.202] E> invalid display type
[2020-12-07 10:02:32] [0002.203] E> cannot find any other nvdisp nodes
[2020-12-07 10:02:32] [0002.203] E> no valid display unit config found in dtb
[2020-12-07 10:02:32] [0002.205] W> display init failed
[2020-12-07 10:02:32] [0002.205] I> Load in CBoot Boot Options partition and parse it
[2020-12-07 10:02:32] [0002.215] E> Error -9 when finding node with path /boot-configuration
[2020-12-07 10:02:32] [0002.215] E> tegrabl_cbo_parse_info: "boot-configuration" not found in CBO file.
[2020-12-07 10:02:32] [0002.222] I> Hit any key to stop autoboot:	4	3	2	1
[2020-12-07 10:02:34] [0004.230] initializing target
[2020-12-07 10:02:34] [0004.230] calling apps_init()
[2020-12-07 10:02:34] [0004.231] starting app kernel_boot_app
[2020-12-07 10:02:34] [0004.250] I> found decompressor handler: lz4-legacy
[2020-12-07 10:02:34] [0004.251] I> decompressing BMP blob ...
[2020-12-07 10:02:34] [0004.255] I> Kernel type = Normal
[2020-12-07 10:02:34] [0004.255] I> Loading kernel-bootctrl from partition
[2020-12-07 10:02:34] [0004.255] I> Loading partition kernel-bootctrl at 0xa42b0000 from device(0x1)
[2020-12-07 10:02:34] [0004.262] W> tegrabl_get_kernel_bootctrl: magic number(0x00000000) is invalid
[2020-12-07 10:02:34] [0004.262] W> tegrabl_get_kernel_bootctrl: use default dummy boot control data
[2020-12-07 10:02:34] [0004.263] I> ########## SD boot ##########
[2020-12-07 10:02:34] [0004.267] I> No sdcard
[2020-12-07 10:02:34] [0004.269] I> -0 params source = 
[2020-12-07 10:02:34] [0004.272] E> Blockdev open: exit error
[2020-12-07 10:02:34] [0004.275] E> SD boot failed, err: 724238353
[2020-12-07 10:02:34] [0004.279] I> ########## USB boot ##########
[2020-12-07 10:02:34] [0004.300] I> USB Firmware Version: 60.06 release
[2020-12-07 10:02:34] [0004.357] I> regulator of usb2-0 already enabled
[2020-12-07 10:02:34] [0004.365] I> regulator of usb2-1 already enabled
[2020-12-07 10:02:34] [0004.374] I> regulator of usb2-2 already enabled
[2020-12-07 10:02:34] [0004.386] I> enabling 'vdd-5v-sata' regulator
[2020-12-07 10:02:35] [0005.455] I> USB 2.0 port 2 new high-speed USB device detected
[2020-12-07 10:02:35] [0005.457] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa069c300, cmd_ring.dma = 0xa069c380
[2020-12-07 10:02:36] [0005.557] I> Start to enumerate device
[2020-12-07 10:02:36] [0005.560] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa069c300, cmd_ring.dma = 0xa069c380
[2020-12-07 10:02:36] [0005.564] I> This device is non-MSD, skip enumeration
[2020-12-07 10:02:36] [0005.564] E> Failed to enumerate USB device
[2020-12-07 10:02:36] [0005.564] E> failed to start xhci controller
[2020-12-07 10:02:36] [0005.565] E> Error in init of XUSB host driver, err: 7979000d
[2020-12-07 10:02:36] [0005.565] E> Failed to initialize device 5-0
[2020-12-07 10:02:36] [0005.568] E> USB boot failed, err: 2037973005
[2020-12-07 10:02:36] [0005.572] I> ########## Fixed storage boot ##########
[2020-12-07 10:02:36] [0005.577] I> Already published: 00010003
[2020-12-07 10:02:36] [0005.581] I> Look for boot partition
[2020-12-07 10:02:36] [0005.584] I> Fallback: assuming 0th partition is boot partition
[2020-12-07 10:02:36] [0005.590] I> Detect filesystem
[2020-12-07 10:02:36] [0005.617] I> Loading extlinux.conf ...
[2020-12-07 10:02:36] [0005.618] I> rootfs path: /sdmmc_user/boot/extlinux/extlinux.conf
[2020-12-07 10:02:36] [0005.660] I> L4T boot options
[2020-12-07 10:02:36] [0005.661] I> [1]: "primary kernel"
[2020-12-07 10:02:36] [0005.661] I> Enter choice: 
[2020-12-07 10:02:39] [0008.662] I> Continuing with default option: 1
[2020-12-07 10:02:39] [0008.662] I> Loading kernel sig file from rootfs ...
[2020-12-07 10:02:39] [0008.662] I> rootfs path: /sdmmc_user/boot/Image.sig
[2020-12-07 10:02:39] [0008.681] I> Loading kernel binary from rootfs ...
[2020-12-07 10:02:39] [0008.681] I> rootfs path: /sdmmc_user/boot/Image
[2020-12-07 10:02:39] [0008.912] I> Validate kernel ...
[2020-12-07 10:02:39] [0008.912] I> T19x: Authenticate kernel (bin_type: 37), max size 0x5000000
[2020-12-07 10:02:39] [0009.225] E> digest on binary did not match!!
[2020-12-07 10:02:39] [0009.225] C> OEM authentication of kernel payload failed!
[2020-12-07 10:02:39] [0009.226] W> Failed to validate kernel binary (err=1077936152, fail=0)
[2020-12-07 10:02:39] [0009.226] W> Security fuse not burned, ignore validation failure
[2020-12-07 10:02:39] [0009.230] I> No kernel-dtb binary path
[2020-12-07 10:02:39] [0009.236] I> A/B: bin_type (38) slot 1
[2020-12-07 10:02:39] [0009.237] I> Loading kernel-dtb_b from partition
[2020-12-07 10:02:39] [0009.237] I> Loading partition kernel-dtb_b at 0x91000000 from device(0x1)
[2020-12-07 10:02:39] [0009.250] I> Validate kernel-dtb ...
[2020-12-07 10:02:39] [0009.250] I> T19x: Authenticate kernel-dtb (bin_type: 38), max size 0x400000
[2020-12-07 10:02:39] [0009.253] I> Loading ramdisk from rootfs ...
[2020-12-07 10:02:39] [0009.255] I> rootfs path: /sdmmc_user/boot/initrd
[2020-12-07 10:02:39] [0009.307] I> Kernel hdr @0xa42b0000
[2020-12-07 10:02:39] [0009.307] I> Kernel dtb @0x90000000
[2020-12-07 10:02:39] [0009.307] I> decompressor handler not found
[2020-12-07 10:02:39] [0009.308] I> Copying kernel image (34332680 bytes) from 0xa42b0000 to 0x80080000 ... [0009.314] I> Done
[2020-12-07 10:02:39] [0009.314] I> Updated bpmp info to DTB
[2020-12-07 10:02:39] [0009.316] I> Ramdisk: Base: 0x92000000; Size: 0x54eb56
[2020-12-07 10:02:39] [0009.316] I> Updated initrd info to DTB
[2020-12-07 10:02:39] [0009.316] W> WARN: Fail to override "console=none" in commandline
[2020-12-07 10:02:39] [0009.320] E> tegrabl_linuxboot_add_disp_param, du 0 failed to get display params
[2020-12-07 10:02:39] [0009.327] E> tegrabl_linuxboot_add_disp_param, du 0 failed to get display params
[2020-12-07 10:02:39] [0009.334] E> tegrabl_linuxboot_add_disp_param, du 0 failed to get display params
[2020-12-07 10:02:39] [0009.342] I> Active slot suffix: _b
[2020-12-07 10:02:39] [0009.345] I> add_boot_slot_suffix: slot_suffix = _b
[2020-12-07 10:02:39] [0009.350] I> Linux Cmdline: console=ttyTCU0,115200 video=tegrafb no_console_suspend=1 earlycon=tegra_comb_uart,mmio32,0x0c168000 gpt usbcore.old_scheme_first=1 tegraid=19.1.2.0.0 maxcpus=8 boot.slot_suffix=_b boot.ratchetvalues=0.4.2 vpr_resize sdhci_tegra.en_boot_part_access=1 
[2020-12-07 10:02:39] [0009.374] I> Updated bootarg info to DTB
[2020-12-07 10:02:39] [0009.378] W> MAC addr invalid!
[2020-12-07 10:02:39] [0009.381] E> Failed to get WIFI MAC address
[2020-12-07 10:02:39] [0009.385] W> MAC addr invalid!
[2020-12-07 10:02:39] [0009.388] E> Failed to get Bluetooth MAC address
[2020-12-07 10:02:39] [0009.392] I> eeprom_get_mac_addr: MAC (type: 2): 48:b0:2d:2b:81:8b
[2020-12-07 10:02:39] [0009.399] W> "plugin-manager" doesn't exist, creating
[2020-12-07 10:02:39] [0009.404] I> Adding /chosen/plugin-manager/cvm
[2020-12-07 10:02:39] [0009.408] W> "chip-id" doesn't exist, creating
[2020-12-07 10:02:39] [0009.412] I> Adding /chosen/plugin-manager/chip-id
[2020-12-07 10:02:39] [0009.417] W> "configs" doesn't exist, creating
[2020-12-07 10:02:39] [0009.421] I> Adding /chosen/plugin-manager/configs
[2020-12-07 10:02:39] [0009.425] W> "ids" doesn't exist, creating
[2020-12-07 10:02:39] [0009.429] I> Adding /chosen/plugin-manager/ids
[2020-12-07 10:02:39] [0009.434] W> "odm-data" doesn't exist, creating
[2020-12-07 10:02:39] [0009.438] I> Adding /chosen/plugin-manager/odm-data
[2020-12-07 10:02:39] [0009.446] W> "memory" doesn't exist, creating
[2020-12-07 10:02:39] [0009.448] I> [0] START: 0x80000000, END: 0xac000000
[2020-12-07 10:02:39] [0009.452] I> [1] START: 0xac004000, END: 0xf09d0000
[2020-12-07 10:02:39] [0009.456] I> [2] START: 0xf09dc000, END: 0xf09e0000
[2020-12-07 10:02:39] [0009.461] I> dram_block larger than 80000000
[2020-12-07 10:02:39] [0009.465] I> [3] START: 0x100000000, END: 0x880000000
[2020-12-07 10:02:39] [0009.470] I> added [base:0x80000000, size:0x2c000000] to /memory
[2020-12-07 10:02:39] [0009.476] I> added [base:0xac200000, size:0x44600000] to /memory
[2020-12-07 10:02:39] [0009.482] I> added [base:0x100000000, size:0x780000000] to /memory
[2020-12-07 10:02:39] [0009.489] I> Updated memory info to DTB
[2020-12-07 10:02:39] [0009.492] E> add_disp_param: failed to get display params for du=0
[2020-12-07 10:02:39] [0009.498] W> "reset" doesn't exist, creating
[2020-12-07 10:02:39] [0009.502] I> NVG: Logical CPU: 0; MPIDR: 0x80000000
[2020-12-07 10:02:39] [0009.507] I> NVG: Logical CPU: 1; MPIDR: 0x80000001
[2020-12-07 10:02:39] [0009.511] I> NVG: Logical CPU: 2; MPIDR: 0x80000100
[2020-12-07 10:02:39] [0009.516] I> NVG: Logical CPU: 3; MPIDR: 0x80000101
[2020-12-07 10:02:39] [0009.521] I> NVG: Logical CPU: 4; MPIDR: 0x80000200
[2020-12-07 10:02:40] [0009.525] I> NVG: Logical CPU: 5; MPIDR: 0x80000201
[2020-12-07 10:02:40] [0009.530] I> NVG: Logical CPU: 6; MPIDR: 0x80000300
[2020-12-07 10:02:40] [0009.535] I> NVG: Logical CPU: 7; MPIDR: 0x80000301
[2020-12-07 10:02:40] [0009.541] W> "misc-data" doesn't exist, creating
[2020-12-07 10:02:40] [0009.544] I> Boot-device: eMMC
[2020-12-07 10:02:40] [0009.547] I> Add boot-sdmmc to plugin-manager/misc-data
[2020-12-07 10:02:40] [0009.552] I> Add storage-sdmmc to plugin-manager/misc-data
[2020-12-07 10:02:40] [0009.558] W> Unknown storage device
[2020-12-07 10:02:40] [0009.561] I> Add serial number:1423620008933 as DT property
[2020-12-07 10:02:40] [0009.567] I> Plugin-manager override starting
[2020-12-07 10:02:40] [0009.571] I> node /plugin-manager/fragement-tegra-wdt-en matches
[2020-12-07 10:02:40] [0009.578] I> node /plugin-manager/fragement-soft-wdt matches
[2020-12-07 10:02:40] [0009.586] I> node /plugin-manager/fragment-pcie-c5-rp matches
[2020-12-07 10:02:40] [0009.590] I> node /plugin-manager/fragment-tegra-ufs-lane10 matches
[2020-12-07 10:02:40] [0009.601] I> Disable plugin-manager status in FDT
[2020-12-07 10:02:40] [0009.602] I> Plugin-manager override finished successfully
[2020-12-07 10:02:40] [0009.604] I> tegrabl_load_kernel_and_dtb: Done
[2020-12-07 10:02:40] [0009.608] E> tegrabl_display_clear: display is not initialized
[2020-12-07 10:02:40] [0009.613] W> Boot logo display failed...
[2020-12-07 10:02:40] [0009.641] I> Kernel EP: 0x80080000, DTB: 0x90000000
[2020-12-07 10:02:40] [    0.000000] Booting Linux on physical CPU 0x0
[2020-12-07 10:02:40] [    0.000000] Linux version 4.9.140-tegra (buildbrain@mobile-u64-4294) (gcc version 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701] (Linaro GCC 7.3-2018.05) ) #1 SMP PREEMPT Tue Oct 27 21:02:46 PDT 2020
[2020-12-07 10:02:40] [    0.000000] Boot CPU: AArch64 Processor [4e0f0040]
[2020-12-07 10:02:40] [    0.000000] OF: fdt:memory scan node memory, reg size 48,
[2020-12-07 10:02:40] [    0.000000] OF: fdt: - 80000000 ,  2c000000
[2020-12-07 10:02:40] [    0.000000] OF: fdt: - ac200000 ,  44600000
[2020-12-07 10:02:40] [    0.000000] OF: fdt: - 100000000 ,  780000000
[2020-12-07 10:02:40] [    0.000000] earlycon: tegra_comb_uart0 at MMIO32 0x000000000c168000 (options '')
[2020-12-07 10:02:40] [    0.000000] bootconsole [tegra_comb_uart0] enabled
[2020-12-07 10:02:47] [    6.167538] cgroup: cgroup2: unknown option "nsdelegate"
[2020-12-07 10:02:48] [    7.721633] iwlwifi 0003:01:00.0: Direct firmware load for iwlwifi-8265-26.ucode failed with error -2
[2020-12-07 10:02:48] [    7.721824] iwlwifi 0003:01:00.0: Falling back to user helper
[2020-12-07 10:02:49] [    8.355952] iwlwifi 0003:01:00.0: Direct firmware load for iwlwifi-8265-25.ucode failed with error -2
[2020-12-07 10:02:49] [    8.356145] iwlwifi 0003:01:00.0: Falling back to user helper
[2020-12-07 10:02:49] [    8.441464] iwlwifi 0003:01:00.0: Direct firmware load for iwlwifi-8265-24.ucode failed with error -2
[2020-12-07 10:02:49] [    8.441667] iwlwifi 0003:01:00.0: Falling back to user helper
[2020-12-07 10:02:49] [    8.444049] iwlwifi 0003:01:00.0: Direct firmware load for iwlwifi-8265-23.ucode failed with error -2
[2020-12-07 10:02:49] [    8.444244] iwlwifi 0003:01:00.0: Falling back to user helper
[2020-12-07 10:02:49] [    8.640653] thermal thermal_zone8: failed to read out thermal zone (-5)
[2020-12-07 10:02:49] [    8.898860] using random self ethernet address
[2020-12-07 10:02:49] [    8.898981] using random host ethernet address
[2020-12-07 10:02:50] [    9.355004] using random self ethernet address
[2020-12-07 10:02:50] [    9.355121] using random host ethernet address
[2020-12-07 10:02:52] 
[2020-12-07 10:02:52] Ubuntu 18.04.5 LTS ubuntu-desktop ttyTCU0
[2020-12-07 10:02:52] 
[2020-12-07 10:02:52] ubuntu-desktop login: [   36.388890] bpmp: mrq 22 took 1476000 us       "
12

Do you ever run any application during this time to cause the board to crash?

According to the log, it took 3 days to have a kernel panic. Please note that I don’t see any gpu error this time.

13

We don’t know what is the cause of this panic that is why we asked for help.
Definitely we are running applications, a deep learning algorithm to process some images with python3.
The reboot happens not necessarily every hour, it works for a couple of hours and sometimes days, and then rebooted.
Sometimes it gets rebooted several times after the first reboot.

14

Hi,

I don’t have any answer for you now. Information is not sufficient.

Please

  1. Try your application on multiple xavier devices and see if every of them can hit this issue.

  2. Try your applications on xavier development kits and see if you can still reproduce this issue.

  3. If you can reproduce your issue on devkit, please provide the application for us to check. We will try to reproduce it too.

  4. Please identify if each reboot gives out the same stack dump of kernel panic.

15

I confirm that we see the same issue with multiple Xavier device.
I also confirm all devices are Xavier development kit.
We have updated the kernel and just added a QMI_WWAN driver to the kernel for LTE that I’m attaching the source code os those updated drivers for your reference.
To get the same reboot debug message I think I need to wait a couple of more days hopefully we see it soon.

option.c (118.9 KB) qmi_wwan.c (44.1 KB) usb_wwan.c (18.1 KB)

16

Hi,

I think you can just remove these drivers and see if you can still reproduce issue.

Is this application using LTE?

17

We have LTE connected but we are not transferring data, LTE is just used for remote connections,

18

Ok, so can we remove them first?

19

I will re-flash the system without adding those modules and see what happens.

1 Like
20

I saw the same issue, but this time the sensor was rebooted a couple of time almost every two minutes, all same kernel panic and CPU: rcu_sched detected stalls on CPUs/tasks

Can you check this with Nvidia team and see what is this for?
I attached both logs:
debug_log.txt (255.6 KB) debug_log.txt (83.7 KB)

21

Hi,
I would like to know

  1. What is the software release you are using? Can you just use “pure” jetpack release and see if this issue still happens? “Pure” means you just download the jetpack from sdkmanager again and do not install anything.

  2. When this error happens in latest log, does your application start? or your just put it idle and doing nothing?

  3. Does every xavier device you have suffer the same error ? How many jetson xavier devkit do you have?

  4. Could you remove the “quiet” in /boot/extlinux/extlinux.conf ? It will enable the full kernel log as dmesg in your uart console.