TX2 unexpected self-reboots

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Yes, there are many posts about this issue on the forum, but they are all closed without solving the problem, so I have to reopen a new topic.

My tx2 (devkit) worked fine over months before, but recently the unexpected self-rebooting problem shows almost 2 times a week. It connects 2 realsense D435 through a usb-hub.

nv_tegra_release: # R32 (release), REVISION: 4.3, GCID: 21589087, BOARD: t186ref, EABI: aarch64, DATE: Fri Jun 26 04:34:27 UTC 2020

The ouputs of below:

root@jetson:~ cat /proc/device-tree/chosen/reset/pmc-reset-reason/reset-level
root@jetson:~ cat /proc/device-tree/chosen/reset/pmc-reset-reason/reset-source
root@jetson:~ cat /proc/device-tree/chosen/reset/pmic-reset-reason/reason
root@jetson:~ cat /proc/device-tree/chosen/reset/pmic-reset-reason/register-value

is:
0; SYS_RESET_N; NIL_OR_MORE_THAN_1_BIT; 0x50

And my question is: What does the register-value 0x50, 0x44, 0x40 and so on actually mean? How can we check it out precisely weather the power is low or overvoltage lockout or undervoltage lockout?

Thank you!

2

You can start from monitoring the uart serial console with below method.

This is the quick way to see what was printed when such sudden reboot happens. Directly checking the reset reason may not directly find out the case.

For example, if system crashed due to kernel panic, the cause may be from lots of driver. But all of them may lead to watchdog to reset the board. Under such situation, the pmc reset reasons are all the same. However, since you mentioned D435, maybe this is from specific usb driver, or maybe not even from usb driver. We need to check the stack dump info directly.

3

Thanks for your reply.

However, due to the requirements of my work scenario, tx2 needs to communicate with two STM32 boards using serial ports J17 and J21, so I have changed the serial console at J21 to a normal serial port according to https://connecttech.com/resource-center/kdb347-reassigning-disable-serial-debug-connect-tech-tx1-carriers/. Will this modification affect the reboot of tx2?

If so, then what‘s the right way for tx2 to open two serial ports at the same time?

4

Is this still a NV devkit? or your custom carrier board?

Are you able to use other serial port for your work and use our original setting to check this issue?

5

Yes, this is the original devkit and original board. Just connect 2 depth cameras via usb3.0 and 2 serial ports via J17 and J21 to the board.

This is the problem: the reboot issue cannot be reproduced stably. According to my experience, there has been no unexpected automatic reboot problem when using one camera and one serial port (J17). Is it because the power requirements of the two cameras are too large and the onboard voltage is too low?

And I also want to know, are there other more suitable pins on the tx2 board that can be used for the second serial port communication but J21?

6

So you are suspecting this issue is due to the uart communication on J21 pin 8,9,10?

7

You can also check below node after the error happens and see if it collects the error log.

/sys/fs/pstore/console-ramoops-0

8

Yes, I suspect it is one of the possible reasons. And I will try console-ramoops-0 next the issue happens.

9

I caught the error again and after tx2’s rebooting the terminal outputs like below:

cat /sys/fs/pstore/console-ramoops-0 
[    3.522682] cgroup: cgroup2: unknown option "nsdelegate"
[    4.835823] using random self ethernet address
[    4.835869] using random host ethernet address
[    5.428409] using random self ethernet address
[    5.428454] using random host ethernet address
[    7.589860] uvcvideo 2-1.1:1.0: Entity type for entity Intel(R) RealSense(TM) Depth Ca was not initialized!
[    7.589874] uvcvideo 2-1.1:1.0: Entity type for entity Processing 2 was not initialized!
[    7.589882] uvcvideo 2-1.1:1.0: Entity type for entity Camera 1 was not initialized!
[    7.600825] uvcvideo 2-1.1:1.3: Entity type for entity Processing 7 was not initialized!
[    7.600840] uvcvideo 2-1.1:1.3: Entity type for entity Extension 8 was not initialized!
[    7.600848] uvcvideo 2-1.1:1.3: Entity type for entity Camera 6 was not initialized!
[    7.614468] uvcvideo 2-1.2:1.0: Entity type for entity Intel(R) RealSense(TM) Depth Ca was not initialized!
[    7.614483] uvcvideo 2-1.2:1.0: Entity type for entity Processing 2 was not initialized!
[    7.614490] uvcvideo 2-1.2:1.0: Entity type for entity Camera 1 was not initialized!
[    7.636370] uvcvideo 2-1.2:1.3: Entity type for entity Processing 7 was not initialized!
[    7.636386] uvcvideo 2-1.2:1.3: Entity type for entity Extension 8 was not initialized!
[    7.636393] uvcvideo 2-1.2:1.3: Entity type for entity Camera 6 was not initialized!
[    8.165278] vdd-1v8: voltage operation not allowed
[    8.165299] sdhci-tegra 3440000.sdhci: could not set regulator OCR (-1)
[    8.185046] vdd-1v8: voltage operation not allowed
[    8.185065] sdhci-tegra 3440000.sdhci: could not set regulator OCR (-1)
[    8.205863] vdd-1v8: voltage operation not allowed
[    8.205880] sdhci-tegra 3440000.sdhci: could not set regulator OCR (-1)
[    8.208263] vdd-1v8: voltage operation not allowed
[    8.208276] sdhci-tegra 3440000.sdhci: could not set regulator OCR (-1)
[    8.307958] vdd-1v8: voltage operation not allowed
[    8.307979] sdhci-tegra 3440000.sdhci: could not set regulator OCR (-1)
[    8.309511] vdd-1v8: voltage operation not allowed
[    8.309527] sdhci-tegra 3440000.sdhci: could not set regulator OCR (-1)
[    8.311033] vdd-1v8: voltage operation not allowed
[    8.311044] sdhci-tegra 3440000.sdhci: could not set regulator OCR (-1)
[   23.365315] uvcvideo 2-1.1:1.0: Entity type for entity Intel(R) RealSense(TM) Depth Ca was not initialized!
[   23.365330] uvcvideo 2-1.1:1.0: Entity type for entity Processing 2 was not initialized!
[   23.365338] uvcvideo 2-1.1:1.0: Entity type for entity Camera 1 was not initialized!
[   23.370091] uvcvideo 2-1.1:1.3: Entity type for entity Processing 7 was not initialized!
[   23.370106] uvcvideo 2-1.1:1.3: Entity type for entity Extension 8 was not initialized!
[   23.370113] uvcvideo 2-1.1:1.3: Entity type for entity Camera 6 was not initialized!
[   23.626264] uvcvideo 2-1.2:1.0: Entity type for entity Intel(R) RealSense(TM) Depth Ca was not initialized!
[   23.626277] uvcvideo 2-1.2:1.0: Entity type for entity Processing 2 was not initialized!
[   23.626284] uvcvideo 2-1.2:1.0: Entity type for entity Camera 1 was not initialized!
[   23.630914] uvcvideo 2-1.2:1.3: Entity type for entity Processing 7 was not initialized!
[   23.630926] uvcvideo 2-1.2:1.3: Entity type for entity Extension 8 was not initialized!
[   23.630932] uvcvideo 2-1.2:1.3: Entity type for entity Camera 6 was not initialized!
[  125.955857] uvcvideo 2-1.1:1.0: Entity type for entity Intel(R) RealSense(TM) Depth Ca was not initialized!
[  125.955869] uvcvideo 2-1.1:1.0: Entity type for entity Processing 2 was not initialized!
[  125.955877] uvcvideo 2-1.1:1.0: Entity type for entity Camera 1 was not initialized!
[  125.964774] uvcvideo 2-1.1:1.3: Entity type for entity Processing 7 was not initialized!
[  125.964787] uvcvideo 2-1.1:1.3: Entity type for entity Extension 8 was not initialized!
[  125.964794] uvcvideo 2-1.1:1.3: Entity type for entity Camera 6 was not initialized!
[  126.233889] uvcvideo 2-1.2:1.0: Entity type for entity Intel(R) RealSense(TM) Depth Ca was not initialized!
[  126.233903] uvcvideo 2-1.2:1.0: Entity type for entity Processing 2 was not initialized!
[  126.233911] uvcvideo 2-1.2:1.0: Entity type for entity Camera 1 was not initialized!
[  126.238537] uvcvideo 2-1.2:1.3: Entity type for entity Processing 7 was not initialized!
[  126.238552] uvcvideo 2-1.2:1.3: Entity type for entity Extension 8 was not initialized!
[  126.238559] uvcvideo 2-1.2:1.3: Entity type for entity Camera 6 was not initialized!
[  184.249955] uvcvideo 2-1.1:1.0: Entity type for entity Intel(R) RealSense(TM) Depth Ca was not initialized!
[  184.249969] uvcvideo 2-1.1:1.0: Entity type for entity Processing 2 was not initialized!
[  184.249977] uvcvideo 2-1.1:1.0: Entity type for entity Camera 1 was not initialized!
[  184.254669] uvcvideo 2-1.1:1.3: Entity type for entity Processing 7 was not initialized!
[  184.254683] uvcvideo 2-1.1:1.3: Entity type for entity Extension 8 was not initialized!
[  184.254690] uvcvideo 2-1.1:1.3: Entity type for entity Camera 6 was not initialized!
[  184.509396] uvcvideo 2-1.2:1.0: Entity type for entity Intel(R) RealSense(TM) Depth Ca was not initialized!
[  184.509408] uvcvideo 2-1.2:1.0: Entity type for entity Processing 2 was not initialized!
[  184.509416] uvcvideo 2-1.2:1.0: Entity type for entity Camera 1 was not initialized!
[  184.513996] uvcvideo 2-1.2:1.3: Entity type for entity Processing 7 was not initialized!
[  184.514009] uvcvideo 2-1.2:1.3: Entity type for entity Extension 8 was not initialized!
[  184.514016] uvcvideo 2-1.2:1.3: Entity type for entity Camera 6 was not initialized!
[  464.124246] uvcvideo 2-1.1:1.0: Entity type for entity Intel(R) RealSense(TM) Depth Ca was not initialized!
[  464.124260] uvcvideo 2-1.1:1.0: Entity type for entity Processing 2 was not initialized!
[  464.124268] uvcvideo 2-1.1:1.0: Entity type for entity Camera 1 was not initialized!
[  464.130103] uvcvideo 2-1.1:1.3: Entity type for entity Processing 7 was not initialized!
[  464.130116] uvcvideo 2-1.1:1.3: Entity type for entity Extension 8 was not initialized!
[  464.130124] uvcvideo 2-1.1:1.3: Entity type for entity Camera 6 was not initialized!
[  469.075533] tegra-xusb 3530000.xhci: WARN Event TRB for slot 5 ep 14 with no TDs queued?
[  519.013234] usb 2-1.2: Disable of device-initiated U1 failed.
[  519.014545] usb 2-1.2: Disable of device-initiated U2 failed.
[  606.685151] uvcvideo 2-1.1:1.0: Entity type for entity Intel(R) RealSense(TM) Depth Ca was not initialized!
[  606.685165] uvcvideo 2-1.1:1.0: Entity type for entity Processing 2 was not initialized!
[  606.685173] uvcvideo 2-1.1:1.0: Entity type for entity Camera 1 was not initialized!
[  606.692619] uvcvideo 2-1.1:1.3: Entity type for entity Processing 7 was not initialized!
[  606.692632] uvcvideo 2-1.1:1.3: Entity type for entity Extension 8 was not initialized!
[  606.692718] uvcvideo 2-1.1:1.3: Entity type for entity Camera 6 was not initialized!
[  608.987765] tegra-xusb 3530000.xhci: WARN Event TRB for slot 5 ep 14 with no TDs queued?
[  663.971636] usb 2-1.2: Disable of device-initiated U1 failed.
[  663.972964] usb 2-1.2: Disable of device-initiated U2 failed.
[  710.978305] tegra-xusb 3530000.xhci: controller firmware hang
[  715.049766] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[  732.364467] Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0
[  732.364483] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 4.9.140-tegra #1
[  732.364489] Hardware name: quill (DT)
[  732.364494] Call trace:
[  732.364506] [<ffffff800808bdb8>] dump_backtrace+0x0/0x198
[  732.364513] [<ffffff800808c37c>] show_stack+0x24/0x30
[  732.364521] [<ffffff800845c7a0>] dump_stack+0x98/0xc0
[  732.364529] [<ffffff80081c1438>] panic+0x11c/0x298
[  732.364537] [<ffffff80081823c4>] watchdog_check_hardlockup_other_cpu+0x11c/0x120
[  732.364543] [<ffffff8008181538>] watchdog_timer_fn+0x98/0x2c0
[  732.364551] [<ffffff80081399e0>] __hrtimer_run_queues+0xd8/0x360
[  732.364556] [<ffffff800813a330>] hrtimer_interrupt+0xa8/0x1e0
[  732.364563] [<ffffff8008bff764>] tegra186_timer_isr+0x34/0x48
[  732.364570] [<ffffff8008122508>] __handle_irq_event_percpu+0x68/0x288
[  732.364575] [<ffffff8008122750>] handle_irq_event_percpu+0x28/0x60
[  732.364580] [<ffffff80081227d8>] handle_irq_event+0x50/0x80
[  732.364586] [<ffffff80081265a0>] handle_fasteoi_irq+0xc8/0x1b8
[  732.364591] [<ffffff80081214f4>] generic_handle_irq+0x34/0x50
[  732.364596] [<ffffff8008121bd8>] __handle_domain_irq+0x68/0xc0
[  732.364601] [<ffffff8008080d44>] gic_handle_irq+0x5c/0xb0
[  732.364606] [<ffffff8008082c28>] el1_irq+0xe8/0x194
[  732.364613] [<ffffff8008ba4ae0>] cpuidle_enter_state+0xb8/0x380
[  732.364618] [<ffffff8008ba4e1c>] cpuidle_enter+0x34/0x48
[  732.364624] [<ffffff8008111cc4>] call_cpuidle+0x44/0x70
[  732.364629] [<ffffff8008112040>] cpu_startup_entry+0x1b0/0x200
[  732.364636] [<ffffff8008092ce8>] secondary_start_kernel+0x190/0x1f8
[  732.364641] [<00000000811621a4>] 0x811621a4
[  732.364649] SMP: stopping secondary CPUs
[  733.415796] SMP: failed to stop secondary CPUs 0,5
[  733.415803] Kernel Offset: disabled
[  733.415808] Memory Limit: none
[  733.415814] trusty-log panic notifier - trusty version Built: 21:16:26 Jun 25 2020 [  733.422066] Rebooting in 5 seconds..
[  734.483087] tegra-xusb 3530000.xhci: Stopped the command ring failed, maybe the host is dead
[  734.519816] tegra-xusb 3530000.xhci: Host not halted after 16000 microseconds.
[  734.519823] tegra-xusb 3530000.xhci: Abort command ring failed

Based on the above info, what could be the cause of the issue? Thank you！

10

It looks like usb firmware hang. So nothing is related to the uart serial.

[ 710.978305] tegra-xusb 3530000.xhci: controller firmware hang

Please try more times and see if it is always such reboot reason.

What is your environment and device to reproduce this issue? Custom board? Devkit? which software release? Which camera? Any app sample code to share?

11

Another reboot occurs, and tegra-xusb 3530000.xhci: controller firmware hang appears again.

  • Devkit
  • nv_tegra_release: # R32 (release), REVISION: 4.3, GCID: 21589087, BOARD: t186ref, EABI: aarch64, DATE: Fri Jun 26 04:34:27 UTC 2020
  • realsense D435
  • and the 2 D435 is connect to tx2 by a usb-hub Bus 001 Device 002: ID 05e3:0610 Genesys Logic, Inc. 4-port hub

How should I do to test the USB firmware hang issue, please?

12

Could you also upgrade to rel-32.6.1 and see if issue is still there? rel-32.4.3 is kind of old, released for like 1.5 years ago.

13

Does that mean I have to re-flash the tx2 with the newest version of JetPack using SDK Manager?

14

Yes, we always debug on the latest release.

15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.