Nano shuts down in headless mode

I am having an interesting problem where a Nano will run for days when connected to a keyboard, monitor and mouse, but will shutdown at what appears to be random times when in headless mode. It will even shutdown with an active jupyter lab session. It does not seem to be an out memory issue. I have gone in through Gnome desktop and disabled the various sleep settings. The Nano has both a hardwired ethernet and WiFi connection. I created the sd image manually using the Nvidia instructions. I am using the 4A power supply from Adafruit :) Not a thermal issues - tried with fanon and then off… All of the jupyter lab applications run just fine and I have even developed some of my own code to control a Tello drone…

Any help would be greatly appreciated!

I am not a Linux expert but have looked through the syslog and kern.log files.

Here is the excerpt from the kern.log right when the system shuts down (mac address removed):

May 24 04:51:58 jetpack-dev-desktop kernel: [21053.870835] wlan0: denied authentication (status 37)
May 24 04:52:09 jetpack-dev-desktop kernel: [21064.875794] wlan0: authenticate with
May 24 04:52:09 jetpack-dev-desktop kernel: [21064.884962] wlan0: send auth to (try 1/3)
May 24 04:52:09 jetpack-dev-desktop kernel: [21064.890234] wlan0: denied authentication (status 37)
May 24 04:52:22 jetpack-dev-desktop kernel: [21077.811001] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
May 24 04:52:23 jetpack-dev-desktop kernel: [21078.850868] wlan0: authenticate with
May 24 04:52:23 jetpack-dev-desktop kernel: [21078.859086] wlan0: send auth to (try 1/3)
May 24 04:52:23 jetpack-dev-desktop kernel: [21078.861888] wlan0: denied authentication (status 37)
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@May 24 04:42:31 jetpack-dev-desktop kernel: [ 0.000000] Booting Linux on physical CPU 0x0
May 24 04:42:31 jetpack-dev-desktop kernel: [ 0.000000] Linux version 4.9.140-tegra (buildbrain@mobile-u64-2713) (gcc version 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701] (Linaro GCC 7.3-2018.05) ) #1 SMP PREEMPT Mon Dec 9 22:47:42 PST 2019
May 24 04:42:31 jetpack-dev-desktop kernel: [ 0.000000] Boot CPU: AArch64 Processor [411fd071]
May 24 04:42:31 jetpack-dev-desktop kernel: [ 0.000000] OF: fdt:memory scan node memory@80000000, reg size 48,
May 24 04:42:31 jetpack-dev-desktop kernel: [ 0.000000] OF: fdt: - 80000000 , 7ee00000
May 24 04:42:31 jetpack-dev-desktop kernel: [ 0.000000] OF: fdt: - 100000000 , 7f200000
May 24 04:42:31 jetpack-dev-desktop kernel: [ 0.000000] OF: fdt:Reserved memory: failed to reserve memory for node ‘fb0_carveout’: base 0x0000000000000000, size 0 MiB
May 24 04:42:31 jetpack-dev-desktop kernel: [ 0.000000] OF: fdt:Reserved memory: failed to reserve memory for node ‘fb0_carveout’: base 0x0000000000000000, size 0 MiB
May 24 04:42:31 jetpack-dev-desktop kernel: [ 0.000000] OF: fdt:Reserved memory: failed to reserve memory for node ‘fb1_carveout’: base 0x0000000000000000, size 0 MiB
May 24 04:42:31 jetpack-dev-desktop kernel: [ 0.000000] OF: fdt:Reserved memory: failed to reserve memory for node ‘fb1_carveout’: base 0x0000000000000000, size 0 MiB
May 24 04:42:31 jetpack-dev-desktop kernel: [ 0.000000] OF: reserved mem: initialized node vpr-carveout, compatible id nvidia,vpr-carveout
May 24 04:42:31 jetpack-dev-desktop kernel: [ 0.000000] OF: reserved mem: initialized node iram-carveout, compatible id nvidia,iram-carveout
May 24 04:42:31 jetpack-dev-desktop kernel: [ 0.000000] OF: reserved mem: initialized node ramoops_carveout, compatible id nvidia,ramoops
May 24 04:42:31 jetpack-dev-desktop kernel: [ 0.000000] cma: Reserved 64 MiB at 0x00000000fac00000
May 24 04:42:31 jetpack-dev-desktop kernel: [ 0.000000] On node 0 totalpages: 1039872
May 24 04:42:31 jetpack-dev-desktop kernel: [ 0.000000] DMA zone: 8192 pages used for memmap
May 24 04:42:31 jetpack-dev-desktop kernel: [ 0.000000] DMA zone: 0 pages reserved
May 24 04:42:31 jetpack-dev-desktop kernel: [ 0.000000] DMA zone: 519168 pages, LIFO batch:31
May 24 04:42:31 jetpack-dev-desktop kernel: [ 0.000000] Normal zone: 8136 pages used for memmap
May 24 04:42:31 jetpack-dev-desktop kernel: [ 0.000000] Normal zone: 520704 pages, LIFO batch:31

Here is the syslog file at the the same time:

May 24 04:52:49 jetpack-dev-desktop NetworkManager[4153]: [1590310369.2254] device (wlan0): supplicant interface state: authenticating -> disconnected
May 24 04:52:49 jetpack-dev-desktop nm-applet[6565]: gtk_widget_destroy: assertion ‘GTK_IS_WIDGET (widget)’ failed
May 24 04:52:49 jetpack-dev-desktop nm-applet[6565]: gtk_widget_destroy: assertion ‘GTK_IS_WIDGET (widget)’ failed
May 24 04:52:49 jetpack-dev-desktop nm-applet[6565]: Can’t set a parent on widget which has a parent
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@May 24 04:42:31 jetpack-dev-desktop systemd-modules-load[1983]: Inserted module ‘bluedroid_pm’
May 24 04:42:31 jetpack-dev-desktop systemd-modules-load[1983]: Module ‘nvhost_vi’ is builtin
May 24 04:42:31 jetpack-dev-desktop systemd-modules-load[1983]: Inserted module ‘nvgpu’
May 24 04:42:31 jetpack-dev-desktop systemd[1]: Starting Flush Journal to Persistent Storage…
May 24 04:42:31 jetpack-dev-desktop systemd-udevd[2657]: Network interface NamePolicy= disabled on kernel command line, ignoring.
May 24 04:42:31 jetpack-dev-desktop systemd[1]: Started Load Kernel Modules.
May 24 04:42:31 jetpack-dev-desktop systemd[1]: Starting Apply Kernel Variables…
May 24 04:42:31 jetpack-dev-desktop systemd[1]: Mounting Kernel Configuration File System…
May 24 04:42:31 jetpack-dev-desktop systemd[1]: Started Set the console keyboard layout.
May 24 04:42:31 jetpack-dev-desktop systemd[1]: Reached target Local File Systems (Pre).
May 24 04:42:31 jetpack-dev-desktop systemd[1]: Reached target Local File Systems.
May 24 04:42:31 jetpack-dev-desktop systemd[1]: Starting Clean up any mess left by 0dns-up…
May 24 04:42:31 jetpack-dev-desktop systemd-sysctl[2691]: Couldn’t write ‘1’ to ‘net/ipv4/tcp_syncookies’, ignoring: No such file or directory
May 24 04:42:31 jetpack-dev-desktop systemd-sysctl[2691]: Couldn’t write ‘1’ to ‘kernel/yama/ptrace_scope’, ignoring: No such file or directory
May 24 04:42:31 jetpack-dev-desktop systemd-sysctl[2691]: Couldn’t write ‘fq_codel’ to ‘net/core/default_qdisc’, ignoring: No such file or directory

Ubuntu 18.04.3
Gnome 3.28.4

Hi,

It seems the syslog just goes into reboot without error. But the timestamp looks out of order. Are you sure this log is correct?

Could you enable serial log and put system in idle and see if it show any log during shutdown?

The you for the reply!

I noticed the timestamp issues also. That has occurred every time there is a random shutdown. Strange. If by serial log, you mean attach a serial able and monitor the system that way, I will give it a go :) How do I put the system in idle?

Hi,

Yeah, I meant enable the serial log so that your host is able to monitor the status.

How do I put the system in idle?

Have you ever observed when and how this issue would happen? I meant if you just turn on the device, set the status to never lock down/suspend, will you see this issue?

HI WayneWWW. I finally had a chance to connect a terminal to the console port. Nothing very interesting. Here is what I observed:

At power on, there is what appears to be a normal boot sequence on the serial console. There were some errors due to no HDMI monitor connected. (Exerpts below)
Followed by a normal kernel startup.

The syslog and kern.log also appear to have a normal boot other than some gnome warnings / error.

With on user logged in (system idle), it will run for several hours and then just power off. There is no additional information in the console. The power button will NOT boot the nano and I have to power cycle (unplug / plug) the nano to get it to boot.

The console log appears to show it rebooting just before it shuts down…

Console
[0007.770] Carveout took -1539323 us
[0007.773] CPU initialization took 2036201 us
[0007.777] Total time taken by TegraBoot 7862410 us

[0007.782] Starting CPU & Halting co-processor

64NOTICE: BL31: v1.3(release):23b153a63
NOTICE: BL31: Built : 22:39:57, Dec 9 2019
ERROR: Error initializing runtime service trusty_fast
[0007.904] RamCode = 0
[0007.909] LPDDR4 Training: Read DT: Number of tables = 2
[0007.914] EMC Training (SRC-freq: 204000; DST-freq: 1600000)
[0007.927] EMC Training Successful
[0007.930] 408000 not found in DVFS table
[0007.936] RamCode = 0

[0008.208] platform_init_timer: DONE
[0008.211] Timer(tick) Init done
[0008.215] osc freq = 38400 khz
[0008.219]
[0008.220] Welcome to L4T Cboot
[0008.223]
[0008.224] Cboot Version: 00.00.2018.01-t210-40c3ff9c
[0008.229] calling constructors
[0008.232] initializing heap

[0009.134] Proceeding to Cold Boot
[0009.137] starting app android_boot_app
[0009.141] Device state: unlocked
[0009.144] display console init
[0009.153] could not find regulator
[0009.176] hdmi cable not connected
[0009.179] is_hdmi_needed: HDMI not connected, returning false
[0009.185] hdmi is not connected
[0009.188] sor0 is not supportDT entry for leds-pwm not found
e[0009.197] d
[0009.198] display_console_init: no valid display out_type
[0009.206] subnode volume_up is not found !
[0009.210] subnode back is not found !
[0009.214] subnode volume_down is not found !
[0009.218] subnode menu is not found !
[0009.222] Gpio keyboard init success
[0009.265] found decompressor handler: lz4-legacy
[0009.280] decompressing blob (type 1)…
[0009.313] display_resolution: No display init
[0009.318] Failed to retrieve display resolution
[0009.322] Could not load/initialize BMP blob…ignoring
[0009.373] decompressor handler not found
[0009.377] load_firmware_blob: Firmware blob loaded, entries=2

Hi Thomas,

Is there any log prior to this one? Below log is the one that system already reboots. Also, just to make sure it is not due to abnormal thermal issue, could you run sudo tegrastats on another ssh console to monitor the thermal condition?

[0007.770] Carveout took -1539323 us
[0007.773] CPU initialization took 2036201 us
[0007.777] Total time taken by TegraBoot 7862410 us

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one. Thanks