Jetson Nano does not boot (stuck on boot logs)

Hello everyone,

I was working on a project with my Nvidia Jetson Nano. I was accessing remotely with NoMachine, when suddenly the connection failed. I tried to reboot it, but the connection could not be re-established. So I connected the Jetson Nano directly to a screen, and after a reboot I saw this screen.

I tried connecting to the Jetson Nano with ssh but I got the following:
ssh -p 4000 root@192.168.2.2
kex_exchange_identification: Connection closed by remote host

I also did the following things: disconnected all the sensors/devices that were connected to the GPiO pins, removed the wifi card.
I am using a 5V 4A power supply. There is a fan connected to the Jetson Nano and mounted on top of the heat sink.

Does anyone know what is wrong and how I can get my Jetson Nano back working again?

Thanks in advance!

Anton

hello anton.haes,

are you working with default JetPack release image, may I know what’s the modification you’d done?
thanks

I am working with the default JetPack release image, downloaded here: https://developer.nvidia.com/embedded/learn/get-started-jetson-nano-devkit#write
I installed it approximately half a year ago, and is probably not up to date. I flashed a new SD-card with the newest version, and when booting I get this screen:


Is this problem hardware related? If so, can it be fixed?

Thanks in advance!

hello anton.haes,

since it also contain SPI parts you should update all together, assume you’re only update image to the SD-card and plug it into your Nano devkit caused the failure.
could you please put Nano device into forced-recovery mode, and installing it again with SDKmanager to the latest JetPack release.
thanks

I tried booting into forced-recovery mode (by shorting the appropriate pins), but apparently it does not boot (the fan is spinning, the green light next to the micro-USB port is on, but I have no display output).

What do you mean by ‘installing it again with SDKmanager to the latest JetPack release’.

Thanks in advance.

Forced recovery is not normal boot, and by itself does nothing more than set the Nano up for flashing. JetPack/SDK Manager is mentioned because this is for flashing the Nano. I saw in your previous logs lots of filesystem errors, and so flashing with a new filesystem is likely the best “next test” (this will erase the SD card though and overwrite with a new install…you could save a copy of the image on your host PC with dd if you do not want to lose this, but there is no guarantee that the filesystem is sufficiently intact to recovery it).

FYI, there were a lot of checksum errors of the filesystem which apparently was too severe for fsck to repair. A dd copy of this image on the host might allow manual repair, but there would be pieces of the system missing.

Where can I find instructions to do that?

Simply flashing again is what I would advise. If you want to try and recover something, then it depends on whether this is an SD card version or an eMMC version. Is this on SD card? If so, do you have an SD card reader you can use with your host PC? If you have a reader, and you monitor “dmesg --follow” during insert of the SD card, what do you see?

I have an SD-card version.
I get this with the SD-card having a new install:

[ 636.043793] usb 1-5: new high-speed USB device number 9 using xhci_hcd
[ 636.197944] usb 1-5: New USB device found, idVendor=05e3, idProduct=0751
[ 636.197952] usb 1-5: New USB device strings: Mfr=3, Product=4, SerialNumber=0
[ 636.197957] usb 1-5: Product: USB Storage
[ 636.197961] usb 1-5: Manufacturer: USB Storage
[ 636.199241] usb-storage 1-5:1.0: USB Mass Storage device detected
[ 636.199407] scsi host3: usb-storage 1-5:1.0
[ 637.227203] scsi 3:0:0:0: Direct-Access Generic STORAGE DEVICE 1404 PQ: 0 ANSI: 6
[ 637.228321] sd 3:0:0:0: Attached scsi generic sg0 type 0
[ 637.411892] sd 3:0:0:0: [sda] 31116288 512-byte logical blocks: (15.9 GB/14.8 GiB)
[ 637.413064] sd 3:0:0:0: [sda] Write Protect is off
[ 637.413068] sd 3:0:0:0: [sda] Mode Sense: 21 00 00 00
[ 637.414135] sd 3:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn’t support DPO or FUA
[ 637.435678] Alternate GPT is invalid, using primary GPT.
[ 637.435707] sda: sda1 sda2 sda3 sda4 sda5 sda6 sda7 sda8 sda9 sda10 sda11 sda12 sda13 sda14
[ 637.440537] sd 3:0:0:0: [sda] Attached SCSI removable disk
[ 638.038376] EXT4-fs (sda1): warning: mounting fs with errors, running e2fsck is recommended
[ 638.040248] EXT4-fs (sda1): recovery complete
[ 638.041808] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[ 638.158496] EXT4-fs error (device sda1): ext4_iget:4784: inode #521217: comm pool: checksum invalid
[ 638.170563] EXT4-fs error (device sda1): ext4_iget:4784: inode #651521: comm pool: checksum invalid
[ 638.252818] EXT4-fs error (device sda1): ext4_iget:4784: inode #657198: comm pool: checksum invalid
[ 638.261422] EXT4-fs error (device sda1): ext4_iget:4784: inode #657204: comm pool: checksum invalid

And this with the SD-card I was using when the Jetson stopped working:

[ 688.292375] usb 1-5: new high-speed USB device number 10 using xhci_hcd
[ 688.449360] usb 1-5: New USB device found, idVendor=05e3, idProduct=0751
[ 688.449366] usb 1-5: New USB device strings: Mfr=3, Product=4, SerialNumber=0
[ 688.449370] usb 1-5: Product: USB Storage
[ 688.449374] usb 1-5: Manufacturer: USB Storage
[ 688.450758] usb-storage 1-5:1.0: USB Mass Storage device detected
[ 688.451029] scsi host3: usb-storage 1-5:1.0
[ 689.482998] scsi 3:0:0:0: Direct-Access Generic STORAGE DEVICE 1404 PQ: 0 ANSI: 6
[ 689.483792] sd 3:0:0:0: Attached scsi generic sg0 type 0
[ 689.604892] sd 3:0:0:0: [sda] 121634816 512-byte logical blocks: (62.3 GB/58.0 GiB)
[ 689.606069] sd 3:0:0:0: [sda] Write Protect is off
[ 689.606073] sd 3:0:0:0: [sda] Mode Sense: 21 00 00 00
[ 689.607225] sd 3:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn’t support DPO or FUA
[ 689.631425] sda: sda1 sda2 sda3 sda4 sda5 sda6 sda7 sda8 sda9 sda10 sda11 sda12 sda13 sda14
[ 689.638267] sd 3:0:0:0: [sda] Attached SCSI removable disk
[ 690.615428] EXT4-fs (sda1): 9 orphan inodes deleted
[ 690.615431] EXT4-fs (sda1): recovery complete
[ 690.642302] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)

I am guessing this system is being powered down incorrectly, e.g., by cutting power. Both SD cards look to be working correctly, and without error. The error is from the ext4 filesystem. Orphan nodes and nodes being recovered is a sign of failure to write content before power is lost. Either intentional power down, or else some sort of crash condition which is so extreme there is no possibility of writing unwritten data. Has the system had power cut while running, or a crash which prevented normal shutdown?

hello anton.haes,

please also access NVIDIA SDK Manager for environment setups.
if you had install with SDKManager before, there’s system image save to your machine locally,
for example, you may check the path for JetPack-4.2.2 release images,
~/nvidia/nvidia_sdk/JetPack_4.2.2_Linux_GA_P3448/Linux_for_Tegra

you may enter the folder and enable flash script to flash your board again, please refer to Flash Script Usage for guidance.
please flash the Nano platform with the JetPack release image completely with below command.
i.e. $ sudo ./flash.sh jetson-nano-qspi-sd mmcblk0p1

While trying to flash the board with the SDKManager I get these errors:

  • 16:55:48 ERROR : Drivers for Jetson : tar: Linux_for_Tegra/bootloader/tos.img: Cannot create symlink to ‘tos-mon-only.img’
  • 16:55:48 ERROR : Drivers for Jetson : : Operation not permitted
  • 16:55:49 ERROR : Drivers for Jetson : tar: Linux_for_Tegra/jetson-nano-qspi.conf: Cannot create symlink to ‘p3448-0000.conf’: Operation not permitted
  • 16:55:49 ERROR : Drivers for Jetson : tar: Linux_for_Tegra/jetson-nano-qspi-sd.conf: Cannot create symlink to ‘p3448-0000-sd.conf’: Operation not permitted
  • 16:55:49 ERROR : Drivers for Jetson :
  • 16:55:49 ERROR : Drivers for Jetson : tar: Linux_for_Tegra/jetson-tx1.conf: Cannot create symlink to ‘p2371-2180-devkit.conf’: Operation not permitted
  • 16:56:00 ERROR : Drivers for Jetson : tar: Linux_for_Tegra/jetson-nano-emmc.conf: Cannot create symlink to ‘p3448-0000-emmc.conf’: Operation not permitted
  • 16:56:00 ERROR : Drivers for Jetson : tar: Exiting with failure status due to previous errors
  • 16:56:00 ERROR : Drivers for Jetson : [exec_command]: /bin/bash -c /tmp/tmp_NV_L4T_DRIVERS_T210_COMP.sh; [error]: exit status 2
  • 16:56:00 INFO : Drivers for Jetson : [ Package Install Finished with Error ]
  • 16:56:00 INFO : Drivers for Jetson : [ 4.00 KB used. Disk Avail: 15.33 GB ]
  • 16:56:00 INFO : Drivers for Jetson : [ NV_L4T_DRIVERS_T210_COMP Install took 36s ]
  • 16:56:00 ERROR : Drivers for Jetson : [error]: Run commands failed at step Install: command /tmp/tmp_NV_L4T_DRIVERS_T210_COMP.sh finished with error
  • 16:56:00 INFO : Drivers for Jetson :
  • 16:56:00 ERROR : Drivers for Jetson : command terminated with error
  • 16:56:00 ERROR : Drivers for Jetson : Install ‘Drivers for Jetson’ failure, command < using adapter to install NV_L4T_DRIVERS_T210_COMP@JETSON_NANO to /media/anton/THEO/anton/nvidia/nvidia_sdk/JetPack_4.4_DP_Linux_DP_JETSON_NANO > terminated with error.
  • 16:56:00 ERROR : Unavailable components detected, please check log
  • 16:56:00 ERROR : File System and OS : File System and OS Skipped. Due to its dependencies fail to install: Drivers for Jetson,
  • 16:56:02 ERROR : Device Mode Host Setup in Flash : Flash skipped due to component(s) download/install failure
  • 16:56:02 ERROR : Flash Jetson Nano : Flash skipped due to component(s) download/install failure
  • 16:56:02 ERROR : Device Mode Host Setup in Target SDK : Flash skipped due to component(s) download/install failure
  • 16:56:02 ERROR : DateTime Target Setup : Flash skipped due to component(s) download/install failure
  • 16:56:02 ERROR : CUDA Toolkit for L4T : Flash skipped due to component(s) download/install failure
  • 16:56:02 ERROR : cuDNN on Target : Flash skipped due to component(s) download/install failure
  • 16:56:02 ERROR : TensorRT on Target : Flash skipped due to component(s) download/install failure
  • 16:56:02 ERROR : OpenCV on Target : Flash skipped due to component(s) download/install failure
  • 16:56:02 ERROR : VisionWorks on Target : Flash skipped due to component(s) download/install failure
  • 16:56:02 ERROR : VPI on Target : Flash skipped due to component(s) download/install failure
  • 16:56:02 ERROR : NVIDIA Container Runtime with Docker integration (Beta) : Flash skipped due to component(s) download/install failure
  • 16:56:02 ERROR : Multimedia API : Flash skipped due to component(s) download/install failure
  • 16:56:02 ERROR : DeepStream : Flash skipped due to component(s) download/install failure

What should I do?

My situation: I’ve been running a Jetson Nano in headless mode since November 2019. I powered it down a few weeks ago because I was reorganizing my small computers. Today when I tried to boot it wasn’t responsive in headless mode (I could not ssh to it). I then connected it to a monitor, keyboard and mouse. The boot process got through the NVIDIA logo screen, but then the boot log showed several failures, including the display environment and NetworkManager. Then it hung up completely while creating a log concerning containerd (another failure). Not sure how to track this down, hoping it’s not a hardware failure.

Have you ever tried to run the SDKM as root (sudo)? If so, then this might have set permissions to root only, and will cause this to fail. Running only as a regular user allows SDKM to choose when to use sudo, but if you ever ran as root, then you probably need to remove that content and run again without sudo. Basically this log line implies lack of correct ownership:

16:55:49 ERROR : Drivers for Jetson : tar: Linux_for_Tegra/jetson-nano-qspi-sd.conf: Cannot create symlink to ‘p3448-0000-sd.conf’: Operation not permitted

The most common reason ownership would be wrong is using sudo at the wrong moment.

You’ll need to get a boot log via serial console:
https://www.jetsonhacks.com/2019/04/19/jetson-nano-serial-console/