Jetson Nano B01 fails booting from USB devices

Autonomous Machines Jetson & Embedded Systems Jetson Nano
,
1

Hi all,

My Jetson Nano fails booting from USB devices.
I plan to change the USB cable between Jetson Nano and SSD, but it probably fails.
Currently, I have no idea how to solve this issue…
Is there any suggestion about this issue?

  • Jetson Nano B01 (4GB)
    • Moudle: P3448-0000 (jetson-nano-devkit)
    • Carrier Board: B01
    • Power Supply: 5V 4A, place a jumper on the power select header (J48)
    • A small fan is connected the fan header (J15)
    • Power/Reset buttons are connected to the button header, auto-on disabled
    • USB-Serial converter is connected to UART on the button header
    • USB Mirco B (J28) is connected to a usb hub which is connected to a host machine
    • LAN cable is connected
    • HDMI/DisplayPort is not connected
  • JetPack 4.5.1
  • NVIDIA SDK Manager 1.5
  • Bootloaders have been updated in the first boot from the sdcard

I tried the following procedures, but none of them succeeded booting:

  • Changing USB port to use
  • Changing USB device
    1. USB 3.2 Gen2 SSD (SSD-PK960U3, KIOXIA) (no English page at this point)
    2. USB SATA adapter (BENFEI USB 3.0-SATA, JMicron) + HDD/1TB
  • Disabling UAS by adding a parameter like usb-storage.quirks=0x152d:0x0578:u
  • Changing the way to build root filesystem on USB device

    1. Flashing on a host machine as described in NVIDIA Jetson Linux Developer Guide - Flashing to a USB Drive

    2. Directly copying from the sdcard to the USB device by running the following commands on Jetson Nano:

      sudo mount /dev/sda1 /mnt
sudo rsync -axHAWX --numeric-ids --info=progress2 --exclude={'/dev','/lost+found','/media','/mnt','/proc','/run','/sys','/tmp'} / /mnt/
sudo mkdir -p /mnt/dev /mnt/media /mnt/mnt /mnt/proc /mnt/run /mnt/sys /mnt/tmp
sudo nano /mnt/boot/extlinux/extlinux.conf  # changes the `root` parameter
  • Re-Flashing bootloaders using SDK manager
  • Booting from the sdcard, then mounting the USB device as the root filesystem
    • Changing the root parameter in /boot/extlinux/extlinux.conf in the sdcard

Boot logs captured on the serial console are listed below:

From logs:

  • U-Boot can read data (/boot/extlinux/extlinux.conf, /boot/initrd and /boot/Image) successfully from the USB device
    • Probably, it loads tegra21x_xusb_firmware from somewhere because the firmware size is same as /lib/firmware/tegra21x_xusb_firmware
  • In the kernel boot sequence, some kind of error occurs depending on situations
    • usb 2-1.4: cmd cmplt err -71 where -71 means EPROTO
    • tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
    • blk_update_request: I/O error, dev sda, sector XXXXXX
    • FAT-fs (loop0): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
    • EXT4-fs error (device sda1): ext4_wait_block_bitmap:523: ...
    • and so on…
    • All errors seem to be related to the USB device

I confirmed that USB devices work properly with Jetson Nano when booting from the sdcard.
I can mount USB devices successfully.

I appreciate your help.

3

Please check if using this method to prepare the rootfs can bypass this issue or not.

https://elinux.org/Jetson/L4T/Boot_From_External_Device

4

@WayneWWW Thank you for reply.

As I wrote in the first comment, I did procedures described in NVIDIA Jetson Linux Developer Guide - Flashing to a USB Drive which seems to be similar procedures described in Jetson/L4T/Boot From External Device - eLinux.org.
Or is there any significant difference between them?

tar is used in Jetson/L4T/Boot From External Device - eLinux.org. On the other hand, rsync is used in NVIDIA Jetson Linux Developer Guide - Flashing to a USB Drive. However, I think that tar or rsync is not an important difference.

I also tried to directly copy files from the sdcard to USB devices. This method seems to be the same as copying files from Linux_for_Tegra/rootfs/.

5

@WayneWWW
I’ve understood the difference between the two methods.
The method described in Jetson/L4T/Boot From External Device - eLinux.org directly changes boot parameters used in bootloaders before U-Boot.

I performed each step but failed booting unfortunately.
The details are described below.

After the step 2:

# changed the label from PARTLABEL to APP
GPT fdisk (gdisk) version 1.0.3

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/sdb: 1875385008 sectors, 894.3 GiB
Model: EXCERIA         
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 23EB86BC-53AA-407D-A3D9-65E814CCF409
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 1875384974
Partitions will be aligned on 2048-sector boundaries
Total free space is 81886 sectors (40.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1           81920      1875384974   894.2 GiB   8300  APP

Before ejecting the USB device, created bootloader/l4t-rootfs-uuid.txt_ext by running the following command:

sudo blkid -o value -s PARTUUID /dev/sdb1 | sudo tee bootloader/l4t-rootfs-uuid.txt_ext

Before the step 4, connected the display, mouse, keyboard, a sdcard containing rootfs and the USB device to Jetson Nano. Then, booted in the recovery mode.
Without the sdcard, flash.sh stops at the following point:

[   2.8410 ] tegradevflash --oem platformdetails storage storage_info.bin
[   2.8420 ] Cboot version 00.01.0000
[   2.8519 ] Saved platform info in storage_info.bin

In the step 5.1, I ran sudo ./flash.sh jetson-nano-devkit external and succeeded flashing.
20210511-jetson-nano-flash.log (14.2 KB)

However, the boot sequence stopped and the setup GUI didn’t shown.
Error messages listed in the first comment were shown on the serial console like below:
20210511-jetson-nano-direct-usb-ssd-boot.log (28.3 KB)

6

I am little bit confused by your case here. It looks like either a rootfs corruption problem or the usb device problem.

Do you have other usb device drive that can try? Because it is weird that you say these two drives can work fine if you boot it from sdcard. If kernel has the capability to read the device correctly, then it should work fine with no such error. Do you see those error from usb if you boot from sdcard with usb drive attached?

[ 20.150832] tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
[ 20.160455] tegra-xusb 70090000.xusb: @00000000ffebcb90 ffe4f000 00000000 04000060 04078001
[ 20.168858] usb 2-1.4: cmd cmplt err -71
[ 20.172800] tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
[ 20.182403] tegra-xusb 70090000.xusb: @00000000ffebcbb0 ffe4f800 00000000 04000060 04048001
[ 20.191746] usb 2-1.4: cmd cmplt err -71
[ 20.196146] usb 2-1.4: cmd cmplt err -71
[ 20.200200] usb 2-1.4: cmd cmplt err -71
[ 20.433336] usb 2-1.4: cmd cmplt err -71

Could you also remove the “quiet” in your extlinux.conf to enable more kernel log from uart console?

7

Yeah, I have the same feelings as you.

When I first faced this issue, I thought something broken. So, I checked with fsck whether the filesystem has been broken or not, retried with another USB device and flashed bootloaders again. However, I had no positive result.

I have a USB sdcard adapter and a 32GB micro sdcard, and retried Jetson/L4T/Boot From External Device - eLinux.org again with them. However, the boot failed with the same errors.
20210512-jetson-nano-usb-sdcard-boot.log (41.5 KB)
The log file above contains flashing logs.

I saved boot logs without quiet:
20210512-jetson-nano-usb-sdcard-boot-without-quiet.log (81.4 KB)
Kernel seems to detect the root partition correctly:

[    6.123628] sd 0:0:0:0: [sda] 60751872 512-byte logical blocks: (31.1 GB/29.0 GiB)
[    6.133779] sd 0:0:0:0: [sda] Write Protect is off
[    6.141149] sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[    6.168970]  sda: sda1
[    6.175883] sd 0:0:0:0: [sda] Attached SCSI removable disk
[    6.186255] Freeing unused kernel memory: 8576K
[    6.217477] Root device found: PARTUUID=2e7e5722-5e3d-41e7-8b37-c4d71199d03a
[    6.685677] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[    6.702159] Rootfs mounted over PARTUUID=2e7e5722-5e3d-41e7-8b37-c4d71199d03a
[    6.744316] Switching from initrd to actual rootfs

Errors occurred after re-mount:

[    7.994105] EXT4-fs (sda1): re-mounted. Opts: (null)
...
[   38.938188] hub 2-1:1.0: hub_ext_port_status failed (err = -71)
[   38.945877] usb 2-1-port1: cannot reset (err = -71)
[   38.952599] usb 2-1-port1: cannot reset (err = -71)
[   38.959325] usb 2-1-port1: cannot reset (err = -71)
[   38.965957] usb 2-1-port1: cannot reset (err = -71)
[   38.972583] usb 2-1-port1: cannot reset (err = -71)
[   38.979080] usb 2-1-port1: Cannot enable. Maybe the USB cable is bad?

USB devices I used and filesystems on them seem not to be broken. Actually, I can mount them on Linux machines including Jetson Nano booted from a sdcard. I can access to files in them even after device/filesystem errors occurred in the Jetson Nano boot sequence. They were auto-mounted on Ubuntu desktop environment without any error when inserting them.

I also checked the filesystems with fsck on the Ubuntu desktop.
fsck reported that there was no broken block.

# KIOXIA SSD
fsck from util-linux 2.31.1
e2fsck 1.44.1 (24-Mar-2018)
/dev/sdc1: clean, 164883/58605568 files, 5145834/234412881 blocks

Output of dmesg on Jetson Nano when inserting and mounting a USB SSD is like below:

[ 1016.395390] usb 1-2.3: new high-speed USB device number 8 using tegra-xusb
[ 1016.438543] usb 1-2.3: New USB device found, idVendor=30de, idProduct=140c
[ 1016.438612] usb 1-2.3: New USB device strings: Mfr=2, Product=3, SerialNumber=1
[ 1016.438658] usb 1-2.3: Product: EXCERIA
[ 1016.438701] usb 1-2.3: Manufacturer: KIOXIA
[ 1016.438741] usb 1-2.3: SerialNumber: 20210128127D
[ 1016.445884] scsi host0: uas
[ 1016.449703] scsi 0:0:0:0: Direct-Access     KIOXIA   EXCERIA          0    PQ: 0 ANSI: 6
[ 1016.504264] sd 0:0:0:0: [sda] 1875385008 512-byte logical blocks: (960 GB/894 GiB)
[ 1016.515369] sd 0:0:0:0: [sda] Write Protect is off
[ 1016.520181] sd 0:0:0:0: [sda] Mode Sense: 43 00 00 00
[ 1016.520743] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 1016.530322] sd 0:0:0:0: [sda] Optimal transfer size 33553920 bytes
[ 1016.536927]  sda: sda1
[ 1016.539814] sd 0:0:0:0: [sda] Attached SCSI disk
[ 1046.798516] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)

No broken block was found usingfsck on Jetson Nano. USB devices seem to work properly on Jetson Nano.

I think that this is NOT a hardware issue. Maybe, the root cause is in software (module, firmware, parameters or other software-related stuff).

8

Do you have other usb pendrive to check this issue instead of using any adapters (usb to sd)?

Also, I would like to know do you have other jetson nano device there?

9

I have a 32GB USB 3.0 memory stick. I’ll try again with it.

I don’t have other jetson nano device.

10

Booting from the USB memory stick failed.

When I selected to use external in the step 5.1, boot failed due to the following errors:

mount: /mnt/: can't find PARTUUID=<UUID>.

I confirmed that the correct PARTUUID was used.

When I selected to use sda1 in the step 5.1, boot failed, too.
In this case, jetson nano booted from the sdcard and tried to mount /dev/sda1.
20210513-jetson-nano-usb-memory-sda1-boot.log (75.0 KB)

11

Hi,

I think we can skip the problem from UUID first. If you know the device will be mounted on sda1 or something else, you can use it first.

UUID is just a method when you don’t know which device name the usb memory stick will be located.

12

As I reported in the previous comment, booting from /dev/sda1 failed. Booting from /dev/sda1 without the sdcard also failed.

I don’t know the exact reason, but I cannot boot using the USB memory stick unfortunately.

13

Hi,

The reason seems to be that your uboot boot target is mmc > usb drive. But on that case, it seems mount rootfs from sda1 is working fine? I don’t see EXT4 error now.

What is the error log if you remove the sdcard? Could you share it?

14

I saw two cases before, but I just reproduced only the case.1:

Case.1: Aborted in the kernel boot sequence
20210513-jetson-nano-usb-memory-sda1-boot-without-sdcard.log (35.9 KB)
U-Boot seems to load initrd and Image from the USB memory stick successfully.

Case.2: Stopped before U-Boot due to some error (cannot reproduce at this time…)

Before trying to reproduce boot failures, I checked the filesystem in the USB memory stick using fsck on Ubuntu desktop. I didn’t see any broken blocks.

Image in the USB memory stick may be broken. I’ll re-create filesystem in the USB memory stick, and try again.

15

I gave up to boot from the USB memory stick. The USB memory stick is very slow and frequently stops working when copying data on Ubuntu desktop.

I have no other memory stick whose size is larger than 32GB. So, I cannot provide additional logs about booting from USB memory sticks, unfortunately.

16

Using another jetson nano, I tried booting from USB devices but never succeeded. I got the same results as reported so far.

This issue is not caused by jetson nano itself, maybe caused by software or my environment.

17

Confirmed that this issue was caused by my USB-Serial converter. Jetson nano booted from USB devices when removing the USB-Serial converter.

The USB-Serial converter was configured for 5V. After changing it to 3.3V, jetson nano boots without any error even when the USB-Serial converter is connected.

We should check the serial device configuration (5V or 3.3V) if we see boot errors which seem to be hardware/device errors when the serial is connected.