JetsonNano flash to USB SSD to boot

I’m trying to flash JetsonNano board and an external USB SSD, so that I can use the external USB SSD instead of the microSD. I find the info in the Developer Guide confusing. In particular the instructions given in ’ Basic Flashing Procedures → To flash the target device to mount a rootfs specified by UUID’ (call it #1) , and the ’ Flashing to a USB Drive → To set up a flash drive manually for booting’ (#2).

My setup:

  • the board is in the Force Recovery mode and connected to the host Linux machine
  • the USD SSD is connected to the host machine
  • both devices are visible i.e.
    __ lsusb
    Bus 002 Device 002: ID 152d:0583 JMicron Technology Corp. / JMicron USA Technology Corp.

    Bus 001 Device 004: ID 0955:7f21 NVidia Corp.

Now per ‘#1
sudo ./flash.sh jetson-nano-qspi external
###############################################################################

L4T BSP Information:

R32 , REVISION: 6.1

###############################################################################

Target Board Information:

Name: jetson-nano-qspi, Board Family: t210ref, SoC: Tegra 210,

OpMode: production, Boot Authentication: ,

Disk encryption: disabled ,

###############################################################################
./tegraflash.py --chip 0x21 --applet “/home/biocodz/Linux_for_Tegra/bootloader/nvtboot_recovery.bin” --skipuid --cmd “dump eeprom boardinfo cvm.bin”
Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands

[ 0.0048 ] Generating RCM messages
[ 0.0098 ] tegrarcm --listrcm rcm_list.xml --chip 0x21 0 --download rcm /home/biocodz/Linux_for_Tegra/bootloader/nvtboot_recovery.bin 0 0
[ 0.0115 ] RCM 0 is saved as rcm_0.rcm
[ 0.0126 ] RCM 1 is saved as rcm_1.rcm
[ 0.0133 ] List of rcm files are saved in rcm_list.xml
[ 0.0284 ]
[ 0.0297 ] Signing RCM messages
[ 0.0366 ] tegrasign --key None --list rcm_list.xml --pubkeyhash pub_key.key
[ 0.0396 ] Assuming zero filled SBK key
[ 0.0603 ]
[ 0.0619 ] Copying signature to RCM mesages
[ 0.0675 ] tegrarcm --chip 0x21 0 --updatesig rcm_list_signed.xml
[ 0.0824 ]
[ 0.0843 ] Boot Rom communication
[ 0.0879 ] tegrarcm --chip 0x21 0 --rcm rcm_list_signed.xml --skipuid
[ 0.0897 ] RCM version 0X210001
xx
klkjlkjlkjlkjlkj (<-- entering some random input here and hitting ENTER - nothing happens)
^CTraceback (most recent call last):
File “./tegraflash.py”, line 1296, in
tegraflash_run_commands()
File “./tegraflash.py”, line 1155, in tegraflash_run_commands
interpreter.onecmd(command)
File “/usr/lib/python3.6/cmd.py”, line 217, in onecmd
return func(arg)
File “./tegraflash.py”, line 785, in do_dump
tegraflash_dump(exports, args)
File “/home/biocodz/Linux_for_Tegra/bootloader/tegraflash_internal.py”, line 1360, in tegraflash_dump
tegraflash_send_tboot(tegrarcm_values[‘–signed_list’])
File “/home/biocodz/Linux_for_Tegra/bootloader/tegraflash_internal.py”, line 2463, in tegraflash_send_tboot
uidlog = run_command(command, True)
File “/home/biocodz/Linux_for_Tegra/bootloader/tegraflash_internal.py”, line 202, in run_command
log = print_process(process, enable_print)
File “/home/biocodz/Linux_for_Tegra/bootloader/tegraflash_internal.py”, line 161, in print_process
output = process.stdout.read(1)
KeyboardInterrupt
Reading board information failed.

The process is stuck (waiting for the input?). After trying some random input, and eventually interrupting, the process gives the error trace as shown above.

Trying to repeat the same command again:
sudo ./flash.sh jetson-nano-qspi external
###############################################################################

L4T BSP Information:

R32 , REVISION: 6.1

###############################################################################
Error: probing the target board failed.
Make sure the target board is connected through
USB port and is in recovery mode.

It doesn’t see the board even though it is still connected to the host and is visible.

Apparently I don’t understand what is supposed to happen when the root device is ‘external’.

Following instruction ‘#2
sudo BOOTDEV=sdb1 ./flash.sh --no-flash jetson-nano-devkit sdb1
generates huge image file (14GB), so that I ran out of memory on my host VM, and on the way reports a load of errors, and eventually the JetsonNano gets disconnected from the host.

Is it possible to flash the USB SSD directly i.e. without generating the image file on the host machine?

I think there are some misunderstand and mistakes here.

  1. jetson-nano-qspi config is to flash the QSPI-NOR on the module only. What is your exact module? Is it emmc based or sdcard based?

  2. “flash.sh” is not able to flash anything to your external drive. This tool can only tell the kernel where to load the file system. That is what “external” is doing. I wrote a post here to explain.
    Boot from external drive - #11

  3. Below error has nothing to explain. Just the board is not in recovery mode. You need to put it in RCM manually again.
    RCM can only has one time chance. For example, if you put a board in RCM, try to flash but fail, then you need to put the board into RCM again before your next flash command.


sudo ./flash.sh jetson-nano-qspi external
###############################################################################

# L4T BSP Information:

# R32 , REVISION: 6.1

###############################################################################
Error: probing the target board failed.
Make sure the target board is connected through
USB port and is in recovery mode.

Hi,
thank you for the prompt response!

  1. my board is JetsonNano rev B01 with SDCard, and it works fine when using the SDCard. I’m aware of the ‘jetson-nano-qspi’ intent to flash the QSPI. Just baffled why it hangs.

  2. I’m aware of the post ‘Boot from external drive’ and it is quite useful.
    So, I gather creating the image using ‘flash.sh’, and then copying the image to the USB SSD using e.g. ‘rsync’ or ‘dd’ is the way to go.

Do I understand correctly that in order to boot and load rootFS from the USD SSB I have to flash QSPI, and create/copy the image containing root FS, kernel, and bootloader components to the USB SSD, and for that I have to use both commands

  • flash.sh jetson-nano-qspi external ← flash QSPI
  • lash.sh --no-flash jetson-nano-devkit sdb1 with subsequent rsync ← create/copy image
    ?

Please still use mmcblk0p1 for QSPI. Your next move with flash.sh --no-flash jetson-nano-devkit sdb1 will tell kernel where to mount the filesystem.

@WayneWWW

So, if I understand correctly, the mmcblk0p1 still needs to be inserted while flashing the QSPI.
The ‘external’ apparently tells the QSPI part of the bootloader to search for the rest of the bootloader on the external device.
I’ll give it a try.

No, this does not happen.There is no such thing. Bootloader will only be on QSPI.

Actually I just want to you to flash the QSPI with the common command we used.

Still struggling with this, also due lack of time can spare on this project.
Using host with native Ubuntu 18.04 now to avoid problems with VMs.
Also setting up my own rootfs to make it lighter than the NVIDIA’s sample one.

  • created rootfs using debootstrap ~/rootfs/bionic/
  • sudo LDK_ROOTFS_DIR=~/rootfs_bionic ./apply_binaries.sh 2>&1 | tee … → Success!
  • connect USBSSD to the host
  • lsblk -dpsf → /dev/sdc1 ext4 usbssd d5118148-4eca-4cc2-8d71-e3d3d2c0e89c
  • sudo mount /dev/sdc1 /mnt/usbssd
  • pushd ~/rootfs_bionic
  • sudo cp -a * /mnt/usbssd/ && sync ← as per the L4T Guide: rootfs_custom.html:copy the file system to an external storage device
  • attach USBSSD to JetsonNano
  • as mentioned the Guide is full of dead ends and unfinished directions: " 6. After copying the content to the external disk or device, unmount the disk and connect it to the target SoC device.
    Proceed to Flashing and Booting the Target Device."
    So, where to proceed from here? Trial and error continues
  • put the JetsonNano into Recovery and attach to the host
  • lsusb → Bus 003 Device 010: ID 0955:7f21 NVidia Corp.
  • pushd ~/Linux_for_Tegra
  • sudo ./flash.sh jetson-nano-devkit external | tee … →
# L4T BSP Information:
# R32 , REVISION: 6.1

# Target Board Information:
# Name: jetson-nano-devkit, Board Family: t210ref, SoC: Tegra 210,
# OpMode: production, Boot Authentication: ,
# Disk encryption: disabled ,

./tegraflash.py --chip 0x21 --applet "/home/biocodz/Linux_for_Tegra/bootloader/nvtboot_recovery.bin" --skipuid --cmd "dump eeprom boardinfo cvm.bin"
Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands

[   0.0018 ] Generating RCM messages
[   0.0043 ] tegrarcm --listrcm rcm_list.xml --chip 0x21 0 --download rcm /home/biocodz/Linux_for_Tegra/bootloader/nvtboot_recovery.bin 0 0
[   0.0055 ] RCM 0 is saved as rcm_0.rcm
[   0.0063 ] RCM 1 is saved as rcm_1.rcm
[   0.0063 ] List of rcm files are saved in rcm_list.xml
[   0.0063 ]
[   0.0063 ] Signing RCM messages
[   0.0084 ] tegrasign --key None --list rcm_list.xml --pubkeyhash pub_key.key
[   0.0093 ] Assuming zero filled SBK key
[   0.0183 ]
[   0.0183 ] Copying signature to RCM mesages
[   0.0209 ] tegrarcm --chip 0x21 0 --updatesig rcm_list_signed.xml
[   0.0236 ]
[   0.0237 ] Boot Rom communication
[   0.0264 ] tegrarcm --chip 0x21 0 --rcm rcm_list_signed.xml --skipuid
[   0.0279 ] RCM version 0X210001
[   0.0719 ] Boot Rom communication completed
[   1.0788 ]
[   1.0789 ] dump EEPROM info
[   1.0827 ] tegrarcm --oem platformdetails eeprom /home/biocodz/Linux_for_Tegra/bootloader/cvm.bin
[   1.0850 ] Applet version 00.01.0000
[   1.1013 ] Saved platform info in /home/biocodz/Linux_for_Tegra/bootloader/cvm.bin
[   1.1780 ]
[   1.1814 ] tegrarcm --reboot recovery
[   1.1831 ] Applet version 00.01.0000
[   1.2010 ]
Board ID(3448) version(400)
Using UUID d5118148-4eca-4cc2-8d71-e3d3d2c0e89c for mounting root APP_ext partition.
copying bctfile(/home/biocodz/Linux_for_Tegra/bootloader/t210ref/BCT/P3448_A00_lpddr4_204Mhz_P987.cfg)... done.
copying bootloader(/home/biocodz/Linux_for_Tegra/bootloader/t210ref/cboot.bin)... done.
copying initrd(/home/biocodz/Linux_for_Tegra/bootloader/l4t_initrd.img)... done.
Making Boot image... done.
Existing sosfile(/home/biocodz/Linux_for_Tegra/bootloader/nvtboot_recovery.bin) reused.
copying tegraboot(/home/biocodz/Linux_for_Tegra/bootloader/t210ref/nvtboot.bin)... done.
copying cpu_bootloader(/home/biocodz/Linux_for_Tegra/bootloader/t210ref/cboot.bin)... done.
copying bpffile(/home/biocodz/Linux_for_Tegra/bootloader/t210ref/sc7entry-firmware.bin)... done.
copying wb0boot(/home/biocodz/Linux_for_Tegra/bootloader/t210ref/warmboot.bin)... done.
Existing tosfile(/home/biocodz/Linux_for_Tegra/bootloader/tos-mon-only.img) reused.
Existing eksfile(/home/biocodz/Linux_for_Tegra/bootloader/eks.img) reused.
copying dtbfile(/home/biocodz/Linux_for_Tegra/kernel/dtb/tegra210-p3448-0000-p3449-0000-b00.dtb)... done.
Copying nv_boot_control.conf to rootfs
sed: can't read /home/biocodz/Linux_for_Tegra/rootfs/etc/nv_boot_control.conf: Not a directory
sed: can't read /home/biocodz/Linux_for_Tegra/rootfs/etc/nv_boot_control.conf: Not a directory
...
generating system.img for booting...
Making system.img...
/tmp/tmp.AwJ8YEyaF6/boot/extlinux/extlinux.conf is not found, exiting...

Why it tries the Linux_for_Tegra/rootfs when the external is specified?

Not sure why you asked where to start… there is just a section called " Flashing to a USB Drive". You can refer to it.
https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/flashing.html#wwpID0E01O0HA

Also, please still make sure your BSP package (Linux_for_Tegra) is a completed one. If you don’t know what doet completed one mean, just let sdkmanager download it.

I read the section Flashing to USB, and found it totally confusing.

  1. nothing tells me that I should ignore Basic Flashing Procedures -> flash the target device to mount a rootfs specified by UUID -> sudo ./flash.sh <board> external. This logic comes first on the list, whence I tried it first
  2. Flashing to a USB Drive -> 2. Connect the flash drive to the host computer. … what? My drive already has the rootfs and IS connected to the SoC as advised in the Guide. See the steps performed in previous message.

I’m baffled at this point how to proceed

It is hard to tell what is the difference between these two tutorial if you have no idea what yourself is doing.

I would suggest you firstly boot to sdcard, check the serial console, dump it here and I will point out what is going on first.

This method is very common to deal with boot issue. So better using this first.

BTW, the error you hit from previous comment has nothing do with whether boot from usb or boot from anything.

That error looks more like your BSP package is incomplete. That is why I said this comment.

Also, please still make sure your BSP package (Linux_for_Tegra) is a completed one. If you don’t know what does completed one mean, just let sdkmanager download it.

Thank you for the info. I have extracted the BSP as

Not sure why it would be incomplete but I’ll double check

The rootfs is still needed.

And you need to run apply_binaries.sh script. This is mentioned in quick start guide document.

If you don’t want to worry about this, just let sdkmanager prepare the package…

I created the rootfs exactly as the Guide advises (to the best of my interpretation at least) and apply_binary.sh has been run successfully as shown in my previous messages.:

  • sudo LDK_ROOTFS_DIR=~/rootfs_bionic ./apply_binaries.sh 2>&1 | tee ...

I need to understand the nitty-gritty details, so I do want to worry about this:-)

Could you clarify why the “LDK_ROOTFS_DIR” is “~/rootfs_bionic”? It should be Linux_for_Tegra/rootfs if you didn’t customize anything else.

I’m working with a custom rootfs that was created at ~/rootfs_bionic/ using debootstrap as shown in my previous messages.

Here is the output of the debug console:
minicom_202112151526.cap (22.3 KB)

Ok, open your own log and check these lines.

This is the uboot bootloader which is responsible for loading the kernel.

U-Boot 2020.04-g46e4604c78 (Jul 26 2021 - 12:09:42 -0700)

SoC: tegra210
Model: NVIDIA Jetson Nano Developer Kit
Board: NVIDIA P3450-0000
DRAM:  4 GiB
MMC:   sdhci@700b0000: 1, sdhci@700b0600: 0
Loading Environment from SPI Flash... SF: Detected mx25u3235f with page size 256 Bytes, erase size 4 KiB, total 4 MiB
*** Warning - bad CRC, using default environment

In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
Hit any key to stop autoboot:  2  1  0 
switch to partitions #0, OK
mmc1 is current device
Scanning mmc 1:1...
Found /boot/extlinux/extlinux.conf
Retrieving file: /boot/extlinux/extlinux.conf
845 bytes read in 27 ms (30.3 KiB/s)
1:	primary kernel
Retrieving file: /boot/initrd
7160133 bytes read in 332 ms (20.6 MiB/s)
Retrieving file: /boot/Image

As you can see, this bootloader firstly found “mmc1” as boot device. mmc1 is the sdcard on your board.
And it found “/boot/extlinux/extlinux.conf”, “/boot/initrd” and “/boot/Image” from mmc1. You can just call them kernel to make it easier to understand.

This step- loading kernel from the drive is what we call “boot from something”.

After kernel is up, there would be a kernel cmdline. looks like below.

tegraid=21.1.2.0.0 ddr_die=4096M@2048M section=512M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 console=ttyS0,115200n8 debug_uartport=lsport,4 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=0x1000@0xff780000 core_edp_mv=1125 core_edp_ma=4000 gpt tegra_fbmem=0x800000@0x92ca9000 is_hdmi_initialised=1 earlycon=uart8250,mmio32,0x70006000 root=PARTUUID=0e165ff1-d050-4ae9-a085-9c5309602862 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 quiet root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0

There is one keyword “root=/dev/mmcblk0p1” inside the cmdline. This line indicates where to mount the filesystem.
In your case, “dev/mmcblk0p1” is still sdcard, so it will load file system from sdcard.

Back to those flash methods.

Actually, Jetson/L4T/Boot From External Device - eLinux.org is just doing the second point. The last parameter in flash.sh will tell the kernel where to mount the file system. It does not really tell uboot where boot the kernel.
This method is general one for all jetson platforms since bootloader may not boot some components due to driver problem. For example, TX2 and TX2-NX uboot are not able to boot from usb.

As for the tutorial from l4t developer guide, it wants to ensure both kernel and rootfs are from the external drive.

Thank you for the information.
However it is still not clear how to make the u-boot to load the OS from the USB SSD.

From the debug console:

...
U-Boot 2020.04-g46e4604c78 (Jul 26 2021 - 12:09:42 -0700)
...
mmc1 is current device     <- SD card apparently
Scanning mmc 1:1...
Found /boot/extlinux/extlinux.conf
...

How do I make sda1 (see below) the current device instead of mmc1?
Do I need to modify the partition configuration file and re-build the u-boot?

The bootloader finds and auto-mounts the SSD but not as the rootfs.
lsblk

sda            8:0    0 238.5G  0 disk
└─sda1         8:1    0 229.1G  0 part /media/biocodz/usbssd  <- how to mount this to /  ?
mtdblock0     31:0    0     4M  0 disk
mmcblk0      179:0    0 119.1G  0 disk
├─mmcblk0p1  179:1    0 119.1G  0 part /
├─mmcblk0p2  179:2    0   128K  0 part

If you don’t insert a sdcard there, then uboot will not be able to find sd and goes to next boot target, that boot target will be usb.