Boot from external drive

,

Hello,

I understand that there are lots of people asking about how to boot from external drive (e.g. usb/NVMe drive). And also lots of methods all around this forum. Thanks for everyone to share their steps to achieve that.

This post is just to give some basic concepts about how jetson platform boots from external drive. And answer some frequently-asked questions.
I believe every method shared on this forum uses the same idea but just in different forms.

1. Jetson is not able to “fully” boot from external drive.

What jetson can do is boot “initrd/kernel” or “mount file system” from your external drive. It means you cannot just format the boot device and leave it empty. There must be bootloader components in your boot device so that the boot process can use the bootloader from it. For sdcard nano or NX, this boot device is QSPI on the module. As for other platforms, it is the emmc on the module.

2. You have to know the difference between “boot kernel from external drive” and “mount rootfs from external drive”

  • Boot initrd/kernel from external drive means the uboot (Nano/TX2 series) or cboot (NX/AGX) will search the kernel and extlinux.conf from your external drive according to the boot order. Once it finds the kernel and extlinux.conf, it will use the info from it to mount the file system.

  • Mount the file system means kernel will mount the file system according to your “root=” in kernel command line.
    For example, the most common usage is root=/dev/mmcblk0p1 which will boot from sdcard (nx/nano) and emmc.

→ Thus, if you just want to increase your disk space, you don’t really need to boot kernel from external drive, you just make sure the file system is mounted from external drive.
→ It means there are lots of variety that you can use. For example, you can boot kernel from usb and mount the file system from NVMe.

3. How to do?
We provide steps inside the developer guide link for booting both kernel and mount rootfs from the same drive.
Please be careful that every platform shares different bootloader capability. For example, TX2 uboot does not support usb .

If you just want to mount the file system from external, then you can check this page.
I am sorry that the title of this page is not precise. It is just mounting the rootfs from external. Not booting from anything.
https://elinux.org/Jetson/L4T/Boot_From_External_Device

4. flash.sh will not flash anything to your external drive.
There were some users asking about why “sudo ./flash.sh … sda1” does not flash data to their usb drive. The truth is flash.sh currently only has capability to flash data to native sdcard (nano/nx) and emmc boot device. What this “sudo ./flash.sh … sda1” doing is just adding “root=/dev/sda1/” to the kernel command line. Which means this is only telling kernel to mount the file system from sda1. Nothing else is installed to the sda1 usb drive.

5. Can I create a USB drive that is similar to the sdcard image on Nano/NX?
Theoretically, flashing the sdcard image to a USB drive can work. Just in case the bootloader(uboot/cboot) is able to find your usb first, but not some other external drives. Also, you have to make sure the root device in kernel command line is not setting to something like mmcblk0p1. Otherwise, your file system will not be mounted correctly.

6. Official OTA upgrade does not support boot from external
Your “boot from external” setting would be gone after you try the apt upgrade. Because our OTA debian package by default only supports the boot from default path (emmc/sd).

7. (update: 2022/2/14) SDKM is able to directly flash to NVMe on jetson after jetpack4.6
This is because the initrd flash tool in this link is implemented and it can directly flash the NVMe drive on jetson of TX2-series/Xavier series.
link

11 Likes

Thanks @WayneWWW for sharing all the steps and details for booting Jetson NX from external drive. This is very helpful.

We were able to boot from SATA drive but run into a problem while installing multimedia and deepstream packages in the external drive. Can you please provide the steps for the same?

Also, we are unable to run sudo apt upgrade to upgrade the packages and also unable to install any new packages e.g. python3-pip etc. This might be due to the issue you have referred to the last point. Is there a work around for the same?

Error log:

:~$ sudo apt install python3-pip
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  dh-python libpython3-dev libpython3.6-dev python3-dev python3-setuptools python3-wheel python3.6-dev
Suggested packages:
  python-setuptools-doc
The following NEW packages will be installed:
  dh-python libpython3-dev libpython3.6-dev python3-dev python3-pip python3-setuptools python3-wheel python3.6-dev
0 upgraded, 8 newly installed, 0 to remove and 0 not upgraded.
6 not fully installed or removed.
Need to get 46.1 MB of archives.
After this operation, 78.8 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://ports.ubuntu.com/ubuntu-ports bionic/main arm64 dh-python all 3.20180325ubuntu2 [89.2 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 libpython3.6-dev arm64 3.6.9-1~18.04ubuntu1.4 [45.1 MB]
Get:3 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 libpython3-dev arm64 3.6.7-1~18.04 [7,328 B]                            
Get:4 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 python3.6-dev arm64 3.6.9-1~18.04ubuntu1.4 [508 kB]                     
Get:5 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 python3-dev arm64 3.6.7-1~18.04 [1,288 B]                               
Get:6 http://ports.ubuntu.com/ubuntu-ports bionic-updates/universe arm64 python3-pip all 9.0.1-2.3~ubuntu1.18.04.5 [114 kB]                  
Get:7 http://ports.ubuntu.com/ubuntu-ports bionic/main arm64 python3-setuptools all 39.0.1-2 [248 kB]                                        
Get:8 http://ports.ubuntu.com/ubuntu-ports bionic/universe arm64 python3-wheel all 0.30.0-0.2 [36.5 kB]                                      
Fetched 46.1 MB in 16s (2,921 kB/s)                                                                                                          
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package dh-python.
(Reading database ... 173372 files and directories currently installed.)
Preparing to unpack .../0-dh-python_3.20180325ubuntu2_all.deb ...
Unpacking dh-python (3.20180325ubuntu2) ...
Selecting previously unselected package libpython3.6-dev:arm64.
Preparing to unpack .../1-libpython3.6-dev_3.6.9-1~18.04ubuntu1.4_arm64.deb ...
Unpacking libpython3.6-dev:arm64 (3.6.9-1~18.04ubuntu1.4) ...
Selecting previously unselected package libpython3-dev:arm64.
Preparing to unpack .../2-libpython3-dev_3.6.7-1~18.04_arm64.deb ...
Unpacking libpython3-dev:arm64 (3.6.7-1~18.04) ...
Selecting previously unselected package python3.6-dev.
Preparing to unpack .../3-python3.6-dev_3.6.9-1~18.04ubuntu1.4_arm64.deb ...
Unpacking python3.6-dev (3.6.9-1~18.04ubuntu1.4) ...
Selecting previously unselected package python3-dev.
Preparing to unpack .../4-python3-dev_3.6.7-1~18.04_arm64.deb ...
Unpacking python3-dev (3.6.7-1~18.04) ...
Selecting previously unselected package python3-pip.
Preparing to unpack .../5-python3-pip_9.0.1-2.3~ubuntu1.18.04.5_all.deb ...
Unpacking python3-pip (9.0.1-2.3~ubuntu1.18.04.5) ...
Selecting previously unselected package python3-setuptools.
Preparing to unpack .../6-python3-setuptools_39.0.1-2_all.deb ...
Unpacking python3-setuptools (39.0.1-2) ...
Selecting previously unselected package python3-wheel.
Preparing to unpack .../7-python3-wheel_0.30.0-0.2_all.deb ...
Unpacking python3-wheel (0.30.0-0.2) ...
Setting up nvidia-l4t-bootloader (32.5.1-20210614115015) ...
ERROR. Cannot open /etc/nv_boot_control.conf for reading.
Cannot install package. Exiting...
dpkg: error processing package nvidia-l4t-bootloader (--configure):
 installed nvidia-l4t-bootloader package post-installation script subprocess returned error exit status 1
Setting up nvidia-l4t-xusb-firmware (32.5.1-20210614115015) ...
ERROR. Cannot open /etc/nv_boot_control.conf for reading.
Cannot install package. Exiting...
dpkg: error processing package nvidia-l4t-xusb-firmware (--configure):
 installed nvidia-l4t-xusb-firmware package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of nvidia-l4t-initrd:
 nvidia-l4t-initrd depends on nvidia-l4t-xusb-firmware (= 32.5.1-20210614115015); however:
  Package nvidia-l4t-xusb-firmware is not configured yet.dpkg: error processing package nvidia-l4t-initrd (--configure):
 dependency problems - leaving unconfigured
No apport report written because the error message indicates its a followup error from a previous failure.
                                                                                                          Setting up python3-wheel (0.30.0-0.2) ...
Setting up libpython3.6-dev:arm64 (3.6.9-1~18.04ubuntu1.4) ...
Setting up python3-pip (9.0.1-2.3~ubuntu1.18.04.5) ...
Setting up nvidia-l4t-kernel (4.9.201-tegra-32.5.1-20210505093723) ...
Using the existing boot entry 'primary'
ERROR. Cannot open /etc/nv_boot_control.conf for reading.
Cannot install package. Exiting...
dpkg: error processing package nvidia-l4t-kernel (--configure):
 installed nvidia-l4t-kernel package post-installation script subprocess returned error exit status 1
No apport report written because MaxReports is reached already
                                                              Setting up python3-setuptools (39.0.1-2) ...
Setting up python3.6-dev (3.6.9-1~18.04ubuntu1.4) ...
Setting up dh-python (3.20180325ubuntu2) ...
Setting up libpython3-dev:arm64 (3.6.7-1~18.04) ...
dpkg: dependency problems prevent configuration of nvidia-l4t-kernel-dtbs:
 nvidia-l4t-kernel-dtbs depends on nvidia-l4t-kernel (= 4.9.201-tegra-32.5.1-20210505093723); however:
  Package nvidia-l4t-kernel is not configured yet.dpkg: error processing package nvidia-l4t-kernel-dtbs (--configure):
 dependency problems - leaving unconfigured
Setting up python3-dev (3.6.7-1~18.04) ...
No apport report written because MaxReports is reached already
                                                              dpkg: dependency problems prevent configuration of nvidia-l4t-kernel-headers:
 nvidia-l4t-kernel-headers depends on nvidia-l4t-kernel (= 4.9.201-tegra-32.5.1-20210505093723); however:
  Package nvidia-l4t-kernel is not configured yet.dpkg: error processing package nvidia-l4t-kernel-headers (--configure):
 dependency problems - leaving unconfigured
No apport report written because MaxReports is reached already
                                                              Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Errors were encountered while processing:
 nvidia-l4t-bootloader
 nvidia-l4t-xusb-firmware
 nvidia-l4t-initrd
 nvidia-l4t-kernel
 nvidia-l4t-kernel-dtbs
 nvidia-l4t-kernel-headers
E: Sub-process /usr/bin/dpkg returned an error code (1)

Error log while flashing Multimedia and Deepstream:

21:43:23 ERROR: Multimedia API - target: E: Unable to correct problems, you have held broken packages.
21:43:23 ERROR: Multimedia API - target: command terminated with error
21:43:23 SUMMARY: Multimedia API - target: First Error: APT system is broken and requires manual fix
21:43:23 DEBUG: running command < true >
21:43:23 INFO: command finished successfully
21:43:23 DEBUG: DeepStream - target: using adapter to install NV_DEEPSTREAM_TARGET_POST_INSTALL_COMP@JETSON_XAVIER_NX to /home/ubuntu-dell/nvidia/nvidia_sdk/JetPack_4.5.1_Linux_JETSON_XAVIER_NX
21:43:23 INFO: DeepStream - target: exec_command: ssh -t -q -o PreferredAuthentications=password -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no localhost@192.168.55.1 "df -B1 / | tail -1 | awk '{ printf \$4 }'"
21:43:23 INFO: DeepStream - target: [target] localhost@192.168.55.1's password:
21:43:24 INFO: DeepStream - target: current working directory is /home/ubuntu-dell/nvidia/nvidia_sdk/JetPack_4.5.1_Linux_JETSON_XAVIER_NX
21:43:24 INFO: DeepStream - target: **********************
21:43:24 INFO: DeepStream - target: exec_command: scp -q -o PreferredAuthentications=password -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no /tmp/tmp_NV_DEEPSTREAM_TARGET_POST_INSTALL_COMP.sh localhost@192.168.55.1:~; ssh -t -q -o PreferredAuthentications=password -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no localhost@192.168.55.1 "set -o pipefail; ~/tmp_NV_DEEPSTREAM_TARGET_POST_INSTALL_COMP.sh | sed 's/^/SDKM_INSTALL_ERROR /' && rm -f ~/tmp_NV_DEEPSTREAM_TARGET_POST_INSTALL_COMP.sh"
21:43:24 INFO: DeepStream - target: [target] localhost@192.168.55.1's password:
21:43:24 INFO: DeepStream - target: [target] localhost@192.168.55.1's password:
21:43:24 INFO: DeepStream - target: *begin sudo apt-get update, 5
21:43:24 INFO: DeepStream - target: [target] [sudo] password for localhost:
21:43:25 INFO: DeepStream - target: 0% [Working] Hit:1 https://repo.download.nvidia.com/jetson/common r32.5 InRelease
21:43:25 INFO: DeepStream - target: 0% [Connecting to ports.ubuntu.com (91.189.91.38)] 0% [1 InRelease gpgv 2,552 B] [Connecting to ports.ubuntu.com (91.189.91.38)] [ Hit:2 https://repo.download.nvidia.com/jetson/t194 r32.5 InRelease
21:43:25 INFO: DeepStream - target: 0% [1 InRelease gpgv 2,552 B] [Connecting to ports.ubuntu.com (91.189.91.39)] 0% [Waiting for headers] 0% [2 InRelease gpgv 2,562 B] [Waiting for headers] Hit:3 http://ports.ubuntu.com/ubuntu-ports bionic InRelease
21:43:26 INFO: DeepStream - target: 0% [2 InRelease gpgv 2,562 B] 0% [Waiting for headers] 0% [3 InRelease gpgv 242 kB] [Waiting for headers] 0% [Waiting for headers] Hit:4 http://ports.ubuntu.com/ubuntu-ports bionic-updates InRelease
21:43:26 INFO: DeepStream - target: 0% [Working] 0% [4 InRelease gpgv 88.7 kB] [Waiting for headers] 0% [Waiting for headers] Hit:5 http://ports.ubuntu.com/ubuntu-ports bionic-backports InRelease
21:43:26 INFO: DeepStream - target: 0% [Working] 0% [5 InRelease gpgv 74.6 kB] [Waiting for headers] 0% [Waiting for headers] Hit:6 http://ports.ubuntu.com/ubuntu-ports bionic-security InRelease
21:43:28 INFO: DeepStream - target: 0% [Working] 0% [6 InRelease gpgv 88.7 kB] 20% [Working] Reading package lists... 0% Reading package lists... 0% Reading package lists... 0% Reading package lists... 4% Reading package lists... 4% Reading package lists... 6% Reading package lists... 6% Reading package lists... 6% Reading package lists... 6% Reading package lists... 6% Reading package lists... 6% Reading package lists... 33% Reading package lists... 41% Reading package lists... 41% Reading package lists... 61% Reading package lists... 61% Reading package lists... 61% Reading package lists... 61% Reading package lists... 61% Reading package lists... 61% Reading package lists... 66% Reading package lists... 66% Reading package lists... 72% Reading package lists... 72% Reading package lists... 72% Reading package lists... 72% Reading package lists... 73% Reading package lists... 73% Reading package lists... 79% Reading package lists... 79% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 86% Reading package lists... 86% Reading package lists... 91% Reading package lists... 91% Reading package lists... 91% Reading package lists... 91% Reading package lists... 92% Reading package lists... 92% Reading package lists... 96% Reading package lists... 96% Reading package lists... 98% Reading package lists... 98% Reading package lists... 98% Reading package lists... 98% Reading package lists... 98% Reading package lists... 98% Reading package lists... 98% Reading package lists... 98% Reading package lists... 98% Reading package lists... 98% Reading package lists... Done
21:43:28 INFO: DeepStream - target: *begin sudo apt-get install -y libssl1.0.0 libgstreamer1.0-0 libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev gstreamer1.0-tools gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav libgstrtspserver-1.0-0 libjansson4 librdkafka1=0.11.3-1build1 , 5
21:43:29 INFO: DeepStream - target: Reading package lists... 0% Reading package lists... 100% Reading package lists... Done
21:43:29 INFO: DeepStream - target: Building dependency tree... 0% Building dependency tree... 0% Building dependency tree... 50% Building dependency tree... 50% Building dependency tree
21:43:29 INFO: DeepStream - target: Reading state information... 0% Reading state information... 0% Reading state information... Done
21:43:29 INFO: DeepStream - target: libgstrtspserver-1.0-0 is already the newest version (1.14.5-0ubuntu1~18.04.1).
21:43:30 INFO: DeepStream - target: 6 not fully installed or removed.
21:43:30 INFO: DeepStream - target: After this operation, 0 B of additional disk space will be used.
21:43:31 INFO: DeepStream - target: Setting up nvidia-l4t-bootloader (32.5.1-20210614115015) ...
21:43:31 INFO: DeepStream - target: Setting up nvidia-l4t-xusb-firmware (32.5.1-20210614115015) ...
21:43:31 INFO: DeepStream - target: Setting up nvidia-l4t-kernel (4.9.201-tegra-32.5.1-20210505093723) ...
21:43:31 INFO: DeepStream - target: Using the existing boot entry 'primary'
21:43:31 INFO: DeepStream - target: installed nvidia-l4t-kernel package post-installation script subprocess returned error exit status 1
21:43:31 INFO: DeepStream - target: dependency problems - leaving unconfigured
21:43:31 INFO: DeepStream - target: nvidia-l4t-kernel-headers
21:43:34 ERROR: DeepStream - target: E: Sub-process /usr/bin/dpkg returned an error code (1)
21:43:34 ERROR: DeepStream - target: command terminated with error
21:43:34 SUMMARY: DeepStream - target: Installation failed.

I am not quite sure why the error there is “Cannot open /etc/nv_boot_control.conf for reading”.

Can you help me check if this file exists on your drive and what is the content inside?

@WayneWWW this file does not exist /etc/nv_boot_control.conf

Then it means something is missing in your file system over the external drive. How did you prepare your rootfs?

Followed the (NVMe + Xavier NX as example) steps from the link to create and mount rootfs from external device - Jetson/L4T/Boot From External Device - eLinux.org

Because I couldn’t connect the external sda drive (SATA) to my host machine, I directly downloaded L4T Driver Package (BSP) and Sample Root Filesystem to (previously flashed) emmc’s Downloads folder and then followed steps 1.2 - 6 (skipped step 5) of Use NVMe + Xavier NX as example section to create the rootfs, partition (sda1) and then flash the device. For flashing, I used sudo ./flash.sh jetson-xavier-nx-devkit-emmc sda1. After it got flashed emmc had only the boot folder

Do you remember to run apply_binaries.sh?

yes we have run apply_binaries.sh

Can you let sdkmanager download one package for you and you just copy that rootfs on your disk? I mean using the package prepared by sdkm instead of download the BSP and rootfs by yourself. You can also check if the package from sdkm has this nv_boot_control.conf or not.

It should be there.

I am checking the launch of the official Jetson Nano image Ubuntu (with kernel 4.9) from an external USB media. It is enough to edit /boot/extlinux/extlinux.conf after writing the image to USB media (replace APPEND root=/dev/mmcblk0p1 with /dev/sda1) and the system starts up and works fine from USB media. But when I try to start another system (Armbian) with the same kernel 4.9 from USB media, I get a root system mount error. When checking the UART log, I see that the kernel does not use APPEND from extlinux.conf, but uses a variant from u-boot (where root=/dev/mmcblk0p1). I.e. the kernel ignores the value of the variable from extlinux.conf. At the same time, the exact same system with the same extlinux.conf values works fine with a USB with a 5.15\16 kernel and correctly installs the kernel command line. Maybe I’m using something wrong with the official Jetson Nano kernel 4.9 in the configuration or settings?

This is confusing. I have tried to follow the instructions from:
https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/flashing.html#wwpID0E0QN0HA

So this is my setup: I have an eMMC 16GB that boots alright. I have flashed my J NX using the SDK Manager.
Now I want to use the NVME to boot from. Or rather, any of the options is ok, mount file system or booting too. But I cant get the flashing to nvme to work.

I connect the Jetson to a linux 18.04 machine via USB and reboots with 9/10 bridged. I remove it after a while.
Then I have seen that nVidia device appears in lsusb. So far so good. BUT I cant get : sudo BOOTDEV=nvme0n1p1 ./flash.sh --no-flash nvme0n1p1 to work I get : bash: board: No such file or directory. Why? I have created the p1 partition and formated as ext4.

any ideas?

Because you didn’t give the board config to flash.sh as parameters. The parameters are not just “board” string. You need to give the one that you use for flashing.

2 Likes

I figured as much but what is that?
looking in the flash.sh and cant understand the name you look for?
Is it Jetson NX? or jetson_nx ??

sudo BOOTDEV=nvme0n1p1 ./flash.sh --no-flash <board> nvme0n1p1 (The board part was removed due to html-formating)

what should I write instead of “<board>”
Also. Where will I run this? On the Jetson where I have flashed the eMMC and booted or on the Host-machine where I have connected the USB cable

I figured it out. Looking in the tegra folder I found the .config files and figured you talked about those. So I replaced <\board> with jetson-xavier and then it continued. Should mention that in “childs” language that it should be like that. Perhaps an example along-side the instruction would be nice.

Thanks for now…

For clarification

Should mention that in “childs” language that it should be like that.

I mean that the instruction should be easier so that a child can understand. Couldn’t edit my previous answer sorry.
Also that is a bit unclear is when you mention the host computer vs. Jetson. As I understand you have to prepare the nvme on a host linux machine. It says so, but not so straight forward

//M

Hi,

  1. I assume that most people would have some basic ideas about how to use flash.sh. So this is not mentioned inside this post. For someone who didn’t use this tool before, you can follow the quick start guide. Flash the board’s emmc or sdcard first as practice.
    https://developer.nvidia.com/embedded/downloads#?search=quick

  2. Actually sdkmanager is able to directly flash to external NVMe after jeptack4.6 for TX2 series and Xavier series. The post above is mostly for users who are using old jetpack release. The concept is still similar. Just the way to prepare NVMe may not be needed anymore.

2 Likes

Hi, I have used the SDKManager now, but need to run the 4.5.1 Jetpack. Is this what controls the flashing options? Cause it seems I cant select the NVME as target only flash to emmc.

Also. About basic knowledge. Previous installations has been using a flash-memory card and is very straight forward since you can flash the Jetpack linux image directly to the memory card. But for SSD it seems to be more tricky. Sorry to say is that the official guides are quite difficult to follow as well as referencing back and forth. The SDK-manager though is Awsome improvement! It would be very nice if you could add the function to create an image from an existing installation too. So you can “backup” your Jetson. This would greatly improve when I need to deploy multiple installations… The regular tools to create image in linux has not been very good since you need to unmount the drive in which the system runs.

Thanks for the feedback though, this is very much appreciated!

1 Like