Boot from external drive

Hello,

I understand that there are lots of people asking about how to boot from external drive (e.g. usb/NVMe drive). And also lots of methods all around this forum. Thanks for everyone to share their steps to achieve that.

This post is just to give some basic concepts about how jetson platform boots from external drive. And answer some frequently-asked questions.
I believe every method shared on this forum uses the same idea but just in different forms.

1. Jetson is not able to “fully” boot from external drive.

What jetson can do is boot “initrd/kernel” or “mount file system” from your external drive. It means you cannot just format the boot device and leave it empty. There must be bootloader components in your boot device so that the boot process can use the bootloader from it. For sdcard nano or NX, this boot device is QSPI on the module. As for other platforms, it is the emmc on the module.

2. You have to know the difference between “boot kernel from external drive” and "mount rootfs from external drive

  • Boot initrd/kernel from external drive means the uboot (Nano/TX2 series) or cboot (NX/AGX) will search the kernel and extlinux.conf from your external drive according to the boot order. Once it finds the kernel and extlinux.conf, it will use the info from it to mount the file system.

  • Mount the file system means kernel will mount the file system according to your “root=” in kernel command line.
    For example, the most common usage is root=/dev/mmcblk0p1 which will boot from sdcard (nx/nano) and emmc.

→ Thus, if you just want to increase your disk space, you don’t really need to boot kernel from external drive, you just make sure the file system is mounted from external drive.
→ It means there are lots of variety that you can use. For example, you can boot kernel from usb and mount the file system from NVMe.

3. How to do?
We provide steps inside the developer guide link for booting both kernel and mount rootfs from the same drive.
Please be careful that every platform shares different bootloader capability. For example, TX2 uboot does not support usb .

If you just want to mount the file system from external, then you can check this page.
I am sorry that the title of this page is not precise. It is just mounting the rootfs from external. Not booting from anything.
https://elinux.org/Jetson/L4T/Boot_From_External_Device

4. flash.sh will not flash anything to your external drive.
There were some users asking about why “sudo ./flash.sh … sda1” does not flash data to their usb drive. The truth is flash.sh currently only has capability to flash data to native sdcard (nano/nx) and emmc boot device. What this “sudo ./flash.sh … sda1” doing is just adding “root=/dev/sda1/” to the kernel command line. Which means this is only telling kernel to mount the file system from sda1. Nothing else.

5. Can I create a USB drive that is similar to the sdcard image on Nano/NX?
Theoretically, flashing the sdcard image to a USB drive can work. Just in case the bootloader(uboot/cboot) is able to find your usb first, but not some other external drives. Also, you have to make sure the root device in kernel command line is not setting to something like mmcblk0p1. Otherwise, your file system will not be mounted correctly.

6. Official OTA upgrade does not support boot from external
Your “boot from external” setting would be gone after you try the apt upgrade. Because our OTA debian package by default only supports the boot from default path (emmc/sd).

5 Likes

Thanks @WayneWWW for sharing all the steps and details for booting Jetson NX from external drive. This is very helpful.

We were able to boot from SATA drive but run into a problem while installing multimedia and deepstream packages in the external drive. Can you please provide the steps for the same?

Also, we are unable to run sudo apt upgrade to upgrade the packages and also unable to install any new packages e.g. python3-pip etc. This might be due to the issue you have referred to the last point. Is there a work around for the same?

Error log:

:~$ sudo apt install python3-pip
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  dh-python libpython3-dev libpython3.6-dev python3-dev python3-setuptools python3-wheel python3.6-dev
Suggested packages:
  python-setuptools-doc
The following NEW packages will be installed:
  dh-python libpython3-dev libpython3.6-dev python3-dev python3-pip python3-setuptools python3-wheel python3.6-dev
0 upgraded, 8 newly installed, 0 to remove and 0 not upgraded.
6 not fully installed or removed.
Need to get 46.1 MB of archives.
After this operation, 78.8 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://ports.ubuntu.com/ubuntu-ports bionic/main arm64 dh-python all 3.20180325ubuntu2 [89.2 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 libpython3.6-dev arm64 3.6.9-1~18.04ubuntu1.4 [45.1 MB]
Get:3 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 libpython3-dev arm64 3.6.7-1~18.04 [7,328 B]                            
Get:4 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 python3.6-dev arm64 3.6.9-1~18.04ubuntu1.4 [508 kB]                     
Get:5 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 python3-dev arm64 3.6.7-1~18.04 [1,288 B]                               
Get:6 http://ports.ubuntu.com/ubuntu-ports bionic-updates/universe arm64 python3-pip all 9.0.1-2.3~ubuntu1.18.04.5 [114 kB]                  
Get:7 http://ports.ubuntu.com/ubuntu-ports bionic/main arm64 python3-setuptools all 39.0.1-2 [248 kB]                                        
Get:8 http://ports.ubuntu.com/ubuntu-ports bionic/universe arm64 python3-wheel all 0.30.0-0.2 [36.5 kB]                                      
Fetched 46.1 MB in 16s (2,921 kB/s)                                                                                                          
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package dh-python.
(Reading database ... 173372 files and directories currently installed.)
Preparing to unpack .../0-dh-python_3.20180325ubuntu2_all.deb ...
Unpacking dh-python (3.20180325ubuntu2) ...
Selecting previously unselected package libpython3.6-dev:arm64.
Preparing to unpack .../1-libpython3.6-dev_3.6.9-1~18.04ubuntu1.4_arm64.deb ...
Unpacking libpython3.6-dev:arm64 (3.6.9-1~18.04ubuntu1.4) ...
Selecting previously unselected package libpython3-dev:arm64.
Preparing to unpack .../2-libpython3-dev_3.6.7-1~18.04_arm64.deb ...
Unpacking libpython3-dev:arm64 (3.6.7-1~18.04) ...
Selecting previously unselected package python3.6-dev.
Preparing to unpack .../3-python3.6-dev_3.6.9-1~18.04ubuntu1.4_arm64.deb ...
Unpacking python3.6-dev (3.6.9-1~18.04ubuntu1.4) ...
Selecting previously unselected package python3-dev.
Preparing to unpack .../4-python3-dev_3.6.7-1~18.04_arm64.deb ...
Unpacking python3-dev (3.6.7-1~18.04) ...
Selecting previously unselected package python3-pip.
Preparing to unpack .../5-python3-pip_9.0.1-2.3~ubuntu1.18.04.5_all.deb ...
Unpacking python3-pip (9.0.1-2.3~ubuntu1.18.04.5) ...
Selecting previously unselected package python3-setuptools.
Preparing to unpack .../6-python3-setuptools_39.0.1-2_all.deb ...
Unpacking python3-setuptools (39.0.1-2) ...
Selecting previously unselected package python3-wheel.
Preparing to unpack .../7-python3-wheel_0.30.0-0.2_all.deb ...
Unpacking python3-wheel (0.30.0-0.2) ...
Setting up nvidia-l4t-bootloader (32.5.1-20210614115015) ...
ERROR. Cannot open /etc/nv_boot_control.conf for reading.
Cannot install package. Exiting...
dpkg: error processing package nvidia-l4t-bootloader (--configure):
 installed nvidia-l4t-bootloader package post-installation script subprocess returned error exit status 1
Setting up nvidia-l4t-xusb-firmware (32.5.1-20210614115015) ...
ERROR. Cannot open /etc/nv_boot_control.conf for reading.
Cannot install package. Exiting...
dpkg: error processing package nvidia-l4t-xusb-firmware (--configure):
 installed nvidia-l4t-xusb-firmware package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of nvidia-l4t-initrd:
 nvidia-l4t-initrd depends on nvidia-l4t-xusb-firmware (= 32.5.1-20210614115015); however:
  Package nvidia-l4t-xusb-firmware is not configured yet.dpkg: error processing package nvidia-l4t-initrd (--configure):
 dependency problems - leaving unconfigured
No apport report written because the error message indicates its a followup error from a previous failure.
                                                                                                          Setting up python3-wheel (0.30.0-0.2) ...
Setting up libpython3.6-dev:arm64 (3.6.9-1~18.04ubuntu1.4) ...
Setting up python3-pip (9.0.1-2.3~ubuntu1.18.04.5) ...
Setting up nvidia-l4t-kernel (4.9.201-tegra-32.5.1-20210505093723) ...
Using the existing boot entry 'primary'
ERROR. Cannot open /etc/nv_boot_control.conf for reading.
Cannot install package. Exiting...
dpkg: error processing package nvidia-l4t-kernel (--configure):
 installed nvidia-l4t-kernel package post-installation script subprocess returned error exit status 1
No apport report written because MaxReports is reached already
                                                              Setting up python3-setuptools (39.0.1-2) ...
Setting up python3.6-dev (3.6.9-1~18.04ubuntu1.4) ...
Setting up dh-python (3.20180325ubuntu2) ...
Setting up libpython3-dev:arm64 (3.6.7-1~18.04) ...
dpkg: dependency problems prevent configuration of nvidia-l4t-kernel-dtbs:
 nvidia-l4t-kernel-dtbs depends on nvidia-l4t-kernel (= 4.9.201-tegra-32.5.1-20210505093723); however:
  Package nvidia-l4t-kernel is not configured yet.dpkg: error processing package nvidia-l4t-kernel-dtbs (--configure):
 dependency problems - leaving unconfigured
Setting up python3-dev (3.6.7-1~18.04) ...
No apport report written because MaxReports is reached already
                                                              dpkg: dependency problems prevent configuration of nvidia-l4t-kernel-headers:
 nvidia-l4t-kernel-headers depends on nvidia-l4t-kernel (= 4.9.201-tegra-32.5.1-20210505093723); however:
  Package nvidia-l4t-kernel is not configured yet.dpkg: error processing package nvidia-l4t-kernel-headers (--configure):
 dependency problems - leaving unconfigured
No apport report written because MaxReports is reached already
                                                              Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Errors were encountered while processing:
 nvidia-l4t-bootloader
 nvidia-l4t-xusb-firmware
 nvidia-l4t-initrd
 nvidia-l4t-kernel
 nvidia-l4t-kernel-dtbs
 nvidia-l4t-kernel-headers
E: Sub-process /usr/bin/dpkg returned an error code (1)

Error log while flashing Multimedia and Deepstream:

21:43:23 ERROR: Multimedia API - target: E: Unable to correct problems, you have held broken packages.
21:43:23 ERROR: Multimedia API - target: command terminated with error
21:43:23 SUMMARY: Multimedia API - target: First Error: APT system is broken and requires manual fix
21:43:23 DEBUG: running command < true >
21:43:23 INFO: command finished successfully
21:43:23 DEBUG: DeepStream - target: using adapter to install NV_DEEPSTREAM_TARGET_POST_INSTALL_COMP@JETSON_XAVIER_NX to /home/ubuntu-dell/nvidia/nvidia_sdk/JetPack_4.5.1_Linux_JETSON_XAVIER_NX
21:43:23 INFO: DeepStream - target: exec_command: ssh -t -q -o PreferredAuthentications=password -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no localhost@192.168.55.1 "df -B1 / | tail -1 | awk '{ printf \$4 }'"
21:43:23 INFO: DeepStream - target: [target] localhost@192.168.55.1's password:
21:43:24 INFO: DeepStream - target: current working directory is /home/ubuntu-dell/nvidia/nvidia_sdk/JetPack_4.5.1_Linux_JETSON_XAVIER_NX
21:43:24 INFO: DeepStream - target: **********************
21:43:24 INFO: DeepStream - target: exec_command: scp -q -o PreferredAuthentications=password -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no /tmp/tmp_NV_DEEPSTREAM_TARGET_POST_INSTALL_COMP.sh localhost@192.168.55.1:~; ssh -t -q -o PreferredAuthentications=password -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no localhost@192.168.55.1 "set -o pipefail; ~/tmp_NV_DEEPSTREAM_TARGET_POST_INSTALL_COMP.sh | sed 's/^/SDKM_INSTALL_ERROR /' && rm -f ~/tmp_NV_DEEPSTREAM_TARGET_POST_INSTALL_COMP.sh"
21:43:24 INFO: DeepStream - target: [target] localhost@192.168.55.1's password:
21:43:24 INFO: DeepStream - target: [target] localhost@192.168.55.1's password:
21:43:24 INFO: DeepStream - target: *begin sudo apt-get update, 5
21:43:24 INFO: DeepStream - target: [target] [sudo] password for localhost:
21:43:25 INFO: DeepStream - target: 0% [Working] Hit:1 https://repo.download.nvidia.com/jetson/common r32.5 InRelease
21:43:25 INFO: DeepStream - target: 0% [Connecting to ports.ubuntu.com (91.189.91.38)] 0% [1 InRelease gpgv 2,552 B] [Connecting to ports.ubuntu.com (91.189.91.38)] [ Hit:2 https://repo.download.nvidia.com/jetson/t194 r32.5 InRelease
21:43:25 INFO: DeepStream - target: 0% [1 InRelease gpgv 2,552 B] [Connecting to ports.ubuntu.com (91.189.91.39)] 0% [Waiting for headers] 0% [2 InRelease gpgv 2,562 B] [Waiting for headers] Hit:3 http://ports.ubuntu.com/ubuntu-ports bionic InRelease
21:43:26 INFO: DeepStream - target: 0% [2 InRelease gpgv 2,562 B] 0% [Waiting for headers] 0% [3 InRelease gpgv 242 kB] [Waiting for headers] 0% [Waiting for headers] Hit:4 http://ports.ubuntu.com/ubuntu-ports bionic-updates InRelease
21:43:26 INFO: DeepStream - target: 0% [Working] 0% [4 InRelease gpgv 88.7 kB] [Waiting for headers] 0% [Waiting for headers] Hit:5 http://ports.ubuntu.com/ubuntu-ports bionic-backports InRelease
21:43:26 INFO: DeepStream - target: 0% [Working] 0% [5 InRelease gpgv 74.6 kB] [Waiting for headers] 0% [Waiting for headers] Hit:6 http://ports.ubuntu.com/ubuntu-ports bionic-security InRelease
21:43:28 INFO: DeepStream - target: 0% [Working] 0% [6 InRelease gpgv 88.7 kB] 20% [Working] Reading package lists... 0% Reading package lists... 0% Reading package lists... 0% Reading package lists... 4% Reading package lists... 4% Reading package lists... 6% Reading package lists... 6% Reading package lists... 6% Reading package lists... 6% Reading package lists... 6% Reading package lists... 6% Reading package lists... 33% Reading package lists... 41% Reading package lists... 41% Reading package lists... 61% Reading package lists... 61% Reading package lists... 61% Reading package lists... 61% Reading package lists... 61% Reading package lists... 61% Reading package lists... 66% Reading package lists... 66% Reading package lists... 72% Reading package lists... 72% Reading package lists... 72% Reading package lists... 72% Reading package lists... 73% Reading package lists... 73% Reading package lists... 79% Reading package lists... 79% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 82% Reading package lists... 86% Reading package lists... 86% Reading package lists... 91% Reading package lists... 91% Reading package lists... 91% Reading package lists... 91% Reading package lists... 92% Reading package lists... 92% Reading package lists... 96% Reading package lists... 96% Reading package lists... 98% Reading package lists... 98% Reading package lists... 98% Reading package lists... 98% Reading package lists... 98% Reading package lists... 98% Reading package lists... 98% Reading package lists... 98% Reading package lists... 98% Reading package lists... 98% Reading package lists... Done
21:43:28 INFO: DeepStream - target: *begin sudo apt-get install -y libssl1.0.0 libgstreamer1.0-0 libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev gstreamer1.0-tools gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav libgstrtspserver-1.0-0 libjansson4 librdkafka1=0.11.3-1build1 , 5
21:43:29 INFO: DeepStream - target: Reading package lists... 0% Reading package lists... 100% Reading package lists... Done
21:43:29 INFO: DeepStream - target: Building dependency tree... 0% Building dependency tree... 0% Building dependency tree... 50% Building dependency tree... 50% Building dependency tree
21:43:29 INFO: DeepStream - target: Reading state information... 0% Reading state information... 0% Reading state information... Done
21:43:29 INFO: DeepStream - target: libgstrtspserver-1.0-0 is already the newest version (1.14.5-0ubuntu1~18.04.1).
21:43:30 INFO: DeepStream - target: 6 not fully installed or removed.
21:43:30 INFO: DeepStream - target: After this operation, 0 B of additional disk space will be used.
21:43:31 INFO: DeepStream - target: Setting up nvidia-l4t-bootloader (32.5.1-20210614115015) ...
21:43:31 INFO: DeepStream - target: Setting up nvidia-l4t-xusb-firmware (32.5.1-20210614115015) ...
21:43:31 INFO: DeepStream - target: Setting up nvidia-l4t-kernel (4.9.201-tegra-32.5.1-20210505093723) ...
21:43:31 INFO: DeepStream - target: Using the existing boot entry 'primary'
21:43:31 INFO: DeepStream - target: installed nvidia-l4t-kernel package post-installation script subprocess returned error exit status 1
21:43:31 INFO: DeepStream - target: dependency problems - leaving unconfigured
21:43:31 INFO: DeepStream - target: nvidia-l4t-kernel-headers
21:43:34 ERROR: DeepStream - target: E: Sub-process /usr/bin/dpkg returned an error code (1)
21:43:34 ERROR: DeepStream - target: command terminated with error
21:43:34 SUMMARY: DeepStream - target: Installation failed.

I am not quite sure why the error there is “Cannot open /etc/nv_boot_control.conf for reading”.

Can you help me check if this file exists on your drive and what is the content inside?

@WayneWWW this file does not exist /etc/nv_boot_control.conf

Then it means something is missing in your file system over the external drive. How did you prepare your rootfs?

Followed the (NVMe + Xavier NX as example) steps from the link to create and mount rootfs from external device - Jetson/L4T/Boot From External Device - eLinux.org

Because I couldn’t connect the external sda drive (SATA) to my host machine, I directly downloaded L4T Driver Package (BSP) and Sample Root Filesystem to (previously flashed) emmc’s Downloads folder and then followed steps 1.2 - 6 (skipped step 5) of Use NVMe + Xavier NX as example section to create the rootfs, partition (sda1) and then flash the device. For flashing, I used sudo ./flash.sh jetson-xavier-nx-devkit-emmc sda1. After it got flashed emmc had only the boot folder

Do you remember to run apply_binaries.sh?

yes we have run apply_binaries.sh

Can you let sdkmanager download one package for you and you just copy that rootfs on your disk? I mean using the package prepared by sdkm instead of download the BSP and rootfs by yourself. You can also check if the package from sdkm has this nv_boot_control.conf or not.

It should be there.