How to build/rebuild U-Boot to boot from USB

dear readers,

i’m scavenging the Nvidia form on how to boot from USB for 2 weeks. As far i can understand so far i have to go out of my comfort-zone and take a deep, real deep dive in to the assembly of the Jetson TX2. It feels like i can’t see the forest trough the trees anymore.

How can i boot form a USB? i would be very grateful if you can explain the steps that i have to take.

i will list all the post i have consulted:
i also consulted JetsonHacks:

what i think my next step will be is:

  1. find a driver for USB2.0/3.0
  2. build custom U-boot package
  3. flash to Jetson TX2
  4. change the extlinux.conf in /boot/extlinux/ on the eMMC
  5. reboot the TX2

where can i find a driver for u-boot?
how do i flash the costum u-boot file whit the rest of JetPack3.1?

if more information is required plz. let me know.

ps: sorry for my English

If your USB port stays in the same mode (same speed, e.g., USB2 mode) during both U-Boot stage and Linux operation, then no boot loader change is needed. If you operate Linux in USB3 mode, then that port cannot be used for booting from unless you either downgrade it to USB2 mode, or else install a USB3 driver in U-Boot (U-Boot uses a generic USB2 driver with no awareness of Tegra hardware).

I don’t know if the code for USB3 (tegra-xhci) is all available for porting to U-Boot or not…I haven’t looked to see if it provided in binary format or source code. You are guaranteed that setting up to run in USB2 mode will work without this complication…not many hard drives are actually fast enough to exceed USB2 speeds without getting into either RAID or perhaps SSDs. What is your USB drive type…fast SSD? Old style platters? RAID? If you didn’t answer “yes” to one of those, then the only down side to forcing USB2 mode is that other devices on that port will also drop back to USB2 speed. On the other hand, if you had a device needing USB3 bandwidth, then you pretty much eliminated using the same port for your root partition as reasonable…it’d be “a bad idea”.

Please give more information on your USB drive relative to drive type and performance. Also give more information on whether you need USB3.

@linuxdev tank you for reacting on my post.

i got a Samsung portable SSD T3 it has a USB3.1 to USB-C. i also have a a Toshiba HDD whit USB3 and a Kingston 64GB flash USB whit USB3.0. it looks like the new stander for USB appliances is USB3. i have red a post ware indeed the solution was to force the USB port to stay USB2. speed is not the most imported thing we need at the moment. our prediction is that the processing of data is a bigger bottleneck than the R/W speed of the USB connection.

reading you comment linexdev, i’m starting to think that it is better to use the PCI lane or the on-board SATA connector. but to be honest we don’t have the time to change. the use of a USB connected drive is a little bit of a quick and dirty solution to our storage problem. but looking back it may had been better if we had used a SD or internal SSD. but that is further not imported.

what is the best way to go from here?

if i limit the USB port to USB2 is there a downside to it other than speed?
how do i limit the USB port?
is it still relevant/beneficial to look for does USB3 (tegra-xhci) binaries?

In what follows I am assuming R28.1. Earlier releases had much in common, but you may see differences with regard to how USB speed is configured. At the end of this I also have a question which NVIDIA will need to answer.

Just FYI, USB3.0 is supported, though not in U-Boot…and USB3.1 is not supported, even in Linux…so USB3.0 would be a speed cap if you were not booting from the drive. Since you are booting from the drive you’ll have to verify it works with USB2 speeds. “lsusb -t” lists speeds of devices where “480M” implies USB2, and “5000M” implies USB3. Other than lower speed I do not believe there is any downside to storage over USB. Some USB3 devices need their full speed and do not have a USB2 compatibility mode, but hard drives do not have this limitation (some USB3 high resolution cameras do not provide a USB2 mode).

If you connect your drive and it shows as 5000M after a normal boot, then expect to lose the drive as a root partition until the port is forced back to USB2 mode.

USB3 in U-Boot would be difficult.

For SD card boot you would not need to worry about any of those restrictions.

Do you have serial console? This will make life much less painful while experimenting. U-Boot has a selection menu where you can pick which boot entry you want and have test entries such that a failure means you just boot the original entry instead…without serial console many failed tests would require a flash to boot again.

Some Information About Boot Targets…

When you flash a Jetson there are many hidden partitions going into the eMMC. Regardless of where your boot configuration is found (extlinux.conf), and regardless of where your root partition goes, the hidden partitions will remain on eMMC.

mmcblk0p1 is the first partition of eMMC, mmcblk1p1 is the first partition of SD card, sda1 is the first partition of the first hard drive. These are possible choices for rootfs, and also for where extlinux.conf is searched for. It is not mandatory that extlinux.conf and rootfs be on the same partition. I recommend keeping it simple and doing a normal flash to eMMC, and then editing extlinux.conf so that after configuration is read Linux will use the USB drive for rootfs.

Think closely about this: If you name mmcblk0p1 as a flash target, then both rootfs and extlinux.conf go on eMMC…but an edit to this extlinux.conf can cause the Linux kernel to boot to the USB drive. You can add a second entry in extlinux.conf (versus editing the existing entry), and suddenly serial console can be used to pick which one you want to boot to. It’s like having a built in rescue system. You can have a separate entry for eMMC, SD card, and sda1…all with the edit of a file and no flash required.

If you choose to flash to mmcblk1p1 (SD card) instead of mmcblk0p1 (eMMC), then the rootfs will still go to this alternate device as expected…what you won’t necessarily expect is that extlinux.conf must now be on the SD card or other media…eMMC will no longer have an extlinux.conf which matters (and in fact everything on the “/boot” of any file system other than the one used as the flash target will be ignored). If you remove the SD card or other media which had extlinux.conf, then the system will no longer be able to find configuration and will no longer boot. You can play around with the environment in U-Boot and have failover behavior, but this probably isn’t what you want. Just flash to mmcblk0p1 and then edit extlinux.conf for the simplest life. Make your rootfs on sda1 or mmcblk1p1 at a later time. If you want to you can make one of these external media a default entry, and then use serial console only when you want to try something new.

So for example this might be your “/boot/extlinux/extlinux.conf” on eMMC:

DEFAULT primary

MENU TITLE p2771-0000 eMMC boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4

LABEL sdcard
      MENU LABEL sdcard mmcblk1p1
      LINUX /boot/Image
      APPEND ${cbootargs} root=/dev/mmcblk1p1 rw rootwait rootfstype=ext4

LABEL sda1
      MENU LABEL sda1
      LINUX /boot/Image
      APPEND ${cbootargs} root=/dev/sda1 rw rootwait rootfstype=ext4

If you interrupt boot early on you get to the U-Boot command line…this is too early…simply type “boot” to continue…then very quickly you get a kernel selection menu…hit any key (quickly) to stop boot from continuing. Enter the number of the entry. In the above you’d be able to pick “2” for SD card, or “3” for SATA. Beware that whatever key you hit to stop boot might still be there, so you might need to hit the backspace before entering the number.

Everything else is about creating the rootfs or setting up USB. If you flashed to mmcblk0p1, then you can do all of this at a later date.

About External Rootfs…

A typical procedure is to create an ext4 partition on the first partition of the USB drive. I did this by extracting Tegra_Linux_Sample-Root-Filesystem_R28.1.0_aarch64.tbz2 there (it was ext4 formatted and extraction done with sudo), then using the R28.1 on it while it was mounted on “/mnt” of my host:

sudo ./ -r /mnt

This puts everything there except for “/boot” content…if you flashed to mmcblk0p1, then you don’t need “/boot” content.

You could also clone an existing install and copy to this partition from the clone.

USB Settings…

Without USB mode edits you will get part way through boot, and then sda1 will fail. This is an artifact of switching modes…you need sda1 to get the rootfs, and you need to lose the rootfs to get a new USB mode…a proverbial “chicken and the egg” dilemma.

This is somewhat more problematic on R28.1 since some of the settings were taken out of the kernel command line and instead put in the device tree. Device tree itself is somewhat more complicated because as of R28.1 the dtb isn’t provided in a file accessed by name in extlinux.conf…this is now in partitions (which as mentioned before will be on eMMC).

Mostly people ask how to make sure USB3 is enabled, but you need the inverse. Would someone at NVIDIA comment on what part of the device tree to change to cause the USB3 port to run only at USB2 mode? Without this booting on a USB device will fail.

@linuxdev your are so much help, tank you.

Your are give me a proper view on my challenge. Your last post is almost the answer. i agree it’s something where Nvidia has to look into. I’m very interested in what Nvidia has to tell us. A simple refrains to the R28.1 manual will not suffice.

My next action plan is:

  1. trying an old USB2 stick
  2. ordering a SD card(hopefully we stay in budget and it's delivered in time)
  3. giving you credits in our report
  4. await Nvidia's answer

Tank you for your help so far.

A USB stick is interchangeable with a USB SATA drive if you use the first partition (it’ll be “/dev/sda1” either way). It’d still be a case of needing to know what device tree changes are needed to keep the USB3 port in USB2 mode when the Linux kernel loads (or vice verse…porting a USB3 driver to U-Boot so kernel and boot loader use the same setup without reset of the USB controller causing loss of the root device).

I know this is an old thread but I’m also looking to boot the TX2 from a USB stick. This is required for firmware upgrade ie boot from USB then upgrade the firmware on the eMMC. I realise it’s simpler to use an SD Card but a USB stick would be so much easier for client field upgrades.
Can anyone tell me if it’s possible and, if so, what the steps are to implement it?
I am using L4T R28.2.1.
Many thanks

I have not kept up with the list of all drivers in U-Boot, but the gist is that the driver must exist in both Linux and U-Boot if you are to boot from that device. Linux pretty much has everything, but U-Boot may be limited.

In particular I know U-Boot has had a SATA over USB driver, but not SATA over PCI…thus a PCIe controller of a disk would fail, but an external USB SATA drive would work. Now it turns out that U-Boot has only USB2 drivers, and not USB3…so if you were to boot to USB SATA with USB2 speeds, and then Linux switches to USB3, it will fail (continuity of service without resetting the controller is required). In that case the Linux USB port would have to be USB2 only and USB3 (within Linux) could not be used for this.

Some of the steps for naming boot configuration have complicated this. In the past U-Boot used the extlinux.conf file for its configuration, but more recent releases have placed some of that content (including many arguments to U-Boot) within a partition via the device tree “chosen” entry. Some of this still works, but I have not sorted through what is or isn’t passed along via extlinux.conf. However, I think there is still something which might help you…

If you are in serial console you can printenv and see a lot of variables. Many of those are used as macros, and one of them is to search for extlinux.conf on various media; if nothing optional is found, it uses the eMMC version. It is possible that the current configuration might work if you just put a “rescue” or alternate boot version on a USB stick while USB is configured to remain USB2 and not USB3. One of the things I am not positive about on newer releases is if USB2 or USB3 can be chosen via extlinux.conf or not (just for the Linux stage…U-Boot will already be USB2).

Caution one for testing: 64-bit extensions are not allowed in the ext4 file system within U-Boot (Linux doesn’t care). The Ubuntu flavors all work by default, so if you have an Ubuntu host, you don’t need to worry about it, and this includes both the Jetson and the PC if you use this for formatting a partition.

The macros will look for the first partition on a given device, and thus if your USB stick has partitions using gdisk (newer UEFS style partitions…fdisk is just older BIOS style), and you put a test rescue system on the first partition which has the “/boot/extlinux/extlinux.conf” file, then you’ll know if U-Boot can handle this. As a test you don’t even need any content other than ext4 with that file…if it stops booting with that plugged in, then you know U-Boot sees it and you won’t need special steps for U-Boot (you could then add your rescue or clone to that USB stick partition, but you might still need to use the device tree to force the port to USB2 mode once in Linux…I can’t confirm if that is required via device tree or not on recent releases).

Thanks, Linuxdev.
In fact I don’t even get to first base here. U-boot fails to start usb ie.

Tegra186 (P2771-0000-500) # usb start
starting USB…
No controllers found

I assume this is because my Jetson TX2 Developer Kit has only USB3 hardware, not USB2.

It has been suggested that we could add USB3 support to u-boot, but I’m not sure how realistic this is.

Alternatively, are you saying above that it is possible to configure the usb3 hardware as usb2, via extlinux.conf?

What would you recommend?

No, the dev kit has USB2, and a port will route to the USB2 or USB3 (xusb) driver/controller depending on which is configured and what connects. I don’t remember how to actually set up the USB2 controller, but if you go to the U-Boot command prompt and look at the “printenv” output (I recommend logging from gtkterm and keeping a copy of printenv), then notice this as one of the boot alternatives:

Tegra186 (P2771-0000-500) # printenv usb_boot
usb_boot=usb start; if usb dev ${devnum}; then setenv devtype usb; run scan_dev_for_boot_part; fi
Tegra186 (P2771-0000-500) # printenv scan_dev_for_boot_part
scan_dev_for_boot_part=part list ${devtype} ${devnum} -bootable devplist; env exists devplist || setenv devplist $defaultdevplist; for distro_bootpart in ${devplist}; do if fstype ${devtype} ${devnum}:${distro_bootpart} bootfstype; then run scan_dev_for_boot; fi; done
Tegra186 (P2771-0000-500) # printenv scan_dev_for_boot     
scan_dev_for_boot=echo Scanning ${devtype} ${devnum}:${distro_bootpart}...; for prefix in ${boot_prefixes}; do run scan_dev_for_extlinux; run scan_dev_for_scripts; done;run scan_dev_for_efi;
Tegra186 (P2771-0000-500) # printenv scan_dev_for_efi 
scan_dev_for_efi=setenv efi_fdtfile ${fdtfile}; for prefix in ${efi_dtb_prefixes}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${efi_fdtfile}; then run load_efi_dtb; fi;done;if test -e ${devtype} ${devnum}:${distro_bootpart} efi/boot/bootaa64.efi; then echo Found EFI removable media binary efi/boot/bootaa64.efi; run boot_efi_binary; echo EFI LOAD FAILED: continuing...; fi; setenv efi_fdtfile

I don’t have an external USB hard drive I can test right now (nor even a thumb drive I can afford to wipe out), but if you use gdisk to put a partition on a USB disk (without any 64-bit extensions…all Ubuntu hosts are ok and do not add those extensions), and then format it as ext4, you should be able to put some files into this which show if boot is working or not. The boot doesn’t even have to succeed, all it has to do is change and you’ll know USB SATA works. It is possible different drivers are involved (since it is U-Boot and not Linux) for USB sticks and SATA drives (I don’t know…both are USB Mass Storage, but there may be differences I don’t know about).

At minimum the device would have to have one partition…the script looks for the first partition, and if you allocate a disk as a whole there won’t be a first partition. At minimum it must be ext4 without 64-bit extensions…U-Boot can’t read 64-bit extensions. At minimum the “/boot/” content must be complete…U-Boot looks for extlinux.conf. If you have an extlinux.conf, then for testing the “mmcblk0p1” should be edited (on the external drive only) to point at “sda1” instead (I forget if it needs “/dev/sda1” as prefix for U-Boot) instead of “mmcblk0p1” (a serial console log of a successful boot to eMMC could tell you which extlinux.conf was found, but it is a lot easier to see what is going on if the extlinux.conf actually points at the SATA device (then run “lsblk -f” after boot and see which device is “/”).

Note: If a USB SATA drive starts in U-Boot, then Linux will fail if it tries to switch to USB3. There may have also been changes in U-Boot since I last tried this, I won’t guarantee the drivers are the same since earlier releases.

Many thanks, Linuxdev, I can confirm your printenv results above.

I have a USB stick on which I have created a single ext4 partition and it mounts to /dev/sda1 when linux is running. However, from u-boot, I can’t see the USB stick. “usb start” fails with the message “No controllers found”.

Surely we need to be able to access the USB stick from u-boot before we can boot from it?

In the past I’ve accessed a USB SATA drive for boot, but it has been a long time. The main trick was that once in Linux the USB mode had to be kept as USB2. There have been issues in seeing the USB driver from U-Boot even back then (despite something in boot working…meaning manual use of USB commands in the U-Boot prompt didn’t always work as expected despite USB working).

What I’m going to suggest is that you create a clone of your existing rootfs, edit the “/boot/extlinux/extlinux.conf” to only point to “/dev/sda1” as the root, and then put the clone onto the USB stick (you could edit extlinux.conf after copy to the stick). There are a number of ways to go about doing so, but beware that an ordinary file copy probably won’t work unless you add a few options to guarantee numeric user ID (ask if you run into an issue doing an ordinary copy). My favorite is to create a partition the same size as the clone, and then use dd to overwrite the partition with the clone at the correct offset (you can test offset with a few bytes of read before you try to write).

I have never booted from a USB stick, but since the first partition would be “sda1” (assuming no other SATA device or USB stick is involved), it should be the same as a USB enclosure of a SATA hard drive.

One thing I worry about is that some of the device tree is now used in early boot stages (this is the part which changed since the last time I booted to a USB SATA drive). That device tree is passed on to U-Boot. One entry in the tree is called “chosen”, and this is the command line to the Linux kernel after edits…what you put in extlinux.conf is not always honored since that change went in. On a newer running system (but not on earlier releases) you can “cd /proc/device-tree/chosen/”, and then “cat bootargs”. The “root=/dev/mmcblk0p1” (the first partition of the eMMC) might override extlinux.conf, but I’m not sure in what way (it would depend on the order in which boot macros scan for devices versus when the device tree kicks in).

If you have a valid ext4 (without 64-bit extensions), then it should be possible for the macros to find and use that version of extlinux.conf…and if that version uses rootfs of “/dev/sda1”, then it should boot to the USB thumb drive. Even if the system hangs it would prove it is doing as you want. Once you’ve booted you can see what actually provides rootfs via “lsblk -f” (which lists mount point).

Just a point of emphasis on 64-bit extensions to ext4: Linux doesn’t care, it is U-Boot which cannot find partitions when those extensions are involved. Ubuntu systems do not use those extensions unless you’ve intentionally set them. Many of the Linux distributions also do not use this, but a few do.

As for needing USB to see the device and thus U-Boot needing a driver you are 100% correct. However, I do recall some time in the past where I was unable to manually load the driver, but actually booting from USB SATA auto loaded this driver correctly. Perhaps the driver has been left out in newer releases. In that case U-Boot would need to be custom compiled, but it is far easier to test with the above clone style copy prior to figuring out customization of U-Boot.

Thanks again, linuxdev,
I have tried your suggestion of cloning onto a usb drive, it doesn’t work for me, although the same procedure on an SD card does work. This leads me to believe it’s not possible on a TX 2 without uBoot changes.
Can anyone out there confirm this for me?
Has anyone managed to get a TX2 to boot from a USB stick/drive?

Hi all,

I referred and try below method on TX2.

But I still meet TX2 boot up issue from USB stick/drive.
Please refer this attachment for boot-up log.
Could anybody help me to fix this question?
Thanks for your help.

teraterm_usb_boot.log (67.9 KB)

Boot was normal up until trying to find “/dev/sda1”. As mentioned in earlier posts there may be a need to make changes to U-Boot if sda1 is to be visible over USB during U-Boot, but in this case configuration was found (presumably on eMMC), and then later attempts to find sda1 failed. In your particular case how is sda1 connected? USB external disk?

If the disk is connected via USB and you do not attempt to boot to the disk, and simply observe the disk after boot, what do you see from:

lsblk -f | egrep '(NAME|sda|blk1p1)'