How to Boot from USB Drive?

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
,
1

So as of L4T 32.5 and per the Developer Guide it appears that flashing to and booting from USB is now officially supported. This however appears to not work with a basic USB drive so I haven’t even attempted booting from SD card, NVMe, or otherwise. Why is this happening? I would appreciate any guidance or explanation in order to get this working.

Current Setup

  • NVIDIA Jetson AGX Xavier Developer Kit - 32GB
  • L4T 32.5.1 flashed and running on the Jetson’s built-in 32GB eMMC
  • Jetson Operating System: Ubuntu 18.04.5 LTS
  • Jetson Kernel: Linux 4.9.201-tegra
  • Jetson Architecture: arm64
  • Host Operating System: Ubuntu 18.04.5 LTS
  • Host Kernel: Linux 5.4.0-42-generic
  • Host Architecture: x86-64
  • Host JetPack: 4.5.1
  • USB Drive: SanDisk Cruzer Glide 64GB (USB 2.0 Interface)

Flashing to a USB Drive - verbatim from Developer Guide

  • cd /home/hogank/nvidia/nvidia_sdk/JetPack_4.5.1_Linux_JETSON_AGX_XAVIER/Linux_for_Tegra
  • sudo parted /dev/sdb mklabel gpt
  • sudo parted /dev/sdb mkpart APP 0GB 32GB
  • sudo mkfs.ext4 /dev/sdb1
  • Put the Jetson device into recovery mode (middle button ~4 seconds… then combo left button ~4 seconds… then release both)
  • Connect USB-C from host to Jetson (port next to power light)
  • sudo BOOTDEV=sda1 ./flash.sh --no-flash jetson-agx-xavier-devkit sda1
  • sudo mount /dev/sdb1 /mnt
  • sudo mkdir tmp_system
  • sudo mount bootloader/system.img.raw ./tmp_system
  • sudo rsync -axHAWX --numeric-ids --info=progress2 --exclude=/proc ./tmp_system/ /mnt
  • sudo umount /mnt
  • sudo umount ./tmp_system
  • Plug the flash drive into the target device and power it on or reboot it (in this case plugged direct into the Jetson’s eSATA + USB 3.1 Type A combo port between the Ethernet and HDMI)

Results on Serial Console
[0002.204] I> Load in CBoot Boot Options partition and parse it
[0002.214] E> Error -9 when finding node with path /boot-configuration
[0002.214] E> tegrabl_cbo_parse_info: “boot-configuration” not found in CBO fil.
[0002.221] I> Using default boot order
[0002.225] I> boot-dev-order :-
[0002.228] I> 1.sd
[0002.229] I> 2.usb
[0002.231] I> 3.nvme
[0002.233] I> 4.emmc
[0002.235] I> 5.net
[0002.237] I> Hit any key to stop autoboot: 4 3 2 1
[0004.245] initializing target
[0004.245] calling apps_init()
[0004.246] starting app kernel_boot_app
[0004.265] I> found decompressor handler: lz4-legacy
[0004.266] I> decompressing BMP blob …
[0004.277] I> Kernel type = Normal
[0004.277] I> Loading kernel-bootctrl from partition
[0004.278] I> Loading partition kernel-bootctrl at 0xa42d0000 from device(0x1)
[0004.284] W> tegrabl_get_kernel_bootctrl: magic number(0x00000000) is invalid
[0004.285] W> tegrabl_get_kernel_bootctrl: use default dummy boot control data
[0004.285] I> ########## SD (0) boot ##########
[0004.286] I> No sdcard
[0004.287] I> -0 params source =
[0004.291] E> Blockdev open: exit error
[0004.294] E> SD boot failed, err: 724238353
[0004.298] I> ########## USB (0) boot ##########
[0004.307] W> No valid slot number is found in scratch register
[0004.308] W> Return default slot: _a
[0004.324] I> USB Firmware Version: 60.06 release
[0004.380] I> regulator of usb2-0 already enabled
[0004.387] I> regulator of usb2-1 already enabled
[0004.394] I> regulator of usb2-2 already enabled
[0004.402] I> enabling ‘vdd-5v-sata’ regulator
[0005.470] I> USB 2.0 port 4 new high-speed USB device detected
[0005.472] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa92d0000, cmd_ring.dma = 0xa92d0040
[0005.573] I> Start to enumerate device
[0005.574] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa92d0000, cmd_ring.dma = 0xa92d0040
[0005.579] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa92d0000, cmd_ring.dma = 0xa92d0040
[0005.581] I>
[0005.581] I> Enumerated USB Device 0781:5575
[0005.581] I>
[0005.582] I> Max LUN = 0
[0006.586] W> xusbh_wait_irq: Timed out! status = 0x00
[0007.589] W> xusbh_wait_irq: Timed out! status = 0x00
[0007.590] E> Bad CSW SIG (0)!
[0007.691] E> USBMSD: Invalid value CSW SIG 0.
[0007.692] E> INQUIRY returned error 0x7c7c1102 …
[0008.696] W> xusbh_wait_irq: Timed out! status = 0x00
[0008.696] E> Bad CSW SIG (0)!
[0008.797] E> USBMSD: Invalid value CSW SIG 0.
[0009.802] W> xusbh_wait_irq: Timed out! status = 0x00
[0009.802] E> Bad CSW SIG (0)!
[0009.903] E> USBMSD: Invalid value CSW SIG 0.
[0010.908] W> xusbh_wait_irq: Timed out! status = 0x00
[0010.908] E> Bad CSW SIG (0)!
[0011.009] E> USBMSD: Invalid value CSW SIG 0.
[0012.013] W> xusbh_wait_irq: Timed out! status = 0x00
[0012.014] E> Bad CSW SIG (0)!
[0012.115] E> USBMSD: Invalid value CSW SIG 0.
[0012.115] W> TEST UNIT READY returned error 0x7c7c1102, retries @ 5 …
[0012.116] I> -0 params source =
[0012.116] E> Blockdev read block: exit error = e0e0104
[0012.116] E> Blockdev read block: exit error = e0e0104
[0012.117] W> Cannot find any partition table for 00050000
[0012.119] I> Detect filesystem
[0012.122] E> Either offset or requested data size from that offset is beyond device size
[0012.130] E> Blockdev read: exit error = e0e0004
[0012.134] I> fs_detect:160: Failed to read superblock
[0012.139] E> USB boot failed, err: 724238349
[0012.139] I> ########## Fixed storage boot ##########
…and goes on to boot successfully from eMMC beyond this point

But booting from the Jetson’s built-in 32GB eMMC is not my intention. I want to boot from the USB drive with L4T 32.5 installed and running on the USB drive… why is this failing?

2

For those wondering if the USB drive might be bad or if there is some issue with the Jetson’s USB port not functioning… I can say this. The USB port on the Jetson works fine once the Jetson boots up from the built-in eMMC. Also, if I boot up the Jetson and from the Jetson reformat the USB drive (gpt partition table and a single full drive ext4 APP partition) and use rsync to clone the Jetson’s APP partition (on the Jetson’s built-in eMMC) over to the APP partition on my newly formatted USB drive… and then reboot the Jetson… I get slightly better results (at least the USB drive can be recognized now) but still failing as follows.

[0002.204] I> Load in CBoot Boot Options partition and parse it
[0002.214] E> Error -9 when finding node with path /boot-configuration
[0002.214] E> tegrabl_cbo_parse_info: “boot-configuration” not found in CBO fil.
[0002.221] I> Using default boot order
[0002.225] I> boot-dev-order :-
[0002.228] I> 1.sd
[0002.229] I> 2.usb
[0002.231] I> 3.nvme
[0002.233] I> 4.emmc
[0002.235] I> 5.net
[0002.237] I> Hit any key to stop autoboot: 4 3 2 1
[0004.245] initializing target
[0004.245] calling apps_init()
[0004.246] starting app kernel_boot_app
[0004.265] I> found decompressor handler: lz4-legacy
[0004.266] I> decompressing BMP blob …
[0004.277] I> Kernel type = Normal
[0004.277] I> Loading kernel-bootctrl from partition
[0004.278] I> Loading partition kernel-bootctrl at 0xa42d0000 from device(0x1)
[0004.284] W> tegrabl_get_kernel_bootctrl: magic number(0x00000000) is invalid
[0004.285] W> tegrabl_get_kernel_bootctrl: use default dummy boot control data
[0004.285] I> ########## SD (0) boot ##########
[0004.286] I> No sdcard
[0004.287] I> -0 params source =
[0004.291] E> Blockdev open: exit error
[0004.294] E> SD boot failed, err: 724238353
[0004.298] I> ########## USB (0) boot ##########
[0004.307] W> No valid slot number is found in scratch register
[0004.308] W> Return default slot: _a
[0004.324] I> USB Firmware Version: 60.06 release
[0004.380] I> regulator of usb2-0 already enabled
[0004.388] I> regulator of usb2-1 already enabled
[0004.396] I> regulator of usb2-2 already enabled
[0004.407] I> enabling ‘vdd-5v-sata’ regulator
[0005.475] I> USB 2.0 port 4 new high-speed USB device detected
[0005.477] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa92d00000
[0005.578] I> Start to enumerate device
[0005.579] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa92d00000
[0005.584] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa92d00000
[0005.586] I>
[0005.586] I> Enumerated USB Device 0781:5575
[0005.586] I>
[0005.587] I> Max LUN = 0
[0005.591] I> Vendor Identification : SanDisk
[0005.591] I> Product Identification : Cruzer Glide
[0005.594] I> Product Revision Level : 1.27
[0005.603] I> READ CAPACITY - last LBA = 125031680, 61050MB
[0005.604] I> -0 params source =
[0005.629] I> Found 1 partitions in USB_MS (instance 0)
[0005.629] I> Look for boot partition
[0005.630] I> Fallback: assuming 0th partition is boot partition
[0005.636] I> Detect filesystem
[0005.643] I> ext4_mount:588: Failed to allocate memory for group descriptor
[0005.643] E> Failed to mount file system!!
[0005.643] I> Loading kernel …
[0005.644] E> Cannot open partition kernel
[0005.644] E> USB boot failed, err: 202184205
[0005.645] I> ########## Fixed storage boot ##########
…and goes on to boot successfully from eMMC beyond this point

But again… not what I wanted booting from the Jetson’s built-in eMMC… Why is this happening and what am I missing here?

5

Hi hogan.k,

Using the same steps on Xavier-32GB, confirmed we can boot from USB drive success.
Suggest you can try another USB drive or run again. Please connect USB drive to USB port directly.
Below is our UART output:

I> ########## USB (0) boot ##########
W> No valid slot number is found in scratch register
W> Return default slot: _a
I> USB Firmware Version: 60.06 release
I> regulator of usb2-0 already enabled
I> regulator of usb2-1 already enabled
I> regulator of usb2-2 already enabled
I> enabling 'vdd-5v-sata' regulator
I> USB 2.0 port 4 new high-speed USB device detected
W> WARNING: event and command not matching, cmd_trb_ptr = 0xa9ad0000, cmd_ring.dma = 0xa9ad0040
I> Start to enumerate device
W> WARNING: event and command not matching, cmd_trb_ptr = 0xa9ad0000, cmd_ring.dma = 0xa9ad0040
W> WARNING: event and command not matching, cmd_trb_ptr = 0xa9ad0000, cmd_ring.dma = 0xa9ad0040
I> 
I> Enumerated USB Device 0781:5580
I> 
I> Max LUN = 0
I> Vendor Identification   :  SanDisk 
I> Product Identification  :  Extreme         
I> Product Revision Level  :  0001
I> READ CAPACITY - last LBA = 62533296, 30533MB
I> -0 params source = 
I> Found 1 partitions in USB_MS (instance 0)
I> Look for boot partition
I> Fallback: assuming 0th partition is boot partition
I> Detect filesystem
I> Loading extlinux.conf ...
I> rootfs path: /usb/boot/extlinux/extlinux.conf
I> L4T boot options
I> [1]: "primary kernel"
I> Enter choice: 
[0176.404] I> Continuing with default option: 1
[0176.404] I> Loading kernel sig file from rootfs ...
[0176.404] I> rootfs path: /usb/boot/Image.sig
[0176.414] I> Loading kernel binary from rootfs ...
[0176.414] I> rootfs path: /usb/boot/Image
6

Hey @carolyuu thanks for the quick response! Do you mind sharing the partition structure of your USB Drive? Do you only have a single ext4 partition labeled APP? Did you follow the steps exactly from my first message as pulled from the Developer Guide or did you run other commands, copy other files, create additional partitions, change device trees, write other images/kernels, etc?

7

Hi hogank,

My steps is following Developer Guide (the same with your first comment).
I don’t change any file, using default image download from SDKM.
List test steps for you reference:

1. Check the flash drvier's device name:
   $ sudo lsblk -p -d | grep sd
2. Create a new GPT:
   $ sudo parted /dev/<sdx> mklabel gpt
3. Add the APP partition:
   $ sudo parted /dev/<sdx> mkpart APP 0GB 16GB
4. Format APP as ext4 partiton:
   $ sudo mkfs.ext4 /dev/<sdx>1
   $ sudo mount /dev/<sdx>1 /mnt
5. Generate the rootfs without flashing the device:
   $ sudo BOOTDEV=sda1 ./flash.sh --no-flash jetson-xavier sda1
   $ sudo mkdir tmp_system
   $ sudo mount bootloader/system.img.raw ./tmp_system
   $ sudo rsync -axHAWX --numeric-ids --info=progress2 --exclude=/proc ./tmp_system/ /mnt
6. Umount flash driver:
   $ sudo umount /mnt
   $ sudo umount ./tmp_system
7. Plug the flash drive into target device and power on.
8

So @carolyuu I noticed in your flash command that you have “jetson-xavier” listed as your board which I’ve seen in examples elsewhere but which doesn’t appear to be a valid board name listed in the table. Is this a board name accepted in an older version of jetpack? I must confess I have not tried to use it myself yet.

Also, speaking of using the “default image download from SDKM” could you elaborate on this (like where is this and what’s the url). I simply installed the full jetpack deal and flashed the Jetson’s built-in eMMC through the desktop tool (it downloaded all the packages and images it needed and so on). So now after all that I’m running the flash.sh script directly from the folder in order to do things like this USB boot (which has no support in the desktop interface). So did the initial flash of the Jetson and building of the images and all have some effect on the files in the jetpack folder such that they wouldn’t work for USB boot now? Is there a place to download the bootloader/system.img.raw on its own?

Also, I’m still curious as to the partition structure of your USB drive (even locations/offsets and specific sizes) as gpt with a single ext4 APP partition filled with a seemingly normal looking Linux file system doesn’t boot at all on my system and isn’t even recognized unless I wait for the OS to boot up (from the eMMC) and I mount the USB drive partition and poke around… then it looks like it should work, no errors or anything registering for the USB drive or it’s partition and no bad blocks etc

9

HI,

Actually, the board names are defined under Linux_for_Tegra. Thus, if you check the *.conf file there, you shall notice both jetson-xavier and jetson-agx-xavier-devkit link to the same file. Thus, it does not matter which file is in use here.

jetson-xavier.conf → p2822-0000+p2888-0004.conf
jetson-agx-xavier-devkit.conf → p2822-0000+p2888-0004.conf

Also, speaking of using the “default image download from SDKM” could you elaborate on this (like where is this and what’s the url).

As for your question, what @carolyuu said is when you flash with sdkm, it will install Linux_for_Tegra on your host machine. She just used that package to run those commands/steps.

10

Okay so I picked up a fresh USB 3.0 SanDisk Ultra 128GB Flash Drive and still experiencing more or less the same issues. Should also add that I decided to try a new drive (and USB 3.0 instead of 2.0) after the Jetson continuously crashed and restarted itself (logs filled with dozens of “resetting high speed usb device” type errors prior to every crash) when attempting to format the old USB 2.0 drive with GParted.

With the new USB 3.0 drive its still failing to boot with a “Cannot open partition kernel” message. Would this be the file at /boot/Image on the APP partition of the USB drive and if so is there a way to verify that this file isn’t corrupt? Also, just to verify, is the ext4 APP partition the only partition that is needed on the USB drive (as the directions would seem to confirm)? Should we leave extra space on the drive (if and when it ever boots… will it need space for other partitions it may create) or can we use the full drive? Not sure if this would help but the current size of what rsync dumped on the APP partition is 4,916,548 bytes with a breakdown as follows.

11440		/bin
44572		/boot
4			/dev
12504		/etc
4			/home
461672		/lib
16			/lost+found
4			/media
4			/mnt
202108		/opt
4			/README.txt
12			/root
136			/run
11236		/sbin
4			/snap
4			/srv
4			/sys
4			/tmp
3956044		/usr
216772		/var
4916548		total

[0004.294] I> ########## USB (0) boot ##########
[0004.303] W> No valid slot number is found in scratch register
[0004.304] W> Return default slot: _a
[0004.320] I> USB Firmware Version: 60.06 release
[0004.376] I> regulator of usb2-0 already enabled
[0004.384] I> regulator of usb2-1 already enabled
[0004.393] I> regulator of usb2-2 already enabled
[0004.404] I> enabling ‘vdd-5v-sata’ regulator
[0005.473] I> USB 2.0 port 4 new high-speed USB device detected
[0005.475] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa92d00000
[0005.575] I> Start to enumerate device
[0005.577] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa92d00000
[0005.582] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa92d00000
[0005.583] I>
[0005.583] I> Enumerated USB Device 0781:5581
[0005.584] I>
[0005.585] I> Max LUN = 0
[0005.588] I> Vendor Identification : USB
[0005.589] I> Product Identification : SanDisk 3.2Gen1
[0005.591] I> Product Revision Level : 1.00
[0005.601] I> READ CAPACITY - last LBA = 240353280, 117360MB
[0005.601] I> -0 params source =
[0005.614] I> Found 1 partitions in USB_MS (instance 0)
[0005.615] I> Look for boot partition
[0005.615] I> Fallback: assuming 0th partition is boot partition
[0005.624] I> Detect filesystem
[0005.631] I> ext4_mount:588: Failed to allocate memory for group descriptor
[0005.631] E> Failed to mount file system!!
[0005.632] I> Loading kernel …
[0005.635] E> Cannot open partition kernel
[0005.639] E> USB boot failed, err: 202184205
[0005.643] I> ########## Fixed storage boot ##########

11

The exact same message also appears (shown below) when I rsync the filesystem from the Jetson itself and then modify the /boot/extlinux/extlinux.conf to root=/dev/sda1. What I’m saying is I booted up the Jetson and plugged the USB drive directly into the Jetson’s USB port and rsync cloned it’s filesystem (APP partition on the Jetson’s internal eMMC). So what is going on here??? Am I missing something here or are there additional steps required to get the Jetson to boot from USB that have been left out???

[0004.293] I> ########## USB (0) boot ##########
[0004.303] W> No valid slot number is found in scratch register
[0004.303] W> Return default slot: _a
[0004.319] I> USB Firmware Version: 60.06 release
[0004.375] I> regulator of usb2-0 already enabled
[0004.383] I> regulator of usb2-1 already enabled
[0004.390] I> regulator of usb2-2 already enabled
[0004.400] I> enabling ‘vdd-5v-sata’ regulator
[0005.468] I> USB 2.0 port 4 new high-speed USB device detected
[0005.469] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa92d0000, cmd_ring.dma = 0xa92d0040
[0005.570] I> Start to enumerate device
[0005.572] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa92d0000, cmd_ring.dma = 0xa92d0040
[0005.576] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa92d0000, cmd_ring.dma = 0xa92d0040
[0005.578] I>
[0005.578] I> Enumerated USB Device 0781:5581
[0005.579] I>
[0005.580] I> Max LUN = 0
[0005.583] I> Vendor Identification : USB
[0005.583] I> Product Identification : SanDisk 3.2Gen1
[0005.586] I> Product Revision Level : 1.00
[0005.596] I> READ CAPACITY - last LBA = 240353280, 117360MB
[0005.596] I> -0 params source =
[0005.608] I> Found 1 partitions in USB_MS (instance 0)
[0005.609] I> Look for boot partition
[0005.609] I> Fallback: assuming 0th partition is boot partition
[0005.619] I> Detect filesystem
[0005.626] I> ext4_mount:588: Failed to allocate memory for group descriptor
[0005.626] E> Failed to mount file system!!
[0005.627] I> Loading kernel …
[0005.630] E> Cannot open partition kernel
[0005.634] E> USB boot failed, err: 202184205
[0005.638] I> ########## Fixed storage boot ##########

12

@WayneWWW and @carolyuu I want to include you both to make sure the documentation and future guidance can be corrected. I have managed to successfully move the root file system to a USB drive and then also to an NVMe drive. The key steps that were missing in the docs, in advice here, and elsewhere was to… at the very end… connect the drive to the Jetson, switch the Jetson to recovery mode, connect the Jetson to your host, and run the following command on your host.

sudo ./flash.sh jetson-agx-xavier-devkit sda1
or
sudo ./flash.sh jetson-agx-xavier-devkit nvme0n1p1

This is all after already cloning the root file system to your drive… from literally anywhere (Linux_for_Tegra/rootfs or Linux_for_Tegra/bootloader/system.img.raw or even just rsync-ing your live root file system from the built-in eMMC while the Jetson is booted up)… and making sure /boot/extlinux/extlinux.conf on the drive has root=/dev/sda1 or root=/dev/nvme0n1p1 instead of root=/dev/mmcblk0p1

As it stands I don’t see any way to just stick your root file system on a USB drive, NVMe, or otherwise… and just reboot… as the Developer Guide states. You have to run the command to make the Jetson target the USB/NVMe as its new root file system drive or you will never be able to use it… as your root file system drive. Honestly I think this whole area of the docs on Flashing and Booting and the CBoot sections need to be completely revamped as they are misleading and riddled with errors. We need to clearly distinguish between “booting from” (completely removing dependence on the built-in eMMC… still a little fuzzy on the boot process but I think this is impossible) and simply “changing your root file system target” (which is all that I did so that I can have space to my heart’s content without ugly hacks). I hope this can serve to help others on their journey to experiment with an otherwise epic platform.

Possibly the most important point of all that I almost forgot. Ignore the errors in the USB section at boot (like those shown in my previous messages) as this seems to have nothing to do with your root file system being mounted correctly. Only after booting up and running lsblk was I shocked to discover that my root file system had in fact mounted from my USB drive despite all the USB errors that seemingly skipped the drive altogether.

Also equally important to note is that you must run this command from your host (with Jetson connected in recovery mode of course) every time you want to change the source/drive of your root file system. You might think you can just pop that USB out and it would boot from the built-in eMMC… but you’d be wrong… boot just stalls out at some point because if you examine the built-in eMMC’s APP partition now you’ll see that it only contains the boot folder and nothing else. Admittedly having to switch the root file system source/drive will be a rare task but still… no fallbacks… come on…

13

Hi,

Let me make a summary before we start this long discussion.
→ Actually, I think you just took another way to avoid the old issue you had. Thus, it seems has nothing to do with document.
I want to find out why the issue existing on your case, could you attach the last boot up log here? Need to consult with some bootloader experts to see what would cause below problem.

005.614] I> Found 1 partitions in USB_MS (instance 0)
[0005.615] I> Look for boot partition
[0005.615] I> Fallback: assuming 0th partition is boot partition
[0005.624] I> Detect filesystem
[0005.631] I> ext4_mount:588: Failed to allocate memory for group descriptor
[0005.631] E> Failed to mount file system!!

Ok, then back to what I want to say.

Since I don’t have the full log for your last comment, I can only guess that your last method here is to only mount the rootfs from usb drive. But your kernel and extlinux.conf are still on emmc. According to your comment, looks like you already know what is the difference between “boot from” and “mount rootfs to”.

In fact, I also posted this wiki page below to guide other users. Actually, this method is still “mount rootfs”. While where the platform “boots from” depends on the boot-order of uboot or cboot.

https://elinux.org/Jetson/L4T/Boot_From_External_Device

For non-Xavier soc platform(TX1/TX2/Nano), the boot flow is cboot → uboot → kernel,
while for Xavier-series (NX/ AGX Xavier), the boot flow is cboot->kernel. (no uboot)

And the capability of whether a bootloader is able to boot from each interface differs between every SoC too… You can see we have a list in the document. TX2 is excluded from this supports.

image
image1154×112 14.8 KB

That is why I wrote the wiki page to only teach others how to “mount rootfs” because it does not depend on the platform, but just purely the ability of kernel. For example, TX2 is able to mount rootfs on usb storage but cannot “boot from” usb.

In your log, you can see there is SD boot → usb boot → fixed storage boot (emmc). This is the boot order. If the cboot finds out file kernel/extlinux.conf on external storage during this scan out, it will boot the kernel from that storage. As for which rootfs to mount, it depends on the extlinux.conf. If the extlinux.conf tells you to boot from usb and then still use the rootfs on emmc, I think it would still work. Though I didn’t try that before.

What @carolyuu did is both boot from usb drive and also mount the rootfs on usb drive. Thus, your error is unexpected to us so need further investigation.

2 Likes
14

Hey @WayneWWW this really makes a lot of sense with the distinction of “mount rootfs” and “boots from” and your summary of the process. I can see now that although its really nice having the rootfs on a bigger disk, updating the kernel down the road could introduce some issues. I’ve attached the boot log from when I had my USB drive attached directly to the Jetson with an exact mirror of my NVMe drive (only difference on the USB drive is in /boot/extlinux/extlinux.conf where root=/dev/sda1 instead of root=/dev/nvme0n1p1 or root=/dev/mmcblk0p1). The NVMe drive ultimately had its rootfs mounted instead of the USB drive and I’m assuming the kernel on the eMMC was booted. Thanks for taking a look into this. I would definitely rather have the kernel and rootfs on the same partition (“boots from” + “mount rootfs” team-up).

bootlog.txt (906.8 KB)

1 Like
15

Hi,

Sorry for misleading. The log I need is the uart log which includes the cboot log.

16

Totally my bad, grabbed the wrong log. Attached should be the one you’re after. Thanks again for looking into this @WayneWWW

uartlog.txt (98.4 KB)

17

Was reading through a thread over here and made me think… based on my review of the uart logs and your explanation I believe the /boot/extlinux/extlinux.conf file is being read from the Jetson’s built in eMMC (and not from the USB or NVMe drive like I want). That being said, what would happen if the file was changed so that linux and initrd both pointed to the NVMe like below? Would this work or is the /dev/nvme0n1p1 not accessible at this point? Also wondering now if this is what the /boot/extlinux/extlinux.conf file on the NVMe is also supposed to look like in order to properly locate the kernel and ramdisk?

LABEL primary
MENU LABEL primary kernel
LINUX /dev/nvme0n1p1/boot/Image
INITRD /dev/nvme0n1p1/boot/initrd
APPEND ${cbootargs} quiet root=/dev/nvme0n1p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4

18

Just tested this using the USB drive and it doesn’t have any effect. Still fails the USB section with “Cannot open partition kernel” and moves on to “boot from” the Jetson’s built-in eMMC (kernel) and mounts the root file system (rootfs) from the NVMe.

19

Just successfully updated the kernel, bootloader, device tree, and a whole bunch of other stuff following the advice in the response below. Simply hit “Reboot later” after all the updates installed, then copied the /boot on the NVMe to the /boot on the Jetson’s built-in eMMC, and then rebooted the Jetson. Obviously this is not the most ideal path and I’m still looking forward to us figuring out the boot issues so that the /boot folder (kernel, ramdisk, device tree, and so on) that’s actually booted from can live on the NVMe as well.

20

Okay now things are getting really interesting. I just updated the kernel (and other stuff) and then mirrored to the built-in eMMC as stated in my previous message. So basically the eMMC and the NVMe are both totally updated. Now I decided to stick in the USB drive (old/non-updated kernel and old/non-updated everything else) and reboot the Jetson to see if any of the new updates had fixed the USB problems (unlikely I guess since CBoot lives elsewhere and I think is the issue?). So the boot process starts and the USB section once again shows the standard “Cannot open partition kernel” but to my surprise as the boot continues the system continuously crashes with “Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00”. If I unplug the USB drive while CBoot is counting down to autoboot then the system boots up as normal (“boots from” eMMC and “mounts rootfs” from NVMe). I attached the UART log from when I allowed it to panic once and then pulled the USB and let it boot successfully. I would expect this if I was trying to mount the rootfs from the USB and the kernel didn’t match the built-in eMMC… which I’m not even doing (I’m mounting the rootfs from the NVMe). I’m even more confused as to why this is happening since the USB section appears to fail as if the USB is skipped. So why does a USB (that supposedly has no partition kernel) cause a kernel panic???

uartlog_2021032101.txt (122.7 KB)

21

Well… I just checked your uartlog.txt posted yesterday from #15.

This is the first time you posting a full log here, so I didn’t notice this before.

Even when the cboot tries to read the extlinux.conf from your emmc, there is a same error as usb boot case. If such case happens, our cboot will initiate a fallback mechanism to read kernel from partition. That is, during the flash process (by flash.sh), our tool not only installs the /boot to your rootfs but also flash a backup kernel into specific partition. When boots up fails, it will use the kernel partition to boot instead.

[0005.637] I> ########## Fixed storage boot ##########
[0005.642] I> Already published: 00010003
[0005.646] I> Look for boot partition
[0005.649] I> Fallback: assuming 0th partition is boot partition
[0005.655] I> Detect filesystem
[0005.670] I> ext4_mount:588: Failed to allocate memory for group descriptor
[0005.671] E> Failed to mount file system!!
[0005.671] E> Invalid fm_handle (0xa06964a8) or mount path passed
[0005.675] I> Fallback: Load binaries from partition
[0005.679] W> No valid slot number is found in scratch register
[0005.685] W> Return default slot: _a
[0005.689] I> A/B: bin_type (37) slot 0
[0005.692] I> Loading kernel from partition

I think that explains why your kernel update method does not work when you try to update it in built-in emmc… because the boot process does not read it at all.

Even in your latest fine boot result from nvme drive. The kernel is still from the partition.

[0005.637] I> ########## Fixed storage boot ##########
[0005.642] I> Already published: 00010003
[0005.646] I> Look for boot partition
[0005.649] I> Fallback: assuming 0th partition is boot partition
[0005.655] I> Detect filesystem
[0005.670] I> ext4_mount:588: Failed to allocate memory for group descriptor
[0005.671] E> Failed to mount file system!!
[0005.671] E> Invalid fm_handle (0xa06964a8) or mount path passed
[0005.675] I> Fallback: Load binaries from partition
[0005.679] W> No valid slot number is found in scratch register
[0005.685] W> Return default slot: _a

Thus, I think we should firstly check why even the emmc boot fails from the beginning… Is it a pure image from sdkmanager?

22

Yep everything on the eMMC drive was direct from the sdkmanager.

  • While I was getting the host set up I had booted up the Jetson just to poke around (so at that point it had the L4T that came pre-installed)
  • Then I flashed the Jetson from the host using the desktop UI and ran the Jetson like that for a bit while contemplating storage issues and “modern” software spreading all over the file system
  • Then I cloned the eMMC APP partition to several other drives (UFS cards, USB drives, and NVMe) while testing booting (but never altered the eMMC)
  • Then yesterday I used the flash utility from the host to set the rootfs to the NVMe (which as a byproduct wiped my eMMC except for the boot folder and even that was slimmed down as compared to my NVMe clone of the original /boot folder) and I didn’t mention this before but I noticed after the flash (that switched the rootfs target to the NVMe) that my /boot/Image and /boot/initrd were no longer the same from the eMMC to the NVMe (although similar in size diff revealed that they were not identical anymore)
  • Then finally today I updated the kernel, bootloader, and so on through the software update utility and then cloned the boot directory from the NVMe to the eMMC (altering it from its original sdkmanager flashed form)