Initrd external USB flash fails

[bsp_dev]

Hi,
I try to flash external USB drive connected to my Orin Nano devkit which hosting an Orin NX SoM.

1. I connected it to the devkit and format it with 1 ext4 partition (/dev/sda1)
2. I used

sudo fdisk -l /dev/sda

to check what is the num_sectors and modified the partitions layout accordingy.
layout file:
flash_l4t_external_usb.xml (9.7 KB)

2. I tried to flash it with the following command (workflow 4 in README_initird_flash.txt):

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device sda1 -c tools/kernel_flash/flash_l4t_external_usb.xml -p “-c bootloader/t186ref/cfg/flash_t234_qspi.xml --no-systemimg” --network usb0 --showlogs --network usb0 jetson-orin-nano-devkit external > flash_usb.log 2>&1

flash log:
flash_usb.log (151.2 KB)

serial output log:
uart.log (16.2 KB)

Thanks

[DaveYYY]

Hi,

I just tried with a 32GB SD card (also modifying sector size and added the -S option) and it also failed.
We will check it internally, but please use a 64GB or larger USB drive now.

[bsp_dev]

Unfortunately I don’t have one right now (only 32GB). I’ll wait for your answer by your permission.

[DaveYYY]

Hi,

I tried with -S 25GiB and failed, but it worked with -S 20GiB.
Can you give it a try?

[DaveYYY]

I got this on my device:

nvidia@tegra-ubuntu:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk1p1   29G  5.6G   22G  21% /
none            3.6G     0  3.6G   0% /dev
tmpfs           3.7G   36K  3.7G   1% /dev/shm
tmpfs           748M   11M  737M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           3.7G     0  3.7G   0% /sys/fs/cgroup
tmpfs           748M   88K  748M   1% /run/user/1000
nvidia@tegra-ubuntu:~$ cat /etc/nv_tegra_release 
# R35 (release), REVISION: 4.1, GCID: 33958178, BOARD: t186ref, EABI: aarch64, DATE: Tue Aug  1 19:57:35 UTC 2023

[bsp_dev]

From this print, your device booted with the rootFS mounted on the nvme and not from the USB drive. Is this correct?

[bsp_dev]

I used the -S 20GiB flag and flashed successfully, but even if i choose to boot from usb manually via uefi, the system boots from nvme like seen in your picture.

[DaveYYY]

@bsp_dev

From this print, your device booted with the rootFS mounted on the nvme and not from the USB drive. Is this correct?

No, I got it from the SD card.

@bsp_dev

I used the -S 20GiB flag and flashed successfully, but even if i choose to boot from usb manually via uefi, the system boots from nvme like seen in your picture.

Choosing boot options in UEFI only controls where the system reads the extlinux.conf and loads the kernel image, but what to mount as the rootfs is still controlled by the kernel command line in extlinux.conf. So please check rootfs=xxx inside extlinux.conf on both your NVMe drive and USB drive, and make sure they point to where they are from respectively.

For example,

rootfs=/dev/sda1

for the config file on the USB drive,

rootfs=/dev/nvme0n1p1

for the config file on the NVMe drive.

[bsp_dev]

So it turns out that both /dev/nvme0n1p1 and /dev/sda1 have the same PARTUUID and that appears in the extlinux file (rootfs=[PARTUUID]). The question is how it happen?

[DaveYYY]

The content of the kernel command line is determined by the following diagram inside Linux_for_Tegra/tools/kernel_flash/README_initrd_flash.txt:

Initrd flash depends on --external-device options and the last parameter <rootdev>
to generate the correct images. The following combinations are supported:
+-------------------+-----------------+-------------------------------------------------------+
| --external-device |       <rootdev> | Results                                               |
+-------------------+-----------------+-------------------------------------------------------+
| nvme*n*p* / sda*  |        internal | External device contains full root filesystem with    |
|                   |                 | kernel commandline: rootfs=PARTUUID=<external-uuid>   |
|                   |                 |                                                       |
|                   |                 | Internal device contains full root filesystem with    |
|                   |                 | kernel commandline: rootfs=PARTUUID=<internal-uuid>   |
+-------------------+-----------------+-------------------------------------------------------+
| nvme*n*p* / sda*  | nvme0n*p* / sd* | External device  contains full root filesystem with   |
|                   |                 | with kernel commandline rootfs=/dev/nvme0n1p1         |
|                   |                 |                                                       |
|                   |                 | Internal device contains minimal filesystem with      |
|                   |                 | kernel command line rootfs=/dev/nvme0n1p1             |
+-------------------+-----------------+-------------------------------------------------------+
| nvme*n*p* / sda*  |       mmcblk0p1 | External device  contains full root filesystem with   |
|                   |                 | with kernel commandline rootfs=/dev/nvme0n1p1         |
|                   |                 |                                                       |
|                   |                 | Internal device contains full filesystem with         |
|                   |                 | kernel command line rootfs=/dev/mmcblk0p1             |
+-------------------+-----------------+-------------------------------------------------------+
| nvme*n*p* / sda*  |        external | External device contains full root filesystem with    |
|                   |                 | kernel commandline: rootfs=PARTUUID=<external-uuid>   |
|                   |                 |                                                       |
|                   |                 | Internal device contains minimal root filesystem with |
|                   |                 | kernel commandline: rootfs=PARTUUID=<external-uuid>   |
+-------------------+-----------------+-------------------------------------------------------+

Hence your command falls into category 4, and results in rootfs=PARTUUID=<external-uuid> on both storage devices.
Just to make sure, do you get to boot from the SD card after modifying rootfs=xxx?

[bsp_dev]

@BSP_User

I don’t use sdcard, only nvme and usb.

Sorry, I just mean USB drive.
So does rootfs=/dev/sda1 work?

@BSP_User

Thank you for the table but I didn’t understand why both device have the same partuuid? does the string is not unique? do I generate it with my command?

That’s because you used jetson-orin-nano-devkit external in your flashing command. It’s just that kernel image is loaded from the USB drive, but the whole rootfs is from the NVMe drive.

[DaveYYY]

Oh sorry, I just accidentally edited your comment instead of making a new one.

[bsp_dev]

1. yes it works

2. so the command should be:

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device sda1 -c tools/kernel_flash/flash_l4t_external_usb.xml -S 20GiB -p “-c bootloader/t186ref/cfg/flash_t234_qspi.xml --no-systemimg” --network usb0 --showlogs --network usb0 jetson-orin-nano-devkit /dev/sda1
?

3. what is the difference between “full root filesystem” and “minimal root filesystem” and if i have both (one in internal storage and one in external) which of them will be used for boot?

[DaveYYY]

@bsp_dev

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device sda1 -c tools/kernel_flash/flash_l4t_external_usb.xml -S 20GiB -p “-c bootloader/t186ref/cfg/flash_t234_qspi.xml --no-systemimg” --network usb0 --showlogs --network usb0 jetson-orin-nano-devkit /dev/sda1
?

Maybe, I guess. I haven’t verified it but you can give it a try.
Also, sda1 instead of /dev/sda1.

@bsp_dev

what is the difference between “full root filesystem” and “minimal root filesystem” and if i have both (one in internal storage and one in external) which of them will be used for boot?

The minimal one is just like the initrd used for flashing external devices, only kernel and other necessary stuff like device drivers, and nothing else. The full one is just what you get normally after flashing. Again, the device to boot from (from which the kernel image and extlinux.conf is read) is determined by UEFI, and is not the same as what to mount as the rootfs, which is determined by rootfs=/dev/xxx in extlinux.conf found on the storage device determined in the previous step.

[bsp_dev]

I tried flashing, as you said, with this:

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device sda1 -c tools/kernel_flash/flash_l4t_external_usb.xml -S 20GiB -p “-c bootloader/t186ref/cfg/flash_t234_qspi.xml --no-systemimg” --network usb0 --showlogs --network usb0 jetson-orin-nano-devkit sda1

It didn’t work. Can you help me with the suitable command for not getting the same partuuid?

[DaveYYY]

So the flashing process was fine, but you still got rootfs= pointing to the NVMe drive?
I’m not sure if such use case is considered in the script, as both of your storage devices are external ones, so if manually changing it to rootfs=/dev/sda1 after flashing works, then just take it.

[bsp_dev]

I’ll verify it one last time and if that works I’ll take it as it is. I’ll be able to update here in about 2 weeks from now

[kayccc]

Is this still an issue to support? Any result can be shared? Thanks

[bsp_dev]

I can confirm the following sequence works:

1. flash the 32GB usb drive using parted and create 1 partition
2. flash with initrd method and the -S 20GiB flag
3. Manually modify extlinux.conf: root=/dev/sda1

Thank you for your support