Flashing separate kernels/rootfs under the same bootloader

[wrogers]

I am attempting to specify multiple kernels on a single bootloader, and then have those kernels each with their own partition that they can have a unique rootfs on.

I have edited the extlinux.conf to add a backup kernel, and then specified a separate partition for each:

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      FDT /boot/dtb/kernel_tegra234-p3701-0005-p3737-0000.dtb
      INITRD /boot/initrd
      APPEND ${cbootargs} root=/dev/mmcblk0p2 rw rootwait rootfstype=ext4

LABEL previous
      MENU LABEL previous kernel
      LINUX /boot/Image-previous
      FDT /boot/dtb/kernel_tegra234-p3701-0005-p3737-0000.dtb
      INITRD /boot/initrd
      APPEND ${cbootargs} root=/dev/mmcblk0p3 rw rootwait rootfstype=ext4

I have a few questions:

1. Is this possible to flash two separate and different rootfs at flashing time to the different partitions?
2. Will the common bootloader files e.g. /boot be placed automatically in mmcblk0p1 in this config?
3. Is it possible to have both the primary and previous kernel share another partition (lets say mmcblk0p4) for certain common rootfs directories (e.g. /home)?

I am aware of rootfs A/B redundancy support, but this is tied to separate bootloaders. My requirements dictate that I have more redundancy than A/B bootloaders, and separate kernel and rootfs redundancy needs to be present for both bootloaders.

[DaveYYY]

@wrogers

Is this possible to flash two separate and different rootfs at flashing time to the different partitions?

NO.
You may only be able to do it after flashing with something like cloning the partition.

@wrogers

Will the common bootloader files e.g. /boot be placed automatically in mmcblk0p1 in this config?

I don’t know what you are referring to here.
What do you mean with be placed automatically?

@wrogers

Is it possible to have both the primary and previous kernel share another partition (lets say mmcblk0p4) for certain common rootfs directories (e.g. /home)?

I think modifying /etc/fstab should do it.

[wrogers]

Thank you for your response.

@DaveYYY

What do you mean with be placed automatically?

In the extlinux.conf file I am specifying each kernel to boot from a different partition. Based on your response, I would first need to flash to mmcblk0p2 and then clone to mmcblk0p3 so both kernels have a bootable rootfs. I assume this means that the boot files such as the kernel images in /boot will be taken from mmcblk0p2 or whatever the partition that was flashed? Or does it require a copy of the /boot directory in each partition to be able to boot?

[DaveYYY]

Hi,

Loading the kernel image and mounting the rootfs are two different things. Where to load the kernel image/dtb is determined in the UEFI boot menu, and it’s totally fine that the partition you read these stuff from is different from the one you want to mount as the rootfs.

For example, in the UEFI boot menu, you can boot from mmcblk0p1, and in the extlinux.conf in that partition, you can add two entries stating that you are going to mount mmcblk0p2 and mmcblk0p3 as the rootfs.

So ideally, you don’t need kernel image in partitions that is only meant to be used as rootfs, but I’d suggest using dd to clone the whole partition so you won’t miss anything.

[wrogers]

@DaveYYY

For example, in the UEFI boot menu, you can boot from mmcblk0p1

1. In that case would mmcblk0p1 be the EFI partition, or is that separate?
2. How would I be able to boot from mmcblk0p1 instead of going through the boot menu? Is there an option in the UEFI source?

[linuxdev]

Normally I see the UEFI VFAT partition as the first partition. Until you mentioned this I never really thought about alternate locations, but on my AGX Orin I see the VFAT exists at “mmcblk0p10”. Jetsons always use the first partition in eMMC as the rootfs. It would seem something has been customized, and I can’t really answer your question, but if you run the command “lsblk -f”, do you see “vfat” associated with mmcblk0p10? If so, then perhaps this is where your content goes, while mmcblk0p1 remains rootfs (or one of them).

[DaveYYY]

@wrogers

In that case would mmcblk0p1 be the EFI partition, or is that separate?

I mean you can load the kernel and device tree from a partition that is different from the one you want to mount as the rootfs. I’m not referring to the EFI partition here.

@wrogers

How would I be able to boot from mmcblk0p1 instead of going through the boot menu? Is there an option in the UEFI source?

I think you will only see bootable devices instead of partitions in UEFI boot menu. So mmcblk0p1 would be used for loading kernel here.

[wrogers]

@linuxdev

if you run the command “lsblk -f”, do you see “vfat” associated with mmcblk0p10

Yes this seems to be the default for eMMC based on the config for the orin devkit.

It looks like the EFI partition is specified sequentially in the default flash_t234_qspi_sdmmc.xml config so it ends up as mmcblk0p10.

<partition name="esp" type="data">
            <allocation_policy> sequential </allocation_policy>
            <filesystem_type> basic </filesystem_type>
            <size> 67108864 </size>
            <file_system_attribute> 0 </file_system_attribute>
            <allocation_attribute> 0x8 </allocation_attribute>
            <percent_reserved> 0 </percent_reserved>
            <filename> ESP_FILE </filename>
            <partition_type_guid> C12A7328-F81F-11D2-BA4B-00A0C93EC93B </partition_type_guid>
            <description> **Required.** EFI system partition with L4T Launcher. </description>
        </partition>

I guess I never really noticed that this file specifies the partition layout and is placing a bunch of the images and dtb files for the bootloader and UEFI all in different partitions.

@DaveYYY

Where to load the kernel image/dtb is determined in the UEFI boot menu

I am now a little confused as to how the Orin knows where the bootloader image and UEFI dtb files are located and what partitions to access as specified in that flash_t234_qspi_sdmmc.xml file.

See gparted:

I suppose I would need to specify the partitions in this xml file before flashing, to allow for space for my extra rootfs and a shared partition.

However, if I do so and make a rootfs on mmcblk0p2, which is normally the bootloader image partition, will the Orin still find the bootloader image if moved to mmcblk0p3?

Also the A_kernel partition looks like it stores the bootloader image. Can you confirm that despite the name, it’s the bootloader and not the kernel image stored there?

@DaveYYY

I think you will only see bootable devices instead of partitions in UEFI boot menu. So mmcblk0p1 would be used for loading kernel here.

What would be the bare minimum file structure for mmcblk0p1 that the UEFI would show as bootable so that I could then mount to another partition? For example, I would assume /boot and /etc at the very least.

[DaveYYY]

@wrogers

I suppose I would need to specify the partitions in this xml file before flashing, to allow for space for my extra rootfs and a shared partition.

However, if I do so and make a rootfs on mmcblk0p2, which is normally the bootloader image partition, will the Orin still find the bootloader image if moved to mmcblk0p3?

I think these partitions are read by UEFI with UUID instead of fixed name like mmcblk0px, so it won’t cause the issue.

@wrogers

Also the A_kernel partition looks like it stores the bootloader image. Can you confirm that despite the name, it’s the bootloader and not the kernel image stored there?

NO.
I’m sure it’s kernel instead of bootloader, but it’s rarely used because kernel/kernel-dtb are read from rootfs by default, instead of from dedicated partitions.
This is configurable in the UEFI boot menu.

@wrogers

What would be the bare minimum file structure for mmcblk0p1 that the UEFI would show as bootable so that I could then mount to another partition? For example, I would assume /boot and /etc at the very least.

If you just want to save disk space, then the basic flavor rootfs should do it.
https://docs.nvidia.com/jetson/archives/r35.4.1/DeveloperGuide/text/SD/RootFileSystem.html#basic-flavor-root-file-system

[wrogers]

@DaveYYY

NO.
I’m sure it’s kernel instead of bootloader, but it’s rarely used because kernel/kernel-dtb are read from rootfs by default, instead of from dedicated partitions.
This is configurable in the UEFI boot menu.

I assumed this was the bootloader image based on Partition Configuration — Jetson Linux Developer Guide documentation.
The A_kernel partition is by default specified with LNXFILE as the file name which according to the doc translates to boot.img.

So then to confirm, the bootloaders are stored and read from the 64MiB QPSI NOR not any partition on the eMMC, and the kernel images/ kernel dtb are read from the rootfs (APP) partition, not the A_kernel or whatever those random partitions specified in the config are?
If so, I can assume I can delete those extra partitions and only keep required ones like ESP?

@DaveYYY

If you just want to save disk space, then the basic flavor rootfs should do it.
Root File System — Jetson Linux Developer Guide documentation

I suppose I was more asking about what file structure is the minimum for the bootloader to consider a bootable rootfs if all I am using it for is directing it to mount to another partition. So you are saying that I would need a full linux file system, but I can minimize it of course by removing unnecessary modules and files like the basic rootfs flavor does?

Also, going back to the original redundancy request, I modified the partitions on the eMMC to test booting a kernel with a rootfs on another partition:

I cloned the original APP partition (mmcblk0p1) to mmcblk0p16. I then modified the extlinux.conf file to use mmcblk0p16 as the rootfs.

TIMEOUT 30
DEFAULT primary

MENU TITLE L4T boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      FDT /boot/dtb/kernel_tegra234-p3701-0005-p3737-0000.dtb
      INITRD /boot/initrd
      APPEND ${cbootargs} root=/dev/mmcblk0p16 rw rootwait rootfstype=ext4 mminit_loglevel=4 console=ttyTCU0,115200 console=ttyAMA0,115200 console=tty0 firmware_class.path=/etc/firmware fbcon=map:0 net.ifnames=0 nospectre_bhb 

I rebooted, but the original APP partition (mmcblk0p1) is still being used. Shouldn’t the bootloader find mmcblk0p1, try to boot from it, but then it reads extlinux.conf which redirects the rootfs mount point to be mmcblk0p16? Any ideas why this does not work?

[DaveYYY]

@wrogers

So then to confirm, the bootloaders are stored and read from the 64MiB QPSI NOR not any partition on the eMMC, and the kernel images/ kernel dtb are read from the rootfs (APP) partition, not the A_kernel or whatever those random partitions specified in the config are?
If so, I can assume I can delete those extra partitions and only keep required ones like ESP?

Supposedly.

@wrogers

I suppose I was more asking about what file structure is the minimum for the bootloader to consider a bootable rootfs if all I am using it for is directing it to mount to another partition. So you are saying that I would need a full linux file system, but I can minimize it of course by removing unnecessary modules and files like the basic rootfs flavor does?

I don’t think we would suggest users modifying the file system itself or have verified your use case, so you may need to figure out it yourself.

@wrogers

I rebooted, but the original APP partition (mmcblk0p1) is still being used. Shouldn’t the bootloader find mmcblk0p1, try to boot from it, but then it reads extlinux.conf which redirects the rootfs mount point to be mmcblk0p16? Any ideas why this does not work?

How did you confirm it failed to load mmcblk0p16?

[DaveYYY]

Based on your requirement, I do think you should consider enabling A/B redundancy:
https://docs.nvidia.com/jetson/archives/r35.4.1/DeveloperGuide/text/SD/RootFileSystem.html#root-file-system-redundancy

, instead of trying some random stuff and ended up asking why it would not work.

[wrogers]

@DaveYYY

How did you confirm it failed to load mmcblk0p16?

I placed a file on mmcblk0p1 “test.txt” to identify it. When I rebooted, the “test.txt” file was still there so it loaded mmcblk0p1.
I was able to use fstab to mount the partition on a separate directory, but trying to mount mmcblk0p16 to root either using fstab or extlinux.conf doesn’t work.

@DaveYYY

Based on your requirement, I do think you should consider enabling A/B redundancy:

Look at my original post:

@wrogers

I am aware of rootfs A/B redundancy support, but this is tied to separate bootloaders. My requirements dictate that I have more redundancy than A/B bootloaders, and separate kernel and rootfs redundancy needs to be present for both bootloaders.

[DaveYYY]

Can you please show the full booting log?

[wrogers]

Attached syslog. Hopefully that’s good enough to see the issue, i.e. that it is not even attempting to mount mmcblk0p16.
syslog.txt (274.3 KB)

Looks like the kernel command line is not being updated with changes to extlinux.conf:

What is the process to have changes to extlinux.conf take effect after flashing?

[DaveYYY]

Hi,

Assuming you are now at mmcblk0p1, can you cat both /proc/cmdline and /boot/extlinux/extlinux.conf to make sure whether they are aligned or not?

[wrogers]

They are not aligned.

Interestingly, I discovered that manually selecting the eMMC as the boot device in the bootloader screen will use the updated extlinux.conf file and boot from mmcblk0p16. However, when I let it boot normally, it just uses mmcblk0p1.

[DaveYYY]

Hi,

I just tested on my local device (AGX Orin DevKit + L4T 35.4.1), and I can successfully mount /dev/mmcblk0p16 as the rootfs, no matter specifying the boot device in UEFI or not, or using device name or PARTUUID in extlinux.conf.

nvidia@tegra-ubuntu:~$ df -h
Filesystem       Size  Used Avail Use% Mounted on
/dev/mmcblk0p16   28G  5.6G   21G  22% /
none              15G     0   15G   0% /dev
tmpfs             15G   36K   15G   1% /dev/shm
tmpfs            3.0G   11M  3.0G   1% /run
tmpfs            5.0M  4.0K  5.0M   1% /run/lock
tmpfs             15G     0   15G   0% /sys/fs/cgroup
tmpfs            3.0G   16K  3.0G   1% /run/user/1000
nvidia@tegra-ubuntu:~$ cat /proc/cmdline 
root=/dev/mmcblk0p16 rw rootwait rootfstype=ext4 mminit_loglevel=4 console=ttyTCU0,115200 console=ttyAMA0,115200 console=tty0 firmware_class.path=/etc/firmware fbcon=map:0 net.ifnames=0 nospectre_bhb 

I’m suggesting the following things:

1. It should not make any difference, but I’m using P3701 SKU 0 while you are using SKU 5.
2. How did you clone the rootfs? I used rsync and it worked fine.
3. Are you using the latest 35.4.1 release?
4. See if using PARTUUID instead of device names helps.