Massflash image with ADDITIONAL_DTB_OVERLAY_OPT option

adit_bhrgv · November 13, 2023, 3:31pm

Hi,

Setup:

BSP 35.4.1
Xavier NX
Flashing on NVME

I am trying to use a custom dtbo which I created and copied into Linux_for_Tegra/kernel/dtb/

This custom dtbo is dedicated to read a string for a Jetson NX manufacturer type since it is “Unknown” by default. It looks something like below:

“custom.dtbo”

/dts-v1/;

/ {
overlay-name = “Manufacturer”;
compatible = “nvidia,p3509-0000+p3668-0000”, “nvidia,p3509-0000+p3668-0001”;
jetson-header-name = “Jetson 40pin Header”;

fragment@0 {
	target-path = "/";

	__overlay__ {
		name = "MyName";
	};
};

};

I created Massflash image by using this dtbo in the arguments for the MFimage creation command as :
ADDITIONAL_DTB_OVERLAY_OPT=“custom.dtbo”

Finally, I flash the Massflash image on Jetson Xavier NX.
However, it does not automatically show an entry into /proc/device-tree/name as this “name” is in the custom.dtbo.

But, I can only get this “/proc/device-tree/name” entry when I manually apply the dtbo via
/opt/nvidia/jetson-io/config-by-hardware.py -n “Manufacturer” and reboot the jetson.

Is there a better alternative to this ? I don’t want to apply this dtbo via jetson-io tool , rather automatically after flashing…Is it possible ?
I also want to set boot-order as NVME as default via “BootOrderNvme.dtbo”
I tried adding ADDITIONAL_DTB_OVERLAY_OPT=“custom.dtbo,BootOrderNvme.dtbo” to Massflash creation command before flashing, but it doesn’t work as well. Am I doing something wrong ?

Thanks
Regards

DaveYYY · November 14, 2023, 2:22am

Hi,

Have you checked if the overlay is being loaded correctly during boot up?
You can build UEFI again to enable debug print to do so.

What do you mean with it doesn’t work? What gets set as the the default booting device?
Do you have any other bootable device plugged in?
The logic is that UEFI will scan for all bootable devices upon boot, and those newly found devices will be added to the top of the list.

adit_bhrgv · November 15, 2023, 9:47am

“It doesn’t work” means, setting BootOrderNvme.dtbo in environment variable ADDITIONAL_DTB_OVERLAY_OPT=“BootOrderNvme.dtbo” while creating massflash image doesn’t lead to uefi default boot entry to nvme.
The first entry in the boot menu is still EMMC.

I also tried fdtoverlay to append the final dtb with kernel/dtb/tegra194-p3668-0001-p3509-0000.dtb with BootOrderNvme.dtbo.

fdtoverlay -i Linux_for_Tegra/kernel/dtb/tegra194-p3668-0001-p3509-0000.dtb -o Linux_for_Tegra/kernel/dtb/tegra194-p3668-0001-p3509-0000-extended.dtb Linux_for_Tegra/kernel/dtb/BootOrderNvme.dtbo

mv Linux_for_Tegra/kernel/dtb/tegra194-p3668-0001-p3509-0000-extended.dtb Linux_for_Tegra/kernel/dtb/tegra194-p3668-0001-p3509-0000.dtb

After fdtoverlay, it contains the nvme entry in kernel/dtb/tegra194-p3668-0001-p3509-0000.dtb as :

firmware {
        uefi {
            variables {
                gNVIDIATokenSpaceGuid {
                    DefaultBootPriority {
                        locked;
                        data = "nvme";
                    };
                };
            };
        };
...
...

The FDT entry in extlinux.conf is :

    FDT /boot/dtb/kernel_tegra194-p3668-0001-p3509-0000.dtb

Is the final dtb tegra194-p3668-0001-p3509-0000.dtb (which I modified with fdtoverlay ) copied to kernel_tegra194-p3668-0001-p3509-0000.dtb as the final FDT entry ?
The contents of Linux_for_Tegra/bootloader on host PC after creating Massflash image has 2 kernel entries:

 kernel_tegra194-p3668-0001-p3509-0000.dtb    
 kernel_tegra194-p3668-0001-p3509-0000.dtb.sb

What is the .sb file ? Because, this file is the same as in the FDT entry on the jetson.
FDT /boot/dtb/kernel_tegra194-p3668-0001-p3509-0000.dtb

which doesn’t have nvme entry in uefi.
That is probably the reason why the BootOrder doesn’t get changed on flashing.

Regards

DaveYYY · November 15, 2023, 11:33am

Hi,

If massflash package is created correctly, please try modifying files in Linux_for_Tegra/mfi_<board>; they are what get actually flashed into devices with massflash.

adit_bhrgv · November 15, 2023, 1:10pm

Could you please also comment on the file:
kernel_tegra194-p3668-0001-p3509-0000.dtb.sb ?

.sb what kind of file is this ? Is it related to secure boot ?
Why this file is being used and copied in the FDT entry but without .sd extension on Jetson ?

DaveYYY · November 15, 2023, 1:23pm

So what comment did you use to flash the device?
Did you enable secure boot?

Then that may be the case.

adit_bhrgv · November 15, 2023, 1:44pm

Yes, we have secure boot enabled.

To generate the Massflash Image:

sudo ADDITIONAL_DTB_OVERLAY_OPT="BootOrderNvme.dtbo" ROOTFS_ENC=1 BOARDID=3668 FAB=301 BOARDSKU=0001 BOARDREV=E.0 ./tools/kernel_flash/l4t_initrd_flash.sh -u $ROOT_DIR/keys/key.pem -v $ROOT_DIR/keys/sbk_key.txt -p "-i ./ekb.key" --external-device nvme0n1p1 -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_enc.xml --external-only -S 40GiB --no-flash --massflash 8 --showlogs jetson-xavier-nx-devkit-emmc nvme0n1p1

To flash the device from Linux_for_Tegra/mfi_ folder :

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --massflash 8 --flash-only

How can I make sure, that .sb file is updated with latest fdtoverlay changes with custom.dtbo and BootOrderNvme.dtbo as applied below ? I see kernel_tegra194-p3668-0001-p3509-0000.dtb has these overlay changes but kernel_tegra194-p3668-0001-p3509-0000.dtb.sb doesn’t.

#Apply custom.dtbo

dtc -O dtb -o Linux_for_Tegra/kernel/dtb/custom.dtbo -@ Linux_for_Tegra/dts/custom.dts
fdtoverlay -i Linux_for_Tegra/kernel/dtb/tegra194-p3668-0001-p3509-0000.dtb -o Linux_for_Tegra/kernel/dtb/tegra194-p3668-0001-p3509-0000-extended.dtb Linux_for_Tegra/kernel/dtb/custom.dtbo

mv Linux_for_Tegra/kernel/dtb/tegra194-p3668-0001-p3509-0000-extended.dtb Linux_for_Tegra/kernel/dtb/tegra194-p3668-0001-p3509-0000.dtb

#Apply BootOrderNvme.dtbo
fdtoverlay -i Linux_for_Tegra/kernel/dtb/tegra194-p3668-0001-p3509-0000.dtb -o Linux_for_Tegra/kernel/dtb/tegra194-p3668-0001-p3509-0000-extended.dtb Linux_for_Tegra/kernel/dtb/BootOrderNvme.dtbo

mv Linux_for_Tegra/kernel/dtb/tegra194-p3668-0001-p3509-0000-extended.dtb Linux_for_Tegra/kernel/dtb/tegra194-p3668-0001-p3509-0000.dtb

DaveYYY · November 15, 2023, 1:56pm

What changes are taking effect now?
Only custom.dtbo or both custom.dtbo and BootOrderNvme.dtbo are gone?

adit_bhrgv · November 15, 2023, 2:05pm

I didn’t flash the jetson again yet but currently what I observe is in Linux_for_Tegra/bootloader on host pc:

kernel_tegra194-p3668-0001-p3509-0000.dtb.sb  -> has nvme change but custom.dtbo change is the previous one
kernel_tegra194-p3668-0001-p3509-0000.dtb  -> has nvme change and also new custom.dtbo change

I guess, kernel_tegra194-p3668-0001-p3509-0000.dtb is regenerated while creating massflash image again with new custom.dtbo ( as we change within custom.dtbo “name” value ) but kernel_tegra194-p3668-0001-p3509-0000.dtb.sb remains the old one. Not sure, why .sb file is not regenerated while creating massflash image.

But in Linux_for_Tegra/mfi_jetson-xavier-nx-devkit-emmc/bootloader : there is only one file and I guess that has to be the final dtb: [Since mfi_ folder, there is no .sb file ]

kernel_tegra194-p3668-0001-p3509-0000.dtb -> it has both BootorderNvme.dtbo and custom.dtbo changes

I will flash the jetson again and post again regarding what changes takes effect…

adit_bhrgv · November 15, 2023, 6:13pm

After flashing, the custom.dtbo change is working correctly.
However, boot order set as “nvme” from BootOrderNvme.dtbo is not reflected in uefi menu and the jetson continues to boot from emmc by default.

However, the boot-order change can be seen in the final dtb file: kernel_tegra194-p3668-0001-p3509-0000.dtb
Is this change not good enough to boot from NVME by default ? or some extra things needed to be done ?

 firmware {

                uefi {

                        variables {

                                gNVIDIATokenSpaceGuid {

                                        DefaultBootPriority {
                                                locked;
                                                data = "nvme";
                                        };
                                };
                        };
                };

                optee {
                        compatible = "linaro,optee-tz";
                        method = "smc";
                        status = "disabled";
                };

                android {
                        hardware = "galen-k414";
                };
        };

DaveYYY · November 16, 2023, 1:32am

Then if you manually change the boot order in UEFI, would it work?
It’s just that settings in dtb do not take effect?

adit_bhrgv · November 16, 2023, 7:52am

Yes, manually selecting boot device as NVME from UEFI Menu works !
But, the dtb changes doesn’t take effect. That means, it didn’t boot from NVME by default. Still the EMMC is reflected as the default boot device.

DaveYYY · November 16, 2023, 8:14am

Hi,

Can you please show me what do you get with df -h after booting into eMMC and NVMe?
Also, would it make any difference if you apply it with L4TConfiguration.dtbo instead of BootOrderNvme.dtbo?
https://docs.nvidia.com/jetson/archives/r35.4.1/DeveloperGuide/text/SD/Bootloader/UEFI.html#customizing-the-default-boot-order-in-the-configuration-file

adit_bhrgv · November 17, 2023, 1:47pm

I don’t have anything on emmc as I use --external-only flag while creating Mass flash image.
However, “df -h” on NVME returns :

df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/crypt_root   39G  8.6G   29G  24% /
/dev/mapper/crypt_UDA   279M   24K  257M   1% /mnt/crypt_UDA
none                    3.3G     0  3.3G   0% /dev
tmpfs                   3.4G     0  3.4G   0% /dev/shm
tmpfs                   684M  2.2M  682M   1% /run
tmpfs                   5.0M  4.0K  5.0M   1% /run/lock
tmpfs                   3.4G     0  3.4G   0% /sys/fs/cgroup
/dev/nvme0n1p1          371M   96M  248M  28% /boot
/dev/sda1               3.7G  224K  3.7G   1% /mnt/hxdongle
tmpfs                   684M     0  684M   0% /run/user/1000```

I checked again that using “ADDITIONAL_DTB_OVERLAY_OPT=“BootOrderNvme.dtbo”” indeed works when I don’t use the --external-only flag in the below command without creating massflash image, rather just directly flashing jetson:

sudo ADDITIONAL_DTB_OVERLAY_OPT="BootOrderNvme.dtbo" ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh -u $ROOT_DIR/keys/key.pem -v $ROOT_DIR/keys/sbk_key.txt -p "-i ./ekb.key" nvme0n1p1 -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_enc.xml -S 40GiB --showlogs jetson-xavier-nx-devkit-emmc nvme0n1p1

But, I don’t want to flash EMMC . Workflow 11, example 2 from README_initrd_flash.txt mentions:

**"Workflow 11: Generate images for internal device and external device seperately"** 
**Example 2: In this example, you want to boot Jetson Xavier NX SD from an**
**attached NVMe SSD. The SD card does not need to be plugged in. You can also**
**apply this if you don't want to use the emmc on the Jetson Xavier NX emmc.**

Do I need to generate qspi-only images for BootorderNvme.dtbo to take effect ?

Regards,

DaveYYY · November 17, 2023, 2:39pm

What if you do

-p "-c bootloader/t186ref/cfg/flash_l4t_t194_qspi_p3668.xml"

instead of --external-only?

adit_bhrgv · November 17, 2023, 2:51pm

sudo ADDITIONAL_DTB_OVERLAY_OPT="BootOrderNvme.dtbo" ROOTFS_ENC=1 BOARDID=3668 FAB=301 BOARDSKU=0001 BOARDREV=E.0 ./tools/kernel_flash/l4t_initrd_flash.sh -u $ROOT_DIR/keys/key.pem -v $ROOT_DIR/keys/sbk_key.txt -p "-i ./ekb.key -c bootloader/t186ref/cfg/flash_l4t_t194_qspi_p3668.xml" --external-device nvme0n1p1 -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_enc.xml  -S 40GiB --no-flash --massflash 8 --showlogs jetson-xavier-nx-devkit-emmc nvme0n1p1

We already have -c option with “./tools/kernel_flash/flash_l4t_nvme_rootfs_enc.xml” conf file, will it be able to accept another "-c bootloader/t186ref/cfg/flash_l4t_t194_qspi_p3668.xml " within -p ?

Does the above command looks good ?

DaveYYY · November 17, 2023, 2:57pm

YES.
I don’t think they contradict each other.

adit_bhrgv · November 17, 2023, 3:58pm

This works now.
The Boot order seems to be updated now with both L4TConfiguration.dtbo or BootOrderNvme.dtbo .

Could you please give a background on excluding --external-only flag and including bootloader/t186ref/cfg/flash_l4t_t194_qspi_p3668.xml ?

What is the reasoning behind this works ? Is it related to boot-flow ? Please refer me to Nvidia documentation

DaveYYY · November 20, 2023, 2:17am

I’m just doing some guessing here.

The boot order is determined by UEFI, which resides in the QSPI memory.
Then you can take a look at the following snippet taken from Linux_for_Tegra/tools/kernel_flash/l4t_flash_from_kernel.sh

1142 function flash_qspi
1143 {
1144 	if [ "${host_mode}" = "0" ] && [ "${external_only}" = "0" ] && is_spi_flash; then
1145 		print_log "Starting to flash to qspi"
1146 		if [ -z "${target_partname}" ]; then
1147 			erase_spi /dev/mtd0
1148 		fi
1149 		write_to_spi
1150 
1151 		print_log "Successfully flash the qspi"
1152 	fi
1153 }

As you can see, if you explicitly specify –external-only, the QSPI memory will not be updated at all.
Also, UEFI reads device tree from the dedicated bootloader-dtb partition, which also resides in the QSPI memory, so checking if it’s taking effect in the rootfs is meaningless.

Therefore, if you don’t want to flash the internal eMMC, and need to update the QSPI memory to update the boot order, instead of –external-only, you need to specify -p "-c bootloader/t186ref/cfg/flash_l4t_t194_qspi_p3668.xml" to tell the script to generate images for only QSPI but not eMMC.

adit_bhrgv · November 20, 2023, 7:30am

Thank you @DaveYYY for the clear explanation.

Topic		Replies	Views
Xavier NX remains in recovery after successfull massflash with QSPI+external images & disk encryption Jetson Xavier NX security	13	649	December 4, 2023
Jetson Board parameters Jetson Xavier NX reflash	15	67	July 29, 2024
Initrd flash boot order Jetson Xavier NX boot	31	2653	October 25, 2022
Native boot from M.2 NVME SSD Jetson Xavier NX nvme	77	21037	October 13, 2021
Workflow 3: Fail to make the massflash backup image, device is orin nx 8gb Jetson Orin NX reflash , board-design	21	1485	March 12, 2024
[MASS FLASH] Flashing Multiple Jetson Orin NX 8GB(QSPI+NVME) Devices Always Failed Jetson Orin NX hw	31	1577	October 27, 2023
Flash error : Jetson Nano 16GB emmc Jetson Nano flash	7	50	December 3, 2024
Jetson Orin Nano doesn't show up with lsusb Jetson Orin Nano usb	39	648	April 10, 2024
Xavier NX disk encryption with NVME SSD Jetson Xavier NX boot , security , nvme	20	1429	June 21, 2023
What version of jetpack has the feature "Kernel Boot Sequence " ?only jetpack4.5? how can we make a usb disk image with an easy way? Jetson Nano boot	56	2527	October 3, 2021

Massflash image with ADDITIONAL_DTB_OVERLAY_OPT option

Related topics