Just bumping this, as I have the same need for my product.
Hi,
We have confirmed that next rel-35.x release will have a overlay dtbo which can change the boot order in it before flashing.
However, this dtbo won’t work with current UEFI software. Thus, only initrd workflow 10 in rel-35.1 could change the boot order at this moment. (or only runtime UEFI menu)
Hey @WayneWWW
how can I change the boot order then with initrd workflow 10?
When I use it to flash, the board always is set to boot from emmc.
Sorry, should be Workflow 11: Manually generate a bootable external storage device. Not workflow 10.
Hey @WayneWWW
okay then I will have only the NVME formatted, as I do now.
Afterwards I should use the normal flash.sh like this?
./flash.sh -c $BASEDIR/boardconfig/$PARTITION_FILE jetson-xavier-nx-devkit-qspi nvme0n1p1
So I’d flash it two times, first the nvme only, and then the QSPI with the correct boot parameter?
If that’s the case I’ll revert to my old “wrong” scripts, and just simply always use the flash.sh for everything except flashing the NVMe part…I hope that will work for A/B too, did not test with that yet. I remember that the UUID should be set when using NVMe.
If your board was flashed by sdkmanager/jetpack with rel-35.1 before, then running workflow 11 would be sufficient.
Running it to flash nvme drive, put it back to device and it will be able to boot from nvme.
I don’t get it.
I assume we get production modules from our distributor.
As I understood those modules are pre-flashed with the standard image on their emmc. So the QSPI should be set to boot from emmc?
When I now just flash the NVMe, would they not still boot from emmc? What you say makes only sense if the production modules come with the UEFI order preconfigured to NVMe > SD/EMMC?
You can try workflow 11 first.
I currently have no option to attach an NVMe disk to the flash PC. There is no connector. I’d have to order one.
But let’s be serious. I am sure that you are misunderstanding what I am trying to achieve.
What makes you think that your suggestion could work?
I have a SOM with QSPI UEFI set to boot from EMMC and a working image in EMMC. So it always boots from emmc and ignores the NVMe.
Why would it make any difference if I change only the content on the NVMe? Workflow 11 does not even have the board connected, so it can’t influence the UEFI boot order. It has simply no effect on the UEFI!
If you can explain my what magic should happen here I’ll set up a PC and test it, otherwise I can’t waste any time and money on such a try.
I could be wrong, but the search order likely exists in eMMC (or more specifically, QSPI NOR memory). Naturally, if you boot without NVMe, then search won’t find the NVMe. However, by changing it to search first for NVMe instead of eMMC, the boot device depends on what is present. If search order says to search first for the NVMe, as set up during QSPI flash, then the presence of an NVMe with a prerequisite extlinux.conf (or just “/boot”?) should in fact cause the NVMe to boot first without there ever being a change to the NVMe. It is true that the extlinux.conf of the NVMe can then chain load a different device, but the initial search order is not found on the NVMe. Also, device tree can also have an effect since it specifies some inherited environment. It might have been something needed on the NVMe which triggers detection, or it might be QSPI. Both work together.
@linuxdev In the case you describe the NVMe content could of course have an effect. But I am quite sure that in my specific case the issue lies somewhere else:
After flashing the search order is set to look for emmc and then for nvme. When I plug in an NVMe device I can boot from it by manually selecting the entry. Due to the search order, the system will always boot from the emmc though, as it has a working system on it. The search order skips boot options which are not found. I can not see how modifying the NVMe can have an effect on the search order and the successful boot of the emmc, which prevents the system from even attempting to boot from the NVMe…
To boot from NVMe I have to do one of these:
- Add the NVMe to the extlinux of the emmc
- Change the UEFI boot order in the QSPI to first look for the NVMe.
Workflow 10 (Flash QSPI and NVMe seperately), does not set the UEFI order correctly, that is a bug in my eyes. And Workflow 11 does work with the NVMe disk directly connected to the host PC. So I fail to understand how reflashing the NVMe with the same content as before would affect the boot order.
In that case you should in theory be able to edit the “extlinux.conf” to name the NVMe as the rootfs, although I’d suggest keeping a second boot entry to still point at the eMMC until you are sure things are as you wish. In some cases there were bugs whereby the “DEFAULT” for extlinux.conf would not correctly go to the desired entry unless that entry is also the first entry in the list, but I think this is probably fixed on the latest of R32.x and R34.x+ (not certain though). Think chain loading, in which it might go through eMMC extlinux.conf first, but still load NVMe first. It isn’t a bad idea to do as you mention, to add NVMe to extlinux.conf in eMMC. UEFI does change things though, and I’ve not experimented with this enough to say anything useful for UEFI configuration.
Do make sure any extlinux.conf in the NVMe does not accidentally name eMMC as rootfs, and do make sure that in both cases (eMMC and NVMe extlinux.conf) that the DEFAULT entry is the first entry.
Incidentally, Jetsons have always had to flash QSPI separately. The boot content software release versions have to match between early boot and rootfs content or it will fail. Simply flashing QSPI again and getting boot to succeed (even if it requires manually selecting NVMe via serial console) is how you verify the software releases are compatible. Anytime it fails to do what you want, then that has to be set up to guarantee it isn’t a mismatch of releases causing it (it is undefined how it will behave when QSPI and other content is mismatched).
So I am just saying that it is a somewhat object-oriented approach. QSPI and NVMe and eMMC are all modular components, and you have make sure each is designed to operate with the other or something “undefined” will occur.
I totally agree with you. In a very basic “student project” setup that workflow would be okay.
The extlinux probably will not work as we use the A/B feature. Afaik the A/B selection is done in UEFI, and not in extlinux.
My point though is that I need a flash procedure which works for production.
I want the flash tools to flash what I asked it to flash.
If I tell the scripts to flash QSPI + NVMe, I’d like to end up with a system which boots from NVMe and not from emmc, just because it already was there before. I can’t ask our production team to do the UEFI stuff manually. We assemble the systems with robots, there is no way we can add a manual step in there which the scripts should perform.
It can’t be that hard to modify the uefi partition before flash to be properly configured?
Maybe I have to compile it with some other options?
I’ve not experimented with massflash, so I am not very useful there. You are probably correct that A/B changes things, but when UEFI determines if a partition is bootable, then it probably also uses extlinux.conf to some extent. Do you have a working setup (regardless of whether it was mass flashed or manually configured) which you can post the extlinux.conf from all related partitions (eMMC and NVMe)? Also, post the output of “/proc/cmdline” in the working case, along with “lsblk -f”. I don’t know if I can spot anything, but it is simple to look at with that information, plus a serial console boot log of the working system. The goal is to see what is used from where, and why it transfers control from one step to another.
Additionally, when you flashed this working example, the target of a command line flash would have a “.conf” file. For example, if I flashed to target “jetson-xavier”, then this would have used “jetson-xavier.conf” for configuration. Whatever target you used, post a copy of that conf file as well.
FYI, flashing a Jetson and telling it NVMe versus eMMC is not about flashing the NVMe…it is about configuring for search.
I am glad for your help, and I think you’re right there is the A/B mechanism which selects if the extlinux from APP or APP_b is used. Afterwards the system continues with the extlinux config.
Extlinux:
TIMEOUT 30
DEFAULT primary
MENU TITLE L4T boot options
LABEL primary
MENU LABEL primary kernel
LINUX /boot/Image
FDT /boot/dtb/kernel_tegra194-p3668-0000-p3509-0000.dtb
INITRD /boot/initrd
APPEND ${cbootargs} root=PARTUUID=c75e9baf-ec9a-49d9-acf4-9cb0be129e74 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0
Cmdline:
root=PARTUUID=c75e9baf-ec9a-49d9-acf4-9cb0be129e74 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0
Anyway, I think getting your head into our use case might be a bit too complicated. Since Nvidia does not support A/B OTA Updates I’ve created a workflow myself. We create a tar file to be copied to the unused A/B partition. Afterwards the extlinux.conf is modified to match the UUID of the next rootfs, the bootcontrl is changed to boot from the other slot. Like this uefi uses slot A/B, reads the extlinux and boots the correct UUID. When you flash the system with ROOTFS_AB the l4t scripts automatically modify the extlinux.conf on each of the system images to match their future UUID.
But the issue is not with here. It’s the default boot order.
I’ve done some research. It looks like no matter if you use flash.sh or initrd_flash.sh the uefi_jetson.bin is always the same. It has not updated in my system since I unpacked the jetpack. Could it be that the default boot order is set here and cannot be changed unless I modify the source here? GitHub - NVIDIA/edk2-nvidia: NVIDIA EDK2 platform support
I think this file is responsible for the order: edk2-nvidia/PlatformBootOrderLib.c at main · NVIDIA/edk2-nvidia · GitHub
Unfortunately the DEFAULT_BOOT_ORDER_STRING does not match what I see on my board…
I don’t know enough to work in detail, but since this is scheduled to work with device tree overlay in the next release, I suspect you are on some deadline to make this work with the current release.
Is that extlinux.conf the same on all partitions, including A/B/eMMC? I didn’t see “lsblk -f” on the running Jetson, but it would be interesting to see UUID information. One of the reasons I mention “/proc/cmdline” is to see if the PARTUUID relates to that in both extlinux.conf and what the kernel actually sees (cmdline in part inherits “${cbootargs}” which can be created and edited or deleted before passing to the Linux kernel…in this case “cmdline” reflects that there was no “${cbootargs}”, at least not at the time when the Linux kernel was loaded to overwrite boot software).
Did you ever edit the boot source you gave the URL to above?
https://github.com/NVIDIA/edk2-nvidia/blob/main/Silicon/NVIDIA/Library/PlatformBootOrderLib/PlatformBootOrderLib.c
I am curious because you said the uefi_jetson.bin is always the same, and this leads to two questions: Was the binary content actually not changed, or was the binary content actually changed, but not behaving as you wish (in which case it was something the binary inherited determining the issue rather than the binary itself)? By what means did you determine uefi_jetson.bin did not change? I don’t know which partition it would be in, but in theory you could use dd to copy the partition, cut off any signature, and search for strings. That string search might indicate if any edits actually did take place. I suspect though that one would have to change the code to not use inherited environment if it is a case of inheritance changing boot order to the wrong order (the next release changes of .dtbo would be to provide an alternate environment inheritance of boot device search).
Hi,
Had some discussion with our UEFI folks. For rel-35.1 UEFI, there is no support to change boot order.
But as linuxdev’s link there, we have a new UEFI version with rel-35 updates branch.
In this version, there is a BootOrderNvme.dts
You can change the boot order with something similar to
gNVIDIATokenSpaceGuid {
26 DefaultBootPriority {
27 data = "nvme, usb, emmc";
28 locked;
29 };
30 };
After adding this, build the dtbo out, put it to Linux_for_Tegra/bootloader and Linux_for_Tegra/kernel/dtb.
Also, need to add dtbo name to your board config file.
Reflash the board after doing all these steps.
SInc Wayne posted while I was writing this post, here just some info for @linuxdev in case he’s still interested:
Here is the blkid (not lsblk since the bootarg uses PARTUUID instead of UUID):
/dev/nvme0n1p1: UUID="00a1c1c8-0c45-48bb-baf8-82c42d2019c7" TYPE="ext4" PARTLABEL="APP" PARTUUID="8859443e-b4cd-430b-9a00-5be584992c6f"
/dev/nvme0n1p2: UUID="4265ffb5-ecbc-4a41-b30d-69177129f890" TYPE="ext4" PARTLABEL="APP_b" PARTUUID="d24a091c-51d0-4eca-bdd1-a731ccb5e881"
/dev/nvme0n1p3: PARTLABEL="kernel" PARTUUID="69a4ff2d-a8af-4dee-975f-895edb0adf1e"
/dev/nvme0n1p4: PARTLABEL="kernel_b" PARTUUID="17da4ce5-68f1-4380-8a6f-cb60326fb566"
/dev/nvme0n1p5: PARTLABEL="kernel-dtb" PARTUUID="313269f9-9607-4cbc-8ed3-77706b6cf453"
/dev/nvme0n1p6: PARTLABEL="kernel-dtb_b" PARTUUID="321d9d1a-a9a0-4f0c-9d9e-d324e1233d4f"
/dev/nvme0n1p7: PARTLABEL="recovery" PARTUUID="4397df74-a43f-4b5e-9871-f702ecc67521"
/dev/nvme0n1p8: PARTLABEL="recovery-dtb" PARTUUID="4b2ea1a9-3c4d-4cf4-8815-874ca6e93f40"
/dev/nvme0n1p9: PARTLABEL="kernel-bootctrl" PARTUUID="6383c2ba-ee2a-4267-9f3a-994cc159367d"
/dev/nvme0n1p10: PARTLABEL="kernel-bootctrl_b" PARTUUID="1b5764e9-5287-48b6-8bba-9c731664fc04"
/dev/nvme0n1p11: PARTLABEL="RECROOTFS" PARTUUID="66a4fb36-1922-4226-b16e-db231c487f7e"
/dev/nvme0n1p12: UUID="AB9B-63AC" TYPE="vfat" PARTLABEL="esp" PARTUUID="35a68213-debc-44a6-8eb7-34650decd866"
/dev/nvme0n1p13: PARTLABEL="UDA" PARTUUID="016374c3-8adc-45ac-b858-cd3ade118133"
/dev/nvme0n1p14: UUID="8a4c7dde-2e79-4402-b3aa-2b51628f77e7" TYPE="ext4" PARTLABEL="censored" PARTUUID="a3d62819-ff30-4747-8d99-1bc4403f8a33"
Extlinux for nvme0n1p1
TIMEOUT 30
DEFAULT primary
MENU TITLE L4T boot options
LABEL primary
MENU LABEL primary kernel
LINUX /boot/Image
FDT /boot/dtb/kernel_tegra194-p3668-0000-p3509-0000.dtb
INITRD /boot/initrd
APPEND ${cbootargs} root=PARTUUID=8859443e-b4cd-430b-9a00-5be584992c6f rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0
Extlinux for nvme0n1p2
DEFAULT primary
MENU TITLE L4T boot options
LABEL primary
MENU LABEL primary kernel
LINUX /boot/Image
FDT /boot/dtb/kernel_tegra194-p3668-0000-p3509-0000.dtb
INITRD /boot/initrd
APPEND ${cbootargs} root=PARTUUID=d24a091c-51d0-4eca-bdd1-a731ccb5e881 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0
####################################################################
Thanks for your answer @WayneWWW ,
Checking your solution, I wanna make sure I get this right
- I expected that the uefi binary would have to be replaced. But it seems like these dtbo are responsible for the boot order.
- In the images directory I get this: BootOrderNvme_Jetson_RELEASE.dtbo, I copy ONLY this file? Does the name matter or will the scripts in 35.1 automatically pick this up? Do I need to copy the UEFI too to make sure that everything works?
- For my solution I’d like to be able to flash emmc and nvme depending on a variable. Can I simply copy the required file or delete it before flashing?
- I expected that the uefi binary would have to be replaced. But it seems like these dtbo are responsible for the boot order.
Both are needed. Old UEFI does not support those dtbo either. I think this answered both your question 1 and 2.
For my solution I’d like to be able to flash emmc and nvme depending on a variable. Can I simply copy the required file or delete it before flashing?
You can use that variable to decide whether the dtbo should be included or not.
my apologies, I did overread a part of your answer. My third question was nonsense of course.
Adding the DTBO to the config file of all “NMVe” configs is sufficient. Thus no variable is required on my side.
I tested flashing only the QSPI with the unmodified NVMe boot order overlay. That is sufficient to make the system boot from NVMe by default.
Thanks a lot for the support!
Now as this all works I’ve got two questions:
- L4TRootfsInfo_Jetson_RELEASE.dtbo and L4TRootfsABInfo_Jetson_RELEASE.dtbo also already exist in the current bootloader directory of the L4T. Do I need to copy these or will the files from the R35.1 be compatible with the UEFI from 35.1-updates?
- will this dtbo be added automatically when flashing nvme with 5.0.2 or will we have to compile the dtbo by hand even after the update?
1 Like