Jetson Nano eMMC U-Boot Field Upgrade

Hello,

I am working with a Jetson Nano based system, using L4T 32.4.2. I recently switched from using an SD card module to an eMMC module as we get closer to production. I have some software to update U-Boot on my devices, which I migrated over from a device we make based on the TX1 module. The U-Boot update process runs on devices in the field and works fine on the TX1 and on the SD card based Nano, but does not work on the eMMC based Nano. For reference, I am not using secure boot (yet), but that is also planned for this device.

From debugging a bit, it seems that TegraBoot will check a signature on the updated copy of U-Boot and it doesn’t match what TegraBoot expects, causing TegraBoot to run the copy of U-Boot stored in the LNX-1 partition instead. Here are the messages I receive from TegraBoot when this happens:

...
[0000.552] *** Booting BFS0.
[0000.555] Read PT from (0:3)
[0000.561] Using BFS PT to query partitions
[0000.565] PT: Partition LNX NOT found !
[0001.055] *** Failing over to KFS1.
[0000.308] [L4T TegraBoot] (version 00.00.2018.01-l4t-2d758fea)

The device does boot after that, but it isn’t using the modified version of U-Boot of course. Currently, I generate my U-Boot binary with a combination of mkbootimg and tegraflash.py calls, as recommended on this forum when I asked about this for my TX1 based product. Here are the commands being used to generate the updated U-Boot binary on the Nano:

sudo ./mkbootimg --kernel ./t210ref/p3450-porg/u-boot.bin --ramdisk initrd-blank --board mmcblk0p1 --output boot.img --cmdline "root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 fbcon=map:0 net.ifnames=0 sdhci_tegra.en_boot_part_access=1 "
sudo ./tegraflash.py --cfg flash.xml --chip 0x21 --skipuid  --bl cboot.bin --bct  P3448_A00_4GB_Micron_4GB_lpddr4_204Mhz_P987.cfg --odmdata 0x94000 --bldtb tegra210-p3448-0002-p3449-0000-b00.dtb --applet nvtboot_recovery.bin --boardconfig t210ref/cfg/board_config_p3448.xml --cmd "sign"

For reference, to initially flash the device I am using the following command:

sudo ./flash.sh -S 536870912 -r jetson-nano-emmc mmcblk0p1

I am aware that there are some new tools in L4T 32.4.2 for updating the lower level boot loaders (CBoot, etc.) as well as the DTB files on the device. I tested that procedure and was not able to get it to work, but in the end I do not want to use that procedure anyways (for several reasons) and just want to support updating U-Boot.

One additional point is that I have tested my update code using an identical copy of the U-Boot binary and that update does “work”, so it doesn’t seem like my install code is broken. It only doesn’t work when I make a change to the U-Boot binary (e.g. rebuild it).

Is there a way to generate a U-Boot binary so that I can support updating only the U-Boot binary in the field?

Thanks,

Chris Richardson

Hi,

I don’t get it. Why not just use “sudo ./flash.sh -r -k LNX -board name- mmcblk0p1” to update the uboot?

Hi WayneWWW,

Thanks for the response. These are devices that are installed in the field and the customer has no way to connect to the USB port to flash new software. Supporting a U-Boot update in the field lets us fix bugs and/or change things after devices have been installed, since it can be very costly for customers to take devices out of the field and send them back to us. I have several other devices where this general process works fine, including a device based on the TX1 module. It also works fine on the Nano dev module with SD card.

I would just like to be able to update U-Boot on the product I am working on now which will have a Nano eMMC/production module.

Thanks,

Chris Richardson

Hi cjwti,

Do you want to update uboot to our new release or to your own binary? Actually, we have OTA method now but this is for upgrading to our latest release.

Also, what is your current method for in-field update?
IMO, You could just use dd to update the uboot.

Hi WayneWWW,

Thanks for the info. The device will be using L4T 32.4.2 and I want to update U-Boot after making a change to it (even just rebuilding it with a clean build). Everything in this case is done with L4T 32.4.2 on an eMMC/production module.

I’m glad you brought up the dd method of updating U-Boot. That is basically what I would like to do, and I believe if you try that you will see the same problem I am seeing. If you flash an eMMC module, then clean/rebuild U-Boot, then try to dd the new version of U-Boot into the LNX partition, TegraBoot will not run that new version and will instead run the version of U-Boot flashed in to the LNX-1 partition.

Please try this on an eMMC module, since it works fine on the SD card module which doesn’t support using the LNX-1 fallback partition. I would expect this to work fine on the eMMC module as well and that is the problem I am trying to solve.

Thanks,

Chris Richardson

Hi,

Do you sign the boot.img before you use dd?

Would you mind sharing the full step to reproduce issue?

Hey WayneWWW,

I do sign the boot.img before it is used. That step is shown in my original post with the mkbootimg and tegraflash command listing. I did manage to put together some reproduction steps you can use. This is all done without any of my software involved. I assume you are familiar with recompiling U-Boot and I didn’t include instructions for that.

<Put the device in recovery mode and connect a serial console>
...

# Switch to base L4T installation folder.
cd ??/Linux_for_Tegra

# Flash the device with initial software.
sudo ./flash.sh jetson-nano-emmc mmcblk0p1

<Take note of TegraBoot KFS0/KFS1 selection and U-Boot date/time banner>
<My device showed the following on the first boot after flashing:>
...
[0000.342] Non secure mode, and RB not enabled.
[0000.358] Csd NumOfBlocks=0
[0000.552] *** Booting BFS0.
[0000.555] Read PT from (0:3)
[0000.561] Using BFS PT to query partitions
[0000.565] PT: Partition LNX NOT found !
[0001.567] *** Booting KFS0.
[0001.570] Read GPT from (0:3)
[0001.573] Using GPT Primary to query partitions
...
U-Boot 2016.07-svn70 (Jun 10 2020 - 16:59:14 -0700)

# Clean and build a new U-Boot binary.
<Follow steps in L4T documentation section "Downloading and Building U-Boot" to do this>
...

# Generate signed binary with some commands I received from NVIDIA in a separate post.
cd ./bootloader
sudo ./mkbootimg --kernel ./t210ref/p3450-porg/u-boot.bin --ramdisk initrd-blank --board mmcblk0p1 --output boot.img --cmdline "root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 fbcon=map:0 net.ifnames=0 sdhci_tegra.en_boot_part_access=1 "
sudo ./tegraflash.py --cfg flash.xml --chip 0x21 --skipuid  --bl cboot.bin --bct  P3448_A00_4GB_Micron_4GB_lpddr4_204Mhz_P987.cfg --odmdata 0x94000 --bldtb tegra210-p3448-0002-p3449-0000-b00.dtb --applet nvtboot_recovery.bin --boardconfig t210ref/cfg/board_config_p3448.xml --cmd "sign"

# Copy the new U-Boot binary over to the device
# Obviously replace your user name and IP address of the device.
scp ./signed/boot.img.encrypt <user@ip>:/home/<user>/

# Now switch over to the device's serial console or SSH in to it and
# make sure which partition contains LNX (U-Boot)
ls -l /dev/disk/by-partlabel/ | grep LNX
lrwxrwxrwx 1 root root 15 Jun  5 11:23 LNX -> ../../mmcblk0p5
lrwxrwxrwx 1 root root 15 Jun  5 11:23 LNX-1 -> ../../mmcblk0p9

# Copy the new U-Boot to the LNX partition we see above (mmcblk0p5)
sudo dd if=./boot.img.encrypt of=/dev/mmcblk0p5

# Reboot to see changes
sudo shutdown -r now

<Now check TegraBoot KFS0/KFS1 selection and U-Boot date/time banner again>
<My device now shows the following:>
[0000.360] Csd NumOfBlocks=0
[0000.552] *** Booting BFS0.
[0000.555] Read PT from (0:3)
[0000.561] Using BFS PT to query partitions
[0000.565] PT: Partition LNX NOT found !
[0001.174] *** Failing over to KFS1.
[0000.281] [L4T TegraBoot] (version 00.00.2018.01-l4t-2d758fea)
[0000.287] Processing in cold boot mode Bootloader 2
...
U-Boot 2016.07-svn70 (Jun 10 2020 - 16:59:14 -0700)

So we can see that updating with a cleanly compiled U-Boot binary from the same source code did not work. TegraBoot rejects the new binary and uses the LNX-1 (KFS1) partition instead. This is the problem I am trying to solve. Without code for TegraBoot it’s pretty difficult to know why it rejects the new image and I hope you can shed some light on this.

Thanks for taking the time to read this and try it out.

Chris Richardson

1.Are you still able to see LNX partition label after the system uses LNX-1 to boot up?
2. Have you tried to upgraded both LNX and LNX-1?

Hey WayneWWW,

Yes I can still see the partition label after it boots up using LNX-1. I have not tried updating both LNX-1 and LNX. My goal is to leave LNX-1 alone, to have as a fallback boot loader in case something happens during the LNX/U-Boot partition write (e.g. a power cut).

Thanks,

Chris

Hi,

I just confirmed with our bootloader team for this.
For TX1 and nano-emmc, we don’t support to update one partition by “dd” or other tools.
Because we implement BFS/KFS mechanism in it. It will calculate images’ CRC values and store in the BCT.
The LNX partition is belong to KFS. In here, if you change the uboot, which is in LNX partition, it mean the KFS’s CRC value will be changed, but the value which was stored in BCT doesn’t be updated accordingly, then the CRC mismatched.

You said you can update uboot on TX1, I supposed this TX1 was flashed with old release,right ? which didn’t have BFS/KFS yet.

And for nano-sd, we don’t store BFS/KFS CRC value in BCT.

You can generate a new booloader payload (BUP), which include your latest uboot, then install it to update all bootloader partitions. Please check BUP info on l4t developer guide.

Hey WayneWWW,

Thanks for the response and information. Our TX1 based device runs 28.2 and does not have the BFS/KFS redundancy, which is why my U-Boot only update works I suppose.

I did mention in my original post that I tried the BUP update and that the install does not work on my device. In addition, that requires a rather large dependency on Python, which I would like to avoid.

That said, getting the updater to run successfully is also a bit of a mess and not really that well documented in the L4T docs as far as I can tell. I had to do all sorts of messing around with the FAB/BOARDREV/SKU environment variables when flashing vs. running the l4t_generate_soc_bup.sh script, in order to make everything line up perfectly so the bl_update_payload file would be accepted by the l4t_payload_updater tool. I also had to modify the l4t_generate_soc_bup.sh script and nv_boot_control.conf file to support my custom board name, neither of which were documented. This alone took me a few days and gave me the impression that these tools are not really ready for production devices.

Once I did get the l4t_payload_updater tool to accept and install the update payload, I then spent a lot more time debugging why it didn’t work. The symptom is that the update succeeds but then neither BFS0 or BFS1 are accepted by TegraBoot, which renders the device un-bootable (without re-flashing). Unfortunately, this basically cemented the sour taste I had regarding the maturity of these tools.

I did the above investigations before I posted my initial question, and hence my reasoning to skip using the BUP tools altogether and use my existing tool set to update the parts of the device software I want to support updating.

If you would like any information to help reproduce the update failure please let me know.

Also as an aside, I feel like I could have avoided a lost of wasted effort here if NVIDIA would release the TegraBoot and/or CBoot source.

I think you need to post the error you’ve met when running BUP updater.

Also as an aside, I feel like I could have avoided a lost of wasted effort here if NVIDIA would release the TegraBoot and/or CBoot source.

Unfortunately, TX1/Nano cboot source will not be released.

Hey WayneWWW,

Understood about the CBoot source. I wonder whether the same situation applies to the TegraBoot source. It looks like TegraBoot is the one having an issue with the BUP update in this case. I don’t get any errors during the update itself, but here is the console output when TegraBoot rejects both BFS0 and BFS1:

[0000.205] [L4T TegraBoot] (version 00.00.2018.01-l4t-2d758fea)
[0000.210] Processing in cold boot mode Bootloader 2
[0000.215] A02 Bootrom Patch rev = 1023
[0000.218] Power-up reason: software reset
[0000.222] No Battery Present
[0000.225] pmic max77620 reset reason
[0000.228] pmic max77620 NVERC : 0x0
[0000.232] RamCode = 0
[0000.234] Platform has DDR4 type RAM
[0000.237] max77620 disabling SD1 Remote Sense
[0000.241] Setting DDR voltage to 1125mv
[0000.245] Serial Number of Pmic Max77663: 0x335d9
[0000.253] Entering ramdump check
[0000.256] Get RamDumpCarveOut = 0x0
[0000.259] RamDumpCarveOut=0x0,  RamDumperFlag=0xe59ff3f8
[0000.264] Last reboot was clean, booting normally!
[0000.269] Sdram initialization is successful
[0000.273] SecureOs Carveout Base=0x00000000ff800000 Size=0x00800000
[0000.279] Lp0 Carveout Base=0x00000000ff780000 Size=0x00001000
[0000.285] BpmpFw Carveout Base=0x00000000ff700000 Size=0x00080000
[0000.291] GSC1 Carveout Base=0x00000000ff600000 Size=0x00100000
[0000.297] GSC2 Carveout Base=0x00000000ff500000 Size=0x00100000
[0000.302] GSC4 Carveout Base=0x00000000ff400000 Size=0x00100000
[0000.308] GSC5 Carveout Base=0x00000000ff300000 Size=0x00100000
[0000.314] GSC3 Carveout Base=0x000000017f300000 Size=0x00d00000
[0000.330] RamDump Carveout Base=0x00000000ff280000 Size=0x00080000
[0000.336] Platform-DebugCarveout: 0
[0000.340] Nck Carveout Base=0x00000000ff080000 Size=0x00200000
[0000.345] Non secure mode, and RB not enabled.
[0000.362] Csd NumOfBlocks=0
[0000.558] *** Booting BFS0.
[0000.560] Read PT from (0:3)
[0000.567] Using BFS PT to query partitions
[0000.571] PT: Partition LNX NOT found !
[0001.054] *** Failing over to KFS1.
[0000.282] [L4T TegraBoot] (version 00.00.2018.01-l4t-2d758fea)
[0000.287] Processing in cold boot mode Bootloader 2
[0000.292] A02 Bootrom Patch rev = 1023
[0000.296] Power-up reason: software reset
[0000.299] No Battery Present
[0000.302] pmic max77620 reset reason
[0000.305] pmic max77620 NVERC : 0x0
[0000.309] RamCode = 0
[0000.311] Platform has DDR4 type RAM
[0000.314] max77620 disabling SD1 Remote Sense
[0000.319] Setting DDR voltage to 1125mv
[0000.322] Serial Number of Pmic Max77663: 0x335d9
[0000.330] Entering ramdump check
[0000.333] Get RamDumpCarveOut = 0x0
[0000.336] RamDumpCarveOut=0x0,  RamDumperFlag=0xe59ff3f8
[0000.341] Last reboot was clean, booting normally!
[0000.346] Sdram initialization is successful
[0000.350] SecureOs Carveout Base=0x00000000ff800000 Size=0x00800000
[0000.356] Lp0 Carveout Base=0x00000000ff780000 Size=0x00001000
[0000.362] BpmpFw Carveout Base=0x00000000ff700000 Size=0x00080000
[0000.368] GSC1 Carveout Base=0x00000000ff600000 Size=0x00100000
[0000.374] GSC2 Carveout Base=0x00000000ff500000 Size=0x00100000
[0000.380] GSC4 Carveout Base=0x00000000ff400000 Size=0x00100000
[0000.385] GSC5 Carveout Base=0x00000000ff300000 Size=0x00100000
[0000.391] GSC3 Carveout Base=0x000000017f300000 Size=0x00d00000
[0000.407] RamDump Carveout Base=0x00000000ff280000 Size=0x00080000
[0000.414] Platform-DebugCarveout: 0
[0000.417] Nck Carveout Base=0x00000000ff080000 Size=0x00200000
[0000.423] Non secure mode, and RB not enabled.
[0000.439] Csd NumOfBlocks=0
[0000.635] *** Booting BFS1.
[0000.637] Read PT from (0:3)
[0000.644] Using BFS PT to query partitions
[0000.648] PT: Partition LNX NOT found !
[0001.129] *** Failing over to KFS2.

The only thing that looks like an error to me is the “PT: Partition LNX NOT found !” message, but that shows up during normal boots as well.

I attached the update log as well. If more information would help, please let me know. Once the board fails to boot I can’t do anything further, but I could get more information about the boot partitions after I perform the update and before I reboot, if that would help.

BootLog_2020-06-10_11_ForumPostBUPUpdateLog.txt (34.3 KB)

Thanks,

Chris Richardson

I wonder whether the same situation applies to the TegraBoot source

Yes, we don’t apply tegraboot src too. Please note that, unlike cboot, tegraboot is never released to public since rel-24.

We will check your log. Thanks.

Hi,

There is no error in your log and our engineer has tried on our side and update uboot is working fine.

  1. Could you share the detailed step of how you hit this issue?
  2. Have you modified anything in tegraflash.py or tegraflash_internal.py that we don’t know?

Hey WayneWWW,

Thanks for checking on this and for the response. I haven’t modified anything in tegraflash.py or tegraflash_internal.py. I did modify l4t_generate_soc_bup.sh to add my board to the t21x_spec list (and to remove all the other boards, to keep the file size down).

I also modified the nv_boot_control.conf file on my board to make the TNSPEC value match the built software, so l4t_payload_updater would run. Here is the contents of my l4t_generate_soc_bup.sh t21x_spec list:

'boardid=3448;fab=400;boardsku=0002;boardrev=;fuselevel_s=1;chiprev=0;board=board_name;rootdev=mmcblk0p2'

The last modification I made is to l4t_bup_gen.func, where I changed the definition of partitions_t210ref so the only partitions in the list are from partitions_t210ref_common.

I put the root FS in mmcblk0p2 (and flash it that way) because I have a small partition before the APP partition, which is used to align the APP partition to the erase block size of 512K. Using mmcblk0p1 there (and using the same value when flashing with flash.sh) does not affect anything as far as I can tell, since I don’t boot with the rootfs directly in eMMC, and instead boot with a ramdisk that I load from mmcblk0p2 via U-Boot.

Thanks,

Chris Richardson

Hi Chris,

Could you firstly follow the step from document without any customization in l4t_generate_soc_bup.sh?

https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%2520Linux%2520Driver%2520Package%2520Development%2520Guide%2Fbootloader_update_nano_tx1.html%23wwpID0E0RE0HA

Hi WayneWWW,

Thanks a lot for looking into this and for your responses and information. The instructions you linked are what I initially followed to start any of this.

Unfortunately since I am using a separate file for my base flashing .conf file (and due to other differences like the FAB and BOARDSKU not matching between what is flashed vs. what is generated at build time), the updater doesn’t find a compatible ‘spec’ to use so it just won’t do anything. I customized l4t_generate_soc_bup.sh so the updater would find a matching spec, and and removed the unused (TX1/Nano QSPI) entries to save space. Here is what it printed just now, but this is after I have already debugged and made the FAB and BOARDSKU the same:

root@device:/opt/autorun# l4t_payload_updater_t210 ./bl_update_payload
BLOB PATH:
/opt/autorun/bl_update_payload

SPEC: 3448-400-0002--0-0-board_name-mmcblk0p2
Compatible SPEC: 3448-400-0002--0-0-board_name-mmcblk0p2

Error. Cannot find matched SPEC in blob.
No changes have been made. Exiting...

After, and since I now know more about how this works, I modified the /etc/nv_boot_control.conf file on my device to have a ‘compatible’ spec that matched one of the default specs in l4t_generate_soc_bup.sh. This let the install proceed, but the result was the same upon booting. Here is the compatible spec I added to nv_boot_control.conf:

COMPATIBLE_SPEC 3448-300-0002--1-0-jetson-nano-emmc-mmcblk0p1

And here is the result after rebooting:

[0000.205] [L4T TegraBoot] (version 00.00.2018.01-l4t-2d758fea)
[0000.211] Processing in cold boot mode Bootloader 2
[0000.216] A02 Bootrom Patch rev = 1023
[0000.219] Power-up reason: software reset
[0000.223] No Battery Present
[0000.226] pmic max77620 reset reason
[0000.229] pmic max77620 NVERC : 0x0
[0000.232] RamCode = 0
[0000.235] Platform has DDR4 type RAM
[0000.238] max77620 disabling SD1 Remote Sense
[0000.242] Setting DDR voltage to 1125mv
[0000.246] Serial Number of Pmic Max77663: 0x335d9
[0000.254] Entering ramdump check
[0000.257] Get RamDumpCarveOut = 0x0
[0000.260] RamDumpCarveOut=0x0,  RamDumperFlag=0xe59ff3f8
[0000.265] Last reboot was clean, booting normally!
[0000.270] Sdram initialization is successful
[0000.274] SecureOs Carveout Base=0x00000000ff800000 Size=0x00800000
[0000.280] Lp0 Carveout Base=0x00000000ff780000 Size=0x00001000
[0000.286] BpmpFw Carveout Base=0x00000000ff700000 Size=0x00080000
[0000.292] GSC1 Carveout Base=0x00000000ff600000 Size=0x00100000
[0000.297] GSC2 Carveout Base=0x00000000ff500000 Size=0x00100000
[0000.303] GSC4 Carveout Base=0x00000000ff400000 Size=0x00100000
[0000.309] GSC5 Carveout Base=0x00000000ff300000 Size=0x00100000
[0000.315] GSC3 Carveout Base=0x000000017f300000 Size=0x00d00000
[0000.331] RamDump Carveout Base=0x00000000ff280000 Size=0x00080000
[0000.337] Platform-DebugCarveout: 0
[0000.340] Nck Carveout Base=0x00000000ff080000 Size=0x00200000
[0000.346] Non secure mode, and RB not enabled.
[0000.363] Csd NumOfBlocks=0
[0000.606] *** Booting BFS0.
[0000.608] Read PT from (0:3)
[0000.615] Using BFS PT to query partitions
[0000.619] PT: Partition LNX NOT found !
[0001.099] *** Failing over to KFS1.
[0000.292] [L4T TegraBoot] (version 00.00.2018.01-l4t-2d758fea)
[0000.298] Processing in cold boot mode Bootloader 2
[0000.303] A02 Bootrom Patch rev = 1023
[0000.306] Power-up reason: software reset
[0000.310] No Battery Present
[0000.312] pmic max77620 reset reason
[0000.316] pmic max77620 NVERC : 0x0
[0000.319] RamCode = 0
[0000.321] Platform has DDR4 type RAM
[0000.325] max77620 disabling SD1 Remote Sense
[0000.329] Setting DDR voltage to 1125mv
[0000.333] Serial Number of Pmic Max77663: 0x335d9
[0000.340] Entering ramdump check
[0000.343] Get RamDumpCarveOut = 0x0
[0000.347] RamDumpCarveOut=0x0,  RamDumperFlag=0xe59ff3f8
[0000.352] Last reboot was clean, booting normally!
[0000.356] Sdram initialization is successful
[0000.361] SecureOs Carveout Base=0x00000000ff800000 Size=0x00800000
[0000.367] Lp0 Carveout Base=0x00000000ff780000 Size=0x00001000
[0000.372] BpmpFw Carveout Base=0x00000000ff700000 Size=0x00080000
[0000.378] GSC1 Carveout Base=0x00000000ff600000 Size=0x00100000
[0000.384] GSC2 Carveout Base=0x00000000ff500000 Size=0x00100000
[0000.390] GSC4 Carveout Base=0x00000000ff400000 Size=0x00100000
[0000.396] GSC5 Carveout Base=0x00000000ff300000 Size=0x00100000
[0000.402] GSC3 Carveout Base=0x000000017f300000 Size=0x00d00000
[0000.418] RamDump Carveout Base=0x00000000ff280000 Size=0x00080000
[0000.424] Platform-DebugCarveout: 0
[0000.427] Nck Carveout Base=0x00000000ff080000 Size=0x00200000
[0000.433] Non secure mode, and RB not enabled.
[0000.450] Csd NumOfBlocks=0
[0000.690] *** Booting BFS1.
[0000.693] Read PT from (0:3)
[0000.700] Using BFS PT to query partitions
[0000.704] PT: Partition LNX NOT found !
[0001.184] *** Failing over to KFS2.

I attached the full install log as before. BootLog_2020-06-15_12_ForumPostBUPUpdateLog01.txt (35.6 KB)

Thanks,

Chris Richardson

Hi,

Per checked with internal engineer, your change to hardcode the FAB/BOARDSKU should be okay. But we wonder the “small partition” before APP may cause problem.

What we want is

  1. Fresh flash log from host device.
  2. BUP generation log from host device.
  3. ls /dev/disk/by-partlabel/ -l