nv_update_engine not updating from R32.2 to R32.2.1

Ursula · December 12, 2019, 4:35am

I’m trying to upgrade a board’s bootloader stack from R32.2 to R32.2.1 with nv_update_engine; things don’t seem to be behaving. I have generated the BUP and I get the following output from nv_update_engine in Linux, all looks good:

Nvidia A/B-Redundancy Update tool Version 1.2
current slot 0
Start updating slot 1
Tegra User Block Device: /dev/disk/by-partlabel
Tegra Boot Block Device: /dev/mmcblk0boot0
HEADER: MAGIC NVIDIA__BLOB__V2
HEX_VALUE 131072
BLOB_SIZE 5178144
HEADER_SIZE 48
NUMBER_OF_ELEMENTS 16
HEADER_TYPE 0
UNCOMP_SIZE 5178144
MB1_RATCHET_LV 0
MTS_RATCHET_LV 0
ROLLBACK_FUSE_LV 0
Device TN Spec: 3310-D01-1000-B.0-1-0-my-board-tx2-mmcblk0p3
Device is fused board.
ENTRY_TABLE:
PART  POS  LEN  VER TNSPEC TYPE UPDATABLE
bpmp-fw  1968  533904  2    1  1
sce-fw  535872  125568  2    1  1
cpu-bootloader  661440  278176  2    1  1
mb2  939616  111648  2    0  1
spe-fw  1051264  95984  2    0  1
secure-os  1147248  313152  2    1  1
mb1  1460400  99664  2    2  1
mts-bootpack  1560064  2082144  2    1  1
mts-preboot  3642208  63104  2    0  1
sc7  3705312  23632  2    1  1
bpmp-fw-dtb  3728944  466240  2  3310-D01-1000-B.0-1-0-my-board-tx2-mmcblk0p3  1  1
bootloader-dtb  4195184  223840  2  3310-D01-1000-B.0-1-0-my-board-tx2-mmcblk0p3  1  1
BCT  4419024  3584  2  3310-D01-1000-B.0-1-0-my-board-tx2-mmcblk0p3  2  1
MB1_BCT  4422608  50016  2  3310-D01-1000-B.0-1-0-my-board-tx2-mmcblk0p3  0  1
kernel  4472624  481680  2    1  1
kernel-dtb  4954304  223840  2  3310-D01-1000-B.0-1-0-my-board-tx2-mmcblk0p3  1  1
Saving Entry table to /opt/ota_package/entry_table
Writing 533904 bytes to /dev/disk/by-partlabel/bpmp-fw_b for bpmp-fw
Writing 125568 bytes to /dev/disk/by-partlabel/sce-fw_b for sce-fw
Writing 278176 bytes to /dev/disk/by-partlabel/cpu-bootloader_b for cpu-bootloader
mb2 write: slot = 1 offset = 1276928 bytes = 111648
spe-fw write: slot = 1 offset = 883712 bytes = 95984
Writing 313152 bytes to /dev/disk/by-partlabel/secure-os_b for secure-os
Writing 2082144 bytes to /dev/disk/by-partlabel/mts-bootpack_b for mts-bootpack
mts-preboot write: slot = 1 offset = 1801216 bytes = 63104
Writing 23632 bytes to /dev/disk/by-partlabel/sc7_b for sc7
Writing 466240 bytes to /dev/disk/by-partlabel/bpmp-fw-dtb_b for bpmp-fw-dtb
Writing 223840 bytes to /dev/disk/by-partlabel/bootloader-dtb_b for bootloader-dtb
MB1_BCT write: slot = 1 offset = 622592 bytes = 50016
Writing 481680 bytes to /dev/disk/by-partlabel/kernel_b for kernel
Writing 223840 bytes to /dev/disk/by-partlabel/kernel-dtb_b for kernel-dtb
Update successful
Setting slot 1 as active boot slot

However, on the reboot, I see:

[0000.159] C> ERROR: Highest Layer Module = 0x40, Lowest Layer Module = 0x40,
Aux Info = 0x0, Reason = 0xd
[0000.139] C> ERROR: Highest Layer Module = 0x40, Lowest Layer Module = 0x40,
Aux Info = 0x0, Reason = 0xd
[0000.139] C> ERROR: Highest Layer Module = 0x40, Lowest Layer Module = 0x40,
Aux Info = 0x0, Reason = 0xd
[0000.139] C> ERROR: Highest Layer Module = 0x40, Lowest Layer Module = 0x40,
Aux Info = 0x0, Reason = 0xd
[0000.140] C> ERROR: Highest Layer Module = 0x40, Lowest Layer Module = 0x40,
Aux Info = 0x0, Reason = 0xd
[0000.140] C> ERROR: Highest Layer Module = 0x40, Lowest Layer Module = 0x40,
Aux Info = 0x0, Reason = 0xd
[0000.140] C> ERROR: Highest Layer Module = 0x40, Lowest Layer Module = 0x40,
Aux Info = 0x0, Reason = 0xd
[0000.194] I> Welcome to MB2(TBoot-BPMP)(version: 01.00.160913-t186-M-00.00-mobile-1b47d5d9)
[0000.248] I> A/B: bin_type (16) slot 0
[0000.252] I> Loading partition bpmp-fw at 0xd7800000
[0000.283] I> A/B: bin_type (17) slot 0
[0000.287] I> Loading partition bpmp-fw-dtb at 0xd79f0000
[0000.490] I> Loading SCE-FW ...
[0000.493] I> A/B: bin_type (12) slot 0
[0000.497] I> Loading partition sce-fw at 0xd7300000
[0000.525] I> Loading APE-FW ...
[0000.528] I> A/B: bin_type (11) slot 0
[0000.531] I> Loading partition adsp-fw at 0xd7400000
[0000.560] I> A/B: bin_type (13) slot 0
[0000.563] I> Loading partition cpu-bootloader at 0x96000000
[0000.591] I> A/B: bin_type (20) slot 0
[0000.595] I> Loading partition bootloader-dtb at 0x8520f400
[0000.622] I> A/B: bin_type (14) slot 0
[0000.626] I> Loading partition secure-os at 0x8530f600
[0001.307] I> A/B: bin_type (21) slot 0
[0001.311] I> Loading partition kernel-dtb at 0x80000000 from device(0x1)
[0001.783] I> Loading kernel/boot.img from built-in storage ...
[0001.788] I> A/B: bin_type (24) slot 0
[0002.558] I> Loading partition kernel at 0x80400000 from device(0x1)

Those error messages don’t look good. And then slot #0 is used instead of slot#1, so the upgrade seems to be getting rejected. The documentation makes reference to a debug build of MB1 “that has [debug] messages available”. Is this available anywhere?

Edit: I don’t see the ERROR messages if I try to do a BUP from R32.2 to R32.2 (i.e. keep the same version of L4T). It looks like upgrading across releases does not work. However, even if I try to do just an R32.2 to R32.2 upgrade, things aren’t working right, as nvbootctrl get-current-slot reports back 0 after the reboot even though slot 1 was actually used for the DTB & u-boot binaries during the boot.

carolyuu · December 12, 2019, 7:08am

Hi Ursula,

Are you flash R32.2 on slot 0? then update R32.2.1 u-boot.bin on slot 1?
Please share your detail steps. Thanks!

Ursula · December 12, 2019, 10:06am

Hi there,

Yes, steps are:

a) flash everything with R32.2 BSP;
b) generate BUP using R32.2.1 BSP;
c) apply BUP using nv_update_engine – see ERROR messages on reboot;

If I instead do:

a) flash everything with R32.2 BSP;
b) generate BUP using R32.2 BSP;
c) apply BUP using nv_update_engine – no ERROR messages, but slot 0 still used for MB2, although slot 1 is correctly used for DTB/u-boot as expected. However, this is still a problem as nvbootctrl / nv_update_engine still think the active boot slot is 0, so slot 1 is not marked as successful by the --verify.

Thanks,
Ursula

JerryChang · January 10, 2020, 3:27am

hello Ursula,

may I know what’s your purpose for enable A/B redundancy, A/B redundancy is a feature added on top of A/B update to provide Bootloader redundancy. it’s prevent the risk of booting failure after a bootloader update.

if you’re going to have software release upgrade, you’ll need to flash the whole image instead.
we’re able to support Over-the-Air (OTA) Update starting from l4t-r32.3.1, you may also refer to Topic 1068980 for more details of OTA update.

please based-on the same l4t software release and developer your bootloader binaries.
one tricks for A/B redundancy verification is corruption one of booloader partition,
you may corrupt the cpu-bootloader as below.
for example,

$ sudo dd if=/dev/zero of=/dev/disk/by-partlabel/cpu-bootloader bs=512

Ursula · January 10, 2020, 10:11am

Hi JerryChang,

I took a look at the OTA update topic that you linked to. This will not work for my use-case as I have a custom board (with modified U-Boot) rather than a standard Jetson TX2 developer kit. So I need to replicate the functionality of the nvidia-l4t-bootloader package but with my own payloads.

I will first retry with R32.3.1 to see if I can still replicate the same issue, and will then update the topic with more information.

Thanks,
Ursula

Ursula · January 13, 2020, 4:27pm

Trying to upgrade from R32.2.1 to R32.3.1 still results in the same problem. Perhaps this is as expected if OTA is only supported from R32.3.1 onwards. How can I generate a BUP with a different set of bootloader code to prove that upgrades to a new version will “really” work once a new L4T is released in the future (e.g. let’s say R32.4)?

Another problem I’m seeing is that nvbootctrl seems to be confused about the active slot. If I do a R32.3.1 to R32.3.1 bootloader update and reboot, I can see from the serial logs that the new slot (slot 1 / _b) is being used for the next boot. All good there. However, the userspace tools still seem to think the boot slot being used for the boot is the old slot, for some reason, so they don’t mark slot 1 as successful:

root@hw:/# nvbootctrl dump-slots-info
magic:0x43424e00,             version: 3             features: 3             num_slots: 2
slot: 0,             priority: 14,             suffix: _a,             retry_count: 7,             boot_successful: 1
slot: 1,             priority: 15,             suffix: _b,             retry_count: 6,             boot_successful: 0
root@hw:/# nv_update_engine --verify
Nvidia A/B-Redundancy Update tool Version 1.2
verifying update
The current slot 0 is marked as boot successful
root@hw:/# nvbootctrl get-current-slot
0

Why is nvbootctrl thinking the current slot is zero, when the current slot is really slot 1? Is the source code to nvbootctrl available?

Thanks,
Ursula

JerryChang · January 14, 2020, 2:40am

hello Ursula,

it seems there’re two questions in the same discussion thread, one for bootloader redundancy the other for software upgrade, let’s separate them for better supports.
suggest you should create another new topic for software upgrade, and OTA questions.
regarding to bootloader redundancy issue, it usually one of *_b partition boot-up failed.
could you please share the whole UART messages, to review the bootloader logs.
thanks

Ursula · January 22, 2020, 3:07am

After a bit more investigation, the nvbootctrl issue I was seeing was due to not passing ${cbootargs} from u-boot into the kernel boot arguments: nvbootctrl was expecting to see “boot.slot_suffix=_b” in there, and wasn’t. After fixing this, all is now good with nvbootctrl.

So my only remaining question on this topic is, going back the original issue; that is, how can I verify that upgrading from R32.3.1 to a later version of bootloader will work successfully, as going from R32.2.1 to R32.3.1 isn’t working?

JerryChang · January 30, 2020, 2:08am

hello Ursula,

it’s a new feature of l4t release r32.3.1, Over-the-Air (OTA) Update.
if you’re based-on l4t-r32.2.x, you’ll need to flash your board through usb port for software upgrade.
thereafter, you’ll be able to have software upgrade without flashing after r32.3.x release.
thanks

Ursula · January 30, 2020, 8:57am

Thanks JerryChang, that’s all clear!