Bootloader Update / Redundancy

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

I am having difficulty getting bootloader updates on my target working. I am trying to follow the directions in the L4T_28.2 documentation on Bootloader Update / Redundancy but it isn’t working and I need some guidance on debugging. Here are the steps that I took:

  1. I updated the slot_metadata.bin by:
  • Editing smd_info.cfg so that it was: 
# SMD metadata information
< VERSION 3 >

#
# Config 2: Enable redundancy support (by removing comments ##)
#
< REDUNDANCY_USER 1 >

# slot info order is important!
# <priority>    <suffix>     <retry_count>  <boot_successful>
15                  _a          7               1
14                  _b          7               1
  • $ sudo nv_smd_generator smd_info.cfg slot_metadata.bin
  • Reprogrammed my board with ./flash.sh
  • Created an upgrade file and attempted an upgrade:
    • sudo ./build_l4t_bup.sh jetson-tx2 mmcblk0p1 (Note I tried the documented methods to do this without a board connected, but it didn't seem to work for me)
    • Used scp to copy the resulting bl_update_payload to /opt/ota_package on my Tx2 based system
    • Ran nv_update_engine (and saw failure "GetEntryTable failed"): 
    $ sudo nv_update_engine --install
Tegra User Block Device: /dev/disk/by-partlabel
Tegra Boot Block Device: /dev/mmcblk0boot0
HEADER: MAGIC NVIDIA__BLOB__V2
HEX_VALUE 131072
BLOB_SIZE 4968688
HEADER_SIZE 48
NUMBER_OF_ELEMENTS 16
HEADER_TYPE 0
UNCOMP_SIZE 4968688
MB1_RATCHET_LV 0
MTS_RATCHET_LV 0
ROLLBACK_FUSE_LV 0
Device TN Spec: P2771-0000-C04-devkit.default
Device is fused board.
ENTRY_TABLE:
PART  POS  LEN  VER TNSPEC TYPE UPDATABLE
bpmp-fw  1968  529440  2    1  1
sce-fw  531408  76992  2    1  1
cpu-bootloader  608400  276384  2    1  1
mb2  884784  100960  2    0  1
spe-fw  985744  81936  2    0  1
secure-os  1067680  62976  2    1  1
kernel  1130656  508304  2    1  1
mb1  1638960  94736  2    2  1
mts-bootpack  1733696  2076832  2    1  1
mts-preboot  3810528  63024  2    0  1
sc7  3873552  21552  2    1  1
BCT  3895104  3584  2  3310-B02-fuselevel_production  2  0
MB1_BCT  3898688  50016  2  3310-B02-fuselevel_production  0  0
bpmp-fw-dtb  3948704  466160  2  3310-B02-fuselevel_production  1  0
bootloader-dtb  4414864  276912  2  3310-B02-fuselevel_production  1  0
kernel-dtb  4691776  276912  2  3310-B02-fuselevel_production  1  0
Writing 529440 bytes to /dev/disk/by-partlabel/bpmp-fw_b for bpmp-fw
Writing 76992 bytes to /dev/disk/by-partlabel/sce-fw_b for sce-fw
Writing 276384 bytes to /dev/disk/by-partlabel/cpu-bootloader_b for cpu-bootloader
mb2 write: slot = 1 offset = 1212416 bytes = 100960
spe-fw write: slot = 1 offset = 819200 bytes = 81936
Writing 62976 bytes to /dev/disk/by-partlabel/secure-os_b for secure-os
Writing 508304 bytes to /dev/disk/by-partlabel/kernel_b for kernel
Writing 2076832 bytes to /dev/disk/by-partlabel/mts-bootpack_b for mts-bootpack
mts-preboot write: slot = 1 offset = 1736704 bytes = 63024
Writing 21552 bytes to /dev/disk/by-partlabel/sc7_b for sc7
GetEntryTable failed 
Fail to write Dependence partitions 
Writing to partitions failed.
OTA Blob update failed. Status: 3
Nvidia A/B-Redundancy Update tool Version 1.1
current slot 0
Start updating slot 1
Slot 1 update failed
  • Ran various nvbootctrl commands in hopes of debugging: 
    • $ sudo nvbootctrl dump-slots-info
magic:0x43424e00,             version: 3             features: 3             num_slots: 2
slot: 0,             priority: 15,             suffix: _a,             retry_count: 7,             boot_successful: 1
slot: 1,             priority: 0,             suffix: _b,             retry_count: 0,             boot_successful: 0

$ sudo bash -c 'nvbootctrl is-slot-bootable 1; if [ $? = 0 ]; then echo yes; else echo no; fi'
no

$ sudo bash -c 'nvbootctrl is-slot-bootable 0; if [ $? = 0 ]; then echo yes; else echo no; fi'
yes

$ sudo nvbootctrl get-number-slots
2

$ sudo nvbootctrl get-current-slot
0

    What else can I do to debug this? Can anyone give me guidance on getting this working? It looks like a lot of the update was successful, but failed midway through. I have a bunch of our products that I would love to update without having to go into recovery mode, and I would love to support this in our production software as well.

    Thanks,
    -David

    2

    David_N,

    I’ll help check this issue.

    Thanks.

    3

    Hi David,

    Did you hit any error when enabling A/B redundancy? Or just had it during updating bl payload?

    4

    Wayne,
    I believe I enabled it by updating the slot_metadata.bin and then reprogramming via RCM. But, I did also try from the embedded system and got the not very helpful error:

    $ sudo bash -c 'nv_update_engine --enable-ab | tee /var/log/nv_update2.log'
Nvidia A/B-Redundancy Update tool Version 1.1
enabling A/B redundancy
Enable A/B command failed. It could be due to A/B being enabled already.
Error: Enable A/B redundancy failed!

    However I think that it is enabled given the output

    $ sudo nvbootctrl get-number-slots
2
    5

    Thanks for sharing. So the result shows redundancy is enabled but log has error…

    I’ll try it on my device as well.

    6

    Wayne,
    I just had a thought to test if was enabled by seeing if there was an error on disable. There wasn’t, and then I was able to enable again successfully:

    $ sudo nv_update_engine --disable-ab
Nvidia A/B-Redundancy Update tool Version 1.1
disabling A/B redundancy
A/B Redundancy has been disabled.

$ sudo nv_update_engine --enable-ab
Nvidia A/B-Redundancy Update tool Version 1.1
enabling A/B redundancy
A/B Redundancy has been enabled.

    The install would still fail after disable/enable.

    7

    Please try below steps for updating.

    #Run below command to generate BUP package
$ sudo [env={value}, …] ./build_l4t_bup.sh [options] <target_board> <rootdev>
#Example for Jetson-TX2 FAB B00 Fused board:
$ sudo FAB=B00 BOARDID=3310 FUSELEVEL=fuselevel_production ./build_l4t_bup.sh jetson-tx2 mmcblk0p1
#The BPU package bl_update_payload can be found in bootloader folder

A. Full flash L4T TOT image 
B. Save entire uart logs and check mb2 & cpubl log

#Run nvbootctrl utility to show slot info:
$ sudo nvbootctrl dump-slots-info

#Update with given <BUP> and reboot until DUT boots into kernel
#To update BUP & reboot:
$ sudo mkdir -p /opt/ota_package
$ sudo cp <path>/bl_update_payload  /opt/ota_package
$ sudo nv_update_engine --install
#Save entire uart logs and check mb2 & cpubl log
    8

    David_N,

    Have you resolved your problem?

    9

    WayneWWW,

    I have been unsuccessful at running “nv_update_engine --install” on my target device using our altered partition map. I decided to return to the my Jetson-Tx2 development kit hardware to establish a baseline.

    1. I installed the Sample Root Filesystem and then performed a "apply_binaries.sh".
    2. Ran "sudo nv_update_engine --enable" on Tx2
    3. Ran "sudo ./build_l4t_bup.sh -u tx2_secureboot.pem jetson-tx2 mmcblk0p1"
    4. copied the bl_update_payload to the Tx2
    5. Ran "sudo nv_update_engine --install" on Tx2

    Using this configuration “nv_update_engine --install” ran to completion and counted down to a reboot, but then the system did not boot. There is no serial console to capture, even after a hard reset or power cycle.

    So I have not resolved this yet ;-( Different problems with baseline setup.

    10

    David_N,

    So your tools are running fine without any error log now. Do you have UART debugger to check the bootloader log?

    11

    hello David_N,

    is your Jetson-Tx2 fused ?
    also, could you please share the whole console messages of all your steps.
    thanks

    12

    Hi David_N,

    Any update? Or it’s not any issue now?

    Thanks