OTA fails on Nx-devkit when system is installed on nvme

david.L · February 10, 2022, 2:28am

Hi:

I want to test the OTA function of NX on NX-DEVKIT. My os is installed on nvme, and the system version is 32.6.

1.Generate the base recovery image and recovery DTB：
sudo ./tools/ota_tools/version_upgrade/build_base_recovery_image.sh jetson-xavier-nx-devkit R32-6 ${BASE_BSP} ${BASE_BSP}/rootfs ${TARGET_BSP}

2.Generate the OTA update payload package：
sudo ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh jetson-xavier-nx-devkit R32-6

3.Unpack the OTA payload package and prepare to start OTA:
sudo ./nv_ota_start.sh /dev/mmcblk0 /ota/ota_payload_package.tar.gz

The result：

The board name in OTA package(jetson-xavier-nx-devkit) does not match current board(jetson-xavier-nx-devkit-qspi)
Failed to run "check_target_board /ota_work TARGET_BOARD"

Is this correct?

WayneWWW · February 10, 2022, 3:03am

How did you set up your system to NVMe previously?

david.L · February 10, 2022, 3:13am

I used script nvsdkmanager_flash.sh.

sudo ./nvsdkmanager_flash.sh -s nvme0n1p1

carolyuu · February 11, 2022, 7:34am

Hi 529161095,

The nvsdkmanager_flash.sh script is use “jetson-xavier-nx-devkit-qspi” config to flash image, so when you run the OTA update, you will get not match target_board errors.

There are two solution can fix this issue:

Replace jetson-xavier-nx-devkit-qspi to jetson-xavier-nx-devkit in /etc/nv_boot_control.conf as below:

TNSPEC 3668-200-0000-G.0-1-2-jetson-xavier-nx-devkit-nvme0n1p1
COMPATIBLE_SPEC 3668-100---1--jetson-xavier-nx-devkit-
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mtdblock0
TEGRA_OTA_GPT_DEVICE /dev/mtdblock0

Use l4t_initrd_flash.sh script to flash image on NVMe.
example:

$ sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 -c ./tools/kernel_flash/flash_l4t_nvme.xml -S 90GiB --showlogs jetson-xavier-nx-devkit nvme0n1p1

Note: The -S size cannot be bigger than “num_sectors” * “sector_size”, please check README for more detail.

david.L · February 23, 2022, 5:55am

I used l4t_initrd_flash.sh script to flash image on NVMe.

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 -c ./tools/kernel_flash/flash_l4t_nvme.xml -S 90GiB --showlogs jetson-xavier-nx-devkit nvme0n1p1

2.The “/etc/nv_boot_control.conf” in nx-devkit :

TNSPEC 3668-200-0000-H.0-1-2-jetson-xavier-nx-devkit-qspi-internal
COMPATIBLE_SPEC 3668-100---1--jetson-xavier-nx-devkit-qspi-
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mtdblock0
TEGRA_OTA_GPT_DEVICE /dev/mtdblock0

I modified as you suggested：

TNSPEC 3668-200-0000-G.0-1-2-jetson-xavier-nx-devkit-nvme0n1p1
COMPATIBLE_SPEC 3668-100---1--jetson-xavier-nx-devkit-
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mtdblock0
TEGRA_OTA_GPT_DEVICE /dev/mtdblock0

3.start OTA

sudo ./nv_ota_start.sh /dev/nvme0n1 /ota/ota_payload_package.tar.gz

The logs is:

Command: ./nv_ota_start.sh /dev/nvme0n1 /ota/ota_payload_package.tar.gz
init_ota_log /ota_log
Create log file at /ota_log/ota_20220223-132941.log
OTA_LOG_FILE=/ota_log/ota_20220223-132941.log
Extract /ota/ota_payload_package.tar.gz
update_nv_boot_control_in_rootfs /ota_work
3668-100---1--jetson-xavier-nx-devkit-
check_prerequisites
get_chip_id chip_id
nv_ota_update_without_layout_change.sh /ota_work/ota_package.tar.gz
Command: nv_ota_update_without_layout_change.sh /ota_work/ota_package.tar.gz
decompress_ota_package /ota_work/ota_package.tar.gz /ota_work
decompress_ota_package: start at 2022年 02月 23日 星期三 13:30:48 CST
Sha1 checksum for /ota_work/ota_package.tar (e6b0363edf6764624033f91835a3cf08a7cb03d2) matches
decompress_ota_package: end at 2022年 02月 23日 星期三 13:33:19 CST
check_target_board /ota_work TARGET_BOARD
get_chip_id CHIP_ID
update_utilities_for_BUP_update /ota_work
enable_a_b_redundancy
Nvidia A/B-Redundancy Update tool Version 2.1
enabling A/B redundancy
A/B Redundancy has been enabled.
get_rootfs_a_b_enabled ROOTFS_AB_ENABLED UNIFIED_AB_ENABLED
ROOTFS_AB_ENABLED=0
UNIFIED_AB_ENABLED=0
check_bsp_version /ota_work BASE_VERSION
get_update_control /ota_work UPDATE_BOOTLOADER UPDATE_ROOTFS
UPDATE_BOOTLOADER=1, UPDATE_ROOTFS=1
update_bootloader /ota_work 0 0x19
get_update_slot bootloader 0 update_slot
update_slot=B
update_bootloader_with_UE /ota_work 0x19
Nvidia A/B-Redundancy Update tool Version 2.1
verifying update
Verify bootloader update begins.
The rotate count has been restored.
The current slot 0 is marked as boot successful
SM: S1
The priority of current slot 0 has been restored.
Verify rootfs update begins.
RootFS A/B is not enabled, verification finishes.
Nvidia A/B-Redundancy Update tool Version 2.1
Got payload file: /ota_work/bl_update_payload
current slot 0
SM: S11
Set slot 1 as unbootable and start updating.
Start running: /usr/sbin/nv_bootloader_payload_updater --no-dependent-partition /ota_work/bl_update_payload
Got update payload: /ota_work/bl_update_payload
Tegra User Block Device: /dev/disk/by-partlabel
Tegra Boot Block Device: /dev/mtdblock0
HEADER: MAGIC NVIDIA__BLOB__V2
HEX_VALUE 16909857
BLOB_SIZE 49900468
HEADER_SIZE 48
NUMBER_OF_ELEMENTS 29
HEADER_TYPE 0
UNCOMP_SIZE 49900468
MB1_RATCHET_LV 0
MTS_RATCHET_LV 0
ROLLBACK_FUSE_LV 0
Device TN Spec: 3668-200-0000-H.0-1-2-jetson-xavier-nx-devkit-nvme0n1p1
Device Compatible Spec: 3668-100---1--jetson-xavier-nx-devkit-
Device TN Spec: 3668-200-0000-H.0-1-2-jetson-xavier-nx-devkit-nvme0n1p1
Device Compatible Spec: 3668-100---1--jetson-xavier-nx-devkit-
Device is fused board.
ENTRY_TABLE:
PART  POS  LEN  VER TNSPEC TYPE UPDATABLE
spe-fw  3528  94960  12897    0  1
mb2  98488  181232  12897    0  1
cpu-bootloader  279720  469840  12897    0  1
secure-os  749560  410560  12897    0  1
bpmp-fw  1160120  856352  12897    0  1
eks  2016472  5136  12897    0  1
adsp-fw  2021608  81312  12897    0  1
rce-fw  2102920  272752  12897    0  1
mts-preboot  2375672  24016  12897    0  1
mts-mce  2399688  143200  12897    0  1
mts-proper  2542888  3430416  12897    0  1
sc7  5973304  65504  12897    0  1
bpmp-fw-dtb  6038808  199808  12897  3668-100-0000--1-2-jetson-xavier-nx-devkit-mmcblk0p1  0  1
bpmp-fw-dtb  6238616  199808  12897  3668-301-0000--1-2-jetson-xavier-nx-devkit-mmcblk0p1  0  0
bootloader-dtb  6438424  208272  12897  3668-100-0000--1-2-jetson-xavier-nx-devkit-mmcblk0p1  0  1
bootloader-dtb  6646696  208272  12897  3668-301-0000--1-2-jetson-xavier-nx-devkit-mmcblk0p1  0  0
VER  6854968  102  12897  3668-100-0000--1-2-jetson-xavier-nx-devkit-mmcblk0p1  0  1
VER  6855070  102  12897  3668-301-0000--1-2-jetson-xavier-nx-devkit-mmcblk0p1  0  0
mb1  6855172  249312  12897  3668-100-0000--1-2-jetson-xavier-nx-devkit-mmcblk0p1  2  1
mb1  7104484  249312  12897  3668-301-0000--1-2-jetson-xavier-nx-devkit-mmcblk0p1  2  0
BCT  7353796  2888  12897  3668-100-0000--1-2-jetson-xavier-nx-devkit-mmcblk0p1  2  1
BCT  7356684  2888  12897  3668-301-0000--1-2-jetson-xavier-nx-devkit-mmcblk0p1  2  0
MB1_BCT  7359572  30240  12897  3668-100-0000--1-2-jetson-xavier-nx-devkit-mmcblk0p1  0  1
MB1_BCT  7389812  30240  12897  3668-301-0000--1-2-jetson-xavier-nx-devkit-mmcblk0p1  0  0
MEM_BCT  7420052  198656  12897  3668-100-0000--1-2-jetson-xavier-nx-devkit-mmcblk0p1  0  1
MEM_BCT  7618708  198656  12897  3668-301-0000--1-2-jetson-xavier-nx-devkit-mmcblk0p1  0  0
kernel  7817364  41666560  12897    1  1
kernel-dtb  49483924  208272  12897  3668-100-0000--1-2-jetson-xavier-nx-devkit-mmcblk0p1  1  1
kernel-dtb  49692196  208272  12897  3668-301-0000--1-2-jetson-xavier-nx-devkit-mmcblk0p1  1  0
Saving Entry table to /opt/ota_package/entry_table
spe-fw write: slot = 1 offset = 1572864 bytes = 94960
mb2 write: slot = 1 offset = 2097152 bytes = 181232
cpu-bootloader write: slot = 1 offset = 13500416 bytes = 469840
secure-os write: slot = 1 offset = 18874368 bytes = 410560
bpmp-fw write: slot = 1 offset = 29491200 bytes = 856352
eks write: slot = 1 offset = 21561344 bytes = 5136
adsp-fw write: slot = 1 offset = 22675456 bytes = 81312
rce-fw write: slot = 1 offset = 24772608 bytes = 272752
mts-preboot write: slot = 1 offset = 2424832 bytes = 24016
mts-mce write: slot = 1 offset = 2686976 bytes = 143200
mts-proper write: slot = 1 offset = 7077888 bytes = 3430416
sc7 write: slot = 1 offset = 11403264 bytes = 65504
bpmp-fw-dtb write: slot = 1 offset = 32112640 bytes = 199808
bootloader-dtb write: slot = 1 offset = 15400960 bytes = 208272
VER write: slot = 1 offset = 33357824 bytes = 102
MB1_BCT write: slot = 1 offset = 720896 bytes = 30240
MEM_BCT write: slot = 1 offset = 1048576 bytes = 198656
Writing 41666560 bytes to /dev/disk/by-partlabel/kernel_b for kernel
Writing 208272 bytes to /dev/disk/by-partlabel/kernel-dtb_b for kernel-dtb
Update bup successfully
SM: S12
Setting slot 1 as active boot slot
Nvidia A/B-Redundancy Update tool Version 2.1
Got payload file: /ota_work/xusb_only_payload
current slot 0
SM: S11
Set slot 1 as unbootable and start updating.
Start running: /usr/sbin/nv_bootloader_payload_updater --no-dependent-partition /ota_work/xusb_only_payload
Got update payload: /ota_work/xusb_only_payload
Tegra User Block Device: /dev/disk/by-partlabel
Tegra Boot Block Device: /dev/mtdblock0
HEADER: MAGIC NVIDIA__BLOB__V2
HEX_VALUE 16909857
BLOB_SIZE 133288
HEADER_SIZE 48
NUMBER_OF_ELEMENTS 1
HEADER_TYPE 0
UNCOMP_SIZE 133288
MB1_RATCHET_LV 0
MTS_RATCHET_LV 0
ROLLBACK_FUSE_LV 0
Device TN Spec: 3668-200-0000-H.0-1-2-jetson-xavier-nx-devkit-nvme0n1p1
Device Compatible Spec: 3668-100---1--jetson-xavier-nx-devkit-
Device TN Spec: 3668-200-0000-H.0-1-2-jetson-xavier-nx-devkit-nvme0n1p1
Device Compatible Spec: 3668-100---1--jetson-xavier-nx-devkit-
Device is fused board.
ENTRY_TABLE:
PART  POS  LEN  VER TNSPEC TYPE UPDATABLE
xusb-fw  168  133120  12897    0  1
Saving Entry table to /opt/ota_package/entry_table
xusb-fw write: slot = 1 offset = 11862016 bytes = 133120
GetEntryTable failed for mb1
Skipping partition: mb1
GetEntryTable failed for BCT
Skipping partition: BCT
GetEntryTable failed for BCT
Skipping partition: BCT
GetEntryTable failed for mb1
Skipping partition: mb1
Update bup successfully
SM: S12
Setting slot 1 as active boot slot
Copy /opt/ota_package/entry_table into /ota_work/entry_table.ota_backup
Bootloader on non-current slot(B) is updated
update_rootfs /ota_work 0
update_rootfs_with_a_b_disabled /ota_work
write_base_recovery /ota_work
Verifying image /ota_work/recovery.img.R32_3_4 with sha1 chksum file /ota_work/recovery.img.R32_3_4.sha1sum
Sha1 checksum for /ota_work/recovery.img.R32_3_4 (bcf90830ae148d659aef6d6a37545ba40a8819e4) matches
Verifying image /ota_work/recovery.dtb.R32_3_4 with sha1 chksum file /ota_work/recovery.dtb.R32_3_4.sha1sum
Sha1 checksum for /ota_work/recovery.dtb.R32_3_4 (e15debad31566963bc76df34731f0c7a32e1abc3) matches
Backed up kernel and kernel-dtb partition under /ota_work before writing them
Writing base recovery image into /dev/disk/by-partlabel/recovery
Read back base recovery image into /ota_work/image.tmp and verify it
Reading 49487872 bytes from /dev/disk/by-partlabel/recovery: 1KB block=48328 remainder=0 offset=49487872
Verifying image /ota_work/image.tmp with sha1 chksum file /ota_work/recovery.img.R32_3_4.sha1sum
Sha1 checksum for /ota_work/image.tmp (bcf90830ae148d659aef6d6a37545ba40a8819e4) matches
Writing base recovery dtb into /dev/disk/by-partlabel/recovery-dtb
Read back base recovery dtb into /ota_work/image.tmp and verify it
Reading 208928 bytes from /dev/disk/by-partlabel/recovery-dtb: 1KB block=204 remainder=32 offset=208896
Verifying image /ota_work/image.tmp with sha1 chksum file /ota_work/recovery.dtb.R32_3_4.sha1sum
Sha1 checksum for /ota_work/image.tmp (e15debad31566963bc76df34731f0c7a32e1abc3) matches
write_kernel_bootctrl /ota_work images-R32-ToT
512+0 records in
512+0 records out
262144 bytes (262 kB, 256 KiB) copied, 0.00452857 s, 57.9 MB/s
Backed up kernel-bootctrl partition under /ota_work before writing them
Writing bootctrl update file into /dev/disk/by-partlabel/kernel-bootctrl
Read back bootctrl update file into /ota_work/image.tmp and verify it
Reading 20 bytes from /dev/disk/by-partlabel/kernel-bootctrl: 1KB block=0 remainder=20 offset=0
backup_extlinux_conf /ota_work
Backing up /boot/extlinux/extlinux.conf
Rootfs is to be updated in recovery kernel once device is rebooted.

4.Then I restarted the device, but the device didn’t boot properly and kept restarting.
The log printed by the serial port is：
start-log.txt (94.5 KB)

Where is the problem?
Thanks！

carolyuu · February 23, 2022, 7:22am

Hi fnaiot,

If you are using ‘l4t_initrd_flash.sh’ script to flash image on NVMe, you don’t need modify ‘nv_boot_control.conf’ file., the config file will show correct data.

What is your BASE_BSP, ToT_BSP and TARGET_BSP?
Are you flash r32.6.1 image on Xavier-NX-devkit?

david.L · February 23, 2022, 7:42am

The BASE_BSP is the default build system by SDKManger.
ToT_BSP and TARGET_BSP is the same path.Compared with the BASE_BSP system, I modified the kernel, device tree, and installed some debs.
They can all be run individually.
I flash R32.6.1 image on NX-devkit.

WayneWWW · February 23, 2022, 7:58am

Hi,

I think the real thing to point out here is you have some customization and we don’t know whether those customization is causing this trouble.

From the beginning, @carolyuu is doing her test under different scenario as your case.

Please use pure jetpack to validate this problem first and then start your customization.

david.L · February 23, 2022, 10:49am

Thanks.
I use pure jetpack to test the OTA.
The BASE_BSP, ToT_BSP and TARGET_BSP is the same path.And it worked.
Maybe there is something wrong with my rootfs.
Is there a way to copy the system.img directly from the device and then make an OTA file?

david.L · February 24, 2022, 9:14am

I know why it fails.

Linux_for_Tegra/tools/kernel_flash/flash_l4t_nvme.xml

The “num_sectors” doesn’t match my nvme.

system · March 10, 2022, 9:15am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.