After the external NVMe is enabled,and rootfs redundancy is enabled, OTA cannot be upgraded

nvidias · January 7, 2022, 2:51am

Hi，

My Steps：

flash external NVMe with rootfs redundant enabled：

sudo ROOTFS_AB=1 ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1 -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_ab.xml   jetson-xavier nvme0n1p1

2.Generate BUP with the follow command ：

sudo  FAB=400 BOARDID=2888 FUSELEVEL=fuselevel_production BOARDSKU=0004 BOARDREV=K.0 ./build_l4t_bup.sh jetson-xavier nvme0n1p1

3.put the bl_update_payload into xavier target folder: /opt/ota_package
4.OTA upgrade with command: sudo nv_update_engine -i , but the result was a failure

the log:

[sudo] password for nvidia:
Nvidia A/B-Redundancy Update tool Version 2.1
current slot 0
SM: S11
Set slot 1 as unbootable and start updating.
Start running: /usr/sbin/nv_bootloader_payload_updater --no-dependent-partition /opt/ota_package/bl_update_payload
Got update payload: /opt/ota_package/bl_update_payload
Tegra User Block Device: /dev/disk/by-partlabel
Tegra Boot Block Device: /dev/mmcblk0boot0
HEADER: MAGIC NVIDIA__BLOB__V2
HEX_VALUE 16909857
BLOB_SIZE 50080934
HEADER_SIZE 48
NUMBER_OF_ELEMENTS 21
HEADER_TYPE 0
UNCOMP_SIZE 50080934
MB1_RATCHET_LV 0
MTS_RATCHET_LV 0
ROLLBACK_FUSE_LV 0
config COMPATIBLE_SPEC not found in /etc/nv_boot_control.conf
Device TN Spec: 2888-400-0004-K.0-1-2-jetson-xavier-nvme0n1p1
config COMPATIBLE_SPEC not found in /etc/nv_boot_control.conf
Device TN Spec: 2888-400-0004-K.0-1-2-jetson-xavier-nvme0n1p1
Device is fused board.
ENTRY_TABLE:
PART  POS  LEN  VER TNSPEC TYPE UPDATABLE
spe-fw  2568  94960  12897    1  1
mb2  97528  181232  12897    1  1
cpu-bootloader  278760  469840  12897    1  1
secure-os  748600  410560  12897    1  1
bpmp-fw  1159160  856352  12897    1  1
eks  2015512  5136  12897    1  1
adsp-fw  2020648  81312  12897    1  1
rce-fw  2101960  272752  12897    1  1
mts-preboot  2374712  24016  12897    1  1
mts-mce  2398728  143200  12897    1  1
mts-proper  2541928  3430416  12897    1  1
sc7  5972344  65504  12897    1  1
bpmp-fw-dtb  6037848  993104  12897  2888-400-0004-K.0-1-2-jetson-xavier-nvme0n1p1  1  1
bootloader-dtb  7030952  253168  12897  2888-400-0004-K.0-1-2-jetson-xavier-nvme0n1p1  1  1
VER  7284120  102  12897  2888-400-0004-K.0-1-2-jetson-xavier-nvme0n1p1  1  1
mb1  7284222  249312  12897  2888-400-0004-K.0-1-2-jetson-xavier-nvme0n1p1  2  1
BCT  7533534  2888  12897  2888-400-0004-K.0-1-2-jetson-xavier-nvme0n1p1  2  1
MB1_BCT  7536422  30864  12897  2888-400-0004-K.0-1-2-jetson-xavier-nvme0n1p1  1  1
MEM_BCT  7567286  198656  12897  2888-400-0004-K.0-1-2-jetson-xavier-nvme0n1p1  1  1
kernel  7765942  42061824  12897    1  1
kernel-dtb  49827766  253168  12897  2888-400-0004-K.0-1-2-jetson-xavier-nvme0n1p1  1  1
Saving Entry table to /opt/ota_package/entry_table
**Slot could not be opened spe-fw**
**spe-fw fail to write**
**Writing to partitions failed.**
OTA Blob update failed. Status: 5
/usr/sbin/nv_bootloader_payload_updater --no-dependent-partition failed.
**Slot 1 update failed**
**Error: installing bootloader updates failed: 1280**

The “nv_boot_control.conf” file contents:

TNSPEC 2888-400-0004-K.0-1-2-jetson-xavier-nvme0n1p1
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mmcblk0boot0
TEGRA_OTA_GPT_DEVICE /dev/mmcblk0boot1

How can I solve this problem?

Thanks!
BR.

nvidias · January 10, 2022, 6:02am

Hi,
Any suggestions?

BR.

WayneWWW · January 10, 2022, 9:40am

Hello,

Just want to double confirm,

Can you confirm the following two things:

Are you using the JP4.6?
Can you dump the eMMC boot partitions and send the binaries to us for analyzing?

Dump command:

sudo dd if=/dev/mmcblk0boot0 of=boot0.bin bs=1M;sync
sudo dd if=/dev/mmcblk0boot1 of=boot1.bin bs=1M;sync

nvidias · January 10, 2022, 10:47am

Hi WayneWWW，

Thanks for your reply！

yes，I use jetpack4.6
2.please see the attachment

Thank！
BR.
boot0.bin (8 MB)
boot1.bin (8 MB)

nvidias · January 11, 2022, 8:21am

Hi,

Is there any progress?

BR.

WayneWWW · January 12, 2022, 3:35am

Hi,

We cannot reproduce issue with the steps you provided.
The cause of your issue is due to missing GPT

Please
a. check definition: Linux_for_Tegra/bootloader/t186ref/cfg/flash_t194_sdmmc_rootfs_ab.xml, partition name “secondary_gpt” should be defined under device type “sdmmc_boot” section.

b. check the flash log after flashing: Linux_for_Tegra/initrdlog/flash_x-x.x_x_yyyymmdd_hhmmss.log, it should include writing secondary_gpt log as following:

writing item=17, 0:3:secondary_gpt, 16760320, 16896, gpt_secondary_0_3.bin, 16896, …

nvidias · January 12, 2022, 4:22am

Hi WayneWWW,

Thank you very much for your reply!

a. Yes,I checked the files and I found "secondary_gpt” under device type “sdmmc_boot” section,I didn’t actually change the file.
The picture for “flash_t194_sdmmc_rootfs_ab.xml” :

b. Yes, I can see the log :

........
Writing kernel-dtb_b partition done
Writing VER partition done
writing item=8, 9:0:recovery, 30199926784, 66060288, recovery_sigheader.img.encrypt, 57296896, fixed-<reserved>-7, b346fdeb70be81e4dd4ace068046e65ce921d8b0
**writing item=17, 0:3:secondary_gpt, 16760320, 16896, gpt_secondary_0_3.bin, 16896, fixed-<reserved>-0, 34d1e9eb17d1dc7e608b059a938969292caf35cb**
writing item=18, 1:3:master_boot_record, 0, 512, mbr_1_3.bin, 512, fixed-<reserved>-0, 694898d1c345bdb31b377790ed7fc0b0db184bf7
Writing recovery partition with recovery_sigheader.img.encrypt
Get size of partition through connection.
writing item=19, 1:3:primary_gpt, 512, 19968, gpt_primary_1_3.bin, 16896, fixed-<reserved>-0, e4598bd20cdba96042cd115ba14d45573a7ed0a7
........

It is important to note that I am using a 500GB NVMe SSD, so I changed the “num_sectors=“976773168” in the” Linux_for_Tegra/tools/kernel_flash/flash_l4t_nvme_rootfs_ab.xml "file and did not change anything else.

How can I fix this?

Thanks!
BR.

WayneWWW · January 12, 2022, 6:57am

What does your xml file look like now? Could you attach it?

nvidias · January 12, 2022, 7:04am

Hi WayneWWW,

Thanks for your reply!

Yes, please see the attchment file:
flash_l4t_nvme_rootfs_ab.xml (9.2 KB)

flash_t194_sdmmc_rootfs_ab.xml (36.1 KB)

Thanks!
BR.

WayneWWW · January 12, 2022, 7:44am

Hi @nvidias ,

Just want to confirm, is this really reproducible in each try? I mean if you have another clean setup AGX and nvme, would you hit same issue?

nvidias · January 12, 2022, 8:03am

Hi WayneWWW,

Thanks for your reply again!

That is a good question! In fact ,I have two Xaviers,each of them can reproduce the problem,
but since I don’t know how many times I repeat it and I’m not sure if it’s “Linux_for_Tegra” 's problem, I will delete the current folder and re-write xavier with SDKManage to get a new “Linux_for_Tegra”.

I’ll try again.
Can you compress the “Linux_for_Tegra” you are using and send it to me for a try?

Thanks!
BR.

WayneWWW · January 12, 2022, 8:06am

Just want to confirm, is "flash_l4t_nvme_rootfs_ab.xml " the only file that you modified or even the flash_t194_sdmmc_rootfs_ab.xml?

nvidias · January 12, 2022, 8:10am

Hi WayneWWW,

Thanks!

Yes, I only modified the "flash_l4t_nvme_rootfs_ab.xml ".

BR.

WayneWWW · January 12, 2022, 8:14am

Looks like we can reproduce it now. Will check.

nvidias · January 13, 2022, 2:54am

Hi WayneWWW,

Do you have any new findings?

My side with a new environment to try, can also reproduce the problem.
This problem can be repeated by modifying or not modifying the file "flash_l4t_nvme_rootfs_ab.xml ".

Thanks!
BR.

WayneWWW · January 13, 2022, 2:55am

This is a bug in initrd flash and we are checking it.

WayneWWW · January 14, 2022, 11:36am

Please try with this patch.

diff --git a/scripts/kernel_flash/l4t_flash_from_kernel.sh b/scripts/kernel_flash/l4t_flash_from_kernel.sh
index 7bae264..ad4b2de 100755
--- a/scripts/kernel_flash/l4t_flash_from_kernel.sh
+++ b/scripts/kernel_flash/l4t_flash_from_kernel.sh
@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright (c) 2020-2021, NVIDIA CORPORATION. All rights reserved.
+# Copyright (c) 2020-2022, NVIDIA CORPORATION. All rights reserved.
 #
 # Permission is hereby granted, free of charge, to any person obtaining a
 # copy of this software and associated documentation files (the "Software"),
@@ -108,21 +108,21 @@
 
 function is_internal_device()
 {
-   if [ "${1}" = "${SDCARD_STORAGE_DEVICE}" ] ||
-      [ "${1}" = "${SDMMC_USER_DEVICE}" ]; then
-          return 0;
-   fi
-   return 1;
+	if [ "${1}" = "${SDCARD_STORAGE_DEVICE}" ] ||
+	   [ "${1}" = "${SDMMC_USER_DEVICE}" ]; then
+			return 0;
+	fi
+	return 1;
 }
 
 function is_gpt_supported_device()
 {
 	if [ "${1}" = "${SDCARD_STORAGE_DEVICE}" ] ||
-      [ "${1}" = "${SDMMC_USER_DEVICE}" ] ||
-	  [ "${1}" =  "${EXTERNAL_STORAGE_DEVICE}" ]; then
-          return 0;
-   fi
-   return 1;
+	   [ "${1}" = "${SDMMC_USER_DEVICE}" ] ||
+	   [ "${1}" =  "${EXTERNAL_STORAGE_DEVICE}" ]; then
+			return 0;
+	fi
+	return 1;
 }
 
 function is_not_qspi()
@@ -131,9 +131,9 @@
 		|| [ "${1}" = "${SDMMC_USER_DEVICE}" ] \
 		|| [ "${1}" = "${SDCARD_STORAGE_DEVICE}" ] \
 		|| [ "${1}" = "${EXTERNAL_STORAGE_DEVICE}" ]; then
-          return 0;
-   fi
-   return 1;
+		return 0;
+	fi
+	return 1;
 }
 
 function erase_spi
@@ -739,9 +739,6 @@
 	echo "Writing ${part_name} partition with ${file_name}"
 	# if this device is emmc's boot partitions
 	if [ "${device_type}" = "${SDMMC_BOOT_DEVICE}" ];then
-		if [ "${part_name}" == "secondary_gpt" ];then
-			start_offset=$((start_offset - SDMMC_BOOT0_SIZE))
-		fi
 		flash_sdmmc_boot_partition "${file_name}" "${part_name}" \
 			"${start_offset}" "${file_size}" "${attributes}" "${sha1_chksum}"
 		res="${?}"
@@ -1085,10 +1082,14 @@
 
 			if [ "${part_name}" = "primary_gpt" ]; then
 				entry_id=0
-				continue
+				if is_gpt_supported_device "${device_type}"; then
+					continue
+				fi
 			elif [ "${part_name}" = "secondary_gpt" ] || [ "${part_name}" = "master_boot_record" ]; then
 				# skip as we have already put these in
-				continue
+				if is_gpt_supported_device "${device_type}"; then
+					continue
+				fi
 			fi
 			if [ -z "${attribute[2]}" ]; then
 				entry_id="$((entry_id + 1))"

nvidias · January 14, 2022, 11:38am

Hi WayneWWW,

Thanks very much for your reply!

I’ll try it right away, thanks!

BR.

nvidias · January 14, 2022, 12:28pm

Hi WayneWWW,

Thanks very much!

It works!

I notice that now I just switch the slot of the Kernel,
but I want to know how can I switch Rootfs A and Rootfs B?

Please give me some advice, thank you!

Thanks!
BR.

nvidias · January 17, 2022, 2:40am

Hi WayneWWW，

Do you have any suggestions on how to switch Rootfs Slot A/B？

Thanks！
BR.