Can't find matching TN Spec in OTA Blob!

Hello Team,

Got following error message during ota:

Nvidia A/B-Redundancy Update tool Version 2.1
Got payload file: /ota_work/bl_only_payload
current slot 0
SM: S11
Set slot 1 as unbootable and start updating.
Start running: /usr/sbin/nv_bootloader_payload_updater --no-dependent-partition /ota_work/bl_only_payload
Start running: /opt/nvidia/l4t-bootloader-config/nv-l4t-bootloader-config.sh -c
2888-401-0004-B.0-1-2-jetson-agx-xavier-devkit-
Got update payload: /ota_work/bl_only_payload
Tegra User Block Device: /dev/disk/by-partlabel
Tegra Boot Block Device: /dev/mmcblk0boot0
HEADER: MAGIC NVIDIA__BLOB__V2
HEX_VALUE 16909857
BLOB_SIZE 11096529
HEADER_SIZE 48
NUMBER_OF_ELEMENTS 33
HEADER_TYPE 0
UNCOMP_SIZE 11096529
MB1_RATCHET_LV 0
MTS_RATCHET_LV 0
ROLLBACK_FUSE_LV 0
Device TN Spec: 2888-401-0004-B.0-1-2-jetson-agx-xavier-devkit-mmcblk0p1
Device Compatible Spec: 2888-401-0004-B.0-1-2-jetson-agx-xavier-devkit-
Can't find matching TN Spec in OTA Blob!
OTA Blob update failed. Status: 3
/usr/sbin/nv_bootloader_payload_updater --no-dependent-partition failed.
Slot 1 update failed
Error: installing bootloader updates failed: 768
Failed to run "/usr/sbin/nv_update_engine -i bl --payload /ota_work/bl_only_payload --no-reboot"
Failed to run "update_bootloader_with_UE /ota_work 0x19"
Failed to run "update_bootloader /ota_work 0 0x19"

Thanks.

My commands to generate ota image:

sudo BOARDID=2888 BOARDSKU=0004 FAB=401 BOARDREV=B.0 FUSELEVEL=fuselevel_production  ROOTFS_AB=1 ./build_l4t_bup.sh jetson-agx-xavier-devkit mmcblk0p1
sudo BOARDID=2888 BOARDSKU=0004 FAB=401 BOARDREV=B.0 FUSELEVEL=fuselevel_production ROOTFS_AB=1 ./tools/ota_tools/version_upgrade/build_base_recovery_image.sh jetson-agx-xavier-devkit R32-7 ${BASE_BSP}/ ${BASE_BSP}/rootfs ${TARGET_BSP}
sudo BOARDID=2888 BOARDSKU=0004 FAB=401 BOARDREV=B.0 FUSELEVEL=fuselevel_production  ROOTFS_AB=1 ./build_l4t_bup.sh jetson-agx-xavier-devkit mmcblk0p1

The result of BUP_generator:

 ./bootloader/BUP_generator.py -c ./bootloader/payloads_t19x/bl_only_payload 
BLOB PATH:
/home/hehehuang/build/image/jetson/32.7.2/Linux_for_Tegra/bootloader/payloads_t19x/bl_only_payload

BLOB HEADER:
       Magic: NVIDIA__BLOB__V2
     Version: v2.1-2021.6-0 (0x01020621)
   Blob Size: 7,724,358 bytes
 Header Size: 48 bytes
 Entry Count: 19 partition(s)
        Type: 0 (0 for update, 1 for BMP)
Uncompressed
   Blob Size: 7,724,358 bytes
   Accessory: 0x0000000000000000

ENTRY TABLE:
|   part_name    |  offset | part_size | version | op_mode |                          tnspec                          |
|         spe-fw |    2328 |     94960 |   3272  |    0    |                                                          |
|            mb2 |   97288 |    181328 |   3272  |    0    |                                                          |
| cpu-bootloader |  278616 |    472048 |   3272  |    0    |                                                          |
|      secure-os |  750664 |    410560 |   3272  |    0    |                                                          |
|        bpmp-fw | 1161224 |    856352 |   3272  |    0    |                                                          |
|            eks | 2017576 |      5136 |   3272  |    0    |                                                          |
|        adsp-fw | 2022712 |     81312 |   3272  |    0    |                                                          |
|         rce-fw | 2104024 |    271952 |   3272  |    0    |                                                          |
|    mts-preboot | 2375976 |     24016 |   3272  |    2    |                                                          |
|        mts-mce | 2399992 |    143200 |   3272  |    2    |                                                          |
|     mts-proper | 2543192 |   3430416 |   3272  |    2    |                                                          |
|            sc7 | 5973608 |     65504 |   3272  |    2    |                                                          |
|    bpmp-fw-dtb | 6039112 |    993088 |   3272  |    0    | 2888-401-0004-B.0-1-2-jetson-agx-xavier-devkit-mmcblk0p1 |
| bootloader-dtb | 7032200 |    209056 |   3272  |    0    | 2888-401-0004-B.0-1-2-jetson-agx-xavier-devkit-mmcblk0p1 |
|            VER | 7241256 |       102 |   3272  |    0    | 2888-401-0004-B.0-1-2-jetson-agx-xavier-devkit-mmcblk0p1 |
|            mb1 | 7241358 |    250432 |   3272  |    2    | 2888-401-0004-B.0-1-2-jetson-agx-xavier-devkit-mmcblk0p1 |
|            BCT | 7491790 |      2888 |   3272  |    2    | 2888-401-0004-B.0-1-2-jetson-agx-xavier-devkit-mmcblk0p1 |
|        MB1_BCT | 7494678 |     31024 |   3272  |    0    | 2888-401-0004-B.0-1-2-jetson-agx-xavier-devkit-mmcblk0p1 |
|        MEM_BCT | 7525702 |    198656 |   3272  |    0    | 2888-401-0004-B.0-1-2-jetson-agx-xavier-devkit-mmcblk0p1 |

Result of /etc/nv_boot_control.conf on the jetson agx xavier board:

cat /etc/nv_boot_control.conf 
TNSPEC 2888-401-0004-B.0-1-2-jetson-agx-xavier-devkit-mmcblk0p1
COMPATIBLE_SPEC 2888-401-0004-B.0-1-2-jetson-agx-xavier-devkit-
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mmcblk0boot0
TEGRA_OTA_GPT_DEVICE /dev/mmcblk0boot1

Unfortunately we have already sold products, so questions are:

  1. Where does nv_bootloader_payload_updater read 2 TN Spec?
  2. Is there any way to force a TN Sepc?
  3. Is there any way to bypass this check?
    Thanks.

Hi hehe.huang,

Are you using the devkit or custom board for AGX Xavier?

What’s you BASE_BSP and TARGET_BSP in OTA update?

Have you referred to the following instruction for image-based OTA update?
NVIDIA Jetson Linux Developer Guide : Over-the-Air Update | NVIDIA Docs

  1. I am using customer boaard, while does changed build commands
  2. BASE_BSP and TARGET_BSP are the same, set to ~/build/Linux_for_Tegra
  3. Yes, I tried exact same instatructions from “Steps Performed on the Host Machine” step 12 and 13. It reported that TN SPEC does not match. I guess that might have something to do with enviroment variables, so added those “sudo BOARDID=2888 BOARDSKU=0004 FAB=401 BOARDREV=B.0 FUSELEVEL=fuselevel_production” in commands

We found there’s a known issue about AGX Xavier SKU4(32GB), and it should be fixed in the latest release (R32.7.4). Please verify image-based OTA for your board with latest R32.7.4.

Hello,
Could you share me the direct fix?
We have already built products based on 32.7.2. To using 32.7.4 means to apply all the works on 32.7.2 again.
Thanks.

May I know what’s the use case of you to update from R32.7.2 to R32.7.2 (the same version)?

Do you want to trigger the image-based OTA for rootfs or bootloader or both?

We added a background program to control the system, e.g., shutdown, heartbeat and so.
Unfortunately there is bug in that program, and currently the only way to update that program is whole system OTA…

btw, we ran OTA on 32.7.2 successfully before, with following patch

diff --git a/tools/ota_tools/version_upgrade/build_base_recovery_image.sh b/tools/ota_tools/version_upgrade/build_base_recovery_image.sh
index 3270c56..6ea494d 100755
--- a/tools/ota_tools/version_upgrade/build_base_recovery_image.sh
+++ b/tools/ota_tools/version_upgrade/build_base_recovery_image.sh
@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright (c) 2019-2021, NVIDIA CORPORATION.  All rights reserved.
+# Copyright (c) 2019-2022, NVIDIA CORPORATION.  All rights reserved.
 #
 # Permission is hereby granted, free of charge, to any person obtaining a
 # copy of this software and associated documentation files (the "Software"),
@@ -28,7 +28,8 @@
 # 4. Copy all these four files to <target BSP L4T dir>/bootloader
 
 # The BASE version can be:
-# "R28.2", "R28.3", "R28.4", "R32.1", "R32.2", "R32.3", "R32.4" "R32.5" "R32.6"
+# "R28.2", "R28.3", "R28.4", "R32.1", "R32.2", "R32.3", "R32.4" "R32.5"
+# "R32.6", "R32.7"
 set -e
 
 KERNEL_DTB_NAME=
@@ -55,12 +56,12 @@
 	echo -ne "\t<target_board>: specify the target board.\n"
 	echo -ne "\t   supported target boards: jetson-tx2-devkit, jetson-agx-xavier-devkit, jetson-xavier-nx-devkit, jetson-xavier-nx-devkit-emmc, jetson-xavier-nx-devkit-tx2-nx.\n"
 	echo -ne "\t<bsp version>: specify the version of the base BSP\n"
-	echo -ne "\t   supported versions: R28-2, R28-3, R28-4, R32-1, R32-2, R32-3, R32-4, R32-5, R32-6\n"
+	echo -ne "\t   supported versions: R28-2, R28-3, R28-4, R32-1, R32-2, R32-3, R32-4, R32-5, R32-6, R32-7\n"
 	echo -ne "\t<base BSP L4T dir>: specify the path of the L4T dir of the base BSP\n"
 	echo -ne "\t<base BSP rootfs dir>: specify the path of the rootfs of the base BSP\n"
 	echo -ne "\t<target BSP L4T dir>: specify the path of the L4T dir of the target BSP\n"
 	echo -ne "Example:\n"
-	echo -ne "\tsudo $0 -u PKCkeyfile -v SBKkeyfile jetson-tx2-devkit R28-3 <R28.3_BSP>/Linux_for_Tegra <R28.3_BSP>/Linux_for_Tegra/rootfs <R32.6_BSP>/Linux_for_Tegra\n"
+	echo -ne "\tsudo $0 -u PKCkeyfile -v SBKkeyfile jetson-tx2-devkit R28-3 <R28.3_BSP>/Linux_for_Tegra <R28.3_BSP>/Linux_for_Tegra/rootfs <R32.7_BSP>/Linux_for_Tegra\n"
 	exit 1
 }
 
diff --git a/tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh b/tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh
index d4086d3..eefd712 100755
--- a/tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh
+++ b/tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh
@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright (c) 2019-2021, NVIDIA CORPORATION.  All rights reserved.
+# Copyright (c) 2019-2022, NVIDIA CORPORATION.  All rights reserved.
 #
 # Permission is hereby granted, free of charge, to any person obtaining a
 # copy of this software and associated documentation files (the "Software"),
@@ -21,7 +21,7 @@
 # DEALINGS IN THE SOFTWARE.
 
 # This script generates OTA package, which used to ota from
-# R28.2/R28.3/R28.4/R32.1/R32.2/R32.3/R32.4/R32.5/R32.6 to R32-ToT.
+# R28.2/R28.3/R28.4/R32.1/R32.2/R32.3/R32.4/R32.5/R32.6/R32.7 to R32-ToT.
 set -e
 
 LINUX_BASE_DIR="$(pwd)"
@@ -74,7 +74,7 @@
 	echo -ne "\tWhere,\n"
 	echo -ne "\t\t<target board>: target board.\n"
 	echo -ne "\t\t  Supported boards: jetson-tx2-devkit, jetson-agx-xavier-devkit, jetson-xavier-nx-devkit, jetson-xavier-nx-devkit-emmc, jetson-xavier-nx-devkit-tx2-nx.\n"
-	echo -ne "\t\t<bsp version>: the version of the base BSP. Supported versions:R28-2, R28-3, R28-4, R32-1, R32-2, R32-3, R32-4, R32-5, R32-6;\n"
+	echo -ne "\t\t<bsp version>: the version of the base BSP. Supported versions:R28-2, R28-3, R28-4, R32-1, R32-2, R32-3, R32-4, R32-5, R32-6, R32-7;\n"
 	echo -ne "\toptions:\n"
 	echo -ne "\t\t-u <PKC key file>: PKC key used for odm fused board\n"
 	echo -ne "\t\t-v <SBK key file>: Secure Boot Key (SBK) key used for ODM fused board\n"
@@ -113,7 +113,7 @@
 		exit 1
 	fi
 
-	# Skip check for the R32.3/R32.4/R32.5/R32.6 that have no partition layout changes
+	# Skip check for the R32.3/R32.4/R32.5/R32.6/R32.7 that have no partition layout changes
 	if [ "${LAYOUT_CHANGE}" == 0 ]; then
 		echo "No immediate partitions and skip it"
 		return 0
diff --git a/tools/ota_tools/version_upgrade/nv_ota_common.func b/tools/ota_tools/version_upgrade/nv_ota_common.func
index 46730a4..ffcf7dd 100644
--- a/tools/ota_tools/version_upgrade/nv_ota_common.func
+++ b/tools/ota_tools/version_upgrade/nv_ota_common.func
@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright (c) 2021, NVIDIA CORPORATION.  All rights reserved.
+# Copyright (c) 2021-2022, NVIDIA CORPORATION.  All rights reserved.
 #
 # Permission is hereby granted, free of charge, to any person obtaining a
 # copy of this software and associated documentation files (the "Software"),
@@ -25,7 +25,7 @@
 _BASE_RECOVERY_IMAGE=
 _BASE_RECOVERY_DTB=
 # Base versions without layout change
-_BASE_VERSIONS_NO_LAYOUT_CHANGE=( "R32-3" "R32-4" "R32-5" "R32-6" )
+_BASE_VERSIONS_NO_LAYOUT_CHANGE=( "R32-3" "R32-4" "R32-5" "R32-6" "R32-7" )
 _EXTLINUX_CONF_BACKUP="extlinux.conf.ota_backup"
 _K_BYTES=1024
 _NV_BOOT_CONTROL_CONF="nv_boot_control.conf"
diff --git a/tools/ota_tools/version_upgrade/nv_ota_update_without_layout_change.sh b/tools/ota_tools/version_upgrade/nv_ota_update_without_layout_change.sh
index bfb8e4f..3af69bd 100755
--- a/tools/ota_tools/version_upgrade/nv_ota_update_without_layout_change.sh
+++ b/tools/ota_tools/version_upgrade/nv_ota_update_without_layout_change.sh
@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright (c) 2021, NVIDIA CORPORATION.  All rights reserved.
+# Copyright (c) 2021-2022, NVIDIA CORPORATION.  All rights reserved.
 #
 # Permission is hereby granted, free of charge, to any person obtaining a
 # copy of this software and associated documentation files (the "Software"),
@@ -331,6 +331,18 @@
 		fi
 		cp -f "${src}" "${dst}"
 	done
+
+	# Also update nv-l4t-bootloader-config.sh as it is invoked by
+	# nv_bootloader_payload_updater now.
+	local bl_cfg_script="nv-l4t-bootloader-config.sh"
+	local bl_cfg_dir="/opt/nvidia/l4t-bootloader-config"
+	# We don't have nv-l4t-bootloader-config.sh for release versions
+	# prior to 32.4.2. Create the particular dir first for this case.
+	if [ ! -d "${bl_cfg_dir}" ]; then
+		mkdir -p "${bl_cfg_dir}"
+	fi
+	cp -f "${work_dir}/${bl_cfg_script}" "${bl_cfg_dir}"
+
 	return 0
 }
 
diff --git a/tools/ota_tools/version_upgrade/ota_board_specs.conf b/tools/ota_tools/version_upgrade/ota_board_specs.conf
index 58ff151..8300229 100644
--- a/tools/ota_tools/version_upgrade/ota_board_specs.conf
+++ b/tools/ota_tools/version_upgrade/ota_board_specs.conf
@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright (c) 2020-2021, NVIDIA CORPORATION. All rights reserved.
+# Copyright (c) 2020-2022, NVIDIA CORPORATION. All rights reserved.
 #
 # Permission is hereby granted, free of charge, to any person obtaining a
 # copy of this software and associated documentation files (the "Software"),
@@ -78,6 +78,8 @@
 JETSON_TX2_DEVKIT_R32_5_ALIAS="jetson_tx2_devkit_ota_emmc_r32_3_4_spec"
 # shellcheck disable=SC2034
 JETSON_TX2_DEVKIT_R32_6_ALIAS="jetson_tx2_devkit_ota_emmc_r32_3_4_spec"
+# shellcheck disable=SC2034
+JETSON_TX2_DEVKIT_R32_7_ALIAS="jetson_tx2_devkit_ota_emmc_r32_3_4_spec"
 
 # jetson-agx-xavier-devkit
 jetson_agx_xavier_devkit_ota_emmc_r32_1_2_spec=(
@@ -119,6 +121,8 @@
 JETSON_AGX_XAVIER_DEVKIT_R32_5_ALIAS="jetson_agx_xavier_devkit_ota_emmc_r32_3_4_spec"
 # shellcheck disable=SC2034
 JETSON_AGX_XAVIER_DEVKIT_R32_6_ALIAS="jetson_agx_xavier_devkit_ota_emmc_r32_3_4_spec"
+# shellcheck disable=SC2034
+JETSON_AGX_XAVIER_DEVKIT_R32_7_ALIAS="jetson_agx_xavier_devkit_ota_emmc_r32_3_4_spec"
 
 # jetson-xaiver-nx-devkit
 jetson_xavier_nx_devkit_ota_sd_r32_4_spec=(
@@ -133,6 +137,8 @@
 JETSON_XAVIER_NX_DEVKIT_R32_5_ALIAS="jetson_xavier_nx_devkit_ota_sd_r32_4_spec"
 # shellcheck disable=SC2034
 JETSON_XAVIER_NX_DEVKIT_R32_6_ALIAS="jetson_xavier_nx_devkit_ota_sd_r32_4_spec"
+# shellcheck disable=SC2034
+JETSON_XAVIER_NX_DEVKIT_R32_7_ALIAS="jetson_xavier_nx_devkit_ota_sd_r32_4_spec"
 
 # jetson-xaiver-nx-devkit-emmc
 jetson_xavier_nx_devkit_emmc_ota_emmc_r32_4_spec=(
@@ -147,6 +153,8 @@
 JETSON_XAVIER_NX_DEVKIT_EMMC_R32_5_ALIAS="jetson_xavier_nx_devkit_emmc_ota_emmc_r32_4_spec"
 # shellcheck disable=SC2034
 JETSON_XAVIER_NX_DEVKIT_EMMC_R32_6_ALIAS="jetson_xavier_nx_devkit_emmc_ota_emmc_r32_4_spec"
+# shellcheck disable=SC2034
+JETSON_XAVIER_NX_DEVKIT_EMMC_R32_7_ALIAS="jetson_xavier_nx_devkit_emmc_ota_emmc_r32_4_spec"
 
 # jetson-xaiver-nx-devkit-tx2-nx
 jetson_xavier_nx_devkit_tx2_nx_ota_emmc_r32_5_spec=(
@@ -156,6 +164,8 @@
 JETSON_XAVIER_NX_DEVKIT_TX2_NX_R32_5_ALIAS="jetson_xavier_nx_devkit_tx2_nx_ota_emmc_r32_5_spec"
 # shellcheck disable=SC2034
 JETSON_XAVIER_NX_DEVKIT_TX2_NX_R32_6_ALIAS="jetson_xavier_nx_devkit_tx2_nx_ota_emmc_r32_5_spec"
+# shellcheck disable=SC2034
+JETSON_XAVIER_NX_DEVKIT_TX2_NX_R32_7_ALIAS="jetson_xavier_nx_devkit_tx2_nx_ota_emmc_r32_5_spec"
 
 # List the supported t18x devices and t19x devices
 T18X_DEVICES=(

A quick way may be to hack nv_bootloader_payload_updater, force it ignoring the TN spec check.
Could we do that? Thanks.

It seems you are just need to update the rootfs with image-based OTA.
You could just add -r parameter during generating OTA payload, which would skip BUP update and you would not hit TN spec issue.

Thanks @KevinFFF , that is really a good suggestion. :)

However, still want a way to update BUP… Some days later we may have to pick up security patches for kernel, at that time we may face similar problem again.

If you are going to update BUP in the future from R32.7.2 to R32.7.4. You would not hit this issue because the TN Spec issue for AGX Xavier SKU4 has been fixed in the R32.7.4.

Hello,

Find something interesting: we tried the same OTA image on an old jetson module purchased last year, it works. But on new modules purchased recently, it fails. However, the /etc/nv_boot_control.conf (TNSPEC) file on both modules are the same.

Hello Kevin,

Checked with our management team: we have run our products based on 32.7.2 for more than half an year, now it is time to deliver to customer. It is too dangerous to update the baseline at such a critical phase, and we may not have time to run enough test again.

Sorry to say that I may still ask for help to show the exact fix, we may have to pick up it directly.

You would not hit the issue if you are going to update it from R32.7.2 to latest R32.7.4.

Does the image-based OTA for rootfs only (with -r parameter) work in your current use case?