OTA tools: broken support of AGX Xavier 64Gb

Latest OTA tools (R35.6.0) that supports Xavier AGX fails to apply OTA update on 64gb xavier version:

Command: ./nv_ota_start.sh /dev/mmcblk0 /home/jetuser/ota_payload_package.tar.gz
Current rootfs is on /dev/mmcblk0
init_ota_log /ota_log
Creating log dir at /ota_log
Create log file at /ota_log/ota_20250222-100405.log
OTA_LOG_FILE=/ota_log/ota_20250222-100405.log
Extract /home/jetuser/ota_payload_package.tar.gz
update_nv_boot_control_in_rootfs /ota_work
2888-402-0005-G.0-1-2-jetson-agx-xavier-devkit-
check_prerequisites
decompress_ota_package ota_package.tar /ota_work
decompress_ota_package: start at Sat Feb 22 10:04:48 UTC 2025
Sha1 checksum for /ota_work/ota_package.tar (fc0e76ba49e15583e1ad96cc5eb0a52203aa7e68) matches
decompress_ota_package: end at Sat Feb 22 10:05:10 UTC 2025
nv_ota_update_with_layout_change.sh /dev/mmcblk0
Command: nv_ota_update_with_layout_change.sh /dev/mmcblk0
check_bsp_version /ota_work BASE_VERSION
check_target_board /ota_work TARGET_BOARD
set_msi_emmc_min_size jetson-agx-xavier-devkit MSI_EMMC_MIN_SIZE
ota_check_rollback /ota_work jetson-agx-xavier-devkit R32-7 /dev/mmcblk0
OTA_PACKAGE version: branch:35 revision:6.0 major.minor:6.0
16384+0 records in
16384+0 records out
8388608 bytes (8.4 MB, 8.0 MiB) copied, 2.45499 s, 3.4 MB/s
boot_device_size=8388608
VER_b's offset is 2203648 and size is 65536
VER's offset is 2269184 and size is 65536
VER_b version: branch:32 revision:7.5 major.minor:7.5
VER version: branch:32 revision:7.5 major.minor:7.5
Check BCT/MB1/MB1_BCT partiton for fresh OTA
Checking BCT partition
Checking MB1 partition
Checking MB1_BCT partition
ver_check_res=0
ota_choose_images /ota_work
COMPATIBLE_SPEC=2888-402-0005-G.0-1-2-jetson-agx-xavier-devkit-
TEGRA_CHIPID=0x19
_BOARD_SPEC_NAME=2888-402-0005-G.0
No image is found for compatible SPEC 2888-402-0005-G.0
Failed to call "copy_board_spec_dep_files /ota_work/internal_device"
Failed to run "ota_choose_images /ota_work"

Could you fix it please? It seems, that problem exist for both 0004 and 0005.

Also, I succesfully applied OTA update with this workaround:

--- a/ota_tools/version_upgrade/ota_multi_board_specs.sh
+++ b/ota_tools/version_upgrade/ota_multi_board_specs.sh
@@ -61,6 +61,9 @@ construct_board_spec_name()
        # board_spec_name=${boardid}-${boardver}-${boardsku}-${boardrev}
        # For example, for jetson-agx-xavier-devkit, board_spec_name can be 2888-400-0001-D.0
        # or 2888-400-0001-E.0.
+       if [ "${boardsku}" == "0005" ]; then
+               boardrev=""
+       fi
        name="${boardid}-${boardver}-${boardsku}-${boardrev}"
        eval "${board_spec_name}=${name}"
        return 0

Hi pim_pam,

Are you using the devkit or custom board for AGX Xavier?
What’s you L4T release before update? (i.e. are you trying to update from r35.6.0 to r35.6.0)

Would you like to perform image-based OTA update for each partitions including rootfs? Or you just want to update bootloader through capsule update?

Please also share the result of the following command on your board.

$ sudo cat /etc/nv_boot_control.conf

Hi! It is custom board with 64gb xavier module, and I am trying to update from 32.7.5 to 35.6.0, all partitions including rootfs.

$ cat /etc/nv_boot_control.conf
TNSPEC 2888-402-0005-G.0-1-2-jetson-agx-xavier-devkit-
COMPATIBLE_SPEC 2888-402-0005-G.0-1-2-jetson-agx-xavier-devkit-
TEGRA_LEGACY_UPDATE false
TEGRA_BOOT_STORAGE mmcblk0
TEGRA_EMMC_ONLY true
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mmcblk0boot0
TEGRA_OTA_GPT_DEVICE /dev/mmcblk0boot1

Also, just for information, on 16Gb devkit OTA works without any problems, and 16gb module on custom board also OK.

from log above, OTA scripts get this SPEC from nv_boot_control.conf:

_BOARD_SPEC_NAME=2888-402-0005-G.0
No image is found for compatible SPEC 2888-402-0005-G.0

but generated ota_package.tar has only these directories for 0005, with null boardrev:

# tar tf ota_package.tar | grep '2888-402-0005.*\/$'
./internal_device/images-R32x-R35i/2888-402-0005-/
./internal_device/images-R35-ToT/2888-402-0005-/
./internal_device/images-R35A-R35i/2888-402-0005-/

sorry, I answered not in reply…

Do you mean the issue is specific to AGX Xavier 64GB(SKU5)?

And there’s no issue after you add this line in ota_multi_board_specs.sh?

Could you elaborate on this? Do you mean something missing for SKU5 in OTA payload?

I would also like to check the full log when you generate the OTA payload for 2888-0005.

yes

And there’s no issue after you add this line in ota_multi_board_specs.sh ?

yes

generate-ota.log (944.7 KB)

Do you mean something missing for SKU5 in OTA payload?

I am not expert in your code :) but I think, payload is fine, but… there is a bug in “using” this payload.

In ota_payload_package.tar.gz → ota_package.tar these directories are generated:

./internal_device/images-*/2888-400-0001-D.0
./internal_device/images-*/2888-400-0001-E.0
./internal_device/images-*/2888-400-0004-
./internal_device/images-*/2888-402-0005-

but applying payload mistakenly tries to using 2888-402-0005-G.0 directory instead of 2888-402-0005-

It seems there’s the bug for AGX Xavier SKU5.

Could you modify the following line of /opt/nvidia/l4t-bootloader-config/nv-l4t-bootloader-config.sh in your board ?

@@ -251,7 +251,11 @@ update_jetson_xavier_spec () {
         board_ver="400"
         return 0
     fi
-
+    if [[ "${board_sku}" == "0005" ]]; then
+        board_rev=""
+        board_ver="402"
+        return 0
+    fi
     if [[ "${board_ver}" == "400" ]]; then
         if [[ "${board_rev}" == "D.0" || "${board_rev}" < "D.0" ]]; then
             board_rev="D.0"

After rebooting the board, please share the result of the following command on your board.

$ cat /etc/nv_boot_control.conf

If its COMPATIBLE_SPEC becomes 2888-402-0005--1-2-jetson-agx-xavier-devkit-, then you can try using the OTA payload you generated before to update the board.

yes, COMPATIBLE_SPEC changed:

$ grep -A 15 'update_jetson_xavier_spec ()' /opt/nvidia/l4t-bootloader-config/nv-l4t-bootloader-config.sh
update_jetson_xavier_spec () {
    if [[ "${board_sku}" == "0004" ]]; then
        board_rev=""
        board_ver="400"
        return 0
    fi

    if [[ "${board_sku}" == "0005" ]]; then
        board_rev=""
        board_ver="402"
        return 0
    fi

    if [[ "${board_ver}" == "400" ]]; then
        if [[ "${board_rev}" == "D.0" || "${board_rev}" < "D.0" ]]; then
            board_rev="D.0"
$ cat /etc/nv_boot_control.conf
TNSPEC 2888-402-0005-G.0-1-2-jetson-agx-xavier-devkit-
COMPATIBLE_SPEC 2888-402-0005--1-2-jetson-agx-xavier-devkit-
TEGRA_LEGACY_UPDATE false
TEGRA_BOOT_STORAGE mmcblk0
TEGRA_EMMC_ONLY true
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mmcblk0boot0
TEGRA_OTA_GPT_DEVICE /dev/mmcblk0boot1

Please also modify those lines in file of you BSP package(<Linux_for_Tegra>/rootfs/opt/nvidia/l4t-bootloader-config/nv-l4t-bootloader-config.sh) and generate OTA payload again.

Please also modify those lines in file of you BSP package(<Linux_for_Tegra>/rootfs/opt/nvidia/l4t-bootloader-config/nv-l4t-bootloader-config.sh) and generate OTA payload again.

done, ota update from 32.7.5 to 35.6.0 successfully finished, thanks.

Is there any chance to have update with fixed bug soon?

Thanks for your update.
We are working on fixing this issue with internal, it may be fixed in the next release.
Before that, please take above modification as WA for AGX Xavier SKU5.

Thanks!

Hi, @KevinFFF, it seems that there are more problems…

After OTA update from 32.7.5 to 35.6.0, I am tryin to do OTA 35.6.0 to 35.6.0, but get this:

Command: ./nv_ota_start.sh /dev/mmcblk0 /home/jetuser/ota_payload_package.tar.gz
Current rootfs is on /dev/mmcblk0
init_ota_log /ota_log
Create log file at /ota_log/ota_20250405-161139.log
OTA_LOG_FILE=/ota_log/ota_20250405-161139.log
Extract /home/jetuser/ota_payload_package.tar.gz
update_nv_boot_control_in_rootfs /ota_work
2888-402-0005--1-2-jetson-agx-xavier-devkit-
Info. eMMC only board jetson-agx-xavier-devkit
Info. Installing mtdblock.
Info. Active boot storage: mmcblk0
Info. Legacy mode: false
TNSPEC 2888-402-0005-G.0-1-2-jetson-agx-xavier-devkit-
COMPATIBLE_SPEC 2888-402-0005--1-2-jetson-agx-xavier-devkit-
TEGRA_LEGACY_UPDATE false
TEGRA_BOOT_STORAGE mmcblk0
TEGRA_EMMC_ONLY true
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mmcblk0boot0
TEGRA_OTA_GPT_DEVICE /dev/mmcblk0boot1
Info: Spec variable TegraPlatformSpec is not found.
Info: Write TegraPlatformSpec with 2888-402-0005-G.0-1-2-jetson-agx-xavier-devkit-.
mount: /opt/nvidia/esp: unknown filesystem type 'vfat'.
Info: Spec variable TegraPlatformCompatSpec is not found.
Info: Write TegraPlatformCompatSpec with 2888-402-0005--1-2-jetson-agx-xavier-devkit-.
mount: /opt/nvidia/esp: unknown filesystem type 'vfat'.
Info. Uninstalling mtdblock.
check_prerequisites
decompress_ota_package ota_package.tar /ota_work
decompress_ota_package: start at Sat 05 Apr 2025 04:12:06 PM UTC
Sha1 checksum for /ota_work/ota_package.tar (f56cbaf66ae08b3a3130c81eef763ccc858c302e) matches
decompress_ota_package: end at Sat 05 Apr 2025 04:12:24 PM UTC
nv_ota_update_without_layout_change.sh
Command: nv_ota_update_without_layout_change.sh
check_target_board /ota_work TARGET_BOARD
get_chip_id CHIP_ID
ota_choose_images /ota_work
COMPATIBLE_SPEC=2888-402-0005--1-2-jetson-agx-xavier-devkit-
TEGRA_CHIPID=0x19
_BOARD_SPEC_NAME=2888-402-0005-
Copy files from ./images-R35-ToT/2888-402-0005-/ to ./images-R35-ToT/
is_rootfs_a_b_enabled ROOTFS_AB_ENABLED ROOTFS_CURRENT_SLOT
ROOTFS_AB_ENABLED=0
ROOTFS_CURRENT_SLOT=0
is_rootfs_encryption_enabled ROOTFS_ENC_ENABLED
ROOTFS_ENC_ENABLED=0
get_update_slot UPDATE_SLOT
mount: /opt/nvidia/esp: unknown filesystem type 'vfat'.
Failed to get current slot
Failed to run "get_update_slot UPDATE_SLOT"

For some reason nvbootctrl get-current-slot fails… Some info about system:

jetuser@pillar2(hss2a000):~$ sudo nvbootctrl get-current-slot
mount: /opt/nvidia/esp: unknown filesystem type 'vfat'.
Error: mount esp to directory /opt/nvidia/esp failed.
0

jetuser@pillar2(hss2a000):~$ sudo nvbootctrl dump-slots-info
mount: /opt/nvidia/esp: unknown filesystem type 'vfat'.
Error: mount esp to directory /opt/nvidia/esp failed.
Current version: 35.6.0
Capsule update status: 0
Current bootloader slot: A
Active bootloader slot: A
num_slots: 2
slot: 0,             status: normal
slot: 1,             status: normal

jetuser@pillar2(hss2a000):~$ cat /etc/nv_tegra_release
# R35 (release), REVISION: 6.0, GCID: 37391689, BOARD: t186ref, EABI: aarch64, DATE: Wed Aug 28 09:12:27 UTC 2024

jetuser@pillar2(hss2a000):~$ cat /etc/nv_boot_control.conf
TNSPEC 2888-402-0005-G.0-1-2-jetson-agx-xavier-devkit-
COMPATIBLE_SPEC 2888-402-0005--1-2-jetson-agx-xavier-devkit-
TEGRA_LEGACY_UPDATE false
TEGRA_BOOT_STORAGE mmcblk0
TEGRA_EMMC_ONLY true
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mmcblk0boot0
TEGRA_OTA_GPT_DEVICE /dev/mmcblk0boot1

Please share the result of the following command on your board.

$ ls -al /dev/disk/by-partlabel|grep esp
lrwxrwxrwx 1 root root  16 Apr  6 22:24 esp -> ../../mmcblk0p40
$ sudo lsblk -f|grep mmcblk0p40
$ ls -al /dev/disk/by-partlabel | grep esp
lrwxrwxrwx 1 root root  16 Jun 17  2024 esp -> ../../mmcblk0p40
lrwxrwxrwx 1 root root  16 Jun 17  2024 esp_alt -> ../../mmcblk0p43

$ sudo lsblk -f | grep mmcblk0p40
|--mmcblk0p40 vfat         7DC8-07BD

Your esp partition is vfat format as expected.
This error looks like mount command does not support vfat format.
Have you customized the rootfs on your board?

We would suggest applying the following patch in BSP package of r35.6.0 before generate OTA payload.

Please also help to clarify if the issue is specific to OTA update.
(i.e. would you hit the similar issue if your flash R35.6.0 to your board directly?)

Oh, thank you for response. Yes, it was custom kernel… I fixed it by changing kernel config:

-CONFIG_VFAT_FS=m
+CONFIG_VFAT_FS=y