Xavier 64GB C: OTA from 4.6.5 to 5.1.4 fails

Previous topic was closed, but problem is not solved, so I created it again.

I retested it multiple times on clean Ubuntu 20.04.6 installation. It is 100% reproducible with original L4T and sample rootfs, without any modifications.

Problematic hardware revision:

cat /etc/nv_boot_control.conf
TNSPEC 2888-402-0005-C.0-1-2-jetson-agx-xavier-devkit-internal
COMPATIBLE_SPEC 2888-402-0005--1-2-jetson-agx-xavier-devkit-
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mmcblk0boot0
TEGRA_OTA_GPT_DEVICE /dev/mmcblk0boot1

Steps to reproduce:

# 1. Prepare Jetpack 4.6.5 Sample RootFS
cd $HOME/jetpack-4.6.5
tar xf jetson_linux_r32.7.5_aarch64.tbz2
sudo tar xpf tegra_linux_sample-root-filesystem_r32.7.5_aarch64.tbz2 -C Linux_for_Tegra/rootfs/
cd Linux_for_Tegra/
sudo apt-get install qemu-user-static
sudo ./apply_binaries.sh
sudo ./tools/l4t_flash_prerequisites.sh
sudo ./l4t_create_default_user.sh -u nvidia -p nvidia1

# apply fix for nv-l4t-bootloader-config.sh from https://forums.developer.nvidia.com/t/ota-tools-broken-support-of-agx-xavier-64gb/325683/10

# 2. Flash Jetpack 4.6.5 Sample RootFS
sudo ./flash.sh jetson-agx-xavier-devkit internal

# 3. Prepare Jetpack 5.1.4 Sample RootFS
cd $HOME/jetpack-5.1.4
tar xf Jetson_Linux_R35.6.0_aarch64.tbz2
tar xf ota_tools_R35.6.0_aarch64.tbz2
sudo tar xpf tegra_linux_sample-root-filesystem_r35.6.0_aarch64.tbz2 -C Linux_for_Tegra/rootfs/
cd Linux_for_Tegra/
sudo ./apply_binaries.sh
sudo ./tools/l4t_flash_prerequisites.sh
sudo .tools/l4t_create_default_user.sh -u nvidia -p nvidia1

# apply fix for nv-l4t-bootloader-config.sh from https://forums.developer.nvidia.com/t/ota-tools-broken-support-of-agx-xavier-64gb/325683/10

# 4. Generate OTA
export BASE_BSP=$HOME/jetpack-4.6.5/Linux_for_Tegra
export TARGET_BSP=$HOME/jetpack-5.1.4/Linux_for_Tegra
BOARD=jetson-agx-xavier-devkit
UPD_FROM=R32-7

cd $TARGET_BSP

# Recovery image
sudo ./tools/ota_tools/version_upgrade/build_base_recovery_image.sh \
    ${BOARD} \
    ${UPD_FROM} \
    ${BASE_BSP} \
    ${BASE_BSP}/rootfs \
    ${TARGET_BSP}

# OTA package
sudo -E ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh \
    ${BOARD} \
    ${UPD_FROM}

# 5. OTA update on xavier 64GB
tar xf ota_tools_R35.6.0_aarch64.tbz2
cd Linux_for_Tegra/tools/ota_tools/version_upgrade
sudo ./nv_ota_start.sh $HOME/ota_payload_package.tar.gz
sudo reboot

# After reboot, dd fails at some point:
...
Writing secondary_gpt partition with gpt_secondary_1_3.bin
Sha1 checksum matched for /mnt/ota_work/internal_device/images-R32x-R35i/gpt_secondary_1_3.bin
read-back partition secondary_gpt to check whether to update it
Offset is not aligned to K Bytes, no optimization is applied
dd if=/dev/mmcblk0 of=/tmp/tmp.img bs=1 skip=31272713728 seek=0 count=16896
dd: /dev/mmcblk0: cannot skip: Invalid argument
0+0 records in
0+0 records out
0 bytes copied, 0.000168192 s, 0.0 kB/s
write /mnt/ota_work/internal_device/images-R32x-R35i/gpt_secondary_1_3.bin into partition secondary_gpt
Offset is not aligned to K Bytes, no optimization is applied
dd if=/mnt/ota_work/internal_device/images-R32x-R35i/gpt_secondary_1_3.bin of=/dev/mmcblk0 bs=1 skip=0 seek=31272713728 count=16896
dd: /dev/mmcblk0: cannot seek: Invalid argument
0+0 records in
0+0 records out
0 bytes copied, 0.000199936 s, 0.0 kB/s
Offset is not aligned to K Bytes, no optimization is applied
dd if=/dev/mmcblk0 of=/tmp/tmp.img bs=1 skip=31272713728 seek=0 count=16896
dd: /dev/mmcblk0: cannot skip: Invalid argument
0+0 records in
0+0 records out
0 bytes copied, 0.000173856 s, 0.0 kB/s
Sha1 checksum does not match (da39a3ee5e6b4b0d3255bfef95601890afd80709 != 1acfb37c81c1020b56c41537d90b809cc85e2c86) for /tmp/tmp.img
Writing secondary_gpt partition done
Failed to run_task Write sdmmc_user secondary_gpt /mnt/ota_work/internal_device/images-R32x-R35i/flash.idx
Error happens when running "l4t_update_partitions.sh jetson-agx-xavier-devkit upgradetasklist.txt"
/mnt/ota_work
Failed to run "ota_update_all_in_recovery /mnt/ota_work"
Reached OTA max retries (0 times)
ERROR: Failed to run command and keep in recovery mode to waiting for check

Rude experimental workaround:

# check storage size on xavier
cat /sys/block/mmcblk0/size
61071360

# replace size in L4T bootloader xml configurations
find bootloader/ -name "*.xml" -type f -exec sed -i 's/61079552/61071360/g' {} +

Then OTA update finishes. But, most likely, now, it is broken for another boards…

At first glance, OTA works with boards that have size 61079552 too. Tested on 2888-400-0001-J.0-1-2-jetson-agx-xavier-devkit-. nVidia, please, look if this fix is OK 🙏

It seems, that only 64GB AGX Xavier “C” can have different size :

cat /sys/block/mmcblk0/size /etc/nv_boot_control.conf
# C
61071360
TNSPEC 2888-402-0005-C.0-1-2-jetson-agx-xavier-devkit-internal
COMPATIBLE_SPEC 2888-402-0005--1-2-jetson-agx-xavier-devkit-
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mmcblk0boot0
TEGRA_OTA_GPT_DEVICE /dev/mmcblk0boot1

# G
61079552
TNSPEC 2888-402-0005-G.0-1-2-jetson-xavier-mmcblk0p1
COMPATIBLE_SPEC 2888-402-0005--1-2-jetson-agx-xavier-devkit-
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mmcblk0boot0
TEGRA_OTA_GPT_DEVICE /dev/mmcblk0boot1

32GB AGX Xaviers:

# P
61079552
TNSPEC 2888-400-0004-P.0-1-2-jetson-xavier-mmcblk0p1
COMPATIBLE_SPEC 2888-400-0004--1-2-jetson-agx-xavier-devkit-mmcblk0p1
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mmcblk0boot0
TEGRA_OTA_GPT_DEVICE /dev/mmcblk0boot1

# R
61079552
TNSPEC 2888-400-0004-R.0-1-2-jetson-xavier-mmcblk0p1
COMPATIBLE_SPEC 2888-400-0004--1-2-jetson-agx-xavier-devkit-mmcblk0p1
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mmcblk0boot0
TEGRA_OTA_GPT_DEVICE /dev/mmcblk0boot1

# L
61079552
TNSPEC 2888-400-0004-L.0-1-2-jetson-xavier-mmcblk0p1
COMPATIBLE_SPEC 2888-400-0004--1-2-jetson-agx-xavier-devkit-mmcblk0p1
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mmcblk0boot0
TEGRA_OTA_GPT_DEVICE /dev/mmcblk0boot1

16GB AGX Xavier:

# J
61079552
TNSPEC 2888-400-0001-J.0-1-2-jetson-agx-xavier-devkit-
COMPATIBLE_SPEC 2888-400-0001-E.0-1-2-jetson-agx-xavier-devkit-
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mmcblk0boot0
TEGRA_OTA_GPT_DEVICE /dev/mmcblk0boot1

hello pim_pam,

let me have confirmation, it’s OTA from 4.6.5 to 5.1.4 fail with 64GB AGX Xavier “C” only, right?
please give it a try to update ota_board_specs.conf for adding this board spec to handling it specially.

yes, correct.

Could you give me some instructions how to do that?

hello pim_pam,

64GB AGX Xavier “C” shows 61071360 sectors, which shows 29.12109375 GiB
for other series, they’re having 61079552 sectors, which shows 29.125 GiB

let’s give it a try to shrink the EMMCSIZE from the board configuration file,
for instance, p2972-0000.conf.common, please try with below for testing.

- EMMCSIZE=31276924928;
+ EMMCSIZE=31245887078;

OTA failed:

line[77]: Update_B1:Write:sdmmc_user:secondary_gpt:images-R32x-R35i/flash.idx
stage=Update_B1 action=Write device=sdmmc_user partition=secondary_gpt index_file=/mnt/ota_work/internal_device/images-R32x-R35i/flash.idx
Active index file is /mnt/ota_work/internal_device/images-R32x-R35i/flash.idx
Number of lines is 90
max_index=89
action=Write item=89, 1:3:secondary_gpt, 31272713728, 16896, gpt_secondary_1_3.bin, 16896, fixed-<reserved>-0, 7e24ac28ff9ee144e054ea28d829b057a6d67757
Do write action
Writing secondary_gpt partition with gpt_secondary_1_3.bin
Sha1 checksum matched for /mnt/ota_work/internal_device/images-R32x-R35i/gpt_secondary_1_3.bin
read-back partition secondary_gpt to check whether to update it
Offset is not aligned to K Bytes, no optimization is applied
dd if=/dev/mmcblk0 of=/tmp/tmp.img bs=1 skip=31272713728 seek=0 count=16896
dd: /dev/mmcblk0: cannot skip: Invalid argument
0+0 records in
0+0 records out
0 bytes copied, 9.008e-05 s, 0.0 kB/s
write /mnt/ota_work/internal_device/images-R32x-R35i/gpt_secondary_1_3.bin into partition secondary_gpt
Offset is not aligned to K Bytes, no optimization is applied
dd if=/mnt/ota_work/internal_device/images-R32x-R35i/gpt_secondary_1_3.bin of=/dev/mmcblk0 bs=1 skip=0 seek=31272713728 count=16896
dd: /dev/mmcblk0: cannot seek: Invalid argument
0+0 records in
0+0 records out
0 bytes copied, 8.5152e-05 s, 0.0 kB/s
Offset is not aligned to K Bytes, no optimization is applied
dd if=/dev/mmcblk0 of=/tmp/tmp.img bs=1 skip=31272713728 seek=0 count=16896
dd: /dev/mmcblk0: cannot skip: Invalid argument
0+0 records in
0+0 records out
0 bytes copied, 8.848e-05 s, 0.0 kB/s
Sha1 checksum does not match (da39a3ee5e6b4b0d3255bfef95601890afd80709 != 7e24ac28ff9ee144e054ea28d829b057a6d67757) for /tmp/tmp.img
Writing secondary_gpt partition done
Failed to run_task Write sdmmc_user secondary_gpt /mnt/ota_work/internal_device/images-R32x-R35i/flash.idx
Error happens when running "l4t_update_partitions.sh jetson-agx-xavier-devkit upgradetasklist.txt"
/mnt/ota_work
Failed to run "ota_update_all_in_recovery /mnt/ota_work"
Reached OTA max retries (0 times)
ERROR: Failed to run command and keep in recovery mode to waiting for check

hello pim_pam,

let’s try changing the sectors size from 61079552 to 61071360 in the following xml files,
for instance,
flash_t194_sdmmc.xml
flash_t194_sdmmc_R32x_R35i.xml
flash_t194_sdmmc_R35A_R35i.xml
flash_t194_sdmmc_rootfs_ab.xml
flash_t194_sdmmc_rootfs_ab_R32x_R35i.xml
flash_t194_sdmmc_rootfs_ab_R35A_R35i.xml

after that, you should re-generate the OTA payload package, and apply the image-based OTA again.

It works, as expected, because it basically the same fix:

# replace size in L4T bootloader xml configurations
find bootloader/ -name "*.xml" -type f -exec sed -i 's/61079552/61071360/g' {} +

besides the flash_t194_uefi_sdmmc_min.xml which seems not used anywhere

Is there any chance to get L4T update with bug fixed? 🙏

hello pim_pam,

actually, we don’t see this issue locally, and there’s no plan for check-in changes.
please see-also Topic 357716, JP-5 is reaching EOL.

eh, it’s disappointing…

actually, we don’t see this issue locally

if it is true, then there are different versions of “C” 64G Xaviers:

ours (more than 20 devices):

cat /sys/block/mmcblk0/size
61071360

yours:

cat /sys/block/mmcblk0/size
61079552

I have no other explanation…