NX OTA failure(Jetpack5.1.2->6.2): A_VER/B_VER CRC32 is empty and QSPI bootloader is not updated

Robotics & Edge Computing Jetson Systems Jetson Orin NX
, ,
1

APX_readonly_VER_CRC_evidence.txt (2.4 KB)

OTA_failure.txt (409.4 KB)

failed_device_uefi_variables.txt (94.3 KB)

failed_device_uefi_readonly_check.txt (39.0 KB)

Hi NVIDIA team,

We are performing an image-based OTA upgrade on Jetson Orin NX 8GB modules:

Jetson Linux R35.4.1 -> R35.5.0 -> R36.5.0
Ubuntu 20.04 -> Ubuntu 22.04
Rootfs: external NVMe

Some devices complete the upgrade successfully, while other devices fail. The current failure rate is approximately 50%.

1. Hardware information

The successful and failed devices use the same module SKU:

Module: Jetson Orin NX 8GB
P-number: P3767-0001
BOARDID: 3767
BOARDSKU: 0001

One failed device inspected through APX/RCM reports:

FAB=303
TegraPlatformCompatSpec=3767--0001--1--jetson-orin-nano-devkit-

We have not yet completed a controlled reproduction test using a Jetson Orin NX 8GB module on the NVIDIA P3768 reference carrier board.

2. Successful and failed states

The successful device reaches:

NVMe rootfs: R36.5
A bootloader chain: R36.5
B bootloader chain: R35.5

The failed device reaches:

NVMe rootfs: R36.5
A bootloader chain: R35.4.1
B bootloader chain: R35.4.1
UEFI firmware: 4.1-33958178

The failed device finally reports:

L4TLauncher: Unable to locate L4T Support protocol: Not Found
UEFI Interactive Shell v2.2

3. OTA log observations

During both OTA stages, the logs show that the rootfs/kernel partitions were written successfully, TEGRA_BL.Cap was copied, and OsIndications was set:

Copying TEGRA_BL.Cap into /opt/nvidia/esp/EFI/UpdateCapsule
Triggering UEFI capsule update by writing ... OsIndications
OTA is successfully completed

However, after every reboot, UEFI remains at version:

4.1-33958178

Therefore, the OTA process successfully stages the Capsule, but the Capsule is not applied to either QSPI bootloader chain.

4. APX/RCM read-only inspection

We performed an APX/RCM read-only inspection of one failed device. No QSPI or NVMe partition was written, erased or modified.

The A_VER and B_VER partitions were read separately from the read-only QSPI MTD device. Both contain:

NV4
# R35 , REVISION: 4.1
BOARDID=3767 BOARDSKU=0001 FAB=303
20260701162332
0x230401
BYTES:85 CRC32:

Results:

A_VER version: R35.4.1
B_VER version: R35.4.1
Stored CRC32 field: empty
Calculated CRC32 for the first 85 bytes: 6241156E

The A_VER and B_VER raw dumps are byte-for-byte identical. The complete A and B bootloader-chain regions also have identical SHA-256 values.

This confirms that neither QSPI bootloader chain was updated to R35.5 or R36.5.

How to solve this problem?

Thanks.

4

hello 3521049662,

per developer guide, Updating Jetson Linux with Image-Based Over-the-Air Update.
it has to perform the OTA update twice, firstly, r35.4.1 → r35.5.0 as base version.
secondly, you may updates from version r35.5.0 to version r36.5.0.

you may see-also similar issue, Topic 362627 and Topic 332557.
let’s have below checks to narrow down this..

  1. nvbootctrl dump-slots-info
  2. /sys/firmware/efi/esrt/entries/entry0/last_attempt_status
  3. The original flash host’s python-is-python3/zlib1g state
  4. Its generated qspi_bootblob_ver.txt
5

Hi Jerry,

Thank you for you reply. Our OTA upgrade is performed in two stages:R35.4.1 → R35.5.0 and R35.5.0 → R36.5.0.

1. Failed device: nvbootctrl

$ sudo nvbootctrl dump-slots-info

Current version: 0.0.1
Capsule update status: 0
Current bootloader slot: A
Active bootloader slot: A
num_slots: 2
slot: 0,             status: normal
slot: 1,             status: normal

2. ESRT last_attempt_status

$ sudo cat /sys/firmware/efi/esrt/entries/entry0/last_attempt_status

6162

3. Ubuntu PC used to generate the OTA archives

$ command -v python
/usr/bin/python

$ python --version
Python 3.12.3

$ python3 --version
Python 3.12.3

$ dpkg -l python-is-python3
ii  python-is-python3  3.11.4-1  all  symlinks /usr/bin/python to python3

$ dpkg -l zlib1g
ii  zlib1g:amd64  1:1.3.dfsg-3.1ubuntu2.1  amd64

$ dpkg -l zlib1g-dev
ii  zlib1g-dev  1:1.3.dfsg-3.1ubuntu2.1  amd64

$ python3 -c 'import zlib;print(zlib.ZLIB_VERSION)'
1.3

4. Generated qspi_bootblob_ver.txt

$ cat bootloader/qspi_bootblob_ver.txt

NV4
# R36 , REVISION: 5.0
BOARDID=3767 BOARDSKU=0000 FAB=000
20260803150704
0x240500
BYTES:85 CRC32:86F4D26F

Thanks.

7

hello 3521049662,

this should be a corrupted pre-existing QSPI VER partition, not an OTA-staging failure.
last_attempt_status=6162 which due to the empty A/B CRC32, causes version parsing to fail.
it seems identical with the forum thread, Topic 362627, forum developer has resolved the issue, which caused by missing python-is-python3 package on the host.

8

Hi,JerryChang,

We found the same root cause described in this topic.

Our original R35.4.1 BSP generated qspi_bootblob_ver.txt with an empty CRC32
field because the original flashing host was missing python-is-python3.

The boards could boot and run normally after the initial flash, but later OTA
Capsule updates failed with ESRT status 6162 because A_VER/B_VER were invalid.

For new boards, we will fix the host environment and regenerate the BSP/QSPI
artifacts.

For deployed boards that already contain invalid A_VER/B_VER data, could NVIDIA
please provide the officially supported recovery method?

翻译

9

hello 3521049662,

unfortunately, another UEFI Capsule update cannot repair this condition; you cannot restore the VER partition remotely.

a target needs to enter forced-recovery mode, you may omit re-flashing external NVMe rootfs.
please re-flash its bootloader partition (i.e. QSPI), here’s sample flash command to flash only QSPI.
for instance,
$ sudo ./tools/kernel_flash/l4t_initrd_flash.sh -p "-c bootloader/generic/cfg/flash_t234_qspi.xml --no-systemimg" jetson-orin-nano-devkit internal

10

Hi, there is actually a fix which can be performed remotely. Custom EFI binary can be automatically executed from UEFI shell and can do the job. We fixed few of such devices in field this way. Please check this repo: GitHub - kyberpunk/nvidia-jetson-corrupted-ver-partition-fix: Fix of corrupted VER partition in Nvidia Jetson · GitHub . But you need to have some kind of remote access - SSH or capability to execute some custom script remotely.

11

Hello, I’m glad to receive your reply. We have noticed this open source project and are currently in the process of verifying it. I will keep you updated on the progress