Update rce firmware without flash

[liviolima80]

Good morning,
I have a system with the following features:

• Orin AGX Dev kit
• 4 GMSL custom cameras
• Jetpack 5.1.3
• Application based on Argus Library

I’m experiencing the same camera problem as described in

AGX Orin camera issues

We had seen issues on SCILs 0 to 2 (CSI 0 to 5 / AB, CD and EF). SCIL 1 problem can be easily replicated on one of the modules, the issues does not seem to affect one brick over others. I tested the semaphore fix on third faulty module and got [ 1883.968405] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 1, flags: 0, err_data 4194404 [ 1883.993429] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 2, flags: 0, err_data 4194404 [ 1884.018468] tegra-camrt…

so after an undefined time (from one to 6 hours) one of the camera connected stop and i have the following error

[ 4114.811194] [RCE] ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] “General error queue is out of sync with frame queue. ts=4138277494048 sof_ts=4138308140928 gerror_code=2 gerror_data=404 notify_bits=20000”

I already tried all the other suggestions as increse clocks and increase CAPTURE_TIMEOUT_MS to 5000.

I want to try to update the Camera RCE firmware as explained here

AGX Orin camera issues

hello jetson-developer, could you please try this rce-fw update for JP-5.1.3 / l4t-r35.5.0, Topic293597_Jun05_rce-fw.7z (199.1 KB) you should replace it with the default firmware binary, i.e. $OUT/Linux_for_Tegra/bootloader/camera-rtcpu-t234-rce.img here’s flash command-line, you may perform partition flash to update the rce-fw on your target. for instance, $ sudo ./flash.sh -r -k A_rce-fw jetson-agx-orin-devkit mmcblk0p1

I cannot reflash the system to Jetpack 5.1.4 neither use the flash script connect the board to USB since it is remotely installed. I can only connect to ssh.

Is it possible to perform the firmware update uploading the .img file to the board and then using some command sequence?

Regards

[JerryChang]

hello liviolima80,

it’s possible to update rce-fw remotely.
since rce-fw belong to bootloader, you may use the OTA payload to update bootloader remotely.
here’s an example, we’ve also tested and verified rce-fw update without re-flashing.
for instance,
(1) Please update rce-fw on your local host machine,
i.e. please update $OUT/Linux_for_Tegra/bootloader/camera-rtcpu-t234-rce.img with the latest rce firmware binary.
(2.1) You should also visit L4T page, jetson-linux-r3550 to download [OTA Tools] package.
(2.2) You may refer to developer guide for Preparing the OTA Payload Package section to prepare the OTA payload, and see-also Steps Performed on the Host Machine for reference.
(3) Running below commands to generate an OTA payload package, here’s an example to create OTA payload for only updating Bootloader for Orin Nano DevKit.
for instance, $ sudo -E ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh --external-device nvme0n1 -b jetson-orin-nano-devkit R35-5
(4) Here’s generated OTA payload, you may deliver this (via ssh) to your remote targets for running bootloader update.
$ ls $OUT/Linux_for_Tegra/bootloader/jetson-orin-nano-devkit/ota_payload_package.tar.gz

[liviolima80]

Hi @JerryChang ,
I tried to follow the steps for OTA payload package creation and I created the package only after fixing the error described in the following link

Image-based OTA package generation failing

Hi, all. I’m currently trying to generate an OTA update image to upgrade a system running 35.2.1 to 35.3.1. I’m using a freshly-generated sample filesystem for each of them, and I’ve followed along with the instructions here. The payload generation completes with the following errors: t19x payload generation complete with 4 failure(s) SUCCESS: cleaned up BUP tmp files prior to payload creation FAILURE: no payload made for config "boardid=2888;fab=400;boardsku=0001;boardrev=D.0;fuselevel_s=1;ch…

Next I try to follow the section Steps Performed on the Jetson Device, but when I luch the update I get the following

orin@ubuntu:~/update/Linux_for_Tegra/tools/ota_tools/version_upgrade$ sudo ./nv_ota_start.sh /ota/ota_payload_package.tar.gz
Command: ./nv_ota_start.sh /ota/ota_payload_package.tar.gz
Current rootfs is on /dev/mmcblk0
init_ota_log /ota_log
Create log file at /ota_log/ota_20240907-144508.log
OTA_LOG_FILE=/ota_log/ota_20240907-144508.log
Extract /ota/ota_payload_package.tar.gz
update_nv_boot_control_in_rootfs /ota_work
3701-300-0000--1--jetson-agx-orin-devkit-
Info. Installing mtdblock.
Info. Active boot storage: mmcblk0
Info. Legacy mode: false
TNSPEC 3701-500-0000-K.0-1-1-jetson-agx-orin-devkit-
COMPATIBLE_SPEC 3701-300-0000--1--jetson-agx-orin-devkit-
TEGRA_LEGACY_UPDATE false
TEGRA_BOOT_STORAGE mmcblk0
TEGRA_EMMC_ONLY false
TEGRA_CHIPID 0x23
TEGRA_OTA_BOOT_DEVICE /dev/mtdblock0
TEGRA_OTA_GPT_DEVICE /dev/mtdblock0
Info: Write TegraPlatformCompatSpec with 3701-300-0000--1--jetson-agx-orin-devkit-.
Info. Uninstalling mtdblock.
check_prerequisites
decompress_ota_package ota_package.tar /ota_work
decompress_ota_package: start at Sat 07 Sep 2024 02:45:12 PM CEST
Sha1 checksum for /ota_work/ota_package.tar (b92fe94707e62e2bd8b90ad75294add9d3d97877) matches
decompress_ota_package: end at Sat 07 Sep 2024 02:45:14 PM CEST
nv_ota_update_without_layout_change.sh
Command: nv_ota_update_without_layout_change.sh
check_target_board /ota_work TARGET_BOARD
get_chip_id CHIP_ID
ota_choose_images /ota_work
COMPATIBLE_SPEC=3701-300-0000--1--jetson-agx-orin-devkit-
TEGRA_CHIPID=0x23
_BOARD_SPEC_NAME=3701-300-0000-
Copy files from ./images-R35-ToT/3701-300-0000-/ to ./images-R35-ToT/
is_rootfs_a_b_enabled ROOTFS_AB_ENABLED ROOTFS_CURRENT_SLOT
ROOTFS_AB_ENABLED=0
ROOTFS_CURRENT_SLOT=0
is_rootfs_encryption_enabled ROOTFS_ENC_ENABLED
ROOTFS_ENC_ENABLED=0
get_update_slot UPDATE_SLOT
UPDATE_SLOT=B
get_update_control /ota_work UPDATE_BOOTLOADER UPDATE_ROOTFS
UPDATE_BOOTLOADER=1, UPDATE_ROOTFS=0
select_uefi_capsule /ota_work
check_bootloader_version /ota_work
update_bootloader /ota_work
trigger_uefi_capsule_update /ota_work jetson-agx-orin-devkit
Mount esp partition on /opt/nvidia/esp
Copying /ota_work/TEGRA_BL.Cap into /opt/nvidia/esp/EFI/UpdateCapsule
Triggering UEFI capsule update by writing \x07\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00 to UEFI variable OsIndications-8be4df61-93ca-11d2-aa0d-00e098032b8c
dd if=/tmp/var_tmp.bin of=OsIndications-8be4df61-93ca-11d2-aa0d-00e098032b8c bs=12

but after a while the ssh consol seems frezeed and the connection is closed. I can’t understand if the the bootloader update has been done. I tried to reboot the device but I cannot see any changes. I see that 2 folder /ota_log and /ota_work have been created and are not empty but the file in ota_log only report the output above of the command nv_ota_start. In /ota_work i have the following content

orin@ubuntu:~$ ls -l /ota_work/
total 27876
-rw-r--r-- 1 root root        5 Sep  7 12:53 base_version
-rw-r--r-- 1 root root       22 Sep  7 12:53 board_name
-rw-r--r-- 1 root root    86016 Sep  7 12:53 BOOTAA64.efi
drwxr-xr-x 3 root root     4096 Sep  7 12:51 external_device
-rw-r--r-- 1 root root     1653 Sep  7 14:45 file_checklist.txt
-rw-r--r-- 1 root root        1 Sep  7 12:53 layout_change
-rwxr-xr-x 1 root root    27831 Sep  7 12:53 nv-l4t-bootloader-config.sh
-rwxr-xr-x 1 root root    21556 Sep  7 12:53 nv_ota_check_version.sh
-rw-r--r-- 1 root root    46645 Sep  7 12:53 nv_ota_common.func
-rw-r--r-- 1 root root     2151 Sep  7 12:53 nv_ota_common_utils.func
-rw-r--r-- 1 root root     2856 Sep  7 12:53 nv_ota_customer.conf
-rwxr-xr-x 1 root root     4291 Sep  7 12:53 nv_ota_preserve_data.sh
-rwxr-xr-x 1 root root    25372 Sep  7 12:53 nv_ota_rootfs_updater.sh
-rwxr-xr-x 1 root root     2741 Sep  7 12:53 nv_ota_run_tasks.sh
-rwxr-xr-x 1 root root     9929 Sep  7 12:53 nv_ota_update_all_in_recovery.sh
-rwxr-xr-x 1 root root     9857 Sep  7 12:53 nv_ota_update_rootfs_in_recovery.sh
-rwxr-xr-x 1 root root     4245 Sep  7 12:53 nv_ota_update.sh
-rwxr-xr-x 1 root root     1971 Sep  7 12:53 nv_ota_validate.sh
-rw-r--r-- 1 root root     1481 Sep  7 12:53 ota_backup_files_list.txt
-rw-r--r-- 1 root root      277 Sep  7 14:45 ota_nv_boot_control.conf
-rw-r--r-- 1 root root 28225809 Sep  7 12:53 TEGRA_BL.Cap
-rw-r--r-- 1 root root       11 Sep  7 12:53 update_control
-rw-r--r-- 1 root root       19 Sep  7 12:53 user_release_version
-rw-r--r-- 1 root root      109 Sep  7 12:53 version.txt

What could I check to see if the process is completed?

Regards

[JerryChang]

hello liviolima80,

here’s checksum of rce-fw reported during kernel initialization,
please examine the checksum since you’re updating rce-fw remotely.
for instance,
this is checksum re-cap from AGX Orin/ r35.4.1
tegra186-cam-rtcpu bc00000.rtcpu: firmware version cpu=rce cmd=6 sha1=8676d22a01443bdbc66581b7dde3b6d6828cc925

[liviolima80]

Hi @JerryChang ,
I tried the whole process again but the result is the same. During the command
sudo ./nv_ota_start.sh /ota/ota_payload_package.tar.gz
the cpu is rebooted, but after the reboot no update is done. The OTA package is correct since if I check the file generate in

orin@ubuntu:~$ cat /ota_work/version.txt
NV4
# R35 , REVISION: 6.0
BOARDID=3701 BOARDSKU=0005 FAB=300
20240909102613
0x230600
BYTES:85 CRC32:F1F927E8

But the checksum of RCE firmware is the old one
tegra186-cam-rtcpu bc00000.rtcpu: firmware version cpu=rce cmd=6 sha1=55ecd57df677fd722e795b49ccb283f065b222a1

Do you have any idea why the OTA process does not complete?

Regards

[JerryChang]

hello liviolima80,

according to comment #4, may I know your commands to generate an OTA payload package.

[liviolima80]

Hi @JerryChang ,
R35.6.0/Linux_for_Tegra$ sudo -E ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh --external-device nvme0n1 -b jetson-agx-orin-devkit R35-5

since I need to upgrade from Jetpack 5.1.3 to Jetpack 5.1.4

Regards

[liviolima80]

I realized that I don’t need to use the option --external-device since the AGX boot from internal emmc.

So I repeated the process but the behaviour is the same, the only difference is that after the command
dd if=/tmp/var_tmp.bin of=OsIndications-8be4df61-93ca-11d2-aa0d-00e098032b8c bs=12

i get a some kernel messages of type
kernel:[ 527.797261] BUG: workqueue lockup - pool cpus=8 node=0 flags=0x0 nice=0 stuck for 73s!

and after a while the cpu reboot.

[JerryChang]

hello liviolima80,

hold-on, why you have to update rce-fw since you’re already moving to JP-5.1.4 release version.
may I double confirm what (or, where it from) binary file you would like to use?

[liviolima80]

Hi @JerryChang ,
I need to move from Jetpack 5.1.3 (current) to 5.1.4 (target) in order to update RCE firmware. I did the OTA procedure using 35.5 as BASE_BSP and 35.6 as TARGET_BSP

Regards

[JerryChang]

so, you should running with Updating Jetson Linux with Image-Based Over-the-Air Update for moving whole L4T version to r35.6.0

[liviolima80]

This is exactly what I tried to do

[JerryChang]

BTW, we’ve tested locally to confirm OTA update from r35.5 to r35.6 success on AGX-Orin.

[liviolima80]

Ok, in order to retry the whole process, is to possible to apply OTA from 35.6 to 35.6?

Now my system is at 35.6 since I dicided to reflash the system with SDK Manager

Regards