OTA update with R35.5.0 [PKC+SBK + Disk encryption enabled]

Continuing the discussion from OTA Update r35.4.1 on an external device nvme0n1p1 failed to update NVMe:

Hello,

I would like to update a jetsonPC from R35.4.1 to R35.5.0 release with (PKC+SBK + Disk encryption) enabled.
I created the payload using below command from R35.5.0 ota_tools :

sudo -E ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh -u jetson.pem  -v sbk_key.txt -i ./ekb.key --external-device nvme0n1 -S 40GiB  jetson-xavier-nx-devkit-emmc R35-4

As a result of the above command, “ota_payload_package.tar.gz” is created successfully. I copied this payload on the /ota/ folder and ran on the jetsonPC :

sudo ./nv_ota_start.sh /ota/ota_payload_package.tar.gz
Logs:


Command: ./nv_ota_start.sh /ota/ota_payload_package.tar.gz
Current rootfs is on /dev/nvme0n1
init_ota_log /ota_log
Creating log dir at /ota_log
Create log file at /ota_log/ota_20240314-160923.log
OTA_LOG_FILE=/ota_log/ota_20240314-160923.log
Extract /ota/ota_payload_package.tar.gz
update_nv_boot_control_in_rootfs /ota_work
3668-100---1--jetson-xavier-nx-devkit-emmc-
Info. Installing mtdblock.
Info. Active boot storage: nvme0n1
Info. Legacy mode: false
TNSPEC 3668-300-0001-B.0-1-2-jetson-xavier-nx-devkit-emmc-
COMPATIBLE_SPEC 3668-100---1--jetson-xavier-nx-devkit-emmc-
TEGRA_LEGACY_UPDATE false
TEGRA_BOOT_STORAGE nvme0n1
TEGRA_EMMC_ONLY false
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mtdblock0
TEGRA_OTA_GPT_DEVICE /dev/mtdblock0
Info: Write TegraPlatformCompatSpec with 3668-100---1--jetson-xavier-nx-devkit-emmc-.
Info. Uninstalling mtdblock.
check_prerequisites
decompress_ota_package ota_package.tar /ota_work
decompress_ota_package: start at Thu Mar 14 16:11:27 UTC 2024
Sha1 checksum for /ota_work/ota_package.tar (1453f5ae154a78ac4d6212fcf7fd1afad93a0b6f) matches
decompress_ota_package: end at Thu Mar 14 16:13:31 UTC 2024
nv_ota_update_without_layout_change.sh
Command: nv_ota_update_without_layout_change.sh
check_target_board /ota_work TARGET_BOARD
get_chip_id CHIP_ID
ota_choose_images /ota_work
COMPATIBLE_SPEC=3668-100---1--jetson-xavier-nx-devkit-emmc-
TEGRA_CHIPID=0x19
_BOARD_SPEC_NAME=3668-100--
Copy files from ./images-R35-ToT/3668-100--/ to ./images-R35-ToT/
is_rootfs_a_b_enabled ROOTFS_AB_ENABLED ROOTFS_CURRENT_SLOT
ROOTFS_AB_ENABLED=0
ROOTFS_CURRENT_SLOT=0
is_rootfs_encryption_enabled ROOTFS_ENC_ENABLED
ROOTFS_ENC_ENABLED=1
get_update_slot UPDATE_SLOT
UPDATE_SLOT=B
get_update_control /ota_work UPDATE_BOOTLOADER UPDATE_ROOTFS
UPDATE_BOOTLOADER=1, UPDATE_ROOTFS=1
select_uefi_capsule /ota_work
check_bsp_version /ota_work BASE_VERSION
update_misc_partitions /ota_work
update_partition_with_alt /ota_work recovery
install_partition_with_alt /ota_work/external_device/images-R35-ToT recovery
prerequisite_check recovery
No image is specified for partition recovery
Skip updating recovery partition as no valid image is found
update_partition_with_alt /ota_work recovery-dtb
install_partition_with_alt /ota_work/external_device/images-R35-ToT recovery-dtb
prerequisite_check recovery-dtb
No image is specified for partition recovery-dtb
Skip updating recovery-dtb partition as no valid image is found
update_partition_with_alt /ota_work esp
install_partition_with_alt /ota_work/external_device/images-R35-ToT esp
prerequisite_check esp
The /ota_work/external_device/images-R35-ToT/esp.img for partition esp is not found
Skip updating esp partition as no valid image is found
update_rootfs /ota_work
update_rootfs_with_a_b_disabled /ota_work
update_rootfs_in_recovery /ota_work
force_booting_to_recovery jetson-xavier-nx-devkit-emmc
Force booting to recovery by writing \x07\x00\x00\x00\x03\x00\x00\x00 to UEFI variable L4TDefaultBootMode-781e084c-a330-417c-b678-38e696380cb9
dd if=/tmp/var_tmp.bin of=L4TDefaultBootMode-781e084c-a330-417c-b678-38e696380cb9 bs=8
1+0 records in
1+0 records out
8 bytes copied, 0.032502 s, 0.2 kB/s
Rootfs is to be updated in recovery kernel once device is rebooted.
check_bootloader_version /ota_work
update_bootloader /ota_work
Bootloader on non-current slot(B) is to be updated once device is rebooted
clean_up_ota_files

“sudo reboot” caused the jetsonPC cut-off the power from monitor and did not wake up at all. Earlier when doing update from R35.4 to R35.4 without PKC+SBK+ disk encryption led to same power off monitor for ~10 mins but it did come up with the NVIDIA logo boot page with Update Progress bar from OTA.

Could you please check if I missed something here or the OTA logs seem OK ?
Unfortunately I could not get the serial logs as they didn’t come on minicom during bootup. It comes only after the jetson is up.

Thanks in advance !

Hi adit_bhrgv,

Are you using the devkit or custom board for Xavier NX?

Could you share the full serial console log at this moment?

It is not the expected result for me.
Serial console log would output from MB1/MB2/bootloader/kernel/rootfs during boot up.
Do you get the serial console log with a serial console cable from debug UART port?

Can you also share the log when you are generating OTA package on the host?

  • I am using a custom board for Xavier NX and it is a fused board with Secureboot,PKC and Kek2 keys

  • I contacted our hardware vendor to know about UART port settings to get the serial console logs.

  • Please find the attached OTA generation logs on the host as well as the OTA payload apply logs on the jetson.

OTA_generation_R35.5.0_disk_encryption_logs.txt (435.4 KB)
OTA_payload_logs_R35.5.0.txt (3.4 KB)

I did several other experiments too. Here are my observations:

  1. When I created OTA payload with -b option to just update the bootloader and the rootfs was not affected, I could actually see the OTA Update % screen on the monitor. However, after the Update % is finished, no login came up(just monitor power went off again). This Update % screen didn’t even pop up when a OTA payload with both bootloader and rootfs was OTAed. I just copied rootfs from R35.4.1 in R35.5.0 Linux_for_Tegra folder. Does this cause any error ? Or in other words, the rootfs in R35.5.0 needs to be generated from BSP R35.5.0 ?

  2. I also tried to generate the OTA payload (bootloader + rootfs )only with SBK+PKC keys and no disk-encryption enabled. But still I could’t get the OTA Update % screen. In this case, the rootfs was also copied from R35.4.1. Here are the logs:

OTA_generation_R35.5.0_pkc_sbk_logs.txt (425.5 KB)
OTA_payload_logs_R35.5.0_sbk_pkc.txt (3.4 KB)

Please extract sample rootfs from R35.5.0 release page to the BSP package of R35.5.0.

Do you enable disk-encryption with R35.4.1on your board before update?

Hi,

I did some other experiments to narrow down the issue (just SBK+PKC but without disk encryption enabled):

  1. “Just update bootloader” with -b option by generating payload in this manner:
sudo -E ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh -b  -u ./jetson.pem  -v  ./sbk_key.txt --external-device nvme0n1 -S 40GiB jetson-xavier-nx-devkit-emmc R35-4
  1. “Update bootloader + rootfs” without -b option by generating payload in this manner:
sudo -E ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh  -u ./jetson.pem  -v  ./sbk_key.txt --external-device nvme0n1 -S 40GiB jetson-xavier-nx-devkit-emmc R35-4

In both the cases, after the OTA Update is finished, the Jetson board goes again to RECOVERY STATE . Could you please tell me what could be reason?. I can say for sure that the keys pkc & sbk are the same as were used to fuse the board.

Bus 001 Device 096: ID 0955:7e19 NVIDIA Corp. APX

Note : The unfused board works fine with OTA UPDATE for both above cases i.e only bootloader, bootloader+rootfs

Hi @KevinFFF ,

Could you please have a look into this ?

Thanks

@KevinFFF We are currently stuck on this and don’t know how to proceed further ? Could you please suggest how can we debug this issue ? Thanks

Please answer this question from me.

May I know why you use sbk_key.txt rather than sbk.key here?

Please also noted that we don’t support image-based OTA update with secureboot enabled, you could find more details in Jetson 35.5.0 image based OTA with UEFI secure boot enabled - #2.

Yes, the disk encryption was enabled before update.
sbk_key.txt essentially contains same 16 byte key . I used .txt here because we used the same to fuse the board earlier.

So, if image-based OTA update is not supported with secureboot enabled; then why does ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh script has -u and -v options to provide pkc and sbk keys ?

Do you mean I can only use image based OTA with disk encryption enabled without secureboot but not both ?

Okay, what we don’t support is “UEFI secureboot”, which is used to protect the payload for UEFI like kernel, initrd…etc.
The secureboot in image-based OTA is supported, which is used to protect the bootloader.

Please share the full serial console log for further check.

I am still in contact with our hardware vendor to get the UART debug logs. Once, I get that I can provide you with that.

Is there any other way by which we can identify the issue as such why after the successfull OTA Update progress bar is finished running(upto 100%), the monitor goes black and Jetson is reset to "Recovery state 0955:7e19 " again. ?

Before OTA-Update:
Image running with BSP 35.4.1 with Secureboot enabled; No disk encryption ( -u jetson.pem -v sbk.key )

Payload creation:
BSP 35.5.0 OTA Payload created with same -u jetson.pem -v sbk.key . OTA Tools version 35.5.0 used to trigger the update “sudo ./nv_ota_start.sh /ota/ota_payload_package.tar.gz”

For info:
Jetson custom board fused with Public key Hash. Kek0 ,Kek1,Kek2 & SBK keys

Thanks again !

There should be too many reasons to cause this, wrong dtb configuration, any kernel panic…etc.
We would need the serial console log to check this behavior in details.

Do you mean that you could perform OTA update from R35.4.1 to R35.5.0 with Secureboot enabled successfully but you hit the boot up issue after update?

Yes, after “sudo reboot”, OTA update is triggered, monitor goes black( connection to HDMi gets lost), after around 5 mins, HDMI gets active again, we see the update progress bar from 0% to 100%.
After that, the jetson goes to Recovery state again. → In this case, the eks_t194.img was created using gen_ekb tool from R35.4.1 (old version) and then copied to R35.5.0/Linux_for_Tegra/bootloader folder before creating the OTA Payload

I also tried generating “eks_t194.img” from gen_ekb tool from R35.5.0(new version), and then copying it to R35.5.0/Linux_for_Tegra/bootloader folder before creating the OTA Payload, in this case, I don’t see any Update Progress bar from OTA and the monitor power is cut off & the jetson doesn’t boot up as before but here it doesn’t stay in Recovery mode.

Which gen_ekb tool should be used to generate eks_t194.img ideally from doing update from R35.4.1 to R35.5.0 ?

If you are generating the OTA package for R35.5.0, please just use gen_ekb tool from this release.

Please also share the full serial console log after you run the reboot command for further check.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.