Unable to flash L4T 36.4.0 on Orin NX with custom carrier - UEFI won't start in RCM boot

Autonomous Machines Jetson & Embedded Systems Jetson Orin NX
,
1

I am developing a product using an Orin NX 16GB module on a custom carrier board. It has a modified kernel, with our own dtb, and custom bootloader dts for BCT, pinmux, padvoltage, pmic, etc.

I have the board successfully flashing and booting with L4T 35.5.0 and 36.3.0.

I have attempted to update to L4T 36.4.0, but I am unable to flash the board using l4t_initrd_flash.sh. I am finding that the process fails at Step 2 as the board doesn’t boot the initrd kernel in RCM boot. Looking at the serial output and comparing to the working L4T 36.3.0, it seems that it fails to run the Jetson UEFI firmware, which is first step in booting the kernel.

Rather than the UEFI firmware starting, I see the following messages on the serial, which implies it just reboots and then nothing happens.

NOTICE:  BL31: v2.8(release):e12e3fa93
NOTICE:  BL31: Built : 21:01:44, Sep 12 2024
I/TC: 
I/TC: Non-secure external DT found
I/TC: OP-TEE version: 4.2 (gcc version 11.3.0 (Buildroot 2022.08)) #2 Fri Sep 13 04:10:17 UTC 2024 aarch64
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
I/TC: Primary CPU initializing
I/TC: Test OEM keys are being used. This is insecure for shipping products!
I/TC: fTPM ID is not enabled.
I/TC: ftpm-helper PTA: fTPM DT or EKB is not available. fTPM provisioning is not supported.
I/TC: Primary CPU switching to normal world boot
��[     6.870815] Camera-FW on t234-rce-safe started
TCU early console enabled.
[     6.917986] Camera-FW on t234-rce-safe ready SHA1=e2238c99 (crt 0.894 ms, total boot 48.115 ms)
����Shutdown state requested 1
Rebooting system ...

I have attached the full log output from the Linux side and the serial output during the flashing process. I am using Ubuntu 22.04 as the development host.

jetpack61_flashing_linux.txt (283.0 KB)
jetpack61_flashing_serial.txt (43.7 KB)

Is this a known issue with Jetpack 6.1, or is there something extra/different I need to do compared to previous releases to make this work?

3

Hi steve.wilkins,

Do you have any customization in UEFI before?
Have you tried using debug UEFI firmware to get more logs?

I/TC: fTPM ID is not enabled.
I/TC: ftpm-helper PTA: fTPM DT or EKB is not available. fTPM provisioning is not supported.
I/TC: Primary CPU switching to normal world boot

Or are you using a fused board since I don’t see these message on my un-fused devkit?

4

I do not have any customization in UEFI, I am using the default version that comes with L4T 36.4.0. The fTPM messages are new since updating to L4T 36.4, and I haven’t knowingly tried to enable anything extra in the build or initrd flashing process.

I could try a debug version of UEFI, although I will have to work out how to build it. I’m a little sceptical if will make a difference, as I don’t even see the first debug message that states the UEFI version.

I don’t believe the Orin SOM has been fused for secure boot. Is there any checks/tools I can use to confirm this?
It has had the USB3 mode in recovery fuse set as per Flashing Support — Jetson Linux Developer Guide documentation

5

Please get a debug UEFI and check what causes the shutdown.

Or you can get a devkit to compare the full boot logs.

We would also suggest you referring to Jetson Orin NX and Nano Series — NVIDIA Jetson Linux Developer Guide 1 documentation if you are developing custom carrier board.

6

Is there a debug UEFI for L4T 36.4.0 available or do I have to build it myself from source? If I have to build it, are there any build instructions?

7

Please refer to the steps in Build with docker · NVIDIA/edk2-nvidia Wiki to build UEFI binary

$ edk2_docker edkrepo clone nvidia-uefi-r36.4.0 NVIDIA-Platforms r36.4.0
$ cd nvidia-uefi-r36.4.0
$ edk2_docker edk2-nvidia/Platform/NVIDIA/Jetson/build.sh
$ cp images/uefi_Jetson_DEBUG.bin <Linux_for_Tegra>/bootloader/uefi_jetson.bin
8

I have built the debug UEFI and tried using this instead. I don’t see any difference on the debug serial output, which indicates the UEFI is not even starting to the point it outputs any debug. I did a search in the Linux_for_Tegra directory and it looks like the “Shutdown state requested” and “Rebooting system …” messages come from the BPMP firmware.

I have also acquired a dev kit carrier that I can use as a comparison. I have been able to flash a standard image L4T 36.4.0 on the dev kit using l4t_initrd_flash.sh to flash to a NVMe, so I know my Orin NX SOM is good and doesn’t have anything fused that would cause a problem. It would seem that something in our carrier board specific bootloader dts files is upsetting the boot process. I know the bootloaders are prebuilt binary blobs and the source is not public. What other options do I have to investigate what is going wrong?

9

Have you reflashed the QSPI to apply the change?
Please share the full serial console log after updating the debug UEFI firmware.

Do you refer to each sections in Jetson Orin NX and Nano Series — NVIDIA Jetson Linux Developer Guide 1 documentation to develop custom carrier board?

����Shutdown state requested 1
Rebooting system ...

From the log you shared, bpmp-fw received the request to reboot the board.
Does your board reboot after you get this message?

10

I don’t believe it is getting to the point where it will flash the QSPI. It fails at the RCM boot stage, where the UEFI has been downloaded to RAM via USB. It needs to boot the initrd kernel from RCM boot to start the flashing process.

I have attached the serial log and output from running l4t_initrd_flash.sh

jetpack61_flashing_debug_uefi_linux.txt (283.0 KB)
jetpack61_flashing_debug_uefi_serial.txt (43.7 KB)

The developer guide has been followed for developing with our custom carrier, although on an earlier L4T release. I will review the latest version.

The board does not appear to reboot. I don’t see any more serial messages. If I run lsusb, I still see the device “Bus 002 Device 101: ID 0955:7323 NVIDIA Corp. APX”.

11

It seems you have customization in board config(my-custom-board.conf), partition layout(my-custom-board-flash_rootfs_ab.xml), DTB(tegra234-p3768-0000+p3767-0000-nv-my-custom-board.dtb)…etc.

Since the current issue is specific to your custom carrier board, could you help to clarify what customization causes this issue?

��[     7.613927] Camera-FW on t234-rce-safe started
TCU early console enabled.
[     7.678640] Camera-FW on t234-rce-safe ready SHA1=e2238c99 (crt 1.351 ms, total boot 66.127 ms)
����Shutdown state requested 1
Rebooting system ...

I’ve also checked this with internal that it is from BPMP-FW received the reboot request.
We suspect it is caused from system hang at this moment.
Could you also help to enable timestamp in your serial console log for us to check if there’s any hang before that shutdown message?