Flashing Newly Built Kernel to Jetson AGX Orin causes Jetson to not boot (with logs)

[jtaghipour]

We have a Jetson AGX Orin dev kit for which we’re attempting to enable the LTC2990 configuration in the kernel. We followed the steps provided on the NVIDIA documentation for how to build and flash a kernel. After flashing the kernel, the Jetson no longer boots up. The power LED indicator turns on and so does the fan, but nothing is displayed on the monitor. Please find attached a document listing all the steps we have followed for building the kernel and flashing it to the Jetson. I have also included all of the log files (the document specifies the log file name for each step).

We would greatly appreciate it if you could let us know what we’re doing incorrectly?

Thank you.
Jetson_Kernel_Modification_Steps.pdf (248.6 KB)
Kernel_Mod_Logs.zip (164.6 KB)

[JerryChang]

hello jtaghipour,

it looks you’re not using identical release image.
please re-flash your target with JetPack-5.0.2, and compile its public release sources, l4t-r35.1

please check developer guide. Flashing a Specific Partition; your steps for flashing kernel is incorrect, since you’re not assign options to the flash script. it’ll full flash the target.

however,
please review your /boot/extlinux/extlinux.conf, it’s by default to check LINUX entry for loading kernel image, to check FDT entry for loading device tree blob via file system.
you may copy the binary files via scp and modify the extlinux configuration file instead of flashing the target.
please see-also developer guide, Switching to a Different Kernel for reference,
thanks

[jtaghipour]

Thank you for your response! When I reflash the target with JetPack-5.0.2, the Jetson does successfully turn on upon completion. However, the SDK manager indicates a failure. I’ve attached the logs. Please advise what should be done to resolve these errors.
SDKM_logs_JetPack_5.0.2_(rev._1)_Linux_for_Jetson_AGX_Orin_modules_2022-10-12_08-40-25.zip (87.2 KB)

[JerryChang]

hello jtaghipour,

I don’t understand, the flash process is complete to boot-up the target.

[ 641.8501 ] tegradevflash_v2 --write B_MEM_BCT mem_coldboot_sigheader.bct.encrypt
[ 641.8543 ] Bootloader version 01.00.0000
[ 642.0075 ] Writing partition B_MEM_BCT with mem_coldboot_sigheader.bct.encrypt [ 243712 bytes ]
[ 642.0089 ] [................................................] 100%
[ 645.0087 ] Flashing completed

...

[ 645.1713 ] Coldbooting the device
[ 645.1759 ] tegrarcm_v2 --chip 0x23 0 --reboot coldboot
[ 645.1791 ] MB2 version 01.00.0000
*** The target t186ref has been flashed successfully. ***
Reset the board to boot from internal eMMC.

may I know what’s the failure.
did you have a display monitor connected? would you please also setup a serial console to check UART logs.
thanks

[jtaghipour]

I used the SDK Manager application. I’ve attached the screenshot of the error. I also setup a serial console to check the Jetson’s UART logs. I’ve attached that as well, and yes, I have a monitor connected to both the host and the Jetson. The Jetson does boot up just fine after the flashing completes, but the SDK manager indicates a failure when I use JetPack version 5.0.2.
uart.log (27.3 KB)

[JerryChang]

hello jtaghipour,

according to uart logs, it seems it’s abort during image flashing process.
could you please try deselect those components, please flashing target only for confirmation.
BTW,
SDKManager it’s able to generate reports, please gather the logs and share them as attachment for reference,
thanks

[jtaghipour]

I have already attached the SDKManager logs in the initial post I created. Which components should I deselect?

[JerryChang]

hello jtaghipour,

you may try deselect all Jetson SDK Components (CUDA, CUDA-X AI, CV, MMAPI…etc) only flashing target for testing.

[jtaghipour]

I tried it without any Jetson SDK components enabled. And it still fails. Please find attached a screenshot of the SDK Manager as well as the logs.

SDKM_logs_JetPack_5.0.2_(rev._1)_Linux_for_Jetson_AGX_Orin_modules_2022-10-20_08-19-34.zip (38.1 KB)

[JerryChang]

hello jtaghipour,

it looks USB communication failed. are you using a virtual machine?
for example,

[   0.0268 ] File rcm_state open failed
[   0.0272 ] ERROR: failed to read rcm_state
...
[   0.8728 ] ERROR: might be timeout in USB write.
Error: Return value 3
Command tegrarcm_v2 --new_session --chip 0x23 0 --uid --download bct_br br_bct_BR.bct --download mb1 mb1_t234_prod_aligned_sigheader.bin.encrypt --download psc_bl1 psc_bl1_t234_prod_aligned_sigheader.bin.encrypt --download bct_mb1 mb1_bct_MB1_sigheader.bct.encrypt
Reading board information failed.

you may see-also Orin Developer Kit User Guide for the [How-to] session to put the developer kit into Force Recovery Mode (RCM).
please also refer to developer guide To Determine Whether the Developer Kit Is in Force Recovery Mode for image flashing.
thanks

[jtaghipour]

I am not using a virtual machine. And I have followed the instructions in those links. The unit is in force recovery mode when I want to flash the unit. I do not have this issue when I flash the unit with the JetPack 5.0.1 version. I only have this issue with JetPack 5.0.2 version which you suggested to use.

[JerryChang]

hello jtaghipour,

we’ve test and confirm Jetpack-5.0.2 to flash AGX Orin successfully.
so, could you please remove Jetpack installation folder and re-download the image to test again.

[jtaghipour]

I’ve removed the Jetpack installation folder and re-downloaded the image multiple times. This time, we went ahead and enabled the SDK components too. The kernel portion installed fine. However, the SDK portion was yielding errors (attached are images of the errors) as well as our connection. It keeps saying that it detects more than one Jetson even though we only have one Jetson connected.

[JerryChang]

you have not even connect Type-C next to 40pin header for image flashing…

[jtaghipour]

After multiple tries, we were able to flash the bare minimum JetPack 5.0.2. And fixed the flash.sh parameters that I missed, we were able to flash the kernel. However, we are still facing issues where the Jetson will not completely boot. It always gets stuck at a certain point:

[ 13.838607] using random self ethernet address
[ 13.845701] using random host ethernet address

Please find attached a write up of what we’ve done as well as all the log files we have.

Jetson_Logs_20221026.zip (148.6 KB)
Jetson_Kernel_Modification_Steps_v2.pdf (1.6 MB)

[JerryChang]

hello jtaghipour,

in page-19 of [Jetson_Kernel_Modification_Steps_v2.pdf], you may copy the built kernel image and copy it to your target instead of flashing the device.
you may check /boot/extlinux/extlinux.conf, it’s LINUX entry to load kernel image, and FDT entry for device tree blob.

besides, according to [Boot_Up.log] in the log attachment, it looks already boot-up file system.
these two messages also included in the normal boot-up Orin/r35.1 kernel logs.
[ 14.055448] using random self ethernet address
[ 14.060261] using random host ethernet address

I don’t see certain failure to cause boot hang.
hence… could you please try removing all connected peripheral devices. or, please try a HDMI display monitor to test again.
thanks

[jtaghipour]

Do you have detailed instructions on where the device tree blob needs to be copied to on the target?

Also, the Jetson AGX Orin dev kit does not have an HDMI port.

[JerryChang]

hello jtaghipour,

you may copy the device tree binary with scp utility. and, modify the extlinux.conf to specify the file path. reboot the system to make the changes take effect.

ya, my bad, there’s only DP port for AGX Orin.