Jetson Orin Nano RT Kernel won't boot

I custom build a RT kernel but I could not boot from it.

I’m 100% following this instruction:
https://docs.nvidia.com/jetson/archives/r36.2/DeveloperGuide/SD/Kernel/KernelCustomization.html
Except the part of signing the image and secure boot because I wasn’t sure how to do it.

Toolchain I used aarch64-glibc-stable-2022.08-1.tar.bz2
Compiled on a x86 ubuntu 22.04 host machine

Then I tried to flash the image with 3 ways:

  1. Nvidia’s SDK manager. After I recompile the image, SDK manager still thinks the image is ready and flashed it to my Jetson Orin Nano, without reverting it back to old image
  2. sudo ./flash.sh jetson-orin-nano-devkit nvme0n1p1
  3. sudo ./l4t_initrd_flash.sh --flash-only jetson-origin-nano-devkit nvme0n1p1

I tried the same process and build a kernel with ./generic_rt_build.sh “disable”, it boots normally. Looks like there’s something wrong with the RT enable specifically

Hi,
6.0 DP is developer preview and there may be potential issue. Please try 5.1.3 and see if you still hits the issue.

Thank you! Do you have instructions on how to install older version of JetPack SDK? I could not download other versions of JetPack other than 6.0 DP. Is it because I’m on Ubuntu 22.04 on my host machine?

This is a bug in our kernel compilation workflow.

NVMe drivers in initrd have to match the kernel you are using, but the current situation is that drivers in initrd are still built for the stock kernel, while you are trying to load the RT kernel, then the NVMe driver fails to load, and you don’t have access to rootfs so booting gets stuck.

Please manually unpack initrd following this method:
https://docs.nvidia.com/jetson/archives/r36.2/DeveloperGuide/SD/FlashingSupport.html#modifying-the-ram-disk

Replace /lib/modules/($uname -r)/ in the unpacked folder with the one you build, like it should originally be 5.15.122-tegra, and replace it with your 5.15.122-rt-tegra. Pack it back into initrd and flash again.

Also, your command for flashing is not right:

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1
-c tools/kernel_flash/flash_l4t_external.xml -p “-c bootloader/generic/cfg/flash_t234_qspi.xml”
–showlogs --network usb0 jetson-orin-nano-devkit internal

We will provide a tool in JetPack 6 GA to automate this process of updating initrd, or we will also provide our pre-built RT kernel on our APT server, so you don’t need to build it yourself.

Thank you! I’ll give it a try!

Yeah It would be awesome if the kernel is on the APT server!

I still couldn’t get it work

I copied ./bootloader/l4t_initrd.img to /tmp and deleted 5.15.122-tegra from lib/modules/ and add 5.15.122-rt-tegra from rootfs/lib/modules to /tmp/temp/lib/modules, recompress /tmp/l4t_initrd.img and copied back to ./bootloader/l4t_initrd.img.

it’s still not booting, with the same error shown

I then further update rootfs/boot/initrd

I also did same for rootfs/boot/initrd. copy rootfs/boot/initrd to /tmp, deleted 5.15.122-tegra from lib/modules/ and add 5.15.122-rt-tegra from rootfs/lib/modules to /tmp/temp/lib/modules, recompress /tmp/initrd and copied back to rootfs/boot/initrd.

it’s still not booting, with the same booting screen shown, and it cannot finish flashing because of timeout

Dump the serial console log.

flash_log.txt (9.9 KB)

I mean this one:

Anyway, this issue should only affect booting but not flashing.
If you fail in the flashing stage, then the issue lies somewhere else.

OK I’ll try to get a debug console setup.

Just want to make sure, there’re 2 initrd in the Jetpack SDK on my host

rootfs/boot/initrd
bootloader/l4t_initrd.img

Do I need to change both? By deleting 5.15.122-tegra from them and add 5.15.122-rt-tegra into them

Thank you very much!

Ideally both.

is the bootloader/initrd automatically created by copying bootloader/l4t_initrd.img during flashing? I explicitly deleted it before flash, but when I flash it appears again.

YES, exactly.

I see. It seems my flash fails because it’s waiting for some sort of response from Jetson Orin Nano board (Flash Jetson Linux - flash: Waiting for target to boot-up), but the target hang at the booting screen, so the flash script timeout and failed.

We still need the log. Description doesn’t really help.

OK I ordered TTL cable arrives tomorrow. I’ll get you the log as soon as I get the cable.

But from what I experimented:

  1. If I don’t change bootloader/l4t_initrd.img, rootfs/boot/initrd, it finishes the flash no problem, but cannot boot
  2. If I only change bootloader/l4t_initrd.img, leave rootfs/boot/initrd as the old one, it finishes the flash no problem, but cannot boot
  3. If I change both rootfs/boot/initrd and bootloader/l4t_initrd.img, it fails to flash

Hi,
You may be trying 5.1.3 and see if it works. Since the host is in Ubuntu 22.04, it may not be able to use SDKManager and please download the packages from the page:

https://developer.nvidia.com/embedded/jetson-linux-r3550
Driver Package (BSP)
Sample Root Filesystem

And follow the steps to extract and flash the image:
Quick Start — NVIDIA Jetson Linux Developer Guide 1 documentation

And then try RT kernel.

I tried 5.1.3 and it works well with non-RT kernel. When switched to RT kernel, it seems it booted but it stuck in a black screen with a curser flashing, I’m not able to input any command. In the tutorial it says

In Orin, fix the black screen. This is not required on Xavier.

  1. Flash the system into your Jetson device.
  2. At the first boot, log in, type sudo depmod -a, and then reboot. The GUI should now be back.

However I was not able to log in during the boot. Where should I type sudo depmod -a command?

Hi,
Please connect uart console and you can execute the command in the console.

Hey, when is the RT installation scheduled to come out in APT? It will be sooo useful.