Trouble building PREEMPT_RT for Jetson Orin Nano Jetpack 6.1 R36.4.0

Hi,

I am trying to build a custom kernel with PREEMPT_RT on my Jetson Orin Nano with Jetpack 6.1 and R36.4.0 and I’m running into issues. I’m on my 4th go. First I compiled the kernel in WSL2, but wasn’t able to copy/install anything on the Jetson because the USB connection would timeout. Next I tried compiling the kernel on my linux laptop and then I executed the flash.sh script but that failed because it couldn’t find the mount point/NVMe. And now I’ve tried to compile the kernel locally on the Jetson Orin Nano itself and then just copied the necessary files from my temporary build folder into /boot/Image, / boot/dtb, and /lib/modules.

I used a modified version of the scripts found here GitHub - hmxf/RTJetson: Preempt-RT Kernel Build Guide for NVIDIA Development Board to work with R36.4 so I could grab the correct files.

Also I looked at the instructions here Kernel Customization — NVIDIA Jetson Linux Developer Guide 1 documentation but wasn’t able to get the new kernel to boot.

Here is the script I used to build the kernel locally on the Jetson Orin Nano followed by the serial console logs.

#!/bin/bash

# Red is 1
# Green is 2
# Reset is sgr0

BUILD_DIR=~/RTJetsonBuild/R36.4.0
cd $BUILD_DIR

tput setaf 2
echo "Extract files"
tput sgr0
sudo tar xpf Jetson_Linux_R36.4.0_aarch64.tbz2
cd Linux_for_Tegra/rootfs/
sudo tar xpf ../../Tegra_Linux_Sample-Root-Filesystem_R36.4.0_aarch64.tbz2
cd ../../
#tar -xvf aarch64--glibc--stable-2022.08-1.tar.bz2
sudo tar -xjf public_sources.tbz2
tar -xjf Linux_for_Tegra/source/kernel_src.tbz2

tput setaf 2
echo "Apply PREEMPT-RT patches"
tput sgr0
sudo ./generic_rt_build.sh enable

tput setaf 2
echo "Compile kernel"
tput sgr0
TEGRA_KERNEL_OUT=kernel_out
mkdir $TEGRA_KERNEL_OUT
cd kernel/kernel-jammy-src
make O=$TEGRA_KERNEL_OUT tegra_defconfig

tput setaf 2
echo "Confirm if these config options are chosen."
echo "General setup -> Preemption Model (Fully Preemptible Kernel (Real-Time))"
echo "Kernel Features -> Timer frequency: 1000 HZ "
echo "If not, choose them in menuconfig interface."
echo "Else, quit menuconfig and compile will auLto start."
echo "Press Return Key to continue........"
tput sgr0
read
make O=$TEGRA_KERNEL_OUT menuconfig
make O=$TEGRA_KERNEL_OUT -j$(nproc)

tput setaf 2
echo "Copying results"
tput sgr0
sudo cp kernel_out/arch/arm64/boot/Image $BUILD_DIR/Linux_for_Tegra/kernel/Image
sudo cp kernel_out/arch/arm64/boot/Image.gz $BUILD_DIR/Linux_for_Tegra/kernel/Image.gz
sudo cp -r kernel_out/arch/arm64/boot/dts/nvidia/* $BUILD_DIR/Linux_for_Tegra/kernel/dtb/
sudo make O=$TEGRA_KERNEL_OUT modules_install INSTALL_MOD_PATH=$BUILD_DIR/Linux_for_Tegra/rootfs/
cd $BUILD_DIR/Linux_for_Tegra/rootfs/
sudo tar --owner root --group root -cjf kernel_supplements.tbz2 lib/modules
sudo mv kernel_supplements.tbz2  ../kernel/

tput setaf 2
echo "Applying binaries"
tput sgr0
cd ..
sudo ./apply_binaries.sh

And here are the serial console logs. These are two attempts I made at booting after compiling the kernel and trying to use it.

serial-console-rt-kernel-log.txt (64.8 KB)
serial-console-rt-kernel-log-2.txt (75.6 KB)

Hi,
For manually building kernel image, please use host PC in Ubuntu 20.04 or 22.04. It may not work properly to run the steps on WSL2.

Ad there are additional steps for building RT kernel. The steps are validated. If you follow it one by one, it should work fine.

Kernel Customization — NVIDIA Jetson Linux Developer Guide 1 documentation

Hi @DaneLLL thanks for your response.

I started from Kernel Customization — Jetson Linux Developer Guide documentation. I manually downloaded the public_sources.tbz2 from Jetson Linux | NVIDIA Developer and extracted it.

The first issue I had came immediately after I unzipped the public_sources, and the issue is that this folder path in the instructions doesn’t exist

$ cd Linux_for_Tegra/source/public

So I had to go to Linux_for_Tegra/source and extract the kernel_src.tbz2 in that folder instead since that’s where it existed.

$ tar –xjf kernel_src.tbz2

Then I followed the steps to export the cross compiler variable according to the instructions

$ export CROSS_COMPILE_AARCH64_PATH=<toolchain-path>
$ export CROSS_COMPILE_AARCH64=<toolchain-path>/bin/aarch64-buildroot-linux-gnu-

Then I created the kernel_out directory according to the instructions. Again I had to do that in Linux_for_Tegra/source/ since the location it was supposed to be done in doesn’t exist (Linux_for_Tegra/source/public ).

Then I ran the following build script acccording to the instructions

$ ./nvbuild.sh -o $PWD/kernel_out

Next the instructions have you setCROSS_COMPILE_AARCH which I did

export CROSS_COMPILE_AARCH=/home/jetson/preempt_rt/R36.4.0/aarch64--glibc--stable-2022.08-1/bin/aarch64-linux-gnu-

The next issue I had here was an error saying that CROSS_COMPILE has not been set. So I had to manually do that since the instructions don’t mention that.

export CROSS_COMPILE=/home/jetson/preempt_rt/R36.4.0/aarch64--glibc--stable-2022.08-1/bin/aarch64-linux-gnu-

Then the script was able to run successfully after I fixed that and the following is the final output of that script to the terminal.

make: Leaving directory '/home/jetson/preemp_rt/R36.4.0/Linux_for_Tegra/source/kernel_out/kernel'
Kernel sources compiled successfully.
Directory "nvethernetrm" is not found, exiting

I went to the next set of instructions

5. Replace Linux_for_Tegra/rootfs/usr/lib/modules/$(uname -r)/kernel/drivers/gpu/nvgpu/nvgpu.ko with a copy of this file:

$kernel_out/drivers/gpu/nvgpu/nvgpu.ko

Butnvgpu.kodoesn’t exist and neither does the path that I’m being asked to replace that with so I can’t follow those instructions either.

Hi,
It may be due to permission of the folder, please try
Building the kernel from kernel source jetpack 6 , jetson orin nx 16GB - #5 by DaneLLL

Hi @DaneLLL Did you read what I wrote in my second reply? I did mention that I was able to get the scripts to run, but I had to make a few changes because the paths outlined in the documentation aren’t correct. I’m able to successfully run the scripts without sudo permissions so I’m not sure what that will do.

I did my due diligence, however, and ran all of those same steps with sudo permissions like you suggested, but I’m in the exact same place. I still don’t have the nvgpu.ko that the documentation says to copy into rootfs. In addition to not having the nvgpu.ko file, there is no rootfs folder after following these instructions. So I can’t do the following steps because the file and folders referenced don’t exist:

5. Replace
Linux_for_Tegra/rootfs/usr/lib/modules/$(uname -r)/kernel/drivers/gpu/nvgpu/nvgpu.ko with a copy of this file:

$ kernel_out/drivers/gpu/nvgpu/nvgpu.ko

Hi,
Please download

Jetson Linux | NVIDIA Developer
Driver Package (BSP)
Sample Root Filesystem

And follow the steps to extract it:

Quick Start — NVIDIA Jetson Linux Developer Guide 1 documentation

So that you have the default system image for Orin Nano developer kit.

Hi @DaneLLL

I have already flashed my jetson with the SDK manager, do these steps play any role in that?

Also if unzipping the Driver Package and Sample Root Filesystem are important part of building and installing PREEMPT_RT kernel then why aren’t they included here Kernel Customization — Jetson Linux Developer Guide documentation?

Also in what order should I do these things? Customizing the kernel says to unzip public_sources, doesn’t say anything about the Driver Package or Sample Root Filesystem. Do I unzip the BSP and the the Sample root file system, and then go back to the kernel customization?

I promise I’ve gone over these instructions thoroughly, but they are all over the place and have had incorrect information at every step so far. And now I have to download other source files that aren’t mentioned in building the RT kernel. If I’m mistaken please correct me. I’m trying my best, it’s been quite a confusing journey so far though.

Thanks

Hi,
Do you mean you don’t have this:

Linux_for_Tegra/rootfs/usr/lib/modules/5.15.148-tegra/updates/nvgpu.ko

Or this:

$kernel_out/drivers/gpu/nvgpu/nvgpu.ko

Hi @DaneLLL

Okay I just now realized that the trouble I had was because I stumbled onto the R35.3.1 Kernel Customization instructions. Somehow I managed to stumble there. And that was the source of a lot of confusion because I was using the R36.4 source files. Silly me.

Okay now I’m on the correct version of the instructions for R36.4 Kernel Customization — NVIDIA Jetson Linux Developer Guide 1 documentation. I think you gave me the instructions for R36.3, which are event slightly different than R36.4.

I’m skipping
Signing and Encrypting the Kernel, the kernel-dtb, and the initrd Binary Files,
Using the Jetson Linux Real-Time Kernel Package, and
Real-Time Kernel Using OTA Update
because I don’t think I need to worry about those.

I’ve made it to this step here Installing the Real-Time Kernel Packages on a Jetson Device. Two questions I have here are

  1. How do I make the tick frequency of the PREEMPT_RT kernel 1000Hz (or whatever the max is)?
  2. How do I actually flash my jetson with this PREEMPT_RT image that I just made? I didn’t that step in the instructions.

I’m confused because I just built the kernel and enabled the real time aspect of it (supposedly according to the instructions) and the next set of instructions doesn’t tell me what to do with the kernel that I just built. And it doesn’t specify how to flash the RT kernel to the jetson. These are the next steps here below, and from the looks of it I just compiled a kernel for no reason because apparently you can just install the RT kernel package with apt:

Open the apt source configuration file in a text editor, for example:

$ sudo vi /etc/apt/sources.list.d/nvidia-l4t-apt-source.list

Add the RT kernel repository:

deb https://repo.download.nvidia.com/jetson/rt-kernel main

Save and close the source configuration file.

Enter the following command:

$ sudo apt update

Install the RT kernel packages:

$ sudo apt install nvidia-l4t-rt-kernel nvidia-l4t-rt-kernel-headers nvidia-l4t-rt-kernel-oot-modules nvidia-l4t-display-rt-kernel

Reboot your Jetson device after the installation is finished:

$ sudo reboot

So what is next after Building the DTBs? That’s the last step I completed.

Hi,
The deb package is for AGX Orin developer kit. For Orin Nano developer kit, would need to follow the steps to manually build it. Please follow the comment to flash Orin Nano developer kit. To make sure the files under Linux_for_Tegra folder is good. And then follow the guidance to build RT kernel:

Kernel Customization — NVIDIA Jetson Linux Developer Guide 1 documentation

And overwrite it to the same files under Linux_for_Tegra. And re-flash developer kit.

You should need to re-build

Kernel Customization — NVIDIA Jetson Linux Developer Guide 1 documentation
Kernel Customization — NVIDIA Jetson Linux Developer Guide 1 documentation

DTB may not be required since it is for deviation between custom board and developer kit.

Hi @DaneLLL

I have manually built everything according to the guide.

I believe I was able to actually successfully build it because it executed uname -a and saw PREEMPT_RT in the output.

$ uname -a
Linux oss 5.15.148-rt-tegra #1 SMP PREEMPT_RT Thu Dec 12 23:51:49 EST 2024 aarch64 aarch64 aarch64 GNU/Linux

I did get a few errors while it was building, though it said it was successful. And also the screen glitches out a little bit sometimes. When booting up or shutting down I’ll see gray static patterns on the screen (sometimes the whole screen, sometimes just a strip across the screen). I’ll send pictures of the screen and the log errors I’m seeing. But I want to get the fastest tick rate possible first and then I’ll rebuild and try flashing again before I share those.

A few more things, when I look at the configuration I see the following

$ zcat /proc/config.gz | grep PREEMPT
CONFIG_HAVE_PREEMPT_LAZY=y
CONFIG_PREEMPT_LAZY=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y
CONFIG_PREEMPT_RCU=y
# CONFIG_DEBUG_PREEMPT is not set
# CONFIG_PREEMPT_TRACER is not set
# CONFIG_PREEMPTIRQ_DELAY_TEST is not set

$ zcat /proc/config.gz | grep HZ
CONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not set
CONFIG_NO_HZ_IDLE=y
# CONFIG_NO_HZ_FULL is not set
# CONFIG_NO_HZ is not set
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250

I would like to not have a “lazy” preemptable kernel like I see in the output above.

CONFIG_HAVE_PREEMPT_LAZY=y
CONFIG_PREEMPT_LAZY=y

I would also like to change some of the frequency settings to have a 1000Hz tick rate. How do I do that? I want the following settings but I don’t know how to set them.

CONFIG_HZ_1000=y
CONFIG_HZ=1000

rather than

CONFIG_HZ_250=y
CONFIG_HZ=250

Also, something that isn’t clear, do I execute the first steps on Quick Start — NVIDIA Jetson Linux Developer Guide 1 documentation which are:

$ tar xf ${L4T_RELEASE_PACKAGE}
$ sudo tar xpf ${SAMPLE_FS_PACKAGE} -C Linux_for_Tegra/rootfs/
$ cd Linux_for_Tegra/
$ sudo ./tools/l4t_flash_prerequisites.sh
$ sudo ./apply_binaries.sh

And then follow the steps on Kernel Customization — NVIDIA Jetson Linux Developer Guide 1 documentation? It isn’t clear if l4t_flash_prerequisites.sh and apply_binaries.sh comes before or after kernel customization.

Hi,
Please enable the config to set tick frequency to 1000Hz:

CONFIG_HZ_1000

And after executing the steps in Kernel Customization, $ sudo ./apply_binaries.sh is not required and please run initrd command to flash Orin Nano developer kit.

Hi @DaneLLL

  1. Do I need to execute these 4 commands before doing the kernel customization? Without adopting the binaries like you suggested of course
    $ tar xf ${L4T_RELEASE_PACKAGE} $ sudo tar xpf ${SAMPLE_FS_PACKAGE} -C Linux_for_Tegra/rootfs/ $ cd Linux_for_Tegra/ $ sudo ./tools/l4t_flash_prerequisites.sh

  2. As far as the tick rate goes, I get that the frequency should be 1000Hz, but are there instructions on where and how to physically do that? Do I change it before I flash the Jetson? Do I change it after I flash the Jetson? Do I just manually edit that configuration file where I saw the rate before or after flashing? Is there a tool that let’s me customize additional aspects of the kernel/ real time tick rate?

Thanks

Hi,

Kernel Customization — NVIDIA Jetson Linux Developer Guide 1 documentation

sudo ./tools/l4t_flash_prerequisites.sh is required after installing the out-of-tree modules. Other steps are not required.

Hi @DaneLLL

I’m confused, earlier you said I need to download the source files and unzip them, now you’re saying vthey’re not needed, just the flash pre requisites after building the out of tree modules.

Also I still am confused about changing the tick rate settings, you still haven’t clarified that.

Forgive me but the instructions you’re giving me are a little bit all over the place, not very clear, and piecemeal. I’m trying to put them together the best I can but I think you might be making lots of assumptions about what I know. I’ve been following the documentation, and the documentation you’ve shared with me, but you’ve pointed me to different steps in different pages that don’t reference each other, and that’s not obvious to someone who isn’t a dev. And now you’re telling me some of the steps you previously said I should do don’t matter. It’s all a little confusing. I know I’m really close.

I do appreciate your help, I am a lot closer than I was before.

Hi,
Do you have the Linux_for_Tegra which is good to flash Orin Nano developer kit to standard kernel now? Maybe it is better you can flash Orin Nano developer kit with standard Jetpack release successfully first.

Please check
No data from Joystick Logitech-f710 - #10 by DaneLLL

Hi @DaneLLL,

Sorry for the delay, I’ll get on that tomorrow with the link you sent and get back to you.

Hi @DaneLLL

I have successfully flashed the plain ole standard Jetpack. I just did the following commands in this order:

$ tar xf ${L4T_RELEASE_PACKAGE}
$ sudo tar xpf ${SAMPLE_FS_PACKAGE} -C Linux_for_Tegra/rootfs/
$ cd Linux_for_Tegra/
$ sudo ./tools/l4t_flash_prerequisites.sh
$ sudo ./apply_binaries.sh

$ sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 \
  -c tools/kernel_flash/flash_l4t_t234_nvme.xml -p "-c bootloader/generic/cfg/flash_t234_qspi.xml" \
  --showlogs --network usb0 jetson-orin-nano-devkit internal

Hi,
You have completed [1]. Please do [2][3][4] and see if you can flash developer kit to RT kernel.