Testing the RT Kernel and Troubleshooting RT Kernel Package Dependencies on Orin NX 8GB (Jetpack 6.2, L4T 36.4)

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin NX
,
1

Good afternoon everyone

Config:

  • Jetpack 6.2
  • Orin Nano NX 8GB @ 20W
  • R36.4
  • PREEMPT_RT

I have one question further down the post, I’ll send my updates on flashing my board with RT-Kernel and some tests. My question is related to the naming of the debian that we populate our rootfs using ./apply_binaries.sh .

When doing the OTA you would do:

sudo apt install nvidia-l4t-rt-kernel nvidia-l4t-rt-kernel-headers nvidia-l4t-rt-kernel-oot-modules nvidia-l4t-display-rt-kernel

But since i’m working on a custom kernel build to be able to flash on NVMe i can’t use the OTA.

I folowed the steps for the custom RT Kernel. Both kernel build and board flashing were successful, and my system is running in RT mode as verified by:

$ cat /sys/kernel/realtime
1
$ uname -r
5.15.148-rt-tegra
$ zcat /proc/config.gz | grep PREEMPT_RT
CONFIG_PREEMPT_RT=y
$ zcat /proc/config.gz | grep CONFIG_HZ
# CONFIG_HZ_PERIODIC is not set
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250

Yes, I left it on 250 Hz, no 1000Hz ticks for now.

Tests with Cyclictest

Under stress and with jetson_clocks activated, 20W (@ 6cores).

plot
plot640×480 5.16 KB

For those wondering the script to create the image was:
"mklatencyplot.bash "

sudo /usr/bin/jetson_clocks
stress-ng --cpu 4 --io 2 --vm 1 --vm-bytes 1G --timeout 60s
# Then run the script which has this inside: 
cyclictest -l60000 -m -Sp90 -i5000 -h1000 -q >output

The Issue

I’m encountering dependency errors when installing a custom Debian package that depends on either nvidia-l4t-kernel or nvidia-l4t-rt-kernel. I use this .deb as a check to confirm if the RT kernel is installed.

Currently, after flashing, the package manager shows:

$ apt list nvidia-l4t-kernel
nvidia-l4t-kernel/stable,now 5.15.148-tegra-36.4.3-20250107174145 arm64 [installed]
$ apt list nvidia-l4t-rt-kernel
Listing... Done

So, only nvidia-l4t-kernel is installed and recognized, not nvidia-l4t-rt-kernel, which makes sense since I did not rename it.

My Questions

  • What changes are needed in the Debian package names or in the apply_binaries.sh process to rename the RT kernel package to (nvidia-l4t-rt-kernel)
  • Is it normal that on a 5 minutes test (60k samples over 200Hz) i got a maximum latency of 358 us? I’ll reflash on a normal Jetpack 6.2 and see the difference under the same test.
  • Is there any test for me to know that everything went down properly?
3

Hi,
Please not to run apply_binaries.sh after you have the customized kernel. The script is only execute while preparing required files under Linux_for_Tegra.

You may refer to

Jetson/L4T/r36.4.x patches - eLinux.org
[Jetpack 6.1/r36.4.0] Self-built kernel is overwritten after sudo apt update && sudo apt upgrade
Enable RT kernel and CONFIG_HZ_1000

4

Hi @DaneLLL

I did run ./apply_binaries.sh before doing my customization.

It’s more about the naming of the debian package than anything else, and from the references that you send I just discovered another potential issue that could emerge with apt update && apt upgrade thanks for that.

Which leads to my question again, shouldn’t we have some way to change the debian package names to a customized one when we customize the kernel, this way even the apt update && apt upgrade would be solved too.

The rest of the post was just to create a forum post to other developers to have a reference.

:)

5

Hi,
The proposal is for example, nvidia-l4t-kernel can be separated to nvidia-l4t-kernel-r36.4.0 and nvidia-l4t-kernel-r36.4.3. Is this understanding correct?

6

Hello,

The proposal is: when flashing a board with a kernel customization, to be able to change the .deb names (have a script to do that).

nvidia-l4t-<custom>-kernel
nvidia-l4t-<custom>-kernel-headers
nvidia-l4t-<custom>-kernel-oot-modules
nvidia-l4t-display-<custom>-kernel

This would avoid the need for the apt-mark hold, and also in my case I would be able to create my other debians that depend on this custom kernel to work.

The proposal that you gave could be useful in other situations, but for me I wanted to know how to customize those names with a custom tag to say that it’s custom, and if that is possible at all.

Thanks for your time.

7

Adding to the post, same test under 5min. but now we see a much higher max time of 2.1ms when flashed on non-rt (this time under 15W just saw that i had 4 cores).

plot
plot640×480 5.27 KB

1 Like
9
  • Jetpack 6.2
  • Orin Nano NX 16GB

My Jetson has a maximum latency of 2200us under the RT kernel. Do you have any optimization methods

10

Can you confirm that you are using the rt-kernel with this commands?

$ cat /sys/kernel/realtime
1
$ uname -r
5.15.148-rt-tegra
$ zcat /proc/config.gz | grep PREEMPT_RT
CONFIG_PREEMPT_RT=y
$ zcat /proc/config.gz | grep CONFIG_HZ
# CONFIG_HZ_PERIODIC is not set
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250

Also, im running on both tests I've sent sudo jetson_clocks` to put the arm clocks at max frequency since the embedded arms used on the jetson go from 2 MHz to 700 KHz when idle.

Other than that you can check if using CONFIG_HZ = 1000 makes it better, i saw some people doing it since it fires the kernel interrupt clock faster to reorganize the scheduler at this rate, I didn’t test it nor know what else that can impact (check that post).

Also, which function are you using to calculate those timings?

1 Like
11

There is also the Note talking about the UEFI service, you can deactivate it and try again.

The UEFI runtime services are enabled by default which may increase latency. We do not recommend using UEFI runtime services while running RT applications. Refer to commit ba7585a4a5580b4cba26ebe76050bd2ef1148948 for more information.

1 Like
12

Thank you for your response. As you can see, I am still in the beginner’s stage in many areas. For the real-time content of Linux, I have only been exposed to it for one month, and English is not my native language, so I am not proficient in it.

Regarding this issue, I am using

sudo cyclictest -m -S -p 80 -i 1000 -d 0

The tests conducted.

On the Jetson platform, my testing performance was very poor.
Whether I am using Nvidia’s official deb package management to update the kernel to the real-time kernel or compiling the kernel according to the manual, the results of both tests are the same as those of a regular kernel.

The same rt tests command, on an x86 host subscribed to Ubuntu Pro, the latency is only in single digits, which is normal.
Of course, on the same x86 host, I did not use the assistance of Ubuntu Pro. I compiled my own Xenomai kernel, and its latency performance was not as good as Ubuntu Pro’s.

So, in fact, I suspect that after applying any real-time patches, I still need to do some operations to truly activate the real-time kernel, and Jetson may be nothing more than that.

13

I’ll risk sounding a little repetitive, but launch this commands and post them here. I’m also curious as why i’m seeing different results.

To check give more info on your jetson, and if the rt is activated:

uname -r
cat /sys/kernel/realtime
zcat /proc/config.gz | grep PREEMPT_RT
cat /etc/nv_tegra_release

To put your jetson on MAXN and activating jetson_clocks
Do this and restart.

sudo nvpmodel -m 0

After reboot:

Test your cyclictest command.
Save results.

sudo jetson_clocks

Test your cyclictest command again.
Save results to compare.

I usually use jtop to see the frequency of the cores and if Jeston Clocks is activated.

Since it’s an arm64 running you could be seeing “wrong” results that are more related to idle cores and power saving policies than your real-time kernel not being activated.

I did run the same command as you on my Orin and got that:

$ sudo jetson_clocks
user@GTW-ONX8-ubuntu:~$ sudo cyclictest -m -S -p 80 -i 1000 -d 0
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 2.66 2.49 1.38 1/880 7765

T: 0 ( 5495) P:80 I:1000 C: 281063 Min:      2 Act:    3 Avg:    3 Max:     117
T: 1 ( 5496) P:80 I:1000 C: 281060 Min:      2 Act:    5 Avg:    3 Max:     131
T: 2 ( 5497) P:80 I:1000 C: 281057 Min:      2 Act:    5 Avg:    4 Max:     123
T: 3 ( 5498) P:80 I:1000 C: 281054 Min:      2 Act:    4 Avg:    4 Max:     154
T: 4 ( 5499) P:80 I:1000 C: 281047 Min:      2 Act:    7 Avg:    4 Max:     138
T: 5 ( 5500) P:80 I:1000 C: 281047 Min:      2 Act:    8 Avg:    4 Max:     137

and without jetson_clocks:

$ sudo cyclictest -m -S -p 80 -i 1000 -d 0
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 5.13 3.44 1.55 7/895 6348

T: 0 ( 3670) P:80 I:1000 C: 273104 Min:      2 Act:    2 Avg:    8 Max:    2112
T: 1 ( 3671) P:80 I:1000 C: 273091 Min:      2 Act:    4 Avg:   10 Max:    1843
T: 2 ( 3672) P:80 I:1000 C: 273049 Min:      2 Act:   36 Avg:   14 Max:    1651
T: 3 ( 3673) P:80 I:1000 C: 273115 Min:      2 Act:   29 Avg:    8 Max:     258
T: 4 ( 3674) P:80 I:1000 C: 273111 Min:      1 Act:   17 Avg:    7 Max:     377
T: 5 ( 3675) P:80 I:1000 C: 273106 Min:      1 Act:   13 Avg:    8 Max:     346

F.Y.I: English is not my first language too. LOL

1 Like
14

Thank you very much for your help. Your verification results have provided me with a very useful reference.

16

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.