PREEMPT_RT and regular kernel show similar latencies using cyclictest

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

Hi!

We are having difficoulties verifying the RT capabilities of our jetson agx xavier using the PREEMPT_RT patch and the manual from here: https://docs.nvidia.com/jetson/archives/r35.4.1/DeveloperGuide/text/SD/Kernel/KernelCustomization.html

We ran the system once without the PREEMPT_RT patch, booted it up, opened a terminal while in the Ubuntu config still, installed and ran cyclictest using this command:

cyclictest --loops=1000000 --interval=100 --histogram=200 --quiet --policy=fifo --priority=99 > nonRT.txt

We then ran this exact test again using a PREEMPT_RT patched variant of the kernel. uname -a correctly shows the “PREEMPT RT” part. Here are the results compared with each other in a histogram:

newplot
newplot1900×911 16.9 KB

As you can see there is no significant difference in latencies which is unexpected since this is a major metric PREEMPT_RT is trying to influence.

Have any of you ran a comparison test like this one? If so what where your results and if they differ from ours: How did you set up the system exactly to get to these better results?

Sincerly,

Jannick Bremm

3

Hi,

Have you run these commands before the test?

echo 100 > /sys/kernel/debug/tegra_mce/rt_window_us
echo 20 > /sys/kernel/debug/tegra_mce/rt_fwd_progress_us
echo 0x7f > /sys/kernel/debug/tegra_mce/rt_safe_mask

You may also want to run

sudo nvpmodel -m 0
sudo jetson_clocks

to ensure performance is not gated by CPU clocks.

4

Hi Dave!

I have not! Sounds very interesting. I will run these tests asap. Until then: Do you mind explaining what these are or where I can find more information about this?

Thanks!

5

Sorry we don’t provide document on it.
They are some registers on Xavier SoC controlling behaviors with RT kernels.

6

newplot(1)
newplot(1)1900×911 18.2 KB

So that apparently solved the issue. Just like that. Did I miss some documentation which could have lead me to this or was my only chance in succeeding asking you? I’m curious because we spent several weeks debugging this without success and I’m wondering if there is more documentation I need to review to be able to properly use this system in the way we intended to.

7

Sorry but it’s all we have:
https://docs.nvidia.com/jetson/archives/r35.4.1/DeveloperGuide/text/SD/Kernel/KernelCustomization.html#using-the-jetson-linux-real-time-kernel-package