xavier PREEMPT rt patch

I apply the rt patch successfully, but the cycle test result is not normal,

nvidia@jetson-xxxxxxxxxxxxx:~$ uname -a
Linux jetson-xxxxxxxxxxxxx 4.9.108-rt76-rt #1 SMP PREEMPT RT Mon Dec 24 04:42:00 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux
nvidia@jetson-xxxxxxxxxxxxx:~$ zcat /proc/config.gz | grep CONFIG_HZ
# CONFIG_HZ_PERIODIC is not set
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000
nvidia@jetson-xxxxxxxxxxxxx:~$ zcat /proc/config.gz | grep PREEMPT
CONFIG_PREEMPT_RCU=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_RT_BASE=y
CONFIG_HAVE_PREEMPT_LAZY=y
CONFIG_PREEMPT_LAZY=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT__LL is not set
# CONFIG_PREEMPT_RTB is not set
CONFIG_PREEMPT_RT_FULL=y
CONFIG_PREEMPT_COUNT=y
# CONFIG_DEBUG_PREEMPT is not set
# CONFIG_PREEMPT_TRACER is not set

the install of the rt os is ok, the result of the cycle test:

nvidia@jetson-xxxxxxxxxxxxx:~/Documents/rt-tests$ sudo ./cyclictest -t 5 -p 80 -n
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.25 0.15 0.17 2/721 6607          

T: 0 ( 6557) P:80 I:1000 C:   2630 Min:      9 Act:   21 Avg:   18 Max:     105
T: 1 ( 6558) P:80 I:1500 C:   1753 Min:      8 Act:   79 Avg:   22 Max:      91
T: 2 ( 6559) P:80 I:2000 C:   1315 Min:      8 Act:   21 Avg:   18 Max:      86
T: 3 ( 6560) P:80 I:2500 C:   1052 Min:      9 Act:   24 Avg:   23 Max:     113
T: 4 ( 6561) P:80 I:3000 C:    876 Min:      9 Act:   16 Avg:   18 Max:     131

a few second later, the Max will increase to almost 800

Is anything wrong?

I haven’t patched, but just speculating. Did you first set “sudo nvpmodel -m 0”? Did you follow that up with “sudo ~nvidia/jetson_clocks.sh”? If not, try those steps, and then run the test again.

Yes, the

sudo nvpmodel -m 0
sudo ~nvidia/jetson_clocks.sh

steps works for Jetson TX2, but failed on Xavier. We still looking for a way to patch on Xavier correctly.

What was the failure message on nvpmodel and jetson_clocks.sh? Both will depend on kernel features which are custom to the Xavier. It is possible that some feature required for these broke or was removed from the RT patch. I could see removing some energy saving feature, but mostly that would be a bad sign if those functions no longer work.

In fact, the “failure” here means there was no improvement before and after turning on the mode.

sudo nvpmodel -m 0
sudo ~nvidia/jetson_clocks.sh

The max latency test by

sudo ./cyclictest -t 5 -p 80 -n

is still higher than 800 which is abnormal.

On the contrary, Jetson TX2 improved greatly in max latency before and after turning on the mode.

So there was no failure message on nvpmodel and jetson_clocks.sh jumped out on the screen obviously.

And the fan worked OK.

the

sudo nvpmodel --query

gives

NV Power Mode: MAXN
0

the

sudo ./jetson_clocks.sh --show

gives

SOC family:tegra194  Machine:jetson-xavier
Online CPUs: 0-7
CPU Cluster Switching: Disabled
cpu0: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
cpu1: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
cpu2: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
cpu3: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
cpu4: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
cpu5: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
cpu6: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
cpu7: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
GPU MinFreq=1377000000 MaxFreq=1377000000 CurrentFreq=1377000000
EMC MinFreq=204000000 MaxFreq=2133000000 CurrentFreq=2133000000 FreqOverride=1
Fan: speed=255

Other config info is the same as the first post

nvidia@jetson-xxxxxxxxxxxxx:~$ uname -a
Linux jetson-xxxxxxxxxxxxx 4.9.108-rt76-rt #1 SMP PREEMPT RT Mon Dec 24 04:42:00 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux
nvidia@jetson-xxxxxxxxxxxxx:~$ zcat /proc/config.gz | grep CONFIG_HZ
# CONFIG_HZ_PERIODIC is not set
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000
nvidia@jetson-xxxxxxxxxxxxx:~$ zcat /proc/config.gz | grep PREEMPT
CONFIG_PREEMPT_RCU=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_RT_BASE=y
CONFIG_HAVE_PREEMPT_LAZY=y
CONFIG_PREEMPT_LAZY=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT__LL is not set
# CONFIG_PREEMPT_RTB is not set
CONFIG_PREEMPT_RT_FULL=y
CONFIG_PREEMPT_COUNT=y
# CONFIG_DEBUG_PREEMPT is not set
# CONFIG_PREEMPT_TRACER is not set

So we are looking for a RT patch that works as Jetson TX2 do.

Looks like performance was indeed at max. One final thing to check is that the direct hardware access libraries are in place. If this shows all “ok”, then there is some other issue, but it is fast to check:

sha1sum -c /etc/nv_tegra_release

Neither the Cortex-A series nor Linux were designed for hard realtime, so even if things work as expected you won’t get better than soft realtime. I won’t be able to offer better advice, but you might mention some specifics on what subset of realtime needs improvement for your specific case (there may be workarounds, but a benchmark test doesn’t really say what specific needs you have).

thank you for your help,

it gives

/usr/lib/aarch64-linux-gnu/tegra/libnvosd.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvgov_camera.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_image.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvomx.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvgov_force.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnveventlib.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmedia.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_utils.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libglx.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvexif.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvrm_gpu.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvtx_helper.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvargus_socketserver.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvscf.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvfnetstorehdfx.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmm_parser.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvrm.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmm_contentpipe.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvdla_utils.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvgov_gpucompute.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvgov_tbc.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvos.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvtnr.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvgov_graphics.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvimp.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvfnet.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvphs.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvavp.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvcapture.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_video.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvfnetstoredefog.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvgov_il.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvodm_imager.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvargus_socketclient.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvtvmr.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvdla_compiler.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvtracebuf.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvidia-egl-wayland.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvdc.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvgov_boot.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvgov_generic.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libtegrav4l2.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmm.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvapputil.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvcameratools.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvcam_imageencoder.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnveglstream_camconsumer.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmm_utils.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvomxilclient.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvwinsys.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvargus.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvgov_spincircle.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvphsd.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnveglstreamproducer.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvdla_runtime.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvll.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvrm_graphics.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvcolorutil.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvddk_2d_v2.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvtestresults.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvcamv4l2.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvcamerautils.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvcamlog.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvparser.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvddk_vic.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvdla_core.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvdla_os.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvgov_ui.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmmlite.so: OK
/usr/lib/aarch64-linux-gnu/libv4l/plugins/libv4l2_nvvidconv.so: OK
/usr/lib/aarch64-linux-gnu/libv4l/plugins/libv4l2_nvvideocodec.so: OK
/usr/lib/xorg/modules/drivers/nvidia_drv.so: OK
/usr/lib/xorg/modules/extensions/libglx.so: OK

In fact, the soft realtime is enough for us now. It will be OK if the max latency benchmark test performs as well as Jetson TX2.

The original working environment is a normal desktop (intel core i5-7500 cpu@3.4GHZx4) with soft realtime OS(PREEMPT RT ubuntu16.04) to control a 12 degree of freedom robot using ethercat and tensorflow. The max latency benchmark test on it remains stable below 100, and works well on it.

To make it smaller, we transfer the environment to Jetson TX2 with a special jetpack including soft realtime OS (PREEMPT RT ubuntu16.04) successfully, whose max latency benchmark test is also remains stable below 100.

It is said that Xavier is much powerful than TX2, so we tried to transfer it with soft realtime OS.
However, the max latency benchmark test shows there may be something wrong with my installation about PREEMPT rt patch. Because the test is also higher that 800 if I failed to install on the original desktop environment.

So we are still looking for a way to install PREEMPT RT patch that the max latency benchmark test on it remains stable below 100. It will be good if the OS is also ubuntu16.04 just as TX2 did, since the gcc and other libs wouldn’t be change and no compatibility problem will happen.

I know this requirement is rare and may not have a solution yet, but thanks for supporting so long any way.

Beyond what was already mentioned there isn’t much I can add. I have experimented with Xavier under different loads and it does remain responsive (not benchmarked, just my feel for the Xavier) under heavier loads than does the TX2. Someone else may have additional comments on making the RT patches work.

I just installed the latest JetPack 4.1.1 Developer Preview for Xavier and it seems already have the PREEMPT installed?
(I didn’t apply any patch)

nvidia@jetson-0423018054325:~/dev/rt-tests-1.3$ uname -a
Linux jetson-0423018054325 4.9.108-tegra #1 SMP PREEMPT Wed Oct 31 15:17:21 PDT 2018 aarch64 aarch64 aarch64 GNU/Linux
nvidia@jetson-0423018054325:~/dev/rt-tests-1.3$ gzip -cd /proc/config.gz | grep PREEMPT
CONFIG_PREEMPT_RCU=y

CONFIG_PREEMPT_NONE is not set

CONFIG_PREEMPT_VOLUNTARY is not set

CONFIG_PREEMPT=y
CONFIG_PREEMPT_COUNT=y

CONFIG_DEBUG_PREEMPT is not set

CONFIG_PREEMPT_TRACER is not set

That is correct. The question from the original poster is about adding more patches. I don’t know how much of that would be present at this current kernel, but for what is present, then the “CONFIG_PREEMPT=y” is definitive.

Hello,

We also plan to use XAVIER with RT.
Is it that with latest jetpack the RT patch is already applied ?
If I understand this thread correctly, although the patch is already in, the performance is not as expected. Is that right ?

Thanks,
ranran

Hi erwin.coumans:

I think the default JetPack 4.1.1 Developer Preview is not PREEMPT RT if you do not apply any patches and rebuild it.

As far as I know, those configure is necessary.

CONFIG_PREEMPT_RT_FULL=y
CONFIG_HZ_1000=y
CONFIG_HZ=1000
CONFIG_NO_HZ_FULL=y

Hi, ranchu,

I believe the patches would work, since it worked on TX2.

The problem may caused by my wrongly compile kernel.

You can have a try to apply the patches , follow the guideline of https://github.com/kozyilmaz/nvidia-jetson-rt/blob/master/docs/README.03-realtime.md