Real-Time Kernel Problems

Hi NVIDIA, I’m looking for a hard real time solution on Jetson Xavier AGX, and I have some questions:

  1. I saw that NVIDIA have a release version for hard real-time for Xavier AGX (NVIDIA Document Link) I want to know that if this is a hard real-time kernel or a soft real-time kernel or a patch of PREEMPT-RT (a type of soft real-time)?

  2. Is there any way to install a custom kernel to a Xavier AGX and NX? E.g: a Linux kernel or a Linux kernel with Xenomai/RTAI.

  3. Is there any protection process similar to prohibiting modification of the source code during the Xavier AGX startup process? If yes, Is there any way that we can by pass this process?

Note: I tried to install Xenomai 3.1 (Xenomai’s introduction link) to Linux 4.9.51, 5.4.72, 5.4.119, 5.4.135 and flash those kernels to the Xavier, but all of them are failed, only the black screen or NVIDIA logo appeared.

Note 2: I tried to install Xenomai 4 (NVIDIA’s problem link) and I saw as same as problem with this guy, I tried with Jetpack 4.3, Jetpack 4.5.1, and Jetpack 4.6 and all of them are failed: 4.3 showed the black screen, 4.5.1 and 4.6 show the error as same as that guy.

When you download the Jetpack on the host Ubuntu machine, you also get all the driver / binary packages needed to build (or re-build) a kernel.
You can check out the source bits that are needed separately (or they may be included now, I haven’t checked.)
You can then start with “make ARCH=arm64 O=$TEGRA_KERNEL_OUT tegra_defconfig” and build and flash the kernel of your dreams! But you must use the kernel version that Nvidia specifically recommends in the jetpack, or all bets are off.
I think the correct instructions can be found by starting here:
https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/kernel_custom.html

The flash scripts have a step where they bundle in the NVIDIA non-GPL binaries into the image to be flashed, and then write them to the file system image, and then do the whole flash-over-USB thing.

There is no protection on the devkits, although the hardware does have fuses that, when configured, can be impossible to unconfigure (depending on specifics.) I highly recommend not turning them on, even for shipping hardware. People who believe their compiled software instructions are worth more than all the knowledge and experience that the engineers that made the software imbue, and thus believe someone could steal the precious family jewels by disassembling said software, will think differently. Anyway, if you don’t turn them on on your devkit, the devkit will not be protection-fused.

I have no idea which specific patches go into the NVIDIA rt kernel, but presumably you can just download and check …

How… did you build and flash those kernels to the Xavier? Those are not supported kernels from NVIDIA. (well, 4.9 is, at least if you get it from the nvidia repos – don’t know if they’re 100% upstream these days, in the past-time I seem to recall that didn’t happen very reliably.)
Did you cross-compile them from some system, using these instructions, just pulling later tags?
Did you download some binary blobs intended for some other system?
Did you use the scripts that add the appropriate board support?
There’s nothing in those kernels twiddling the nvidia-specific bits in the hardware by default.

So, if you use the nvidia “here’s how to build a kernel” instructions, and then download/apply the Xenomai patches, and if they apply cleanly and don’t use features that are unsupported on Xavier/ARM64, then that could work. More progress reports on that would be interesting!
Given that the Xavier has its own custom set of support peripherals, as well as a custom ARM core implementation, I wouldn’t hold out too much hope, though …

Also you need export below setting for kernel building.

export LOCALVERSION=-tegra

Hi @snarky @ShaneCCC, this is my information:

How… did you build and flash those kernels to the Xavier?
=> I tried to build the kernel directly from Xavier (Xenomai 3) or cross-compile from ubuntu 20 host (Xenomai 4).
Did you cross-compile them from some system, using these instructions, just pulling later tags?
=> I built the kernel directly on the Xavier and copy the dtb/image/source_modules_after_build to the host device and flash the xavier (flash kernel or flash kernel-dtb). And yes, I tried both the first tag and the latest tag.
Did you download some binary blobs intended for some other system?
=> No
Did you use the scripts that add the appropriate board support?
=> Yes, I use the recommend command for the board Xavier AGX.
Also you need export below setting for kernel building: export LOCALVERSION=-tegra
=> Yes, I did this.

Unluckily, 4.9.51 is the only version of Linux 4.9 which supported by Xenomai 3 for arm64 system. The others versions are 4.19 and 5.4.

It sounds like you did the things according to the instructions.
I guess if you want to run Xenomai on Jetson, you’ll either have to debug why 4.9.51 doesn’t work, or try to port it to 4.9.253 and see what happens.

I still think you’re likely to run into problems with the peripherals on the Jetson being different than the tested/supported peripherals for Xenomai, as the ARM64 world doesn’t have the benefit of more-or-less standard ACPI systems…

It’d be good if a nvidia person could answer your question about which particular changes are in the “real time” kernel variant, because I want to know, too!