PREEMPT-RT patches for Jetson Nano

Hi,

I’m trying to compile a kernel with PREEMPT-RT patches for the Jetson Nano. For this, I followed the kernel building instructions in https://docs.nvidia.com/jetson/l4t/ and the only extra step I’m doing is to apply the real-time patches included in the BSP source (in the folder rt-patches) before making the configuration:

for i in rt-patches/*.patch; do patch -p1 < $i; done

Then, I select the following options in menuconfig:

General setup -> Timer subsystem -> Timer tick handling -> Full dynticks system (tickless)
Kernel Features -> Preemption  Model: Fully Preemptible Kernel (RT)
Timer frequency: 1000 HZ

The compilation is successful, however, when I try the kernel in the board it only shows the NVIDIA logo (apparently a boot-loop). First I tried replacing the Image, dtb and modules in a previously installed microSD (with the official Nano image), but then I created my own image using the create-jetson-nano-sd-card-image.sh script:

sudo ./apply_binaries.sh
sudo ./create-jetson-nano-sd-card-image.sh -o jetson_nano.img -s 12G -r 100

However, in both cases the result is the same, I’m not able to boot.

Is there any problem with the included real-time patches or am I doing something wrong?

Regards

Hi ajcalderony69bi,

We haven’t tested on Nano yet, our team is investigating now, the status will be posted once clarified, so stay tuned.

Thanks

Thanks kayccc!
Just for the record, I also tried applying the patches using the rt_patch.sh script, and I even tried this solution: https://devtalk.nvidia.com/default/topic/1048841/jetson-agx-xavier/fix-for-rt-patch-sh-and-realtime-patches-on-xavier/post/5329675/#5329675 however, the result is the same, the board doesn’t boot. I tried creating an image without applying the real-time patches and everything works fine, so apparently there is something wrong with the patching process.

Regards

It would be great if the default Jetson Nano images (and TX2, Xavier) would use PREEMPT-RT patches by default, since most robotics usage benefits from it.

Does/did NVIDIA consider this?

Personally I would prefer a more vanilla kernel then a further specialized one. Not all of us plan on using these boards in robotics.

Ideally we would be able to boot a kernel off kernel.org and go from there.

Hi folks.

I tried to perform the rt patch application on my Jetson Nano and as the previous posts it seemed to work, that is it kernel compiled modules compiled and installed. After put into my Jetson and started the board it simply died.

I attached the serial console to figure out what happened, and I got the console messages, seems the bootloaders are ok, the board seems to suffer resetting after loading the kernel. that is it when Uboot drops the classical message: “starting the kernel…” after that the board goes back to TegraBoot stage.

Below is a print of uboot’s serial console in exact moment the board reboots:

Found /boot/extlinux/extlinux.conf
Retrieving file: /boot/extlinux/extlinux.conf
239 bytes read in 127 ms (1000 Bytes/s)
p3450-porg eMMC boot options
1:	primary kernel
Enter choice: 1:	primary kernel
Retrieving file: /boot/initrd
0 bytes read in 99 ms (0 Bytes/s)
Retrieving file: /boot/Image
34336776 bytes read in 1570 ms (20.9 MiB/s)
append: tegraid=21.1.2.0.0 ddr_die=4096M@2048M section=512M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 console=ttyS0,115200n8 debug_uartport=lsport,0 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=0x1000@0xff780000 core_edp_mv=1075 core_edp_ma=4000  root=/dev/mmcblk0p1 rw rootwait console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0    root=/dev/mmcblk0p1 rw rootwait console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4 root=/dev/mmcblk0p1 rw rootwait
## Flattened Device Tree blob at 83100000
   Booting using the fdt blob at 0x83100000
   reserving fdt memory region: addr=80000000 size=20000
   Using Device Tree in place at 0000000083100000, end 0000000083173e9a

Starting kernel ...        <b>*****Here where the board reboots</b>

[0000.158] [TegraBoot] (version 00.00.2018.01-l4t-33e7fa82) <b>*****TegraBoot again</b>

I hope this extra information helps during your investigation, I’m planning to run some very complex control algorithms on that board and the RealTime Linux is very important to me.

Thanks

Felipe

you sure u-boot isnt over running memory with an image to large to load ? is there a fast flash message on the console just before the reboot ?

Hate to break it to you, but the preempt patch is coming to main-line Linux - and you should be cheering that. It’s not a “specialized kernel” - it’s a patch that takes the bad coding practices out of the existing kernel which otherwise makes determinism impossible (like grabbing a spin-lock then having the thread go to sleep, or not inheriting the priority of a thread you interrupted causing the CPU to stall and crash).

If you are writing code today that does not coexist with a fully preemptible kernel in Linux - you are just delaying your refactoring pain and spreading it to other programmers.

Moderator: I don’t see this issue as closed, failing to boot with the preempt_rt patch applied indicates that Nvidia has some important work to do.

Linking this to the thread on LinuxCNC usage (here).

Just for the record, today I tried using the source of the newest L4T 32.2. The results are the same, I am able to compile the kernel and to create an image for the Jetson Nano. However, I am not able to boot using that image.

This is the method I followed using L4T 32.2 sources. My host machine has Ubuntu 18.04.2 and I worked on my home folder:

# Install required packages
sudo apt-get update
sudo apt-get install libncurses5-dev build-essential bc lbzip2

# Work folder
mkdir jetson_nano
cd jetson_nano

# Download L4T Jetson Driver Package:
https://developer.nvidia.com/embedded/dlc/Jetson-210_Linux_R32.2.0-0

# Download L4T Sample Root File System:
https://developer.nvidia.com/embedded/dlc/Tegra_Linux_Sample-Root-Filesystem_Nano

# Download L4T Sources:
https://developer.nvidia.com/embedded/dlc/public_sources_Nano

# Download GCC Tool Chain for 64-bit BSP:
https://developer.nvidia.com/embedded/dlc/l4t-gcc-7-3-1-toolchain-64-bit

# Prepare files
sudo tar xpf Jetson-210_Linux_R32.2.0-0.tbz2
cd Linux_for_Tegra/rootfs/
sudo tar xpf ../../Tegra_Linux_Sample-Root-Filesystem_Nano.tbz2
cd ../../

tar -xvf l4t-gcc-7-3-1-toolchain-64-bit.tar.xz
tar -xjf public_sources_Nano.tbz2
tar -xjf public_sources/kernel_src.tbz2

# Apply PREEMPT-RT patches
cd kernel/kernel-4.9/
./scripts/rt-patch.sh apply-patches

# Build kernel
TEGRA_KERNEL_OUT=jetson_nano_kernel
mkdir $TEGRA_KERNEL_OUT
export CROSS_COMPILE=$HOME/jetson_nano/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-
make ARCH=arm64 O=$TEGRA_KERNEL_OUT tegra_defconfig
make ARCH=arm64 O=$TEGRA_KERNEL_OUT -j4

# Prepare files for creating image
sudo cp jetson_nano_kernel/arch/arm64/boot/Image $HOME/jetson_nano/Linux_for_Tegra/kernel/Image
sudo cp -r jetson_nano_kernel/arch/arm64/boot/dts/* $HOME/jetson_nano/Linux_for_Tegra/kernel/dtb/
sudo make ARCH=arm64 O=$TEGRA_KERNEL_OUT modules_install INSTALL_MOD_PATH=$HOME/jetson_nano/Linux_for_Tegra/rootfs/

cd $HOME/jetson_nano/Linux_for_Tegra/rootfs/
sudo tar --owner root --group root -cjf kernel_supplements.tbz2 lib/modules
sudo mv kernel_supplements.tbz2  ../kernel/

# Create Jetson Nano image
cd ..
sudo ./apply_binaries.sh
sudo ./create-jetson-nano-sd-card-image.sh -o jetson_nano.img -s 12G -r 100

After that, I flashed the resulting image into my micro SD card using Etcher. However, when I tried it in my Jetson Nano it only shows the Nvidia logo, then black screen then Nvidia logo again and so on, in a bootloop.

Hello ajcalderont,

As I mentioned in my other comment, we never were able to boot PREEMPT_RT kernel on NANO and TX1. Since I never saw anything on the serial console, I could never figure out the reason behind that.

Thank you both for your updates, that was the understanding I read from the final posts to this thread - I too have tried with both a TX1 and Nano without success. Reading into other users of PREEMPT_RT on Linux with NVidia graphics drivers, it seems that there may be some issues with the Nvidia specific code using poor coding practices in Linux, causing crashes and hangs when the functions PREEMPT_RT replaces with a different method - breaks the way they’ve written their code.

And in response to this single point, Nvidia is selling a robot platform for use specifically with this device - so that fact the a basic Linux package for clean motion control and deterministic-timing code-flow used widely in safety-critical Linux systems is not well supported, is rather unnerving. So is Nvidia’s apparent lack of concern about this. “Concern” here would be a response like - “Oh, we’ve studied Linux and robotics and found that the defacto standard for ensuring that code in a robotics or safety critical system in Linux is not working correctly with our hardware that we’re selling-into/advertising-as a robotics sales channel.”

Note that the last response from Nvidia on this thread was 3 months ago - and this should have been triaged by now. I understand that it can be a lot of work to go through one’s code to make it fully-preemptable, but it’s happening in Mainline very soon, so they will have to come to peace with that… As it looks, they are setting themselves up to be caught with their pants down when that module officially rolls into the base kernel later this year.

Thank you for starting this thread. I’m having the same issue with a TX1 devkit and the PREEMPT_RT kernel.
I can see the message “Starting kernel …” in the serial console, then the messages stop for a second or two, and finally the board reboots and TegraBoot starts again.

This happened both with “Fully Preemptible Kernel (RT)” and “Preemptible Kernel (Basic RT)” settings. I tried L4T 28.2, 28.3 and 32.2 kernels - all with the same result. I also tried disabling and enabling some debugging/tracing features related to locks, timers, scheduling and preemption. Unfortunately, not even a hint of why the kernel won’t boot appeared.

The Jetson boards initially seem like a great choice for easy embedded data processing. We were hoping to use these boards in our project, but we really need to get the OS latencies down into tens of microseconds range. This seems feasible only with an RT kernel.

From what I understand of how the PREEMPT_RT patch works, they replace some functions with “better” functions - but it’s up to the individual SW engineers to not use bad coding practices with what is left. If the drivers rely on bad coding practices like grabbing a spin-lock and then letting that thread go to sleep, then the interrupting thread being blocked by the spinlock, they will hang the system. I’m guessing that’s what’s happening here, as soon as the driver starts loading something gets interrupted while it’s holding onto a lock and - boom, system hangs. Just a WAG, but I get the feeling some core Tegra code is probably years overdue for a revisit/refactor, as they started the Tegra Development back in the 2.x kernel days and they’ve probably done the bare minimum patches to get L4T running at all.

Another shameless reminder to Nvidia, you have to deal with this as PREEMPT_RT is coming to the base kernel, and soon. Burying your head in the sand is just going to prolong the pain.

Given the absolute lack of response from Nvidia since April, I’m probably going to move to an Intel or AMD platform which does support PREEMPT_RT - projects/development cycles can’t wait 4+ months with no forecast on how much time remains.

Very disappointing support IMHO. Are you seeing this Jensen, or anyone of importance at Nvidia?

Saw this on another thread. (Posted 08/12/2019 01:37 AM)

https://devtalk.nvidia.com/default/topic/1059380/jetson-nano/which-rt-linux-patch-installed-for-jetson-nano-/post/5372194/

I got about the same information through the grapevine on Thursday evening. “End of year” is still kinda non-committal and vague. End of which year? Calendar year, fiscal year, Neptune year? Assuming “calendar year 2019” that’ll mean I will have waited about a year to actually start doing the work towards finding out if I can use a Jetson Nano for its purchased purpose. And at best, that’s still just an assumption (though reasonable-ish).

Would have been better if the moderator/nVidia had put an actual calendar date on it - intellectual honesty is the best way to get buy-in from customers/developers.

[duplicate]

It’s unfortunate that NVIDIA is making us wait this long just to test if their product is viable for our use case. It’s even worse that they had to have known about the PREEMPT-RT issue with Jetson Nano/TX1, but they didn’t disclose this information.

It was said that the PREMPT-RT patch will be supported by the end of the year.
So while waiting for that Nvidia release, I am trying to make basic improvements to my application.
Not applying the full patch, just passing “threadirqs” as a kernel parameter fails. (but isolcpus works)
as anybody tried “threadirqs”?

Any updates on this?