Preempt-RT patch on Jetson kernel

Has anyone tried to apply the Preempt-RT kernel patches to the Jetson kernel? On the Xavier the kernel is 4.9.108 and the patches I find are 4.9.115. This leads me to think its probably not going to work properly.

You can diff the basic kernel between .108 to .115 and see what changed, to see whether anything that the preempt-rt patches want to change was affected in between.

Turns out the RT-Patches come with the kernel source. Now if I can only remember how to use patch for multiple patch sets. There are quite a few of them. Found the kernel source for RT85 and built that but had some issues. So looks like you need to actually patch the source to get it to work properly.

You might try “kdiff3” (KDE…“kdiff3-qt”) or “meld”.

I think the option which people often need to understand is the “-p#”. This is basically how many of the parent leading directory levels need to be ignored. A patch might name “a/something/to/diff” and “b/something/to/diff”. If you don’t have “a/” and “b/”, then you would want to ignore that with “-p1”. If the patch already fits your directory structure, then you might use “-p0”.

Also, redirects are typical instead of naming all things as an argument. Example:

patch -p1 < ../some.patch

Hi danpollock,

There’s an NVIDIA script to apply all the patches under kernel-4.9/rt-patches.

cd kernel-4.9
./scripts/rt-patch.sh apply-patches

To revert back

cd kernel-4.9
./scripts/rt-patch.sh revert-patches

The corresponding RT patch which is also provided by NVIDIA is 4.9.98-rt76. If you use the RT patch from kernel.org then it will need some more tweaking to adjust to NVIDIA code changes, plus grab the NVIDIA specific (search nvidia.com) patches from kernel-4.9/rt-patches to get everything working.

I was using the script, which at first seemed to work fine but turns out a couple of things are not working as expected.

First of all, the script is modifying tegra_gnu_linux_defconfig but actually, as seen in the tutorials, the recommended for Xavier is tegra_defconfig. Using tegra_gnu_linux_defconfig the Jetson Xavier is booting but the peripherals (keyboard and mouse) were not working. There are quite a lot of differences between both configs, so I can’t say what is causing this issue.

Using tegra_defconfig and changing it manually (directly on the .config generated file or “make menuconfig”) turns out to work well. After booting up:

$ uname -a
Linux jetson-xxxxxxxxxxx 4.9.108-rt76 #1 SMP PREEMPT RT Wed Oct 31 00:54:19 CET 2018 aarch64 aarch64 aarch64 GNU/Linux

But when I run some rt-tests (https://wiki.linuxfoundation.org/realtime/documentation/howto/tools/rt-tests) then it doesn’t behave as I would expect.

$ sudo ./cyclictest --mlockall --smp --priority=80 --interval=200 --distance=0
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 7.42 5.77 2.95 9/763 9490           

T: 0 ( 9010) P:80 I:200 C:1674232 Min:      9 Act:   34 Avg:   34 Max:    1254
T: 1 ( 9011) P:80 I:200 C:1674146 Min:     10 Act:   32 Avg:   33 Max:    1171
T: 2 ( 9012) P:80 I:200 C:1673768 Min:     10 Act:   23 Avg:   34 Max:    1283
T: 3 ( 9013) P:80 I:200 C:1673971 Min:     11 Act:   26 Avg:   37 Max:    1267

The Max latencies shouldn’t be so high, therefore I guess there is something still missing or wrong configured in the kernel.

Has someone the same results or am I missing something? Any hint?

I have the same problemhttps://devtalk.nvidia.com/default/topic/1045608/jetson-agx-xavier/xavier-preempt-rt-patch/, did you fixed it ?

I mentioned something in the duplicate thread before seeing this thread:
https://devtalk.nvidia.com/default/topic/1045608/jetson-agx-xavier/xavier-preempt-rt-patch/

Seems the problem is still not solved yet.

After trying several days, I successfully apply Preempt-RT patch on Xavier. I refer to two github repos.

  1. https://github.com/kozyilmaz/nvidia-jetson-rt/blob/master/docs/README.03-realtime.md
  2. https://github.com/jtagxhub/jetpack-agx-build
$ mkdir my-kernel && cd my-kernel
$ git clone https://github.com/jtagxhub/jetpack-agx-build.git build
$ . build/envsetup.sh
$ l4tout_setup
$ cd /sources/kernel/kernel/kernel-4.9
$ for i in rt-patches/*.patch; do echo $i; done
$ for i in rt-patches/*.patch; do patch -p1 < $i; done
$ cd ~/my-kernel
$ bspsetup 
$ kdefconfig
$ kmenuconfig # set PREEMPT_RT_FULL=y HZ_1000=y
$ ksavedefconfig
$ flash

Instead of using official rt-patch.sh which use tegra-gnu-linux-defconfig I follow my first reference and use tegra-defconfig. @jorge.navarro I use same command that you mention above with following configuration suggested by Nvidia to maximize my performance.

sudo /usr/sbin/nvpmodel -m 0
sudo ${HOME}/jetson_clocks.sh

Here is my result.

policy: fifo: loadavg: 0.62 0.71 0.51 1/679 8460           

T: 0 ( 8431) P:80 I:200 C:2037270 Min:      4 Act:    8 Avg:    8 Max:     501
T: 1 ( 8432) P:80 I:200 C:2037242 Min:      4 Act:    8 Avg:    8 Max:     514
T: 2 ( 8433) P:80 I:200 C:2037211 Min:      5 Act:    8 Avg:    8 Max:     500
T: 3 ( 8434) P:80 I:200 C:2037218 Min:      4 Act:    9 Avg:    8 Max:     449
T: 4 ( 8435) P:80 I:200 C:2037218 Min:      5 Act:    8 Avg:    8 Max:     472
T: 5 ( 8436) P:80 I:200 C:2037191 Min:      4 Act:    8 Avg:    8 Max:     502
T: 6 ( 8437) P:80 I:200 C:2037192 Min:      3 Act:    8 Avg:    7 Max:     459
T: 7 ( 8438) P:80 I:200 C:2037153 Min:      3 Act:   12 Avg:    7 Max:     532

I think the result is much better real-time than yours. However, I think the result is still not well for hard real-time application. Hope my comment here will help you guys.

Great! Thanks for sharing!

Hi,

I would like to test kernel-rt, as descrived above by posutsai. I want to compare cycletest results. But, I’m not able to get it running on Xavier. I need to find what is wrong in my procedure.

Searching for solutions I found next urls quite interesting:

First problems were overcome thanks to:

sudo apt install bzip2=1.0.6-8.1 libbz2-1.0=1.0.6-8.1
sudo apt-get install libncurses5
sudo apt-get install libncurses5-dev

I replace “tegra_gnu_linux_defconfig” by “tegra_defconfig” in:

  1. ~/my-kernel/sources/kernel/kernel/kernel-4.9/scripts/rt-patch.sh
  2. ~/my-kernel/Xavier/Linux_for_Tegra/rootfs/usr/src/linux-headers-4.9.140-tegra-linux_x86_64/kernel-4.9/scripts
  3. ~/my-kernel/Xavier/Linux_for_Tegra/rootfs/usr/src/linux-headers-4.9.140-tegra-ubuntu18.04_aarch64/kernel-4.9/scripts

Below, my commands in the terminal:

cd sources/kernel/kernel/kernel-4.9
for i in rt-patches/*.patch; do echo $i; done
for i in rt-patches/*.patch; do patch -p1 < $i; done
kdefconfig
kmenuconfig # no changes 
ksavedefconfig
kbuild -a	> Log_E_kbuild.txt
#  ... (some warnings were printed but no errors)
flash > Log_E_flash.txt

Flash command seems good but Xavier is not able to boot. Boot messages, read by serial, are:

$ sudo minicom -D /dev/ttyUSB3 -8 -b 115200
#...                                            
[    4.576774] Console: switching to colour frame buffer device 240x67          
[    4.588873] tegradc 15200000.nvdisplay: fb registered                        
[    4.594150] gpio tegra-gpio wake71 for gpio=98(M:2)                          
[    4.595103] tegra_nvdisp_handle_pd_enable: Unpowergated Head0 pd             
[    4.602487] brd: module loaded                                               
[    4.607809] tegra_nvdisp_handle_pd_enable: Unpowergated Head1 pd             
[    4.614681] loop: module loaded                                              
[    4.617082] tegra_profiler: version: 1.132, samples/io: 46/26                
[    4.623067] tegra_profiler: auth: init                                       
[    4.626971] nct1008_nct72 4-004c: starting parse dt                          
[    4.627438] THERMAL EST: found 3 subdevs                                     
[    4.627443] THERMAL EST num_resources: 0                                     
[    4.627448] [THERMAL EST subdev 0]                                           
[    4.627453] [THERMAL EST subdev 1]                                           
[    4.627458] [THERMAL EST subdev 2]                                           
[    4.628017] thermal thermal_zone7: Registering thermal zone thermal_zone7 fot
[    4.628020] THERMAL EST: thz register success.                               
[    4.628999] Parent Clock set for DC plld3                                    
[    4.629909] THERMAL EST: end of probe, return err: 0                         
[    4.631721] ufs_tegra 2450000.ufshci: freq-table-hz property not specified   
[    4.631736] ufs_tegra 2450000.ufshci: ufshcd_populate_vreg: Unable to find vd
[    4.632915] ufs_tegra 2450000.ufshci: wakeup init done, cdirq 459            
[    4.633073] gpio tegra-gpio-aon wake25 for gpio=32(EE:0)                     
[    4.649032] scsi host0: ufshcd                                               
[    4.653552] hisi_sas: driver version v1.6                                    
[    4.660439] libphy: Fixed MDIO Bus: probed                                   
[    4.661542] tun: Universal TUN/TAP device driver, 1.6                        
[    4.661545] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>           
[    4.663045] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k               
[    4.663047] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.              
[    4.663103] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k  
[    4.663105] igb: Copyright (c) 2007-2014 Intel Corporation.                  
[    4.663182] igbvf: Intel(R) Gigabit Virtual Function Network Driver - versiok
[    4.663184] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.              
[    4.663250] Intel(R) 10GbE PCI Express Linux Network Driver - version 4.6.4  
[    4.663253] Copyright(c) 1999 - 2017 Intel Corporation.                      
[    4.665159] eqos 2490000.ether_qos: no PHY interrupt found                   
[    4.669849] eqos 2490000.ether_qos: Setting local MAC: 0 4 4b cb a1 c6       
[    4.669991] libphy: dwc_phy: probed                                          
[    4.673551] PPP generic driver version 2.4.2                                 
[    4.673870] PPP BSD Compression module registered                            
[    4.673874] PPP Deflate Compression module registered                        
[    4.673890] PPP MPPE Compression module registered                           
[    4.673894] NET: Registered protocol family 24                               
[    4.673988] usbcore: registered new interface driver r8152                   
[    4.674052] usbcore: registered new interface driver asix                    
[    4.674093] usbcore: registered new interface driver ax88179_178a            
[    4.674136] usbcore: registered new interface driver cdc_ether               
[    4.674174] usbcore: registered new interface driver net1080                 
[    4.674215] usbcore: registered new interface driver cdc_subset              
[    4.674257] usbcore: registered new interface driver zaurus                  
[    4.674311] usbcore: registered new interface driver cdc_ncm                 
[    4.674744] VFIO - User Level meta-driver version: 0.3                       
[    4.676782] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver       
[    4.676802] ehci-pci: EHCI PCI platform driver                               
[    4.676860] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver           
[    4.676872] ohci-pci: OHCI PCI platform driver                               
[    4.676938] ohci-platform: OHCI generic platform driver                      
[    4.685958] tegradc 15200000.nvdisplay: hdmi: tmds rate:148351K prod-settingm
[    4.686680] tegra-xusb 3610000.xhci: USB2 port 0 has OTG_CAP                 
[    4.686685] tegra-xusb 3610000.xhci: USB3 port 2 has OTG_CAP                 
[    4.687252] tegradc 15200000.nvdisplay: hdmi: get RGB quant from REG program.
[    4.687266] tegradc 15200000.nvdisplay: hdmi: get YCC quant from REG program.
[    4.689876] extcon-disp-state external-connection:disp-state: cable 48 state1
[    4.689880] Extcon AUX2(HDMI) enable                                         
[    4.690701] Could not get extcon-dev /host1x/nvdisplay@15210000:typec0(0)    
[    4.691186] Could not get extcon-dev /host1x/nvdisplay@15220000:typec1(1)    
[    4.692331] Could not get extcon-dev /xhci@3610000:id(0)                     
[    4.696417] usbcore: registered new interface driver uas                     
[    4.696527] usbcore: registered new interface driver usb-storage             
[    4.696698] usbcore: registered new interface driver usbserial               
[    4.702054] tegra-xudc-new 3550000.xudc: PMQOS CPU boost enabled             
[    4.702117] Could not get extcon-dev /xudc@3550000:vbus(0)                   
[    4.705497] ucsi_ccg 1-0008: FW1 Version: gn v12.05, [Base 3.1.

And Xavier halt there.

¿Does anybody see how to solve it?

Thanks.

Hello Leguim,

Please share the complete serial console output in case that has any info.

BTW, how did you flash the PREEMPT_RT kernel on the Xavier? Did you flash the system wholly with sudo ./flash.sh jetson-xavier mmcblk0p1

On another note, Concurrent Real-Time has release RedHawk Linux for NVIDIA’s Jetson AGX Xavier.

RedHawk Linux is a leading Real-Time Operating System for x86 and arm64 systems. It provides guaranteed response time of less than 50 microseconds on the AGX Xavier.

Hello anish.aney,

I already knew about RedHawk. It looks interesting and I will try to know more.

Yes, I have used ‘sudo ./flash.sh jetson-xavier mmcblk0p1’ for flashing Xavier.

Next posts have atached relevant outputs.

Seial console output is atached.
Log_D_serial.txt (391 KB)

Flashing output is atached.
Log_D_flash.txt (9.23 KB)

Hello leguim,

Thank you for sharing the logs.

Looking at the serial console logs, it looks like the run_posix_cpu_timers() needs to be called with IRQS disabled. However, I am not sure where and which function disables the IRQs. I will need to take a look at that more.

We are able to boot the preempt-rt kernel on our end. I know that we had to go to -rt94 patch just because that was easier to patch, and may be better in terms of bug fixes. Anyway, I will try to reproduce this issue at my end.

And may be someone else can comment on this.

Hello,

Finally it is working:

uname -a
Linux host 4.9.108-rt76+ #2 SMP PREEMPT RT Fri Jul 19 10:25:36 CEST 2019 aarch64 aarch64 aarch64 GNU/Linux

Cyclictest results:

sudo cyclictest --mlockall --smp --priority=80 --interval=200 --distance=0
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.11 0.51 0.30 1/447 6864

T: 0 ( 6856) P:80 I:200 C: 594297 Min:      3 Act:    7 Avg:    7 Max:     330
T: 1 ( 6857) P:80 I:200 C: 594278 Min:      4 Act:    6 Avg:    7 Max:     314
T: 2 ( 6858) P:80 I:200 C: 594265 Min:      3 Act:    5 Avg:    6 Max:     304
T: 3 ( 6859) P:80 I:200 C: 594253 Min:      3 Act:    6 Avg:    6 Max:     316
T: 4 ( 6860) P:80 I:200 C: 594236 Min:      4 Act:    7 Avg:    7 Max:     302
T: 5 ( 6861) P:80 I:200 C: 594225 Min:      4 Act:    7 Avg:    7 Max:     298
T: 6 ( 6862) P:80 I:200 C: 594209 Min:      3 Act:   11 Avg:    7 Max:     307
T: 7 ( 6863) P:80 I:200 C: 594190 Min:      4 Act:    9 Avg:    7 Max:     301

I was not able to run a kernel 4.9.140 with rt-patch applied. But it was quite straight with 4.9.108. Below my steps:

  1. Use Ubuntu (64-bit) installer for JetPack 4.1.1 DP 2018/11/08.
  2. Install and downgrade some libs:
    sudo apt install bzip2=1.0.6-8.1 libbz2-1.0=1.0.6-8.1
    sudo apt-get install libncurses5
    sudo apt-get install libncurses5-dev
    
  3. Follow compiling instructions: https://developer.ridgerun.com/wiki/index.php?title=Xavier/JetPack_4.1/Compiling_Code/Kernel * When prompted use this tag for get kernel 4.9.108: 'tegra-l4t-r31.1' * Modify rt-patch:
    cd ~/JetPack-L4T/Xavier/Linux_for_Tegra/sources/kernel/kernel-4.9/scripts/rt-patch.sh
    grep -rl tegra_gnu_linux_defconfig | xargs sed -i 's/tegra_gnu_linux_defconfig/tegra_defconfig/g'
    
    • Apply rt-patch before compile
    • Set Xavier:
    sudo /usr/sbin/nvpmodel -m 0
    sudo ${HOME}/jetson_clocks.sh
    
  4. Flash Jetson

Thanks !

Good to hear that, Leguim!

I still haven’t had a chance to replicate your issue if it’s not just a patching issue. However, I feel that it could be something related to locking. A spin_[un]lock() variant might have been used where a raw_spin_[un]lock() should have been used.

One way to figure out is to look for all the rt-patches/* patches which touch the file(s) defining update_process_times() and/or run_posix_cpu_timers().

Another way is to download either 4.9.115-rt93 or 4.9.115-rt94 from the RT mainline, apply that patch cleanly and then search for “nvidia.com” in rt-patches/ dir of L4T kernel. This will bring in the “working” PREEMPT_RT patch from RT mainline as well as apply any NVIDIA/Jetson specific code developed by NVIDIA.