Jetson AGX HDMI Display is not working after applying RT Patch

Hi,
I am using Jetson AGX Xavier DVK kit.
Jetpack : JetPack_4.4_Linux_JETSON_AGX_XAVIER
kernel source : kernel 4.9

Steps I followed :

  1. Flashed Jetson using SDK manager, after flashing is complete created a user using desktop. and able to use the system.
  2. Build the kernel using kernel sources followed the steps from below link :
    https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide%2Fkernel_custom.html%23
    after successful image built. copied the output Image to Tegra_For_Linux
    Replace <release_packagep>/Linux_for_Tegra/kernel/Image with a copy of:
    $TEGRA_KERNEL_OUT/arch/arm64/boot/Image.
    Used sudo ./flash.sh jetson-xavier mmcblk0p1 to flash the jetson.
    after flashing is complete system booted normally and display comes up.
  3. Applied RT patch using ./scripts/rt-patch.sh apply-patches
    while building the kernel selected choice 5 for
    Preemption Model
  1. No Forced Preemption (Server) (PREEMPT_NONE)
  1. Voluntary Kernel Preemption (Desktop) (PREEMPT_VOLUNTARY)
  2. Preemptible Kernel (Low-Latency Desktop) (PREEMPT__LL) (NEW)
  3. Preemptible Kernel (Basic RT) (PREEMPT_RTB) (NEW)
  4. Fully Preemptible Kernel (RT) (PREEMPT_RT_FULL) (NEW)
    choice[1-5]: 5
    Once the kernel build is complete. copied the Image to Linux_For Tegra same as above:
    system comes up with display user account is created using display then system reboots during this time. Display will be OFF :
    On HDMI display console log is available. no dispalay output.
    By using the UART terminal log I am able to login :
    root@jetson:~# uname -a
    Linux jetson 4.9.140-rt93-tegra #3 SMP PREEMPT RT Thu Jan 21 15:21:26 IST 2021 aarch64 aarch64 aarch64 GNU/Linux
    I have attached the console logs. Please let me know If missed anything.
    With_RT (26.6 KB)
    as well attached the display output image.

Thanks,
Prasanna

Hi,

I guess the correct term should be “GUI desktop is not up”. If you can see the console words on the monitor, then HDMI is working fine.

Could you try to restart gdm3 service and see if it can up or not?

If it cannot, could you share your syslog and point out the timestamp that you trigger gdm3 service?

Hi Wayne,

Yes. “GUI desktop is not up”.

I have attached a log. Which includes. gdm3 restart, service status after restart. and the syslog data from gdm3 service restart. Help us to resolve the issue.

Thanks,
Prasanngdmservice.txt (1.4 MB)

Hi,

Could you share the result of “lsmod” and dmesg?

Hi,

Thanks for quick response. Please find attached log for lsmod and dmesg.

lsmod_dmesg (215.3 KB)

Thanks,
Prasanna

root@jetson:~# lsmod
Module Size Used by

lsmod is empty so gpu drvier is gone.

This issue is due to the uname -r does not match the module folder name (/lib/modules).

When building the kernel, export LOCALVERSION=-tegra to make the uname -r matches the module folder.

Thanks, I will rebuild kernel and modules, Let you know the outcome.

Best Regards,
Prasanna

Hi Wayne,

Below is the version of working and Non-working builds.
Working : Downloaded using SDK Manager. Linux_For_tegra
jetson@jetson:~$ uname -a
Linux jetson 4.9.140-tegra #4 SMP PREEMPT Fri Jan 22 18:32:06 IST 2021 aarch64 aarx

jetson@jetson:~$ uname -r
4.9.140-tegra

Non-working : Kernel Built using steps followed in below link:
https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/kernel_custom.html#wwpID0E0WD0HA

Image file replaced Step#5 GUI is not up
root@jetson:~# uname -a
Linux jetson 4.9.140-rt93-tegra #6 SMP PREEMPT RT Sun Jan 24 09:14:59 IST 2021 aarx
root@jetson:~#

root@jetson:~# uname -r
4.9.140-rt93-tegra

If I Install kernel modules also by using step#7 GUI is not up with Below Error in terminal.
NOTE : Verified that lib/modules contains “4.9.140-rt93-tegra”

[ 16.138331] Please complete system configuration setup on desktop to proceed…
[ 23.477197] nvgpu: 17000000.gv11b __nvgpu_timeout_expired_msg_cpu:94 [ERR]
[ 23.477455] nvgpu: 17000000.gv11b gr_gk20a_ctx_wait_ucode:528 [ERR] 0
[ 23.477646] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:129 [ERR] 0
[ 23.477794] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:131 [ERR] 0
[ 23.477953] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:133 [ERR] 0
[ 23.478110] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:135 [ERR] 0
[ 23.478268] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:137 [ERR] 0
[ 23.478646] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:139 [ERR] 0
[ 23.479375] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:141 [ERR] 4
[ 23.480096] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:143 [ERR] 5
[ 23.485464] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:145 [ERR] 0
[ 23.494655] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:147 [ERR] 0
[ 23.504362] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:149 [ERR] 0
[ 23.513728] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:151 [ERR] 0
[ 23.523616] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:155 [ERR] 0
[ 23.534111] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:155 [ERR] 0
[ 23.544370] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:155 [ERR] 1
[ 23.554350] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:155 [ERR] 0
[ 23.564562] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:155 [ERR] 0
[ 23.574808] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:155 [ERR] 0
[ 23.585211] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:155 [ERR] 0
[ 23.595188] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:155 [ERR] 0
[ 23.605687] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:155 [ERR] 0
[ 23.615920] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:155 [ERR] 0
[ 23.626157] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:155 [ERR] 0
[ 23.636398] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:155 [ERR] 0
[ 23.647157] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:155 [ERR] 0
[ 23.657400] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:155 [ERR] f
[ 23.668422] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:155 [ERR] 0
[ 23.678921] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:155 [ERR] 0
[ 23.689682] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:159 [ERR] 0
[ 23.698873] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:161 [ERR] 0
[ 23.708407] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:163 [ERR] 0
[ 23.717424] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:169 [ERR] 0
[ 23.726962] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:175 [ERR] 0
[ 23.736846] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:181 [ERR] 4
[ 23.746907] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:187 [ERR] 0
[ 23.756189] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:193 [ERR] 0
[ 23.766071] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:200 [ERR] e
[ 23.775956] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:206 [ERR] 0
[ 23.785585] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:200 [ERR] e
[ 23.795732] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:206 [ERR] 0
[ 23.805619] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:200 [ERR] e
[ 23.815161] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:206 [ERR] 0
[ 23.825307] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:200 [ERR] e
[ 23.835283] nvgpu: 17000000.gv11b gk20a_fecs_dump_falcon_stats:206 [ERR] 0
[ 243.119321] INFO: task Xorg:5951 blocked for more than 120 seconds.
[ 243.119469] Not tainted 4.9.140-rt93-tegra #6
[ 243.119549] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this me.
[ 243.122909] Kernel panic - not syncing: hung_task: blocked tasks
[ 243.122923] CPU: 1 PID: 669 Comm: khungtaskd Not tainted 4.9.140-rt93-tegra #6
[ 243.122924] Hardware name: Jetson-AGX (DT)
[ 243.122925] Call trace:
[ 243.122929] [] dump_backtrace+0x0/0x198
[ 243.122932] [] show_stack+0x24/0x30
[ 243.122938] [] dump_stack+0x98/0xbc
[ 243.122943] [] panic+0x11c/0x29c
[ 243.122948] [] watchdog+0x314/0x3d0
[ 243.122952] [] kthread+0xec/0xf0
[ 243.122955] [] ret_from_fork+0x10/0x30
[ 243.122961] SMP: stopping secondary CPUs
[ 243.122976] Kernel Offset: disabled
[ 243.122978] Memory Limit: none
[ 243.122980] trusty-log panic notifier - trusty version Built: 21:16:17 Jun 25 2.
����Shutdown state requested 1
Rebooting system …

I am not getting what steps I am missing.
For reference : Kernel source tag used : tegra-l4t-r32.4.4

Thanks,
Prasanna

Hi,

So you rebuild the whole kernel modules or you just copy the original one to a new folder?

Hi.
I rebuild the kernel and modules Step #1 to #7 of he below link.
https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/kernel_custom.html#wwpID0E0WD0HA

In Step#3 I used ./scripts/rt-patch.sh apply-patches for applying RT Patch.

Thanks,
Prasanna

ok. That’s sound like an issue to me. I will try to reproduce this error my side.

If you have time, could you try other options in rt-patch.sh and see if you can reproduce same issue?

Sure. But rt-patch.sh has only two options.:
apply_rt_patches and revert_rt_patches

I can Obtain the Kernel Sources with Git and follow the whole procedure. again. will that help ?

Before that I will revert RT and replace kernel, modules and dt will check once. whether it works as normal without RT.

I meant these options.

got it. I will try

Hi prasanna,

We can’t reproduce your issue with r32.4.4 on Xavier.
List our steps for you reference:

  1. Run the RT-patch:
$ ./scripts/rt-patch.sh apply-patches
PREEMPT RT patches successfully applied for Auto!
PREEMPT RT patches successfully applied for L4T!
  1. Building the kernel
  2. Copy $TEGRA_KERNEL_OUT/arch/arm64/boot/Image to JetPack-4.4.1/Linux_for_Tegra/kernel/Image
  3. Install kernel modules:
$ sudo make ARCH=arm64 O=$TEGRA_KERNEL_OUT modules_install INSTALL_MOD_PATH=/JetPack-4.4.1/Linux_for_Tegra/rootfs/
  1. Flash image
  2. After boot to Desktop and check:
    $ uname -a
    Linux tegra-ubuntu 4.9.140-rt93-tegra #1 SMP PREEMPT RT Mon Jan 25 13:05:18 CST 2021 aarch64 aarch64 aarch64 GNU/Linux
  3. Also check dmesg, there are no nvgpu errors.

Please try to use clean kernel sources and build again.

Hi,
Thanks I will follow the instructions and I will update.
I suspect the kernel and Jetpack version.
Jetson SDK downloaded version for Linux_for_Tegra is Jetson_Linux_R32.4.3_aarch64. and kernel sources is tegra-l4t-r32.4.4.

I will keep both same versions and check.

Thanks,
Prasanna

Hi,
I downloaded Jetpack and as well kernel sources a fresh.
Now I referred the link JetPack Archive | NVIDIA Developer
to keep Jetpack ad kernel sources of same versions:
My current Jetpack version is : JetPack_4.4.1_Linux_JETSON_AGX_XAVIER

Kernel sources I obtained using ./source_sync.sh
and the tag is tegra-l4t-r32.4.4

gcc version : gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu.

Host Build PC : Linux 5.4.0-64-generic #72~18.04.1-Ubuntu

I followed multiple exercises :

  1. To check whether the kernel build and the steps followed are working properly with my build environment. I tried build the kernel and modules without RT and flashed Jetson. - Works properly. GUI is up. Confirmed that the kernel is built by using sources : root@jetson:~# uname -a
    Linux jetson 4.9.140-tegra #1 SMP PREEMPT Mon Jan 25 23:50:09 IST 2021 aarch64.

  2. Applied RT patches as mentioned by you but kernel modules are not installed :
    a. make ARCH=arm64 O=$TEGRA_KERNEL_OUT tegra_defconfig
    b. Apply RT Patch
    /sources/kernel/kernel-4.9# ./scripts/rt-patch.sh apply-patches
    PREEMPT RT patches successfully applied for Auto!
    PREEMPT RT patches successfully applied for L4T!

c. make ARCH=arm64 O=$TEGRA_KERNEL_OUT -j24
Selected option 5 : Fully Preemptible Kernel (RT) (PREEMPT_RT_FULL) (NEW)
Missed Timer Offsets Histogram (MISSED_TIMER_OFFSETS_HIST) [N/y/?] (NEW) - n

d. copied only kernel image
Copy $TEGRA_KERNEL_OUT/arch/arm64/boot/Image to JetPack-4.4.1/Linux_for_Tegra/kernel/Image

This time system booted properly GUI is not up, since modules are not loaded:
root@jetson:~# uname -a
Linux jetson 4.9.140-rt93-tegra #2 SMP PREEMPT RT Tue Jan 26 00:46:26 IST 2021 aarch64 aarch64 aarch64 GNU/Linux
root@jetson:~# lsmod
Module Size Used by

Then, In the same build I installed the modules and flashed Jetson. Now system is booted, GUI is not up. and I received following Error : task systemd:1 blocked for more than 120 seconds.
Please refer the attachments :
i. With_RT_No_modules With_RT_No_modules (193.0 KB)
ii. With_RT_Modules : With_RT_Modules (78.9 KB)
after sometime system got rebooted with below error Refer :
iii.Sys_RebootSys_reboot (1.3 KB)


3. While applying RT patch instead of option 5. I tried other options 1 - 4. not able to build the kernel.
…/scripts/Makefile.build:649: recipe for target ‘drivers/net’ failed
make[2]: *** [drivers/net] Error 2
/home/Build/sources/kernel/kernel-4.9/Makefile:1104: recipe for target ‘drivers’ failed
make[1]: *** [drivers] Error 2
make[1]: Leaving directory ‘/home/Build/sources/kernel/kernel-4.9/tegra_image’
Makefile:171: recipe for target ‘sub-make’ failed
make: *** [sub-make] Error 2

I am not getting. What is missing.

Thanks for help

I am not sure why you have the option 1~5 to choose but we don’t see it.
Will check it again.

Do you always apply rt-patch after that “make … tegradefconfig” is done?

No not always.
one time only.
defconfig.
apply patc
build kernel