Jetson AGX HDMI Display is not working after applying RT Patch

prasanna · January 21, 2021, 11:07am

Hi,
I am using Jetson AGX Xavier DVK kit.
Jetpack : JetPack_4.4_Linux_JETSON_AGX_XAVIER
kernel source : kernel 4.9

Steps I followed :

Flashed Jetson using SDK manager, after flashing is complete created a user using desktop. and able to use the system.
Build the kernel using kernel sources followed the steps from below link :
Welcome — Jetson Linux<br/>Developer Guide 34.1 documentation
after successful image built. copied the output Image to Tegra_For_Linux
Replace <release_packagep>/Linux_for_Tegra/kernel/Image with a copy of:
$TEGRA_KERNEL_OUT/arch/arm64/boot/Image.
Used sudo ./flash.sh jetson-xavier mmcblk0p1 to flash the jetson.
after flashing is complete system booted normally and display comes up.
Applied RT patch using ./scripts/rt-patch.sh apply-patches
while building the kernel selected choice 5 for
Preemption Model

No Forced Preemption (Server) (PREEMPT_NONE)

Voluntary Kernel Preemption (Desktop) (PREEMPT_VOLUNTARY)
Preemptible Kernel (Low-Latency Desktop) (PREEMPT__LL) (NEW)
Preemptible Kernel (Basic RT) (PREEMPT_RTB) (NEW)
Fully Preemptible Kernel (RT) (PREEMPT_RT_FULL) (NEW)
choice[1-5]: 5
Once the kernel build is complete. copied the Image to Linux_For Tegra same as above:
system comes up with display user account is created using display then system reboots during this time. Display will be OFF :
On HDMI display console log is available. no dispalay output.
By using the UART terminal log I am able to login :
root@jetson:~# uname -a
Linux jetson 4.9.140-rt93-tegra #3 SMP PREEMPT RT Thu Jan 21 15:21:26 IST 2021 aarch64 aarch64 aarch64 GNU/Linux
I have attached the console logs. Please let me know If missed anything.
With_RT (26.6 KB)
as well attached the display output image.

DC2A26AE-3849-42D4-B477-A195408C12792048×1536 514 KB

Thanks,
Prasanna

WayneWWW · January 22, 2021, 2:38am

Hi,

I guess the correct term should be “GUI desktop is not up”. If you can see the console words on the monitor, then HDMI is working fine.

Could you try to restart gdm3 service and see if it can up or not?

If it cannot, could you share your syslog and point out the timestamp that you trigger gdm3 service?

prasanna · January 23, 2021, 1:16pm

Hi Wayne,

Yes. “GUI desktop is not up”.

I have attached a log. Which includes. gdm3 restart, service status after restart. and the syslog data from gdm3 service restart. Help us to resolve the issue.

Thanks,
Prasanngdmservice.txt (1.4 MB)

WayneWWW · January 23, 2021, 1:35pm

Hi,

Could you share the result of “lsmod” and dmesg?

prasanna · January 23, 2021, 1:44pm

Hi,

Thanks for quick response. Please find attached log for lsmod and dmesg.

lsmod_dmesg (215.3 KB)

Thanks,
Prasanna

WayneWWW · January 23, 2021, 1:46pm

root@jetson:~# lsmod
Module Size Used by

lsmod is empty so gpu drvier is gone.

This issue is due to the uname -r does not match the module folder name (/lib/modules).

When building the kernel, export LOCALVERSION=-tegra to make the uname -r matches the module folder.

prasanna · January 23, 2021, 1:49pm

Thanks, I will rebuild kernel and modules, Let you know the outcome.

Best Regards,
Prasanna

prasanna · January 24, 2021, 7:32am

Hi Wayne,

Below is the version of working and Non-working builds.
Working : Downloaded using SDK Manager. Linux_For_tegra
jetson@jetson:~$ uname -a
Linux jetson 4.9.140-tegra #4 SMP PREEMPT Fri Jan 22 18:32:06 IST 2021 aarch64 aarx

jetson@jetson:~$ uname -r
4.9.140-tegra

Non-working : Kernel Built using steps followed in below link:
https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/kernel_custom.html#wwpID0E0WD0HA

Image file replaced Step#5 GUI is not up
root@jetson:~# uname -a
Linux jetson 4.9.140-rt93-tegra #6 SMP PREEMPT RT Sun Jan 24 09:14:59 IST 2021 aarx
root@jetson:~#

root@jetson:~# uname -r
4.9.140-rt93-tegra

If I Install kernel modules also by using step#7 GUI is not up with Below Error in terminal.
NOTE : Verified that lib/modules contains “4.9.140-rt93-tegra”

[ 16.138331] Please complete [ 23.477197] nvgpu: 17000000.gv11b [ 23.477455] nvgpu: 17000000.gv11b [ 23.477646] nvgpu: 17000000.gv11b [ 23.477794] nvgpu: 17000000.gv11b [ 23.477953] nvgpu: 17000000.gv11b [ 23.478110] nvgpu: 17000000.gv11b [ 23.478268] nvgpu: 17000000.gv11b [ 23.478646] nvgpu: 17000000.gv11b [ 23.479375] nvgpu: 17000000.gv11b [ 23.480096] nvgpu: 17000000.gv11b [ 23.485464] nvgpu: 17000000.gv11b [ 23.494655] nvgpu: 17000000.gv11b [ 23.504362] nvgpu: 17000000.gv11b [ 23.513728] nvgpu: 17000000.gv11b [ 23.523616] nvgpu: 17000000.gv11b [ 23.534111] nvgpu: 17000000.gv11b [ 23.544370] nvgpu: 17000000.gv11b [ 23.554350] nvgpu: 17000000.gv11b [ 23.564562] nvgpu: 17000000.gv11b [ 23.574808] nvgpu: 17000000.gv11b [ 23.585211] nvgpu: 17000000.gv11b [ 23.595188] nvgpu: 17000000.gv11b [ 23.605687] nvgpu: 17000000.gv11b [ 23.615920] nvgpu: 17000000.gv11b [ 23.626157] nvgpu: 17000000.gv11b [ 23.636398] nvgpu: 17000000.gv11b [ 23.647157] nvgpu: 17000000.gv11b [ 23.657400] nvgpu: 17000000.gv11b [ 23.668422] nvgpu: 17000000.gv11b [ 23.678921] nvgpu: 17000000.gv11b [ 23.689682] nvgpu: 17000000.gv11b [ 23.698873] nvgpu: 17000000.gv11b [ 23.708407] nvgpu: 17000000.gv11b [ 23.717424] nvgpu: 17000000.gv11b [ 23.726962] nvgpu: 17000000.gv11b [ 23.736846] nvgpu: 17000000.gv11b [ 23.746907] nvgpu: 17000000.gv11b [ 23.756189] nvgpu: 17000000.gv11b [ 23.766071] nvgpu: 17000000.gv11b [ 23.775956] nvgpu: 17000000.gv11b [ 23.785585] nvgpu: 17000000.gv11b [ 23.795732] nvgpu: 17000000.gv11b [ 23.805619] nvgpu: 17000000.gv11b [ 23.815161] nvgpu: 17000000.gv11b [ 23.825307] nvgpu: 17000000.gv11b [ 23.835283] nvgpu: 17000000.gv11b [ 243.119321] INFO: task Xorg:5951 [ 243.119469] Not tainted [ 243.119549] “echo 0 > [ 243.122909] Kernel panic [ 243.122923] CPU: 1 PID: [ 243.122924] Hardware name: [ 243.122925] Call trace:
[ 243.122929] [] dump_backtrace+0x0 [ 243.122932] [] show_stack+0x24/0x [ 243.122938] [] dump_stack+0x98/0x [ 243.122943] [] panic+0x11c/0x29c
[ 243.122948] [] watchdog+0x314/0x3 [ 243.122952] [] kthread+0xec/0xf0
[ 243.122955] [] ret_from_fork+0x10 [ 243.122961] SMP: stopping [ 243.122976] Kernel Offset: [ 243.122978] Memory Limit: none
[ 243.122980] trusty-log panic ��Shutdown state requested 1
Rebooting system … system configuration setup on desktop to proceed…
__nvgpu_timeout_expired_msg_cpu:94 [ERR]
gr_gk20a_ctx_wait_ucode:528 [ERR] 0
gk20a_fecs_dump_falcon_stats:129 [ERR] 0
gk20a_fecs_dump_falcon_stats:131 [ERR] 0
gk20a_fecs_dump_falcon_stats:133 [ERR] 0
gk20a_fecs_dump_falcon_stats:135 [ERR] 0
gk20a_fecs_dump_falcon_stats:137 [ERR] 0
gk20a_fecs_dump_falcon_stats:139 [ERR] 0
gk20a_fecs_dump_falcon_stats:141 [ERR] 4
gk20a_fecs_dump_falcon_stats:143 [ERR] 5
gk20a_fecs_dump_falcon_stats:145 [ERR] 0
gk20a_fecs_dump_falcon_stats:147 [ERR] 0
gk20a_fecs_dump_falcon_stats:149 [ERR] 0
gk20a_fecs_dump_falcon_stats:151 [ERR] 0
gk20a_fecs_dump_falcon_stats:155 [ERR] 0
gk20a_fecs_dump_falcon_stats:155 [ERR] 0
gk20a_fecs_dump_falcon_stats:155 [ERR] 1
gk20a_fecs_dump_falcon_stats:155 [ERR] 0
gk20a_fecs_dump_falcon_stats:155 [ERR] 0
gk20a_fecs_dump_falcon_stats:155 [ERR] 0
gk20a_fecs_dump_falcon_stats:155 [ERR] 0
gk20a_fecs_dump_falcon_stats:155 [ERR] 0
gk20a_fecs_dump_falcon_stats:155 [ERR] 0
gk20a_fecs_dump_falcon_stats:155 [ERR] 0
gk20a_fecs_dump_falcon_stats:155 [ERR] 0
gk20a_fecs_dump_falcon_stats:155 [ERR] 0
gk20a_fecs_dump_falcon_stats:155 [ERR] 0
gk20a_fecs_dump_falcon_stats:155 [ERR] f
gk20a_fecs_dump_falcon_stats:155 [ERR] 0
gk20a_fecs_dump_falcon_stats:155 [ERR] 0
gk20a_fecs_dump_falcon_stats:159 [ERR] 0
gk20a_fecs_dump_falcon_stats:161 [ERR] 0
gk20a_fecs_dump_falcon_stats:163 [ERR] 0
gk20a_fecs_dump_falcon_stats:169 [ERR] 0
gk20a_fecs_dump_falcon_stats:175 [ERR] 0
gk20a_fecs_dump_falcon_stats:181 [ERR] 4
gk20a_fecs_dump_falcon_stats:187 [ERR] 0
gk20a_fecs_dump_falcon_stats:193 [ERR] 0
gk20a_fecs_dump_falcon_stats:200 [ERR] e
gk20a_fecs_dump_falcon_stats:206 [ERR] 0
gk20a_fecs_dump_falcon_stats:200 [ERR] e
gk20a_fecs_dump_falcon_stats:206 [ERR] 0
gk20a_fecs_dump_falcon_stats:200 [ERR] e
gk20a_fecs_dump_falcon_stats:206 [ERR] 0
gk20a_fecs_dump_falcon_stats:200 [ERR] e
gk20a_fecs_dump_falcon_stats:206 [ERR] 0
blocked for more than 120 seconds.
4.9.140-rt93-tegra #6
/proc/sys/kernel/hung_task_timeout_secs” disables this me.
- not syncing: hung_task: blocked tasks
669 Comm: khungtaskd Not tainted 4.9.140-rt93-tegra #6
Jetson-AGX (DT)
/0x198
30
bc
d0
/0x30
secondary CPUs
disabled
notifier - trusty version Built: 21:16:17 Jun 25 2.

I am not getting what steps I am missing.
For reference : Kernel source tag used : tegra-l4t-r32.4.4

Thanks,
Prasanna

WayneWWW · January 24, 2021, 7:41am

Hi,

So you rebuild the whole kernel modules or you just copy the original one to a new folder?

prasanna · January 24, 2021, 7:53am

Hi.
I rebuild the kernel and modules Step #1 to #7 of he below link.
https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/kernel_custom.html#wwpID0E0WD0HA

In Step#3 I used ./scripts/rt-patch.sh apply-patches for applying RT Patch.

Thanks,
Prasanna

WayneWWW · January 24, 2021, 8:00am

ok. That’s sound like an issue to me. I will try to reproduce this error my side.

If you have time, could you try other options in rt-patch.sh and see if you can reproduce same issue?

prasanna · January 24, 2021, 8:16am

Sure. But rt-patch.sh has only two options.:
apply_rt_patches and revert_rt_patches

I can Obtain the Kernel Sources with Git and follow the whole procedure. again. will that help ?

Before that I will revert RT and replace kernel, modules and dt will check once. whether it works as normal without RT.

WayneWWW · January 24, 2021, 8:18am

I meant these options.

prasanna · January 24, 2021, 8:19am

got it. I will try

carolyuu · January 25, 2021, 5:48am

Hi prasanna,

We can’t reproduce your issue with r32.4.4 on Xavier.
List our steps for you reference:

Run the RT-patch:

$ ./scripts/rt-patch.sh apply-patches
PREEMPT RT patches successfully applied for Auto!
PREEMPT RT patches successfully applied for L4T!

Building the kernel
Copy $TEGRA_KERNEL_OUT/arch/arm64/boot/Image to JetPack-4.4.1/Linux_for_Tegra/kernel/Image
Install kernel modules:

$ sudo make ARCH=arm64 O=$TEGRA_KERNEL_OUT modules_install INSTALL_MOD_PATH=/JetPack-4.4.1/Linux_for_Tegra/rootfs/

Flash image
After boot to Desktop and check:
$ uname -a
Linux tegra-ubuntu 4.9.140-rt93-tegra #1 SMP PREEMPT RT Mon Jan 25 13:05:18 CST 2021 aarch64 aarch64 aarch64 GNU/Linux
Also check dmesg, there are no nvgpu errors.

Please try to use clean kernel sources and build again.

prasanna · January 25, 2021, 6:14am

Hi,
Thanks I will follow the instructions and I will update.
I suspect the kernel and Jetpack version.
Jetson SDK downloaded version for Linux_for_Tegra is Jetson_Linux_R32.4.3_aarch64. and kernel sources is tegra-l4t-r32.4.4.

I will keep both same versions and check.

Thanks,
Prasanna

prasanna · January 26, 2021, 11:18am

Hi,
I downloaded Jetpack and as well kernel sources a fresh.
Now I referred the link JetPack Archive | NVIDIA Developer
to keep Jetpack ad kernel sources of same versions:
My current Jetpack version is : JetPack_4.4.1_Linux_JETSON_AGX_XAVIER

Kernel sources I obtained using ./source_sync.sh
and the tag is tegra-l4t-r32.4.4

gcc version : gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu.

Host Build PC : Linux 5.4.0-64-generic #72~18.04.1-Ubuntu

I followed multiple exercises :

To check whether the kernel build and the steps followed are working properly with my build environment. I tried build the kernel and modules without RT and flashed Jetson. - Works properly. GUI is up. Confirmed that the kernel is built by using sources : root@jetson:~# uname -a
Linux jetson 4.9.140-tegra #1 SMP PREEMPT Mon Jan 25 23:50:09 IST 2021 aarch64.
Applied RT patches as mentioned by you but kernel modules are not installed :
a. make ARCH=arm64 O=$TEGRA_KERNEL_OUT tegra_defconfig
b. Apply RT Patch
/sources/kernel/kernel-4.9# ./scripts/rt-patch.sh apply-patches
PREEMPT RT patches successfully applied for Auto!
PREEMPT RT patches successfully applied for L4T!

c. make ARCH=arm64 O=$TEGRA_KERNEL_OUT -j24
Selected option 5 : Fully Preemptible Kernel (RT) (PREEMPT_RT_FULL) (NEW)
Missed Timer Offsets Histogram (MISSED_TIMER_OFFSETS_HIST) [N/y/?] (NEW) - n

d. copied only kernel image
Copy $TEGRA_KERNEL_OUT/arch/arm64/boot/Image to JetPack-4.4.1/Linux_for_Tegra/kernel/Image

This time system booted properly GUI is not up, since modules are not loaded:
root@jetson:~# uname -a
Linux jetson 4.9.140-rt93-tegra #2 SMP PREEMPT RT Tue Jan 26 00:46:26 IST 2021 aarch64 aarch64 aarch64 GNU/Linux
root@jetson:~# lsmod
Module Size Used by

Then, In the same build I installed the modules and flashed Jetson. Now system is booted, GUI is not up. and I received following Error : task systemd:1 blocked for more than 120 seconds.
Please refer the attachments :
i. With_RT_No_modules With_RT_No_modules (193.0 KB)
ii. With_RT_Modules : With_RT_Modules (78.9 KB)
after sometime system got rebooted with below error Refer :
iii.Sys_RebootSys_reboot (1.3 KB)

3. While applying RT patch instead of option 5. I tried other options 1 - 4. not able to build the kernel.
…/scripts/Makefile.build:649: recipe for target ‘drivers/net’ failed
make[2]: *** [drivers/net] Error 2
/home/Build/sources/kernel/kernel-4.9/Makefile:1104: recipe for target ‘drivers’ failed
make[1]: *** [drivers] Error 2
make[1]: Leaving directory ‘/home/Build/sources/kernel/kernel-4.9/tegra_image’
Makefile:171: recipe for target ‘sub-make’ failed
make: *** [sub-make] Error 2

I am not getting. What is missing.

Thanks for help

WayneWWW · January 26, 2021, 11:22am

I am not sure why you have the option 1~5 to choose but we don’t see it.
Will check it again.

WayneWWW · January 26, 2021, 11:29am

Do you always apply rt-patch after that “make … tegradefconfig” is done?

prasanna · January 26, 2021, 11:32am

No not always.
one time only.
defconfig.
apply patc
build kernel

Topic		Replies	Views
Preempt-RT patch on Jetson kernel Jetson AGX Xavier preempt_rt	35	8319	August 23, 2023
PREEMPT-RT patches for Jetson Nano Jetson Nano	38	12643	October 14, 2021
Jetson AGX Compiling Kernel Jetson AGX Xavier compile	36	1563	December 6, 2021
Can anyone help me to find file and folder while build kernel? Jetson AGX Xavier kernel	8	3019	October 18, 2021
building TX1 kernel from source Jetson TX1	30	18661	June 8, 2016
Jetson TK1 on Percepto : Failed flashing ardbeg (download command failed NvError 0x120002) Jetson TK1	22	4201	April 25, 2017
[HowTo] Build own kernel for Jetson TK1 Jetson TK1	92	56801	March 3, 2019
Can not Build Kernel Jetson TX1	8	1543	March 7, 2019
Reflash of Jetson AGX to upgrade ubuntu version, unable to "install SDK componenets on your ... Module" Jetson AGX Xavier sdkm , reflash	28	2063	May 14, 2023
NEED HELP: Updating the L4T from 21.4 to 21.5 on the eMMC fails Jetson TK1	88	6437	October 18, 2021

Jetson AGX HDMI Display is not working after applying RT Patch

Related topics