After flash into jetson orin nx with preempt_rt.Board cannot boot normal

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin NX
,
1

I have the same problems on a jetson Orin nx developer’s kit and my custom board.
I’m currently facing a problem. I can burn the image normally following the preempt_rt patch tutorial at Kernel Customization — NVIDIA Jetson Linux Developer Guide. However, after the burn is completed, the system fails to boot properly. It gets stuck at the following interface for a while and then the screen goes black. Then it re-enters the same interface and repeats the above process continuously. Currently, flash a normal Linux image works fine.

7778f47ab913e36b0b666333f896c620
7778f47ab913e36b0b666333f896c6201706×1279 132 KB

2

Hello @ChrisCatuso,

Hope everything is going great!

Quick couple of questions:

  1. Does it only happen if you add the patches ? Have you tried first just building the source code as they come from NVIDIA? This might give you some insight if it could be an issue with the patch or the procedure being followed.
  2. Are you able to connect with the board through UART? There you can get the boot logs and that could have some details on what is going on that causes the system to be stuck in a loop.
  3. Besides the preempt_rt patches, have you applied any other changes to the kernel?

best regards,
Andrew
Embedded Software Engineer at ProventusNova

4

Hi,
I appreciate the speedy response.

  1. Yes, it happens after i apply "./generic_rt_build.sh “enable” to rebuild kernel ,modules and dtbs. And i tried first to flash with no change to flash the download bsp image to board, it normally boot up.
  2. I choose to use uart. then i think i found the problem. The board have no emmc .but the log below show that:
**Device /dev/mmcblk?p1 does not exist*
*Checking whether device /dev/sd?1 exist*
*Device /dev/sd?1 does not exist*
*Checking whether device /dev/nvme?n1p1 exist*
*Looking for OTA work directory on the device(s): /dev/nvme0n1p1*
*[   15.325930] mount /dev/nvme0n1p1 /mnt*
*[   15.466441] EXT4-fs (nvme0n1p1): 2 orphan inodes deleted*
*[   15.466451] EXT4-fs (nvme0n1p1): recovery complete*
*[   15.468941] EXT4-fs (nvme0n1p1): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.*
*[   15.470930] is_boot_part_for_disk_enc /dev/nvme0n1p1 /mnt*
*[   15.483031] OTA work directory /mnt/ota_work is not found on /dev/nvme0n1p1*
*mount: /mnt: special device /dev/mmcblk0 does not exist.*
*[   15.496373] Finding OTA work dir on internal storage device*
*[   15.497614] mount /dev/mmcblk0 /mnt*
*[   15.500413] Failed to mount /dev/mmcblk0 on the /mnt*
*[   15.501581] Failed to run "mount_ota_work_partition /dev/mmcblk0 /mnt"*
*[   15.502766] OTA work directory is not found on internal and external storage devices*
*bash: cannot set terminal process group (-1): Inappropriate ioctl for device*
*bash: no job control in this shell*
*bash-5.1# **

Thank you again.

5

Hi,
We have validated the steps:

Jetson/L4T/r36.4.x patches - eLinux.org
Enable RT kernel and CONFIG_HZ_1000

Please follow the steps and see whether the device can be booted successfully.

6

Hi,

I appreciate the speedy response.

  1. If the step CONFIG_HZ_1000 is necessary for RT-kernel;
  2. if i want to apply rt-patch in board as the guidence Kernel Customization — NVIDIA Jetson Linux Developer Guide show “Real-Time Kernel Using OTA Update”, why jetson orin nx shows as follow:
Info: Write TegraPlatformCompatSpec with 3767-000-0000--1--jetson-orin-nano-devkit-super-.

Info: The esp is already mounted to /boot/efi.

Starting kernel post-install procedure.

Rootfs AB is not enabled.

ERROR. Procedure for A_kernel update FAILED.

Cannot install package. Exiting...

dpkg: error processing package nvidia-l4t-rt-kernel (--configure):

installed nvidia-l4t-rt-kernel package post-installation script subprocess returned error exit status 1

dpkg: dependency problems prevent configuration of nvidia-l4t-rt-kernel-oot-modules:

nvidia-l4t-rt-kernel-oot-modules depends on nvidia-l4t-rt-kernel (= 5.15.148-rt-tegra-36.4.0-20241014173246); however:

Package nvidia-l4t-rt-kernel is not configured yet.

dpkg: error processing package nvidia-l4t-rt-kernel-oot-modules (--configure):

dependency problems - leaving unconfigured

dpkg: dependency problems prevent configuration of nvidia-l4t-rt-kernel-headers:

nvidia-l4t-rt-kernel-headers depends on nvidia-l4t-rt-kernel (= 5.15.148-rt-tegra-36.4.0-20241014173246); however:

Package nvidia-l4t-rt-kernel is not configured yet.

Cleaning up the temporary directory for updating the initrd..

Errors were encountered while processing:

nvidia-l4t-rt-kernel

nvidia-l4t-rt-kernel-oot-modules

nvidia-l4t-rt-kernel-headers

nvidia-l4t-display-rt-kernel

E: Sub-process /usr/bin/dpkg returned an error code (1)
7

Sorry, no effort with your link to Enable RT kernel and CONFIG_HZ_1000, this time locked in below log.Normal log for no-rt linux can get into.
log_for_1t_disnormal.txt (74.1 KB)
log_for_1t_normal.txt (78.9 KB)

8

okay! it enter the bash after times of restart.but my screen is still locked in the phtot that i support the first time i ask.
Compare with the following normal log:

[    5.509047] nvme 0004:01:00.0: Adding to iommu group 5
[    5.509542] nvme nvme0: pci function 0004:01:00.0
[    5.509789] nvme 0004:01:00.0: enabling device (0000 -> 0002)
[    5.510258] tegra194-pcie 141e0000.pcie: Adding to iommu group 6
[    5.513046] tegra194-pcie 141e0000.pcie: host bridge /bus@0/pcie@141e0000 ranges:
[    5.513068] tegra194-pcie 141e0000.pcie:      MEM 0x3000000000..0x3227ffffff -> 0x3000000000
[    5.513077] tegra194-pcie 141e0000.pcie:      MEM 0x3228000000..0x322fffffff -> 0x0040000000
[    5.513082] tegra194-pcie 141e0000.pcie:       IO 0x003e100000..0x003e1fffff -> 0x003e100000
[    5.513595] tegra194-pcie 141e0000.pcie: iATU unroll: enabled
[    5.513598] tegra194-pcie 141e0000.pcie: Detected iATU regions: 8 outbound, 2 inbound
[    5.547760] nvme nvme0: allocated 64 MiB host memory buffer.
[    5.566560] nvme nvme0: 8/0/0 default/read/poll queues
[    5.572994] nvme nvme0: Ignoring bogus Namespace Identifiers
[    5.577531]  nvme0n1: p1 p2 p3 p4 p5 p6 p7 p8 p9 p10 p11 p12 p13 p14 p15
[    5.851494] EXT4-fs (nvme0n1p1): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[    5.859746] Rootfs mounted over PARTUUID=119c35f1-4b99-44b7-9a67-fd86985ecc71
[    5.868878] Switching from initrd to actual rootfs
[    5.946899] systemd[1]: System time before build time, advancing clock.
[    5.973822] NET: Registered PF_INET6 protocol family

the abnormal log appears that it cannot switch to nvme rootfs normally.:

[   15.174307] pcieport 0007:00:00.0: Adding to iommu group 6
[   15.175687] pcieport 0007:00:00.0: PME: Signaling with IRQ 194
[   15.177472] pcieport 0007:00:00.0: AER: enabled with IRQ 194
[   15.177780] pci_bus 0007:01: busn_res: [bus 01-ff] is released
[   15.177869] pci 0007:00:00.0: Removing from iommu group 6
[   15.177878] pci_bus 0007:00: busn_res: [bus 00-ff] is released
insmod /lib/modules/5.15.148-rt-tegra/kernel/drivers/nvme/host/nvme-core.ko 
insmod /lib/modules/5.15.148-rt-tegra/kernel/drivers/nvme/host/nvme.ko 
[   15.195936] nvme 0004:01:00.0: Adding to iommu group 5
[   15.196330] nvme nvme0: pci function 0004:01:00.0
[   15.196554] nvme 0004:01:00.0: enabling device (0000 -> 0002)
[   15.205178] nvme nvme0: allocated 16 MiB host memory buffer.
[   15.216391] nvme nvme0: 8/0/0 default/read/poll queues
[   15.221242]  nvme0n1: p1 p2 p3 p4 p5 p6 p7 p8 p9 p10 p11 p12 p13 p14 p15
insmod /lib/modules/5.15.148-rt-tegra/kernel/drivers/thermal/tegra/tegra-bpmp-thermal.ko 
insmod /lib/modules/5.15.148-rt-tegra/kernel/drivers/pwm/pwm-tegra.ko 
insmod /lib/modules/5.15.148-rt-tegra/kernel/drivers/hwmon/pwm-fan.ko 
Checking whether device /dev/mmcblk?p1 exist
Device /dev/mmcblk?p1 does not exist
Checking whether device /dev/s[   15.25d?17804]  existFindin
g OTA work dir on external storage devices
Device /dev/sd?1 does not exist
Checking whether device /dev/nvme?n1p1 exist
Looking for OTA work directory on the device(s): /dev/nvme0n1p1
[   15.277304] mount /dev/nvme0n1p1 /mnt
[   15.310067] EXT4-fs (nvme0n1p1): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[   15.312018] is_boot_part_for_disk_enc /dev/nvme0n1p1 /mnt

But my uart console shows that my nvme is normal while my rootfs not mount ssuccessful?

bash-5.1# mkdir /tmp/nvme_root
bash-5.1# mount /dev/nvme0n1 /tmp/nvme_root  
nvme0n1     nvme0n1p11  nvme0n1p14  nvme0n1p3   nvme0n1p6   nvme0n1p9
nvme0n1p1   nvme0n1p12  nvme0n1p15  nvme0n1p4   nvme0n1p7   
nvme0n1p10  nvme0n1p13  nvme0n1p2   nvme0n1p5   nvme0n1p8   
bash-5.1# mount /dev/nvme0n1p1 /tmp/nvme_root
bash-5.1# [ 3115.811055] EXT4-fs (nvme0n1p1): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.

bash-5.1# 
bash-5.1# ls /tmp/nvme_root/
README.txt  boot  etc   lib         media  opt   root  sbin  srv  tmp  var
bin         dev   home  lost+found  mnt    proc  run   snap  sys  usr
bash-5.1# cat /proc/cmdline
root=/dev/initrd rw rootwait mminit_loglevel=4 console=ttyTCU0,115200 firmware_class.path=/etc/firmware fbcon=map:0 net.ifnames=0 bl_prof_dataptr=2031616@0x471E10000 bl_prof_ro_ptr=65536@0x471E00000
9

Hi,
Enabling CONFIG_HZ_1000 is not necessary. It is an example we validate RT kernel on developer kit. Please refer to the steps and try.

10

Hi,
Dear DaneLLL
I follow the step but enter the bash with no authority to mount for nvme rootfs.Howver, I can get the nvme mount while I enter the bash. I guess the initramfs is not build success.

11

Hi,
Please also apply this and try again:

Jetson/L4T/r36.4.x patches - eLinux.org
[Orin] Display drivers install to the wrong path with official document instructions

This is a known issue. Please manually copy the ko files to correct path.

13

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.