Optimizing Kernel Boot Time on the Jetson Xavier NX

Robotics & Edge Computing Jetson & Embedded Systems Jetson Xavier NX
1

Hello Everyone, I am working on Optimizing the Boot time for the Xavier NX Device. The current boot time on my Xavier NX takes approx 36 seconds from old power-on to begin showing the home screen (Password required for login is removed). However, I would like to reduce this as much as possible.
I was researching through the Nvidia Jetson Developer Guide: Kernel Optimization Chapter and there are several methods mentioned there to reduce the boot-time

  • I was able to find one method that could work which was Disabling Console over UART by editing the p3668.conf.common file and removing the specification → CMDLINE_ADD=“console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0”; on line 103 (Could be different for different users)
    I wanted to confirm whether is this the right way ?

  • I would also like to know if there are any more methods applicable which would help in reducing the boot-time particularly of the Xavier NX because I could not find out a way to apply the other given methods

Looking forward to the responses.

2

I can’t completely answer your question, but here is some related info…

Before you remove serial console you should save a full serial console boot log. The reason I say this is that the boot log provides timestamps from both before Linux loads and after Linux loads. You can use those to track what needs optimizing. Before you save that log be sure to remove the word “quiet” from your “/boot/extlinux/extlinux.conf” so the log is full.

About that, consider that at the point when the kernel Image file is loaded (which the log shows) the boot transfers from earlier boot stages tot he Linux kernel actually taking over. If you see something in need of optimizing once the Linux kernel takes over, then it is easier to deal with. If you see something in need of optimizing during earlier boot stages, then you might have to rebuild some software.

The serial console prints at speed 115200 (115.2kbps), and boot might wait for something to print, or else the overhead of printing might slow down boot. So this could help, but because of its utility, you are advised to not do so until you’ve tracked down every other optimization. If you remove serial console messages, then you’ve also cut off some of the tools useful to finding bottlenecks.

If it turns out you don’t see any specific slow spots, e.g., where it waits to allow a user to interrupt boot in case they want to pick a different Linux kernel, then possibly removing serial console could help. Keep in mind that removing serial console during Linux kernel run is separate from removing console in earlier boot stages…they are separate applications. If you have a problem with serial console slowing things down, then it is trivial to disable serial console in the Linux stage, but not as trivial (would require flash I think) to remove in those earlier stages…but if only one of the stages is a problem, then why disable all of them? You can enable/disable serial console separately for Linux and for boot (or both at the same time if you follow the documents on serial console disable and flash).

So…get a serial console boot log from the start. I couldn’t tell you everything you can do with that, but if you post a copy of the log, then it might be possible to post some ideas based on that.

Note: The “console=ttyTCU0,115200n8” is the serial UART console. The “console=tty0” is displaying boot messages on the text terminal, not the serial console. Both could slow things down, but the text mode console only runs in Linux and would not help to disable it if latency is in a boot stage prior to Linux running.

3

Hello @linuxdev,
Thank you for your response. With your suggestion I have removed the quite keyword from the extlinux.conf file and captured the boot-log
The boot-log has been captured using the following commands

  • nvidia@nvidia-desktop:~$ journalctl --list-boots

  • nvidia@nvidia-desktop:~$ journalctl --boot=97897c866c2c40e380c6f90302e865ba > bootlog.txt

bootlog.txt (244.6 KB)

From what I could understand from the log is that it shows all functions/services of the Linux kernel being loaded and the most time consuming is the Gnome Desktop Environment Initialization. I have attached the boot-log for your reference as well.

4

Can you log from serial console for the full boot log? The log you gave should be good for the time when Linux is running, but it completely ignores all of the time taken in boot stages before Linux actually runs. A serial console log includes not only what you’ve included, but also the content before Linux ever starts.

Regarding what is shown, the log starts at 09:10:44, and continues more or less uniformly to 09:11:14. I’ve cut that time field and passed it to uniq, and for reference this is the list of those time stamps with duplicates removed:

09:10:44
09:10:45
09:10:46
09:10:47
09:10:48
09:10:49
09:10:50
09:10:51
09:10:52
09:10:53
09:10:54
09:10:55
09:10:56
09:10:57
09:10:58
09:11:00
09:11:03
09:11:04
09:11:05
09:11:06
09:11:09
09:11:10
09:11:13
09:11:14

Looks like the longest wait between timestamp changes is 3 seconds, while the total time span is 30 seconds. Whatever time is being spent in boot, beyond the 30 seconds, is due to the time in boot stages prior to Linux running. It would be quite useful to know what those serial console timestamps are.

5

Hello @linuxdev,
Thanks for your reply. I tried to get the log from the serial console but I am unable to do so. When I connect the device through the micro-usb port, the device gets listed as /dev/ttyACM0. Then I log into the serial console using putty/minicom using the following command:

  • sudo putty /dev/ttyACM0 -serial -sercfg 115200,8,n,1,N or sudo minicom -D /dev/ttyACM0 -8 -b 115200

The issue is, when I reboot the XNX, I get disconnected from the serial console and hence, I am not able to see the log.

I tired to search for another way to get the the boot log and came across the dmesg utility. I generated the log and went through it, but I could find the contents similar to the jounralctl command. However, I am still attaching that for your reference.

dmesg.log (68.3 KB)

I would like to know if there is any other possible to obtain the complete boot log.

6

I suspect this is an issue of PuTTY running on Windows. There is no actual “disconnect” with a serial port…some setting within PuTTY is apparently set up to disconnect upon logout…check PuTTY settings. If you can boot to Linux on a PC and monitor from there, then that should give you a continuous log. You should be able to start a log before ever applying power, and then when you apply power it should just start logging without any login required.

Note that there is no possibility of journalctl (or dmesg) logging content from boot stages since Linux is not even running at those times.

Note: I think the NX probably has the same serial console instructions as the Nano (for SD card dev kit models). For that see:
https://www.jetsonhacks.com/2019/04/19/jetson-nano-serial-console/

7

Hi @linuxdev ,
Thanks for providing the link. I was actually using the same link for using the serial console. I am using Ubuntu 20.04 with putty/minicom.
In the video you can observe that when the user types the dmesg --follow command and connects the XNX via USB (XNX has no power), the device pops up showing that it is connected to ttyUSB0. However for me nothing of that sort appears. That is why this process is not working for me. When there is no power to the XNX, no communication is happening with the device.
The XNX that I am using is a custom model from a third party OEM. Maybe that could be the reason. I might require some additional module to that and then I guess it should be possible.

8

If you were to monitor a serial console during boot of a dev kit, then you would see log output during boot without any need to run the “dmesg” command. In fact, dmesg does not exist while in boot loader stages. In the case of a third party carrier board the pins will possibly be routed differently (it is after all a different carrier board layout), and the serial console might require a device tree modification to run either (A) before the Linux kernel is loaded, and/or (B) once the Linux kernel is running. I suspect that your carrier board does not have the correct device tree if serial console does not show output prior to the Linux kernel running, or if output requires using dmesg.

Have you flashed this board before? If this is for a third party carrier board, then you must use their board support package. Flashing the NVIDIA dev kit software would cause parts of the carrier board to fail or be inaccessible (due to the module not knowing the correct routing of pins).

9

Hi @linuxdev ,
Yes I have flashed this board to boot directly from the nvme. But I used Jetpack 4.6 and BSP provided by them to flash the board.

10

Things should work ok then in this case. One thing I don’t know though is if the supplier of your BSP enabled serial console logging in stages prior to the Linux kernel. If this were a dev kit, then the log should look something like this in the middle of boot (this is actually from a TX2, but the log would be similar with the NX):

Retrieving file: /boot/extlinux/extlinux.conf
213 bytes read in 115 ms (1000 Bytes/s)
p2771-0000 eMMC boot options
1:	primary kernel
Enter choice: 1:	primary kernel
Retrieving file: /boot/Image
34048008 bytes read in 855 ms (38 MiB/s)
append: root=/dev/mmcblk0p1 rw rootwait console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 video=tegrafb no_console_suspend=1 earlycon=uart8250,mmio32,0x3100000 nvdumper_reserved=0x2772e0000 gpt usbcore.old_scheme_first=1 tegraid=18.1.2.0.0 maxcpus=6 boot.slot_suffix= boot.ratchetvalues=0.2031647.1 bl_prof_dataptr=0x10000@0x275840000 sdhci_tegra.en_boot_part_access=1 root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4
## Flattened Device Tree blob at 80000000
   Booting using the fdt blob at 0x80000000
   reserving fdt memory region: addr=80000000 size=10000
   Using Device Tree in place at 0000000080000000, end 0000000080058fd4

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x100

All content up to and including the note of reading file “/boot/Image” (or a partition image) is from the bootloader stage or earlier. If this is where your log starts, then you might need to contact the carrier board vendor to find out how to enable logging in those stages. This sounds a bit backwards since no logging improves performance during those stages, but it fails to give you timestamps. Those timestamps tell you where boot time is spent, and so you’d want to enable the log, fix anything taking too long, and then disable the log.

If the log does show earlier content, then this is valuable information which is never available in dmesg. Does your serial console show earlier boot content?

11

Hi @linuxdev,
Thank you for providing the snippet. This will help me as a reference to what I should be looking for. I guess I need to contact the vendor for the configuration of the serial port.
As you can see the from the log it starts from
[ 0.000000] Booting Linux on physical CPU 0x0

The only way I can log into the serial console is when the board powers-on and the login screen appears. Before that, the serial console doesn’t work. Hence the serial console doesn’t show anything (no boot content).

12

Looks like the BSP from the carrier board manufacturer disabled serial console prior to Linux running.

13

Hi @linuxdev,
I am waiting for an answer from the carrier board manufacturer regarding the serial console, However I made some changes to the system which reduced the boot time from 36 to 19 seconds

  • I replaced the Gnome desktop with a light window manager called Xfce

  • In /boot/extlinux/extlinux.conf file, I removed the console=tty0 argument for displaying boot messages on the text terminal

I will get back to you as soon as I get an answer from them.

14

Hi @linuxdev,
I would like to ask do you have any idea, how to disable the Nvidia Splash Screen ? When I boot up the board the Nvidia Logo appears twice and I want to disable both of them.

15

I’ve never changed the splash screen. I know many people have changed this to a customized splash image, so although it wouldn’t help boot time, for sure you could replace this with something like a pure black background. Likely you could completely stop the splash screen, but I couldn’t tell you how (this would probably be a good question in a new thread).

16

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.