Nv-l4t-bootloader-config systemd service fails

Autonomous Machines Jetson & Embedded Systems Jetson Nano
1

I just installed and updated the latest Jetson Nano release and if I run systemctl --failed I see that the nv-l4t-bootloader-config.servie is failing.

I tried to dig into the script and I managed to find that is caused by

dpkg-reconfigure nvidia-l4t-bootloader

Anyone an idea what is going wrong here?

Mär 09 10:07:39 cluster-node03 nv-l4t-bootloader-config.sh[6403]: debconf: (TERM is not set, so the dialog frontend is not usable.)
Mär 09 10:07:39 cluster-node03 nv-l4t-bootloader-config.sh[6403]: debconf: falling back to frontend: Readline
Mär 09 10:07:39 cluster-node03 nv-l4t-bootloader-config.sh[6403]: debconf: unable to initialize frontend: Readline
Mär 09 10:07:39 cluster-node03 nv-l4t-bootloader-config.sh[6403]: debconf: (This frontend requires a controlling tty.)
Mär 09 10:07:39 cluster-node03 nv-l4t-bootloader-config.sh[6403]: debconf: falling back to frontend: Teletype
Mär 09 10:07:40 cluster-node03 nv-l4t-bootloader-config.sh[6403]: Starting bootloader post-install procedure.
Mär 09 10:07:40 cluster-node03 nv-l4t-bootloader-config.sh[6403]: ERROR. Procedure for bootloader update FAILED.
Mär 09 10:07:40 cluster-node03 nv-l4t-bootloader-config.sh[6403]: Cannot install package. Exiting...
2

Hi goertz,

Could you share the detailed step you’ve done to hit this error? also tell us the purpose of these steps if possible.

Thanks.

3

Hi,

Ok, I’ll give a summary of what I have done.

I don’t need the graphical stuff so I just boot to text mode

systemctl set-default multi-user.target

Just to be sure that I’m on high power mode

nvpmodel -m 0

Disable swap for kubernetes setup (read that this is sometimes neccessary)

swapoff -a

I set nvidia as my default docker runtime to have GPU enabled kubernetes

vim /etc/docker/daemon.json

“default-runtime”: “nvidia”,

Update the system to the latest and greates

apt update && apt upgrade -y && apt dist-upgrade -y && apt autoclean -y && apt autoremove -y

Add myself to the docker group

sudo usermod -aG docker $USER
sudo newgrp docker

reboot

That was all after the reboot I did my obligatory

systemctl --failed

and found the issue above.

For Debugging I also set the default target back to graphical. This lead to exact the same result.

5

Sorry for late reply. I think setting docker here is the cause of your issue.

Could you check?

6

Hi, I’m not sure what you mean by setting docker. Do you mean adding myself to the docker group oder setting nvidia as default runtime for docker?

7

Hi,

Sorry that I am not very familiar with docker.
What I wanted to point out is I guess you are trying to run docker on jetson nano but you hit the error, right?

Then my problem is it necessary to run nvidia-l4t-bootloader deb file in your usecase? This service is used for OTA update for jetson nano bootloader.

8

I’m not sure if it is necessary to have the nv-l4t-bootloader-config.service not failing. Everything seems to work fine even with this service failing, but on my systems I normally don’t like failing services, so I wanted to report it to be investigated.

9

I am wondering such tool may conflict with docker environment. Such tool needs to access the partition on system directly. If this is not needed in your case, I would say you could disable this service by systemctl first.

10

I did not really configure anything docker related, ist mostly what comes out of the box from the latest image. But I will disable the service for now (which I already have).

11

Oh, do you mean you didn’t enable any docker container yet but just run the steps from above comments?

Actually, I think the one that might cause problem is either the docker or the

apt update && apt upgrade -y && apt dist-upgrade -y && apt autoclean -y && apt autoremove -y

If this does not cause by docker, then probably the apt commands are the cause. I don’t think disable graphical mode or nvpmodel would be the cause.

I will check it with my device. Thanks for report.

15

And one more thing to check… which release do you see this error?

16

I installed the JetPack 4.3 SD card image from the download page.

Oh, do you mean you didn’t enable any docker container yet but just run the steps from above comments?

Yes, the commands above where all that was necessary on a vanilla cloned sd card. No other command ran before.

19

Hi goertz,

We can’t reproduce this issue by following your comment #3 steps on r32.3.1/Nano B01.
Is this still an issue at your side? Any result can be shared?

20

Hi @kayccc,

I’ve just seen I have the same problem on AGX Xavier after a problematic OTA upgrade where it ran out of disk space. This might give a way for reproducing this.

I had not really noticed till now, but this might be related to several problems i have been facing with Argus since then.
I get:

sudo systemctl --failed
  UNIT                             LOAD   ACTIVE SUB    DESCRIPTION                      
● nv-l4t-bootloader-config.service loaded failed failed Configure QSPI bootloader service
● systemd-modules-load.service     loaded failed failed Load Kernel Modules

Seems reasonable to me to first try fixing the former one.

I have tried @WayneWWW’s command without improvement.

How can I debug this ?
Is it possible to fix it without flashing ?
Or download only the required partition/image for flashing (preferably from Xavier itself) ? Sadly, I have low internet BW so far and a new SDK manager and SW takes almost one day to download…Flashing previous version and OTA upgrade is almost the same :-(

22

Hi Honey_Patouceul,

Actually we didn’t provide any solution yet. The command you pasted is just a step from another user and I was trying to figure out why he could reproduce this issue.

So do you mean this issue could happen if the disk space is running out? Also, what is the exact error you will hit when running the OTA? It must have some error that makes you to check the systemctl, right?

23

Hi @WayneWWW,

Yes, it happened when I upgraded OTA from R32.3.1 to R32.4.2 with about 3.5 GB of free space in eMMC.
The upgrade first downloaded about 2.5G of packages and then started the upgrade without any warning about space requirement.

After it had installed a few packages, I have been warned by Ubuntu that the system was running out of space. I thought I could reboot in order to clean cache from already installed packages. When I tried to reboot, I got a popup window about nv-l4t-bootloader install asking me to confirm. I did.
Upon reboot, my rootfs was full, and I’ve had to use serial console for deleting folders and make some room for install to terminate. I had noticed that the apt cache was full of both and new verions of packages. I did fix that with the commands from the above mentioned post and I was thinking it was cleared.
Only one month later I’ve seen that this system unit was failing. It is globally working, only had some problems with Argus, not sure if it is related or if it was because I had low disk space.

[EDIT: dpkg-reconfigure shows no error:

dpkg-reconfigure nvidia-l4t-bootloader
2888-400-0001-E.0-1-2-jetson-xavier-mmcblk0p1
Starting bootloader post-install procedure.
Update bootloader completed.
Reboot the target system for changes to take effect.

But it keeps failing after reboot.]

Attached serial_log for early boot, dmesg and syslog for further details.

logs

xavier_serial.log (28.8 KB)
dmesg.log (74.3 KB)
syslog.log (280.1 KB)

24

Only one month later I’ve seen that this system unit was failing. It is globally working, only had some problems with Argus,

Do you mean you cannot tell if this OTA is successful or not but just notice there are some problem running Argus?

I guess the disk space is the reason we didn’t reproduce this issue since we always re-flash the board before doing any test. Let me try this on our device first.

25

Yes, I didn’t notice other problem except some locks or crashs with argus, but I was able to workaround these.
I have just noticed a few days ago after an aborted install of some software leading to modules failing to be found…I just noticed an error and found the bootloader package status.

I cannot tell if tho OTA had been successful, but I doubt it was. Furthermore, I notice the following:

dpkg -l | grep nvidia
ii  libnvidia-container-tools                  0.9.0~beta.1                                     arm64        NVIDIA container runtime library (command-line tools)
ii  libnvidia-container0:arm64                 0.9.0~beta.1                                     arm64        NVIDIA container runtime library
ii  nvidia-container-csv-cuda                  10.2.89-1                                        arm64        Jetpack CUDA CSV file
ii  nvidia-container-csv-cudnn                 8.0.0.145-1+cuda10.2                             arm64        Jetpack CUDNN CSV file
ii  nvidia-container-csv-tensorrt              7.1.0.16-1+cuda10.2                              arm64        Jetpack TensorRT CSV file
ii  nvidia-container-csv-visionworks           1.6.0.501                                        arm64        Jetpack VisionWorks CSV file
ii  nvidia-container-runtime                   3.1.0-1                                          arm64        NVIDIA container runtime
ii  nvidia-container-toolkit                   1.0.1-1                                          arm64        NVIDIA container runtime hook
ii  nvidia-docker2                             2.2.0-1                                          all          nvidia-docker CLI wrapper
ii  nvidia-l4t-3d-core                         32.4.2-20200408182620                            arm64        NVIDIA GL EGL Package
ii  nvidia-l4t-apt-source                      32.4.2-20200408182620                            arm64        NVIDIA L4T apt source list debian package
ii  nvidia-l4t-bootloader                      32.4.2-20200408182620                            arm64        NVIDIA Bootloader Package
ii  nvidia-l4t-camera                          32.4.2-20200408182620                            arm64        NVIDIA Camera Package
rc  nvidia-l4t-ccp-t186ref                     32.3.1-20191209230245                            arm64        NVIDIA Compatibility Checking Package
ii  nvidia-l4t-configs                         32.4.2-20200408182620                            arm64        NVIDIA configs debian package
ii  nvidia-l4t-core                            32.4.2-20200408182620                            arm64        NVIDIA Core Package
ii  nvidia-l4t-cuda                            32.4.2-20200408182620                            arm64        NVIDIA CUDA Package
ii  nvidia-l4t-firmware                        32.4.2-20200408182620                            arm64        NVIDIA Firmware Package
ii  nvidia-l4t-graphics-demos                  32.4.2-20200408182620                            arm64        NVIDIA graphics demo applications
ii  nvidia-l4t-gstreamer                       32.4.2-20200423124225                            arm64        NVIDIA GST Application files
ii  nvidia-l4t-init                            32.4.2-20200408182620                            arm64        NVIDIA Init debian package
ii  nvidia-l4t-initrd                          32.4.2-20200408182620                            arm64        NVIDIA initrd debian package
ii  nvidia-l4t-jetson-io                       32.4.2-20200408182620                            arm64        NVIDIA Jetson.IO debian package
ii  nvidia-l4t-jetson-multimedia-api           32.4.2-20200408182620                            arm64        NVIDIA Jetson Multimedia API is a collection of lower-level APIs that support flexible application development.
ii  nvidia-l4t-kernel                          4.9.140-tegra-32.4.2-20200408182620              arm64        NVIDIA Kernel Package
ii  nvidia-l4t-kernel-dtbs                     4.9.140-tegra-32.4.2-20200408182620              arm64        NVIDIA Kernel DTB Package
ii  nvidia-l4t-kernel-headers                  4.9.140-tegra-32.4.2-20200408182620              arm64        NVIDIA Linux Tegra Kernel Headers Package
ii  nvidia-l4t-multimedia                      32.4.2-20200408182620                            arm64        NVIDIA Multimedia Package
ii  nvidia-l4t-multimedia-utils                32.4.2-20200408182620                            arm64        NVIDIA Multimedia Package
ii  nvidia-l4t-oem-config                      32.4.2-20200408182620                            arm64        NVIDIA OEM-Config Package
ii  nvidia-l4t-tools                           32.4.2-20200408182620                            arm64        NVIDIA Public Test Tools Package
ii  nvidia-l4t-wayland                         32.4.2-20200408182620                            arm64        NVIDIA Wayland Package
ii  nvidia-l4t-weston                          32.4.2-20200408182620                            arm64        NVIDIA Weston Package
ii  nvidia-l4t-x11                             32.4.2-20200408182620                            arm64        NVIDIA X11 Package
ii  nvidia-l4t-xusb-firmware                   32.4.2-20200408182620                            arm64        NVIDIA USB Firmware Package

Are config files from nvidia-l4t-ccp-t186ref 32.3.1-20191209230245 expected to be there ?

Glad to help providing a new test case ;-)

26

Hi,

Please provide below info. Thanks!

  1. show me the /etc/nv_boot_control.conf
  2. attach the /opt/ota_package/bl_update_payload.log
27 
cat /etc/nv_boot_control.conf
TNSPEC 2888-400-0001-J.0-1-2-jetson-xavier-mmcblk0p1
COMPATIBLE_SPEC 2888-400-0001-E.0-1-2-jetson-xavier-mmcblk0p1
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mmcblk0boot0
TEGRA_OTA_GPT_DEVICE /dev/mmcblk0boot1

Note that this file has date of OTA, while attched bl_update_payload.log has date from 2 days ago when I’ve relaunched the apt commands and dpkg-reconfigure nvidia-l4t-bootloader.
bl_update_payload.log (15.9 KB)