Nv-l4t-bootloader-config systemd service fails

I am wondering such tool may conflict with docker environment. Such tool needs to access the partition on system directly. If this is not needed in your case, I would say you could disable this service by systemctl first.

I did not really configure anything docker related, ist mostly what comes out of the box from the latest image. But I will disable the service for now (which I already have).

Oh, do you mean you didn’t enable any docker container yet but just run the steps from above comments?

Actually, I think the one that might cause problem is either the docker or the

apt update && apt upgrade -y && apt dist-upgrade -y && apt autoclean -y && apt autoremove -y

If this does not cause by docker, then probably the apt commands are the cause. I don’t think disable graphical mode or nvpmodel would be the cause.

I will check it with my device. Thanks for report.

And one more thing to check… which release do you see this error?

I installed the JetPack 4.3 SD card image from the download page.

Oh, do you mean you didn’t enable any docker container yet but just run the steps from above comments?

Yes, the commands above where all that was necessary on a vanilla cloned sd card. No other command ran before.

Hi goertz,

We can’t reproduce this issue by following your comment #3 steps on r32.3.1/Nano B01.
Is this still an issue at your side? Any result can be shared?

Hi @kayccc,

I’ve just seen I have the same problem on AGX Xavier after a problematic OTA upgrade where it ran out of disk space. This might give a way for reproducing this.

I had not really noticed till now, but this might be related to several problems i have been facing with Argus since then.
I get:

sudo systemctl --failed
  UNIT                             LOAD   ACTIVE SUB    DESCRIPTION                      
● nv-l4t-bootloader-config.service loaded failed failed Configure QSPI bootloader service
● systemd-modules-load.service     loaded failed failed Load Kernel Modules              

Seems reasonable to me to first try fixing the former one.

I have tried @WayneWWW’s command without improvement.

How can I debug this ?
Is it possible to fix it without flashing ?
Or download only the required partition/image for flashing (preferably from Xavier itself) ? Sadly, I have low internet BW so far and a new SDK manager and SW takes almost one day to download…Flashing previous version and OTA upgrade is almost the same :-(

Hi Honey_Patouceul,

Actually we didn’t provide any solution yet. The command you pasted is just a step from another user and I was trying to figure out why he could reproduce this issue.

So do you mean this issue could happen if the disk space is running out? Also, what is the exact error you will hit when running the OTA? It must have some error that makes you to check the systemctl, right?

Hi @WayneWWW,

Yes, it happened when I upgraded OTA from R32.3.1 to R32.4.2 with about 3.5 GB of free space in eMMC.
The upgrade first downloaded about 2.5G of packages and then started the upgrade without any warning about space requirement.

After it had installed a few packages, I have been warned by Ubuntu that the system was running out of space. I thought I could reboot in order to clean cache from already installed packages. When I tried to reboot, I got a popup window about nv-l4t-bootloader install asking me to confirm. I did.
Upon reboot, my rootfs was full, and I’ve had to use serial console for deleting folders and make some room for install to terminate. I had noticed that the apt cache was full of both and new verions of packages. I did fix that with the commands from the above mentioned post and I was thinking it was cleared.
Only one month later I’ve seen that this system unit was failing. It is globally working, only had some problems with Argus, not sure if it is related or if it was because I had low disk space.

[EDIT: dpkg-reconfigure shows no error:

dpkg-reconfigure nvidia-l4t-bootloader
2888-400-0001-E.0-1-2-jetson-xavier-mmcblk0p1
Starting bootloader post-install procedure.
Update bootloader completed.
Reboot the target system for changes to take effect.

But it keeps failing after reboot.]

Attached serial_log for early boot, dmesg and syslog for further details.

logs

xavier_serial.log (28.8 KB)
dmesg.log (74.3 KB)
syslog.log (280.1 KB)

Only one month later I’ve seen that this system unit was failing. It is globally working, only had some problems with Argus,

Do you mean you cannot tell if this OTA is successful or not but just notice there are some problem running Argus?

I guess the disk space is the reason we didn’t reproduce this issue since we always re-flash the board before doing any test. Let me try this on our device first.

Yes, I didn’t notice other problem except some locks or crashs with argus, but I was able to workaround these.
I have just noticed a few days ago after an aborted install of some software leading to modules failing to be found…I just noticed an error and found the bootloader package status.

I cannot tell if tho OTA had been successful, but I doubt it was. Furthermore, I notice the following:

dpkg -l | grep nvidia
ii  libnvidia-container-tools                  0.9.0~beta.1                                     arm64        NVIDIA container runtime library (command-line tools)
ii  libnvidia-container0:arm64                 0.9.0~beta.1                                     arm64        NVIDIA container runtime library
ii  nvidia-container-csv-cuda                  10.2.89-1                                        arm64        Jetpack CUDA CSV file
ii  nvidia-container-csv-cudnn                 8.0.0.145-1+cuda10.2                             arm64        Jetpack CUDNN CSV file
ii  nvidia-container-csv-tensorrt              7.1.0.16-1+cuda10.2                              arm64        Jetpack TensorRT CSV file
ii  nvidia-container-csv-visionworks           1.6.0.501                                        arm64        Jetpack VisionWorks CSV file
ii  nvidia-container-runtime                   3.1.0-1                                          arm64        NVIDIA container runtime
ii  nvidia-container-toolkit                   1.0.1-1                                          arm64        NVIDIA container runtime hook
ii  nvidia-docker2                             2.2.0-1                                          all          nvidia-docker CLI wrapper
ii  nvidia-l4t-3d-core                         32.4.2-20200408182620                            arm64        NVIDIA GL EGL Package
ii  nvidia-l4t-apt-source                      32.4.2-20200408182620                            arm64        NVIDIA L4T apt source list debian package
ii  nvidia-l4t-bootloader                      32.4.2-20200408182620                            arm64        NVIDIA Bootloader Package
ii  nvidia-l4t-camera                          32.4.2-20200408182620                            arm64        NVIDIA Camera Package
rc  nvidia-l4t-ccp-t186ref                     32.3.1-20191209230245                            arm64        NVIDIA Compatibility Checking Package
ii  nvidia-l4t-configs                         32.4.2-20200408182620                            arm64        NVIDIA configs debian package
ii  nvidia-l4t-core                            32.4.2-20200408182620                            arm64        NVIDIA Core Package
ii  nvidia-l4t-cuda                            32.4.2-20200408182620                            arm64        NVIDIA CUDA Package
ii  nvidia-l4t-firmware                        32.4.2-20200408182620                            arm64        NVIDIA Firmware Package
ii  nvidia-l4t-graphics-demos                  32.4.2-20200408182620                            arm64        NVIDIA graphics demo applications
ii  nvidia-l4t-gstreamer                       32.4.2-20200423124225                            arm64        NVIDIA GST Application files
ii  nvidia-l4t-init                            32.4.2-20200408182620                            arm64        NVIDIA Init debian package
ii  nvidia-l4t-initrd                          32.4.2-20200408182620                            arm64        NVIDIA initrd debian package
ii  nvidia-l4t-jetson-io                       32.4.2-20200408182620                            arm64        NVIDIA Jetson.IO debian package
ii  nvidia-l4t-jetson-multimedia-api           32.4.2-20200408182620                            arm64        NVIDIA Jetson Multimedia API is a collection of lower-level APIs that support flexible application development.
ii  nvidia-l4t-kernel                          4.9.140-tegra-32.4.2-20200408182620              arm64        NVIDIA Kernel Package
ii  nvidia-l4t-kernel-dtbs                     4.9.140-tegra-32.4.2-20200408182620              arm64        NVIDIA Kernel DTB Package
ii  nvidia-l4t-kernel-headers                  4.9.140-tegra-32.4.2-20200408182620              arm64        NVIDIA Linux Tegra Kernel Headers Package
ii  nvidia-l4t-multimedia                      32.4.2-20200408182620                            arm64        NVIDIA Multimedia Package
ii  nvidia-l4t-multimedia-utils                32.4.2-20200408182620                            arm64        NVIDIA Multimedia Package
ii  nvidia-l4t-oem-config                      32.4.2-20200408182620                            arm64        NVIDIA OEM-Config Package
ii  nvidia-l4t-tools                           32.4.2-20200408182620                            arm64        NVIDIA Public Test Tools Package
ii  nvidia-l4t-wayland                         32.4.2-20200408182620                            arm64        NVIDIA Wayland Package
ii  nvidia-l4t-weston                          32.4.2-20200408182620                            arm64        NVIDIA Weston Package
ii  nvidia-l4t-x11                             32.4.2-20200408182620                            arm64        NVIDIA X11 Package
ii  nvidia-l4t-xusb-firmware                   32.4.2-20200408182620                            arm64        NVIDIA USB Firmware Package

Are config files from nvidia-l4t-ccp-t186ref 32.3.1-20191209230245 expected to be there ?

Glad to help providing a new test case ;-)

Hi,

Please provide below info. Thanks!

  1. show me the /etc/nv_boot_control.conf
  2. attach the /opt/ota_package/bl_update_payload.log
cat /etc/nv_boot_control.conf
TNSPEC 2888-400-0001-J.0-1-2-jetson-xavier-mmcblk0p1
COMPATIBLE_SPEC 2888-400-0001-E.0-1-2-jetson-xavier-mmcblk0p1
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mmcblk0boot0
TEGRA_OTA_GPT_DEVICE /dev/mmcblk0boot1

Note that this file has date of OTA, while attched bl_update_payload.log has date from 2 days ago when I’ve relaunched the apt commands and dpkg-reconfigure nvidia-l4t-bootloader.
bl_update_payload.log (15.9 KB)

Hi,

According to the log, the OTA updated successfully.

Does that nv-l4t-bootloader-config service always fail after reboot?

Yes.

Hi Honey_Patouceul,

Looks like we’ve found the reason.

Actually the service run successful. This service is used to update QSPI images, but your device is Xavier, no QSPI, so the service will exit directly. But our code issues “exit 1” for it, then the system supposed it as a failure.

We’ve pushed a new patch in nv-l4t-bootloader-config.sh to set the exit 1 to “exit 0”.

--- a/rfs/opt/nvidia/l4t-bootloader-config/nv-l4t-bootloader-config.sh
+++ b/rfs/opt/nvidia/l4t-bootloader-config/nv-l4t-bootloader-config.sh
@@ -92,7 +92,7 @@
 	# installed_deb_ver=32.2.0-20190514154120
 	installed_deb_ver="$(dpkg -l | grep "${PACKAGE_NAME}" | awk '/'${PACKAGE_NAME}'/ {print $3}')"
 	if [ "${installed_deb_ver}" == "" ]; then
-		exit 1
+		exit 0
 	fi
 
 	# 2. get main deb_version
@@ -205,7 +205,7 @@
 		-x "${T186REF_UPDATER}" ]]; then
 		read_t186ref_qspi_ver_info
 	else
-		exit 1
+		exit 0
 	fi
 
 	if update_qspi_check; then
1 Like

Hi Wayne,

Thanks, it worked.
Not sure if this affects all Xavier or only a particular hw version.

Thanks anyway for the fix not requiring a reflash.

It should only affect Nano and NX.