Error while updating to r35.4 using Debian packages

Autonomous Machines Jetson & Embedded Systems Jetson Xavier NX
1

I’ve tried following the instructions to upgrade my Xavier NX given here (“1.3.2 Upgrade Jetpack”):

https://docs.nvidia.com/jetson/jetpack/install-jetpack/index.html#upgrade-jetpack

I have updated my sources.list, apt update, apt upgrade, apt dist-upgrade. Then I get an error, which doesn’t go away with the suggested --fix-broken command. Here is what I get:

# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n]
Setting up nvidia-l4t-kernel (5.10.120-tegra-35.4.1-20230801124926) ...
Using the existing boot entry 'primary'
3668-100---1--jetson-xavier-nx-devkit-
Info. Active boot storage: nvme0n1
Info. Legacy mode: true
TNSPEC 3668-200-0000-H.0-1-0-jetson-xavier-nx-devkit-mmcblk0p1
COMPATIBLE_SPEC 3668-100---1--jetson-xavier-nx-devkit-
TEGRA_LEGACY_UPDATE true
TEGRA_BOOT_STORAGE nvme0n1
TEGRA_EMMC_ONLY false
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mtdblock0
TEGRA_OTA_GPT_DEVICE /dev/mtdblock0
Starting kernel post-install procedure.
Starting legacy update procedure.
ERROR. Procedure for kernel update FAILED.
Cannot install package. Exiting...
dpkg: error processing package nvidia-l4t-kernel (--configure):
installed nvidia-l4t-kernel package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
nvidia-l4t-kernel
E: Sub-process /usr/bin/dpkg returned an error code (1)

It seems that the command that fails in the post-install is a call to /usr/sbin/nv_bootloader_payload_updater. If I try to call this directly I get this:

root@estonia:/opt/ota_package# /usr/sbin/nv_bootloader_payload_updater
Start running: /opt/nvidia/l4t-bootloader-config/nv-l4t-bootloader-config.sh -c
3668-100---1--jetson-xavier-nx-devkit-
Tegra User Block Device: /dev/disk/by-partlabel
Tegra Boot Block Device: /dev/mtdblock0
Tegra GPT Block Device: /dev/mtdblock0
HEADER: MAGIC NVIDIA__BLOB__V3
HEX_VALUE 16975394
BLOB_SIZE 50801622
HEADER_SIZE 48
NUMBER_OF_ELEMENTS 20
HEADER_TYPE 0
UNCOMP_SIZE 50801622
MB1_RATCHET_LV 0
MTS_RATCHET_LV 0
ROLLBACK_FUSE_LV 0
Device TN Spec: 3668-200-0000-H.0-1-0-jetson-xavier-nx-devkit-mmcblk0p1
Device Compatible Spec: 3668-100---1--jetson-xavier-nx-devkit-
Device TN Spec: 3668-200-0000-H.0-1-0-jetson-xavier-nx-devkit-mmcblk0p1
Device Compatible Spec: 3668-100---1--jetson-xavier-nx-devkit-
Get Fuse mode failed. Status: 1
OTA Blob update failed. Status: 1
/usr/sbin/nv_bootloader_payload_updater Failed.

At this point it has copied the new kernel into place. But I haven’t tried to reboot it! I am nervous that this error is real and it will fail to come back up.

Can anyone suggest what I should try to resolve this?

Some things to note:

  • I have an NVME SSD. The original Ubuntu is still on the SD card, and the NVME has a symlink from /boot to the Ubuntu /boot, since the bootloader reads the kernel from the SD card.

  • I’ve not used this board for a while; I’ve likely missed some updates.

Suggestions anyone?

3

Is it a DevKit or a custom carrier board?
What L4T version are you upgrading from?

4

It’s a dev kit.

I’m not certain of the old version, but it may have been r32.x? Kernel version if 4.9.140.

5

Then please re-flash the device.
You won’t be able to upgrade from JetPack 4 to JetPack 5 via APT.

6

Thanks for the reply. Since my original post I found this page which claims to describe how to upgrade from 4.x to 5.x by first reflashing the QSPI:

https://docs.nvidia.com/jetson/archives/r35.1/DeveloperGuide/text/SD/FlashingSupport.html?highlight=upgrade%20jetpack#to-upgrade-jetpack-4-x-to-jetpack-5-x-on-jetson-xavier-nx-p3668-0000

That tells me to download the QSPI image from a linked web page, but that page now refers to JetPack 6, not JetPack 5, and the archived JetPack 5 page doesn’t seem to have a QSPI image download that I can see. (Am I missing something?)

I hesitate to upgrade to JetPack 6 because it is only a “preview” and I would prefer stability over latest features. I note that the JetPack 6 preview doesn’t support “OTA” (i.e. apt) updates to future versions.

Is there any chance of upgrading to JetPack 5 by following the instructions that I linked to above?

Thanks, Phil.

7

One further question:

If I just install a new SD card image for JetPack 5 or 6, will that work? That won’t have modified the QSPI flash, right?

8

JetPack 6 does not support Xavier series, so you don’t even need to consider whether you should do it.
For the QSPI image for Xavier NX, please check the page for JetPack 5.1.2:

https://developer.nvidia.com/embedded/L4T/r35_Release_v1.0/QSPI-img/Jetson_Xavier_NX_QSPI_35.1.gz

9

Definitely NO.

10

Thanks for the link to the QSPI download - though I still can’t see where that is on the actual web page! (Edit: found it, it’s hidden in a box that appears when you click on a “>”.)

If I now write that (with flash_cp), will I then be able to boot from a new SD card image?

11

image
image1340×760 56.9 KB

The image is for 35.1, so I’m not sure if it works with 35.4.1; maybe not.
As long as you have a Ubuntu host PC, the safest way is still use SDK Manager to flash the device; that way, you can make sure version of QSPI bootloader and OS on SD cards are matched.

12

https://developer.nvidia.com/embedded/L4T/r35_Release_v1.0/QSPI-img/Jetson_Xavier_NX_QSPI_35.1.gz

Slightly worryingly, that “.gz” file is actually a .tar.bz2…

13

YES, but I don’t feel like that makes any difference.

14

I’m currently downloading the SD card image - very slowly, over my amazingly-slow internet connection. However I realise that it is likely too large for the SD card that I currently have, which is 16 GB, and I don’t seem to have a larger one.

Remember that I have an NVMe SSD attached. So, I’m contemplating whether I can use the boot partitions from the new SD image and the root filesystem from NVMe.

I think this has a chance of working, but where does the kernel command line indicating the root device come from? How do I change it from mtdblk0p1 to nvme0n1p1? Is that hidden one of the other partitions in the SD image, like kernel-bootctrl, or something? (Another option would be NFS root.)

I also see that in my current old SD image there is a 105MB partition called RECROOTFS; is that a recovery root filesystem? If I create an SD card image with a truncated main root filesystem, can I boot with this recovery filesystem and get a prompt?

(Where are the docs for this?)

Many thanks for your help DaveYYY!

Cheers, Phil.

15

You cannot mix bootloader and rootfs coming from different L4T releases.
Please don’t to this.

I don’t know what’s the purpose here.

16

Anyway, the easiest way would still be flashing with SDK Manager.
It’s not worth it digging into so many options that won’t work when you just want to upgrade the device.

17

the easiest way would still be flashing with SDK Manager.

I understand, but please believe I’m not completely mad! I’m limited by e.g. having a Debian ARM, not Ubuntu, Linux host system that may or may not be able to run SDK Manager, and not having enough free disk space anyway. So I could, for example, install Ubuntu in a VM on my Mac, but then I would need to get USB forwarding to work.

You cannot mix bootloader and rootfs coming from different L4T releases.

The root FS on the NVMe drive is already upgraded to r35.4 using apt - but hasn’t been rebooted yet. If that is broken in some way, I could copy the FS from the new SD image to a new partition on the NVMe drive.

Any idea about where on the SD card image the kernel command line comes from?

18

Check /boot/extlinux/extlinux.conf.
However, if you are sure the NVMe drive is fully upgraded to 35.4, you can just select booting devices in UEFI menu.

19

You don’t necessarily need Ubuntu; that’s only applied when you are using SDK Manage because of package dependencies.
I think flashing scripts should also work on other Linux distros:
https://docs.nvidia.com/jetson/archives/r35.4.1/DeveloperGuide/text/SD/FlashingSupport.html

20

Check /boot/extlinux/extlinux.conf .

But that’s in the root filesystem, isn’t it? So it can’t be read to determine where the root filesystem is…

(Or is /boot a mount point for a different partition? It doesn’t seem to be on my existing system, but maybe I’ve screwed that up at some point.)

However, if you are sure the NVMe drive is fully upgraded to 35.4, you can just select booting devices in UEFI menu.

Ah, of course, there is now UEFI! I guess that is the answer.

Wish me luck!

21

I have got it (mostly??) working!

Basically, flashing the new QSPI image from within the old system and then restarting with the JetPack 5 SD image does work.

It took me longer than it should have because:

  • I only had a 16 GB SD card, and the new image is slightly larger. So I decided to “hack” the image down to below 16 GB; I mounted it on another system and removed 3 GB of /usr/local/cuda stuff that I don’t need currently. But then, although I know in principle how to shrink the filesystem and partition, I managed to get it subtly wrong.

  • Then the network wasn’t working… and it took me an embarrassingly long time to realise that this wasn’t a missing kernel module or incompatible device tree, but rather than I hadn’t plugged the ethernet cable back. Ooops.

  • It took me a long time to discover where in UEFI I could add kernel command line arguments (i.e. root=/dev/nvmexxx). It’s in a sub-menu called “NVIDIA Resources”, or somesuch.

So I think that I now have functional system! Yay! Of course I haven’t tested everything yet so I may still discover bugs.

Thanks for your help.

1 Like