New Jetpack 5.1.3 Xavier NX flash and apt upgrade results in broken setup

Autonomous Machines Jetson & Embedded Systems Jetson Xavier NX
,
1

Using my Xavier NX board I flashed Jetpack 5.1.3 to NVME disk, completed the setup and had a working board.

Upon reboot I had 140+ packages waiting for update. sudo atp upgrade and then reboot results in the board startup process failing. the messages that appear most relevant are


depmod: WARNING: could not open modules.order at /lib/modules/5.10.192-tegra: No such file or directory
tegra_ccc 3960000.tegra_cec: Sent <Text View On> res: -113
tegra_ccc 3960000.tegra_cec: tegra_cec_init Done.
depmod: WARNING: could not open modules.builtin at /lib/modules/5.10.192-tegra: No such file or directory
No root-device: Mount failed
cannot set terminal process group (-1): Inappropriate ioctl for device
no job control in this shell

The start up leaves me at a bash prompt as the root user and an ls of /lib/modules/5.10.192-tegra does show that there are no files with those names.

2

Dump the boot log by using this would clarify the situation.

3

xavier_5.1.3_boot.txt (61.9 KB)

Here is the debug port dump when starting the board.

4

So what is that “root= PARTUUID=1a0e2c42-333b-4d74-ba06-842ebc54a57c” boot drive on your device? This is the one that is missing so cannot boot.

5

That is /dev/nvme0n1p1 It has the PARTLABEL="APP" TYPE="ext4

7

[ 6.115945] No root-device: Mount failed

It looks like the UUID cannot be found. How about your just changed the kernel cmdline in your /boot/extlinux/extlinux.conf to root=/dev/nvme0n1p1?

8

Interesting… ls /boot shows no such file or directory

After a fresh flash everything worked. I suspect it was that one of these packages that were the culprit

  "nvidia-l4t-bootloader"
  "nvidia-l4t-configs"
  "nvidia-l4t-display-kernel"
  "nvidia-l4t-initrd"
  "nvidia-l4t-kernel"
  "nvidia-l4t-kernel-dtbs"
  "nvidia-l4t-kernel-headers"
  "nvidia-l4t-tools"
  "nvidia-l4t-xusb-firmware"

Here is a dump of the apt upgrade command run immediately after flashing
FAILED_UPGRADE.txt (57.8 KB)

9

What is the first boot option from your UEFI? This seems to be a sdcard module, are you sure the UEFI tries to boot from nvme but not that sdcard?

Did you ever check that before?

10

I did check, NVME drive is listed, as first and I select it explicitly. The SD card actually won’t select. I tried in the boot menu but it won’t accept the command. It is listed as a device though.

11

This currently has my attention. It was one of the last lines in the upgrade command output

update-initramfs: Generating /boot/initrd.img-5.10.192-tegra
W: Couldn't identify type of root file system for fsck hook
12

BTW, just a clarification .

/boot is under the nvme drive. So if you want to modify it, you need to mount it up first as your are operating in initrd shell.

13

facepalm yes…after mounting and opening up /boot/extlinux/extlinux.conf I saw root= was empty. You were right, setting it root=/dev/nvme0n1p1 put my board is back up and in a running state.

So does that mean that the warning in apt upgrade

Couldn't identify type of root file system for fsck hook

Could have been the culprit? I’m just trying to understand how the boot/extlinux/extlinux.conf was modified.

14

Actually the exact things you need to do for me is give step by step commands which you have run to reproduce this error.

What was the jetpack version before running this.

If there is indeed an error, then we will help check.

15

Also, a workaround to “avoid” this issue for now is remove the nvidia related apt source file from your /etc/apt /sources.list.

16
  1. SDK Manager Flash 5.1.3 Jetpack using Runtime setup and NVME setting
  2. Complete runtime setup on board
  3. Using SDK Manager establish USB connection to finish installation of other Jetson dependencies
  4. Reboot board. Board reboots as expected
  5. sudo apt update. 140+ packages are upgradeable. See FAILED_UPGRADE.txt attachment for specific packages and versions
  6. sudo apt upgrade. (everything looks ok but that one fsck hook warning)
  7. sudo reboot
  8. Error as we discussed.
18

Hello, I am experiencing exactly the same issue on my Xavier NX, following exactly the same steps as described by @drew22. After a clean install of Jetpack 5.1.3 on a NVME disk (no SDCard inserted) using the sdkmanager, if I run sudo apt update && sudo apt upgrade and reboot, the Jetson hangs during startup with the same messages described in the first post (note: there is no issue to reboot prior to the apt upgrade call).

It happend twice in 2 days. The first time I was updating from Jetpack 3.x to 5.1.3, the second time was after the first occurance of the issue, so I was “updating” from Jetpack 5.1.3 to Jetpack 5.1.3.

I am not sure I understood what to do exactly at this point to still be able to boot (appart reflashing the NX entirely with sdkmanager).

Edit: probably not related but after every reboot with Jetpack 5.1.3 I get a Ubuntu notification telling me “nvidia-l4t-bootloader Post Install Notification. Reboot is required to complete the installation.” The message appear even after multiple reboots.

20

Please add this patch to /sbin/nv-update-extlinux and it will fix this issue.

diff --git a/rfs/usr/sbin/nv-update-extlinux b/rfs/usr/sbin/nv-update-extlinux
index e450a36..e1bc435 100755
--- a/rfs/usr/sbin/nv-update-extlinux
+++ b/rfs/usr/sbin/nv-update-extlinux
@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 
-# SPDX-FileCopyrightText: Copyright (c) 2020-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-FileCopyrightText: Copyright (c) 2020-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: MIT
 #
 # Permission is hereby granted, free of charge, to any person obtaining a
@@ -36,12 +36,11 @@
     # strip extra spaces if any
     current = re.sub(' +', ' ', current.rstrip())
 
-    params = re.findall( r'([^=^\s]+=[^=]+)(?:[\s]|$)', current)
-    head = current.replace(' '.join(params), '')
+    params = current.split()
     for param in additions:
         if param not in params:
             params.append(param)
-    return head.rstrip() + ' ' +  ' '.join(params)
+    return ' '.join(params)
 
 def add_extlinux_entry(image_path, label, mlabel, kernel_params):
     # Check extlinux.conf
1 Like
21

How do we add this to the jetson xaiver NX with 5.1.3 when its in the “broken” state? I am having the same issue with the exact same process that @drew22 did. Should I re flash and do this fix or is there a way to do this without it booting?
Should I sudo apt upgrade then apply this fix before rebooting?

Thank you

22

This patch is to prevent the issue happening, but not fix it when it is already broken.

23

Just sharing my way to resolve this after getting the same issue

Mount the nvme ssd on a working PC , navigate to /boot/extlinux/extlinux.conf
Found out that there is an extra space after the “root=” which causes the UDID being not recognized.
Delete that extra space and mount the ssd back to jetson.