Jetson Orin Nano: Flashing custom rootfs backup with flash.sh causes boot failure (root=PARTUUID issue / kernel panic)

Robotics & Edge Computing Jetson Systems Jetson Orin Nano
, ,
1

Hello,

I am working with a Jetson Orin Nano Developer Kit (P3767-0004) running JetPack 6.2 (L4T R36.4.x).
I attempted to create a golden image by backing up the root filesystem from a configured Jetson and restoring it onto another device.

Below are the exact steps I performed and the problem I encountered
1. Steps I followed

(A) On the source Jetson (working system)

I created a rootfs backup:

sudo tar --numeric-owner -cpzf nvme_backup_rootfs.tar.gz --one-file-system /

Then I copied the backup to the host PC.

(B) On the host PC

I extracted the backup into Linux_for_Tegra/rootfs:

sudo rm -rf Linux_for_Tegra/rootfs/*
sudo tar xpf nvme_backup_rootfs.tar.gz -C Linux_for_Tegra/rootfs --numeric-owner

I modified rootfs/boot/extlinux/extlinux.conf to replace:

root=PARTUUID=xxxx

with:

root=/dev/nvme0n1p1

(because the target device has a different PARTUUID).

(C) Flashing

I flashed using:

sudo ./flash.sh -r jetson-orin-nano-devkit nvme0n1p1

The flash completed successfully.

2. The problem

When I reboot the Jetson after flashing, it does not boot.
Instead, it shows the following errors on the display:

VFS: Cannot open root device "PARTUUID=xxxx" or unknown-block(0,0)
mount: /mnt/sysroot: mount point does not exist
cp: cannot create regular file '/mnt/sysroot/etc/resolv.conf'
Switching from initrd to actual rootfs
Kernel panic - not syncing: Attempted to kill init!

Even though I changed root=/dev/nvme0n1p1 in extlinux.conf, during boot the kernel still tries to use the old PARTUUID.

The system enters a kernel panic and resets repeatedly.

I also tried to backup via sudo ./tools/backup_restore/l4t_backup_restore.sh -e nvme0n1 -b jetson-orin-nano-devkit but it failed, error at step 3:

  •                                 *

  • Step 3: Start the flashing process *

  •                                 *

Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for device to expose ssh …Timeout
Device should have booted into initrd kernel now. However, the host cannot connect to its ssh server
Command failed: ping6 -c 1 fe80::1%enx42f3fa974e19. Check your network settings (firewall, VPN,…) so that it can ping fe80::1%enx42f3fa974e19
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether d0:50:99:84:a2:19 brd ff:ff:ff:ff:ff:ff
inet 192.168.6.39/24 brd 192.168.6.255 scope global dynamic noprefixroute enp2s0
valid_lft 449874sec preferred_lft 449874sec
inet6 fe80::434a:72f7:39a5:671a/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 72:6e:20:f9:0d:eb brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::706e:20ff:fef9:deb/64 scope link
valid_lft forever preferred_lft forever
10: br-f6824b08c474: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 76:5f:53:a8:35:5a brd ff:ff:ff:ff:ff:ff
inet 172.18.0.1/16 brd 172.18.255.255 scope global br-f6824b08c474
valid_lft forever preferred_lft forever
inet6 fe80::745f:53ff:fea8:355a/64 scope link
valid_lft forever preferred_lft forever
45: enx42f3fa974e19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
link/ether 42:f3:fa:97:4e:19 brd ff:ff:ff:ff:ff:ff
inet6 fc00:1:1::1/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::40f3:faff:fe97:4e19/64 scope link
valid_lft forever preferred_lft forever
inet6 fe80::2/64 scope link
valid_lft forever preferred_lft forever
Cleaning up…

My goal:
I want to clone the entire software environment (source code, libraries, Python environment, CUDA/TensorRT setup, system configuration, etc.) from one Jetson Orin Nano and flash the exact same environment onto multiple other Jetson Orin Nano devices.

I need the correct, NVIDIA-supported workflow for creating a Golden Image / Custom RootFS Image on JetPack 6.x (L4T R36.x) that can be flashed to many Jetsons reliably (NVMe boot, P3767-0004).

Thanks

3

*** Please note that this reply is generated by LLM automatically ***

Based on the provided context, it appears that you are trying to create a custom root file system (RootFS) image for JetPack 6.2 (L4T R36.x) on Jetson Orin Nano devices. You want to backup the root file system from a configured Jetson, modify it, and then restore it onto other Jetson Orin Nano devices.

The issue you encountered is that the kernel is trying to use the old PARTUUID instead of the new one you specified in the extlinux.conf file during boot. This is causing the system to enter a boot loop and reset repeatedly.

To resolve this issue, you can try the following steps:

  1. Verify the PARTUUID: Make sure that the PARTUUID you specified in the extlinux.conf file matches the actual PARTUUID of the target device’s NVMe drive. You can check the PARTUUID by running the command blkid on the target device.
  2. Update the extlinux.conf file: Update the extlinux.conf file to reflect the correct PARTUUID of the target device’s NVMe drive.
  3. Reflash the device: Reflash the device using the flash.sh script, making sure to specify the correct --rootfs option to point to the modified RootFS image.
  4. Verify the boot process: Verify that the system boots correctly and uses the new PARTUUID by checking the boot logs and the dmesg output.

Regarding the creation of a Golden Image, you can follow these steps:

  1. Backup the root file system: Backup the root file system from a configured Jetson Orin Nano device using the sudo ./tools/backup_restore/l4t_backup_restore.sh -b -e nvme0n1 -d jetson-orin-nano-devkit command.
  2. Modify the RootFS image: Modify the RootFS image to include any customizations, such as source code libraries, Python environment, CUDA, TensorRT, and system configuration.
  3. Create a new RootFS image: Create a new RootFS image by running the sudo ./tools/backup_restore/l4t_backup_restore.sh -r -e nvme0n1 -d jetson-orin-nano-devkit command.
  4. Flash the new RootFS image: Flash the new RootFS image onto multiple Jetson Orin Nano devices using the flash.sh script.

Note that you may need to modify the flash.sh script to accommodate any customizations you made to the RootFS image.

In comparison to the provided context, the l4t_backup_restore.sh script is used to backup and restore the root file system, while the flash.sh script is used to flash the device with the new RootFS image. The extlinux.conf file is used to specify the boot configuration, including the PARTUUID.

Additionally, the jetson-orin-nano-devkit.conf file is used to configure the device for flashing, and the flash_l4t_t234_nvme.xml file is used to specify the flashing configuration for the NVMe drive.

It is also worth noting that the nvautoflash.sh script is a wrapper script for flash.sh that can be called without specifying a device name, and the nvmassflashgen.sh script is used to enable simultaneous flashing to multiple target devices.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

4

Please let me know if the direction I followed is correct or not (compress the sample jetson and extract it to the rootfs folder, then run flash.sh. The command runs successfully but when I plug in the monitor, it doesn’t boot up. Is it because of the old UUID?)

5

Hi,

Please use this script instead, and ensure the firewall is disabled. Also, confirm that you are running on native Ubuntu rather than a virtual machine.

Thank you.

6

Hi,

As I shared above, I turned off the firewall, stopped udisks2.service, stopped docker, … Then ran sudo ./tools/backup_restore/l4t_backup_restore.sh -e nvme0n1 -b jetson-orin-nano-devkit but it always failed at step 3

7

Hi,

Some questions to confirm:

  • Is your ubuntu a native or a virtual machien?

Thanks

8

sorry for the lack of data, my machine is Ubuntu 22.04, not a virtual machine

9

Update: I used another jetson device and luckily it worked, the backup to another device also worked fine, thanks for the help