How to make image snapshot of AGX Orin and restore it?

Hi, I own two Jetson AGX Orin.
I constantly install stuff and modify files so I need a quick way to make a full disk image of one AGX Orin, so I can restore it on the other one. (Installing everything from scratch using the Nvidia SDK takes too much time).

I’m looking for an official solution (command or tool) from Nvidia.

I’ve tried several solutions, but no one worked so far.

  1. I tried to boot a live ARM64 Linux Image from a USB pen drive and use the dd command to make a full image file of the disk. I tried to boot live Ubuntu 20.04.6, Ubuntu 22.04.3, Ubuntu Fedora Workstation 64-38-1.6, and Manjaro minimal-generic 23.02. But none of them resulted to boot. (tried both Rufus and Ventoy)

  2. I tried to make an image snapshot of the running distro using the Distroshare-Ubuntu-Imager .sh script but I got an error. This guy opened an issue on Github but I don’t see Fat32 partitions on my AGX Orin, so I don’t know how to solve it.

  3. I didn’t try RemasterSys or SystemBack, which basically do the same. Could they work?

Thanks in advance.

Cloning is probably what you want, but I should first describe what is going on.

Jetsons do not have a BIOS. You’ll find the equivalent is done in software, and on Jetsons with eMMC (and AGX Orin is such a model), that means in various partitions. The actual operating system is the “rootfs” (or label “APP”) partition. All other partitions are related to either the equivalent of what a BIOS would do, or else to booting. When you customize a Jetson, most people are in need of backup and restore of only the rootfs/APP partition (someone who customizes UEFI might need more).

When you flash a Jetson the content in every partition other than the rootfs/APP is more or less standard. If you use the same L4T release for flashing each time (you can find that release with “head -n 1 /etc/nv_tegra_release”; JetPack/SDK Manager is a GUI front end for flashing L4T, which is in turn Ubuntu plus NVIDIA drivers), then you actually want those partitions to stay constant. A given rootfs/APP partition must be compatible with the other content, and if you flash using the same release of L4T, plus your clone of rootfs/APP which came from that release originally, then it should work each time. This probably won’t contain modifications to UEFI software or special setup.

The lack of BIOS is also why bootable Linux distributions don’t usually work. This is of course also an aarch64/arm64 architecture, so live distributions, even if adjusted for Jetson boot, would not work unless they are the correct architecture. The short answer is that you won’t find a live bootable distribution which succeeds on a Jetson unless someone specifically designed it for that. You need a Linux host PC for cloning or flashing (consider that during a flash, unlike a desktop PC, you are essentially also flashing the CMOS BIOS).

In recovery mode a Jetson becomes a custom USB device. This does not change the Jetson unless the flash software runs. This same mode is also used for cloning to the desktop PC. For an Orin with the current L4T R35.x you would want to use an Ubuntu 20 host PC, typically not a VM. A VM can be made to work, but you have to correctly configure for USB which disconnects and reconnects during the flash (a typical VM will fail during reconnect if not configured correctly, or if you get lucky). In reality, it is the JetPack/SDK Manager software which has most of the limitations on which version of Linux works for flash. The real software behind the flash is known as the “driver package”, and this runs on command line in many more cases than does JetPack/SDKM. JetPack/SDKM has some advantages though, so it is recommended in most cases.

If you’ve installed JetPack on your host PC, it will produce a directory:
~/nvidia/nvidia_sdk/JetPack...version.../Linux_for_Tegra/

This location is where the driver package lives and where flash commands are run from. The version should be from the same L4T version as on the system. L4T and JetPack versions are tied together, so if you’ve picked one, then you’ve picked the other. You can find the L4T and JetPack version here, and both URLs go to the same web page:

The rootfs/APP partition is a very large file when cloned or generated. If your AGX Orin has a 28 GB partition, then the “raw” clone is also 28 GB. There is also a “sparse” clone. The sparse clone is essentially the content which actually exists on the partition, and so if there is 3 GB of content, then it would be 3 GB, but as the partition fills the sparse partition approaches the size of the raw partition. A clone gives you both as files. This means that if you have a 28 GB partition, and it is nearly full, then your host PC will be saving twice as much (56 GB in the example). Any copy or move of that file (if crossing devices) is a very long process. So before you clone be careful to have a lot of spare disk space.

The recovery button on a Jetson works like a shift key on a keyboard when trying to get a capital letter. You hold the recovery button down, and then tap the key it modifies, and let go. The power on and power reset buttons are what the recovery button modifies. So to go into recovery mode you hold down the recovery button, and then either power up or tap the power reset button, and let go of the recovery button. Then the correct USB port will turn into a custom device, and you connect that to the host PC. From then on anything you do with the flash software will work, but I have to warn you that you cannot perform two operations in a row without resetting recovery mode (for example, you cannot clone then flash, nor can you clone twice if you did not reset recovery mode in-between operations).

On command line this would clone from a recovery mode Jetson using the same flash software release as that which produced the clone (actually, you can clone from any release, but restore should use a correct release):
sudo ./flash.sh -r -k APP -G my_backup.img jetson jetson-agx-orin-devkit mmcblk0p1

What you’ll get are my_backup.img, a sparse clone, and my_backup.img.raw, a raw clone. A sparse image can be created from a raw image, and I tend to delete the sparse image. If renamed to system.img, and placed in “Linux_for_Tegra/bootloader/” while flashing with the correct command, either image will flash (the sparse image is faster because it is smaller). However, the raw image can be loopback mounted, examined, edited, so on.

An example to see what is on a raw image:

sudo mount -o loop my_backup.img.raw /mnt
cd /mnt/etc
head -n 1 nv_tegra_release
cd -
sudo umount /mnt

During loopback mount, if you’ve updated your Jetson and want to update the clone without a full clone, then you can use rsync over ssh to your host PC and update the clone that way. Or you can mount the clone and use it as a sysroot in a user space cross compile. If you were to transition to an L4T release not compatible with that rootfs, you could still copy the “/home”, or any non-release-dependent content from the clone.

When not in use I use “bzip2 -9” to compress the raw clone. This takes a very long time, but saves a lot of space (it’s still huge and takes a long time to move or copy across devices; be careful to not overwrite this since it takes so much time).

Also, Jetsons use an ext4 filesystem. FAT32 tends to be a Windows thing, although it is used for an EFI partition when using UEFI boot (you’ll see it under lsblk -f as vfat). ext4 is a vast improvement over NTFS, and NTFS is a vast improvement over VFAT, but the simplicity of VFAT (meaning subject to corruption without a journal) makes it useful in small and efficient partitions, especially if used mostly read-only (and EFI partitions are normally written once and never again touched).

Incidentally, dd is the right idea, as it creates a partition which is exactly the same as the raw clone. However, dd does this on a running system, which implies temporary files, cache, logs, so on, might also be duplicated. The user who is logged in will look to be already logged in from a dd clone. A recovery mode clone shows content in a properly shut down state (if you shut down without correct procedure, then the clone will also save the corruption created from the improper shutdown…don’t yank the power, use software to shutdown).

If you want to restore your Jetson with a clone, then perform the same recovery mode setup, copy the image to “Linux_for_Tegra/bootloader/system.img” (use that file name), and do this with the same L4T release:
sudo ./flash.sh -r jetson jetson-agx-orin-devkit mmcblk0p1

Notice that this still uses the “-r” option, “reuse image”. The “-r” is used during clone to stop image creation since no image is actually useful when not flashing. If you leave out the “-r”, then system.img will be overwritten and lost.

There are other interesting ways to use a clone as well, this barely touches on its uses.

2 Likes

Many thanks for the detailed reply!
Since I don’t have too much space on my host PC, I adapted the commands to generate the image directly on my external hard drive and then flash from that dir.
The image generation process is OK, but the flashing process fails.
I also repeated the flashing process after copying the “system.img” file into the “flash.sh” directory, but it still failed with this error:

Error return value 3
Command tegradevflash_v2 --pt flash.xml.bin --create
Failed flashing t186ref

These are the commands I gave

  1. “cd” to the jetpack dir

cd /home/user/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/

  1. Backup image

sudo ./flash.sh -r -k APP -G /media/user/externalHDD/new/my_backup_agx.img jetson jetson-agx-orin-devkit mmcblk0p1

  1. renamed the image to “system.img”

  2. flash the image

sudo ./flash.sh -r /media/user/externalHDD/my_backup_agx.img jetson jetson-agx-orin-devkit mmcblk0p1

Not in any particular order:

  • The flash.sh script accesses a lot of other content, so you cannot use that by itself.
  • If you use an external drive, then you should be sure to format it as ext4. Don’t use a Windows style filesystem, they have no ability to correctly work with Linux filesystem permissions.
  • There are ways to mount your external drive to “~/nvidia/nvidia_sdk/JetPack_...version.../Linux_for_Tegra/” rather than moving the software to the disk, OR:
  • There are ways to install NVIDIA flash software to an independent location (typically not for use with SDK Manager, though I’m sure you could do that, but command line on a different location is simple).
  • Flash of course requires the Jetson to be in recovery mode with the correct USB cable attached.
  • After renaming the backup image to become “system.img”, it needs to be placed in the “Linux_for_Tegra/bootloader/” directory for the case of using it to flash. If in the “Linux_for_Tegra/” directory, the system.img file will not be visible for “flash.sh -r” operations.

It’s probably worthwhile explaining some options for use with flashing and using an external device (always assuming the device partition is formatted as type ext4).


Option 1:

JetPack/SDK Manager is just a frontend to the actual flash software. If you go to the URL for the release you are interested in, then you’ll find you can download the “driver package” and “sample root filesystem” manually. Regardless of where your external drive is mounted you could manually unpack the driver package there as a regular user. This produces the “Linux_for_Tegra/” directory, and aside from the “Linux_for_Tegra/rootfs/”, everything there is from the driver package. This is the flash software.

You could then unpack the sample root filesystem within “Linux_for_Tegra/rootfs/with sudo (as root). This is the purely Ubuntu content. To complete the sample root filesystem install, you’d go to “Linux_for_Tegra/” and run this command:
sudo ./apply_binaries.sh

Optionally, if you want to set up your user login name and pass before flashing, you could also run:
sudo ./tools/l4t_create_default_user.sh

With the above, if you exclude using the GUI JetPack for flashing, you could now flash and clone from your external drive regardless of where it is mounted. This would tend to be portable (at least on command line) to several distributions and releases of Linux.


Option 2:

You could take your installation which already exists on the host PC’s “Linux_for_Tegra/”, and use rsync to clone this to your external drive which is mounted somewhere else at the moment. You’d probably first delete the temporary content, which consists of:

  • bootloader/system.img
  • bootloader/system.img.raw
    (you’re going to replace that content each time you flash without “-r”, and since you are restoring a clone, you’re going to have to replace it anyway)

Then you’d mount that partition at:
~/nvidia/nvidia_sdk/JetPack...version.../Linux_for_Tegra/

While the external drive is mounted there the actual original disk content is hidden and invisible. All system.img and other temp content goes on the disk which is actually mounted. **Option 1:**In theory you could remove the original disk content, or leave it there as backup for when the external disk is not mounted.


It is JetPack and SDK Manager which prefer the “~/nvidia/nvidia_sdk/JetPack...version.../” directory prefix. JetPack/SDKM also have some other reserved locations, such as within the Downloads/ directory, but these are all front end aids to the actual flash software. The driver package is the flash software, JetPack is the GUI layer which uses the driver package, and SDKM is a “smart” network layer used with JetPack to aid automatic download and install of JetPack, the driver package, and the sample rootfs. As soon as you work on command line you no longer care about JetPack/SDKM so long as you don’t want automated help.

Incidentally, there is actually another reason to use JetPack. The option to install CUDA and some other software is via JetPack using ssh to the Jetson after flash completes. You can manually use the apt mechanism to install those things as well if you know what they are. In the case of a clone, if that software was already on the cloned system at the time of cloning, then you don’t need this anyway because the software is already installed as soon as the clone is installed.

1 Like

Hi @linuxdev
Thank you so much for the explanations in Options 1 and 2 for flashing the image stored on an external HDD!

In the end, to avoid other possible errors, I decided to free up some space on my HDD’s PC and then flash the system.img file from the /bootloader dir.
(To gain some extra space I deleted an old system.img.raw file in /bootloader, now GParted says that my PC HDD has 22Gbs of free space. (Should be more since the file I deleted with rm was about 55Gb and I don’t see anything in the Ubuntu recycle bin folder)… Anyways,

I still get this:

[450.9917] writing partition APP with system.img [46366421632 bytes]
[450.9972][ ] 000%

the process hangs at 000% for many minutes and then I get this:

Error return value 3
Command tegradevflash_v2 --pt flash.xml.bin --create
Failed flashing t186ref

Also, why does the terminal show 46366421632 bytesif the image I’m trying to flash is 43.2 Gb in file explorer?

Thanks again in advance for any help

I hate to ask, but do you have the exact size of the original system.img.raw? Or do you still have access to the system the clone is from? If the latter, then you could run “blockdev --getsize64 /dev/mmcblk0p1”. I ask because I can think of some possibilities:

  • If you a VM for the host PC, then this causes some USB errors.
  • If the partition is not the same exact size as the default, then it might need to have the size specified. This is done with the “-S <size>” option. It might be stated in MiB or GiB. If the size is evenly divisible by 1024 twice, then that is the size in MiB; if divisible three times by 1024, then this is the size in GiB. So let’s say for example that it works out to 28GiB. In that case it would be something similar to:
    sudo ./flash.sh -r -S 28GiB jetson-agx-orin-devkit mmcblk0p1
    (the sparse file will not give the true size; the raw file or the actual partition will give the true size)

I noticed that your command is attempting 46366421632 bytes, which is not evenly divisible by 1024. I think on Orin you might get be able to just specify bytes, e.g., “-S 46366421632”, but I’m not sure if that works or not. It’s pretty much guaranteed though that if it is evenly divisible by 1024, then it would work. Thus I’m hoping you can find the exact size first. If not, then you might just try “-r -S 46366421632” and see what happens.

Incidentally, I never use GUI tools to delete files. I hate when it uses some scheme to allow undelete. You might just go to “Linux_for_Tegra/bootloader/” and “ls system.img*” and see what is there. Anything you do to look at byte sizes should be on command line so you can actually get the true byte size.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.