How to make image snapshot of AGX Orin and restore it?

Cloning is probably what you want, but I should first describe what is going on.

Jetsons do not have a BIOS. You’ll find the equivalent is done in software, and on Jetsons with eMMC (and AGX Orin is such a model), that means in various partitions. The actual operating system is the “rootfs” (or label “APP”) partition. All other partitions are related to either the equivalent of what a BIOS would do, or else to booting. When you customize a Jetson, most people are in need of backup and restore of only the rootfs/APP partition (someone who customizes UEFI might need more).

When you flash a Jetson the content in every partition other than the rootfs/APP is more or less standard. If you use the same L4T release for flashing each time (you can find that release with “head -n 1 /etc/nv_tegra_release”; JetPack/SDK Manager is a GUI front end for flashing L4T, which is in turn Ubuntu plus NVIDIA drivers), then you actually want those partitions to stay constant. A given rootfs/APP partition must be compatible with the other content, and if you flash using the same release of L4T, plus your clone of rootfs/APP which came from that release originally, then it should work each time. This probably won’t contain modifications to UEFI software or special setup.

The lack of BIOS is also why bootable Linux distributions don’t usually work. This is of course also an aarch64/arm64 architecture, so live distributions, even if adjusted for Jetson boot, would not work unless they are the correct architecture. The short answer is that you won’t find a live bootable distribution which succeeds on a Jetson unless someone specifically designed it for that. You need a Linux host PC for cloning or flashing (consider that during a flash, unlike a desktop PC, you are essentially also flashing the CMOS BIOS).

In recovery mode a Jetson becomes a custom USB device. This does not change the Jetson unless the flash software runs. This same mode is also used for cloning to the desktop PC. For an Orin with the current L4T R35.x you would want to use an Ubuntu 20 host PC, typically not a VM. A VM can be made to work, but you have to correctly configure for USB which disconnects and reconnects during the flash (a typical VM will fail during reconnect if not configured correctly, or if you get lucky). In reality, it is the JetPack/SDK Manager software which has most of the limitations on which version of Linux works for flash. The real software behind the flash is known as the “driver package”, and this runs on command line in many more cases than does JetPack/SDKM. JetPack/SDKM has some advantages though, so it is recommended in most cases.

If you’ve installed JetPack on your host PC, it will produce a directory:
~/nvidia/nvidia_sdk/JetPack...version.../Linux_for_Tegra/

This location is where the driver package lives and where flash commands are run from. The version should be from the same L4T version as on the system. L4T and JetPack versions are tied together, so if you’ve picked one, then you’ve picked the other. You can find the L4T and JetPack version here, and both URLs go to the same web page:

The rootfs/APP partition is a very large file when cloned or generated. If your AGX Orin has a 28 GB partition, then the “raw” clone is also 28 GB. There is also a “sparse” clone. The sparse clone is essentially the content which actually exists on the partition, and so if there is 3 GB of content, then it would be 3 GB, but as the partition fills the sparse partition approaches the size of the raw partition. A clone gives you both as files. This means that if you have a 28 GB partition, and it is nearly full, then your host PC will be saving twice as much (56 GB in the example). Any copy or move of that file (if crossing devices) is a very long process. So before you clone be careful to have a lot of spare disk space.

The recovery button on a Jetson works like a shift key on a keyboard when trying to get a capital letter. You hold the recovery button down, and then tap the key it modifies, and let go. The power on and power reset buttons are what the recovery button modifies. So to go into recovery mode you hold down the recovery button, and then either power up or tap the power reset button, and let go of the recovery button. Then the correct USB port will turn into a custom device, and you connect that to the host PC. From then on anything you do with the flash software will work, but I have to warn you that you cannot perform two operations in a row without resetting recovery mode (for example, you cannot clone then flash, nor can you clone twice if you did not reset recovery mode in-between operations).

On command line this would clone from a recovery mode Jetson using the same flash software release as that which produced the clone (actually, you can clone from any release, but restore should use a correct release):
sudo ./flash.sh -r -k APP -G my_backup.img jetson jetson-agx-orin-devkit mmcblk0p1

What you’ll get are my_backup.img, a sparse clone, and my_backup.img.raw, a raw clone. A sparse image can be created from a raw image, and I tend to delete the sparse image. If renamed to system.img, and placed in “Linux_for_Tegra/bootloader/” while flashing with the correct command, either image will flash (the sparse image is faster because it is smaller). However, the raw image can be loopback mounted, examined, edited, so on.

An example to see what is on a raw image:

sudo mount -o loop my_backup.img.raw /mnt
cd /mnt/etc
head -n 1 nv_tegra_release
cd -
sudo umount /mnt

During loopback mount, if you’ve updated your Jetson and want to update the clone without a full clone, then you can use rsync over ssh to your host PC and update the clone that way. Or you can mount the clone and use it as a sysroot in a user space cross compile. If you were to transition to an L4T release not compatible with that rootfs, you could still copy the “/home”, or any non-release-dependent content from the clone.

When not in use I use “bzip2 -9” to compress the raw clone. This takes a very long time, but saves a lot of space (it’s still huge and takes a long time to move or copy across devices; be careful to not overwrite this since it takes so much time).

Also, Jetsons use an ext4 filesystem. FAT32 tends to be a Windows thing, although it is used for an EFI partition when using UEFI boot (you’ll see it under lsblk -f as vfat). ext4 is a vast improvement over NTFS, and NTFS is a vast improvement over VFAT, but the simplicity of VFAT (meaning subject to corruption without a journal) makes it useful in small and efficient partitions, especially if used mostly read-only (and EFI partitions are normally written once and never again touched).

Incidentally, dd is the right idea, as it creates a partition which is exactly the same as the raw clone. However, dd does this on a running system, which implies temporary files, cache, logs, so on, might also be duplicated. The user who is logged in will look to be already logged in from a dd clone. A recovery mode clone shows content in a properly shut down state (if you shut down without correct procedure, then the clone will also save the corruption created from the improper shutdown…don’t yank the power, use software to shutdown).

If you want to restore your Jetson with a clone, then perform the same recovery mode setup, copy the image to “Linux_for_Tegra/bootloader/system.img” (use that file name), and do this with the same L4T release:
sudo ./flash.sh -r jetson jetson-agx-orin-devkit mmcblk0p1

Notice that this still uses the “-r” option, “reuse image”. The “-r” is used during clone to stop image creation since no image is actually useful when not flashing. If you leave out the “-r”, then system.img will be overwritten and lost.

There are other interesting ways to use a clone as well, this barely touches on its uses.

1 Like