Clone entire TX2

Also:

nvidia@tegra-ubuntu:~$ cat /proc/mounts
/dev/root / ext4 rw,relatime,data=ordered 0 0
devtmpfs /dev devtmpfs rw,relatime,size=7973052k,nr_inodes=1993263,mode=755 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/debug cgroup rw,nosuid,nodev,noexec,relatime,debug 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
tmpfs /run/user/1001 tmpfs rw,nosuid,nodev,relatime,size=803988k,mode=700,uid=1001,gid=1001 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
gvfsd-fuse /run/user/1001/gvfs fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1001,group_id=1001 0 0
nvidia@tegra-ubuntu:~$

You should be able to clone everything in the “Name” column of that “gdisk -l” output. You may want to save a copy of this for future reference as well since it shows offsets and size…if you were to manually reconstruct something at a later date it might come in handy.

In R28.1 of a TX2 the mmcblk0p15 “kernel-dtb” seems to be the device tree where both U-Boot and the Linux kernel look, but I don’t know if other stages in boot prior to U-Boot might look at device tree in other directories.

Most of what you see in “/proc/mounts” involves pseudo file systems which don’t exist except in RAM. The ones you get with “grep ext4 /proc/mounts” are the ones with files you can back up. If you clone the “APP” (rootfs) partition I believe this will include all real file system mounts unless you’ve customized something (such as mounting a different file system type from a custom eMMC partition).

when i’m using :

sudo ./flash.sh -r jetson-tx2 mmcblk0p1

the image always was overwritten by the original image, how i can make sure that i’m flashing my backup !

anyone can help me please ?

Thanks
Ebraheem

The command “sudo ./flash.sh -r jetson-tx2 mmcblk0p1” will flash without generating a “new” rootfs…the file currently existing in “bootloader/system.img” will be the source. If this file is not valid or is missing, then the flash program probably won’t tell you…it just copies bytes present. This command does flash brand new all of the other support partitions, e.g., the boot loader.

You must create a clone with the Jetson in recovery mode. If R28.1, then the command is something like:

sudo ./flash.sh -r -k APP -G backup.img jetson-tx2 mmcblk0p1

…which would produce files “backup.img” and “backup.img.raw”. A raw image is uncompressed and quite useful for editing and examining, and can be used directly as a replacement to system.img (flash would take significantly longer). The non-raw image can also replace system.img…you can’t edit or decompress this (neither Linux nor Android sparse file system tools will work on it), but the flash is much faster.

Note that clones and system.img.raw (any raw file) is extremely large. Check disk space with something like “df -H -t ext4” before flashing. A single raw image can easily approach 30GB. A sparse image depends on file system content, but a basic sparse image is typically another 2GB.

1 Like

We are working on jetson tx2 28.2.1 and we followed above steps to clone the image. Clone was successful and system.img created and that we replaced in “/bootloader”.

So with following command:

sudo ./flash.sh -r -k APP -G backup.img jetson-tx2 mmcblk0p1

It flashes successfully and previous cloned data retains when we flash cloned image.

But we want to add some logs using printk(), in Linux driver files and want these modified changes should reflect when we flash with cloned image, but we are not able to get those changes. So what flash commands or steps needs to be followed so that these changes reflected with cloned image.

The clone process would have produced both “backup.img” and “backup.img.raw”. The “raw” image is the full image without any kind of pseudo compression. This “raw” image is the one you can edit, but it will be much larger and slower to work with. If you don’t use the raw image you can’t edit since the sparse image is not compatible with any available tools. So long as the image is renamed “system.img” before flashing to restore the flash tools won’t care if the base image is raw or sparse…raw just takes longer.

The raw image can be loopback mounted. I’m going to assume you named it “backup.img.raw”. You can mount loopback devices anywhere, but I’ll use the traditional “/mnt”.

First a comment on “losetup”, the command for loopback: If you use “losetup --find”, then the command will name the first unused loop device. If the device does not yet exist, then the device will not be created unless you use sudo. Some commands both find and create…take note of when you need sudo…query does not require sudo, but create does.

For example:

# Find first unused device and create:
sudo losetup --find
# Cover the file and echo the name...create if needed, use existing if not needed:
sudo losetup --show /where/ever/the/file/is/backup.img.raw /mnt
# Now browse "/mnt" as if it were "/" of the image.
# Perhaps edit content somewhere, e.g., add a README.txt file mentioning edits you've added.
# Now be sure nothing is still holding "/mnt" and detach:
cd
sudo umount /mnt
# Alternative umount if loop0 is the device:
sudo umount /dev/loop0
# Now detach all unused loop devices just for safety...don't want a loop device leak:
sudo losetup -d /dev/loop0
# Now do your flash of clone in usual manner after putting "backup.img.raw" in the right place as name "system.img"

Incidentally, if you were to update that system with apt or add something, then you could rsync to the loopback mounted raw file to update it as well without a full clone.

Almost forgot, but very important: The kernel itself may be named via extlinux.conf in “/boot”, but newer releases tend to add components to partitions when those components are needed prior to mounting the rootfs. As such you will need to specifically find out if the kernel is actually used from the rootfs on the version of L4T you use, or if the flash tool is needed to place it in a partition. For L4T version see “head -n 1 /etc/nv_tegra_release”. It doesn’t hurt to try to see if a clone updates the kernel, but if it doesn’t, then probably the kernel isn’t the one in “/boot” and you need to investigate if it is instead in a partition.

I have upgraded to JetPack 4.2 via SDK Manager on ubuntu 18.04 because old linux versions as host systems via VM weren’t very stable for me.

SDK Manager doc says there is no CLI for Jetson devices. Does anybody know how to clone an entire TX2 with the SDK Manager toolchain on a 18.04 host?

I have not yet tried cloning via SDK Manager, but do know that the SDK Manager itself does not do flashes…like JetPack, it is a front end. The driver package does the actual flash and any cloning. Somewhere within your SDK Manager directory will be a “Linux_for_Tegra/” subdirectory, and within that “flash.sh”. You can try that with the TX2 in recovery mode and the micro-B USB cable connected.

For details see:
[url]https://devtalk.nvidia.com/default/topic/1028867/jetson-tx2/clone-entire-tx2/post/5234090/#5234090[/url]

Thank you, it worked, on R32.

For reference: I used the backup method as descriobed here: Jetson/TX2 Cloning - eLinux.org
And then I symlinked my compressed backup.img as bootloader/system.img and used the following command on a blank TX2:

sudo ./flash.sh -r jetson-tx2 mmcblk0p1

I used these instructions to get a clone of my existing Tx1, and it works flawlessly, with one exception.

Eventhough it clones the Tx1 rootfs, it’s not a “whole” or “master” image. It doesn’t clone/flash the kernel, device tree and uboot with these commands.

Are there any command to both “clone” and “flash” the tx1 completely(including dt and kernel as well)?
This is still useful to clone the rootfs, but I need to flash the kernel and devicetree everytime after. Then configure u-boot env. variables.

There is no “total disk” clone/flash using recovery mode (there used to be back in the days of the TK1). Life is now complicated by signing of partitions, and so there are still tools for preserving partitions, but those tools require a running system. If the process is not correct, then the Jetson won’t boot and will need to be flashed from scratch.

If you wish to clone all of the regular partitions you can use “dd”. You wouldn’t want to use dd to clone the rootfs while it is being used, but none of the other partitions have any issue with this. If you were booted to an SD card (and thus not using the rootfs on eMMC), then this would copy the entire eMMC:

sudo dd if=/dev/mmcblk0 of=clone.bin bs=512

Even if you use the rootfs and boot normally you could clone the other non-APP (non-rootfs) partitions one at a time, e.g.:

sudo dd if=/dev/mmcblk0p2 of=p2.bin bs=512

Restoring is a different matter. You shouldn’t mix releases of rootfs and different other partitions, but if you have booted system you could for example restore a non-rootfs partition like this example:

sudo dd if=p2.bin of=/dev/mmcblk0p2 bs=512

Now in theory, if you had booted a Jetson to an SD card to avoid using the rootfs from eMMC, and had cloned with “sudo dd if=/dev/mmcblk0 of=clone.bin bs=512”, then you could restore:

sudo dd if=clone.bin of=/dev/mmcblk0 bs=512

Any failure means you have to flash fresh since the system must boot before dd can be used.

I tried to use the tutorial from https://elinux.org/Jetson/TX2_Cloning to perform cloning of Jetson TX2.

$ sudo ./flash.sh -r -k APP -G backup.img jetson-tx2 mmcblk0p1
$ sudo cp backup.img.raw bootloader/system.img
$ sudo ./flash.sh -r -k APP jetson-tx2 mmcblk0p1

Unfortunately, it didn’t produce the “whole” or “master” image, since it didn’t clone some part of the system, .ie device tree (/dev) and etc as stated by hburaksaruhan. So is there any option to copy the “whole” system of Jetson including its device tree etc as an image? Because when I tried your solution:

sudo dd if=/dev/mmcblk0 of=clone.bin bs=512

it still not copied the “whole” system. I would appreciate any ideas.

That article is for cloning the root filesystem image, not the entire disk. Consider as background that a PC motherboard would have a BIOS, and it is the BIOS which makes it possible for all of the different motherboards to use basically one uniform interface to install an operating system from. Embedded systems do not have a BIOS, and thus that content must be custom for every board. In the case of Jetsons, this is in the form of various partitions in addition to the rootfs. If you were to clone everything, then this is more or less like backing up not only the disk of a PC, but also the BIOS itself. Trying to flash with just the disk, when the BIOS is empty or for another board, will cause failures.

In the past those other partitions were not signed, and you could simply save those partitions through mechanisms such as “dd”. Because of signing you should expect to have to flash those partitions (even if you use a rootfs clone) using the designated tools. If the signature is not correct, then the boot partitions will be rejected.

So long as the release the clone is taken from is used to flash the clone you should be ok and it should work. As soon as you fail either the signing of partitions, or if the version of those other partitions were not from the same release as the rootfs, I would expect failure. There are minor patch releases where I would expect the same rootfs to work across, but those are the exceptions and not the common case.

Thank you for the explanation.

In that case, if I have two Jetson TX2, A and B. If I want to clone, I can only do that by cloning A and restore it to A using dd. But if I want to clone from A and restore it to B alias ghosting, it will be impossible since it has a different signed partition? Is this the correct explanation? Because, when I tried to dd clone from A restore it to B, I got this message:

EXT4-fs error (device mmcblk0p1): ext4_dx_csum_verify:448: inode #131775: comm gdm-session-wor: dir seems corrupt? Run e2fsck -D.
EXT4-fs error (device mmcblk0p1): dx_probe:743: inode #131775: comm gdm-session-wor: Directory index failed checksum
EXT4-fs error (device mmcblk0p1): ext4_iget:4591: inode #140795: comm 90gpg-agent: checksum invalid

The partitions you copied via dd were signed before flashing. The signature will be invalid on a different Jetson. This is why it fails. You can clone the rootfs and use that clone on another Jetson of the same model and release.

So the next question becomes one of whether you have customized the non-rootfs partitions? If not, then a normal flash using the clone will correctly place those other partitions. If it turns out that you have customizations in those other partitions, then you will need to replace the correct image in the flash software (an unsigned image), and then flash with both the alternate non-rootfs images and the rootfs image (which would sign the non-rootfs content during flash).

Is there a particular reason you need to clone the non-rootfs content?

Thank you again for the response.

I have two Jetson TX2s, A and B, with the same model and release. [bold]Both Jetsons[/bold] at the beginning has the capability of [bold]reading analog cameras[/bold] (which its device kernels are inside the /dev right?). Just in case something goes wrong, we want to have a clone of a Jetson with the same capability of reading analog camera. So, I tried to flash [bold]reset Jetson B[/bold] (using SDK Manager) and tried to [bold]clone from A[/bold], so both can have the same capability to read the analog cameras.

These are what I tried to perform cloning (from Host):

$ sudo ./flash.sh -r -k APP -G backup.img jetson-tx2 mmcblk0p1
$ sudo cp backup.img.raw bootloader/system.img
$ sudo ./flash.sh -r -k APP jetson-tx2 mmcblk0p1

But that one didn’t copy the /dev, instead only the APP rootfs.

So, I tried to perform DD (inside Jetson A):

$ sudo dd if=/dev/mmcblk0 of=clone.bin bs=512

And used this (inside Jetson B):

$ sudo dd if=clone.bin of=/dev/mmcblk0 bs=512

But, I got this message:

EXT4-fs error (device mmcblk0p1): ext4_dx_csum_verify:448: inode #131775: comm gdm-session-wor: dir seems corrupt? Run e2fsck -D.
EXT4-fs error (device mmcblk0p1): dx_probe:743: inode #131775: comm gdm-session-wor: Directory index failed checksum
EXT4-fs error (device mmcblk0p1): ext4_iget:4591: inode #140795: comm 90gpg-agent: checksum invalid

For now, I still haven’t met my goal yet. I’m still wondering [bold]what’s the correct procedure to perform clone[/bold], at least for the device kernel, and if possible the APP rootfs. And I have another question, so are these statements correct?

  1. DD could perform clone on Jetson TK1, but not on Jetson TX2, because Jetson TX2 has signed partition.
  2. There is no way to perform clone for the whole system (I mean all the partitions - not including BIOS, like the ones listed in sudo gdisk -l).

For now, I will try to use these:

$ sudo ./flash.sh -r -k kernel-dtb -G backup.img jetson-tx2 mmcblk0p28
$ sudo cp backup.img bootloader/system.img
$ sudo ./flash.sh -r -k kernel-dtb jetson-tx2 mmcblk0p28

The other thing that I will try is these:

$ sudo dd if=/dev/mmcblk0p28 of=clone28.bin bs=512
$ sudo dd if=clone28.bin of=/dev/mmcblk0p28 bs=512

Best regards,
Jeff

  1. Correct, TK1 was not signed. Technically you could copy the signed partition of other releases, but the content would be useless for restoring to a different Jetson.

  2. Correct in the sense that there is no way to perform a useful clone of the non-rootfs partitions. Note though that all of those non-rootfs partitions are part of boot, and yet ignored after boot loads the Linux kernel. Although I phrased that those “other” partitions are used because of a lack of a BIOS, they are intermingled in function and none are specifically a “BIOS”. The “gdisk -l” could show binary partitions, but even PC boot uses more than individual partitions (a PC boot disk will have content not visible as a partition, and may use other partitions during boot, but partitions do not contain everything used for PC boot). Either way, other than customization of boot properties, the rootfs will contain the entire operating system. The only part which is “normally” modified by end users would be the kernel or device tree.

If kernel and device tree are combined into the flash software (the “driver package”) on the PC, then any flash will contain those updates. If you also replace the system.img with a clone of the rootfs, then you will correctly flash all of these modifications to the next Jetson you flash. The process will properly sign partitions. If you didn’t modify the kernel or device tree, then you will get 100% cloning by simply telling the flash to reuse the rootfs which you provided from the clone (caveat: the JetPack/SDKM release should match the release which was used on the system you cloned).

Notes:

Beware that you cannot clone (at least in a useful way) the non-rootfs partitions. This is the job of the “driver package” (which JetPack/SDK Manager downloads and is a front end to). While flashing the rootfs from the correct version of JetPack/SDKM/L4T those other partitions will be correctly added (the rootfs can be from a clone).

So far as the rootfs goes you definitely don’t want to clone all files since not all files are actual files. Many files, including those in “/dev”, “/sys”, and “/proc”, do not exist on the disk/eMMC. Those files are actually drivers in RAM programmed to have a file interface.

As an example, consider if a naive file backup program were to create a file named “/backup/dev/random”, and tried to back up by coping all content inside of the real “/dev/random” to “/backup/dev/random”. The file will produce data until the end of time. Similar for “/dev/zero”. Backing up files requires a special tool, e.g., “rsync”, which understands files versus pseudo-files.

Note that if you dd a corrupt copy of a partition, then you will get a corrupt copy. If you use dd on a filesystem while it is changing and being used, then most likely you will get a corrupt (or at least incorrect) clone. If temp files or anything else changes on the filesystem while you use dd to copy it, expect errors. The clone process gets around this (if the filesystem is not corrupt) by not using the rootfs in recovery mode (there is a mini-operating system with minimal function while in recovery mode, and that operating system does not depend on eMMC).

The process of flashing provides a fresh install of all of those non-rootfs partitions. So long as you use the same release you can consider this a restore to the same version you had originally. Boot content modifications, such as a changed kernel and device tree, would need to be considered separately.

This process you mentioned is correct if you want the pristine non-rootfs partitions along with a clone of the original rootfs (warning, I amended this since rootfs is mmcblk0p1…I failed to read this closely enough):

$ sudo ./flash.sh -r <s>-k kernel-dtb</s><b>-k APP</b> -G backup.img jetson-tx2 <s>mmcblk0p28</s>mmcblk0<b>p1</b>
$ sudo cp backup.img bootloader/system.img
$ sudo ./flash.sh -r <s>-k kernel-dtb</s> jetson-tx2 <s>mmcblk0p28</s>mmcblk0<b>p1</b>

EDIT: The kernel-dtb can be flashed as originally named, but won’t be cloned in that manner since it is signed. One would add the unsigned dtb to the right place in the “Linux_for_Tegra/” tree of the driver package on the host PC prior to flash, or you would flash the modified (not cloned) device tree as you first mentioned. The point is that you can flash the dtb using the flash.sh tools, but you cannot clone in a useful way with those tools. Only the rootfs can be cloned in a useful way since it is not signed. If you want more information on device trees though, just ask.

Note: Clone will also produce “backup.img.raw”. This too could be used in place of “bootloader/system.img”. The result will be the same, except that the “backup.img.raw” is much larger and is very useful for examining, modifying, so on (the smaller file is of use because it flashes faster and takes less disk space on the host PC…when it expands the two are an exact match).

Note: You can fsck.ext4 the loopback mounted backup.img.raw. The “backup.img” can only be used for flash. You can create a new “backup.img” from the “backup.img.raw” using the “bootloader/mksparse” program (with NULL byte fill pattern, the default of mksparse).

Hi, linuxdev. Thank you again for the great and detailed explanation.

Best regards,
Jeff

Btw, I did just make some edits to #27. The part with “…if you want the pristine non-rootfs partitions…” I edited since I was only thinking of rootfs and not kernel-dtb at the time I did the copy/paste. The gist is that you can add a modified kernel-dtb the way you mentioned, but cannot necessarily use a cloned copy of kernel-dtb on a different Jetson since it is signed. FYI, “system.img” is always the rootfs and can only be replaced by a clone of a rootfs, never a clone from a kernel-dtb.