How to clone a TX2? Get a unit that won't boot after!

I follow this:

https://elinux.org/Jetson/TX2_Cloning

It works but my unit is missing a third-party DTB to enable the cameras. So I then flash the DTB via:

sudo ./flash.sh -r -k kernel-dtb jetson-tx2 mmcblk0p1

Which now results in a machine that won’t boot. Why?

EDIT: After reading another thread, could I instead backup the DTB partition from unit 1 and flash that into unit 2 provided they are identical hardware and release (28.2.1)?

Do beware that if you have a module which never arrived as a developer kit, then nothing at all is flashed to it and you need a full flash. The clone can be used in place of generating a new rootfs, but if you were to flash only a rootfs, then all of the boot content would be missing.

Beware that when you flash for a custom board, then the default content (the DTB you mention) will be incorrect for your carrier board unless you’ve put this in place prior to flash. Each carrier board manufacturer has their own instructions for DTB file setup.

If the DTB of one unit is also of the same release as what the second unit has been flashed with, then in theory this could work to make the second TX2 a duplicate of the first TX2. A requirement would be that all of the other boot content be in place.

I wonder if the TX2 needed a default flash first before I flash the backup system image (rootfs) and the dtb (kernel-dtb).

Note that I do:

sudo ./flash.sh -r -k kernel-dtb -G ~/backup_dtb.img jetson-tx2 mmcblk0p1

(and system.img is in the right spot)

It should flash both simultaneously which would do the trick provided they are both stock TX2 boards from NVidia and had an initial flash of the same release.

That would make sense to me.

I don’t know if it is possible to back up both a dtb and rootfs in the same clone. Something I myself am not clear on is if backup up a dtb partition via clone would back up the signing as well. It suddenly occurs to me that perhaps requirements for signing could cause a cloned dtb to be rejected…don’t know.

But I would definitely suggest that whatever release was used to flash the TX2 which was a clone source be used on the next TX2 in order to put all of the correct signed content on the new TX2. The only difference is that you could place a copy of the clone at “bootloader/system.img” and flash via this to do a full flash which uses the clone for rootfs:

sudo ./flash.sh -r jetson-tx2 mmcblk0p1 2>&1 | gawk '{gsub("[0-9][0-9]+[/][0-9[0-9]+ bytes sent..",".");print}' | tee log.txt

The result would be a fresh flash of everything, but instead of generating a rootfs from the sample rootfs it would use your clone. Any signing would be taken care of. I’ve added logging to the above command so you’ll know where each device tree file is from. The gawk reduces the progress bar logging (you may need to “sudo apt-get install gawk” if it isn’t already on your system). Any DTB flash could then be performed as a second step (make sure to restart the TX2 in recovery mode again before the second operation).

I guess then the procedure might be:

  • Flash a stock vanilla release from the clone target, i.e. if your clone is from R28.2.1 first flash a stock vanilla R28.2.1 (complete flash, everything)

  • Reboot in forced recovery mode

  • Copy your rootfs system.img into your Linux_for_Tegra flash environment

  • sudo ./flash.sh -r -k kernel-dtb jetson-tx2 mmcblk0p1

That will flash the backup of the DTB and system.img from the reference unit in one feel swoop.

Has anyone tried this?

I tried it and it failed but I DID NOT flash a stock vanilla release and I NOW know that the target TX2 was on a newer release initially than my backup which maybe why ultimately the DTB flash caused issues?

You would leave out the “-k kernel-dtb” when flashing the rootfs. You would add the “-k kernel-dtb” when flashing just that device tree.

The flash.sh option naming is a bit misleading. The “-r” says to “reuse” the “bootloader/system.img”, but what this really does is to “not regenerate bootloader/system.img” (if a system.img is still there this file will remain untouched…if no file is there, then none will be created). Whether or not the rootfs is flashed is not influenced by this option. The “-k kernel-dtb” is for flashing one device tree, but if you left off “-r” it would generate a new system.img even though it wouldn’t flash it (the “-k” says this is all you will flash). Device tree and rootfs flashes are separate, just make sure you have the device tree in place when you run “sudo ./flash.sh -r jetson-tx2 mmcblk0p1” and this will flash your device tree and the clone.

It is correct that having the wrong base release combined with some component of another release is often the cause for boot failure. There are some pieces you might be able to get away with keeping, but in terms of a full flash with mixed versions the failure rate will be very high.

Should you put your device tree in place in the flash software, and put your clone in place, then use only the “-r” option, you will do a complete flash with your clone as the rootfs.

The last sentence is exactly what I’m referring to. I’m trying it now.

The key here is to completely flash the same release as the reference system on the target FIRST before then flashing the rest.

I realize that could first install the new DTB on the system and then just flash the rootfs which I know works and would get me the equivalent result.

Flashing the reference system first works. However, you can save time and flash the full reference system using the clone rootfs at the same time. Any time you “sudo ./flash.sh -r jetson-tx2 mmcblk0p1” you have performed a full flash of every single partition…the “-r” has an effect only on whether the sample rootfs is used or the current “bootloader/system.img”. Such a flash replaces all content and not just the rootfs filesystem.

But it doesn’t…I don’t know why.

I back up the root filesystem on the reference system. I install this in a stock Linux_for_Tegra distro as well as copy in the third-party DTB.

I do a flash -r for everything and the target TX2 won’t boot. WTF?!

@NVidia: What is the official way to clone a TX2? Can we have a set of instructions once and for all?

We don’t suggest to try the command you started in #3. If cloning is working in #1, and then update dtb got failure. Could you share the boot log?

Sorry, you mean the boot.log of the working unit I’m trying to clone?

Yes, want to check the log that you gets failure based on case in #1.

You used the clone image to flash a new tx2 → successful
Then you update only the dtb file → fail and I want to check this log.

No that’s not what happened:

I backed up the rootfs of our reference system which had a base R28.2.1 image flashed with third-party DTB changes. I then coped that backup into bootloader/system.img (Replacing the stock one).

I then put a unit 2 in USB REcovery mode and did a complete reflash of that system, i.e. flash -r jetson-tx2 mmcblk etc from the same directory (which had the backed up image now in it as well as the third-party DTB). Unit 2 rebooted into a black screen which was highly unexpected.

I asking if what I did should work? I assume yes it should.

If the original system and destination system were both R28.2.1, then a complete flash would have occurred with the clone rootfs replacing the sample rootfs. If the source system of the rootfs had device tree changes, then those changes would still be missing, but if you had replaced the new flash dtb files with the same dtb which produced the working source system dtb, then this would have been a true and complete copy.

The method of cloning non-rootfs files may not be a true clone and could complicate things since those files are signed before flashing. Imagine if you’ve cloned a partition which was signed, and used this from the flash tool…a signed image would have been signed a second time and would not be valid. An exact copy of a partition with a device tree does not copy just the tree, it also copies any signature in the partition for releases which use signing.

Were there device tree changes needed for the source rootfs? If so, did you use the source system’s dtb, or did you use the source system’s signed (cloned) dtb?

Did not work. That is EXACTLY how I understand the process without doing a deeper dive into flash.sh.

I used the sourced system’s dtb (the one distributed by the vendor).

If they gave you the unsigned dtb (which is how it probably is sent…sending a signed dtb from a clone would be unusual), and you flashed like this, then it should work.

What does the dtb change do? Is this a devel carrier board with custom hardware (e.g., camera), or is the carrier board itself custom?

For the sake of debugging, if you flash a system without any clone, but do use the new dtb file, does the system still boot?

Devel carrier board for three CSI cameras (Leopard Imaging).

Yes, the system boots. That’s what has me so confused. I’m going to try with R32.1 now. I’m wondering if this is a flash.sh bug?

Here’s an oddball question: Do I have to ALWAYS run apply_binaries.sh before flash? I didn’t think so since I thought that was only used for stuffing ‘rootfs’ with the NVidia system “extras” that would already be in my system.img from the reference system.

When you run flash manually you need to run “sudo ./apply_binaries.sh” once after unpacking the sample rootfs. It doesn’t hurt to run this a second time, but unless the rootfs is replaced this isn’t necessary to run again.

In the case of a clone, if apply_binaries.sh was run before creating the system which has the clone, then the clone itself will already contain those files.

My suggestion (for debugging, not necessarily as an end solution) is that if you can flash with the modified device tree and simultaneously generate a new rootfs (versus using the clone), and if the system still boots, then this is when you want to add a step to install the clone rootfs using the same modified device tree which worked when not using the clone. If this still fails, then a serial console log of boot would be useful. Preferably save a serial console log of boot for the case when (a) the device tree is modified, but rootfs is freshly generated, and a second serial console log for the case when (b) both modified device tree and the old rootfs (from clone) are used.

Hmm, your steps sound no problem, but I don’t understand why you are running “-r -k kernel-dtb -G” in comment #1
Since there are some duplicated files here, please help confirm if my understanding is correct or not.

  1. Flash the board with original system.img (system.img.0) from jetpack with 3rd party dtb (3rd-dtb). System works fine.

  2. Use -r -k APP -G to back up it from unit 1 as system.img.1.

  3. Use sysmtem.img.1 for unit 2. → what is your flash command here? and where do you put 3rd-dtb and this system.img.1 ? The same folder where you flash unit 1?