How to create whole eMMC image?

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Is there a way to get flash.sh to create an image of what the entire eMMC contents should be? I see it creates system.img.raw, which is just the rootfs image. I would like the partition table and all the other partitions too.

I want to update a TX2 from R27 to R32 or newer. But the USB micro AB port is broken. In fact I have three Jetson TX2 boards. All have broken USB ports. It is not a robust design!

So I can not have flash.sh flash the board using the USB interface. But the boards boot, so I can use the software on them to flash them. I can boot them into Linux running on an SD card via U-Boot. Then I can flash eMMC from Linux while running from the SD card.

But there are 29 partitions! Surely there is some way to create a full eMMC image with partition tables, the rootfs, and the other 28 partitions.

2

Unfortunately, there are some partitions that are not able to be created as a file on host…
I think you may need a new carrier board.

3

Not what you were asking, but it is common for people using the micro-OTG port, and needing it to be a bit more rugged, to carefully use epoxy around the connector to help bond it to the carrier board. Once use of recovery mode is gone there isn’t much you can do in terms of flashing, and that connector is mandatory for recovery mode.

4

Why is that? I’m familiar with eMMC and everything on it can be duplicated in a file.

Are you referring to the two physical eMMC boot partitions or the RPMB partition on the eMMC? Those can still be flashed from u-boot or Linux, even if they aren’t part of the same flash image as the main area of the eMMC.

Or does the flash process using the host do some sort of encryption or signing of the data during the flash process, using the Tegra processor through the USB interface, which uses a secret only known to the Tegra, which means it can’t be done on the host without a connection to the processor to be flashed?

5

Those partitions are signed. I’m not sure where, but I believe unique information in the ROM is used to say if this is valid or not (this is a topic I’d personally be interested in hearing about…whether signing really needs to compare to a specific module or not). My thought is that signed content of one module will fail when used with a different module.

6

It seems easy to check. Just download the eMMC from two different TX2s running the same update and compare.

If the contents of one or more of the partitions (which ones?) is signed, is there a way to generate the signature from u-boot or Linux?

It is surprising that the only way to update the software is via an external device using USB. It means the board can not update itself in the field. Is this not extremely limiting? Even just for board development. I always create a self-hosted update system as one first phases of a new product, just so those doing hardware characterization can easily keep their software up to date.

7

I have only one TX2 (I have no way to test). There might also be a degree of compatibility until fuses are burned.

So far as I know every boot related partition (pretty much everything other than rootfs) is now signed (this was not always the case). The U-Boot binary would be signed, CBoot would be signed, any device trees in a partition would be signed, so on. The signing is normally performed on the host PC during the flash process. You can sign on command line on the host PC as well. If the fuse is burned, then you must know the correct information to sign, or else any flash will fail to boot.

I agree that this is somewhat limiting, but needs for security are driving some of those designs. The other part of this is that unlike a desktop PC, embedded systems do not have a BIOS/UEFI to present a uniform boot interface (and this is why GRUB cannot be ported to do this). All of those partitions are essentially a replacement for what is equivalent to a PC’s CMOS BIOS. With a hardware BIOS protection is also simpler (though not necessarily without its own flaws). “It gets complicated”.

8

I have gotten two of the TXs repaired, so hopefully I can reflash them tomorrow and can compare the two images.

I don’t think security is really a good answer as to why a TX2 can’t be flashed (supposedly… I remain unconvinced) from Linux or U-Boot. I have made secure systems which can update themselves. The update payload must be signed, the bootloader, kernel, rootfs, etc. must be signed, the contents to be flashed are decrypted and re-encrypted on the device using a device specific key, etc. It’s entirely possible for a device to update itself this way and I’ve made such systems. The device specific encryption makes it harder, but I don’t see evidence the Tegra is doing that.

I’m guessing unmodified TX2s are fused in a way that causes them to ignore signature failures. Or to allow booting unsigned images, which is more common in my experience, but maybe Tegra doesn’t do that. But either way, it is still entirely possible to construct an image of the entire eMMC with unsigned or signed-with-incorrect key data in it that will boot.

While embedded systems usually don’t use UEFI, and pretty much never the legacy PC BIOS, that’s again hardly an problem for getting them to updates themselves.

Every ARM-based SoC I’ve used has a built in bootloader in the SoC’s ROM. I think in Tegra this is MB1, but I’m not sure of that. While simple, they can do enough to load another bootloader from one (or more) of the storage interfaces present in the SoC. That’s all you need.

9

Here are my understanding and the reason I think it is not possible to work w/o flashing… Please point me out if anything wrong

First, you want to directly use some simple method like dd to write the partition from old rel-27 content to rel-32.
However, you have no existing TX2 that is already in rel-32, which means you cannot use dd to write out some binary files for your other TX2 devices.

Second, the only method remaining now is try to create each binary on host and copy it to device to do dd. Then the problem here is the official tool is not able to create such binary for every partition. Some partition cannot be generated through flash.sh.

And no need to mention the partition layout is changed between rel-27 and rel-32. Also, AFAIK, some partitions are not revealed as /dev/mmcblk*.

10

I’m aware I’d need to get flash.sh to create the binary to flash. That’s what I was asking. A way to do that. Seems it does not have that ability.

A change in partition layout is not a problem. An image of the entire eMMC would contain the partition table and all the partitions.

I’m not aware of any way to place data on an eMMC in a way that is not visible to Linux. Even such details as the ECSD register contents are available in sysfs. Is there some other non-volatile storage other than eMMC that must be updated?

I also do not see why flash.sh can not generate an image of what is to be flashed. Other than the use of a device specific secret key stored in the Tegra SoC to encrypt or sign, what possible thing could be done that can place data in eMMC on the target but could not place that same data into a file on the host?

12

If you flash on command line, then you can log (and see which files are signed, padded, flashed, and in which order). For example:
sudo ./flash.sh jetson-tx2 mmcblk0p1 2>&1 | tee log_flash.txt

Examine the logs and take note of which partition content is signed and used and the order of adding these. You will find that the rootfs is generated, but the other partitions already exist and may be copied in to a different location as they are signed (leaving the original unmodified). To see some candidate preexisting partition images:
find /where/ever/it/is/Linux_for_Tegra/bootloader -name "*.bin"

Images may also need to be padded with NULL bytes on the end to match a partition size (directly or indirectly).

Part of the disk content will be from the partitioning command (metadata), and not from an actual partition. The rest will be the padded/signed images being concatenated. If your disk layout is already correct, then a concatenated dd of the padded/signed partitions in the correct order would probably do what you want. I’m not sure if metadata from a different partition scheme could be easily “synthesized” on the host PC (loopback understands block devices, but not necessarily how to emulate an SSD or eMMC for non-partition content…having layout already correct avoids the issue).

Note that if you have a running Jetson, and you create a file on your host PC via dd of the entire “/dev/mmcblk0”, then you could in fact use this with dd to write the entire Jetson. You might need to extract pieces of the image (perhaps also with dd), remove the signature and padding, sign this, pad this, and overwrite those same bytes in the larger image with the new signed partition data (basically all of this is to sign the content of individual non-rootfs partitions). What I still do not know is if signed content on one Jetson can be used on another Jetson (perhaps it can if fuses are not set). However, I cannot test this, I only have one TX2.

As far as self flashing goes (actual flash, not copying images), the flash software is x86, and not arm64. Now if for some reason NVIDIA were to release this content as arm64, then you could flash from another Jetson. However, that Jetson being flashed would still need to be in recovery mode, and thus the target could not be simultaneously running the flash software. Any self-contained solution would need a fully running system adapted to work without being in recovery mode. Thus, a recovery mode in combination with releasing the flash tools for arm64 would work and be simplest for the end user, but it wouldn’t be easy for NVIDIA.

13

Metadata from a different partition layout can be easily made on the host. Standard GPT partition tools, e.g. sgdisk, can create the MBR and GPT tables in a (sparse) file image. Such an image can be made with truncate. Or one could just write code in Python, it is very easy. There is absolutely no need to use loopback devices or root access to do this.

Placing the individual partitions in such a whole flash image is also trivial. It can be done with dd using seek and conv=notrunc options, or trivial python code. Even better, when using mke2fs, use the offset option to place the sparse ext[234]fs image into the correct location directly without the need to create a system.img file first.

I do not believe it is the case that there are partitions that can not be created on the host, unless there is some encryption that I do not know about. What encryption I see so far is done host side.

It is true that eMMC is not just one image. Usually eMMC has multiple areas: user area, two boot areas, and a RPMB area. Normally people only know about user area, so maybe the use of the boot area is where this notion, apparently false, that an image of eMMC can not be made has come from. The boot areas appear in Linux as /dev/mmcblk0boot[01] and can also be accessed in U-Boot.

I am also highly confident, based on the logs and also knowing in detail how this works on other systems, that one does not really flash eMMC via the USB recovery interface. The USB recovery interface is more limited and only supports uploading data into the processor’s address space, i.e. sram or dram, and programming registers. So the USB interface is used to load in a bootloader and run it. Then that bootloader uses USB to communicate with the host to load and flash data.

So this idea that one can only write to all of eMMC via USB recovery is nonsense. eMMC is always being written by software running on the CPU. It just uses the USB2 port, which fragile, slow, and hard to use remotely, and can only communicate with proprietary software on a host running as root that is also hard to use remotely. It could easily be more capable software, like Linux, that does this flashing, using the more robust, faster, and easier to use remotely gigabit ethernet or wifi interfaces and with far more possibilities in how the process is controlled.

The boot area has 18 partitions. This includes the mb1 and mb2 bootloaders. There is also a secondary GPT at the end, even though it does not appear to have a primary GPT or the protective MBR. Maybe this is some kind of proprietary nvidia modified GPT partitioning system. 13 of these partitions are encrypted. The two mb1 partitions appear to use a different encryption format than the other 11 encrypted partitions.

The user area has 33 partitions. Plus the MBR, GPT and 2nd GPT. Of these 5 are not flashed (fbname, recovery, recovery backup, recrootfs, and uda). Most are encrypted. The kernel-bootctrl, bmp, and APP are not. I bet one could hack the system with a fuzzed splash image, since that data is not signed or encrypted. The sc7 warmboot partitions appear to use a different format than the other encrypted partitions.

I haven’t yet compared two TX2s to see what differences there are. It does not look like device specific encryption is being used.

14

I do not know if the open source tools for working with a sparse image will flash properly, but they might. In the past I’ve not found open source tools compatible with doing the reverse (I’ve not been able to take a sparse clone and read it with open source sparse image tools). It would be useful to know if this works out in testing.

Recovery mode (which may have changed over time), when used with the flash software, typically downloads a micro operating environment. I don’t know about now, but in the past this was a combination of fastboot and a “3p server” (these only go into RAM). The actual content and names will have changed since then, but I’ll still call it this in what follows (the binary content uploaded in a flash would still have that same functionality). That content had to be uploaded into the Jetson by the flash software before flash begins, and I have no knowledge of what is in that particular environment other than it being some sort of micro operating system capable of read/write of the eMMC. Flash software talks to this “micro o/s environment”, and it is the micro o/s environment which flashes the eMMC. If not for this upload of the micro o/s prior to flash recovery mode would not be able to flash. Basically what you just said, but fastboot is not the only software uploaded to the recovery mode Jetson.

In older releases the flash software supported read/write of individual partitions, offsets, or the entire disk. The TK1 was the last device/hardware which supported this. After moving beyond the R21.x release (which was for the 32-bit platforms) to R27.x or newer (which is for 64-bit platforms) the flash software lost the ability to do detailed read/write at arbitrary locations (at least on command line…I have no doubt the internal API can still do so, but the user interface to this no longer included that ability at R27.x+).

Even now people can write individual partitions, but the content of those partitions has strong dependencies to the rootfs content. It just happens that when flashing a custom rootfs it is best to also flash the non-rootfs content, but you could get away with not flashing the non-rootfs if that content is compatible. In every case the content must now be signed correctly, and this is the best way to make sure both dependencies and signing requirements are met. For most people this non-rootfs content won’t change anything.

Unfortunately, there is no ability to upload a bootloader or other content into memory via ethernet unless the operating system is running. The “dream come true” would be if recovery mode had the ability to flash over the ethernet after getting through some security confirmation steps. Since this does not exist, if the system is not already up and running, one must have physical access for USB. Should an ssh session dd update fail and the system become unbootable, any fix would require travel to the physical location to repair the install for a dd-based flash.

What you mentioned about signing on the local host PC is true, but the complications and risks mean you’d need to be very confident that this won’t fail before attempting it. I agree that it would be outstanding and very useful if ethernet could be used in place of USB, or even if a self-contained thumb drive could be used for flash without a host PC, but at this time it isn’t possible without significant changes. You’d need to have ROM based content to replace the fastboot and 3p server upload (the name of this content is probably changed in newer releases, but without some sort of micro o/s environment the eMMC cannot be accessed in recovery mode).

As far as I know the actual content of the other partitions is not encrypted, and is only signed. Jetsons seem to have hardware methods to validate the signing, which is much simpler than actual encryption/decryption. I don’t know if the splash image touches memory the signing covers, but I would be surprised if an altered splash image would be given the opportunity to attempt load (meaning that even if the image could be used as malware it seems likely a failed signature would stop the image from ever loading…or for that matter, even a valid different logo would likely be refused). The acid test is to try this out on a system with the burned fuses enforcing the signature (which I do not have). Without burning the fuses I am thinking some aspects of signing may be ignored.

Many of the complications have come out because the NVIDIA embedded devices have been around a very long time and have inherited from the past. The ARM trusted boot system is new compared to the first Tegra devices, and is basically added on to rather than being created from scratch. I have no doubt that if everything were developed again from scratch the whole maintenance lifecycle would be simplified, but current hardware is what is being supported.