Flashing the Xavier NX's eMMC without a Host PC

Hi everyone,

for a special application, we would like to flash the Xavier NX “in the field” where only a microcontroller is available which has its own flash where the image is stored.

AFAIK we need to use the TegraBoot for this, however, the Documentation doesn’t really disclose a lot of information about how the Bootloader exactly works (is there another Doc where this is described in more detail? Do we need to work through the flash.sh script?)

So generally we would like to know whether it is possible to flash the eMMC of the Xavier NX with a microcontroller?
We think it should work by sending the correct commands to the Jetson (probably via UART?) to set the Bootloader into TegraBoot mode and then provide the Image.
Is this possible?

Thanks for your help and time!

Hi,

  1. You can search similar problems on forum first. There have been similar posts before.

  2. I am not sure which part you want to flash here. We have OTA method to upgrade those components or you can also use the bl updater to upload them.

https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/bootloader_update.html#

Hi @WayneWWW,

I was looking for this the last few days but found nothing… I assume I just wasn’t looking for the right keywords. I’ll give it another go.

We can’t use OTA since we work with the assumption that we need to flash the Jetson when the Image is corrupted for whatever reason. So we want to flash a complete image (which is provided on some other memory) via a microcontroller onto the Xavier NX’s eMMC.

I hope this clarifies our problem

Thanks!
Clemens

Hi,

I believe currently there is no method for your scenario. If the image corrupted, there is A/B mechanism to give a backup mechanism. Maybe this can help your case.

https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/bootloader_update_agx_tx2.html#

1 Like

Hi,

thats a pitty, we would really need that. But maybe we can find a solution using the A/B mechanism! This sounds promising! thank you!

1 Like

Hi @WayneWWW

reviewing the A/B mechanism we were unsure of what kind of redundancy this offers.
Does the A/B redundancy offer a redundancy of just the Bootloader or the complete partition (including the bootloader and the image of the OS)
We are wondering about this because the base Image of the Xavier NX is 14GB in size and the eMMC is only 16GB so the A/B Redundancy with the complete image would just not work on the internal memory.

What we are looking for, is a mechanism to “flash”/“replace” the OS, assuming that the bootloader it self is not corupted.

There is no way for your request yet.

Also, dose your “OS” mean root file system?

Ah I see, yes RFS Redundancy would do the trick. That is also outlined in the A/B mechanism description.

We have a bunch of further questions regarding this. Would it be ok if I contact you via DM?

Hi,

Please directly ask your question here.

Hi,

ok thanks.

  1. The A/B mechanism (with enabled RFS redundancy) keeps the Bootloader and the RFS redundant on two partitions, is that correct? (question is just to see if I understand this correctly)

  2. Does the A/B mechanism also allow for more than double redundancy, thus becoming an A/B/C… mechanism?

  3. Assuming that the RFS is located on the eMMC, and is filling up more than half of the storage space, how would the mechanism handle the redundancy? (also see next question)

  4. Is it possible to setup the A/B mechanism utilizing multiple storage devices (e.g. A is on eMMC, B is on external SPI flash) NOTE: NVME is not an option, only SPI flash can be external storage for our configuration.

  5. Does the A/B mechanism attempt to repair the corrupted image if it detects that one slot seems to have a corrupted image?

These questions will get us started on some further points.
Thanks in advance for your help!

Hi,

  1. The rootfs redundancy is not yet a ready feature. The redundant bootloaders and redundant RFS are kept on 2 different slots

  2. Currently there is only 2 and user side is not able to extend it.

3,4 Since this is a feature that not yet ready, this is still under discussion.

  1. A/B mechanism will fallback to working slot in case corruption is detected on the other slot. But it is up to user side to implement on when to attempt to again update the corrupted slot.

Hi,

ok so if it’s not ready we won’t plan with RFS redundancy for now.

We have further questions:

  1. Is there a timeline on when RFS redundancy would be ready?

  2. For understanding again without RFS redundancy the A/B mechanism only keeps the Bootloader redundant, correct?

  3. Scenario: A/B is keeping the bootloaders redundant and at least one of them works. However, the RFS is corrupted. Is it possible for us to replace/repair the RFS with a working image just utilizing a µController?

I keep asking about different µController opportunities to really make sure we understand the possibilities and get confirmation whether these are possible or not.

  1. This would be ready in next release.

  2. Correct.

  3. We don’t have experience about µController, could you tell us what is that for?

I can’t disclose it completely, however, I can outline the following details which should describe the problem adequately:

We are using the Jetsons in an application where a corruption of the eMMC is likely due to environmental factors. To mitigate and handle this we want to implement a way of fixing potentially corrupted memory. However we will not be able to access the hardware. Due to power constraints we will not be able to provide a host PC that can flash the Jetsons. Therefore, we want to use a microcontroller (e.g. an Atmel or STM chip) that has its own memory where a boot image is stored. The microcontroller can be connected to the Jetson either via UART/SPI/I2C/CAN. The memory where a working copy of the system is stored is connected to the microcontroller. We want this chip to fix (=flash/overwrite) a corrupted RFS or idealy even the Bootloader, when it is instructed to do so.

We envision the microcontroller to send commands to the Jetson and setting its Bootloader into TegraBoot mode where flashing is possible, and then provide the image that shall be flashed. Which commands these should be is something we need to find out maybe you can help us with this?

Our assumption is that this should work since a host PC does nothing else but sending commands to the Jetson when attempting to flash it, so we think this should be possible. However, our intention was to ask whether this was done before or NVIDIA thinks this being possible or not.

I hope this explains our intentions better. Do you have further questions? Is anything of this possible?

FYI, the flash software itself, if flashing from the standard JetPack/SDKM (even if just command line) has binary executables which are only for desktop PC architecture (e.g., amd64/x86). Operations you might want to perform in recovery mode will mandate the flash software “driver package”, which in turn can only run from a Linux x86/amd64 PC.

Non-recovery mode operations can be performed by an external microcontroller, e.g., serial console or triggering restart in recovery mode. This might or might not be useful to you, and depends a lot on whether the original system is bootable. Directly running repairs (e.g., fsck) won’t work in recovery mode, and won’t be available since the eMMC cannot be directly operated on via an external microcontroller without the aid of a running Jetson.

Something I’ve been fascinated by is whether or not a “USB-to-internet” device exists which would forward a remote USB over the internet. There are many reasonably inexpensive devices which forward USB over a LAN (ethernet), and could possibly work, but I have not seen any which say they function over the internet (as opposed to a LAN…a LAN is much higher quality and performance compared to going over the internet to somewhere else in the world). Such a device, if it could also trigger power, reset, and recovery mode pins, and do so securely, would allow pretty much any Jetson to be flashed from anywhere in the world.

Thanks for the extensive answer!

Well, the USB to LAN solution will not work for us either because, while we can tell an external µController to perform certain tasks. There will be no internet connection available. I know this sounds weird. But there is now way that there will be any internet connection.

So we have to resort to a non µC solution if I understand you correctly because we need to run certain exectuables.

@linuxdev

well now that I think of it.
Just because the (or more like “a”) flash software exists and is written for x86 doesn’t mean that it cant be done via a µController. The flash software will, in the end, only send commands to the Jetson. So why can’t they not be sent by a µController? Is the Flash software so extremely extensive that it is basically impossible to realize a simple flasher in a µController?

My questions might sound repetitive, I just want to understand the problem at hand and see whether we really have no way around using another x86 PC (which we just cant)

You are correct, but the source code is not available to recompile to the other architecture. Combine this with the protocol not being public, and it becomes an enormous effort to do this without x86 hardware. Perhaps if your microcontroller were x86, e.g., an Intel Atom (or any lower power x86/amd64) running Linux, then it wouldn’t matter.

Sending commands with some other architecture, e.g., an arm64 microcontroller, is all good and fine, but only if you know what the commands are. You’d probably have to contact the NVIDIA sales team to get access to that protocol information (and if they were to release that information, then there would likely be NDA restrictions).

1 Like

Ah, see that was important to know. Thanks.
Well if none of this is open source there is no feasible way.
I think we will still try to contact them. Thanks for your help!

@WayneWWW
I assume that @linuxdev is correct about the fact that the source of the flash software is closed.
Is there any way that NVIDIA would provide this to us (maybe under NDA)?
Do you have the contact of a person or department, responsible for such requests?