Can a Jetson TX2 device flash itself?
Sub-question: Can it flash itself in the TX2 devkit?
Sub-question: Can it flash itself if a theoretical carrier board has the USB bus routed to itself?
Given the operation of recovery mode, I suspect the answer to all the above is no. Also because I haven’t been able to find any posts indicating it can be done.
The short answer is “no”. When a PC gets an operating system installed it has a BIOS/UEFI to aid as a uniform interface to the hardware…embedded systems do not have this, which complicates flashing. Instead, during boot, a temporary form of boot loader runs (fastboot in RAM customized for this device). The Jetson becomes a custom USB device and the PC has a driver which understands the device (fastboot has the ability to understand a USB serial interface, and custom “glue” goes between this and flashing). The driver itself is an x86_64 Linux executable.
I am very interested in the long answer, if it’s not also “no”.
I don’t believe the presence/absence of a BIOS makes flashing more complicated, except, if UEFI’s secure boot is in use you have to flash signed images instead of unsigned. It does look like Jetson TX2 modules use secure booting, just based on a quick read of flash.sh output.
Flashing is flashing: you overwrite the image that’s stored in non-volatile storage with a new image. PCs can flash their own BIOs. The Jetson TX2 should be able to overwrite all of its own non-volatile storage. However, the system currently is to invoke recovery mode, which only operates over USB, and replaces normal OS operation. (Recovery mode is running instead of Ubuntu.) I’m not sure what requiring recovery mode means for the Jetson TX2 system: are parts of the non-volatile storage not accessible from Ubuntu under normal operation? Is the signing complicating things in some manner that makes it not possible to flash everything from Ubuntu?
I would normally assume that flashing all of non-volatile storage is possible, except nVidia has chosen to make the recovery mode + flash.sh method the officially recommended method.
Jetson TX2s are marketed as an IoT device, see [url]https://devblogs.nvidia.com/jetson-tx2-delivers-twice-intelligence-edge/[/url] for example. IoT typically means remote deployment, in my specific case it means remote deployment with zero people. Therefore, remote firmware upgrades are required. In a product with one Jetson TX2 and nothing else, remote firmware upgrades must be accomplished via self-flashing.
That a Jetson TX2 can’t flash all of its non-volatile storage from Ubuntu makes it unsuitable as an IoT device component. Which makes the marketing something of an untruth.
Flashing a Jetson without an existing running operating system is only possible in recovery mode with the driver package talking to the custom USB device and the driver package runs only x86_64 Linux. Even with JTAG I do not believe you would be able to flash without recovery mode.
If you don’t mind a lack of safety, and you have a system running on alternate media, e.g., booted to the SD card, then you can play tricks with dd. However, such tricks get far more difficult than it sounds because there are other partitions and software involved beyond the rootfs. Perhaps changing the rootfs on one release to a different rootfs of the very same release wouldn’t be too difficult, but upgrading from say R27.1 to R28.1, or from R28.1 to R28.2 via dd gets extremely difficult and errors which can’t be recovered from without a real flash are likely.
When a desktop PC upgrades from a DVD it typically runs a temporary o/s in RAM and relies on BIOS-provided services for disk management (at least initially while setting up for the kernel load). Imagine if every single PC motherboard made had to have its own custom boot loader and if GRUB did not exist except for cases where the manufacturer built a version of GRUB for each and every model made. In many ways this is what U-Boot does for embedded systems…whatever configuration is used depends on specific hardware and its wiring which will differ for even slightly different models even if they use the same architecture.
So that’s the long answer…there is no true self-flash. A Jetson is good for edge computing, but may not be a good fit for IoT devices needing to self flash.
That’s taken care of during production. We don’t ship devices with no firmware. :-)
We’re investigating how much we can update from the TX2 itself via direct partition rewrites. It’s the only option. As you said, unsafe, but nVidia hasn’t provided appropriate tooling here.
Most/all IoT devices need to self-flash.
Thanks for the help.
Perhaps including an additional low cost single board computer into your solution will provide the self flash capability, e.g. X86 SBC running Ubuntu with the Jetpack installed for reflashing the Jetson and interconnected I/O to put the Jetson into the correct mode. The X86 SBC is responsible for keeping Jetpack updated as a background task. While this will increase cost and power consumption, it could be sleeping most of the time.
Regards, H
I appreciate the suggestion, and it’s a possibility for future products, although the trade-offs make it unlikely. Thanks.
However, that doesn’t provide Jetson-self-flash capability, which is what I need for the current product.
Actually there were times when self-flashing was a question of changing files on a system partition. You got a kernel, device tree, initrd, rootfs. All on single SD card. Easy to access. Easy to swap. Update and reboot, done.
But today? Forget about it. You are required to press the recovery button and use an X86-64 based computer, because the binary tools exists only for this platform. There are encrypted device tree blobs in the eMMC, without which the system wont boot, because each newer L4T release will flash the bootloader (which in above metioned terms is really equivalent to the bios/uefi), and provides a set abstraction to the actual system loaded. But no. Nvidia choose not to support full system loading from SD and requires that the DT portion is flashed onto the cpu module’s eMMC. Without any clear reasoning why to do that. But the proof that it does not need to be that way is in the earlier L4T releases. A deployed application does not need to flash its bootloader/BIOS, but should be possible to flash itself. And that is no longer possible as it was before.
I do not believe there is any technical obstacle to provide features the users of this platform want, but there is a seriously bad management of these products in terms of support and development. Many things are left out undocumented, many things are getting more and more obfuscated. Instead of providing a platform which can evolve and develop, we are fed with false promises, impossible to achieve features and buggy hardware. There are uncorrectable bugs which prevent the TX1 and TX2 to be used in our serious application and we are quite tired of working with such a black box.
I always wonder why manufacturers are unable to listen to their users and try to push out products which do not serve the market. If anybody reads this from NVidia, please make a change and start supporting the products properly. Open up the documentation, open up your secret bug/errata/workaround database. We want to use these products but the only one stopping us is unfortunately you, NVidia.
So far as I know the device tree is signed, which isn’t the same as encrypted. There is a question out on how to sign using open source tools…if that can be answered then dd of partitions will be practical again. I have not looked yet at the changes from R28.2 to R28.2.1, but I would guess that whatever major release comes next it’ll have more support for backup partitions which would make self flash safer (I see the dual list of partitions now, but I don’t think it is fully implemented with redundancy yet…but I could be wrong since I have not examined R28.2.1 yet).