How to upgrade a TX2

I would like to manage a full update of my TX2 remotely.
I’m using Jetpack 3.3.

I would like to update kernel and device tree direclty from the running system for the next reboot.

So I can’t use ./flash script and can’t go to recovery mode.

Does it Exist a mean to do that?

The only official way is in recovery mode. There is no “safe”, and no “easy” way to do this.

Note that “dd” can be used to replace partitions on a running system if the new content is the correct size (meaning smaller replacement partitions can be NULL padded on the end to the right size, and the dd copied…but larger partitions would require moving the entire content so it isn’t practical). The rootfs itself cannot be “dd” updated on a running system. This is complicated by the need to sign some of the content (if the signature isn’t valid, then boot will not proceed).

Can you give exact details of what your physical access is and what kind of connectivity you might have? Is this the dev carrier board? Is this a release update or just a few files? Knowing exact details might make it possible to see a workaround.

Thanks for your detailed answer.
I will try to give more details in oreder to have a more precise answer.

  • the TX2 is embedded in an OEM board.
  • The TX2 is on a network with many others
  • It has no internet connection. If is a really hardened point we can plug it onto the web during update.
  • A remote connection using a compuuter on the same network can push an update package over STFP or FTP.
  • The force recovery button is not accessible.
  • I would like to update kernel, device tree and rootfs.
  • All modifications are from myself (recompiled kernel, updated device-tree).
  • The rootfs update is not mandatory because I know it’s more difficult and I can run scripts to update the rootfs directly from itself (adding new packages, updating conf files, etc.)
  • I also can reboot on a update dedicated kernel to apply update adn reboot on the updated kernel after if there is something like that.
  • I have already flash the kernel using the dd command and it worked but when I do the same thing with the device tree it fails; I used the command
sudo dd if=tegra186-quill-p3310-1000-c03-00-base.dtb of=/dev/mmcblk0p15
  • I have a SSH connexion available
  • At the end, I would like to push update using SFTP and start the update using my REST interface.
    thanks for your answers.

First, beware that if you do anything which doesn’t allow boot, and you don’t have physical access for the recovery button, then it is pretty much “game over” and you’ll have to physically recover the Jetson.

For whichever partition you might update with dd, the release version of L4T must match. If you update a rootfs, then the release used for the rootfs must match the release used for all of the other partitions, and vice versa.

In more recent releases (past R28.1) the device tree must first be signed. See:
https://devtalk.nvidia.com/default/topic/1035668/jetson-tx2/how-to-disable-dtb-encryption-of-l4t-28-2/post/5295198/#5295198

I do not know if the same signature works on all Jetsons. Would someone from NVIDIA be able to state whether a device tree signed via the above URL’s information allows dd copy to any TX2 which is using the same L4T release version? Is any part of that signing tied to a specific TX2?

Does your situation make it possible for another computer to provide serial console access? If it does, then you have more choices on kernel update with safety. For example, if you have an SD card slot, then you can put a rescue system in this, and then have this boot only if serial console is used to select the alternate entry. This would still be dependent upon the “/boot” directory of the eMMC, but it would give you a chance to fully boot without mounting eMMC; this, in turn, would allow you to dd copy a new rootfs in place. This isn’t entirely useful though since you need to use the same release.

On the other hand, if you are booted to an SD card, then you could copy an entire dd image of the eMMC directly to the eMMC of the given system. Instead of a partition at a time, you’d dd the entire mmcblk0.

There are things to look out for when doing a dd image copy of either the rootfs (mmcblk0p1) or the entire eMMC (mmcblk0). For example, if the MAC address of the ethernet were put in a udev entry for naming an ethernet, then it would not work on the next Jetson because the MAC address would differ. You might want to provide different passwords, and a dd of the password/shadow/group files would clone those and not use the original passwords.

Keep in mind that a rootfs can also be updated with rsync like any normal backup/restore operation. Having another computer available for this will do the job. Having an SD card for this will do the job, but since you don’t have physical access, then you could consider iSCSI mount of a remote system (this is not a simple thing to set up…I would normally suggest that route for higher performance over NFS, but it isn’t without a learning curve).

From the above, can you narrow in on what might be the direction you are interested in? For example, is a rescue SD card out of the question? Do you have more than one local TX2 to test creating a signed device tree to dd to both and validate that they can use the same device tree?

First, thanks for your huge and detailed answer !

I will answer to your questions first :

  • I have a SD port but I can’t access to it
  • I can’t use tty serial communication with my board.
  • The only link I have is Ethernet.
In more recent releases (past R28.1) the device tree must first be signed

Thanks for the information, it could explain why my dd command haven’t work. I must make a try

I do not know if the same signature works on all Jetsons

I have got many TX2 so I can make a try for that.

Keep in mind that a rootfs can also be updated with rsync like any normal backup/restore operation.

I’m completely agree with that, that’s why the update of the rootfs is not mandatory at all.

For the kernel there is no special encryption signature or something else ?

The kernel is still the “/boot/Image” file. Even if you don’t have serial console, should you use a new kernel, you might consider adding an extlinux.conf entry and adding an alternate Image name. Should things need to be physically accessed later on, then it would be no more than picking a valid entry and editing that file to fix (you would have required physical access, but the fix without this is a clone/edit/restore or flash operation).

There is some kernel partition, and I can’t say I know for sure how that partition is used (I believe it is being worked on for use with signing, but not yet actually used…I could be wrong). U-Boot will look for the eMMC’s “/boot/extlinux/extlinux.conf”, and this in turn points at “/boot/Image”. Be very very certain you have the correct “uname -r” reporting after a boot (the CONFIG_LOCALVERSION during kernel config). Then try testing on another TX2 (you could in fact purposely change CONFIG_LOCALVERSION to convince yourself that a simple file copy of Image will do the trick).

Ok thanks a lot for those information, I will make some tries.

Hello,
I have made some tries.
Here are may results :

  • I flash from scratch a TX2.
  • I rebuild a new DTS
  • I sign / encrypt the DTB file
  • Using dd I update the DTB onto the partition
  • The board boots sucessfully !
  • I try to flash another board with the same DTB, the board doesn’t boot anymore :(

I must check for command line parameters to manage correctly signatures to keep them and don’t regenerate at each build. Someone has already done that ?

Was everything on the other TX2 an exact match? This includes the L4T release (see “head -n 1 /etc/nv_tegra_release”…if they were both flashed with the same JetPack, then L4T release would be the same) and carrier board model.

FYI, I have only one TX2, and so I can’t test if signing one file allows copy to multiple TX2s. Someone from NVIDIA will need to comment on whether signing has any dependence upon specific hardware, e.g., if signing is in the context of some serial number reported in recovery mode.

It was the same L4T release but I had rebuild the kernel and other stuffs between two flashes.
I think when you use the flash command if you don’t provide key file used last time, the flash tool re-generate new one each time.

When you use the command “sign” it re-use the last signatures

I would be interested if anyone from NVIDIA would be able to expand knowledge on how signatures might or might not work or fail when passed between two different systems. I personally have no way to experiment with this.

Me too !!

This might be of interest: