Boot Jetson TX2 from SD Card - require to know config changes

Hi All,
The NVIDIA Jetson TX2 Module datasheet mentions the Boot sources for the board as Internal eMMC and USB(Reset recovery).

Is there a way in which the Jetson TX2 board can be made to boot from an SD Card which has the boot image? Any config change needed to be done for this?

Please advise.

Thanks
M

The easiest way…the way which I prefer since it leaves the eMMC available for rescue…is to chain load. On the eMMC “/boot/extlinux/extlinux.conf” the APPEND key/value pair offers the command line the Linux kernel uses. In that the “root=” determines where Linux will find its root partition (prior to that U-Boot determines where to look for extlinux.conf separately from where Linux will look).

“mmcblk0p1” is the first partition of eMMC; “mmcblk1p1” is the first partition of the SD card. If “root=mmcblk1p1”, then Linux will look at SD card instead of eMMC (U-Boot will still require eMMC first partition to have extlinux.conf, and editing files in “/boot/” of SD will have no effect).

When manually flashing on command line you would normally use a command such as this:

sudo ./flash.sh -S 29318MiB jetson-tx2 mmcblk0p1

…if you instead name “mmcblk1p1” instead of “mmcblk0p1”, then U-Boot will look for extlinux.conf on SD card instead of eMMC. Note that there are differences in boot with a missing SD card when extlinux.conf is on eMMC versus when it is on the SD card, which is why I prefer to flash normally and then edit “root=” (versus flashing to mmcblk1p1). In this latter scheme of flashing to mmcblk1p1 any edits to extlinux.conf on the eMMC will be ignored…the inverse of changing only the “root=”.

During a manual command line flash you normally unpack the sample rootfs, and then run “sudo ./apply_binaries.sh”. If you use the “sudo ./apply_binaries.sh -r /some/where/else”, then the binaries go to the alternate location. You might use your PC to mount an SD card there, unpack sample rootfs onto it, and the apply_binaries.sh…this will give you a complete file system on the SD card which can then be used directly on the Jetson. Be sure the SD card is using ext4 on the first partition (GPT partitions are used, e.g., from gdisk). If for some reason your host uses 64-bit ext4 extensions you may need to mkfs.ext4 using options to disable those extensions.

If you use a serial console you can make multiple boot entries and pick which one you want at boot time. You can also drop into the U-Boot environment, run “help” to see commands, and examine some of the variables which are boot macros used to determine boot device and order. Typing “boot” would continue to a normal boot.

you may also use the kangalows jetsonhacks examples: http://www.jetsonhacks.com/2017/01/26/run-jetson-tx1-sd-card/
http://www.jetsonhacks.com/2017/03/31/install-samsung-ssd-nvidia-jetson-tx2/

As I understand it the Jetsonhacks example is only about the Root filesystem, and it isn’t about loading the kernel or anything else. All that will still be loaded from the eMMC.

Outside of flashing directly from a host machine I don’t see anyway of accomplishing doing a complete boot from an SD card (u-boot, kernel, rootfs). Unless doing a drive image from the eMMC to the SD card works, but it doesn’t seem like it would.

I’m a little baffled why it isn’t pretty standard to boot completely from the SD card where the eMMC is left alone as recovery/test. At least for development purposes because the Jetpack versions constantly evolves. That way the new jetpack version can be tested before committing to it.

It would also allow people to test out complete SD images that someone else creates. Like where I work we shipped TX2’s to multiple sites to have people test things out with. I should be able to save a compressed drive image on a network somewhere that any site with the TX2 can test out what’s on the drive image. Without having any concern about what’s on the eMMC. As everything is booted from the SD card.

When your flash target is mmcblk0p1 extlinux (and “/boot” files) are read from eMMC. When the flash target is mmcblk1p1, then extlinux (and “/boot” files) are read from SD card. U-Boot binary and some of its environment are stored in hidden partitions of eMMC regardless of root partition location.

To see partitions try this:

sudo gdisk -l /dev/mmcblk0
# OR:
sudo blkid

There is a possibility of cloning or writing individual partitions. The main caveat is that your partition data must fit in the size of the current partition size if writing…I don’t think there is any guarantee the same size partition will be valid across releases. Second, not all U-Boot or hidden partition content will be compatible across releases. So clone and restore within a single release version (such as R28.1) should be fine, but mixing releases (such as R27.1, R24.2.1, and R28.1) will not necessarily work.

On a prior release for the TK1 there was an alias for the entire eMMC, “all”. It would be nice if newer releases were to add this ability back in (subtle feature request hint! :). However, each individual image concatenated in the correct order is the same as “all”. When cloning on a TK1 with “all” you can use dd at the right offsets and get a subset of the image which corresponds exactly to cloning just that one partition…you can go back and forth between concatenating individual partitions or taking subsets of multiple partitions and be guaranteed of exactly the same results.

If you want to put all hidden partitions on SD card I don’t believe this is possible. You could of course put those partitions on SD card, and while booted to SD card, use those partition for dd replication onto eMMC…those partitions though would not be active until you reboot since the SD card hidden partitions would basically be just data and not participate in the boot process.

Testing a root partition (APP partition) from SD card is easy, just change the extlinux.conf on eMMC after flash to mmcblk0p1 to “root=/dev/mmcblk1p1”, or if using SD card boot partition, then changing to this on the SD card version of extlinux.conf (the extlinux.conf on SD card could be named during flash, yet have it point at eMMC instead of itself…the corollary of being on eMMC and pointing at SD card is interchangeable).

Additional note: U-Boot environment variables can be set for a search order. One can say search first for extlinux.conf on eMMC, then other devices; or you could alter the variables to search first on SD card, and if not there, only then boot to eMMC version. Much of the boot behavior is customizable from the serial console U-Boot shell.

Here is what I did.

I followed the jetson hacks guide to create the rootfs on the SD card. This didn’t really give me what I wanted because as soon as I stuck the SD card into a new TX2 it didn’t boot. Which was expected, but I did it just to verify.

So then I flashed from the host to the TX2 (in recovery mode) to mmcblk1p1. That didn’t actually seem to flash to the SD card at all. In fact that same rootfs that was on there was still there, along with no other partitions. I did this before I saw your comment, and your comment explains this.

To be perfectly clear is it possible to boot COMPLETELY from the SD card? Where I can actually delete the eMMC?

Is this R28.1 (check with “head -n 1 /etc/nv_tegra_release”)? Does the SD card have “/boot/extlinux/extlinux.conf”, and if so, what is its content?

The answer to COMPLETELY booting from SD is no, it cannot. The boot loader and environment remain on eMMC regardless of which media the boot loader looks to for further configuration. Flashing to mmcblk1p1 changes extlinux.conf and whether it points to eMMC’s version of extlinux.conf or whether it points to the SD card’s extlinux.conf. Boot loader binary remains on eMMC. Only the root partition of eMMC can be deleted.

Correct, it’s R28.1 and the extlinux.conf in the SD card was already setup to boot off the SD.

Thanks for the clarification on the booting from the SD. That confirms what this document says in that you can’t put u-boot anywhere, but in the eMMC.

http://www.tsuremote.net/Karekinada/jetson/nvl4t_docs/nvl4t_docs/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/l4t_uboot_guide.html

I’m not sure why they made that decision because if the eMMC ever becomes corrupt there is no way to boot the unit with exception to the USB recovery mode. That’s fine on the development system, but not production systems.

So did you sucessfully boot Jetson TX2 R28.1 from external SD card om the end?

I am trying to use flash.sh file but it doesn’t work…

Flash to alternate boot targets works, but beware you’ll need to set up the rootfs on the SD card separately. Flash to eMMC puts the rootfs in place, but flash to a non-eMMC target simply places the boot loader setup such that it will look at the SD card when the time is right. No rootfs on the SD card implies it can’t boot. Have you looked to see if the first partition of this device has an ext4 mountable partition on it?

Yes, actually I am trying to flash it onto an alternative usb drive. And the first partition is ext4.

But how to setup rootfs on my external drive?

I followed the guide: http://developer2.download.nvidia.com/embedded/L4T/r28_Release_v1.0/Docs/Jetson_TX1_and_TX2_Developer_Kits_User_Guide.pdf?-ZK6wEk9kUv-s-WkEraresdATr5Y_8vlia5bYdOqS9lQbfdYHZZQFvsmwxM2Wa9gvSUqH2ua3Fxo661-d4s2b8aLIvvMDsQT4F0YFa4OFIpPh-YHxItZehR2l2Uhq1776-Xf7R62oiDk527IvEvVQvgnWeqFCbTDX2hHZHJSFmDFAKIuUzxgMKoHfcKVMXsYWf

And I have folders required (/bootloader, /kernel, /rootfs, /nv_tegra) in the repository: JetPack3.1/64_TX1/Linux_for_Tegra_64_tx1

total 132K
drwxr-xr-x  6 yerong yerong 4.0K Dec 24 00:45 ./
drwxrwxr-x  3 yerong yerong 4.0K Dec 23 23:55 ../
-rwxr-xr-x  1 yerong yerong  18K Jul 20 02:45 apply_binaries.sh*
drwxr-xr-x  3 yerong yerong 4.0K Dec 24 01:40 bootloader/
-rw-r--r--  1 yerong yerong 2.7K Jul 20 02:45 e2220-1170.conf
-rwxr-xr-x  1 yerong yerong 2.5K Jul 20 02:45 elf-get-entry.py*
-rwxr-xr-x  1 yerong yerong  51K Jul 20 02:45 flash.sh*
lrwxrwxrwx  1 yerong yerong   22 Jul 20 02:45 jetson-tx1.conf -> p2371-2180-devkit.conf
drwxr-xr-x  3 yerong yerong 4.0K Jul 20 02:46 kernel/
drwxr-xr-x  3 yerong yerong 4.0K Jul 20 02:46 nv_tegra/
-rw-r--r--  1 yerong yerong 2.9K Jul 20 02:45 p2371-0000.conf
-rw-r--r--  1 yerong yerong 2.8K Jul 20 02:45 p2371-2180.conf
-rw-r--r--  1 yerong yerong 2.9K Jul 20 02:45 p2371-2180-devkit-24x7.conf
-rw-r--r--  1 yerong yerong 2.9K Jul 20 02:45 p2371-2180-devkit.conf
drwxr-xr-x 21 yerong yerong 4.0K Dec 23 23:57 rootfs/
-rwxr-xr-x  1 yerong yerong 9.3K Jul 20 02:45 source_sync.sh*

Those directories are on the host. The content which is in rootfs needs to be copied into the root of the SD card’s first partition. Had this been to eMMC, then flash would have done that copy for you.

Normally rootfs is constructed by unpacking the sample rootfs into it as root, and then using the apply_binaries.sh script to copy the hardware-accelerated version of libraries into this as root. Lastly, the flash.sh script itself adds content to the “rootfs/boot/” directory depending on what your flash options were. If that content was copied there, then all you need to do is recursively copy rootfs (with sudo) into that first partition.

Do you have file “rootfs/boot/extlinux/extlinux.conf”? If so, then it likely has a “root=/dev/sda1” entry for the case of having flashed to target sda1. Had there never been a flash, then extlinux.conf wouldn’t be there; had flash been to target mmcblk0p1, then it would have “root=/dev/mmcblk0p1”.

NOTE: Many distributions do not use 64-bit ext4 extensions. Ubuntu 14.04 and 16.04 are in this category. If you use one of the more recent distributions, and if those extensions are enabled, then U-Boot will not be able to see the file system even if you did everything correctly. If your host file “/etc/mke2fs.conf” shows “64bit” or “metadata_csum”, then you know the default ext4 won’t be valid for your Jetson due to U-Boot. You can still mkfs.ext4 from the host, but you’d need to pass options “-O ^metadata_csum,^64bit” to mkfs.ext4 to tell it to not use those options.

I am wondering what are the steps for flashing the OS directly to sda1 on the tx1 side?

My rootfs/boot/extlinux/extlinux.conf shows ‘mmcblk0p1’:

TIMEOUT 30
DEFAULT primary

MENU TITLE p2371-2180 eMMC boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      INITRD /boot/initrd
      APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait

When I run the command:

sudo ./flash.sh jetson-tx1 mmcblk0p1

I can directly flash the OS to eMMC as normal installation.

Yes, you are right, if I run the command:

sudo ./flash.sh jetson-tx1 mmcblk0p1

on the desktop side in the repository rootfs, the /boot/extlinux/extlinux.conf reads

TIMEOUT 30
DEFAULT primary

MENU TITLE p2371-2180 eMMC boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      INITRD /boot/initrd
      APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait

and if I run the command

sudo ./flash.sh jetson-tx1 sda1

extlinux.conf in rootfs(on desktop side) reads

TIMEOUT 30
DEFAULT primary

MENU TITLE p2371-2180 USB boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      INITRD /boot/initrd
      APPEND ${cbootargs} root=/dev/sda1 rw rootwait

Here is what I did: I firstly copied the system.img.raw to the thumb drive on the desktop side.
Then I run the flash.sh with the thumb drive pluged on the Jetson side.

sudo ./flash.sh jetson-tx1 sda1

And it still doesn’t work saying that Jetson cannot configure graphic driver properly.

As a summary:

[First try]

1.Copy system.img.raw to the thumb drive on the desktop;
2.Flash OS to eMMC
3.Edit extlinux.conf on Jetson eMMC to sda1

Gives output

random: nonblocking pool is initialized
cannot find fireware....retry after 1 second
Direct firemware load for tergra21x_xusb_firemware failed with error -2
Falling back to user helper
cannot find fireware....retry after 1 second
Direct firemware load for tergra21x_xusb_firemware failed with error -2
Falling back to user helper
cannot find fireware....retry after 1 second
Direct firemware load for tergra21x_xusb_firemware failed with error -2
Falling back to user helper
cannot find fireware....retry after 1 second
Direct firemware load for tergra21x_xusb_firemware failed with error -2
Falling back to user helper

[Second try]

1.Copy system.img.raw to the thumb drive on the desktop side;
2.On the desktop side, run flash.sh with thumb drive inserted on Jetson (and with the micro-USB connecting Jetson and desktop);

sudo ./flash.sh jetson-tx1 sda1

Gives pop-out error
Cannot configure graphics driver properly and Jetson asked me to do the configuration manually.

And after I “OK”, Jetson asked me “what would you like to do?”


Then I could not press arrow keys or press the enter key but press ESC and the low graphics error repeats over and over again.

[/quote]

One thing I have forgotten to check…if the USB port used for the flash drive changes between USB2 mode and USB3 mode during boot (the controller, not the device), then that USB port cannot be used as a rootfs device. U-Boot only understands USB2, and the full-sized USB port will switch to USB3 mode after boot, but the micro-USB port will stay USB2. Are you using the micro-USB port during boot? If not, then try that out.

I’m not sure what those GUI displays are from, but it would seem to be a case of missing the correct drivers…which in turn could be an install issue, or a rootfs setup issue, or perhaps losing the USB port during boot (which implies a loss of reading drivers from disk).

I think I succeed in booting R24.1 (JetPack 2.3) from my thumb drive previously. I hope this indicates that my USB port and thumb drive should work on Jetson (TX1) board.

I believe by default R24.1 has both USB ports in USB2 mode…this would cause continuity of service for any rootfs device over USB and work well. Newer L4T may have a port which is USB2 in U-Boot and then switch to USB3 under Linux…this will break boot of a rootfs device over that USB port since it momentarily drops access to the device until it re-enumerates.

I found this is possibly happening in my case:

I tried to put /usr/local in my thumb drive and editing fstab as

UUID=178f9428-9292-4024-8011-93fafb90f17c /usr/local ext4 noauto,user,exec,owner,exec 0 0

And reboot the system and the thumb drive does not mount automatically.

The /usr/local looks as this at the beginning

nvidia@tegra-ubuntu:/usr/local$ ll
total 8
drwxr-xr-x 2 root root 4096 Dec 27 07:46 ./
drwxr-xr-x 11 root root 4096 May 3 2016 ../

It was after I mounted the thumb drive manually through the Disk Utility, the /usr/local looks normal:

nvidia@tegra-ubuntu:/usr/local$ ll
total 59388
drwx------ 16 nvidia nvidia 4096 Dec 27 07:45 ./
drwxr-xr-x 11 root root 4096 May 3 2016 ../
drwxr-xr-x 2 root root 4096 Apr 20 2016 bin/
lrwxrwxrwx 1 root root 8 Dec 27 07:40 cuda -> cuda-8.0/
drwxr-xr-x 11 root root 4096 Dec 27 07:40 cuda-8.0/
drwxr-xr-x 2 root root 4096 Apr 20 2016 etc/
drwxr-xr-x 2 root root 4096 Apr 20 2016 games/
-rw-rw-r-- 1 nvidia nvidia 1595408 Nov 6 2016 get-pip.py
drwxr-xr-x 2 root root 4096 Apr 20 2016 include/
drwxr-xr-x 4 root root 4096 May 3 2016 lib/
drwx------ 2 root root 16384 May 6 2016 lost+found/
lrwxrwxrwx 1 root root 9 Apr 20 2016 man -> share/man/
-rw-rw-r-- 1 nvidia nvidia 1335034 Dec 26 06:58 numba-0.36.2.tar.gz
drwxrwxr-x 2 nvidia nvidia 4096 Dec 26 07:20 openssl/
drwxrwxr-x 22 nvidia nvidia 4096 Dec 26 07:53 openssl-master/
-rw-rw-r-- 1 nvidia nvidia 17070797 Dec 26 07:16 openssl-master.zip
drwxrwxr-x 16 nvidia nvidia 4096 Dec 26 07:59 Python-3.5.1/
-rw-rw-r-- 1 nvidia nvidia 20143759 Dec 26 07:01 Python-3.5.1.tgz
-rw-rw-r-- 1 nvidia nvidia 20566643 Dec 26 06:50 Python-3.5.2.tgz
drwxr-xr-x 2 root root 4096 Apr 20 2016 sbin/
drwxr-xr-x 8 root root 4096 May 3 2016 share/
drwxr-xr-x 2 root root 4096 Apr 20 2016 src/
drwx------ 4 nvidia nvidia 4096 Dec 26 07:35 .Trash-1001/

Do you know how to fix the port issue here?

The “noauto” in fstab says to not auto-mount. Basically you’d still need to “sudo mount /usr/local”, and if that UUID is present, then it will mount.

If you say to always mount, and if the SD card isn’t there, then it will hang and never finish booting. Since you are specifying by UUID, then this is the only SD card which will satisfy the requirement unless you manually force this UUID onto other SD cards. If for whatever reason you forget this UUID and lose the SD card, then your system will likely be unbootable beyond the point of where it mounts file systems (you’d have to test to be sure…but test on one you can afford to lose, e.g., after cloning).

I’m not sure where you would do it, but udev should be able to see a disk with this exact UUID and auto mount on “/usr/local” when the disk is detected as present. You might look around in “/etc/udev/rules.d/”. Note that the system places its default rules in “/lib/udev/rules.d/” and that if the same file name is seen in “/etc/udev/rules.d/”, then the “/etc” version is used. You could put a symbolic link in the “/etc” version pointing at the “/lib” version and it would essentially have no effect…or you could copy a file from the “/lib” version, put it in the “/etc” area, and then edit this file…the rule would be changed until you remove the “/etc” version. It’s a way to test without messing up system files.