How to Boot from NVMe SSD?

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
,
44

Hi everyone,
I have used system installed on SSD with @jocover 's method without any problem for some weeks. But today Ubuntu Update downloaded some updates from nvidia repository ( this and this ), requested to reboot and then Xavier failed to boot system, hanging on initrd step.
Log from debug port: afterupdate-ubuntu-exp2(tgr)-cutecom.log (27.6 KB)
I tried backup kernel (original), but it panic. afterupdate-ubuntu-orig-cutecom.log (56.2 KB)

There was strange thing in log. After update root=/dev/mmcblk0p1 parameter is back!!! Kernel tried to switch rootfs on /dev/mmcblk0p1, that didn’t contain system in @jocover 's method.

[0015.280] I> Linux Cmdline: root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4 video=tegrafb no_console_suspend=1 earlycon=tegra_comb_uart,mmio32,0x0c168000 gpt tegra_fbmem=0x800000@0xa069c000 lut_mem=0x2008@0xa0697000 usbcore.old_scheme_first=1 tegraid=19.1.2.0.0 maxcpus=8 boot.slot_suffix=_b boot.ratchetvalues=0.4.2 vpr_resize sdhci_tegra.en_boot_part_access=1

So, device block with cboot device tree on eMMC was rewrited. I think it’s nvidia-l4t-bootloader package doing.

I had to reflash Xavier from JetPack to repair boot. ( sudo ./flash.sh jetson-xavier external)
System on SSD didn’t take any damage and after reboot system loaded with rootfs from SSD. I copied /boot folder from SSD ( updated from repository kernel and new device trees installed there ) to eMMC and rebooted.
Now Xavier works like before.

So, it should be noted, that now Nvidia firmware updates doesn’t take into account if system was flashed with sudo ./flash.sh jetson-xavier external and reflashing from external PC with JetPack will be needed after update.
Systemd method should be resistant against updates as it doesn’t use bootloader features.
In all cases coping from /boot of SSD to /boot on eMMC will be required.

45

But today Ubuntu Update downloaded some updates from nvidia repository ( this and this ), requested to reboot and then Xavier failed to boot system, hanging on initrd step.

IMO it’s safer to just put commonly used stuff on a ssd and not the entire of /. I’m not sure how much benefit you get from having stuff like /etc on an ssd, since it’s mostly a bunch of text files.
My partition layout looks like:

nvme0n1      259:11   0 931.5G  0 disk
├─nvme0n1p1  259:12   0 119.2G  0 part /tmp
├─nvme0n1p2  259:13   0 323.9G  0 part /home
├─nvme0n1p3  259:14   0   250G  0 part /usr
└─nvme0n1p6  259:15   0 238.4G  0 part /var/lib/docker

Those are just ext4 partitions on a GPT disk, but you may wish to use something fancier like LVM or btrfs with subvolumes for easier resizing.

1 Like
46

Yes, if you update system, you need to manually copy /boot of SSD to /boot on emmc

upgrade
sudo mount /dev/mmcblk0p1 /mnt
cp -r /boot/* /mnt/boot/

47

Why this method is not working for Xavier NX, using
sudo ./flash.sh jetson-xavier-nx-devkit mmcblk0p1

48

My Jetson Xavier NX still boot from microSD instead of nvme ssd following the same approach, I do not know what went wrong?

50

Hi, @jocover
Sorry for the late answer.

  1. Update, mentioned in my previous post ( 4.4 DP), reflashed CBO partition with default root device path root=/dev/mmcblk0p1 . In this case boot folder copy couldn’t help. I don’t know, was it my local error or previous update feature :)

  2. According current update to JetPack 4.4 - L4T R32.4.3 production release:
    In autoupdate window I have choosen “Reboot later” and have manually copied boot folder to eMMC. Xavier have rebooted succesfully and even with customized kernel, that I have used.
    Interesting features:
    This update regenerated extlinux.conf file ( /boot/extlinux/extlinux.conf on SSD). It succesfully parsed my manually edited config (was: extlinux.conf.bak.log (883 Bytes) , now: extlinux.conf.log (1.2 KB) )
    root= parameter was added to APPEND:
    Before update:

LABEL primary
MENU LABEL primary kernel
LINUX /boot/Image-exp2
INITRD /boot/initrd.img-4.9.140-tegra
APPEND ${cbootargs} quiet

After update:

LABEL primary
MENU LABEL primary kernel
LINUX /boot/Image-exp2
INITRD /boot/initrd.img-4.9.140-tegra
APPEND ${cbootargs} quiet root=PARTUUID=84bce66b-f20c-49f0-a6dd-d2d402a01189 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4

Kernel command line from dmesg:

[ 0.000000] Kernel command line: console=ttyTCU0,115200 video=tegrafb no_console_suspend=1 earlycon=tegra_comb_uart,mmio32,0x0c168000 gpt tegra_fbmem=0x800000@0xa069c000 lut_mem=0x2008@0xa0697000 usbcore.old_scheme_first=1 tegraid=19.1.2.0.0 maxcpus=8 boot.slot_suffix=_b boot.ratchetvalues=0.4.2 vpr_resize sdhci_tegra.en_boot_part_access=1 quiet root=PARTUUID=84bce66b-f20c-49f0-a6dd-d2d402a01189 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4

So, in [JetPack 4.4 - L4T R32.4.3 production release] root device path from CBO partition is not used. I think it’s related to Issue 2926568 from 32.4.3 Release Notes . Problem with init script in initrd has been solved by NVIDIA by removing source of first root= parameter :). Update switched system from your method to boot from SSD to APPEND method.

Your method should be tested on latest JetPack. external option could be unsupported now.

51

Hi, @crazy_yorick ,
Thanks for your sharing, I follow the steps and run the scripts from “GitHub - jetsonhacks/rootOnNVMe: Switch the rootfs to a NVMe SSD on the Jetson Xavier NX and Jetson AGX Xavier” to switch the rootfs to SSD successfully.
Because the SD Card is still needed for booting, I am now trying to simplify the SD Card, remove the files that has nothing to do with booting and make the SD small. I find the fold usr/ fold contains a lot of files. Could you help specify that which files we need keep for booting and the other files can be removed?

52

Hi, @laowanggeorge ,
Rootfs switching method is implemented with booting from system on eMMC (SDCard in your case) in SSD fail cases in mind. There are units, executing in parallel with setssdroot.service , including udevd, thus, most system folders on SD card, except “home”, are required (systemd method pros and cons).

If you don’t need to keep spare system and use JetPack 4.4 then IMHO it’s better to use extlinux.conf APPEND method.
In that case only “boot” folder required on SD Card APP partition. “Pros and cons” link above is outdated for APPEND method. Latest cboot in JetPack 4.4 (not DP!!) don’t add root=.. parameter from device tree on CBO partition, so problem with two root=.. parameter in cmdline is solved.
After flashing with latest JetPack there are extlinux.conf ( /boot/extlinux/extlinux.conf) on eMMC/SD card. It already contains APPEND parameter, something like this:

LABEL primary
MENU LABEL primary kernel
LINUX /boot/Image
INITRD /boot/initrd
APPEND ${cbootargs} quiet root=‘some device’ rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4

So, to place rootfs on SSD with APPEND method you can try 2 ways:

  1. Simple, but I haven’t tested it on latest JetPack. Follow @jocover’s method. As result, only boot folder on eMMC will be created.

  2. Straightforward method:

  • Flash device from SDKManager
  • Boot device, configure OS.
  • Open extlinux.conf ( /boot/extlinux/extlinux.conf) and in APPEND string replace
    root='some device' with root=PARTUUID=your SSD partuid
  • Copy rootfs from eMMC/SD to SSD
  • Reboot device. It should boot to rootfs on SSD.

After OTA update of bootloader (when after ubuntu autoupdate new window with warning appears), you should click “Reboot later”, mount SD card and copy /boot folder from SSD to SD card. If you build custom kernel or edit extlinux.conf always syncronise boot folders on SSD and SD card. Because OTA update parse extlinux.conf on SSD and can fail if this folder have unexpected files, but flashing bootloader-related partitions (CBO and others) on eMMC/SD will succeed and reflashing device will be needed. I have already bricked my Xavier once with just additional device tree file in boot folder on SSD :( .

53

Regarding @jocover’s method can somebody explain what the procedure is to accomplish this? I understand what each individual cli command does, but I have the faintest idea how to accomplish this with SDK Manager.

Is there a hidden command line interface/shell/ini config file?

Trying to grasp the caveats.

54

I can’t help with most of this, but understand that SDK Manager is just a front end for downloading software, running the flash software (as a backend), and other dependency resolving. If you’ve used SDKM to flash once, then you will have this directory:
~/nvidia/nvidia_sdk/JetPack...version.../Linux_for_Tegra/

The “Linux_for_Tegra/” content is what you could have downloaded separately as the “driver package”. This is the true flash software. You don’t even need SDKM to download and unpack this, SDKM is just a convenience for this.

Within “Linux_for_Tegra/” there is an initially blank subdirectory, “Linux_for_Tegra/rootfs/”. When flashing, it is this content which makes up the root filesystem. The downloadable package for “sample root filesystem” is normally unpacked there (using sudo), and is an entirely 100% pure Ubuntu. This lacks some of the needed drivers for Jetsons, and so from “Linux_for_Tegra/”, running command “sudo ./apply_binaries.sh” adds those drivers. This allows NVIDIA’s content to be maintained separately and not be mixed with Ubuntu except when the end user chooses to do so.

The “flash.sh” command is normally how one would flash on command line, and this is in the “Linux_for_Tegra/” directory which is installed when unpacking the driver package.

Flashing does not install any of the optional “extras”, e.g., no CUDA or samples. This is the part where SDKM helps as it can do this for you without many of the steps you might need to understand to add these packages yourself. This includes optional packages to both host PC and Jetson.

Most of what the jocover method does is simply from the host side without involving SDKM other than having the “Linux_for_Tegra/” content. When he gets to this line it is the first time SDKM is involved, and then only for the purpose of downloading the “Linux_for_Tegra/” content:
sudo ./flash.sh jetson-xavier external

Do you have “Linux_for_Tegra/”? If so, then you have no more requirements (within his post) for using SDKM. If you don’t have “Linux_for_Tegra/”, then you could skip SDKM and separately download the driver package and sample rootfs, prepare with a single “sudo apply_binaries.sh”, and from then on run as many flashes as you want without ever touching SDKM.

Note that during a flash the “Linux_for_Tegra/rootfs/” content is almost exactly what is used to create the root filesystem on the Jetson. The exception to being an exact copy is from some of the “/rootfs/boot” content being edited prior to flash, where the edits depend on the argument passed to flash.sh. In this example the argument is “external”, whereas normally it would be “mmblk0p1”. Part of the changes from this argument change is that the “rootfs/boot/extlinux/extlinux.conf” will be edited to point at your partition’s UUID instead of pointing at “/dev/mmcblk0p1” during boot.

Special instructions are almost always referring to something within the “Linux_for_Tegra/” directory. The other commands could be run from anywhere on the host PC for the first part of his instructions. The rest of the commands are from the running Jetson after flash completes and you’ve performed the first boot setup (it is hard to set up if you don’t have a user account, and this is part of first boot setup).

2 Likes
55

Thank you, a very helpful reply! This is something I can work with. Appreciated!

56

I tried on my TX2 board, JetsonHacks’ github repo code works on my TX2 with a Samsung 981A NVME SSD too.

1 Like
57

This worked for me.
Root On NVME
I can also add that it is a nice way of cloning one machine onto another.
Just do

  • Clone from NVME back onto root SD card (you need to alter the copy-rootfs-ssd.sh)
  • Swap with a clean SSD and apply procedure as described.
1 Like
58

Thanks Jetson hacks, your Github repo (rootOnNVME) works totally fine on Jetson AGX Xavier with WD blue SN550 500GB ssd.

59

Thank you so much. It wonder work for me.

60

it might not work for upgrading the tegra release - it might get broken if the upgrade is executed from ssd nvme

61

I successfully had my AGX running from an m.2 SSD. I then updated to the latest jetpack and have run into the following issue:

If the m.2 ssd is inserted, the AGX does not start up. If i remove the SSD then it works. I take it that its trying to load up this, now incompatible SSD? How can i stop it from trying to load from there so that I can format it?

62

Probably due to jetpack4.6, which starts to support boot from NVMe ssd in cboot. And its priority is higher than the emmc. You can try to change the boot order.

https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/flashing.html

1 Like
63

@WayneWWW yes changing the boot order worked thank you.

Great, so with Jetpack4.6, would I conduct the JetsonHacks ./copy-rootfs-ssd.sh script to copy the emmc to the SSD, and then change the boot order so that the SSD is the priority? Basically there is no longer a need for the ./setup-service.sh script work around described here anymore?

64

Actually, jetsonhacks is not official website maintained by us. I am not sure if their tool is able to work in each release.

Our official method is on this document.

https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/flashing.html#wwpID0E0QN0HA