How to Boot from NVMe SSD?

dkreutz · May 22, 2020, 4:13pm

I can confirm that the first method does work on Xavier NX Dev Kit - i made it running just yesterday on my NX.

You need to check and adapt the name of the NVMe device (iirc it is /dev/nvme0n1 instead of /dev/nvme0n1p1)

erikneffca · May 22, 2020, 5:37pm

@dkreutz thx for confirming - I’ll give it a try!

crazy_yorick · May 24, 2020, 10:45am

Hello,
I’ve found how to mount SSD as rootfs with extlinux.conf APPEND, but in still requires initrd modification (modification of this method).

Main problem: initrd created by JetPack 4.4 has init script, that search for root device first as device name format, then as PARTUUID format. Then only “PARTUUID*”, “mmcblk*”, “sd*”, “nfs” are acepted as legal rootdev string.
init[62…173]:

rootdev=“$(sed -ne 's/.\broot=/dev/([abcdefklmnps0-9])\b./\1/p’ < /proc/cmdline)"
if [ “${rootdev}” == “” ]; then
uuid_regex=‘[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}’
rootdev="$(sed -ne "s/.\broot=(PARTUUID=${uuid_regex})\b.*/\1/p” < /proc/cmdline)"
fi

if [ “${rootdev}” != “” ]; then
echo “Root device found: ${rootdev}” > /dev/kmsg;
fi
if [[ “${rootdev}” == PARTUUID* ]]; then
-------
elif [[ “${rootdev}” == mmcblk* ]]; then
-------
elif [[ “${rootdev}” == sd* ]]; then
-------
elif [[ “${rootdev}” == “nfs” ]]; then
-------
else
echo “No root-device: Mount failed” > /dev/kmsg;
exec /bin/bash;
fi

crazy_yorick · May 24, 2020, 11:28am

Working second method step by step:

Flash device (Xavier AGX or NX (tested on AGX only)) with JetPack 4.4.
Unpack initrd:
Create directory in user folder. Create there 2 scripts:
unpack.sh:

#!/bin/bash
gzip -cd …/initrd | cpio -imd

pack.sh:

#!/bin/bash
find . -print0 | cpio --null --quiet -H newc -o | gzip -9 -n > …/initrdtossd

Make them executable. Create folder to unpack and cd there.

sudo mkdir dir_initrdssd
sudo cp /boot/initrd ./
cd dir_initrdssd
sudo …/unpack.sh

Patch init
Initrd source will be in dir_initrdssd folder. Patch init script with following patch init.patch:initpatch.log (1.4 KB)
(only .log text files are accepted by forum :( )
Patch changes priority to PARTUUID over device name and adds "nvme" as accepted rootfs name.
Pack back

sudo …/pack.sh

initrdtossd in folder with scripts will be created. Copy this file to /boot

sudo cp …/initrdtossd /boot/

Edit extlinux.conf
Find PARTUID of nvme partition with future rootfs.

lsblk -o NAME,PARTUUID
Output needed:
nvme0n1
└─nvme0n1p1 XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

Open /boot/extlinux/extlinux.conf

sudo gedit /boot/extlinux/extlinux.conf

Add APPEND line to primary boot and create backup boot
Example:

TIMEOUT 30
DEFAULT primary

MENU TITLE L4T boot options

LABEL primary
MENU LABEL primary kernel
LINUX /boot/Image
INITRD /boot/initrdtossd
APPEND ${cbootargs} root=PARTUUID=XXXXXXXX-XXXX-XXXX-XXXX->XXXXXXXXXXXX quiet

LABEL backup
MENU LABEL backup kernel
LINUX /boot/Image
INITRD /boot/initrd
APPEND ${cbootargs} quiet

DON’T FORGET to copy rootfs to SSD!!!.
Reboot and PROFIT

P.S. In blockquote “[two points]/” changed to “…/”. I don’t know how to fix it.

dkreutz · May 24, 2020, 7:36pm

@crazy_yorick can you give pros and cons for both methods?

crazy_yorick · May 26, 2020, 8:50pm

Ok.
Methods:

systemd service
In this case systemd service setssdroot.service runing before local-fs-pre.target is used ( after systemd-remount-fs.service from initrd, if initrd with systemd is used) . So it’s one of the first runing units when system starts up. But, there are parallel targets swap.target, cryptsetup-pre.target , various low-level services. (systemd bootup ) . This service executes script, which switches rootfs with systemctl call /bin/systemctl --no-block switch-root ${CHROOT_PATH}.

Pros:

No initrd or extlinux.conf required. So, work with JetPack 4.2, where kernel is located in special partition.
After building new kernel, default initrd update workflow is required, no manual modification of initrd (i.e. flash.sh ...).If system doesn’t use initrd, we need only install modules on SSD and eMMC.
If /dev/nvme0n1p1 is absent, default rootfs is used, because setssdroot.service will not be executed.

Cons:

It’s still hack:) .
There are units, executing in parallel with setssdroot.service, including udevd, thus, for some devices modules will be loaded from rootfs on eMMC while others from NVMe. That’s why we need to install same modules on both.
I don’t understand why, but we need copy of setssdroot.service and setssdroot.sh on NVMe, otherwise service completes with error (but still switches rootfs).

extlinux.conf APPEND
In this case APPEND line with new “root=…” parameter is added to extlinux.conf. At kernel loading stage cboot appends this line to kernel cmdline. So cmdline have 2 “root=…” parameters: first - “root=/dev/mmcblk0p1” from device tree for cboot, second - custom, appended with extlinux.conf directive.

Pros:

Normal system bootup sequence. Main and backup kernel could be different.
All modules are loaded from initrd or rootfs on NVMe only. One needs only to copy new kernel image and it’s initrd to /boot/ directory on eMMC and configure /boot/extlinux/extlinux.conf there. No modules installation for new kernel on eMMC required. (Except when the same kernel image is used as backup).
Backup kernel is set by extlinux.conf :

LABEL backup
MENU LABEL backup kernel
LINUX /boot/Image
INITRD /boot/initrd
APPEND ${cbootargs} quiet

It can have it’s own initrd and only it’s modules are installed on eMMC.

Cons:

Only JetPack 4.4 now is supported, because cboot with extlinux.conf parsing is required.
After building new kernel and updating initrd , init script should be patched manualy in case of default JetPack initrd. According @gtj, Dracut generated initrd works without modification
If NVMe is absent, linux is stuck in initrd. To boot from backup kernel and rootfs on eMMC command from debug terminal should be used.

vishwakarma.rhl · May 28, 2020, 2:59am

Thanks to @crazy_yorick and our friend at JetsonHacks (GitHub - jetsonhacks/rootOnNVMe: Switch the rootfs to a NVMe SSD on the Jetson Xavier NX and Jetson AGX Xavier),
I was able to seamlessly run provided scripts on a AGX Xavier board today to transfer rootfs on a Crucial P1 500GB M2 NVMe SSD which was available at /dev/nvme0n1p1

crazy_yorick · May 30, 2020, 4:56pm

Hi everyone,
I was able to use initrd produced by ubuntu for second method ( extlinux.conf APPEND), thus removing manual manipulations with initrd.
Steps:

Flash device (Xavier AGX or NX (tested on AGX only)) with JetPack 4.4.
Add update-initramfs hook to add xusb firmware to initrd.
a. Create file tegra-xusbfw with following content (Remove log extention tegra-xusbfw.log (306 Bytes) ):

#!/bin/sh
PREREQ=“”
prereqs()
{
echo “$PREREQ”
}

case $1 in
prereqs)
prereqs
exit 0
;;
esac

. /usr/share/initramfs-tools/hook-functions

copy_exec /lib/firmware/tegra19x_xusb_firmware /lib/firmware
copy_exec /lib/firmware/tegra18x_xusb_firmware /lib/firmware

b. Make file executable and copy to /usr/share/initramfs-tools/hooks directory.
c. Generate initrd:

sudo update-initramfs -c -v -k all

d. Check in log that “tegra19x_xusb_firmware” was added.
e. At this step you should have file initrd.img-4.9.140-tegra in /boot directory.

Edit /boot/extlinux/extlinux.conf

sudo gedit /boot/extlinux/extlinux.conf

Edit INITRD parameter and add APPEND parameter to primary record. Should be something like this (extlinux.conf.log (383 Bytes) ):

TIMEOUT 30
DEFAULT primary

MENU TITLE L4T boot options

LABEL primary
MENU LABEL primary kernel
LINUX /boot/Image
INITRD /boot/initrd.img-4.9.140-tegra
APPEND ${cbootargs} root=PARTUUID=XXXXXXXX-XXXX-XXXX-XXXX->XXXXXXXXXXXX quiet

LABEL backup
MENU LABEL backup kernel
LINUX /boot/Image
INITRD /boot/initrd
APPEND ${cbootargs} quiet

DON’T FORGET to copy rootfs to SSD!!!.
Reboot

Kernel don’t rises any errors while booting from this initrd.
It would be good if someone from NVIDIA check if ubuntu generated initrd didn’t miss any other modules.

gtj · June 4, 2020, 3:44pm

It’s very strange that usb firmware is needed to get to an nvme device but if that works, great!

crazy_yorick · June 4, 2020, 7:37pm

USB firmware is needed because otherwise the usb port will not work.
If you use initrd generated without firmware hook, system do will boot, you will see login screen, but without keyboard it’s useless :).

P.S. I will try APPEND parameter combinations later on weekends.

gtj · June 4, 2020, 9:06pm

Yeah but what’s that got to do with nvme unless you’re using a usb<>nvme bridge. I may have gotten lost between this thread and the one on the NX forum. Are you using usb<>nvme or pcie<>nvme?

crazy_yorick · June 5, 2020, 12:01am

I use NVMe in M.2 key M slot (PCIe x4).
All above was to differentiate 2 variants of APPEND in extlinux.conf method (2)

2.1 With initrd, installed while flashing with JetPack 4.4.
This case requares patching init script in initrd, as default init script prioritirizes root=/dev/[somedevice] over root=PARTUUID=.... .

With default init APPEND below doesn’t work for some still unknown reason. (Boot stops at initrd’s console, as if there is no device, but I see /dev/nvme0n1p1 there ???)

…
INITRD /boot/initrd
APPEND ${cbootargs} root=/dev/nvme0n1p1

In case

INITRD /boot/initrd
APPEND ${cbootargs} root=PARTUUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

first root=/dev/mmcblk0p1 is prioritirized and system boots to rootfs on eMMC.

2.2 With initrd generated by update-initramfs in ubuntu.
In this case Xavier is flashed with JetPack 4.4, then new initrd is generated on Xavier itself and extlinux.conf edited to point on new initrd and APPEND root=PARTUUID= added.

…
INITRD /boot/initrd.img-4.9.140-tegra
APPEND ${cbootargs} root=PARTUUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

This initrd lacks xusb firmware, but has no problems with parsing root=xxx parameter. So linux boots to rootfs on NVMe well, but USB ports are not functional, so no mouse or keyboard could be used.
To add xusb firmware, hook is registered (this post) before update-initramfs.
Thus 2.2 method is most traditional from Linux view, IMHO.

gtj · June 5, 2020, 12:33pm

Ah OK, that makes sense. The usb firmware wasn’t needed to make the nvme work, but to make the mouse and keyboard work. Sorry. I was getting confused…

jocover · June 5, 2020, 4:47pm

My method,use sdkmanager

on jetson xavier
init nvme ssd and read partuuid

sudo parted /dev/nvme0n1
mklabel gpt
mkpart primary 2048s 100%
quit

sudo mkfs.ext4 /dev/nvme0n1p1
sudo blkid /dev/nvme0n1p1

example my device partuuid
/dev/nvme0n1p1: UUID=“b11f5460-ec67-4f4a-9c41-8cffd4b11df4” TYPE=“ext4” PARTLABEL=“primary” PARTUUID=“532ead91-c15f-4412-83b5-e5675ff21773”

clone emmc to ssd
sudo dd if=/dev/mmcblk0p1 of=/dev/nvme0n1p1

back to linux host

echo ‘532ead91-c15f-4412-83b5-e5675ff21773’ > bootloader/l4t-rootfs-uuid.txt
sudo ./flash.sh jetson-xavier external

on jetson xavier
resize ssd size

sudo resize2fs /dev/nvme0n1p1

upgrade
sudo mount /dev/mmcblk0p1 /mnt
cp -r /boot/* /mnt/

crazy_yorick · June 7, 2020, 11:56pm

Thanks, @jocover . I have repeated your method. It works great and doesn’t requires any system modifications, unlike APPEND methods. IMHO, it’s the best method in cases where restoration system on eMMC is not required.

I think it should be noted that after sudo ./flash.sh jetson-xavier external all data on eMMC will be rewrited and APP partition consists only boot folder:

./
├── [4.0K] boot
│ ├── [4.0K] dtb
│ │ ├── [269K] tegra194-p2888-0001-p2822-0000.dtb
│ │ └── [4.0K] tegra194-p2888-0001-p2822-0000.dtb.sig
│ ├── [4.0K] extlinux
│ │ └── [ 733] extlinux.conf
│ ├── [ 33M] Image
│ ├── [5.3M] initrd
│ └── [269K] tegra194-p2888-0001-p2822-0000.dtb
└── [ 16K] lost+found [error opening dir]
4 directories, 6 files

crazy_yorick · June 8, 2020, 12:46am

Some remarks about kernel command line and initrd.
In @jocover case:

Kernel command line: root=PARTUUID=[MY-PARTUUID] rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4 video=tegrafb no_console_suspend=1 earlycon=tegra_comb_uart,mmio32,0x0c168000 gpt tegra_fbmem=0x800000@0xa069c000 lut_mem=0x2008@0xa0697000 usbcore.old_scheme_first=1 tegraid=19.1.2.0.0 maxcpus=8 boot.slot_suffix= boot.ratchetvalues=0.4.2 vpr_resize sdhci_tegra.en_boot_part_access=1 quiet

First root= is from cboot. Now it’s allready in root=PARTUUID= format and for original initrd APPEND ${cbootargs} root=/dev/mmcblk0p1 in extlinux.conf should enable to load rootfs from eMMC (that must be copied there manualy).

About original initrd. It’s main feature - it doesn’t contain any modules, only libs and firmware, so it works with modified kernels ( original dmesg.orig.log (75.6 KB) , build on Xavier (added KVM hw support like there) dmesg.orig-exp2.log (75.9 KB) ):
ls-initrd.orig.log (7.6 KB)

./
…
├── [4.0K] lib
│ ├── [4.0K] aarch64-linux-gnu
│ │ ├── [123K] ld-2.23.so
…
│ │ └── [ 18K] libuuid.so.1.3.0
│ ├── [4.0K] firmware
│ │ ├── [122K] tegra18x_xusb_firmware
│ │ └── [122K] tegra19x_xusb_firmware
│ └── [ 28] ld-linux-aarch64.so.1 → aarch64-linux-gnu/ld-2.23.so
├── [4.0K] mnt
…
22 directories, 137 files

I also have tested with initrd generated by update-initramfs (with xusbfw hook).
Original kernel work fine with initrd generated for it.
Builded on Xavier kernel with it’s initrd failed to boot. Cboot failed to load initrd. I will create new topic for this problem later. initrd-ubuntu-exp2-cutecom.log (23.5 KB)
But modified kernel booted with update-initramfs generated initrd without any taunted modules (dmesg.ubuntu-exp2tgr.log (75.8 KB) ).
Main difference of generated initrd - it’s size. ls-initrd.ubuntu.log (39.2 KB) ls-initrd.ubuntu-exp2.log (39.3 KB)

LDighera · June 12, 2020, 9:28pm

Thank you for sharing your information.

As I did not have a system running Ubuntu, I was unable to use the SDK method to install JetPack on my Jetson AGX Xavier. I flashed the image to a uSD card, and boot from it.

I would prefer to boot from NVMe SSD. Are any changes required to move the rootfs from uSD to SSD?

Any suggestions about what to do with the eMMC?

ADVAthanksNCE

LDighera · June 25, 2020, 6:48pm

Jim Brenen shows how to Run From NVMe SSD in this YouTube video.

You can download the scripts from Jim’s github repository.

crazy_yorick · June 27, 2020, 7:06pm

Hi everyone,
I have used system installed on SSD with @jocover 's method without any problem for some weeks. But today Ubuntu Update downloaded some updates from nvidia repository ( this and this ), requested to reboot and then Xavier failed to boot system, hanging on initrd step.
Log from debug port: afterupdate-ubuntu-exp2(tgr)-cutecom.log (27.6 KB)
I tried backup kernel (original), but it panic. afterupdate-ubuntu-orig-cutecom.log (56.2 KB)

There was strange thing in log. After update root=/dev/mmcblk0p1 parameter is back!!! Kernel tried to switch rootfs on /dev/mmcblk0p1, that didn’t contain system in @jocover 's method.

[0015.280] I> Linux Cmdline: root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4 video=tegrafb no_console_suspend=1 earlycon=tegra_comb_uart,mmio32,0x0c168000 gpt tegra_fbmem=0x800000@0xa069c000 lut_mem=0x2008@0xa0697000 usbcore.old_scheme_first=1 tegraid=19.1.2.0.0 maxcpus=8 boot.slot_suffix=_b boot.ratchetvalues=0.4.2 vpr_resize sdhci_tegra.en_boot_part_access=1

So, device block with cboot device tree on eMMC was rewrited. I think it’s nvidia-l4t-bootloader package doing.

I had to reflash Xavier from JetPack to repair boot. ( sudo ./flash.sh jetson-xavier external)
System on SSD didn’t take any damage and after reboot system loaded with rootfs from SSD. I copied /boot folder from SSD ( updated from repository kernel and new device trees installed there ) to eMMC and rebooted.
Now Xavier works like before.

So, it should be noted, that now Nvidia firmware updates doesn’t take into account if system was flashed with sudo ./flash.sh jetson-xavier external and reflashing from external PC with JetPack will be needed after update.
Systemd method should be resistant against updates as it doesn’t use bootloader features.
In all cases coping from /boot of SSD to /boot on eMMC will be required.

mdegans · June 27, 2020, 7:18pm

But today Ubuntu Update downloaded some updates from nvidia repository ( this and this ), requested to reboot and then Xavier failed to boot system, hanging on initrd step.

IMO it’s safer to just put commonly used stuff on a ssd and not the entire of /. I’m not sure how much benefit you get from having stuff like /etc on an ssd, since it’s mostly a bunch of text files.
My partition layout looks like:

nvme0n1      259:11   0 931.5G  0 disk
├─nvme0n1p1  259:12   0 119.2G  0 part /tmp
├─nvme0n1p2  259:13   0 323.9G  0 part /home
├─nvme0n1p3  259:14   0   250G  0 part /usr
└─nvme0n1p6  259:15   0 238.4G  0 part /var/lib/docker

Those are just ext4 partitions on a GPT disk, but you may wish to use something fancier like LVM or btrfs with subvolumes for easier resizing.