How to Boot from NVMe SSD?

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
,
1

Hi,

I’ve installed a Samsung NVMe 500Gb SSD to the M.2 type M slot of the Xavier and I was able to format the partition and mount and copy the root files over as described by JetsonHacks[1] but I noticed that the file /boot/extlinux/extlinux.conf is missing so I’m not sure how to proceed.

Any ideas on how to configure it to boot from the NVMe SSM? Thanks

[1] Develop on SSD - NVIDIA Jetson TX1 and Jetson TX2 - JetsonHacks

2

In my case [I used NVME PCI-E SSD], I did not manage to boot from it. But, I approached TX2, and may be the situation got changed with Xavier hardware and M.2 type M slot introduction.

3

Xavier does not use uboot and boot flow is different, see this post:

i.e., rootfs path is specified in bootarg of Xavier dtb .

and this post:

Custom kernel without uboot (Solved) - Jetson AGX Xavier - NVIDIA Developer Forums

i.e., you need to generate new “boot.img” and flash it to emmc partition 29.

You can try change path of rootfs in cmdline from /dev/mmcblk0p1 to /dev/nvmenx assuming nvme driver is already loaded during Xavier boot process.

This approach should work for SATA and SD card but I’m not sure about nvme.

4

I was planning to boot from NVMe until I tested read/write speed of the Xavier SD vs the WD Black NVMe. SD is actually faster, so I’m leaving it as boot drive, using the NVMe for data partition. Probably will bind mount /home to the NVMe.

5

NVME is definitely faster than internal EMMC and SD card.
Using "hdparm -t -T ", I got:

/dev/mmcblk0p1 (internal EMMC): Bufferd reads = 292MB/s

/dev/mmcblk1p1 (Sandisk Ultra U1 SD card): Bufferd reads = 81MB/s

/dev/nvme0n0 (WD Black NVME M.2): Bufferd reads = 879MB/s

Using “gnome-disks” (default 10MB sample size) I got 1.7GB/s read speed and 547MB/s write speed for WD Blck M.2 SSD. I got 320MB/s read speed for internal EMMC (gnome-disk can’t unmount EMMC for write tests).

6

I was measuring speeds by the dd method reading from /dev/zero and writing to /dev/null at various bs sizes (NVMe vs the internal EMMC) (not SD sorry for confusion. In any case, I don’t think much will be shaved off the boot time booting from the NVMe.

Booting in Power mode 2 might give more improvements if the boot time is limited by CPU rather than storage speed. I haven’t measured boot time yet. Only running headless so far.

7

I was trying to boot from nvme but for now on I’ve been just able to configure it and use as a normal external disk.

If you have a look at flash.sh there is no NVME entry for <root_device>. It seems that it only supports eMMC, SDCARD, USB Device or NFS. Only changing bootargs to point to nvme0n1p1 is not working for me.

Can someone please tell me why is not possible to boot from nvme SSD? Is it a hardware limitation, no uboot support, whatever else?

1 Like
8

I think it is because cboot does not implement this part.

9

And there is any willing by Nvidia to support this feature in further releases? In my humble opinion, it would be more than great if you consider it.

1 Like
10

Nvidia support,
Doesn’f See like the Xavier supports
Boot from nvme right now.
Any way to do this?
What about boot from PCIE?

11

Not sure for Xavier, but IIRC for TX2 I had to build a kernel with CONFIG_PCI_TEGRA=y (and CONFIG_BLK_DEV_NVME=y) instead of modules for booting a Linux kernel mounting its rootfs from NVME SSD (It was using mmc0 for early boot and extlinux.conf for kernel args and image, though).
Has anyone tried that ?

12

Dear all,

How to boot AGX Xavier that just came out from the box?
Is it need sdcard to boot?
When I put AGX Xavier with a HDMI screen, a keyboard and a mouse then power it up, it does not appear anything on the screen. Keyboard caps lock does not on/ off.

I have followed https://developer.download.nvidia.com/assets/embedded/secure/jetson/xavier/docs/jetson_agx_xavier_developer_kit_user_guide.pdf

How to Install JetPack
Installing JetPack to your developer kit requires these steps, which are detailed in the
sections below:
 Download JetPack installer to your Linux host computer.
 Connect your developer kit to the Linux host computer.
 Put your developer kit into Force Recovery Mode.
 Run JetPack installer to select and install desired components.
Download Installer to the Host Computer
You must have a Linux host computer to run JetPack installer and flash the developer
kit. Supported host operating systems are:
 Ubuntu Linux x64 Version 18.04 or 16.04
Download the latest JetPack installer to the Linux host from the JetPack page on the
Jetson Developer Site.
Note: The installer can flash and update software on a target Jetson device, but it
cannot not run directly on that device. Whether or not a Jetson device is
present, you can use JetPack installer to update software on the Linux host.
Connect Developer Kit to the Host Computer
 Use the included USB cable to connect the Linux host computer to the front USB
Type-C connector on the developer kit.
 Connect a display, keyboard, and mouse to your Jetson AGX Xavier Developer Kit
(see Physical Configuration Instructions, above).
 Connect the developer kit and Linux host computer to the same network.
 Connect the included AC adapter to the developer kit’s DC jack.
Put Developer Kit into Force Recovery Mode
The developer kit must be in Force USB Recovery Mode (RCM) so the installer can
transfer system software to the Jetson AGX Xavier module.
1. Connect the developer kit as described above. It should be powered off.
2. Press and hold down the Force Recovery button.
3. Press and hold down the Power button.
4. Release both buttons.
Run the Installer
JetPack installer includes a Component Manager, allowing you to choose what to install
on the Linux host computer and/or the Jetson Developer Kit.

I have run the SDK Manager on my ubuntu 18.04 host, at step 03, it gives:

Could not connect to the target. Verify that:
1. The device is connected to this host machine with a USB cable.
2. Ubuntu 'System configuration wizard' is completed on the device. (How do I make this one?)
3. Jetson's Ubuntu OS is up and running (I cannot boot the AGX Xavier).
When ready, click 'Flash' to continue.

I assume the above error message that I can boot the AGX Xavier and it ready to receive flash from SDM Manager?

Please help me. Thank you very much in advance.

Warmest regards,
Suryadi

13

No SD card needed for boot (and probably should be avoided while flashing).

When an Xavier is in recovery mode with the correct USB to host, then the following will show something:

lsusb -d 0955:7019

(0955 is the NVIDIA USB registry manufacturer ID, and 7019 is the Xavier product ID)

Basically anything which brings up power or cycles power while the recovery button is held down will put the Xavier in recovery mode. At this point the Xavier is a custom USB device and the driver package running on the host PC will be able to work with the device. People using a VM usually have problems due to the way USB is passed through, so that would be the first thing to ask: Is this a VM? I am going to guess “no” since even a VM usually sees the device at the start.

14

Hello,
It maybe not the answer you all want, but I’d like to share my solution of this problem.
Since it seems it is difficult to mount SSD partition for booting a Xavier device,
I took a way to mount the original filesystem on eMMC as readonly and overlay another read-writable filesystem located on NVMe SSD.
Although it slows down read/write speed a bit, the volume of system filesystem is expanded to the volume of SSD (for my Xavier, it is 1TB.).

Please have a look at my Github repo, it includes all script files and one-liner script to setup the system I proposed.

Hope it will help you!

15

P.S.

The script I posted above actually has one problem on shutdown, where unmounting the overlayed partition fails (though it works without any problem as for now).
For anyone who has any advice or answer, please create a pull request to my repo or just posting to this thread.
It’d definitely help all in the world! ;)

16

Hi,
I have modified furushchev’s solution in “true” systemd way and become able to mount SSD drive as rootfs without overlay and reboot problems.

  1. Xavier was flashed and software was installed using SDKManager.
  2. Rootfs from eMMC was copied to SSD ( with one ext4 partition in my case): 
    #!/bin/bash
sudo mount /dev/nvme0n1p1 /mnt
sudo rsync -aAXv / --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} /mnt
  3. Then we need systemd unit, loading just after unmounting initrd image and mounting real rootfs, before any other targets running with systemd from real rootfs. Something like following: setssdroot.service: 
    [Unit]
Description=Change rootfs to SSD in M.2 key M slot (nvme0n1p1)
DefaultDependencies=no
Conflicts=shutdown.target
After=systemd-remount-fs.service
Before=local-fs-pre.target local-fs.target shutdown.target
Wants=local-fs-pre.target
ConditionPathExists=/dev/nvme0n1p1
ConditionVirtualization=!container

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/setssdroot.sh

[Install]
WantedBy=default.target

    I added condition to start only if SSD is installed ( /dev/nvme0n1p1 exists).

    setssdroot.sh:

    #!/bin/sh
NVME_DRIVE="/dev/nvme0n1p1"
CHROOT_PATH="/nvmeroot"

INITBIN=/lib/systemd/systemd
EXT4_OPT="-o defaults -o errors=remount-ro -o discard"

modprobe ext4
#modprobe fuse

mkdir -p ${CHROOT_PATH}
mount -t ext4 ${EXT4_OPT} ${NVME_DRIVE} ${CHROOT_PATH}

cd ${CHROOT_PATH}
/bin/systemctl --no-block switch-root ${CHROOT_PATH}
  4. New target was installed with following command: 
    #!/bin/sh
sudo cp setssdroot.service /etc/systemd/system
sudo cp setssdroot.sh /sbin
sudo chmod 777 /sbin/setssdroot.sh
systemctl daemon-reload
sudo systemctl enable setssdroot.service

As a result, after reboot systemd mounts /dev/nvme0n1p1 as “/”. We are working with system (except kernel and initrd) from SSD and have “recovery” system on eMMC.

4 Likes
17

If rootfs from SSD is mounted, you will see SD drive on desktop (Ubuntu gives SD icon to eMMC).
systemd-analize result attached.

Now we need to find how to load new kernel from SSD with flashed kernel’s kexec :).

systemd-analyze.png
systemd-analyze.png1962×10690 937 KB

18

it is really works :)
Thank a lot , @crazy_yorick

19

3 posts were split to a new topic: Does the Nvidia Xavier support NVME GEN4 SSD?

20

Can some one from Nvidia please confirm/provide the supported way of doing this?

Thank you.