Booting TX2 with NVMe M.2 SSD as root filesystem

I am trying to boot the TX2 from a root filesystem inside an NVMe M.2 SSD drive. Initially, with the system fully booted from eMMC, can it can recognise the NVMe device. However, when I configured the NVMe as a root filesystem, it seems that the kernel doesn’t detect NVMe drive at early boot stage, leading to a freeze at [9 seconds uptime]

Waiting for root device /dev/nvme0n1p1...

Hardware

  • Jetson TX2 Developer Kit
  • Samsung PM961 128GB M.2 NVMe - plugged into the PCIe slot via an M.2 adapter
  • Jetpack 3.2

Here are the steps I tried.

First, the latest JetPack 3.2 (L4T R28.2) was installed. Then, I plugged in the SSD. It is recognised properly on a fully booted Ubuntu from the eMMC, so I paritioned it with ext4. Then, I copied the content of the root directory from eMMC to the SSD using rsync. I also tried to chroot into it and it worked okay.

I edited extlinux.conf to add an entry for a root filesystem on the SSD.

LABEL SSD
MENU LABEL eMMC kernel SSD root
LINUX /boot/Image
APPEND ${cbootargs} <b>root=/dev/nvme0n1p1</b> rw rootwait rootfstype=ext4

Since the current version of U-Boot does not support NVMe, the kernel image is still loaded from the eMMC, and only the root filesystem is loaded from the SSD.

I checked the kernel config and found that it has NVMe support built in, not as a module.

$ gunzip < /proc/config.gz | egrep CONFIG_BLK_DEV_NVME
CONFIG_BLK_DEV_NVME=y
# CONFIG_NVMEM is not set

I tried to investigate the kernel log using serial console in two different scenarios: normal boot from eMMC and modified boot with rootfs on SSD.

First, I checked the normal boot process that loads the kernel and rootfs from the eMMC. What I found is that the NVMe driver is loaded after the PCIe driver, and this happens at a later boot stage ~20 seconds uptime. This is much later time after the root filesystem has been mounted.

Second, when the rootfs is configured to be on the SSD, then the kernel fails to detect the NVMe partition. Hence the freeze.

<b>Boot from eMMC Kernel & rootfs</b>
<kernel loaded>
... 9 seconds ....
<eMMC is mounted at />
... 20 seconds ...
<PCIe driver loaded>
<NVMe driver loaded>
<SSD mounted at /mnt>

<b>Boot from eMMC Kernel & <u>SSD</u> rootfs</b>
<kernel loaded>
... 9 seconds ...
Waiting for root device /dev/nvme0n1p1...
... nothing happened ...

I had a look at the other threads before and there seems to be an exact same problem here.

In case of TX1, there seems to be a solution by modifying the initrd script. However, the TX2 doesn’t come with a pre-built initrd, so I am not sure where to start with this (and whether this is a potential solution or not).

Note that at this stage it is U-Boot which needs to read and support NVMe, not Linux. I do not believe U-Boot can do this, though perhaps there are options to rebuild U-Boot to do this. Once the Linux kernel and its NVMe drivers are loaded I’m sure Linux has no issue. Someone else may know if or how U-Boot could be rebuilt to support NVMe, but I couldn’t say if it is possible.

baug55,

I am checking the capability of NVMe driver in uboot. Thanks for you patience.

We believe this does not support in our bootloader. However, I found some 3rd party resources to enable nvme in bootloader. May this can help.

is that still an issue to boot from nvme pci connected?
How to call boot select menu at all? Presuming I mirrored the internal emmc to sdcard and to nvme - is there a way to somehow invoke the manual boot selection menu? I tried to add entries to /extlinux.conf but that just still boot and wont show the menu for selection of a boot source

Serial console is how you do that. If you interrupt early on you go to a command line and need to run “boot” to continue. A short time later you get a chance to interrupt again and pick any of the extlinux.conf entries. I suspect that whatever you’re trying to do it is just a matter of timing when you interrupt U-Boot.

Also, if you interrupt at the boot choice selection screen, then you have to hit the backspace to erase whatever key you interrupted with, and then type in the number of the entry.

Note that if the boot selection entry isn’t there, then you are probably defaulting to a rootfs from a different partition than the one you think it is using. E.G., you might be using the extlinux.conf of mmcblk0p1, or of mmcblk1p1 instead.

got it