Booting from NVMe on TX2

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

I am trying to boot from an NVMe SSD on a TX2 board running 27.1, but I’m not having any success. I have tried the following methods:

  • Simply modifying the extlinux.conf file on the emmc so that root points to a partition on the nvme 'root=/dev/nvme01n1p1'. 
    TIMEOUT 30
DEFAULT primary

MENU TITLE p2771-0000 eMMC boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      APPEND fbcon=map:0 net.ifnames=0 console=tty0 OS=l4t consol=ttyS0,115200n8 memtype=0 video=tegrafb no_console_suspend=1 earlycon=uart8250,mmio32,0x03100000 gpt tegraid=18.1.2.0.0 tegra_keep_boot_clocks maxcpus=6 android.kerneltype=normal adroidboot.serialno=0335115020673 vpr_resize root=<b>/dev/nvme0n1p1</b> rw rootwait
  • Modifying the initrd as outlined in https://devtalk.nvidia.com/default/topic/1002763/jetson-tx1/auvudea-j120-pcie-nvme-r24-2-1-booting-from-ssd/post/5122567/, and then adding a line in the extlinux.conf file above to point to the new initrd. 
    INITRD /boot/initrd_new

Both of these methods cause the boot sequence to hang, requiring re-flashing of the TX2.

Does anyone know a way around this?

2

How is the drive connected? SATA? USB? PCIe? M.2?

3

It’s a 1 TB Samsung NVMe SSD 960 Pro M.2 connected through PCIe.

4

When you boot normally and have this drive in the M.2 slot, does it show up under lspci? Or lsblk? If not, then this can be added. If added as a module you may require an initrd…directly integrated into the kernel would simplify things.

If the drive shows up and can be used in a non-boot application you will know the kernel is not the issue. Once that is known you will need to determine if NVMe over PCI is supported by U-Boot (which I can’t answer). Perhaps someone here knows if NVMe over PCI is supported by U-Boot.

5

When booted normally (emmc as root), the NVMe drive shows up using both lspci and lsblk, and it functions normally. It seems like the issue is with U-Boot. In the Nvidia documentation, they mention that,
"There are four example extlinux.conf files provided in the L4T release for each supported board:

  • extlinux.conf.emmc
  • extlinux.conf.sdcard
  • extlinux.conf.usb
  • extlinux.conf.nfs"
    • I don't know if the absence of NVMe means it's not supported or just not supplied as an example...
    6

    I don’t know specifically about NVMe in U-Boot, but there is a related issue which might aid understanding the issue.

    SATA drives under USB are supported as a boot mechanism, but not SATA drives on the SATA port…this is because the port controller itself is not supported under U-Boot, but USB2 and SATA protocols are supported. Similarly, some of the embedded systems connect various controllers through USB instead of over PCI…so those controllers function only if USB runs.

    If a controller works on USB in U-Boot, it will only work in USB2 mode because USB3 requires a different driver (there are two controllers, one for USB2 and older, another for USB3…the port is routed to the correct USB controller). Since USB3 mode is not supported in U-Boot, if you switch from USB2 mode to USB3 mode as the Linux kernel takes over any USB SATA drive will be lost…you’d have to be in USB2 mode under both U-Boot and Linux in order for continuity of service to be achieved while transitioning from U-Boot to Linux.

    I do not know if the M.2 port support for PCI and NVMe is supported within U-Boot…someone else will have to answer this, along with what it might take to add that support if it isn’t there. You’ve validated the Linux side as working, so it is this U-Boot infrastructure which needs clarification.

    7

    My TX2 can boot from SD card, eMMC and SATAII port, but not from PICe NVMe (using PCIe to M.2 adapter).

    8

    It may be that it is a case of U-Boot needing to see the controller and support the drive type. Under Linux, if you boot normally and only look at the drive as extra storage, do you see the controller with “lspci”? Do you see the drive with “lsblk”?

    9

    I think that is because current kernel does not support PICe as internal module, but loaded as external after booting process finished.
    When I booted from SATA2 (or eMMC), NVMe SSD is recognized and mounted.
    But when I boot from NVMe SSD, it hangs, it cannot find PICe driver!

    10

    $ lsblk
    NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
    sda 8:0 0 111.8G 0 disk
    ├─sda1 8:1 0 100.3G 0 part /
    └─sda2 8:2 0 11.5G 0 part [SWAP]
    sdb 8:16 0 1.8T 0 disk
    └─sdb1 8:17 0 1.8T 0 part /media/DATA
    mmcblk0rpmb 179:24 0 4M 0 disk
    mmcblk0boot0 179:8 0 4M 1 disk
    mmcblk0boot1 179:16 0 4M 1 disk
    mmcblk0 179:0 0 29.1G 0 disk
    ├─mmcblk0p1 179:1 0 28G 0 part /media/eMMC
    ├─mmcblk0p2 179:2 0 4M 0 part
    ├─mmcblk0p3 179:3 0 256K 0 part
    ├─mmcblk0p4 179:4 0 512K 0 part
    ├─mmcblk0p5 179:5 0 3M 0 part
    ├─mmcblk0p6 179:6 0 2K 0 part
    ├─mmcblk0p7 179:7 0 604K 0 part
    ├─mmcblk0p8 259:0 0 500K 0 part
    ├─mmcblk0p9 259:1 0 2M 0 part
    ├─mmcblk0p10 259:2 0 6M 0 part
    ├─mmcblk0p11 259:3 0 2M 0 part
    ├─mmcblk0p12 259:4 0 128M 0 part
    ├─mmcblk0p13 259:5 0 32M 0 part
    ├─mmcblk0p14 259:6 0 64M 0 part
    ├─mmcblk0p15 259:7 0 512K 0 part
    ├─mmcblk0p16 259:8 0 256M 0 part
    └─mmcblk0p17 259:9 0 646.7M 0 part
    nvme0n1 259:10 0 111.8G 0 disk
    └─nvme0n1p1 259:11 0 111.8G 0 part /media/NVMe
    $ lspci
    00:01.0 PCI bridge: NVIDIA Corporation Device 10e5 (rev a1)
    01:00.0 Non-Volatile memory controller: Device 1987:5007 (rev 01)

    11

    My sda is connected to SATA II port, booted as / (Samsung 2.5 inch SSD)
    My sdb is connected to USB 3.0 port, as data storage (Seagate 2.5 inch 2TB HD)

    I really hope I can boot from nvme0n1, which I can use it as boot-backup of sda.

    or boot from nvme0n1 as /, sda as backup.

    12

    Assuming U-Boot works with nvme (and I don’t know if it does), then the module idea as cause is probably correct. You could start with “/proc/config.gz” and compile a new kernel matching this other than having the required nvme being integrated instead of as a module. Alternatively (and this would be far more work) you could create an initial ramdisk and use the INITRD in extlinux.conf to load this.

    13

    “you could create an initial ramdisk and use the INITRD in extlinux.conf to load this” --.
    that is the path I want to go.

    But I cannot find any sample initial ramdisk working with current kernel.

    Could you advise me? really appreciated.

    14

    I have not created an initrd from scratch. Basically you need the bare bones init files which will be required for execution (the list of which I do not know), plus the module directory with the particular modules. When initial load is complete it performs a pivot_root to the real file system.

    As for the steps to create or examine an initrd, basically this should help:
    [url]https://devtalk.nvidia.com/default/topic/1011027/jetson-tk1/jetson-tx1-initrd/post/5157554/#5157554[/url]

    There are several custom initrd web URLs, but most center on grub and x86 (typically live DVDs). The content after grub is done would probably be the same (we use U-Boot, x86 uses GRUB…initrd is only loaded in RAM by U-Boot and other than the mechanics of uncompressing it matter to U-Boot…the rest is a Linux requirement).

    Here is the web search I used, though it isn’t specific to a Jetson:
    [url]linux ubuntu "u-boot" custom initrd - Google Search

    Here is the kernel.org doc:
    [url]https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt[/url]

    For a more specific example I suggest you download the R24.2.1 driver package. The “Linux_for_Tegra/bootloader/” directory contains file “initrd”, which during flash is copied directly to the “/boot” of the rootfs. The information of the first URL will give you a way to unpack and examine the content of this initrd. This particular initrd will not work in R28.1 or R28.2 since it is for a different kernel, but the structure should be close to what you want if you copy instead from the R28.1 or R28.2 (whichever you are using). If you run normally and mount an ext4 partition from the NVMe, and then run “lsmod”, then you will see which modules are required. Name this initrd with the “INITRD” entry of extlinux.conf and it should load this prior to starting the kernel. Try first without changing rootfs just to see it work. Once it works with no changes you should be able to rename the “rootfs=” to a new partition. The initrd itself and extlinux.conf will have to be on the mmcblk0p1 partition, but the rest of rootfs will end up on the NVMe.

    In some of the more recent releases you will see an INITRD entry, and an initrd file, but the initrd file is zero bytes…it has no content. You should be able to drop your initrd in, but I strongly suggest giving it a new name and leaving the default zero byte version untouched in case of the need to use the original kernel for testing.

    You do want to try to keep content minimal. In the 32-bit version space was extremely limited, and although you have much more room in 64-bit ARM the combined content of both initrd and modules must still be within the 128MB direct branch limit (I’m not sure how much is reserved for modules and how much is reserved for initrd…perhaps it is partitioned at 64MB but I’ve never checked.

    If you create a log via serial console of a normal boot prior to working on the initrd, and then a log of any initrd boot, you can more or less “diff” the boot logs to use as a comparison while figuring out what the content needs to be.

    15

    Thanks a lot, I will follow the document and try it later on.

    16

    Hello

    Has anyone successfully accomplished this and could share a step-by-step guide to help those of us (including me) stuck!
    I too have a Tx2 /w Samsung NVMe PRO + PCIE to M.2 Converter.

    Running Jetpack 3.2

    lspci:
    00:01.0 PCI bridge: NVIDIA Corporation Device 10e5 (rev a1)
    01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a804

    lsblk:
    nvme0n1 259:20 0 477G 0 disk
    └─nvme0n1p1 259:21 0 477G 0 part

    So i can mount it fine inside of ubuntu… Read/Write etc… But can’t successfully boot off the NVMe drive.

    17

    One thing I don’t know about is if U-Boot itself supports NVMe…I’m pretty sure this version does not. You wouldn’t need to support NVMe in U-Boot if “/boot” (and extlinux.conf) are on the eMMC (and initrd can provide module access in support of NVMe after U-Boot is done and Linux is starting, but the initrd too would need to be on eMMC).

    Is your “/boot” on eMMC? Is your current “root=/dev/mmcblk0p1”?

    18

    hey @linuxdev

    Yea currently it’s all on eMMC…

    mmcblk0 179:0 0 29.1G 0 disk
    ├─mmcblk0p1 179:1 0 28G 0 part /

    19

    Are any of the NVMe functions of the kernel built as a module? If that is the issue, then building a new kernel with that directly built in instead of as a module would be a simple solution. Making an initrd is not simple.

    20

    I have no personal experience with nvme on Jetson, but what you may be able to acheive is:

    • early boot from MMC0
    • u-boot from MMC0 reading extlinux and Linux images from MMC0p1/boot
    • booting a Linux image from MMC0p1, just passing root=/dev/nvme0n1p1 to the booting Linux kernel should be ok.
      With R28.2-DP, I would use an entry like this in MMC0p1:/boot/extlinux/extlinux.conf:
    LABEL NVME
    MENU LABEL NVME kernel
    LINUX /boot/Image
    APPEND root=/dev/nvme0n1p1 rw rootwait console=ttyS0,115200n8 console=tty0 OS=l4t fbcon=map:0 net.ifnames=0 memtype=0 video=tegrafb no_console_suspend=1 earlycon=uart8250,mmio32,0x03100000 nvdumper_reserved=0x2772e0000 gpt tegraid=18.1.2.0.0 tegra_keep_boot_clocks maxcpus=6 boot.slot_suffix= boot.ratchetvalues=0.1.1 androidboot.serialno=0334916010240 bl_prof_dataptr=0x10000@0x277040000 sdhci_tegra.en_boot_part_access=1 root=/dev/nvme0n1p1 rw rootwait rootfstype=ext4

    [EDIT: removed args tegra_fbmem2=0x140000@0x969e9000 lut_mem2=0x2008@0x969e6000 that may be causing problems, especially when no HDMI monitor is connected at boot.]

    You may run

    cat /proc/cmdline

    to get kernel args from your running kernel and just replace mmcblk0p1 with nvme0n1p1.

    • Booting kernel will mount nvme0n1p1 as rootfs (this kernel seems to be able to drive your nvme disk). Of course, you would have cloned an image of L4T rootfs there before.

    With serial console and host, you can try this alternate config; and if it works fine, set it as default in MMC0p1:/boot/extlinux/extlinux.conf.