I can see that: kernel/drivers/nvme/host/nvme.ko has been loaded into the kernel
If you have the same /lib/modules and /lib/firmware (should be the case if you’ve cloned) on nvme and mmc0, it might work.
Otherwise, you may have to build a kernel embedding this driver as built’in.
Additional note: after cloning your mmc0p1 image into nvme0n1p1, you may use gparted to check fs and correct filesystem size to partition size. It is important there to give a new UUID…having still the same as MMC0p1 may lead to problems with udev being confused (symptom is shutdown fails with red screen of the death).
Additional note: If reading the NVMe is required in order to load a partition, then the module must be loaded before this can be done. If the module is on NVMe, then NVMe must be mounted before NVMe can be loaded. It’s a bit like a cartoon of a dog chasing his own tail (sometimes entertaining, but only a very clever or very tired dog will figure this out).
One solution is to build NVMe directly in to the kernel. No module, it’s always there.
Another solution is to have an initrd file system with the module in it in “/boot”…and run from this until the module is loaded, followed by a pivot_root to the actual NVMe.
The reason why the “/lib/modules/$(uname -r)/” module won’t work is that U-Boot doesn’t know to put this in RAM. U-Boot only puts the kernel and device tree in RAM (it will put initrd in RAM if it is told to…which is very convenient if that RAM holds the NVMe module).
i’m really hoping someone can post the exact steps to get this working to help the broader community…and perhaps Nvidia include it as an option in the future jetpack releases
Just use the existing “/proc/config.gz”, then add the NVMe feature as integrated instead of as a module. Put the new kernel in place (probably with an alternate name and boot entry so you can keep the original).
Do you know if there’s an article around that shows one how to do that? sorry new to linux
Honey_Patouceul, could you show us after you successfully mount nvme0n1p1 as rootfs,
$ df -h
The result?
Really appreciated.
Hey all…
I tried Honey_Patouceul’s suggestion of “cat /proc/cmdline” then appending the contents to my /boot/extlinux/extlinux.conf to look like this:
TIMEOUT 30
DEFAULT NVME
MENU TITLE p2771-0000 eMMC boot options
LABEL NVME
MENU LABEL NVME kernel
LINUX /boot/Image
APPEND root=/dev/nvme0n1p1 rw rootwait console=ttyS0,115200n8 console=tty0 OS=l4t fbcon=map:0 net.ifnames=0 memtype=0 video=tegrafb no_console_suspend=1 earlycon=uart8250,mmio32,0x03100000 nvdumper_reserved=0x2772e0000 gpt tegra_fbmem2=0x800000@0x969e9000 lut_mem2=0x2008@0x969e6000 tegraid=18.1.2.0.0 tegra_keep_boot_clocks maxcpus=6 boot.slot_suffix= boot.ratchetvalues=0.2.1 androidboot.serialno=0323817014277 bl_prof_dataptr=0x10000@0x277040000 sdhci_tegra.en_boot_part_access=1 root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4
LABEL EMMC
MENU LABEL EMMC kernel
LINUX /boot/Image
APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4
But no luck…
I get “Waiting for root device /dev/nvme0n1p1…” during the boot and it just hangs
Cannot find firmware … retry after 1 second
Direct Firmware load for tegra18x_xusb_firmware failed with error -2
Failling back to user helper
“Waiting for root device /dev/nvme0n1p1…” – mostly, root cause is, kernel did not find or load the driver for the device, it cannot be recognized or initialized.
Sorry, I wrote too quickly yesterday, thinking that your kernel was able to drive nvme and your kernel module was just adding some features. As said, I have no experience with nvme. Sorry if I’ve added confusion.
Obviously, as pointed out by @Linuxdev, if nvme cannot be drived, no chance to get module from there.
If you have rootfs ready on nvme, last step would be building your kernel image with nvme support and then use this image for booting with rootfs on nvme by setting nvme-enabled kernel image name in extlinux.conf entry.
You can do this on natively on TX2.
I am away from my TX2 and won’t be able to test any advice before tomorrow.
Thanks Honey_Patouceul!
Please let us know how you go… Hopefully we can get this NVME issue solved once and for all :)
It might be useful to know that building kernels and kernel modules are among the most common questions in Linux. With Windows you get updates only from the provider of the software or through Windows update…under Linux you have access to everything, you just have to know how to “turn on” that feature (especially with embedded systems…they don’t enable everything a desktop does, they cannot afford to do so).
The kernel build itself is standard among all the different Linux systems. Many documents you see in tutorials apply equally to Jetsons. Installing the kernel differs because it uses U-Boot instead of GRUB (GRUB requires a BIOS…U-Boot is its own BIOS). Building the kernel for a Jetson only differs if you are cross-compiling (building on one system with another system as the target). If you build directly on the Jetson (and most versions can…R24.x was an exception), then you don’t even need to cross-compile. Most of the Jetson tutorials use cross-compile since many developers want to work on a fast host (the TX2 is unusually fast though…building on a host isn’t actually much of a speed gain if you set the Jetson clocks at max and enable the Denver cores).
The documentation download with the rest of the flash software downloads has a section on “kernel customization” which covers this. eLinux.org has a number of tutorials specific to the TX2, but do watch to see if the release the information covers is the same as which you are running (e.g., an older R24.2 release would have different instructions versus R28.1):
https://developer.nvidia.com/embedded/dlc/l4t-documentation-28-1
https://elinux.org/Jetson_TX2
https://elinux.org/Jetson_TX2#System_Tools
http://www.jetsonhacks.com/2017/03/25/build-kernel-and-modules-nvidia-jetson-tx2/
You will find that some of the custom scripts out there started in earlier L4T versions when compiling was more complicated.
With all of those tutorials there is an important concept, which although subtle and simple, is important. The kernel has certain features (most in fact) which are integrated directly in the kernel. Other features are modular (built as a module) and load or unload on demand (you can’t build everything as a module, there are limitations). When you are on a running system the command “uname -r” shows the complete version number of the kernel, and the kernel looks for modules at:
/lib/modules/$(uname -r)/
# example:
cd /lib/modules/$(uname -r)/
ls
The first part of “uname -r” is from the kernel source. The suffix is from a setting which is custom to whoever compiled the kernel: CONFIG_LOCALVERSION. If the “uname -r” is “4.4.38-tegra”, then the kernel is source version “4.4.38”, and someone set CONFIG_LOCALVERSION to “-tegra” prior to compiling it. This is useful because unless a kernel has the same configuration as the modules you don’t know the modules are actually compatible. If the resulting “uname -r” does not match where modules are installed to, then the kernel will not be able to find its modules. Since you are changing an integrated feature I suggest making a custom CONFIG_LOCALVERSION and just installing all of the modules the new alternate location.
Kernel compile also requires configuring the kernel. There are all kinds of features for all kinds of hardware which the Jetson does not use, and even which a desktop PC can’t use. A single kernel source is shipped and you pick features via a configuration. On a running Jetson the file “/proc/config.gz” is not a real file…it is actually an image in RAM you can copy somewhere else, gunzip it to uncompress it, and then look at it with a common text editor. This is an exact match to your running system (with one exception) and the best place to start. You will see references to making a default config (e.g., “make tegra18_defconfig”), but don’t do that if you have a “/proc/config.gz”…config.gz is better, it matches what you already have. You’d still have to copy the file to the correct place and rename it “.config”.
That one exception in being an exact match is that config.gz does not set “CONFIG_LOCALVERSION”.
Before starting, assuming it is Ubuntu (such as what Jetsons run), you’ll want to install package “libncurses5-dev” (“sudo apt-get install libncurses5-dev”). When you look at the tutorials there will references to configuration editing. This is done from one of several editors, libncurses5-dev makes it possible to use “make menuconfig” or “make nconfig”. From these you can correctly explore your configuration as it is and make edits or changes (without “libncurses5-dev” those menu edit targets don’t work…with “libncurses5-dev” you will see an editor pop up when you run the command).
In the case of installing realize you don’t need to flash anything. There is a file “/boot/extlinux/extlinux.conf”. This file is human readable text. There is a boot entry, and if you have more than one, then you can pick which one at boot time via serial console (life with kernel development and Jetson development is much safer and flexible if you have a serial console set up). Modules just get a copy to the correct structure in “/lib/modules/$(uname -r)/”. The kernel Image file gets copied (preferably with an updated name and edit to extlinux.conf) to “/boot”.
Note: Some systems use zImage, or uImage, or bzImage. This are variations of compression and other setup. The file Image gets built first before any of those. The Jetson doesn’t use those others, you just need file Image.
As intimidating as it might seem it turns out to be fairly simple if you just have a few setup steps done. Then you don’t really need to repeat those and you can have any feature you want which happens to be compatible with your system. Try building the kernel, cleaning up after a build (“make mrproper” even cleans out your “.config”, “make clean” cleans out most things but not all).
You may be interested in finding (within the driver package…which is something JetPack downloads) the “Linux_for_Tegra/source_sync.sh” script. This uses tags to download source, and for R28.1 or newer, this actually downloads some hardware files related to the Jetson which are not in the pure kernel source download. When you turn on certain features specific to the Jetsons you might need those extra files. Copy this script anywhere (or use it where it is…JetPack does), and run this to get kernel source (you can run this from on your Jetson if you copy it there):
./source_sync.sh -k tegra-l4t-r28.1
…this version will also append a “+” to the end of your expected “uname -r”. This might also build some extra subdirectories which “make clean” and “make mrproper” do not clean up after.
NVME block device support is probably not built’in in R27.1 kernel image, but it seems it is built’in in R28.2-DP:
gunzip -c /proc/config.gz |grep CONFIG_BLK_DEV_NVME
CONFIG_BLK_DEV_NVME=y
So you may upgrade your TX2 to L4T R28.2 and should be able to use it as explained above.
Note that I’ve edited post #20. Seems it is not a good thing to specify
tegra_fbmem2=0x140000@0x969e9000 lut_mem2=0x2008@0x969e6000
as kernel args although shown by R28.2-DP1 /proc/cmdline.
(https://devtalk.nvidia.com/default/topic/1030098/jetson-tx2/tx2-fails-to-boot-without-hdmi-monitor-with-rootfs-on-sata-ssd/post/5244358/#5244358)
approaching implementing boot from ssd nvme 4lanes pci-e
[url]https://devtalk.nvidia.com/default/topic/1036590/jetson-tx2/how-to-quot-adding-1pb-storage-to-jetson-quot-/?offset=2#5265982[/url]
I used the same config as honey said. Booted successfully.
I am on Jetson TX2 and L4t 4.3. Linux jtx2-desktop 4.9.140-tegra
for new guys: follow these :
- connect and format the nvme drive to ext4 and mount
- use this command to copy rootfs to nvme : sudo cp -ax / ‘/root/to/nvme’ && sync
- sudo gedit /boot/extlinux/extlinux.conf
- continue honey instructions ( quoted)
try to have access to serial console to better debug problems if existed.
1 Like