Can not mount disk on bootup using fstab

Hello

I have an AGX Orin board with a custom kernel that I compiled. Everything works correctly.
I added an SSD Nvme on my AGX Orin board, and added an entry in /etc/fstab to mount it on boot::

# <file system> <mount point>             <type>          <options>                               <dump> <pass>
/dev/root            /                     ext4           defaults                                     0 1
/dev/nvme0n1	/opt/mnt	   ext4           defaults                                     0 0

However, it is not mounted after bootup.
When checking dmesg logs I can see that it has issues loading nvme_core module in the beginning, but then it works:

[    8.028903] nvme_core: disagrees about version of symbol module_layout
[   18.346787] nvme 0004:01:00.0: Adding to iommu group 56
[   18.347067] nvme nvme0: pci function 0004:01:00.0
[   18.347125] nvme 0004:01:00.0: enabling device (0000 -> 0002)
[   19.065480] nvme nvme0: allocated 128 MiB host memory buffer.
[   19.067549] nvme nvme0: 12/0/0 default/read/poll queues

It seems that the nvme_core it is trying to load on bootup is not the same as the one it is loading after the system boot finishes.

I had similar issues with Orin NX & Nano. The problem was that it was using the module in initrd which is different from the one compiled in /lib/modules/(uname -r)

Is it a similar issue ?

Best

It sounds like CONFIG_LOCALVERSION was incorrect, or the actual software release of the kernel source was not a match to the previous version. initrd could be related, but while running, what do you see from “uname -r”? Did you replace the Image file? Did you replace the modules?

A bit of trivia is that the initrd does not have the kernel in it, but as you have noticed, it has some modules in it. Those modules would have to match the running kernel, but unless those modules are needed for boot (e.g., for loading the rootfs on /…which is not what you are doing), then it won’t matter. An example common use of an initrd is to allow a new filesystem type (e.g., XFS is an example alternate to ext4) that the bootloader itself does not understand. During boot the kernel might be made available as binary data, but modules must be loaded from a filesystem, and if the module itself is what makes the filesystem readable (and the bootloader does not itself understand that filesystem), then you would have a paradox with needing to load XFS to be able to understand XFS. However, when the load is complete, the initrd verison of the rootfs performs a pivot, and the actual disk “/” becomes the rootfs. Other disks, such as those mounted on /opt/mnt, never know about the initrd, nor would a module failure in the initrd cause such a failure. “/opt” is not part of boot.

This suggests you did not build your kernel to be able to load existing modules on the actual rootfs. CONFIG_LOCALVERSION not matching is the number one cause of this (and “uname -r” will tell us what it is). Using the wrong kernel source would also cause this, or using a conflicting configuration would make module load fail.

Thanks for your answer @linuxdev .

The thing is that the kernel is able to load all compiled modules, including nvme_core, but after boot has completed.

What I mean is that when the system is running, I can see nvme_core loaded correctly. However, I have this error saying that it does not match module_layout at the very beginning of the boot process.

I found an interesting post on this error:
https://unix.stackexchange.com/questions/768570/insmod-fails-module-does-not-match-running-kernel-but-module-has-been-compiled

According to that (I have not verified):

I figured out what was causing the symbol error. That was coming from not loading nvme-common first. nvme-core depends on nvme-common so you have to load it first and doing so gets me past that error.

As far as I know udev does not manage this unless you are using a USB external disk. I also don’t know if this is correct or not, but I suspect it is correct:
https://paulgraydon.co.uk/posts/2009-01-26-changing-the-order-modules-load-in-ubuntu/

Be very careful about changing any driver via altering if it is in the form of a module versus integrated (if you swap this you risk a non-booting system if you do it incorrectly). It is much safer to use a method that gets modules to load in the order you want versus rebuilding a kernel.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.