XFS filesystem as kernel module doesn't work

Autonomous Machines Jetson & Embedded Systems Jetson Nano
#1

I compiled XFS and F2FS as kernel modules on my Jetson Nano. F2FS works like a charm, XFS doesn’t load and I am at a loss why that is the case.

Here is exactly what I did:

cd /media/data

uname -a
# 4.9.140

cat /etc/nv_tegra_release
# R32.4.3 210

wget https://developer.nvidia.com/embedded/L4T/r32_Release_v4.3/Sources/T210/public_sources.tbz2

tar jxvf public_sources.tbz2
cd Linux_for_tegra/source/public
tar jxvf kernel_src.tbz2
mv kernel /media/data
cd /media/data
rm -rf Linux_for_Tegra
cp /proc/config.gz .
zcat config.gz > kernel/kernel-4.9/.config
cd kernel/kernel-4.9
vim .config
# CONFIG_LOCALVERSION="-tegra"
# CONFIG_XFS_FS=m
# CONFIG_F2FS_FS=m
make prepare
# Accepted all defaults
make modules
sudo cp fs/xfs/xfs.ko /lib/modules/4.9.140-tegra/kernel/fs/xfs/
sudo cp fs/f2fs/f2fs.ko /lib/modules/4.9.140-tegra/kernel/fs/f2fs/
sudo depmod
sudo modprobe xfs
sudo modprobe f2fs

f2fs loads without error, xfs on the other hand gives this error message:

sudo modprobe -v xfs
# insmod /lib/modules/4.9.140-tegra/kernel/fs/xfs/xfs.ko
# modprobe: ERROR: could not insert 'xfs': Unknown symbol in module, or unknown parameter (see dmesg)

dmesg
# Relevant lines from dmesg
#[..] xfs: Unknown symbol iomap_file_buffered_write (err 0)
#[..] xfs: Unknown symbol iomap_truncate_page (err 0)
#[..] xfs: Unknown symbol iomap_file_dirty (err 0)
#[..] xfs: Unknown symbol iomap_page_mkwrite (err 0)
#[..] xfs: Unknown symbol iomap_fiemap (err 0)
#[..] xfs: Unknown symbol iomap_zero_range (err 0)

For completeness sake I’d like to mention two additional things:

  1. I wasn’t quite sure what the outlier is: The fact that F2FS does work or the fact that XFS doesn’t. I chose another random module to try out. I decided for MINIX FS, compiled it as above and it loads without error.

  2. I already did this whole dance a couple of weeks ago when I still was running R32.4.2 210. I used wget https://developer.nvidia.com/embedded/L4T/r32_Release_v4.2/Sources/T210/public_sources.tbz2 to download the kernel sources. The outcome was exactly the same.

#2

I have not checked what prerequisites are needed for XFS, but if you edited this manually, then perhaps one of the dependencies are missing. CONFIG_LOCALVERSION has no dependencies, and I mostly edit this with vim as well, but XFS could be more complicated. I’d suggest verifying with either “make nconfig”, or looking at the specific Kconfig file, that dependencies are not missing (unless you used an option editor to make the change and were only illustrating the change with vim).

This is evidence (more or less a “smoking gun” proof) that a dependency is not met:

# modprobe: ERROR: could not insert 'xfs': Unknown symbol in module, or unknown parameter (see dmesg)

Your procedure was almost correct. I suggest repeating it, but using a config editor for the enable of XFS and anything other than CONFIG_LOCALVERSION. Editors are aware of dependencies in the Kconfig.

1 Like
#3

Thank you, this is a good idea. Unfortunately in this case I think XFS’ prerequisites seem to me to be fulfilled.

I created a new .config with make nconfig and it had exactly the same modules selected as my manually modified version. There were no additional modules selected. The module docs shown in nconfig for XFS say:

 |         | Symbol: XFS_FS [=m]                                                 
 |         | Type  : tristate                                                   
 |         | Prompt: XFS filesystem support                                     
 |         |   Location:                                                        
 |         |     -> File systems                                                 
 |         |   Defined at fs/xfs/Kconfig:1                                       
 |         |   Depends on: BLOCK [=y] && (64BIT [=y] || LBDAF [=n])           
 |         |   Selects: EXPORTFS [=y] && LIBCRC32C [=m] && FS_IOMAP [=y]

So BLOCK, 64BIT, EXPORTFS and crucially FS_IOMAP are compiled in. LIBCRC32C is a module but lsmod tells me that libcrc32c is loaded.

Now the dmesg output hints at a problem with FS_IOMAP, but /proc/config.gz clearly says FS_IOMAP=y. So I’m still stumped…

#4

What are the exact “symbols” which it says are still missing? What does lsmod show at that time?

1 Like
#5

lsmod output:

Module                  Size  Used by
minix                  37956  0
binfmt_misc            13103  1
f2fs                  502392  0
libcrc32c               1721  0
btrfs                1156495  1
xor                     7364  1 btrfs
raid6_pq               92714  1 btrfs
zram                   26166  4
spidev                 13282  0
nvgpu                1579891  18
bluedroid_pm           13912  0
ip_tables              19441  0
x_tables               28951  1 ip_tables

You can see that minix and f2fs loaded;-)
Missing symbols are the ones from the dmesg output from above.
They all start with iomap and I probably should have said that the list is complete. When I do sudo modprobe xfs I get only the six lines I listed no other additional complaints.

#6

When you used the config editor this should have triggered enabling other modules or kernel features for these:

…did you install those iomap modules prior to loading the XFS module? If the features were enabled as integrated (instead of being in the format of a module), did you replace the entire kernel?

Once those were in place, did the list of missing symbols not change?

1 Like
#7

When you used the config editor this should have triggered enabling other modules or kernel features for these:

FS_IOMAP is y in the config I got out of /proc/config.gz and which I used as the basis for my make nconfig configuration, so given the prerequisites for XFS I wouldn’t expect it to select any additional modules or kernel features.

did you install those iomap modules prior to loading the XFS module?

No, I didn’t, as they should be compiled into the kernel.

If the features were enabled as integrated (instead of being in the format of a module), did you replace the entire kernel?

No I didn’t change the kernel. This is a fairly fresh system.
The most drastic change it encountered was the upgrade from 32.4.2 to 32.4.3.
Apart from that I sudo apt installed a bunch of packages before and after the upgrade and built the three modules and then copied them into /lib/modules/… No system changes otherwise. I have another Jetson Nano which is still 32.4.2 and has had no changes apart from a few installed packages at all. Maybe I should try it there…

Once those were in place, did the list of missing symbols not change?

I don’t know how to get them in place. I guess that’s my problem. First of all, how do I find out for sure which module provides them? If it is indeed FS_IOMAP, why are the symbols not known to the XFS module while /proc/config.gz claims FS_IOMAP is compiled into the kernel? That’s the mystery.

I think I cannot load them as module as long as they are compiled into the kernel, so trying FS_IOMAP=m (if that is even possible) wouldn’t help.
Also I would like to avoid to replace my kernel image if I can avoid it.

#8

You are correct:

…thus there is probably something additional in need of config.

The approach of using the existing config.gz, and of checking what is present through that file, is the best approach. I will suggest that you continue with this.

On the other hand, missing symbols can be somewhat obscure. Actual code issues can prevent symbols from being used, and so can versioning issues which are unrelated to actual code. I do not know if this is the issue, but the following illustrates…

Normally an Image (the integrated kernel which will eventually load modules) is responsible for the output of the “uname -r” command. The kernel will only look for modules in “/lib/modules/$(uname -r)/kernel/”, and any module installed anywhere else will never be found. Conversely, a module compiled under a different “uname -r”, but purposely moved to the running system’s “/lib/modules/$(uname -r)/kernel/”, would normally be refused loading due to mismatched versions (there are ways to make this possible, those methods are not recommended).

One of the most common mistakes for someone new to kernel compiling is to not set up the matching “CONFIG_LOCALVERSION” (should normally be what you already set it to, CONFIG_LOCALVERSION="-tegra"). There is actually another location which can (unknown to the developer) alter this by adding a “+” to the end of the result of an otherwise perfect “uname -r” match. Within the kernel itself is a file, “scripts/setlocalversion”. If that automatically appends a “+”, then your “uname -r” match will actually fail. You can guarantee this never gets in the way if you edit that file and change function “scm_version()” to “return” right after variable declaration:

scm_version()
{
        local short
        short=false
        return

You could make that change as a guarantee it isn’t a localversion issue, and I suspect it is not, but it wouldn’t hurt to rule this out. If you make this change, then within the main kernel source, be sure to run “sudo make mrproper” before using the source. You would also need to make sure your “make modules_prepare” (or more likely “make O=/some/where modules_prepare”) is performed again.

Just a side note: It does not hurt, when changing anything, to completely recursively delete any temp directories used for kernel output, and then recreate the empty location before configuring and building again.

In the case where the symbols already exist as an “=y” feature there is no need to install anything. I do not believe it is the case, but there is a possibility in some circumstances of being able to load a module, but having the integrated ("-y") feature fail from being the wrong version. What we are guaranteed though is that the XFS module has prerequisites which are missing…what we don’t know is the reason why. You are quite correct that these should not also be loaded as a module since the feature (in theory) already exists.

You are also correct to resist installing the base kernel Image if not needed. Adding modules is fairly safe, but changing the Image might or might not be straightforward.

If you perform a clean reboot, and monitor “dmesg --follow”, what do you now see from an attempt to insmod the module?

Also, copy the “/proc/config.gz” somewhere, gunzip it, and then rename it:
mv config config_$(uname -r).txt
…and attach to the forum (it is truly just a plain text file, so attaching it should work). We will look closer at the missing symbols and the existing symbols.

1 Like
#9

I made a new attempt today with all your proposals considered. A detailed transcript of what I did is below and the relevant output files are attached.

What I noticed is that I made a mistake yesterday: There is no CONFIG_IOMAP_FS=y in /proc/config.gz. I must have looked at the wrong file yesterday. At least that explains nicely why the module doesn’t load. Sanity restored.

Next thing I tried was to build IOMAP as a module but I could not find it anywhere in the make nconfig TUI. It somehow gets automatically selected when I select XFS, but only to be included into the kernel image and not as a module. I tried to manually set CONFIG_IOMAP_FS=m but that - unsurprisingly - didn’t build a iomap.ko. Is it possible to build IOMAP as a module? If not, what are my options?

One last thing: I found an already built xfs.ko in the initrd.img that ships with L4T but it too complains about the missing IOMAP symbols when I try to load it. I find that strange as I would have expected all modules that come with in the initrd to be loadable with the matching kernel.

# Straight after reboot do:

sudo mount /dev/disk/by-label/data /media/data
cd /media/data

dmesg > dmesg-after-boot.txt

lsmod > lsmod.txt

rm -rf kernel
rm public_sources.tbz2
rm uname-r
rm config.gz

cp /proc/config.gz .
gunzip config.gz
mv config config_$(uname -r).txt

uname -r > uname-r.txt

# There is just a single kernel version
# 32.4.2 and 32.4.3 have the same kernel version
ls -hlatr /lib/modules > lib-modules.txt

ls -hlatr /boot > boot.txt

# Make sure there is no xfs.ko already
sudo find / -name 'xfs.ko' 2>/dev/null > find-xfs-ko.txt

# Find out the right kernel version
uname -a
# 4.9.140

# Find out the right L4T and hardware version
cat /etc/nv_tegra_release
# R32.4.3 210

# Use this information to build the correct download URL
wget https://developer.nvidia.com/embedded/L4T/r32_Release_v4.3/Sources/T210/public_sources.tbz2


tar jxvf kernel_src.tbz2
mv kernel /media/data
cd /media/data
rm -rf Linux_for_Tegra
cp config_$(uname -r).txt kernel/kernel-4.9/.config
cd kernel/kernel-4.9
#sudo make mrproper
vim scripts/setlocalversion
# scm_version()
#{
#       local short
#       short=false
# Added a return here
#       return


make nconfig
# CONFIG_LOCALVERSION="-tegra"
# CONFIG_XFS_FS=m

diff config_$(uname -r).txt kernel/kernel-4.9/.config

make scripts
make prepare
# Accepted all defaults

make modules_prepare

make modules
sudo make modules_install
sudo depmod
dmesg > dmesg-before-modprobe.txt
sudo modprobe xfs
dmesg > dmesg-after-modprobe.txt

diff dmesg-before-modprobe.txt dmesg-after-modprobe.txt > dmesg.diff

Output of lsmod after the reboot:
lsmod.txt (349 Bytes)

Uncompressed /proc/config.gz
config_4.9.140-tegra.txt (162.5 KB)

Difference in dmesg before and after trying to load the XFS module
dmesg.diff.txt (394 Bytes)

#10

The implication is that you will need to install a new kernel Image and all modules associated with it. You might get away with reusing the modules if “uname -r” remains constant, since you are adding a feature instead of removing one, but it is also possible something is rearranged in the kernel which the previous modules are not set up for.

Some features cannot be a module, they are more or less invasive throughout the kernel. One of the most common examples is that virtual memory in the form of swap cannot be a module.

There are times when the initrd needs your updates too. If the ability to read the XFS filesystem is required to read the module and “/boot” content, then the initrd needs this. If XFS is simply used for some other partition, e.g., “/usr/local”, then it is ok to be missing those symbols in the initrd.

I have not looked up the symbol, but if it is only selectable as a non-module, then you must install a new Image and probably new modules, preferably with a new CONFIG_LOCALVERSION, e.g.:
CONFIG_LOCALVERSION=-tegra-xfs
…which would put modules in:
/lib/modules/4.9.140-tegra-xfs/
…and would then advise the new Image be named “Image-4.9.140-tegra-xfs”, at least in “/boot” (there may be other requirements in the initrd, but with extlinux.conf being used in a Nano, the file in “/boot” can simply be named in the config).

Many distributions will have certain commonly used features available as a module. When going to a non-PC it is likely that the kernel config is cut down from the PC, but not necessarily every module left over has any real meaning. You would have to ask NVIDIA about the XFS module in the initrd, but it would be easy to overlook this since adding XFS support (and CONFIG_IOMAP_FS) increases kernel size in the embedded world where space is not as generous as for a PC.

Before working on the various XFS, keep in mind the following: If the bootloader or boot environment needs to read content in “/boot” (and it does since it reads “/boot/extlinux/extlinux.conf”, and also usually the Image), then those boot stages need drivers for XFS once “/boot” (and actually some less known parts of the filesystem) must be read. The default U-Boot does not have this driver, and so trying to load a kernel from XFS would fail. The initrd is a kind of adapter since it is only a tar archive and not a full filesystem, and the bootloader understands this. But what happens if U-Boot is trying to read XFS in order to find the extlinux.conf to decide to use the initrd? Obviously it works for some situations, but don’t mess with the initrd unless you have backed up, it may not be as straightforward as it seems.

Once you can mount an XFS partition somewhere which is not related to booting, then you know you have an Image which at least potentially could work in the initrd. Or reevaluate whether your boot partition itself can remain ext4, in which case you still need a new Image, but you won’t need a new initrd, and the bootloader won’t care.

1 Like
#11

The implication is that you will need to install a new kernel Image […]

That’s what I was afraid of. At least now I know that this is the way forward.
Luckily I don’t need to boot from XFS and I already built a kernel Image.
Now I have to figure out how to boot the device with my own Image and have a plan B ready if it does not boot with my kernel…

Thanks a lot for your help!

#12

There are different Image install procedures, depending on which Jetson it is, and depending on release version. Generally speaking though, if you have a serial console available, then here is what I would recommend…

Leave the original “/boot/Image” in place. Copy your new Image with a version on its name. If you expect the new “uname -r” will be “4.9.140-tegra-test” instead of “4.9.140-tegra” (because you set CONFIG_LOCALVERSION="-tegra-test), then you could copy this as file “/boot/Image-4.9.140-test”.

Then, edit the “/boot/extlinux/extlinux.conf”. Leave the default entry until you’ve tried selecting and have succeeded with the test image. The extra block in “extlinux.conf” will be nearly a copy of the old one, other than label and file name edits:

LABEL test
      MENU LABEL test
      LINUX /boot/Image-4.9.140-tegra-test
      APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4

Now, if you interrupt boot at the U-Boot kernel selection point, then entry “2” will boot your kernel…otherwise it will boot the original. If booting entry “2” works, then you could make this the default, and leave entry “1” as is for future use:

TIMEOUT 30
DEFAULT test

MENU TITLE p2771-0000 eMMC boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4

LABEL test
      MENU LABEL test
      LINUX /boot/Image-4.9.140-tegra-test
      APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4

Note that the time you have to interrupt U-Boot is rather narrow. If you hit a button far too early, then nothing happens. If you hit a button a bit too early, then you drop into the U-Boot command line (your timing would have almost been correct), and can continue booting via the command “boot”. Very shortly after this, if you hit any key, it’ll offer to let you pick the kernel. You’ll have to use the backspace key to get rid of whatever keystroke you used to interrupt with, but then you can pick “2” and the enter key.

If the above fails, then you are still booting the original kernel and modules. If the above works because you have both Image and modules in correctly, then you can alternate to the new “default”. After that it is up to you if you want to leave the original in place or not.

EDIT: Note that this will not change any Image from an initrd.

1 Like
#13

I got it to work! I compiled a new kernel as you suggested. As I haven’t had read your last reply I just copied it over the old image and crossed my fingers during the reboot. Luckily it worked;-) XFS up and running.

Thank you very much, you helped me lot!