Custom kernel compile driver module size issue

I tried to recompile the kernel and I noticed that the custom compiled kernel driver under /lib/modules/ are much larger than the stock driver size.
My driver size can be all the way up to 1GB and the stock driver size only

test@test:/lib/modules$ du -sh *
965M	4.9.140-user
67M	    4.9.140-tegra

I checked all the files generated (.ko files) they are similar but with much larger size. Wondering what caused this and how I can reduce the driver size similar to stock one.

The modules probably have debug symbols and are not stripped. This may be of interest:
https://linux-tips.com/t/how-to-strip-linux-kernel-modules/472/2
…which says:
strip --strip-debug module.ko
(change the name “module.ko” for the real name, not sure if you can combine this with find)

1 Like

Thank you very much! After some research after you point me the right direction I found to use

INSTALL_MOD_STRIP=1

in make modules_install process is easier for me. the Makefile already has this procedure and it is cross compiler friendly.
The only issue is that the official document about cross-compile needs some update, from

$ sudo make ARCH=arm64 O=$TEGRA_KERNEL_OUT modules_install \
    INSTALL_MOD_PATH=<top>/Linux_for_Tegra/rootfs/

to

$ make ARCH=arm64 O=$TEGRA_KERNEL_OUT modules_install \
    INSTALL_MOD_PATH=<top>/Linux_for_Tegra/rootfs/ \
    INSTALL_MOD_STRIP=1

First, you don’t need root privilege to do the module install because it is cross-compile the rootfs is under user space.
Second, INSTALL_MOD_STRIP=1 will do the cross platform strip automatically for you.

Now the driver size is much smaller and the OS can fit in 16GB sd card.

test@test:/lib/modules$ du -sh *
71M	4.9.140-user
67M	4.9.140-tegra

Thanks,

Xishan

This won’t work the way you expect it to. Everything else here is correct (and very useful), but sudo is not based on user space (a virtual memory space) versus kernel space (a physical memory space). The “rootfs/” content location used for flash image creation must be an exact mirror of the installation about to occur, and that includes permissions. Non-root users are never allowed to modify kernel or kernel modules, and so if installing to “rootfs/” succeeds, then modules are being added to “rootfs/lib/modules/$(uname -r)/” by a non-root user…should that succeed, the implication is something has gone very wrong (those files, when loaded, become part of the kernel).

An illustration of that point comes when people try to flash from an NTFS or VFAT partition (which contains “Linux_for_Tegra/rootfs/”). These filesystems have no concept of Linux “group” permissions, and even less concept of “suid” or “guid” or “sticky bit” permissions. The “sudo” command itself must be correctly set with the right suid permission, or the system will refuse to use it…copying the sudo binary from an NTFS or VFAT partition implies that permission is lost when creating the ext4 image, and flash will succeed, but all use of sudo (or anything requiring root permission) will then fail (you’d have to reflash).

Another example is that if the unpack of the sample rootfs into “Linux_for_Tegra/rootfs/” were performed without “sudo”. The same problem would occur, but the end user could then certainly install custom kernel modules directly to this without error (and sudo would once again fail without knowing why).

Normally one would not work as root/suid when installing software, but if you are able to install modules to any destination which is part of a “lib” directory, or a module directory, then it means something went wrong with permissions.

The reason I mention this is that a number of people have at times done everything else correctly, and the flash succeeded, but then there were odd problems. Different reasons exist why permissions in “rootfs/” might be wrong, but once it happens, it can take quite some time to figure out why an otherwise perfect procedure still works incorrectly. Always treat “Linux_for_Tegra/rootfs/” as a complete operating system, including permissions.

I can understand your concern about the cross compile roofts user problem. there are some understanding about the linux user privilege here:

first when we cross-compile the system, if you use sudo you literately get two users involved on host computer with the cross compile procedure. Will these two users have same environment setup? usually no. This will become problem esp. when I try to do the cross compiler strip. you have to have root user has the same cross compile environment settings. Usually we don’t want to mess up with root user.

Second, cross compiling (make modules_install) under user won’t have any issue. anytime you transfer new stuff to client computer you can always change the user at this step, including generating the image file or package file. I can see during next step if we want to save modules as a file we can

tar --owner root --group root -cjf $DEV_DIR/kernel/kernel_supplements.tbz2 lib/modules

here all files are ready to go.

third. if you still don’t think it is good before you package or make image of the rootfs. you can change the owner of module lib files right after you generate them. it is still easier and safe compared with multi-user compiling.

You are correct. There is a reason though that I mentioned this: While cross compiling I saw “O=/where/ever/it/was/Linux_for_Tegra/rootfs/”, and this implies copying files into “rootfs/lib/modules/$(uname -r)/kernel/”. That destination is owned by root, and if a regular user is able to copy to that point, then it means any image generated from that “rootfs/” will have invalid permissions allowing write into the module area. I was wondering how it was possible to copy files into that subdirectory without sudo.

Manually copying from the PC’s “rootfs/lib/modules/$(uname -r)/kernel/” to the Jetson would, as you mention, not be a problem. Generating a new image for flash using an invalid rootfs permission though would be a problem you wouldn’t find the consequences of until it is too late.

When I actually compile modules I always use the “O=/some/where”, and never ever compile as sudo/root. modules_install, if not to the end location, I also run without sudo. Then I copy files separately with sudo like you do.