Custom kernel build, menuconfig flag selection

[bkawlatow]

Hello,

I am trying to build the custom kernel with USB gadget functionality. As I concluded from previous posts on this forum, I need to enable few flags in the kernel .config file. I am doing that by using menuconfig editor. The problem I experience is that I can enable those flags as modules (=m), but when I want to mark them as statically built-in into kernel, they seem to be mutually-exclusive (in the editor I can not select both flags as =y).
I would like to avoid editing the .config file “by hand”.

Is it a problem with the menuconfig tool or I just can not have those two modules “built-in” for some reason?

Cheers

[linuxdev]

I don’t know your particular option requirements, but generally speaking:

• Most drivers/symbols can be built as either a module or integrated.
• Not all drivers/symbols can be in the format of a module, and not all can be in the format of integrated.
• The reason for using the menuconfig editor (or other editors, e.g., nconfig) is that these editors understand the dependencies. They also understand what can or cannot be in the form of a module or integrated. The editor will not be wrong about this.
• If you write down the unexpected dependencies, and then go to that item in the editor (which is why I like nconfig: It has a symbol search function), you can verify if the item itself is listed as having either format available.

This is entirely a Linux kernel mechanic. The kconfig system is rather nice compared to most dependency systems, I’ve only grown to like it more rather than less over the years. Kconfig itself only acts as a “bookkeeper”, and does not actually cause the requirement to be only integrated or only a module; this is part of the code itself living behind the symbol (every feature has a “symbol”; symbol tends to be how one refers to the identifier despite most symbols corresponding to a driver). If you look at an example, like support for virtual memory (swapping), then this feature is so entangled in everything that you simply cannot build it as a module. Some authors also only build one format because that is all they use, and it takes extra code to build for both formats. I suspect Gadget has only integrated format due to being invasive or complicated, but I could be wrong.

If you must change an “integrated” (“=y”) feature, then you should also change the CONFIG_LOCALVERSION, and build and install all modules. Essentially you start with a copy of the base configuration, and instead of using “-tegra” (reflected in the output suffix of the command “uname -r”), change it to something named after the change. For example, you might change it to “-gadget” (in which case the suffix to “uname -r” will be “-gadget”).

Whenever you change the integrated feature set it is likely previous modules will no longer load in, and so you will want to install 100% of all modules.

A side-effect which complicates life is that if you have an initrd for boot, then you have to get the new modules into the initrd as well. Modules are found under:
/lib/modules/$(uname -r)/kernel

[bkawlatow]

Thank you @linuxdev for the detailed explanation, it answers all of my questions.

[linuxdev]

I forgot to mention: It is a good idea to keep the old kernel Image and modules. In that case you’d edit the “/boot/extlinux/extlinux.conf” to add a second boot option which is a duplicate of the original, except for this:

• You would do something like rename the original Image to Image-tegra.
• The duplicated entry would need a new label and menu label. E.g., the original Image, which is now Image-tegra, might say it is “olddefault” instead of “default”; the Image would be renamed in that entry to “Image-tegra”.
• You’ll end up with a new “uname -r”, and so verify that you still have the “/lib/module/$(uname -r)/kernel” which the Image-tegra will look for (you simply keep what was already there, it is the kernel itself looking for this location, and the old Image will look there regardless of what the file name is).
• You’ll verify that after installing all modules you have both the “Image-gadget” and a new “/lib/modules/$(uname -r)/kernel” (one with the “uname -r” suffix being “-gadget”).
• The original default entry will now name “Image-gadget” instead of “Image”.

If you used an initrd to boot, then you need a new initrd because the old modules won’t load in the new kernel. You’d keep your old initrd, but you could rename it to something with a suffix of “-tegra”, and then edit that in the boot entry of extlinux.conf. Any new boot entry could have the “-gadget” name for the new initrd if you use an initrd. That old initrd would have some subset of modules from the original entry, and so that old initrd won’t need any edit to boot the old kernel regardless of what you name the Image-tegra. It is the renamed new copy of the initrd which would have new modules.

Or you could create one initrd that has both sets of modules (-tegra and -gadget), but I advise against that. initrd space is precious. Better yet, if you don’t have an initrd, you don’t even need to think about it. People booting external rootfs devices, e.g., NVMe or SSD, would be the only ones worrying about this.

[bkawlatow]

@linuxdev I came across the problem where my “uname -r” is image-gadget, but the path to kernel modules looks like/lib/module/image-tegra/kernel . When is this path generated, and where does it take the image-tegra part from? Is the bootloader somehow checking the “uname -r” from an image and unpacking kernel files to that path? The only thing I changed is LOCALVERSION=-gadget flag before building the kernel.
The problem I experience is insmod looking for kernel modules in “/lib/module/$(uname -r)/kernel” by default, and it does not find the location.
I temporarily changed the name of my kernel modules directory to “/lib/module/image-gadget/kernel” just to check if OS modules were built correctly and if my HID script works. The good news that it does what it’s supposed to :)

[linuxdev]

The kernel itself is usually “/boot/Image” (that’s an uncompressed kernel; lots of compressed variants exist, but usually in “/boot”). It is this file which determines the output of “uname -r”. And it is the output of “uname -r” which always determines the subdirectory of “/lib/modules/” to search for that kernel’s modules.

If you installed a kernel Image (preferably with some edited name like “Image-gadget”), then then actual name of the Image file will never have an effect on where that Image looks. The location is more or less hard coded into the Image at the time of compile, just as is the “uname -r”. If you used CONFIG_LOCALVERSION as “-gadget”, and if the installed kernel searches another location, then you are guaranteed that your kernel is not the one booting.

The simplest case is if the entire filesystem is a single ext4 partition. This implies that if Image has an ext4 driver integrated into it (and the default kernels do), then you can always access modules both in “/boot” and in “/lib/modules/$(uname -r)/”.

The next step to complicate this is if you have a different “/boot” partition versus the partition where “/lib/modules/” is found. Now if you have a separate boot (“/boot”) and rootfs (“/”, which contains “/lib/modules/”), then it is rare to have any issue. Sometimes if a separate disk is involved, then the driver for the disk controller itself might mean that the Image can’t mount “/lib/modules/”, but this wouldn’t be due to lacking ext4. What I’m trying to illustrate is that there is a chain of drivers required to access “/lib” (and thus also “/lib/modules”), and if anything in the chain is missing, then modules are not accessible. It follows that if both “/boot” and “/” are the same ext4 partition, that the ability to load the kernel already guarantees the ability to find modules (and if the modules are valid, then to load those modules).

Here is the real reason I mention all of that detail: It is common in the Linux world to boot the Image from some simple device, but then to put everything else on some more complicated device. An example is using /boot as ext4, but using some other filesystem type on “/” (e.g., XFS filesystem is common for large file multi-media systems). The kernel would not be able to mount an XFS “/” to find its own modules if the XFS is itself in the form of a module. You end up with the proverbial “chicken and the egg” dilemma. Thus an initial ramdisk was created as a generic adapter.

An initrd (initial ramdisk) is just a very simple filesystem that lives entirely in RAM. All Linux bootloaders and boot chains that I know of understand ext4 and initrd by default. If your Image is on ext4 of “/boot”, then the bootloader executes this; if the early “/” (which includes the init script and module directory “/lib/modules/$(uname -r)”) is an initrd with the XFS or RAID or volume manager or disk controller drivers, then the Image will have access to those modules regardless of the feature being in the form of a module. A small subset of boot-critical modules are very often placed in that initrd, and then modules can be found. Once modules are loaded, then the init process that run from the initrd will perform a pivot of the root device to change “/” from being the initrd" to “/” instead referring to the actual disk device. Doing so implies you no longer have to have the “/” device drivers in the kernel Image itself.

I suspect that if your kernel is not using the “uname -r” which you altered this to be (the “-gadget” version), then the Image you think is loading is not actually loading.

I suspect that if your kernel is using the proper “uname -r”, but it is failing to find modules which are installed to the new “/lib/modules/$(uname -r)/” location, then it might be due to an initrd not being set up to contain the new modules it needs.

Installing a module which is not required for boot implies you never need to put that module in the initrd. Installing a module related to boot implies you must modify the initrd to put any modules needed for boot into the new “/lib/modules/$(uname -r)/”.

If you have modified the original Image, and changes occur which go beyond adding only modules, then you should consider that you need to install all modules again to be sure they are compatible with this new kernel config (otherwise they might be found, and yet still fail to load). If the original initrd is used (and your boot might not use an initrd), this new kernel Image means not only do you need to install modules into “/lib/modules/$(uname -r)”, you must also install some of them to the initrd.

Does your boot use an initrd (initial ramdisk)? Look in “/boot/extlinux/extlinux.conf”. Specifically, look in the boot entry for that Image, and see if a file is listed in the key/value pair “INITRD”, e.g., maybe it shows “INITRD /boot/initrd”. If so, and if there are modules it fails to load from the new “/lib/modules/$(uname -r)/”, and if the failed modules result in something going wrong during boot, then you have to add the new modules to the initrd to complete the new kernel install.

Also, what is the exact response to the command “uname -r”?

[bkawlatow]

So my uname -r shows 4.9.253gadget but the only directory in /lib/modules is 4.9.253-tegra

I think I know what might went wrong. I followed this instruction for custom kernel build, but under “Copy kernel, device tree and modules into jetpack” section, it looks like the step which copies the modules into new rootfs is missing. The new modules that I added through menuconfig are in /l4t-gcc/Linux_for_Tegra/source/public/modules/lib/modules/4.9.253gadget. In the directory from nvidia_sdk (/nvidia/nvidia_sdk/JetPack_4.6.1_Linux_JETSON_NANO_TARGETS/Linux_for_Tegra/rootfs/lib/modules) there is only 4.9.253-tegra folder.
Should I just manually overwrite the whole directory with modules and then build the whole image again?

Also, does the OS Image contain the rootfs as well, or the installer just copies the rootfs into SD card?

Content of my /boot/extlinux/extlinux.conf, I didn’t make any changes to it yet.

EDIT:
As I thought, the problem was caused by me forgetting to copy newly created modules by sudo cp -r $JETSON_NANO_KERNEL_SOURCE/modules/lib/modules/* rootfs/lib/modules
System installer is populating rootfs from the nvidia source folder

After copying the modules, my /lib/modules/$(uname -r)/ path contains two directories, Image-tegra and Image-gadget

[linuxdev]

I would have used CONFIG_LOCALVERSION of “-gadget”; the hyphen is something you must add yourself. The reason you don’t have “/lib/modules/4.9.253gadget” is because the kernel does not install this; you have to install modules. Keep in mind that there is an option to customize the “make modules_install” command.

The default of that command assumes you are installing to the same system that you run the command on. If you are cross compiling or compiling for another system, then this would be a "bad idea"™. If you use the INSTALL_MOD_PATH option, then it could go something like this from any computer to create a clean directory containing things you want to copy over to some other computer (you might use other options as well, and I’m assuming you are already at your kernel build source location with other options set up):

mkdir ~/modules_out
make INSTALL_MOD_PATH="~/modules_out" modules_install

If you did that, then the entire path of modules and such would then exist at “~/modules_out/”. There would exist for example:
~/modules_out/lib/modules/4.9.253gadget/kernel/...

You could copy the entire “4.9.253gadget/” recursively to the Jetson at “/lib/modules/”, and now you would have “/lib/modules/4.9.253gadet/...” and all of the module content. For this to work you have to have already propagated any configuration with something like the “make Image” or “make modules_prepare” step first, and would have had to have actually run the “make modules” step. If the modules have been built, then the modules_install step conveniently puts them in one place so you can copy them.

Sometimes you won’t see all steps because unless you build a new Image you don’t need to create that directory (it would already exist). Once you replace the Image itself with something using a new “uname -r”, then you must install everything.

The original “4.9.253-tegra/” subdirectory is for the default Image file that would be used if you had not switched to a custom kernel Image. You can leave that there in case you want to keep the original kernel as a backup boot option. However, keep in mind that any change on the host PC Linux_for_Tegra/ means you need to flash. Editing boot content before flashing might be a bit more involved than editing something else. You can put a new Image and modules directly onto the Jetson and not flash every time (which is more convenient if you are not manufacturing).

You do not overwrite. If you copy the 4.9.253gadget module content into “/lib/modules/”, then you will have both. The original kernel (if booted) would use the original 4.9.253-tegra, while the new kernel (if booted) would use the 4.9.253gadget content.

Terminology can seem a bit confusing. The SD card is a filesystem image, not a file. The kernel Image is an actual file that resides in “/boot” of the filesystem image. If you alter or install the SD card image it is quite different than installing file Image into “/boot”.

When you build a kernel you have the option to build different formats. Mainly those formats are different kinds of compression or debug information. This always builds a file “Image”, and other files are often derived from this. For example, there is a compressed zImage and bzImage file, which are just different compressed versions of the Image file (which is not compressed). This is completely different than the filesystem image (I don’t capitalize this “image” because it is like saying you are taking a snapshot from a camera; the other Image is the kernel which resides somewhere on the filesystem).

If you flash, then the flash will overwrite the “Linux_for_Tegra/rootfs/boot/” content, including the extlinux.conf at “Linux_for_Tegra/rootfs/boot/extlinux/extlinux.conf”. The Image at “Linux_for_Tegra/rootfs/boot/Image” will also get overwritten. Probably modules at “Linux_for_Tegra/rootfs/lib/modules/4.9.253-tegra/” will be overwritten (you would not want to expect those modules to be anything but what is set up for the NVIDIA-defined configuration). Other files you’ve added by different names would likely be preserved, but if extlinux.conf has been overwritten, then you won’t see those other boot options without editing.

Just put the content directly on the Jetson, don’t overwrite the original content, but instead add new content. Examples (on the Jetson itself):

• Leave “/boot/Image” alone. Add an additional “/boot/Image-gadget” (via renaming the new Image and leaving alone the old Image).
• Edit “/boot/extlinux/extlinux.conf” to duplicate the old default boot entry, and edit the new entry to say Image-gadget instead of Image. Whichever entry is picked, then it will use the old or new Image or Image-tegra. If Image-tegra is the new kernel, then it will look for modules from /lib/modules/4.9.253gadget/. If you boot Image, then it will look for the old /lib/modules/4.9.253-tegra/. Both have a place since you are adding a new entry and not replacing the original.
• If you have compiled a new set of modules, but not copied them to the Jetson, then the new Image-gadget will be missing modules when it boots. You simply have to copy your built modules there.
• One way to copy is to use scp recursively over the network if you have networking between the host PC you built on and the Jetson. If you built these on the Jetson itself, then you just have to “make modules_install” without setting any optional INSTALL_MOD_PATH. I would still use INSTALL_MOD_PATH to some custom location to examine before using them, and then recursively copy.

If you have an SD card rootfs (the filesystem image), then you can easily just copy straight to that on the host PC.

Before giving more details, are you ok to mount your SD card on the host PC and copy directly to it? It should automount, and you’d have to be familiar with where it mounts. Also, since you will be writing to the SD, you have to properly umount it before you unplug it. Is this ok, and do you have your kernel build source still set up?

[bkawlatow]

I truly appreciate your help; you provide us with so much valuable information!

I did it in two ways regarding flashing the new OS so far.
First from Windows host using Balena Etcher and .img image. I used the JetPack 4.6 image from the Nvidia download page, as well as the image I created by myself on a Linux host using Nvidia SDK.
The second way I flashed the board was by using the .flash script from the SDK tools directory on the Linux host. Both ways were successful but they did not include duplicating the boot entry, I will try that a bit later.

The very weird thing right now is, after some time (a day or two) I noticed a message *** System restart required ***. I do not know if this was the direct reason, but after the reboot my uname -r returns 4.9.337-tegra instead of 4.9.253gadget. I noticed that once already, but I was so confused back then that I thought I flashed the wrong image.

Did that happen because I apt-get upgraded the system?

[linuxdev]

“System restart required” is usually because of a package update. This update might have been for the kernel itself. If your kernel file is named something like “Image-gadget”, then the update won’t overwrite the kernel since the default is “Image”. If you have customized the “/boot/extlinux/extlinux.conf”, and added a new entry, and that entry is an additional boot entry, then I think that extlinux.conf would be preserved (but maybe not; extlinux.conf is a configuration file, and customized versions are normally left alone, but I can’t guarantee that the update will leave that file alone).

Incidentally, if there is an initrd file, then that too might be updated, and so if you use a customization, or need to guarantee one remaining, then you might keep a copy (don’t just rename the original) of the initrd which is itself renamed to something with the “-gadget” file name. Then, in the initrd, in the customized duplicate entry for the new kernel, make sure the INITRD key/value pair names this new file (even if the file is a copy of the original without editing, the renamed file might be useful to protect from updates).

If it turns out that your modified Image-gadget is still there, then it is likely just a case of editing extlinux.conf to name your version of the file as the default boot entry again.