Issues Building Custom Kernel 36.4 new Jetson Orin Nano Dev Kit

Hi @linuxdev ,

I also compile the kernel with the commands manually, just like @EduardoSalazar96 specified in a previous post: Issues Building Custom Kernel 36.4 new Jetson Orin Nano Dev Kit - #10 by EduardoSalazar96.

So for building the kernel, from sources directory (a directory created in Linux_for_Tegra with all the sources, as you can see in the commands), I would execute make -C kernel, but as this does not compile everything with the new config file, I execute now from kernel-jammy-src the command make -j$(nproc), which creates the Image of 41MB with the added modules.

Even if I give it a target ‘kernel’, it rewrites the .config file:

ubuntu@xxx:~/nvidia/nvidia_sdk/JetPack_6.1_Linux_JETSON_ORIN_NANO_TARGETS/Linux_for_Tegra/sources$ make -C kernel kernel
make: Entering directory '/home/ubuntu/nvidia/nvidia_sdk/JetPack_6.1_Linux_JETSON_ORIN_NANO_TARGETS/Linux_for_Tegra/sources/kernel'
================================================================================
Building kernel-jammy-src sources
================================================================================
make \
        ARCH=arm64 \
        -C /home/ubuntu/nvidia/nvidia_sdk/JetPack_6.1_Linux_JETSON_ORIN_NANO_TARGETS/Linux_for_Tegra/sources/kernel/kernel-jammy-src  \
        LOCALVERSION=-tegra \
        defconfig
make[1]: Entering directory '/home/ubuntu/nvidia/nvidia_sdk/JetPack_6.1_Linux_JETSON_ORIN_NANO_TARGETS/Linux_for_Tegra/sources/kernel/kernel-jammy-src'
  HOSTCC  scripts/kconfig/conf.o
  HOSTLD  scripts/kconfig/conf
*** Default configuration is based on 'defconfig'
#
# configuration written to .config
#

It seems that make -C kernel and make -C kernel kernel is the same command in the Makefile, and it is hardcoded to always compile the defconfig, or at least that seems for me:

ubuntu@xxx:~/nvidia/nvidia_sdk/JetPack_6.1_Linux_JETSON_ORIN_NANO_TARGETS/Linux_for_Tegra/sources/kernel$ make help
================================================================================
Usage:
   make or make kernel   # to build kernel
   make install          # to install kernel image and in-tree modules
   make clean            # to make clean kernel source
================================================================================

This is the Makefile, take a look at the KERNEL_DEF_CONFIG variable, and the kernel target:

# SPDX-FileCopyrightText: Copyright (c) 2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: BSD-3-Clause

KERNEL_SRC_DIR ?= kernel-jammy-src
KERNEL_DEF_CONFIG ?= defconfig

MAKEFILE_DIR := $(abspath $(shell dirname $(lastword $(MAKEFILE_LIST))))
kernel_source_dir := $(MAKEFILE_DIR)/$(KERNEL_SRC_DIR)

ifdef KERNEL_OUTPUT
O_OPT := O=$(KERNEL_OUTPUT)
$(mkdir -p $(KERNEL_OUTPUT))
kernel_image := $(KERNEL_OUTPUT)/arch/arm64/boot/Image
else
kernel_image := $(kernel_source_dir)/arch/arm64/boot/Image
endif

NPROC ?= $(shell nproc)

# LOCALVERSION : -tegra or -rt-tegra
version = $(shell grep -q "CONFIG_PREEMPT_RT=y" \
    ${kernel_source_dir}/arch/arm64/configs/${KERNEL_DEF_CONFIG} && echo "-rt-tegra" || echo "-tegra")

.PHONY : kernel install clean help

kernel:
	@echo   "================================================================================"
	@echo   "Building $(KERNEL_SRC_DIR) sources"
	@echo   "================================================================================"
	$(MAKE) \
		ARCH=arm64 \
		-C $(kernel_source_dir) $(O_OPT) \
		LOCALVERSION=$(version) \
		$(KERNEL_DEF_CONFIG)

The actual source directory of the kernel has a more complex Makefile, I think that the default target would be something like make -j$(nproc) Image, because it generates an Image and compiles some modules [M], these are the targets:

ubuntu@xxx:~/nvidia/nvidia_sdk/JetPack_6.1_Linux_JETSON_ORIN_NANO_TARGETS/Linux_for_Tegra/sources/kernel/kernel-jammy-src$ make help
Cleaning targets:
  clean           - Remove most generated files but keep the config and
                    enough build support to build external modules
  mrproper        - Remove all generated files + config + various backup files
  distclean       - mrproper + remove editor backup and patch files

Configuration targets:
  config          - Update current config utilising a line-oriented program
  nconfig         - Update current config utilising a ncurses menu based program
  menuconfig      - Update current config utilising a menu based program
  xconfig         - Update current config utilising a Qt based front-end
  gconfig         - Update current config utilising a GTK+ based front-end
  oldconfig       - Update current config utilising a provided .config as base
  localmodconfig  - Update current config disabling modules not loaded
                    except those preserved by LMC_KEEP environment variable
  localyesconfig  - Update current config converting local mods to core
                    except those preserved by LMC_KEEP environment variable
  defconfig       - New config with default from ARCH supplied defconfig
  savedefconfig   - Save current config as ./defconfig (minimal config)
  allnoconfig     - New config where all options are answered with no
  allyesconfig    - New config where all options are accepted with yes
  allmodconfig    - New config selecting modules when possible
  alldefconfig    - New config with all symbols set to default
  randconfig      - New config with random answer to all options
  yes2modconfig   - Change answers from yes to mod if possible
  mod2yesconfig   - Change answers from mod to yes if possible
  listnewconfig   - List new options
  helpnewconfig   - List new options and help text
  olddefconfig    - Same as oldconfig but sets new symbols to their
                    default value without prompting
  tinyconfig      - Configure the tiniest possible kernel
  testconfig      - Run Kconfig unit tests (requires python3 and pytest)

Other generic targets:
  all             - Build all targets marked with [*]
* vmlinux         - Build the bare kernel
* modules         - Build all modules
  modules_install - Install all modules to INSTALL_MOD_PATH (default: /)
  dir/            - Build all files in dir and below
  dir/file.[ois]  - Build specified target only
  dir/file.ll     - Build the LLVM assembly file
                    (requires compiler support for LLVM assembly generation)
  dir/file.lst    - Build specified mixed source/assembly target only
                    (requires a recent binutils and recent build (System.map))
  dir/file.ko     - Build module including final link
  modules_prepare - Set up for building external modules
  tags/TAGS       - Generate tags file for editors
  cscope          - Generate cscope index
  gtags           - Generate GNU GLOBAL index
  kernelrelease   - Output the release version string (use with make -s)
  kernelversion   - Output the version stored in Makefile (use with make -s)
  image_name      - Output the image name (use with make -s)
  headers_install - Install sanitised kernel headers to INSTALL_HDR_PATH
                    (default: ./usr)

Static analysers:
  checkstack      - Generate a list of stack hogs
  versioncheck    - Sanity check on version.h usage
  includecheck    - Check for duplicate included header files
  export_report   - List the usages of all exported symbols
  headerdep       - Detect inclusion cycles in headers
  coccicheck      - Check with Coccinelle
  clang-analyzer  - Check with clang static analyzer
  clang-tidy      - Check with clang-tidy

Tools:
  nsdeps          - Generate missing symbol namespace dependencies

Kernel selftest:
  kselftest         - Build and run kernel selftest
                      Build, install, and boot kernel before
                      running kselftest on it
                      Run as root for full coverage
  kselftest-all     - Build kernel selftest
  kselftest-install - Build and install kernel selftest
  kselftest-clean   - Remove all generated kselftest files
  kselftest-merge   - Merge all the config dependencies of
                      kselftest to existing .config.

Devicetree:
* dtbs             - Build device tree blobs for enabled boards
  dtbs_install     - Install dtbs to /boot/dtbs/
  dt_binding_check - Validate device tree binding documents
  dtbs_check       - Validate device tree source files

Userspace tools targets:
  use "make tools/help"
  or  "cd tools; make help"

Kernel packaging:
  rpm-pkg             - Build both source and binary RPM kernel packages
  binrpm-pkg          - Build only the binary kernel RPM package
  deb-pkg             - Build both source and binary deb kernel packages
  bindeb-pkg          - Build only the binary kernel deb package
  snap-pkg            - Build only the binary kernel snap package
                        (will connect to external hosts)
  dir-pkg             - Build the kernel as a plain directory structure
  tar-pkg             - Build the kernel as an uncompressed tarball
  targz-pkg           - Build the kernel as a gzip compressed tarball
  tarbz2-pkg          - Build the kernel as a bzip2 compressed tarball
  tarxz-pkg           - Build the kernel as a xz compressed tarball
  perf-tar-src-pkg    - Build perf-5.15.148.tar source tarball
  perf-targz-src-pkg  - Build perf-5.15.148.tar.gz source tarball
  perf-tarbz2-src-pkg - Build perf-5.15.148.tar.bz2 source tarball
  perf-tarxz-src-pkg  - Build perf-5.15.148.tar.xz source tarball

Documentation targets:
 Linux kernel internal documentation in different formats from ReST:
  htmldocs        - HTML
  latexdocs       - LaTeX
  pdfdocs         - PDF
  epubdocs        - EPUB
  xmldocs         - XML
  linkcheckdocs   - check for broken external links
                    (will connect to external hosts)
  refcheckdocs    - check for references to non-existing files under
                    Documentation
  cleandocs       - clean all generated files

  make SPHINXDIRS="s1 s2" [target] Generate only docs of folder s1, s2
  valid values for SPHINXDIRS are: PCI RCU accounting admin-guide arm arm64 block bpf cdrom core-api cpu-freq crypto dev-tools devicetree doc-guide driver-api fault-injection fb filesystems firmware-guide fpga gpu hid hwmon i2c ia64 ide iio infiniband input isdn kbuild kernel-hacking leds livepatch locking m68k maintainer mhi mips misc-devices netlabel networking nios2 openrisc parisc pcmcia power powerpc process riscv s390 scheduler scsi security sh sound sparc spi staging target timers trace translations usb userspace-api virt vm w1 watchdog x86 xtensa

  make SPHINX_CONF={conf-file} [target] use *additional* sphinx-build
  configuration. This is e.g. useful to build with nit-picking config.

  Default location for the generated documents is Documentation/output

Architecture specific targets (arm64):
* Image.gz      - Compressed kernel image (arch/arm64/boot/Image.gz)
  Image         - Uncompressed kernel image (arch/arm64/boot/Image)
  install       - Install uncompressed kernel
  zinstall      - Install compressed kernel
                  Install using (your) ~/bin/installkernel or
                  (distribution) /sbin/installkernel or
                  install to $(INSTALL_PATH) and run lilo

  tegra_prod_defconfig        - Build for tegra_prod
  tegra_recovery_chain_defconfig - Build for tegra_recovery_chain

  make V=0|1 [targets] 0 => quiet build (default), 1 => verbose build
  make V=2   [targets] 2 => give reason for rebuild of target
  make O=dir [targets] Locate all output files in "dir", including .config
  make C=1   [targets] Check re-compiled c source with $CHECK
                       (sparse by default)
  make C=2   [targets] Force check of all c source with $CHECK
  make RECORDMCOUNT_WARN=1 [targets] Warn about ignored mcount sections
  make W=n   [targets] Enable extra build checks, n=1,2,3 where
                1: warnings which may be relevant and do not occur too often
                2: warnings which occur quite often but may still be relevant
                3: more obscure warnings, can most likely be ignored
                Multiple levels can be combined with W=12 or W=123

Execute "make" or "make all" to build all targets marked with [*] 
For further info see the ./README file

In my case, I do not need to configure anything else in the SD card, only flash it with the new Image, as far as I know, no additional data in the SD card if you mean this, always formatted.

With adding kernel changes you mean that, for example, I can compile a default kernel, boot it, and later just compile the two remaining modules and copy them to the SD card? is this possible? It might do the trick for me.

I think there is no longer a problem with the CONFIG_LOCALVERSION, as when the OS is booted, there is only one /lib/modules directory, with the -tegra suffix, and the uname -r returns the same name. When it booted however, it got stuck in OEM installation, but I could open the TTY and make sure the previous things were okay, and could verify that I could load the two networking modules.

Compiling the kernel from the source directory, when it is done the installation, now there is no missing symbol with the modprobe, which is strange as it supposedly is missing something because there is a problem in the OEM installation now.

In the last tests I was updating the initramfs just in case with sudo ./tools/l4t_update_initrd.sh, as they show in the documentation for the release: Kernel Customization — NVIDIA Jetson Linux Developer Guide 1 documentation

I did some tests with the Nvidia scripts for compilation, but it seemed more complex for debugging, so I now prefer just executing myself the commands for precision.

Thank you.

Is the above the actual command? This would be incorrect. For a Jetson you want to “make -C <location> Image” to build the kernel. Then, if you are building modules also, “make -C <location> modules”. All of that assumes you already configured. Note that when you build Image it also propagates the configuration throughout the source; if you were building only modules, then assuming you have created your configuration correctly, you would also have to use “make -C <location> modules_prepare” before building modules. Building Image target does the same thing as modules_prepare target, and then builds Image; modules build correctly only after configuration is fully in place.

If and only if the original configuration has the same integrated (“=y”) features, you just need to copy the modules as ordinary file copy to the correct subdirectory of that kernel’s “uname -r”. I’m going to pretend you are building module “foo”, and that in the kernel source tree, the location is “drivers/bar/” (when built the module ends up in the kernel source’s “drivers/bar/foo.ko”). The “modules_install” target places that as a file copy to the “/lib/modules/$(uname -r)/kernel/” prefix, and would then append the destination suffix “drivers/bar/” before copy to “/lib/modules/$(uname -r)/kernel/bar/foo.ko”.

If you are not replacing the Image, and if the original integrated features match for compile (including “CONFIG_LOCALVERSION” since this determines install path), then just copy the module file and then run “sudo depmod -a”. Optionally, reboot. There is no need to change the Image because you are not changing it.

When you build a new kernel Image (the “=y” part), and if the source code version is the same, but integrated features change, then you would want a new CONFIG_LOCALVERSION, and 100% of all modules installed to the new “/lib/modules/$(uname -r)/”.

Some of the setup for installing to the flash content can also change device tree, initrd, extlinux.conf, and other “meta” boot content. You don’t always want that. If you have a working setup and you are installing only modules, then why change any of that? The one exception is that if your new kernel module is required during boot stages, then you’d have to rebuild the initrd to include that module. Another time to rebuild the initrd is if the boot target is changing, e.g., going from an eMMC boot to NVMe boot, or changing from SD card boot to NVMe boot; if the boot target did not change, and the module was not required to boot, then why mess with the initrd?

So far as the “uname -r” not changing, this implies either (A) the kernel did not change, or (B) the new kernel has the same source code version and the same CONFIG_LOCALVERSION. This latter case (B) has a danger: The original modules will also be what it tries to use, and if that new kernel changed integrated features, then there is a significant chance that the modules won’t be valid with that configuration (e.g., some might not load, and some might load but be missing some dependency).

A kernel’s source code version and list of integrated (“=y”) features should be considered a “lock” with a shape. The modules should be thought of as having been compiled as a “key” for that “lock” (in reality it is just that the ABI must match, and changing things can change the location and/or signature of items in that ABI; sometimes they can still work, but guarantees are gone).