Hi Nvidia Team,
We are developing a custom carrier board with AGX Orin and BSP based on L4T 36.x and are utilizing a custom Real-Time (PREEMPT_RT) kernel build.
We have a requirement where our customers need to compile out-of-tree (OOT) modules natively on the target device. This requires the target board to have the matching RT kernel headers present, with the build and source symlinks in /lib/modules/5.15.148-rt-tegra/ correctly pointing to them.
The Issue:
When building the standard (non-RT) kernel, running apply_binaries.sh will deploy the standard, non-RT L4T kernel headers.
Because the RT headers are not installed by the script, the RT kernel’s build and source symlinks on the flashed device remain broken (often pointing back to the host machine’s source paths, e.g., /home/$user/.../kernel-jammy-src). This prevents our customers from natively compiling modules.
Questions:
-
What is the recommended L4T workflow to package custom RT kernel headers and install them into the rootfs prior to flashing?
-
Is there a specific flag or environment variable for apply_binaries.sh (or nvbuild.sh) to make it deploy the RT headers instead of the standard ones?
-
If apply_binaries.sh does not support this natively for RT kernels, is the recommended workaround to manually generate headers and modules (make modules) from our RT source before flashing?
Any guidance on the standard procedure for this would be greatly appreciated.
Thank you,
Vrushti
Hi,
Please try
Jetpack 5.1 - Real-time kernel and modules - #6 by Gorgo90
The command for Jetpack 6.2.2 r36.5 is
/lib/modules/5.15.185-rt-tegra$ sudo ln -sf /usr/src/linux-headers-5.15.185-tegra-ubuntu22.04_aarch64/3rdparty/canonical/linux-jammy/kernel-source/ build
Hi DaneLL,
Can you confirm whether path /usr/src/linux-headers-5.15.185-tegra-ubuntu22.04_aarch64/3rdparty/canonical/linux-jammy/kernel-source/ would be containing standard kernel headers or rt kernel headers?
Hi,
On Jetpack 6 r36, RT kernel in enabled by executing
$ ./generic_rt_build.sh "enable"
And related configs are enabled. Don’t have any update in header files. So the header files is supposed to be identical.
Hi DaneLLL,
According to the standard Nvidia documentation, the Linux_for_Tegra/apply_binaries.sh script is responsible for creating the necessary symlinks and extracting the kernel headers into the target’s root filesystem using the kernel_headers.tbz2 archive. This process is handled by the following block in the script:
Bash
echo "Extracting the kernel headers to ${LDK_ROOTFS_DIR}/usr/src"
# The kernel headers package can be used on the target device as well as on another host.
# When used on the target, it should go into /usr/src and owned by root.
# Note that there are multiple linux-headers-* directories; one for use on an
# x86-64 Linux host and one for use on the L4T target.
EXTMOD_DIR=ubuntu22.04_aarch64
KERNEL_HEADERS_A64_DIR="$(tar tf "${LDK_KERN_DIR}/kernel_headers.tbz2" | grep "${EXTMOD_DIR}" | tail -1 | cut -d/ -f1)"
KERNEL_VERSION="$(echo "${KERNEL_HEADERS_A64_DIR}" | sed -e "s/linux-headers-//" -e "s/-${EXTMOD_DIR}//" | cut -d- -f1-2)"
KERNEL_SUBDIR="kernel-$(echo "${KERNEL_VERSION}" | cut -d. -f1-2)"
install ${INSTALL_ROOT_OPTS} -m 0755 -d "${LDK_ROOTFS_DIR}/usr/src"
pushd "${LDK_ROOTFS_DIR}/usr/src" > /dev/null 2>&1
# This tar is packaged for the host (all files 666, dirs 777) so that when
# extracted on the host, the user's umask controls the permissions.
# However, we're now installing it into the rootfs, and hence need to
# explicitly set and use the umask to achieve the desired permissions.
(umask 022 && tar -I lbzip2 --no-same-permissions -xmf "${LDK_KERN_DIR}/kernel_headers.tbz2")
The critical limitation here is that kernel_headers.tbz2 strictly packages the headers for the standard, non-RT kernel. It does not include the Real-Time (PREEMPT_RT) kernel headers.
Consequently, even though apply_binaries.sh successfully installs these standard headers onto the device, any out-of-tree modules compiled natively against them will fail to load on an RT system. When attempting to modprobe the compiled module, the running RT kernel will immediately reject it due to a module_layout / vermagic mismatch.
This limitation and resulting module failure was observed by Jetpack 5.1 - Real-time kernel and modules - #6 by Gorgo90
Hi,
We thought kernel_headers.tbz2 is identical to the header files sync’ed from nv-tegra · GitLab Probably certain header files are missing as you mention. In this condition, please try to manually copy the header files from source code to Linux_for_Tegra/rootfs
Hi DaneLLL,
Thank you for the suggestion, but simply manually copying the raw header files from the source code to the rootfs will not resolve the issue for compiling out-of-tree modules on the target.
The issue is not just that files are missing, but rather that the build environment inside the standard kernel_headers.tbz2 is configured for the non-RT kernel. Building external modules natively on the device requires a fully prepared kernel tree (make modules_prepare), which includes the RT-specific .config, the generated include/generated/ files, and critically, the Module.symvers file generated during the RT kernel build.
If we only copy the raw source code into the rootfs, the target board will lack the built scripts and the RT Module.symvers needed to pass the module_layout / vermagic check when loading the module.
Since apply_binaries.sh extracts a pre-packaged kernel_headers.tbz2 that contains the .config and Module.symvers for the standard kernel, it will still hit a vermagic mismatch when running the RT kernel.
Hi,
The two files are in
3rdparty/canonical/linux-jammy/kernel-source/Module.symvers
3rdparty/canonical/linux-jammy/kernel-source/.config
If you copy the full 3rdparty/canonical/linux-jammy/kernel-source/, the files are included.
Thanks DaneLLL for the suggestion, let me try to build rt kernel inside this source and confirm the headers installation.