Jetpack 5.1 - Real-time kernel and modules

[Gorgo90]

I’m trying to get the correct steps to compile a working real-time kernel for my Jetson NX and I want to compile a networking module (8821cu) on-board eventually.

As reference I’m following the instructions here (Kernel Customization — Jetson Linux Developer Guide documentation) and the commands posted here (No display with PREEMPT_RT patches) that are sometimes linked in other similar threads.
The commands are the same but the last one, where I use:

sudo tools/l4t_create_default_user.sh -u nvidia -p nvidia -a -n nvidia-desktop --accept-license
sudo ./flash.sh --no-flash p3449-0000+p3668-0001-qspi-emmc mmcblk0p1
sudo ./flash.sh -r p3449-0000+p3668-0001-qspi-emmc mmcblk0p1

I’m cross-compiling the kernel and modules from a Ubuntu 20.04 host and I’m using the toolchain 2020.08 (9.3.0).

I extracted the sources (manually-downloading-and-expanding-kernel-sources) and Sample_Rootfs + L4T from the website.

Before installing modules, rootf/lib/modules/ doesn’t exist, as expected.

When I run sudo ./apply_binaries.sh it installs all the NVIDIA packages and also some modules in rootfs/lib/modules/5.10.104-tegra/ (without rt).

gorgo@razer:/media/gorgo/PopOS/home/gorgo/NX_5_1_HYAI/Linux_for_Tegra$ ls -la rootfs/lib/modules/5.10.104-tegra/
total 2012
drwxr-xr-x 4 root root 4096 mar 28 15:02 .
drwxr-xr-x 3 root root 4096 mar 28 15:02 …
lrwxrwxrwx 1 root root 69 gen 25 03:33 build → /usr/src/linux-headers-5.10.104-tegra-ubuntu20.04_aarch64/kernel-5.10
drwxr-xr-x 3 root root 4096 mar 28 15:02 extra
drwxr-xr-x 9 root root 4096 mar 28 15:02 kernel
-rw-r–r-- 1 root root 507529 mar 28 15:02 modules.alias

Later with

cd $ROOT/Linux_for_Tegra/source/public/kernel_out
sudo make INSTALL_MOD_PATH=$ROOT/Linux_for_Tegra/rootfs O= modules_install

It installs the modules in rootfs/lib/modules/5.10.104-rt63-tegra/. (with rt)

gorgo@razer:/media/gorgo/PopOS/home/gorgo/NX_5_1_HYAI/Linux_for_Tegra$ ls -la rootfs/lib/modules/5.10.104-rt63-tegra/
total 2016
drwxr-xr-x 3 root root 4096 mar 28 15:04 .
drwxr-xr-x 4 root root 4096 mar 28 15:04 …
lrwxrwxrwx 1 root root 82 mar 28 15:04 build → /media/gorgo/PopOS/home/gorgo/NX_5_1_HYAI/Linux_for_Tegra/source/public/kernel_out
drwxr-xr-x 9 root root 4096 mar 28 15:04 kernel
-rw-r–r-- 1 root root 515446 mar 28 15:04 modules.alias
-rw-r–r-- 1 root root 506829 mar 28 15:04 modules.alias.bin
-rw-r–r-- 1 root root 27993 mar 28 15:04 modules.builtin
-rw-r–r-- 1 root root 50452 mar 28 15:04 modules.builtin.alias.bin
-rw-r–r-- 1 root root 30390 mar 28 15:04 modules.builtin.bin
-rw-r–r-- 1 root root 166915 mar 28 15:04 modules.builtin.modinfo
-rw-r–r-- 1 root root 84507 mar 28 15:04 modules.dep
-rw-r–r-- 1 root root 134914 mar 28 15:04 modules.dep.bin
-rw-r–r-- 1 root root 187 mar 28 15:04 modules.devname
-rw-r–r-- 1 root root 45918 mar 28 15:04 modules.order
-rw-r–r-- 1 root root 829 mar 28 15:04 modules.softdep
-rw-r–r-- 1 root root 202723 mar 28 15:04 modules.symbols
-rw-r–r-- 1 root root 252534 mar 28 15:04 modules.symbols.bin
lrwxrwxrwx 1 root root 90 mar 28 15:04 source → /media/gorgo/PopOS/home/gorgo/NX_5_1_HYAI/Linux_for_Tegra/source/public/kernel/kernel-5.10

I find it strange that build and source point to a local folder, differently from the build folder in 5.10.104-tegra. Why?

Flash log file:
flashLog.log (80.5 KB)

Once the flash is done, the NX boots with some errors you see in the screen below:

After booting I can access to the NX via SSH (and via GUI as well):

nvidia@nvidia-desktop:~$ uname -a
Linux nvidia-desktop 5.10.104-rt63-tegra #1 SMP PREEMPT Tue Mar 28 11:28:03 +04 2023 aarch64 aarch64 aarch64 GNU/Linux

nvidia@nvidia-desktop:~$ lsmod
Module Size Used by
fuse 131072 3
lzo_rle 16384 16
lzo_compress 16384 1 lzo_rle
zram 32768 4
realtek 24576 1
loop 40960 1
nvgpu 2760704 43
snd_soc_tegra186_asrc 40960 1
snd_soc_tegra186_dspk 20480 2
snd_soc_tegra210_ope 36864 1
snd_soc_tegra186_arad 28672 2 snd_soc_tegra186_asrc
snd_soc_tegra210_iqc 16384 0
snd_soc_tegra210_mvc 20480 2
snd_soc_tegra210_afc 20480 6
snd_soc_tegra210_mixer 49152 1
snd_soc_tegra210_admaif 131072 1
snd_soc_tegra210_dmic 20480 4
snd_soc_tegra210_adx 32768 4
snd_soc_tegra210_amx 36864 4
snd_soc_tegra_pcm 16384 1 snd_soc_tegra210_admaif
snd_soc_tegra210_i2s 24576 6
snd_soc_tegra210_sfc 61440 4
aes_ce_blk 36864 0
crypto_simd 24576 1 aes_ce_blk
cryptd 32768 1 crypto_simd
aes_ce_cipher 20480 1 aes_ce_blk
ghash_ce 28672 0
sha2_ce 20480 0
sha256_arm64 28672 1 sha2_ce
sha1_ce 20480 0
pwm_fan 24576 0
snd_soc_tegra_machine_driver 16384 0
snd_soc_spdif_tx 16384 0
leds_gpio 16384 0
snd_hda_codec_hdmi 61440 4
snd_soc_tegra210_adsp 774144 1
snd_soc_tegra_utils 32768 3 snd_soc_tegra210_admaif,snd_soc_tegra_machine_driver,snd_soc_tegra210_adsp
snd_hda_tegra 16384 0
max77620_thermal 16384 0
ina3221 24576 0
snd_soc_simple_card_utils 24576 1 snd_soc_tegra_utils
snd_hda_codec 135168 2 snd_hda_codec_hdmi,snd_hda_tegra
snd_soc_tegra210_ahub 1245184 3 snd_soc_tegra210_ope,snd_soc_tegra210_sfc
nvadsp 118784 1 snd_soc_tegra210_adsp
tegra_bpmp_thermal 16384 0
tegra210_adma 28672 2 snd_soc_tegra210_admaif,snd_soc_tegra210_adsp
snd_hda_core 94208 3 snd_hda_codec_hdmi,snd_hda_codec,snd_hda_tegra
userspace_alert 16384 0
nv_imx219 20480 0
spi_tegra114 32768 0
binfmt_misc 24576 1
nvmap 217088 263 nvgpu
ip_tables 36864 0
x_tables 53248 1 ip_tables

build and source are still linking to my host machine:

nvidia@nvidia-desktop:~$ ls -larth /usr/lib/modules/5.10.104-rt63-tegra/build
lrwxrwxrwx 1 root root 82 Mar 28 2023 /usr/lib/modules/5.10.104-rt63-tegra/build → /media/gorgo/PopOS/home/gorgo/NX_5_1_HYAI/Linux_for_Tegra/source/public/kernel_out

nvidia@nvidia-desktop:~$ ls -larth /usr/lib/modules/5.10.104-rt63-tegra/source
lrwxrwxrwx 1 root root 90 Mar 28 2023 /usr/lib/modules/5.10.104-rt63-tegra/source → /media/gorgo/PopOS/home/gorgo/NX_5_1_HYAI/Linux_for_Tegra/source/public/kernel/kernel-5.10

I updated/upgraded the packets.
I rebooted and then I cloned the 8821cu repo in my home directory.

Now the repo install-driver.sh fails when it looks for the kernel headers:

#check to see if the correct header files are installed
if [ ! -d “/lib/modules/$(uname -r)/build” ]; then
echo “Your kernel header files aren’t properly installed.”
echo “Please consult your distro documentation or user support forums.”
echo “Once the header files are properly installed, please run "sudo ./${SCRIPT_NAME}"”
exit 1
fi

As said above, my build link is broken.
I cannot neither compile it by sources for the same reason:

nvidia@nvidia-desktop:~/8821cu-20210916$ make
make ARCH=arm64 CROSS_COMPILE= -C /lib/modules/5.10.104-rt63-tegra/build M=/home/nvidia/8821cu-20210916 modules
make[1]: *** /lib/modules/5.10.104-rt63-tegra/build: No such file or directory. Stop.
make: *** [Makefile:2501: modules] Error 2

In the documentation (Kernel Customization — Jetson Linux Developer Guide documentation) > Preparing to Build External Kernel Modules you claim to use /usr/src/linux-headers-$(uname -r)-ubuntu20.04_aarch64 but it doesn’t exist. /usr/src/linux-headers-5.10.104-tegra-ubuntu20.04_aarch64/ exists.

What’s missing here?
Thank you for the support!

[DaveYYY]

Hi, which device are you using by saying “Jetson NX”, is a Xavier NX module?
Also did you mean that your device booted successfully with the RT kernel patch,
but you failed to build the custom module because header files were missing?

[Gorgo90]

Hi @DaveYYY
Yes It’s a Xavier NX on a HYAI 310UAV (even if the problem is generic for any carrier board; I didn’t apply any patches but the RT patch).

Also did you mean that your device booted successfully with the RT kernel patch,
but you failed to build the custom module because header files were missing?

Correct. I had no problem with the flashing process but I’m having problem with installing external modules.
The kernel’s default 8821cu drivers are outdated and I usually install the module from this repository (GitHub - morrownr/8821cu-20210916: Linux Driver for USB WiFi Adapters that are based on the RTL8811CU, RTL8821CU, RTL8821CUH and RTL8731AU Chipsets - v5.12.0.4) without the RT patch.
Unfortunately I cannot either because I don’t have the kernel headers installed (I read you don’t support them via apt anymore and I’m not sure how to properly install them) and also because I cannot use make due to the error above (/lib/modules/5.10.104-rt63-tegra/build is broken).

As plan B I’m trying to cross-compile it on my x64 host, following the NVIDIA doc but the Makefile doesn’t seems to generate any ko file from the repo. I’ve already issued the problem to its maintainer.
I guess my procedure is correct though:

gorgo@razer:~/8821cu-20210118$ make ARCH=arm64 CROSS_COMPILE=~/Downloads/l4t-gcc/bin/aarch64-buildroot-linux-gnu- -C /media/gorgo/PopOS/home/gorgo/NX_5_1_HYAI/Linux_for_Tegra/source/public/kernel_out M=$(pwd)
make: Entering directory '/media/gorgo/PopOS/home/gorgo/NX_5_1_HYAI/Linux_for_Tegra/source/public/kernel_out'
  MODPOST /home/gorgo/8821cu-20210118/Module.symvers
make: Leaving directory '/media/gorgo/PopOS/home/gorgo/NX_5_1_HYAI/Linux_for_Tegra/source/public/kernel_out'

[DaveYYY]

Hi, we are also missing header files after building the RT kernel,
and we’ll check with our internal team to provide a solution.
Also want to know if delete the original build folder link to the host and copy the one from /usr/lib/modules/5.10.104-tegra/ to /usr/lib/modules/5.10.104-rt63-tegra/ works?

[Gorgo90]

Hi, glad you can reproduce the problem :)
Yes I changed the symbolic link to build with

sudo ln -s /usr/src/linux-headers-5.10.104-tegra-ubuntu20.04_aarch64/kernel-5.10/ build

nvidia@nvidia-desktop:~$ ls -la /lib/modules/5.10.104-rt63-tegra/
total 2032
drwxr-xr-x 4 root root 4096 Mar 28 12:50 .
drwxr-xr-x 4 root root 4096 Mar 28 11:04 …
lrwxrwxrwx 1 root root 70 Mar 28 12:47 build → /usr/src/linux-headers-5.10.104-tegra-ubuntu20.04_aarch64/kernel-5.10/
drwxr-xr-x 3 root root 4096 Mar 28 11:12 extra
drwxr-xr-x 9 root root 4096 Mar 28 11:04 kernel
-rw-r–r-- 1 root root 516702 Mar 28 12:50 modules.alias
-rw-r–r-- 1 root root 508078 Mar 28 12:50 modules.alias.bin
-rw-r–r-- 1 root root 27993 Mar 28 11:04 modules.builtin
-rw-r–r-- 1 root root 50452 Mar 28 12:50 modules.builtin.alias.bin
-rw-r–r-- 1 root root 30390 Mar 28 12:50 modules.builtin.bin
-rw-r–r-- 1 root root 166915 Mar 28 11:04 modules.builtin.modinfo
-rw-r–r-- 1 root root 84743 Mar 28 12:50 modules.dep
-rw-r–r-- 1 root root 135276 Mar 28 12:50 modules.dep.bin
-rw-r–r-- 1 root root 187 Mar 28 12:50 modules.devname
-rw-r–r-- 1 root root 45918 Mar 28 11:04 modules.order
-rw-r–r-- 1 root root 829 Mar 28 12:50 modules.softdep
-rw-r–r-- 1 root root 203414 Mar 28 12:50 modules.symbols
-rw-r–r-- 1 root root 253332 Mar 28 12:50 modules.symbols.bin
lrwxrwxrwx 1 root root 90 Mar 28 11:04 source → /media/gorgo/PopOS/home/gorgo/NX_5_1_HYAI/Linux_for_Tegra/source/public/kernel/kernel-5.10

In this way I can manually build from sources:

nvidia@nvidia-desktop:~/8821cu-20210916$ make
make ARCH=arm64 CROSS_COMPILE= -C /lib/modules/5.10.104-rt63-tegra/build M=/home/nvidia/8821cu-20210916 modules
make[1]: Entering directory ‘/usr/src/linux-headers-5.10.104-tegra-ubuntu20.04_aarch64/kernel-5.10’
CC [M] /home/nvidia/8821cu-20210916/core/rtw_cmd.o
CC [M] /home/nvidia/8821cu-20210916/core/rtw_security.o
…

No errors with install:

nvidia@nvidia-desktop:~/8821cu-20210916$ sudo make install
install -p -m 644 8821cu.ko /lib/modules/5.10.104-rt63-tegra/kernel/drivers/net/wireless/
/sbin/depmod -a 5.10.104-rt63-tegra

But when loaded is gives me this error:

nvidia@nvidia-desktop:~/8821cu-20210916$ sudo dmesg | grep 8821
[sudo] password for nvidia:
[ 10.177618] 8821cu: disagrees about version of symbol module_layout

May it be due to different gcc between host and Jetson?
To build the RT kernel I used the recommended toolchain 2020.08 (9.3.0) while the Jetpack 5.1 uses 9.4.0.
If not, I don’t why I’m getting this error

[DaveYYY]

Hi, we’ve been tracking this issue internally, and will let you know when there’s update on it.

[Gorgo90]

@DaveYYY
Hi, any news on this issue? We need them to correctly build a new module in our Orin NX (which has the same problem above)

[DaveYYY]

Hi,

sorry that because the priority of this issue is relatively lower, we haven’t been able to allocate resources for fixing it.
Please try if directly building the kernel on Jetson devices works.