Followup to TC/Enabling sch_netem

Thanks! I figured it out. I’m going to post a rough draft (may modify them as necessary) of my notes here in case it helps someone else since I saw the question has come up before and it isn’t documented directly anywhere. Again, this is based on https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/kernel_custom.html#wwpID0E06C0HA and this/other threads on this forum!

If the necessary modules are not included, the following error will occur when issuing TC commands:

RTNETLINK answers :No such file or directory

First, download the most recent source (driver package) for the L4T Jetson Nano kernel from https://developer.nvidia.com/embedded/linux-tegra :

$ curl -L https://developer.nvidia.com/embedded/l4t/r32_release_v5.1/r32_release_v5.1/sources/t210/public_sources.tbz2 -o public_sources.tbz2

$ tar -xvf public_sources.tbz2 # untars to ~/Linux_for_Tegra 

$ cd Linux_for_Tegra/source/public
$ tar –xvjf kernel_src.tbz2

Start following the steps for “Building the NVIDIA Kernel”. Note: The purpose of this is to generate the .config file, which lists kernel modules and configuration – building the entire kernel is NOT necessary as only extra modules need to be loaded. An alternative that could be explored but is not included in the above documentation could be to copy the .config file from /proc/config.gz (the existing config file for the kernel) or from a Jetson where the kernel has already been built from the L4T source.

$ sudo apt install build-essential bc

$ export TEGRA_KERNEL_OUT=<outdir> # In this case, <outdir> is ~/kernel_test

$ cd ~/Linux_for_Tegra/source/public/kernel/kernel-4.9
$ mkdir -p $TEGRA_KERNEL_OUT
$ make ARCH=arm64 O=$TEGRA_KERNEL_OUT tegra_defconfig

$ make ARCH=arm64 O=$TEGRA_KERNEL_OUT -j4 #4 for the number of parallel processes since the Jetson Nano has 4 CPUs. This step takes a while since it's building the entire kernel from source.

Not entirely sure if the next steps are required if only adding modules, but they were performed too

Replace ~/Linux_for_Tegra/kernel/Image with a copy of TEGRA_KERNEL_OUT/arch/arm64/boot/Image

$ cp -Tr TEGRA_KERNEL_OUT/arch/arm64/boot/Image ~/Linux_for_Tegra/kernel/Image

Replace the contents of Linux_for_Tegra/kernel/dtb/ with the contents of TEGRA_KERNEL_OUT/arch/arm64/boot/dts/

$ cp -Tr TEGRA_KERNEL_OUT/arch/arm64/boot/dts/ Linux_for_Tegra/kernel/dtb/

Once your kernel is built in $TEGRA_KERNEL_OUT, you should have a .config file there. It says “DO NOT EDIT”. Edit it anyway by changing the following lines:

$ sudo vi $TEGRA_KERNEL_OUT/.config

# CONFIG_NET_SCH_NETEM is not set → CONFIG_SCH_NETEM=m

# CONFIG_NET_SCH_TBF is not set → CONFIG_NET_SCH_TBF=m

This allows modules to be built separately rather than into the kernel image, so they can be loaded into the existing kernel.

Unclear on if these steps are required or not for loading individual modules rather than building the kernel image:

Install the modules using the config files.

$ cd ~/Linux_for_Tegra/source/public/kernel/kernel-4.9

$ sudo make ARCH=arm64 O=$TEGRA_KERNEL_OUT modules_install INSTALL_MOD_PATH=~/Linux_for_Tegra/rootfs/

$ make ARCH=arm64 O=$TEGRA_KERNEL_OUT -j4 modules_prepare

Make the modules – the path used here is ~/Linux_for_Tegra/rootfs/lib/modules/4.9.201/source/net/sched, which contains the .c and .o source files for the sch_netem and sch_tbf modules.

$ make ARCH=arm64 -C $TEGRA_KERNEL_OUT M=~/Linux_for_Tegra/rootfs/lib/modules/4.9.201/source/net/sched

The above should create the kernel object files (.ko) for the modules.

Then, use depmod to add the module to the kernel dependencies, and modprobe to load the module. There are multiple ways to do this, I followed this.

$ sudo ln -s /lib/modules/4.9.201-tegra/kernel/net/sched/sch_tbf.ko /lib/modules/`uname -r`

$ sudo ln -s /lib/modules/4.9.201-tegra/kernel/net/sched/sch_netem.ko /lib/modules/`uname -r`

$ sudo depmod -a

$ sudo modprobe sch_netem

$ sudo modprobe sch_tbf

$ lsmod #verify the modules are loaded

Then, TC commands should work without the above error.

3 Likes