Followup to TC/Enabling sch_netem

This is a followup to:
Traffic Control in Ubuntu doesn't work
and:
How to enable "CONFIG_NET_SCH_NETEM" in tegra_defconfig? - #2 by juns

I was wondering how to add the sch_netem module to the Jetson Nano kernel to enable traffic control (TC) commands. Possibly with a more straightforward/beginner-friendly explanation than previous posts on this topic. For example, how do you find and configure tegra_defconfig? What steps are required after that?

I am using the following environment:

NVIDIA Jetson Nano (Developer Kit Version)
L4T 32.5.1 [ JetPack UNKNOWN ]
Ubuntu 18.04.5 LTS
Kernel Version: 4.9.201-tegra
CUDA 10.2.89
CUDA Architecture: 5.3
OpenCV version: 4.1.1
OpenCV Cuda: NO
CUDNN: 8.0.0.180
TensorRT: 7.1.3.0
Vision Works: 1.6.0.501
VPI: ii libnvvpi1 1.0.15 arm64 NVIDIA Vision Programming Interface library

Thanks!

tegra_defconfig is a config file in kernel source. This file tells which driver should be built/complied.

Thus, to get tegra_defconfig, you need to download the kernel source first.

Our download center (dlc) has “L4T sources”.

and the steps to build kernel is as below.

https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/kernel_custom.html#wwpID0E06C0HA

Thanks! I was able to get through step 8 in the Building the NVIDIA Kernel section and built the kernel + modules from the L4T Downloaded source to the directory $TEGRA_KERNEL_OUT.

I understand where the kernel source needs to come from either the existing kernel or downloading it from L4T.

Now I’m unclear on how to actually add the “out of tree” module.

After the modules_install command (step 7 of Building the NVIDIA Kernel) , there’s a kernel_supplements.tbz2 in ~/Linux_for_Tegra/rootfs/.

The Preparing to Build External Kernel Modules section states: “On a Jetson system you can use the files in this directory (/usr/src/linux-headers-$(uname -r)-ubuntu18.04_aarch64) to build out-of-tree modules without further action” . How do I use the files here to do this?

I was under the impression that all modules were already built in step 4 of Build the NVIDIA Kernel ($ make ARCH=arm64 O=$TEGRA_KERNEL_OUT -j<n>).

The module I want (sch_netem) appears to be in the directory (~/Linux_for_Tegra/source/public/kernel/kernel-4.9/net/sched/sch_netem.c). But, building the kernel creates the .config file with the line # CONFIG_NET_SCH_NETEM is not set, so I’m not seeing it in $TEGRA_KERNEL_OUT/net/sched when I built the kernel using steps in the documentation.

I see there are there are separate instructions, for Building External Kernel Modules:

$ cd <path_to_module_source>
$ make ARCH=arm64 –C <kernel_directory> M=$(pwd)

But I’m unclear on what <path_to_module_source> and <kernel_directory> are supposed to be after completing the previous steps and if this is necessary here.

TL;DR – I believe I’ve built the entire kernel and (out of tree) modules from source – how do I actually find/use the sch_netem module? If not, what modifications do I need to do for building the kernel and modules from source?

Thanks again!

Also:
I am thinking I may need to edit the .config after the step:

$ make ARCH=arm64 O=$TEGRA_KERNEL_OUT tegra_defconfig

to include CONFIG_NET_SCH_NETEM=y, then run:

 $ make ARCH=arm64 O=$TEGRA_KERNEL_OUT -j<n> modules_prepare

However, I’m still unclear on where/how the module is installed and how to enable it in the kernel/what source directory and kernel directory to reference. E.g where do I find the modules and how do I add the module to /lib/modules/4.9.201-tegra?

Did you follow the " Preparing to Build External Kernel Modules" section?

Do you know there is a driver package “Linux_for_Tegra” which will be installed on sdkamanger on your host?

Under the “Linux_for_Tegra” there will be rootfs/lib/modules/4.9.201-tegra

Yes, I followed it – it only instructs to run:

$ make ARCH=arm64 O=$TEGRA_KERNEL_OUT -j<n> modules_prepare

If on a Jetson system (I’m doing these steps on the same Jetson that needs to use the module).

I see the sch_netem.c file for the module in Linux_for_Tegra/rootfs/lib/modules/4.9.201/source/net/sched, and the sch_netem.o file in the $TEGRA_KERNEL_OUT/net/sched directory after doing the above step, but the step after in “Building External Kernel modules” references a .ko file? Additionally, it doesn’t seem aarch64-linux-gnu-strip works with .o files.

Where do I find the .ko for the module/verify this was built correctly, and how do I load/use it as a module?

  1. The document is based on host to do cross compilation

  2. If you set CONFIG_NET_SCH_NETEM=y ,then it will be built into kernel image. That is why you don’t see it as a ko file. If you want it to be a ko file, you need to set it as ‘CONFIG_NET_SCH_NETEM=m’ .
    You can find such info regarding linux kernel over the Internet.

Thanks, makes sense. I tried both modules_prepare and modules_install with CONFIG_NET_SCH_NETEM=m and am still not finding the .ko file though.

Still unclear on how “The installed modules can be used to provide the contents of /lib/modules/<kernel_version> on the target system” – I was assuming this is the .ko for the extra module.

I don’t think that part needs your worry… it is actually just copy the ko file to the path /lib/modules… maybe write some config to let modprobe can work fine.

Thus, maybe you should check why the make process does not build the driver you want.

Thanks! I figured it out. I’m going to post a rough draft (may modify them as necessary) of my notes here in case it helps someone else since I saw the question has come up before and it isn’t documented directly anywhere. Again, this is based on https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/kernel_custom.html#wwpID0E06C0HA and this/other threads on this forum!

If the necessary modules are not included, the following error will occur when issuing TC commands:

RTNETLINK answers :No such file or directory

First, download the most recent source (driver package) for the L4T Jetson Nano kernel from L4T | NVIDIA Developer :

$ curl -L https://developer.nvidia.com/embedded/l4t/r32_release_v5.1/r32_release_v5.1/sources/t210/public_sources.tbz2 -o public_sources.tbz2

$ tar -xvf public_sources.tbz2 # untars to ~/Linux_for_Tegra 

$ cd Linux_for_Tegra/source/public
$ tar –xvjf kernel_src.tbz2

Start following the steps for “Building the NVIDIA Kernel”. Note: The purpose of this is to generate the .config file, which lists kernel modules and configuration – building the entire kernel is NOT necessary as only extra modules need to be loaded. An alternative that could be explored but is not included in the above documentation could be to copy the .config file from /proc/config.gz (the existing config file for the kernel) or from a Jetson where the kernel has already been built from the L4T source.

$ sudo apt install build-essential bc

$ export TEGRA_KERNEL_OUT=<outdir> # In this case, <outdir> is ~/kernel_test

$ cd ~/Linux_for_Tegra/source/public/kernel/kernel-4.9
$ mkdir -p $TEGRA_KERNEL_OUT
$ make ARCH=arm64 O=$TEGRA_KERNEL_OUT tegra_defconfig

$ make ARCH=arm64 O=$TEGRA_KERNEL_OUT -j4 #4 for the number of parallel processes since the Jetson Nano has 4 CPUs. This step takes a while since it's building the entire kernel from source.

Not entirely sure if the next steps are required if only adding modules, but they were performed too

Replace ~/Linux_for_Tegra/kernel/Image with a copy of TEGRA_KERNEL_OUT/arch/arm64/boot/Image

$ cp -Tr TEGRA_KERNEL_OUT/arch/arm64/boot/Image ~/Linux_for_Tegra/kernel/Image

Replace the contents of Linux_for_Tegra/kernel/dtb/ with the contents of TEGRA_KERNEL_OUT/arch/arm64/boot/dts/

$ cp -Tr TEGRA_KERNEL_OUT/arch/arm64/boot/dts/ Linux_for_Tegra/kernel/dtb/

Once your kernel is built in $TEGRA_KERNEL_OUT, you should have a .config file there. It says “DO NOT EDIT”. Edit it anyway by changing the following lines:

$ sudo vi $TEGRA_KERNEL_OUT/.config

# CONFIG_NET_SCH_NETEM is not set → CONFIG_SCH_NETEM=m

# CONFIG_NET_SCH_TBF is not set → CONFIG_NET_SCH_TBF=m

This allows modules to be built separately rather than into the kernel image, so they can be loaded into the existing kernel.

Unclear on if these steps are required or not for loading individual modules rather than building the kernel image:

Install the modules using the config files.

$ cd ~/Linux_for_Tegra/source/public/kernel/kernel-4.9

$ sudo make ARCH=arm64 O=$TEGRA_KERNEL_OUT modules_install INSTALL_MOD_PATH=~/Linux_for_Tegra/rootfs/

$ make ARCH=arm64 O=$TEGRA_KERNEL_OUT -j4 modules_prepare

Make the modules – the path used here is ~/Linux_for_Tegra/rootfs/lib/modules/4.9.201/source/net/sched, which contains the .c and .o source files for the sch_netem and sch_tbf modules.

$ make ARCH=arm64 -C $TEGRA_KERNEL_OUT M=~/Linux_for_Tegra/rootfs/lib/modules/4.9.201/source/net/sched

The above should create the kernel object files (.ko) for the modules.

Then, use depmod to add the module to the kernel dependencies, and modprobe to load the module. There are multiple ways to do this, I followed this.

$ sudo ln -s /lib/modules/4.9.201-tegra/kernel/net/sched/sch_tbf.ko /lib/modules/`uname -r`

$ sudo ln -s /lib/modules/4.9.201-tegra/kernel/net/sched/sch_netem.ko /lib/modules/`uname -r`

$ sudo depmod -a

$ sudo modprobe sch_netem

$ sudo modprobe sch_tbf

$ lsmod #verify the modules are loaded

Then, TC commands should work without the above error.