building TX1 kernel from source

Autonomous Machines Jetson & Embedded Systems Jetson TX1
1

I tried to build TX1 kernel following this link:

http://elinux.org/Jetson_TX1

When I entered:

make ARCH=arm64 CROSS_COMPILE=aarch64-unknown-linux-gnueabihf- O=build/jetson-tx1 -j32

I got:

aarch64-unknown-linux-gnueabihf-gcc: command not found

Has anyone built TX1 kernel from source?

2

Are you desiring to build upstream Linux kernel or L4T kernel? Which toolchain have you installed? Perhaps you are missing a symlink.

See this page of the thread [url]https://devtalk.nvidia.com/default/topic/894945/jetson-tx1/jetson-tx1/10[/url] for building L4T kernel.

The instructions are contained within the L4T documentation, available here: http://developer.download.nvidia.com/embedded//L4T/r23_Release_v1.0/Tegra_Linux_Driver_Package_Documents_R23.1.1.tar

After extracting the tarball, open l4t_getting_started.html and check out the “Building the NVIDIA Kernel” section.

3

Thanks for the link.

There are many good information in the thread “TX1”, sometime hard to find the right one in a long thread.

The information in

http://developer.download.nvidia.com/embedded//L4T/r23_Release_v1.0/Tegra_Linux_Driver_Package_Documents_R23.1.1.tar

is very helpful, e.g., the building PC should be Linux 32.

My Chromebook CB5 has 32 bit Ubuntu, I’ll use it to build kernel for TX1 (or Shield TV).

4

Unless your cross compiler is in your default search path, you’ll need to use the full path. So for example, if you have “aarch64-unknown-linux-gnueabihf-gcc” located in “/usr/local/cross_compile/bin/”, you’d have to set to “/usr/local/cross_compile/aarch64-unknown-linux-gnueabihf-”.

5

The elinux.org link does not provide information about cross compiler, I don’t believe I have aarch64 cross compiler installed by default. I’ll follow the links provided by dusty_nv to find ways to install cross compiler for L4T kernel.

6

I don’t know yet if it works, as I’m testing right now, but you may want to try the most recent linaro tool chain:
https://releases.linaro.org/components/toolchain/binaries/latest-5.2/

You’ll need both the aarch64-linux-gnu and also arm-linux-gnueabihf. The aarch64 is the 64-bit, while gnueabihf is the 32-bit variant for some legacy stuff. Kernels will compile with just the gcc-linaro, the runtime and sysroot shouldn’t be required for kernels (regular applications in user space would need this). On my system I configured with “./configure --prefix=/usr/local/gcc-linaro-5.2-2015.11-x86_64_aarch64-linux-gnu” and “./configure --prefix=gcc-linaro-5.2-2015.11-x86_64_arm-linux-gnueabihf”.

The R23.1 documents have a section in them for kernel cross-compile, I’m restating it here for convenience:

To build the Tegra Kernel
1. Export the following environment variables:
$ export CROSS_COMPILE=<crossbin>
$ export CROSS32CC=<cross32bin>gcc
$ export TEGRA_KERNEL_OUT=<outdir>
$ export ARCH=arm64
Where:
•<crossbin> is the prefix applied to form the path to the tool chain for cross compilation targeting arm64, e.g., gcc. For a Linaro tool chain, it will look something like:
<linaro_install>/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-
Note: This example requires GCC 4.9 or above.
•<cross32bin> is the prefix applied to form the path to the tool chain for cross compilation targeting arm32, e.g., gcc. For a CodeSourcery tool chain, it will look something like:
<csinstall>/arm-2009q1-203-arm-none-linux-gnueabi/bin/arm-none-linux-gnueabi-
And CROSS32CC would be:
<csinstall>/arm-2009q1-203-arm-none-linux-gnueabi/bin/arm-none-linux-gnueabi-gcc
Note: This example requires GCC 4.7 or above.
•<outdir> is the desired destination for the compiled kernel.
2. Execute the following commands to create the .config:
$ cd <myworkspace>/<kernel_source>
$ mkdir $TEGRA_KERNEL_OUT
Where <kernel_source> directory contains the kernel sources.
•For Tegra X1, Jetson TX1, use:
$ make O=$TEGRA_KERNEL_OUT tegra21_defconfig
Where <myworkspace> is the parent of the Git root.
3. Execute the following commands to build the kernel:
$ make O=$TEGRA_KERNEL_OUT zImage
4. Execute the following command to create the kernel device tree components:
$ make O=$TEGRA_KERNEL_OUT dtbs
5. Execute the following commands to build the kernel modules (and optionally install them)
$ make O=$TEGRA_KERNEL_OUT modules
$ make O=$TEGRA_KERNEL_OUT modules_install INSTALL_MOD_PATH=<your_destination>
6. Copy both the uncompressed (Image) and compressed (zImage) kernel images over the ones present in the ‘kernel’ directory of the release.
7. Archive the kernel modules created in Step 4 using the tar command and the filename that is used for the kernel modules TAR file in the same kernel directory of the release. When both of those TAR files are present, you can follow the instructions provided in this document to flash and load your newly built kernel.

This post shows a workaround for a bug which would otherwise halt compile:
https://devtalk.nvidia.com/default/topic/894945/jetson-tx1/jetson-tx1/post/4744666/#4744666

The mix of 32-bit and 64-bit is the reason why you need a general “CROSS_COMPILE” pointing to a directory plus name prefix, plus “CROSS32CC” pointing directly at the 32-bit gcc executable full path. The TEGRA_KERNEL_OUT is just a temp scratch directory to work in to keep compiles and configs separate from the source (just set this and use “make O-$TEGRA_KERNEL_OUT …” anywhere you have set up for scratch space).

Don’t forget under General Setup to add to the Local version, e.g., “-g3a5c467”.

7

Thanks for the links.

I’m stuck at building tool chain from “crosstool-ng: Summary” when there are many questions from

./ct-ng oldconfig

which I’m not sure how to answer.

It’s mush easier to install pre-built tool chain.

How many files do I need to download from “releases.linaro.org”, are the 2 “gcc*tar.xz” files the only ones I need? How do I install those 64 and 32 bits compilers?

8

The “tar.xz” uses “7z” to unzip; the “tar.bz2” is bunzip2. Both are the same thing but different compression. eabihf and aarch64 define if it is 32-bit ARMv7 or 64-bit ARMv8a. Currently, TX1 is a mix of 32-bit and 64-bit so far as software goes, so you need both (either both of the .bz2 or both of the .xz).

FYI, I don’t think crosstool is sufficient. All I saw was ARMv7 on it, which would imply only 32-bit.

9

Just finished this exercise adding swap to my system. Several things can throw you off when following the directions mentioned above.

If you follow the directions using make O=$TEGRA_KERNEL_OUT tegra21_defconfig will not enable wifi in your new kernel. The file tegra21_defconfig is in the kernel_source/arch/arm64/configs and you will have to edit it manually to get wifi up and rolling.

You might think you could just do a make menuconfig and solve that problem. Not so. When you do that and try and do the make it will stop and tell you to run a make mrproper which will erase your .config file and you will have to go back to the make tegra21_defconfig step so make zImage will run. The thing is you need that .config file to be in $TEGRA_KERNEL_OUT not in kernel_sources so use the O=$TEGRA_KERNEL_OUT if you try and use make menuconfig to customize the .config file.

The last step where you install modules. I found this to be necessary to build the kernel_supplements.tbz2 file that is required. Make a directory in your $TEGRA_KERNEL_OUT directory I called mine Modules and set that directory as the install point. Then you just go into that dir and tar it up for your kernel_supplements.tbz2 file. You might also need to make a tmp directory in your JetPack/Linux-for_Tegra kernel directory. Copy the kernel_headers.tbz2 into that and untar it. Check the name to make sure it matches the name of your kernel mine changed to -dirty for some reason I’m not sure off So change the name of the directory made with the untarring and tar it back up. Move that back into the kernel directory and kill the tmp directory before flashing. BE SURE AND RUN apply_binaries.sh before flashing otherwise all your work is for naught as the kernel will not change. Then you can just run the jetpack normally and reflash your TX1.

Now if you don’t want to mess with that part of the procedure that makes the .config file do that part on your TX1 and move the resulting .config file to the cross-compile box. Put it in the $TEGRA_KERNEL_OUT directory. Or you can get it by doing a zcat /proc/config.gz > config which will give you the config file for the running kernel on the TX1. You can do a make menuconfig using that file or just edit it by hand. The stuff that isn’t implemented is commented out and easy to spot.

Make sure you have copied Image and zImage from kernel_sources/arch/arm64/boot into Jetpack/Linux_for_Tegra/kernel and also the dts directory from there into the dtb directory in kernel. That stuff plus the two tbz2 files are necessary before running apply_binaries.sh to add them to the dummy filesystem that is going to be flashed.

The instructions in the pdf work perfectly with these little additions. Now to figure out why opencv stuff wont compile on the tk1 once you put jetpack2 on it. Also breaks the tx1. Thinks it needs opencv_ocl which is opencl and doesn’t even run on the jetson. I’ll set up one of my tk1’s tomorrow without the jetpack2.0 on it and compare the cmake files and see if I can figure it out.

I finally was able to make crosstool-ng create a toolchain but its a real pain. Using the Linaro compilers its easy. I just have two directories in my home dir. ARM32-Toolchain and ARM64-Toolchain. Using the env variables to point to them and you are all set.

1 Like
10

Forgive me for being blunt, but NEVER edit a config file manually. I’m glad it worked for you but it’s more likely to screw things up and others won’t know how to help you. If you’re running into a problem with the config post the exact steps you are taking, so the community can suggest the proper way to fix it or patch the kernel tree properly.

Did you forget the ARCH variable when you did make menuconfig?

11

Thanks everyone for great instructions and links, I was able to download/install cross compilers and implement 2 workarounds:

#(1). To Fix ‘error: r7 cannot be used in asm here’ problem

add he followin

KBUILD_CFLAGS_KERNEL := -fomit-frame-pointer

to line 378 of makefile

#(2). To fix error “logical not is only applied to the left hand side of comparison”

Chnage line 1065 of “tegra21_clocks.c” to:

c->state = ((!is_lp_cluster()) == (c->u.cpu.mode == MODE_G)) ? ON : OFF;

setup environments:

export CROSS_COMPILE=$HOME/crosstools/gcc-linaro-5.2-2015.11-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-
export CROSS32CC=$HOME/crosstools/gcc-linaro-5.2-2015.11-x86_64_arm-linux-gnueabihf/bin/arm-linux-gnueabi-gcc
export TEGRA_KERNEL_OUT=outdir
export ARCH=arm64

and make kernel and modules:

make O=$TEGRA_KERNEL_OUT tegra21_defconfig
make O=$TEGRA_KERNEL_OUT zImage
make O=$TEGRA_KERNEL_OUT dtbs
make O=$TEGRA_KERNEL_OUT modules

I just want to make sure the next step is for rootfs, not the kernel:

make O=$TEGRA_KERNEL_OUT modules_install INSTALL_MOD_PATH=<your_destination>

where <your_destination> should be “rootfs” directory.

12

No I didn’t forget the arch variable. Those are the default config files they don’t have wifi enabled. I want my default config file with wifi enabled.

I don’t know but I’ve edited many config files in the 20+ years I’ve been hacking on unix. You stick with the “Rules” I’ll get stuff done. I mentioned the other ways to do it if you paid attention. The pdf is unclear that you can do anything other than use make tegra21_defconfig.

By the way the config files that are generated are marked as such and you shouldn’t edit those except in extreme situations but these aren’t marked that way at all. In fact that was the way unix kernels were configured in the past. Just by editing a text file. All this other stuff is just to make it easy for the masses. There is a dictionary of all the config file keywords on the net so its just not that hard to manually config a kernel. If you look there are several defconfig files I just added my own custom one. Now you have to know a little about what config words require other config words to be enabled but that comes with time. And if I screw things up I just reverse the steps and go another way.

If you build a kernel according to the instructions and you flash only to discover you have no wifi now what way do you suggest to go? You do a make menuconfig like you’ve done hundreds of times and run into another error that fixing nukes your .config file and you have to start over. Do you read hundreds of pages of stuff that has no bearing on the actual problem or do you cut to the chase fix the problem and learn some stuff along the way? I don’t want to be an expert on building kernels I want to build robots. So the quickest way is the way I’m going.

The only clue is the tegra21_defconfig file. Find that read it and everything becomes clear why your wifi doesnt’ work. No need to google or get out the books. Copy the original file to a new filename edit the file build the kernel find out it works then if you want you can find out the “correct” way to do it. Which I did and documented.

I could have just put the steps in there with out the background or other ways to get around it but then others have no idea how I went about figuring out what was causing the problem. Now they know if they have a problem with the kernel they can simply read that modified file and see what’s up quickly without going to all the trouble of wading through all those screens on menuconfig. They also now have the knowledge to compare the .config file with the tegra21_defconfig file to diagnose problems. The know where the tegra21_defconfig file is located and what it is. Its not a mystery any more. They have somewhere to go when running make tegra21_defconfig doesnt’ give them the expected results.

If you go at Linux following all the “rules” you are defeating the purpose of an open source OS. And leaving yourself with no out when those “rules” don’t work. Because you never went outside the lines and learned the underpinnings of the “rules”. I could have limited it to a do this do that piece but that only plays to a limited audience.

There was no problem it was a narrative of my experience trying to follow the directions in the pdf. And in my opinion it needs to be fleshed out a bit. Someone that doesn’t usually build kernels is going to think those commands are gospel and when they end up with no wifi arent’ going to have an idea how to go about making it work. Several differences between doing it on another machine than doing it on the target machine that most people are familiar with. There is nothing to fix in the kernel tree. The standard arm defconfig files don’t have wifi enabled. If you want wifi and want to use the commands in that pdf to build your new kernel that is the only way to end up with working wifi. I don’t expect Nvidia to change them either they would be going against standards. Any change would be in the pdf itself so the community wouldn’t get involved in any way.

Griz

13

This is the first time I noticed “make mrproper” when building kernels. Can makefile be modified such that “make mrproper” is bypassed?

I’ll need a “.config” file from Shield TV to make new kernel for Shiled TV.

14

You can apply many of the “make” commands with “O=$TEGRA_KERNEL_OUT” modifying where the config is. I think you can do this with “make mrproper” and “make menuconfig”. If you aren’t using that output location switch, it’ll refer to the kernel source tree itself. I’d suggest “make O=$TEGRA_KERNEL_OUT mrproper” and then “make O=$TEGRA_KERNEL_OUT menuconfig” as a test, or even substitute make menuconfig and make tegra21_defconfig with simply copying your .config to the O=$TEGRA_KERNEL_OUT directory. Without the “O=” expect messages to apply to the source tree, with “O=” many messages refer to the output build tree.

So I’d recommend something like this (assumes you set your environment variable TEGRA_KERNEL_OUT already), run from the kernel source tree:

make mrproper
make O=$TEGRA_KERNEL_OUT mrproper
...copy your .config to $TEGRA_KERNEL_OUT...or:
make O=$TEGRA_KERNEL_OUT menuconfig
...or:
make O=$TEGRA_KERNEL_OUT tegra21_defconfig
...so on
15

Good suggestions, I’ll try

make O=$TEGRA_KERNEL_OUT mrproper

and

make O=$TEGRA_KERNEL_OUT menuconfig

16

I extracted and ubuntu arm64 tarbal into /arm64-chroot and used that to build / rebuild the kernel.

The provided kernel is perfectly able to load arm64 binaries and it saves all the hassle associated with running a cross toolchain.

17

I was also able to use archlinux Generic aarch64 rootfs for L4T kernel on Shield TV.

Can you provide some details on how did you extract ubuntu arm64 tarbal into /arm64-chroot? Did you extract ubuntu arm64 tarbal from an Intel PC?

18

What changes are needed to enable wifi?

19

Some of the TX1s had a firmware file in /lib/firmware/brcm/ causing issues. Not all TX1s have this firmware issue, and I’m not sure what shipped or didn’t ship with the issue, nor do I know if all R23.1 flashes would produce this incorrect firmware file. The gist is that if this file is incorrect you’ll have a large amount of CPU use resulting in a sluggish system, while wifi fails. See:
https://devtalk.nvidia.com/default/topic/901180/jetson-tx1/my-jetson-tx1-is-not-able-to-connect-to-wifi/

I don’t have wireless to test yet, but I believe the unit I have (not yet flashed) works with WiFi (at least when it was enabled it saw networks and didn’t consume CPU). The kernel config on this unflashed unit shows:

# grep WIFI config-3.10.67-g3a5c467 
# CONFIG_TEGRA_PREPOWER_WIFI is not set
CONFIG_WIFI_CONTROL_FUNC=y
CONFIG_IWLWIFI=m
CONFIG_IWLWIFI_OPMODE_MODULAR=y
# CONFIG_IWLWIFI_DEBUG is not set
# CONFIG_IWLWIFI_DEVICE_TRACING is not set
CONFIG_IWLWIFI_DEVICE_TESTMODE=y
CONFIG_IWLWIFI_P2P=y
# CONFIG_RTLWIFI is not set
# CONFIG_MWIFIEX is not set
# CONFIG_CSR_WIFI is not set
20

Thanks for clarification of wifi issue, I thought it was kernel configuration issue.