Custom Kernel compilations

Evidently 24 hours/day isn’t sufficient time to figure out how to do anything useful with your development kit…

Anyway, since the Jetpack seems to know how to build a kernel is there a way to break into the installer process and change the default kernel configuration settings? Any help making trivial changes without spending hours of research would be helpful…

Jetpack isn’t actually required (or all that useful) in terms of putting in a new kernel. Back when fastboot was used, any kernel change required flash…but all of the current generation uses u-boot, so it is a simple file copy (and perhaps extlinux.conf edit on the Jetson). So far as I know though, the installer never compiles or modifies a kernel, it only chooses among the sample rootfs and possibly the kernel/firmware indicated by flash options (of which the kernel would be pre-compiled and not built on demand).

There are many details which can be answered here, but to give a more specific answer, what are your exact needs, e.g., build a kernel with different options? Info on installing that kernel? So on.

for now, what I really want to do is add loadable parallel port driver modules and the user mode SPI device driver so that I can use the SPI available on J21 as well as a card on the PCIe. Since the tools are different from what I’ve used before I was following a tutorial for doing something similar on the TK1. Using the kernel sources on a USB drive I was able to do the usual make menuconfig, select what I want and save it but of course building the kernel on the target fails.

Of course the nVidia documentation doesn’t mention how to create a non-default kernel using the host tools and even trying to follow what is there is almost useless to a novice. Before wasting time trying to work on what I bought the TX1 for I’d like to have the sense that I can do basic tasks. It’s not as if I haven’t done this before for other arm targets but the vendors were a bit kinder about providing supporting tools, environments, and useful documentation.

In most Linux kernel build cases a single compiler is used, either natively or via cross-compile. Those cases, including the Jetson TK1 (emphasis “K”, not “X”), are easy to deal with since all of the tools are generally available natively on the system which will use the kernel. Anything cross-compiled adds at least a slight complexity, as a cross-compiler tool chain needs to be installed, and a few environment variables may need to be set in order to tell the build that the host isn’t what the target is.

On the Jetson TX1 (emphasis “X”) the kernel build environment is slightly different for now; I would expect the environment to be simplified two releases from now (currently L4T R23.1 is the JTX1 release…another release should be coming soon, but environment simplification is not expected until the release after that). The complication is that parts of the software are running 64-bit, while other parts are running 32-bit. Thus two compilers are required. This would make native compile of a kernel on a JTX1 more complicated than cross-compile via an x86_64 host.

The document describing kernel cross-compile is on this page:
https://developer.nvidia.com/embedded/linux-tegra

The specific document is “Jetson TX1 Developer Kit User Guide”, direct URL is:
http://developer.download.nvidia.com/embedded/L4T/r23_Release_v1.0/NVIDIA_Jetson_TX1_Developer_Kit_User_Guide.pdf

Within ths document, see “Building the NVIDIA Kernel”.

If you are using an Ubuntu desktop host, I believe you simply download and install the cross-compile tools via ordinary package management. I’m using Fedora, so I manually download Linaro compilers. My compilers are located on my local host under these paths:

/usr/local/gcc-linaro-5.2-2015.11-x86_64_aarch64-linux-gnu
/usr/local/gcc-linaro-5.2-2015.11-x86_64_arm-linux-gnueabihf

An excerpt from this guide:

1. Export the following environment variables:
$ export CROSS_COMPILE=<crossbin>
$ export CROSS32CC=<cross32bin>gcc
$ export TEGRA_KERNEL_OUT=<outdir>
$ export ARCH=arm64

For my setup (adjust for your install locations), assuming output to “/home/me/build”, this means:

1. Export the following environment variables:
$ export CROSS_COMPILE=/usr/local/gcc-linaro-5.2-2015.11-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-
$ export CROSS32CC=/usr/local/gcc-linaro-5.2-2015.11-x86_64_arm-linux-gnueabihf/bin/arm-linux-gnueabihf-gcc
$ export TEGRA_KERNEL_OUT=/home/me/build
$ export ARCH=arm64

Note that CROSS32CC contains a full path to a 32-bit gcc program, while CROSS_COMPILE is a prefix to an entire set of 64-bit programs starting with “arch64-linux-gnu-”.

Tip on cross-compile of kernels: Many of the kernel build commands respond to “TEGRA_KERNEL_OUT”. Not all examples use this variable, but “make menuconfig”, “make clean”, and “make mrproper” respond to this. Don’t forget to use this when cross-compiling for all of those helper commands.

The rest will be the usual Linux kernel procedures for whatever build there is, e.g., start with a known working “.config”. If the only modifications are in the format of a module, then your $TEGRA_KERNEL_OUT location should produce the modules you need…modifying non-module config may require more extensive copy instructions. Because the full version naming of a default Jetson TX1 has suffix “-g3a5c467”, you will want to set this up to match in the CONFIG_LOCALVERSION (should be make menuconfig “general” settings, and then second line down); should you modify more than modules, you will want to use a modification of “-g3a5c467” and go for complete kernel install procedures (this is not custom to Jetson, it is simply one of the details for manual build and install of Linux kernels).

I am trying to simply build the kernel ‘out-of-the-box’ without any modifications.
I am cross-compiling from a desktop pc running 32-bit linux os. Here’s my uname -a:
Linux endo-sw2-desktop 3.13.0-43-generic #72-Ubuntu SMP Mon Dec 8 19:35:44 UTC 2014 i686 i686 i686 GNU/Linux

I was able to download via apt-get both the aarch64-linux-gnu tools and the arm-linux-gnueabihf tools and set the environment variables appropriately. However, I get compiler errors during the kernel build. First I get a ton of warnings during the compile of kuser32.S of this nature:

aarch64-linux-gnu-gcc -Wp,-MD,arch/arm64/kernel/.kuser32.o.d -nostdinc -isystem /usr/lib/gcc-cross/aarch64-linux-gnu/4.8/include -I/home/endo-sw2/linux-for-tegra-x1/sources/kernel/arch/arm64/include -Iarch/arm64/include/generated -I/home/endo-sw2/linux-for-tegra-x1/sources/kernel/include -Iinclude -I/home/endo-sw2/linux-for-tegra-x1/sources/kernel/arch/arm64/include/uapi -Iarch/arm64/include/generated/uapi -I/home/endo-sw2/linux-for-tegra-x1/sources/kernel/include/uapi -Iinclude/generated/uapi -include /home/endo-sw2/linux-for-tegra-x1/sources/kernel/include/linux/kconfig.h -D__KERNEL__ -mlittle-endian -I/home/endo-sw2/linux-for-tegra-x1/sources/kernel/arch/arm/mach-tegra/include -Iarch/arm/mach-tegra/include -D__ASSEMBLY__ -gdwarf-2 -c -o arch/arm64/kernel/kuser32.o /home/endo-sw2/linux-for-tegra-x1/sources/kernel/arch/arm64/kernel/kuser32.S
In file included from /home/endo-sw2/linux-for-tegra-x1/sources/kernel/include/asm-generic/unistd.h:1:0,
from /home/endo-sw2/linux-for-tegra-x1/sources/kernel/arch/arm64/include/uapi/asm/unistd.h:16,
from /home/endo-sw2/linux-for-tegra-x1/sources/kernel/arch/arm64/include/asm/unistd.h:50,
from /home/endo-sw2/linux-for-tegra-x1/sources/kernel/arch/arm64/kernel/kuser32.S:32:
/home/endo-sw2/linux-for-tegra-x1/sources/kernel/include/uapi/asm-generic/unistd.h:32:0: warning: “__NR_io_setup” redefined [enabled by default]
#define __NR_io_setup 0
^
In file included from /home/endo-sw2/linux-for-tegra-x1/sources/kernel/arch/arm64/kernel/kuser32.S:31:0:
/home/endo-sw2/linux-for-tegra-x1/sources/kernel/arch/arm64/include/asm/unistd32.h:510:0: note: this is the location of the previous definition
#define __NR_io_setup 243
^

Then, later, I get a compiler error when trying to compile vdso32/vgettimeofday.c:

/home/endo-sw2/linux-for-tegra-x1/tools/toolchain-k1/bin/arm-linux-gnueabihf-gcc -Wp,-MD,arch/arm64/kernel/vdso32/.vgettimeofday.o.d -nostdinc -isystem /usr/lib/gcc-cross/aarch64-linux-gnu/4.8/include -I/home/endo-sw2/linux-for-tegra-x1/sources/kernel/arch/arm64/include -Iarch/arm64/include/generated -I/home/endo-sw2/linux-for-tegra-x1/sources/kernel/include -Iinclude -I/home/endo-sw2/linux-for-tegra-x1/sources/kernel/arch/arm64/include/uapi -Iarch/arm64/include/generated/uapi -I/home/endo-sw2/linux-for-tegra-x1/sources/kernel/include/uapi -Iinclude/generated/uapi -include /home/endo-sw2/linux-for-tegra-x1/sources/kernel/include/linux/kconfig.h -I/home/endo-sw2/linux-for-tegra-x1/sources/kernel/arch/arm64/kernel/vdso32 -Iarch/arm64/kernel/vdso32 -D__KERNEL__ -mlittle-endian -I/home/endo-sw2/linux-for-tegra-x1/sources/kernel/arch/arm/mach-tegra/include -Iarch/arm/mach-tegra/include -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -fno-delete-null-pointer-checks -O1 -fconserve-stack -fno-pic -fno-reorder-blocks -fno-ipa-cp-clone -fno-partial-inlining -Wframe-larger-than=2048 -fno-stack-protector -Wno-unused-but-set-variable -fno-omit-frame-pointer -fno-optimize-sibling-calls -fno-var-tracking-assignments -g -fno-inline-functions-called-once -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -Werror -shared -fPIC -fno-common -fno-builtin -march=armv7-a -nostdlib -Wl,-soname=linux-vdso32.so.1 -Wl,–hash-style=sysv -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(vgettimeofday)" -D"KBUILD_MODNAME=KBUILD_STR(vgettimeofday)" -c -o arch/arm64/kernel/vdso32/vgettimeofday.o /home/endo-sw2/linux-for-tegra-x1/sources/kernel/arch/arm64/kernel/vdso32/vgettimeofday.c
/home/endo-sw2/linux-for-tegra-x1/sources/kernel/arch/arm64/kernel/vdso32/vgettimeofday.c: In function ‘clock_gettime_fallback’:
/home/endo-sw2/linux-for-tegra-x1/sou
rces/kernel/arch/arm64/kernel/vdso32/vgettimeofday.c:56:1: error: r7 cannot be used in asm here
}
^
make[3]: *** [arch/arm64/kernel/vdso32/vgettimeofday.o] Error 1
make[2]: *** [arch/arm64/kernel/vdso32] Error 2
make[1]: *** [arch/arm64/kernel] Error 2
make: *** [sub-make] Error 2

I’m a newbie. Never compiled a kernel before so I’m stuck on what to do next. Any help is appreciated.

I think you will find 32-bit hosts have issues with both compile and flash software. It’s quite possible the other issues are a mix of legitimate and ignorable errors/warnings, but I don’t think you’ll be able to tell until your build host is 64-bit. In theory x86_64 Ubuntu 14.04 tools work out of the box, but i686 is not “known” as working.

linuxdev,

I really appreciate your reply. It seems that every time I ask a question I’m referred to documentation that I’ve already been through a number of times. These are sparse on detail and assume that the reader has been through the flow with the Ubuntu tools enough times to “read between the lines” and spot errors. You say to “start with a known good .config” … how does one then modify the .config to create a different kernel? Do I make a menuconfig ( using the cross-compiler is counter-intuitive ) just as if it were on the target and proceed from there?

It sure would be nice if the next release of the tegra linux were 64-bit…

There are two possible places to start with a known good config. The first place is to look at your running system. There is a feature in kernel config which causes the kernel to export its configuration to “/proc/config.gz” (and this feature is enabled in Jetson L4T distributions). That file is a compressed exact copy of the running kernel config, other than the CONFIG_LOCALVERSION being blank. Boot the Jetson, copy /boot/config.gz to your desktop host, and gunzip it to decompress.

Or, there is a make target within the kernel source tree which contains a known starting config for each architecture. This is not as good as the running /proc/config.gz unless you happen to know that the running kernel used this to start with. In the case of JTX1, this is actually guaranteed. On a cross-compile, this may be easier to use; in the case of not having a running system, this is probably the only starting point. Info follows.

The kernel source subdirectory “arch/arm64/configs/” contains default configuration for a number of 64-bit ARM systems. In the case of Jetson TX1, the SoC is the tegra210 (tegra21x series). You will notice this directory has “tegra21_defconfig”. If you go to build your kernel (don’t forget to set TEGRA_KERNEL_OUT), you can use tegra21_defconfig as a target…this will place a “.config” file in your build location (I typically run “make mrproper” first, then “make tegra21_defconfig”, followed by “make menuconfig”).

Just for trivia, the Jetson TK1 is a tegra124 SoC (tegra12x series), and uses “arch/arm/configs/tegra12_defconfig”.

The very next JTX1 L4T release will not be pure 64-bit, but the release after that is expected to be pure 64-bit. Once that occurs, native compile should simplify life.

Advice: Whenever you get a new Jetson system, always save an unmodified read-only copy of the /proc/config.gz to your host for reference. Each time you change a config, save a read-only copy of that config with a descriptive name edit. Preferably any saved config file name should contain the CONFIG_LOCALVERSION extension, e.g., I save my original /proc/config.gz as:

config-3.10.67-g3a5c467.gz

Ok. Perhaps I’m a bit slow; I’m not getting the host vs target kernel configuration using menuconfig thing… but since I already created a new .config file on the target using menuconfig can I just copy the new .config file over to the host before I compile the kernel the first time?

Your advice on saving the config.gz is appreciated… learning from mistakes is good; learning from the mistakes of others is better…

The config should be from the Jetson (target), or tegra21_defconfig (which happens to be the same thing in this case). Host is just the x86_64 desktop system, and you have to specify some of those environment variables listed at the top of this thread so it doesn’t get mixed up and think you’re working on config for the local desktop. “make menuconfig” works no matter where you compile from…it’s just that those variables will silently switch where it thinks you’re working.

The difference between a default (initialized) config and no config is enormous. If you created something under “make menuconfig”, but that config did not start matching your Jetson (an uninitialized config), then you’ll be far better off simply throwing away the uninitialized config and getting a proper basic start config in place prior to making those changes. Many options become available or unavailable based on other options…starting with a known working case is almost mandatory even if you’ve been building kernels for years.

So linuxdev uses Fedora for host development. Has anyone used Mint 17.3 (linux 3.19) with the MATE desktop? It’s a lot more comfortable for me to use…

Hello Again,
I was also wondering if trying to compile the kernel on a host running 32-bit linux was the problem. I updated another desktop to 64-bit Ubuntu 14.04 and tried again but it errored out in the exact same way. Next, I’m going to try installing and using the Linaro tools on this host since there are pre-built binaries available. Any other ideas would be appreciated. Thanks.

I don’t think 32-bit desktops will work…they may work for part of the install, but fail on other parts (I don’t know which parts would work/fail…I would be especially worried about compiling a 64-bit kernel on 32-bit, but who knows…maybe it would work). Definitely desktop CUDA would fail, 32-bit is only supported in CUDA 6.5 and before.

@dstillman, what is the exact failure? Compile stage? Is your 64-bit desktop completely “standard”?

I’ve given up on the 32-bit host compile for now. Yes my 64-bit desktop is standard Ubuntu 14.04 LTS. I used apt-get to pull down the aarch64-linux-gnu and arm-linux-gnueabihf tools.

I am getting a compile time error trying to build the kernel. It appears to be the first time it tries to use the 32-bit toolchain (pointed to by the CROSS32CC environment variable). It is trying to comile vgettimeofday.c in the arch/arm64/kernel/vdso32 folder of the kernel sources. The error is:

…/kernel/arch/arm64/kernel/vdso32/vgettimeofday.c
…/kernel/arch/arm64/kernel/vdso32/vgettimeofday.c: In function ‘clock_gettime_fallback’:
…/kernel/arch/arm64/kernel/vdso32/vgettimeofday.c:56:1: error: r7 cannot be used in asm here

I think it is related to how the unistd32.h (in arch/arm64/include/asm) and unistd.h (in include/asm-generic) files inter-relate. There are some macro definitions in there that seem to enforce a compatibility mode between the 32-bit and 64-bit targets that maybe isn’t working right in my build.

These errors are quite often related to configuration. Was the configuration from tegra12_defconfig or perhaps the running /proc/config.gz? If changes were added to this, what were the changes? This will help for others to verify where the error is.

I’m using tegra21_defconfig as instructed in the documentation.

Update: I just found part of my problem. I was using the gcc-arm-linux-gnueabihf tools to provide the CROSS32CC gcc and apparently that toolset is only for arm v7 and earlier. When I went to the linaro site, I found an armv8l toolset that I could use to satisfy the CROSS32CC tool requirement. This got me past that problem and much further in the kernel build. However, it still didn’t complete the build. It now has this compile-time error:

…/kernel/drivers/platform/tegra/tegra21_clocks.c: In function ‘tegra21_cpu_clk_init’:
…/kernel/drivers/platform/tegra/tegra21_clocks.c:1065:31: error: logical not is only applied to the left hand side of comparison [-Werror=logical-not-parentheses]
c->state = (!is_lp_cluster() == (c->u.cpu.mode == MODE_G)) ? ON : OFF;

Not really sure why the compiler thinks this is error-worthy but it’s easily fixable… but, really? Should I fix this? Aren’t these kernel sources actually supposed to be buildable as-is? I just downloaded the sources directly from the nvidia developer site: https://developer.nvidia.com/embedded/linux-tegra - Kernel sources under “Source Packages”

It is actually intentional to use two seemingly incompatible tool chains. The architecture of 32-bit is ARMv7, while the 64-bit architecture is ARMv8a. Within these two tool chains, however, they need the same base version. So for example, a Linaro 32-bit version 4.9 will not mix with a Linaro 64-bit version 5.1. For my own purposes, I’m using Linaro 5.2-2015-11; one of them is aarch64-linux-gnu (ARMv8a with Linux calling convention), the other is arm-linux-gnueabihf (ARMv7 with Linux hard float calling convention)…both are compatible to use together with matching version 5.2-2015-11.

FYI, every change in options/kernel-config changes which code is compiled…changes imply previously untouched code may suddenly end up being checked and spitting out warnings. It is quite likely that warnings and/or errors change if either a different compiler is used (a lot of change is going into Linaro 5.x tool chains to support ARMv8a every single time Linaro releases) or if the kernel configuration changes anything at all.

One starting place which can cause configuration differences without realizing it is if some old configuration is left over. I always use “make mrproper” (or in this case, “make O=build_location… mrproper”), followed by any new configuration. The make mrproper deletes all configuration and starts clean (this is not a problem if you want tegra21_defconfig or if you saved a copy of a custom .config).

What’s the difference between the armv8l-linux-gnueabihf and aarch64-linux-gnu toolchain then? I thought the armv8l was for 32-bit armv8l (as described on the linaro website)? The X1 is ARMv8 and the arm-linux-gnueabihf is described as being only for ARMv7 and earlier. So how is that working for you? You’re developing on X1 right?

I am definitely using a matching 32-bit / 64-bit toolchain (both from the same linaro release). But I chose the linaro 5.1-2015.08 release. I can try moving up to 5.2-2015 but I still think I need to use the armv8l version.

As for building the kernel, I do this after every -failed- attempt:

make O=build_location mrproper
rm all_build_location_files

start a new shell
set my environment variables again

make O=build_location tegra21_defconfig
make V=1 O=build_location zImage

Thanks for your help. Hoping for some success soon…

ARMv8a is 64-bit, ARMv8 is 32-bit. ARMv7 is also 32-bit. Note the “a” of ARMv8a. I do not know how compatibility works between 32-bit ARMv8 and 32-bit ARMv7 (it is possible an ARMv7 compiled program may run in a user space designed for 32-bit ARMv8). Linaro seems to be losing the distinction of 32-bit ARMv7 and 32-bit ARMv8 in their version 5.x naming. However, it remains constant that a JTX1 kernel compile under R23.1 requires what amounts to two different compilers.

Some of the notation you see might point out that ARM CPUs are not manufactured by ARM, but licensed for others to make. I’m not positive, but in some places you may see notation related to little-endian “l” versus big-endian (I’ve never seen a big-endian ARM CPU). I’ve never been able to confirm if this is truly what the “l” notation is, although I suspect it is for little-endian.

As far as other notations go, it just depends where you look as to what it means. For these purposes, “aarch64”, “arm64” (kernel source “arch/arm64”), and “ARMv8a” are synonymous. “ARMv7” and “arm” (kernel source “arch/arm/”) are also synonymous (in some cases 32-bit ARMv8 could possibly also be synonymous…I can’t point out where this is true or not).

Within tool chains one has to consider the environment as well. Calling convention is either bare metal (“none”), or Linux. Within ARMv7, tools have supported various options, the one which crops up a lot is “eabihf”, which is the “E” ABI with use of the “hard float” (hardware floating point). Whether the “a” of “ARMv8a” is shown or not may just depend where you look…explicitly stating 64-bit would imply the “a”. 32-bit ARMv8 also seems have the same gnueabihf option as ARMv7.

Beware that 64-bit support in embedded tool chains is relatively new. So far as I know, Linaro does the best at this. Linaro started their backport of mainline release 5.x gcc into their own 5.x series not too long ago, and 5.x is required for 64-bit support. Every single release of Linaro within the 5.x series has significant and major improvements related to ARMv8a. The improvements are rapid and is probably one of the few places it is worth keeping up with the constant releases for people working with ARMv8a. Release GCC 5.2-2015.11-2 was announced Jan. 29, 2016, and is shown here:
http://releases.linaro.org/components/toolchain/binaries/5.2-2015.11-2/
http://releases.linaro.org/components/toolchain/gcc-linaro/5.2-2015.11-2/

I’ve successfully test compiled JTX1 kernel via 5.2-2015.11 (no “-2”). I suspect there are major improvements just between the 2015 November release and the more recent patch to this release. I suspect that if you are using a Linaro of the 5.1 series that you would benefit by going to the 5.2 series (these are available as precompiled binaries, although unlikely via apt-get). This may even solve some errors or warnings.

Ok, thanks. I will try the 5.2-2015.11 since you’ve had success with that. If that doesn’t work I’ll try the latest -2 releas.