Cross-compiling user-space apps using GCC-11/G++11

[therealmatiss]

Hello! I’ve been developing TX2 apps via chroot emulation for some time now but I wanted to try cross-compilation. My main thing is that I want to cross-compile user-space applications (with Argus, Qt, etc) and I am using C++20 standard.

Currently I’m compiling with GCC-11 & G+±11 on Jetson itself, so as there’s no official toolchain of GCC-11, G+±11 from Nvidia, I decided to try just installing on Ubuntu 22.04: sudo apt install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu which both contain 11.4 versions (g++-aarch64-linux-gnu --version).

Then I created a minimal .cpp project with CMake, which uses Argus. I do have entire Jetson sysroot (which I’m using to compile natively and via chroot, so all libraries are there).

I also created a toolchain.cmake file:

# Set the system name
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR aarch64)

# Specify the cross compiler
set(CMAKE_C_COMPILER aarch64-linux-gnu-gcc)
set(CMAKE_CXX_COMPILER aarch64-linux-gnu-g++)
set(CMAKE_LINKER aarch64-linux-gnu-ld)

# Specify the sysroot (change the path if necessary)
set(CMAKE_SYSROOT /home/user/rootfs)

# Ensure that the cross-compiler can find the correct headers and libraries
set(CMAKE_FIND_ROOT_PATH /home/user/rootfs)

# Search for programs in the build host directories
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)

# For libraries and headers in the target directories
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

After that I ran Cmake like this and it was successful! It found Argus, correct aarch64 compilers and everything.
cmake -DCMAKE_TOOLCHAIN_FILE=../aarch64-toolchain.cmake ..

When I run make, it builds fine but it fails on linking. I noticed that the ldd --version is different between Ubuntu 22.04 and Jetson rootfs but I’m kinda stuck on this issue. It looks like a problem with glibc. Could anyone help me or just tell me if it’s not worth it to do this?

This is the make output:

[ 33%] Building CXX object CMakeFiles/argus_oneshot.dir/ArgusHelpers.cpp.o
[ 66%] Building CXX object CMakeFiles/argus_oneshot.dir/main.cpp.o
[100%] Linking CXX executable argus_oneshot
/usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/user/rootfs/lib/aarch64-linux-gnu/libdl.so.2: undefined reference to `_dl_vsym@GLIBC_PRIVATE'
/usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/user/rootfs/lib/aarch64-linux-gnu/libpthread.so.0: undefined reference to `__libc_dlclose@GLIBC_PRIVATE'
/usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/user/rootfs/lib/aarch64-linux-gnu/libpthread.so.0: undefined reference to `__libc_current_sigrtmax_private@GLIBC_PRIVATE'
/usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/user/rootfs/lib/aarch64-linux-gnu/libpthread.so.0: undefined reference to `_dl_make_stack_executable@GLIBC_PRIVATE'
/usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/user/rootfs/lib/aarch64-linux-gnu/libpthread.so.0: undefined reference to `__libc_vfork@GLIBC_PRIVATE'
/usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/user/rootfs/lib/aarch64-linux-gnu/libpthread.so.0: undefined reference to `__libc_thread_freeres@GLIBC_PRIVATE'
/usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/user/rootfs/lib/aarch64-linux-gnu/libpthread.so.0: undefined reference to `__libc_pthread_init@GLIBC_PRIVATE'
/usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/user/rootfs/lib/aarch64-linux-gnu/libpthread.so.0: undefined reference to `__libc_allocate_rtsig_private@GLIBC_PRIVATE'
/usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/user/rootfs/lib/aarch64-linux-gnu/libdl.so.2: undefined reference to `_dl_addr@GLIBC_PRIVATE'
/usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/user/rootfs/lib/aarch64-linux-gnu/libpthread.so.0: undefined reference to `__libc_dlopen_mode@GLIBC_PRIVATE'
/usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/user/rootfs/lib/aarch64-linux-gnu/libpthread.so.0: undefined reference to `__libc_dlsym@GLIBC_PRIVATE'

Any help appreciated!

[ShaneCCC]

Don’t have idea for it. However we are using ubuntu 18.04 for cross compile.

[therealmatiss]

Yes, I understand that by default you use 18.04 but is it possible to use GCC-11 and G+±11 for cross-compilation? I need newest C++ features for my program.

[linuxdev]

I can’t help, but will add some ideas.

To build user space apps, you basically need three things:

• Compiler (64-bit ARM; aarch64/arm64).
• Linker (64-bit ARM; aarch64/arm64).
• Libraries (64-bit ARM; aarch64/arm64).

When you switch to cross compiling, you need:

• Cross-compiler (runs on x86, outputs 64-bit ARM; aarch64/arm64).
• Cross-linker (runs on x86, outputs 64-bit ARM; aarch64/arm64).
• Libraries (aarch64/arm64). Also known as “sysroot” because it isn’t a cross library.

I think that if the user space libraries on the Jetson are used, then it won’t matter about which C++ standard you are using. For example, standard library headers and any C++ libraries you link to can be unmodified. The headers have to match the libraries, but if the libraries are already built, then you won’t need anything new. The libraries won’t care what they load into as long as the ABI standard itself doesn’t change, and I am guessing this doesn’t change. This means that on the Jetson you could install any ordinary dev package for headers along with any library you might need to use or link against. A raw clone of this can be loopback mounted and used for that purpose.

The cross-compiler and cross-linker will normally already exist, although someone might provide it as source code. Don’t know. Your desktop PC should be able to install this on a new enough release of Ubuntu, but I have no idea how new “new enough” is. If that is not the case, then you get the cross-compiler as source and build it on that host PC Ubuntu version.

You should then be able to build a static version of the cross-linker which does not itself link to any libraries.

Together you now have cross-tools, and a loopback mountable clone for sysroot. In the past Linaro has always produced these products for cross-tools (I don’t know who actually produces GCC 11 cross-tools…where did you get your GCC 11 compiler?). They normally provide a very very minimal sysroot, and it is a bit of a nightmare to try to build everything you need yourself. However, if you already have a Jetson with those libraries you intend to use, and the dev headers, you don’t need that. Perhaps there is a change in some library which is needed to work with that C++ standard, but most of them won’t really care…they are usually C libraries and linked as such. You can examine any particular library which fails, but if you have a clone, then it means that when you update the actual Jetson to include that library, it is then possible to simply rsync the updates on the Jetson to the loopback mounted clone on the host PC over ssh.

If the clone is on external USB, then it is possible you can clone and switch where the clone is and rsync without networking/ssh.

[therealmatiss]

Thanks for the explanation! However, I don’t have a problem cross-compiling in general, I have copied entire sysroot and it works with official GCC-7 linaro toolchain. Newer GCC/G++ versions are the problem because Linaro toolchains doesn’t exist with newer GCC/G++ versions.

The GCC-11 compiler with toolchain for aarch64 was on some APT repository I found. At the end I managed to cross-compile the application but unfortunately the toolchain with GCC-11 had a different glibcversion, which it seems impossible to fix this issue, maybe using Crosstool-NG building your own toolchain is possible, but I don’t think I want to go so far. GCC-11 compiler exists on Jetson natively, which works fine. Currently I’m using chroot emulation compiling with GCC-11. I was just trying to find even faster way because even with AMD Ryzen 7900X it takes about 5 minutes to compile my project from scratch.

[therealmatiss]

Yes, it shouldn’t matter which Ubuntu you use for cross-compilation. However if you read my reply to @linuxdev, this looks impossible because there is no toolchain for Jetson with GCC-11/G+±11 aarch64 compilers. The toolchain I found just has different glibc version so it’s a no go.

This is unfortunate because it means cross-compilation isn’t possible on Ubuntu 18.04 Jetsons with C++20 features - the best way is chroot emulation I guess, if you want any faster compilation than native (if you have powerful PC)