cross compiler version problem

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Hi,

I am doing cross compile my code with the library OpenCV in my Ubuntu16.4 for Jetson TX2.
However, I encounter the error when I compile my code which call Opencv.
Here is the error log:

/home/jetson/my_jetson_doc/gcc_bin/install/bin/aarch64-unknown-linux-gnu-g++ -std=c++11 -std=c++11 -O3 -DNDEBUG -L /home/my_project/engined_cross_compile-half-way -Xlinker -Map=engined.map -L/usr/local/cuda/targets/aarch64-linux/lib -L/home/my_project/engined_cross_compile-half-way/lib/aarch64 -L -nostartfiles -lstdc++ CMakeFiles/engined.dir/main.c.o CMakeFiles/engined.dir/mqtt.c.o CMakeFiles/engined.dir/hopseq.c.o CMakeFiles/engined.dir/detect_all_freq.c.o CMakeFiles/engined.dir/capture_manager.c.o CMakeFiles/engined.dir/lic.c.o CMakeFiles/engined.dir/cfg.c.o CMakeFiles/engined.dir/input_manager.c.o CMakeFiles/engined.dir/usd_segmentation.c.o CMakeFiles/engined.dir/usd_tx.c.o CMakeFiles/engined.dir/parse_capture_msg.c.o CMakeFiles/engined.dir/data_visualization.c.o CMakeFiles/engined.dir/cuda_interface.c.o -o engined -rdynamic 3rd/libevent/lib/libevent.a 3rd/tinycthread/libtinycthread.a 3rd/queue/libmpmcqueue.a -lpthread engine/libengine.a -lm utils/libutils.a 3rd/mqtt/src/libpaho-mqtt3cs-static.a -lssl -lcrypto -lfftw3f -lfftw3f_threads 3rd/libyaml/libyaml_static.a -lopencv_calib3d -lopencv_core -lopencv_dnn -lopencv_features2d -lopencv_flann -lopencv_imgproc -lopencv_ml -lopencv_objdetect -lopencv_photo -lstdc++ -lcudart -lcufft /usr/local/cuda/targets/aarch64-linux/lib/libcufft.so …/dronecuda/build/libdronecufft.a /usr/local/lib/libopencv_calib3d.so -lpthread -lc -ldl -lpthread -Wl,-rpath,/usr/local/cuda/targets/aarch64-linux/lib:/usr/local/lib
/home/my_project/engined_cross_compile-half-way/lib/aarch64/libopencv_dnn.so：for ‘std::__cxx11::basic_string<char, std::char_traits, std::allocator >::compare(unsigned long, unsigned long, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) const@GLIBCXX_3.4.21’undefined reference.

/home/my_project/engined_cross_compile-half-way/lib/aarch64/libopencv_core.so：for ‘VTT for std::__cxx11::basic_stringstream<char, std::char_traits, std::allocator >@GLIBCXX_3.4.21’undefined reference.

/home/my_project/engined_cross_compile-half-way/lib/aarch64/libopencv_calib3d.so：for ‘std::__throw_out_of_range_fmt(char const*, …)@GLIBCXX_3.4.20’undefined reference.

/home/my_project/engined_cross_compile-half-way/lib/aarch64/libopencv_dnn.so：for ‘std::__cxx11::basic_string<char, std::char_traits, std::allocator >::find_first_not_of(char const*, unsigned long, unsigned long) const@GLIBCXX_3.4.21’undefined reference.

/home/my_project/engined_cross_compile-half-way/lib/aarch64/libopencv_dnn.so：for ‘std::__cxx11::basic_string<char, std::char_traits, std::allocator >::assign(char const*)@GLIBCXX_3.4.21’undefined reference.

/home/my_project/engined_cross_compile-half-way/lib/aarch64/libopencv_core.so：for ‘std::__cxx11::basic_ostringstream<char, std::char_traits, std::allocator >::basic_ostringstream(std::_Ios_Openmode)@GLIBCXX_3.4.21’undefined reference.

/home/my_project/engined_cross_compile-half-way/lib/aarch64/libopencv_dnn.so：for ‘std::__cxx11::basic_string<char, std::char_traits, std::allocator >::~basic_string()@GLIBCXX_3.4.21’undefined reference.

I googled it. The issue is caused by the different version of “LIBCXX” in my Ubuntu and Jetson TX2.

In my Ubuntu I use the aarch64 cross compile tool chain from Jetson TX2 webside.This tool chain is old.
The version of “LIBCXX” is “GLIBCXX_3.4.19”. I used “strings /usr/lib/aarch64-linux-gnu/libstdc++.so.6 |grep LIBCXX” to get the version number.

In Jetson TX2 board the version of “GLIBCXX_3.4.21”.

The root cause of the issue is the incompatible of different version of compiler.
Please help check if there is new cross compile tool chain which use “GLIBCXX_3.4.21”.
If there is not such toolchain, what can I do? Downgrade the gcc version in Jetson TX2?
It is really urgent issue.Thank you.

Best Regards,
Steven

2

Hi Steven,
We don’t support cross compile. Can you compile it on target TX2 board?

3

You could of course add what is needed to build natively on the TX2, clone the TX2, loopback mount the clone on your host, and then use that content for sysroot. Perhaps native compile would be easier.

4

My code is huge. If I compile on Jetson TX2, it will take a lot of time.That time is beyond tolerance.
The compile time will be 5 minutes which will kill my current project.

5

@linuxdev Could you please explain how to use clone to compile in the host？

6

Where files are placed depends on the particular Linux distribution (I’m using Fedora, not Ubuntu…YMMV), but basically you’ll note there is usually a library directory naming scheme something like this on a desktop PC (this is just an example and not an actual case…this is more about the methodology):

/usr/lib
/usr/lib64

The “lib64” or “lib” would tend to be the native architecture of the PC. Within this foreign architectures tend to get subdirectories with names like “aarch64-linux-gnu/”. A PC might have a 32-bit foreign architecture specifically labeled something like “i386” or “i686”. If you were to look at the files installed for the aarch64 cross tools, then some of this is the executable environment, including cross linkers which run on the native PC architecture, but which requires files of the aarch64 architecture as an argument instead of the native architecture.

Most of the time what you will be missing are the libraries which the linker wants to link against. In your case it seems you need libc in the aarch64-linux-gnu format. I’ll stick to libraries since this is what most people find missing, and this applies directly to libc. The tools are there, but the libraries and related linkage targets are missing.

Let’s say that your build, if it had occurred natively on the Jetson, linked against “libc.so.6”. It might be that when cross compiling, instead of finding libc.so.6 in “/lib”, or “/lib64”, it would instead insert this architecture into the search path and would become something like:

/usr/lib/aarch64-linux-gnu/libc.so.6

In fact, if you were to install the “sysroot” files of this architecture, then probably all of the libraries would be in “/usr/lib/aarch64-linux-gnu/”. libc, libm, libstdc++…all libraries of this architecture, being foreign, would be segregated like this into the architecture dependent directory name. Your cross tools would be configured to search for aarch64 link targets there.

A standard sysroot is only the very minimal content needed to build the other content…which builds yet more content…and eventually you get all of the files you need to link against on your PC even if the aarch64 foreign architecture version was not available via the standard package manager. Perhaps you had the libc present already, but it was the wrong version, or perhaps your tools didn’t know the link request was in the context of “aarch64-linux-gnu” (probably not a problem since you are using the standard package system to install the cross tools).

You could manually copy each file you find missing one at a time to the “aarch64-linux-gnu” directory if you know where it is and the tools are configured to look there. For a single directory and developing for only a single product this might be reasonable. Until there is a version upgrade and you must work on different versions depending on the software.

You could instead put all the runtime files on the Jetson via the normal package tool, and add the dev packages as well (such as for headers), clone it, and then substitute a symbolic link for “/usr/lib/aarch64-linux-gnu/” to point at the correct lib directory on the loopback mounted clone.

So if your Jetson has a clone, and you mount it like this:

sudo -s
mkdir /usr/local/loopback_tx2
mount -o loop,ro ./your_clone.img.raw /usr/local/loopback_tx2
cd /usr/lib
# You probably don't have this directory, but just in case:
mv aarch64-linux-gnu original_aarch64-linux-gnu
ln -s /usr/local/loopback_tx2/usr/lib/aarch64-linux-gnu .
exit

Your entire library setup from your Jetson is now available for your cross tools. This library setup has exactly the same library versions as your Jetson down to the last bit. If you were to rsync your clone to a Jetson which had recently been updated, then your PC is suddenly upgraded to again be an exact match of the Jetson.

If you have a different release, say both R28.1 and R28.2.1, then the clone you choose to mount becomes your environment. Have multiple coworkers doing simultaneous development and want to give them what it takes to match your build environment? Just put the symbolic links in, and pass around a thumb drive with the clone.

As a summary, the basic components of a cross compile environment includes the compiler. This is basically all you need for a kernel. It’s bare metal. Then for user space you add a cross linker. This linker is exactly the same as on the foreign system, except that it executes on a PC using foreign architecture libraries. The hard part is that you now need all of the libraries and headers from this foreign system…if the link requirements are not very basic, then you will not find packages prebuilt for your PC host. Clones make a nice substitute because they really are from the package manager…just not the PC’s package manager.

I also happen to be an advocate for cloning…it’s amazing how much better that is as a backup versus ordinary backup tools. Mounting the clone with the “ro” argument makes it read-only for safety.

I almost forgot…there are also tricks you can use with the UUID of a thumb drive or SD card so that it auto mounts read-only to the location you reserved for aarch64.

7

@linuxdev Thank you very much for your detailed information. Acyually, I think I use same way with your description to do cross compile. I copy many libs from my Jetson TX2 board to my host machine which x86 ubuntu 16.04. With these libs from Jetson board compiler can find all the libraries(include cuda, opencv,etc). The problem happened while linking all these libraries from Jetson boards.

In my Ubuntu I use the aarch64 cross compile tool chain from Jetson TX2 webside.This tool chain is old.
The version of “LIBCXX” is “GLIBCXX_3.4.19”. I used “strings /usr/lib/aarch64-linux-gnu/libstdc++.so.6 |grep LIBCXX” to get the version number.In Jetson TX2 board the version of “GLIBCXX_3.4.21”.
The root cause of the issue is the incompatible of different version of compiler. I compile opencv libs with new g++ version in Jetson board. The host machine has the older g++ version.
To resolve this issue,I cope the libstdc++.so.6 from Jetson board to host machine. However,the cross compiler tool chain which is downloaded from Nvidia website cannot recognize this libstdc++.so.6.

So I have two questions about this issue.
1.Do you think I can use the compiler which is cloned from Jetson TX2 board to do cross compile? If not, the g++ linker in cross compile tool chain from Nvidia website is older than ubuntu 16.04. So these opencv libs compiled in Jetson board with new g++ compiler have conflicts with the old cross compile tool chain. I don’t know how to resolve this issue.
2.Why the libstdc++.so.6 which is cloned from Jetson board cannot be recognized by cross compiler? So many libs from Jetson board can be recognized.I don’t know why only this lib have such issue.

8

What compiler version is it? If the compiler is “something-some-version-gcc” (I don’t have your install to know how it names the arm64 gcc compiler), then what do you see from “something-some-version-gcc --version”?

If you do the same thing from the particular TX2 system using its native compiler, “gcc --version”, and compare the two, perhaps we can decide if there is an ABI difference in those two flavors.

If it turns out that the libraries being linked against already match the Jetson’s, but the linker and compiler versions on the host’s cross tools are incompatible, then this indicates changing sysroot won’t help…the tools themselves would need to change. For that case you might end up installing Linaro tools directly from their downloads instead of via the Ubuntu package manager.

The compiler on the Jetson is arm64 and won’t run on x86_64. Linaro will have what you need if what you already have doesn’t work…but you need to know what versions you are dealing with.

There is software involved from the 4.x gcc series, from the 5.x gcc series, from the 6.x gcc series, and the modern support for recent desktop PCs is 7.x gcc (and on Xavier one needs to install a compatibility gcc-6 compiler even for native builds of the kernel if trying to match the running system’s kernel). Lots of history, but much of the older Jetson software was from the 4.x gcc series. One of the problems of LTS (“Long Term Support”) Ubuntu systems is that if your host is not the same LTS version, then your developer system moves in a different time frame for upgrades and the two systems will quickly become incompatible…then you are left adding tools manually.