Tensorflow c library on Drive-PX2

Hello,
we successfully build the libtensorflow_cc.so on the Drive-PX2 and on the host system (Ubuntu 16.04, Driveinstall 5.0.5.0 with Driveworks-0.6, tensorflow version 1.4.1).
But we were not able to cross-compile anything that used this libraries on the host for the Drive-PX2.

This are the steps we have done:

  • Compiling libtensorflow_cc.so (and libtensorflow_framwork.so) on the drive-PX2 with the following command (Successfull, also tested with the ‘hello_tf’ example from the tensorflow-webpage):
bazel build -c opt --local_resources 8192,2.0,1.0 --verbose_failures --config=cuda --incompatible_load_argument_is_label=false //tensorflow:libtensorflow_cc.so
  • Copy compiled files libtensorflow_cc.so and libtensorflow_framwork.so to the host-PC to the folder <VIBRANTE_PDK>/targetfs/lib/aarch64-linux-gnu
  • Copy c_api.h file from <tensorflow_source>/tensorflow/c/c_api.h to <VIBRANTE_PDK>/include/tensorflow/c/c_api.h

Then our cross-compile test with the ‘hello_world’-sample from the driveworks/samples folder:
-modified main.c: Add

printf("Hello from TensorFlow C library version %s\n", TF_Version());

in main()
and

#include <tensorflow/c/c_api.h>

to the include-section

  • added
target_link_libraries(${PROJECT_NAME} PRIVATE ${LIBRARIES} tensorflow_cc tensorflow_framework)

to the CMakeLists.txt in the ‘hello_world’ - Folder.

This is the output of the make-command:

Scanning dependencies of target sample_hello_world
[ 70%] Building CXX object src/hello_world/CMakeFiles/sample_hello_world.dir/main.cpp.o
[ 70%] Linking CXX executable sample_hello_world
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::assign(char const*)@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_replace_aux(unsigned long, unsigned long, unsigned long, char)@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_framework.so: undefined reference to `std::basic_istream<char, std::char_traits<char> >& std::operator>><char, std::char_traits<char>, std::allocator<char> >(std::basic_istream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_stringstream()@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_framework.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::rfind(char, unsigned long) const@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::reserve(unsigned long)@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `vtable for std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::find(char, unsigned long) const@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_framework.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::substr(unsigned long, unsigned long) const@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_replace(unsigned long, unsigned long, char const*, unsigned long)@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::_M_sync(char*, unsigned long, unsigned long)@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_framework.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(unsigned long, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `VTT for std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_framework.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::append(char const*)@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::find(char const*, unsigned long, unsigned long) const@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_framework.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string()@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_erase(unsigned long, unsigned long)@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_append(char const*, unsigned long)@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `typeinfo for std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(unsigned long, unsigned long, char const*) const@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `VTT for std::__cxx11::basic_istringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::resize(unsigned long, char)@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_framework.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::find_last_of(char const*, unsigned long, unsigned long) const@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate(unsigned long, unsigned long, char const*, unsigned long)@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `vtable for std::__cxx11::basic_istringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_framework.so: undefined reference to `std::thread::_M_start_thread(std::shared_ptr<std::thread::_Impl_base>, void (*)())@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_framework.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::find_first_of(char const*, unsigned long, unsigned long) const@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_ostringstream()@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_framework.so: undefined reference to `std::basic_istream<char, std::char_traits<char> >& std::getline<char, std::char_traits<char>, std::allocator<char> >(std::basic_istream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char)@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::swap(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_framework.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::find_first_not_of(char const*, unsigned long, unsigned long) const@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `lgammaf@GLIBC_2.23'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(char const*) const@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `vtable for std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_framework.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct(unsigned long, char)@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `std::random_device::_M_init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `VTT for std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned long&, unsigned long)@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `vtable for std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4.21'
/home/avlsfr/NVIDIA/Drive/5050L_SDK/DriveSDK/drive-t186ref-linux/targetfs/lib/aarch64-linux-gnu/libtensorflow_cc.so: undefined reference to `lgamma@GLIBC_2.23'
collect2: error: ld returned 1 exit status
src/hello_world/CMakeFiles/sample_hello_world.dir/build.make:107: recipe for target 'src/hello_world/sample_hello_world' failed
make[2]: *** [src/hello_world/sample_hello_world] Error 1
CMakeFiles/Makefile2:2295: recipe for target 'src/hello_world/CMakeFiles/sample_hello_world.dir/all' failed
make[1]: *** [src/hello_world/CMakeFiles/sample_hello_world.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

Any hint how to solve this problem? Any hint how to compile/link tensorflow for cross-compiling?

We also successfully builded the libtensorflow_cc.so on the host-system for the host system, but using this file gives architecture-errors, as expected.

Dear felix.elrod,

I’m not sure why you try to install TF on DPX2 because DPX2 is a deployment device.
The workflow for exporting TF model to UFF on the host, then running optimization on DrivePX2 using TensorRT.
It is better to use TensorRT on the PX2 with your TF trained models. Thanks.

Hi felix.elrod,

I’ve decided to continue the discussion just to emphasise to you and the NVIDIA dev. community how “professional” are the answers from NVIDIA support so my post here may cause them to change in a future. I hope NVIDIA could just improve the support service because we are blaming them here more frequently.

First off all you are not suppose to share with NVIDIA your application software use case: as disclosure of such information may violate the company technical information sharing policy. So NVIDIA support should not ask such questions from you.

Second thing is that the problem you are mentioning is not just related to the TensorFlow library as you might want to have such hybrid compilation schema where native and cross compile stuff has to be linked together.

We also would like to have TensorFlow library compiled and linked with the other libraries from DriveWorks SDK as well as to link with some third-party libraries that are already compiled directly on the target Drive-PX2 Ubuntu 16.04 OS so we can’t have them recompiled from source to align with the gcc version in the DriveWorks SDK.

So the problem is a fundamental compatibility issue with the GCC version support in the DriveWorks SDK which NVIDIA has to somehow resolve in a future. So DriveWorks SDK should upgrade to the GCC v 5.4.0 to align with the current OS (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.10) they have supplied with the Drive-PX2 platform.

Dear SteveNV,

could you please answer the question why the DriveWorks SDK is been released with the major GCC incompatibility issue which restricts users basically using the hybrid linking schema which felix.elrod has clearly explained in his question?

Something unrelated here:
TensorRT does not have full support of available TenworFlow operations so it is almost impossible to export complex TensorFlow models to tensorRT.

Another thing is: even you successfully port your TensorFlow models through the custom tensorRT layers development process - it does not give optimal performance as TensorRT can’t fuse custom tensorRT CUDA kernels with the rest of the operations so the tensorRT advertised optimisation feature is useless. This is why I guess people want to use TensorFlow library directly on the Drive-PX2 platform.

To summarise: rather than advertising raw product (tensorRT) it might be better to support more sophisticated and mature off-the-shelf products such as TensorFlow or Mx-Net with their NNVM compiler which does the same thing as tensorRT and even more.

Thanks