How can we compile the TF wheels on a VM for speed up?

Robotics & Edge Computing Jetson Systems Jetson AGX Xavier
1

Hi,

Right now the compile times are taking 24 hours approx when compiling TF 2.1.0 using Bazel 0.27.1 on python 3.7

I was able to compile TF1.13 wheel but without TensorRT because there were errors about compiling for TensorRT

I wanted to make a VM (no GPU, its the VM Player) with Ubuntu 18.04 and install the packages like CUDA 10.0, cuDNN 7.6.3, TensorRT 6. I was able to install all of these and started the build for TensorFlow 2.1.0 for the Jetson Xavier.

But near the end i get errors like: crosstool_wrapper_driver_is_not_gcc failed: error executing command

I basically followed these instructions: Building Tensorflow 1.13 on Jetson Xavier

I did the edits to include aarch64 etc

But i dont know if im missing something else. I’m running the build again this time with gcc-7 and will post the output when done as a follow up comment.

What I would like to know is can I make such a VM with everything loaded on it so that I can make the wheels for the Xavier? Im using Jetpack 4.3 and I matched whatever is installed on there on my VM.

Ive been reading about cross compile but there are no specific info about the Xavier itself

This will save me hours as the compile time on the Xavier itself is around 24 hours and eats a huge amount of disk space

Thank you!
Moth

2

Follow up: The compile failed even with gcc-7

Here is the output:

  • ./tensorflow/compiler/xla/service/hlo_computation.h: In instantiation of ‘tensorflow::Status xla::HloComputation::AcceptOrdered(xla::DfsHloVisitorBase, absl::Span<xla::HloInstruction const>) const [with HloInstructionPtr = const xla::HloInstruction*]’:
  • ./tensorflow/compiler/xla/service/hlo_computation.h:572:61: required from here
  • ./tensorflow/compiler/xla/service/hlo_computation.h:546:29: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
  • TF_RET_CHECK(order.size() == instruction_count());
  • ./tensorflow/core/platform/macros.h:87:47: note: in definition of macro ‘TF_PREDICT_FALSE’
  • #define TF_PREDICT_FALSE(x) (__builtin_expect(x, 0))
  •                                            ^
  • ./tensorflow/compiler/xla/service/hlo_computation.h:546:3: note: in expansion of macro ‘TF_RET_CHECK’
  • TF_RET_CHECK(order.size() == instruction_count());
  • ^
  • ./tensorflow/compiler/xla/array.h: In instantiation of ‘tensorflow::int64 xla::Array::dim(tensorflow::int64) const [with T = int; tensorflow::int64 = long long int]’:
  • ./tensorflow/compiler/xla/array2d.h:71:44: required from ‘tensorflow::int64 xla::Array2D::height() const [with T = int; tensorflow::int64 = long long int]’
  • ./tensorflow/compiler/xla/service/computation_placer.h:47:45: required from here
  • ./tensorflow/compiler/xla/array.h:406:13: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
  •  CHECK(n < sizes_.size());
  • ./tensorflow/core/platform/macros.h:87:47: note: in definition of macro ‘TF_PREDICT_FALSE’
  • #define TF_PREDICT_FALSE(x) (__builtin_expect(x, 0))
  •                                            ^
  • ./tensorflow/compiler/xla/array.h:406:5: note: in expansion of macro ‘CHECK’
  •  CHECK(n < sizes_.size());

  •  ^
  • ERROR: /home/xavier/tensorflow/tensorflow/core/kernels/BUILD:1289:1: C++ compilation of rule ‘//tensorflow/core/kernels:tile_ops’ failed (Exit 4)
  • x86_64-linux-gnu-gcc-7: internal compiler error: Killed (program cc1plus)
  • Please submit a full bug report,
  • with preprocessed source if appropriate.
  • See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
  • Target //tensorflow/tools/pip_package:build_pip_package failed to build
  • Use --verbose_failures to see the command lines of failed build steps.
  • INFO: Elapsed time: 1452.433s, Critical Path: 88.90s
  • INFO: 4938 processes: 4938 local.
  • FAILED: Build did NOT complete successfully
  • xavier@ubuntu:~/tensorflow$
4

Hi,

Just want to clarify,
You try to cross-compile a TensorFlow package for Jetson on an x86 system?
If yes, you can just cross-compile it on a general Ubuntu host rather than a VM.

May I know why do you want to run the compiling on a VM?

Thanks.

5

Hi, its because i run a windows workstation and for that reason is why im making a VM that I can then launch whenever i need to build future wheels.

So is there anything specific that I have to do so that it builds the wheel for an aarch64 instead of an x86 wheel? That part is a bit unclear for me

6

Try gcc-5 - this worked for me when compiling TF 1.14 on a Jetson Nano.

7

Yeah i was trying to compile TF2.1 but i gave up as too many errors. Im trying to compile TF1.14 on a separate Ubuntu desktop (not native compile) it failed with gcc-5 so im trying with gcc-4.8 now lets see how it goes.