Install PyTorch with Python 3.8 on Jetpack 4.4.1

[patrick.nagelm8t1h]

I would like to install PyTorch with Python 3.8 on Jetpack 4.4.1. Unfortunately I ran into the following problem while building PyTorch from source:

[ 76%] Building NVCC (Device) object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/torch_cuda_generated_DistributionCauchyKernel.cu.o
/home/xavier/pytorch/c10/cuda/CUDAMathCompat.h: In static member function ‘static scalar_t at::native::copysign_kernel_cuda(at::TensorIterator&)::<lambda()>::<lambda()>::<lambda(scalar_t, scalar_t)>::_FUN(scalar_t, scalar_t)’:
/home/xavier/pytorch/c10/cuda/CUDAMathCompat.h:46:24: internal compiler error: Segmentation fault
   return ::copysignf(x, y);
                        ^
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
CMake Error at torch_cuda_generated_CopysignKernel.cu.o.Release.cmake:281 (message):
  Error generating file
  /home/xavier/pytorch/build/caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/./torch_cuda_generated_CopysignKernel.cu.o


caffe2/CMakeFiles/torch_cuda.dir/build.make:89106: recipe for target 'caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/torch_cuda_generated_CopysignKernel.cu.o' failed
make[2]: *** [caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/torch_cuda_generated_CopysignKernel.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....

Those are the steps I applied:

git clone https://github.com/pytorch/pytorch.git
cd pytorch
git submodule --init --recursive

export USE_NCCL=0
export USE_DISTRIBUTED=0 
export USE_QNNPACK=0
export USE_PYTORCH_QNNPACK=0
export TORCH_CUDA_ARCH_LIST="5.3;6.2;7.2"
export PYTORCH_BUILD_VERSION=1.7.0
export PYTORCH_BUILD_NUMBER=1

python3.8 -m pip install -r requirements.txt
python3.8 setup.py build

I tried it out two different versions of cmake, namely 3.10.2 and 3.18.5. Also I applied the patch that was referred to in this post for the PyTorch installation with Python 3.6 - PyTorch for Jetson - version 1.7.0 now available, but none of those measures helped.

Any advice would be appreciated.

[dusty_nv]

Hi @patrick.nagelm8t1h, I haven’t build PyTorch for Python 3.8 myself, so some folks from the community may have more suggestions - have you tried building a specific version of PyTorch as opposed to PyTorch master? i.e. changing your git clone command to git clone --recursive --branch v1.7.0 http://github.com/pytorch/pytorch

@jonathan.boisclair was recently able to build PyTorch 1.7 for Python 3.8 on 20.04, but on 18.04 I don’t think you would need the additional patch that they did: https://forums.developer.nvidia.com/t/pytorch-for-jetson-version-1-7-0-now-available/72048/583

Did you try mounting additional SWAP memory on your Nano, in case it is running out of memory and causing the compiler crash?

[patrick.nagelm8t1h]

Hey @dusty_nv, thank you for your quick response. With your help I was able to build PyTorch 1.7 for Python 3.8 on Jetpack 4.4.1 (Jetson Xavier). It actually helped to switch to the 1.7 branch of PyTorch. Here are the steps that worked for me:

Easiest way:

git clone --recursive --branch 1.7 http://github.com/pytorch/pytorch
cd pytorch
python3.8 -m pip install -r requirements.txt
python3.8 setup.py install

From where I was coming:

git clone https://github.com/pytorch/pytorch.git
cd pytorch
git fetch
git checkout 1.7
git submodule update --init --recursive
python3.8 -m pip install -r requirements.txt
python3.8 setup.py build
python3.8 setup.py install

[zhongzisha]

An temp solution would be as follows, to modify the file (c10/cuda/CUDAMathCompat.h) with the following function:

    __MATH_FUNCTIONS_DECL__ float copysign(float x, float y) {
         return static_cast<float>(::copysign(static_cast<double>(x), static_cast<double>(y)));
    }

But I’m not sure if this would lead to problems on performance. Waiting for better solution.

[zhongzisha]

I’ve tried to compile the pytorch-1.8 (the master branch on Dec 12, 2020) on JetPack 4.3 (cuda-10.2, cudnn-8) image (ubuntu 18.04) using the gcc-7.5, gcc-8.4, gcc-9.3 and gcc-10.1.

Finally, only the gcc-9.3 can be used to compile the code and generate the final wheel.

The generated wheels can be found in https://icedrive.net/1/76FFGZiyDO.

[xianmei.lei]

I am installing Torch 1.7 with python3.8 to Xavier NX (JetPack 4.4) on Ubuntu 18.04 using @patrick.nagelm8t1h’s advice. I got an error when I ran python3.8 setup.py install after python3.8 setup.py build.

Install the project…
– Install configuration: “Release”
– Set runtime path of “/home/costar/Documents/pytorch/torch/lib/libtorch_cuda.so” to “$ORIGIN:/usr/local/cuda/lib64:/usr/lib/aarch64-linux-gnu/openmpi/lib”
– Set runtime path of “/home/costar/Documents/pytorch/torch/lib/libtorch.so” to “$ORIGIN:/usr/local/cuda/lib64”
– Set runtime path of “/home/costar/Documents/pytorch/torch/test/cuda_atomic_ops_test” to “$ORIGIN:/usr/local/cuda/lib64”
– Set runtime path of “/home/costar/Documents/pytorch/torch/test/cuda_complex_test” to “$ORIGIN:/usr/local/cuda/lib64”
– Set runtime path of “/home/costar/Documents/pytorch/torch/test/cuda_complex_math_test” to “$ORIGIN:/usr/local/cuda/lib64”
– Set runtime path of “/home/costar/Documents/pytorch/torch/test/cuda_integer_divider_test” to “$ORIGIN:/usr/local/cuda/lib64”
– Set runtime path of “/home/costar/Documents/pytorch/torch/test/cuda_half_test” to “$ORIGIN:/usr/local/cuda/lib64”
– Set runtime path of “/home/costar/Documents/pytorch/torch/test/cuda_distributions_test” to “$ORIGIN:/usr/local/cuda/lib64”
– Set runtime path of “/home/costar/Documents/pytorch/torch/test/cuda_optional_test” to “$ORIGIN:/usr/local/cuda/lib64”
– Set runtime path of “/home/costar/Documents/pytorch/torch/test/cuda_packedtensoraccessor_test” to “$ORIGIN:/usr/local/cuda/lib64”
– Set runtime path of “/home/costar/Documents/pytorch/torch/test/cuda_vectorized_test” to “$ORIGIN:/usr/local/cuda/lib64”
– Set runtime path of “/home/costar/Documents/pytorch/torch/test/cuda_generator_test” to “$ORIGIN:/usr/local/cuda/lib64”
– Set runtime path of “/home/costar/Documents/pytorch/torch/lib/libtorchbind_test.so” to “$ORIGIN:/usr/local/cuda/lib64”
– Set runtime path of “/home/costar/Documents/pytorch/torch/lib/libjitbackend_test.so” to “$ORIGIN:/usr/local/cuda/lib64”
////
CMake Error at test_jit/cmake_install.cmake:90 (file):
file RPATH_CHANGE could not write new RPATH:
$ORIGIN:/usr/local/cuda/lib64:/usr/local/cuda/lib64/stubs
to the file:
/home/costar/Documents/pytorch/torch/bin/test_jit
////
No valid ELF RPATH or RUNPATH entry exists in the file; Failed to load
section headers.
Call Stack (most recent call first):
caffe2/cmake_install.cmake:3191 (include)
cmake_install.cmake:111 (include)
Makefile:61: recipe for target ‘install’ failed
make: *** [install] Error 1
Traceback (most recent call last):
File “setup.py”, line 727, in
build_deps()
File “setup.py”, line 314, in build_deps
build_caffe2(version=version,
File “/home/costar/Documents/pytorch/tools/build_pytorch_libs.py”, line 62, in build_caffe2
cmake.build(my_env)
File “/home/costar/Documents/pytorch/tools/setup_helpers/cmake.py”, line 346, in build
self.run(build_args, my_env)
File “/home/costar/Documents/pytorch/tools/setup_helpers/cmake.py”, line 141, in run
check_call(command, cwd=self.build_dir, env=env)
File “/usr/lib/python3.8/subprocess.py”, line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command ‘[‘cmake’, ‘–build’, ‘.’, ‘–target’, ‘install’, ‘–config’, ‘Release’, ‘–’, ‘-j’, ‘2’]’ returned non-zero exit status 2.

Any advice?

[zhongzisha]

Hi, @xianmei.lei, I didn’t meet the problem you met. When I was compiling the pytorch from the source, I set export BUILD_TEST=0 before the compilation, so the test-related compilation might be skipped.

Did you try the method in PyTorch for Jetson - version 1.7.0 now available?

[xianmei.lei]

Thanks @zhongzisha. I tried that. Pytorch 1.7 was installed but I needed python3.8 to run my application.

[xianmei.lei]

I retried with a fresh reinstallation of Jetpack 4.4.1 for python3.8 and torch1.7 using @patrick.nagelm8t1h’s solution again. Here is the error this time:

Run Build Command:"/usr/bin/make" “cmTC_7d06d/fast”
/usr/bin/make -f CMakeFiles/cmTC_7d06d.dir/build.make CMakeFiles/cmTC_7d06d.dir/build
make[1]: Entering directory ‘/home/costar/pytorch1.7/build/CMakeFiles/CMakeTmp’
Building CXX object CMakeFiles/cmTC_7d06d.dir/src.cxx.o
/usr/bin/c++ -DCAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING -std=c++14 -o CMakeFiles/cmTC_7d06d.dir/src.cxx.o -c /home/costar/pytorch1.7/build/CMakeFiles/CMakeTmp/src.cxx
/home/costar/pytorch1.7/build/CMakeFiles/CMakeTmp/src.cxx:1:10: fatal error: glog/stl_logging.h: No such file or directory
#include <glog/stl_logging.h>
^~~~~~~~~~~~~~~~~~~~
compilation terminated.

Some posts said it is a bug for installing Torch from source. Some said cuDNN needs to be above version 6 so jetpack 4.4 comes with cuDNN v8, right?

apt-get install libgoogle-glog-dev libgflags-dev doesn’t help. Please advice!

[SamuelWei]

I install following your suggestion, and got the error below.
any advise would be appreciated

– Private Dependencies : pthreadpool;cpuinfo;nnpack;XNNPACK;/usr/lib/aarch64-linux-gnu/libnuma.so;fp16;aten_op_header_gen;foxi_loader;rt;fmt::fmt-header-only;gcc_s;gcc;dl
– Configuring incomplete, errors occurred!
See also “/home/workspace/pytorch/build/CMakeFiles/CMakeOutput.log”.
See also “/home/workspace/pytorch/build/CMakeFiles/CMakeError.log”.
Traceback (most recent call last):
File “setup.py”, line 727, in
build_deps()
File “setup.py”, line 314, in build_deps
build_caffe2(version=version,
File “/home/workspace/pytorch/tools/build_pytorch_libs.py”, line 54, in build_caffe2
cmake.generate(version,
File “/home/workspace/pytorch/tools/setup_helpers/cmake.py”, line 330, in generate
self.run(args, env=my_env)
File “/home/workspace/pytorch/tools/setup_helpers/cmake.py”, line 141, in run
check_call(command, cwd=self.build_dir, env=env)
File “/home/samuel/archiconda3/envs/torchreid/lib/python3.8/subprocess.py”, line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command ‘[‘cmake’, ‘-GNinja’, ‘-DBUILD_PYTHON=True’, ‘-DBUILD_TEST=True’, ‘-DCMAKE_BUILD_TYPE=Release’, ‘-DCMAKE_INSTALL_PREFIX=/home/workspace/pytorch/torch’, ‘-DCMAKE_PREFIX_PATH=/home/samuel/archiconda3/envs/torchreid/lib/python3.8/site-packages’, ‘-DNUMPY_INCLUDE_DIR=/home/samuel/archiconda3/envs/torchreid/lib/python3.8/site-packages/numpy/core/include’, ‘-DPYTHON_EXECUTABLE=/home/samuel/archiconda3/envs/torchreid/bin/python3.8’, ‘-DPYTHON_INCLUDE_DIR=/home/samuel/archiconda3/envs/torchreid/include/python3.8’, ‘-DPYTHON_LIBRARY=/home/samuel/archiconda3/envs/torchreid/lib/libpython3.8.so.1.0’, ‘-DTORCH_BUILD_VERSION=1.7.0’, ‘-DUSE_DISTRIBUTED=0’, ‘-DUSE_NCCL=0’, ‘-DUSE_NUMPY=True’, ‘-DUSE_PYTORCH_QNNPACK=0’, ‘-DUSE_QNNPACK=0’, ‘/home/workspace/pytorch’]’ returned non-zero exit status 1.

[xianmei.lei]

@SamuelWei

/home/workspace/pytorch/build/CMakeFiles/CMakeError.log

You can see the content of the error which logged here: /home/workspace/pytorch/build/CMakeFiles/CMakeError.log

[1341035127]

I have successfully built PyTorch with Python 3.8 according to your method. Thank you very much.

[b3hroozam]

the cuda10.2 does not support gcc greater than 8 version.
the error is

[zhongzisha]

I forget how to solve this problem at that time. Maybe you can find something in this link: CUDA 10.2 incompatible with GCC 9.3 and Clang 9.0 · Issue #3654 · espressomd/espresso (github.com).