Hi,
I have a jetson orin nano 8gb development kit. When I try to build the l4t-pytorch container, torchaudio fails to build.
– L4T_VERSION=35.4.1
– JETPACK_VERSION=5.1.2
– CUDA_VERSION=11.4.315
– LSB_RELEASE=20.04 (focal)
I run the command
omar@ubuntu:~/jetson-containers$ ./build.sh --name=arsd_test l4t-pytorch
I get the following error
.
.
.
[149/182] cd /torchaudio/build/temp.linux-aarch64-cpython-38/third_party/sox && /usr/local/lib/python3.8/dist-packages/cmake/data/bin/cmake -E make_directory /torchaudio/build/temp.linux-aarch64-cpython-38/third_party/sox/CMakeFiles && /usr/local/lib/python3.8/dist-packages/cmake/data/bin/cmake -E touch /torchaudio/build/temp.linux-aarch64-cpython-38/third_party/sox/CMakeFiles/opusfile-complete && /usr/local/lib/python3.8/dist-packages/cmake/data/bin/cmake -E touch /torchaudio/build/temp.linux-aarch64-cpython-38/third_party/sox/src/opusfile-stamp/opusfile-done
[150/182] /usr/local/cuda/bin/nvcc -forward-unknown-to-host-compiler -DINCLUDE_KALDI -DINCLUDE_RIR -DUSE_CUDA -Dlibtorchaudio_EXPORTS -I/torchaudio -I/torchaudio/third_party/kaldi/src -I/torchaudio/third_party/kaldi/submodule/src -isystem /usr/local/lib/python3.8/dist-packages/torch/include -isystem /usr/local/lib/python3.8/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/cuda/include -DONNX_NAMESPACE=onnx_c2 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_86,code=compute_86 -Xcudafe --diag_suppress=cc_clobber_ignored,--diag_suppress=integer_sign_change,--diag_suppress=useless_using_declaration,--diag_suppress=set_but_not_used,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=implicit_return_from_non_void_function,--diag_suppress=unsigned_compare_with_zero,--diag_suppress=declared_but_not_referenced,--diag_suppress=bad_friend_decl --expt-relaxed-constexpr --expt-extended-lambda -O3 -DNDEBUG -std=c++17 -Xcompiler=-fPIC -D_GLIBCXX_USE_CXX11_ABI=1 -MD -MT torchaudio/csrc/CMakeFiles/libtorchaudio.dir/iir_cuda.cu.o -MF torchaudio/csrc/CMakeFiles/libtorchaudio.dir/iir_cuda.cu.o.d -x cu -c /torchaudio/torchaudio/csrc/iir_cuda.cu -o torchaudio/csrc/CMakeFiles/libtorchaudio.dir/iir_cuda.cu.o
FAILED: torchaudio/csrc/CMakeFiles/libtorchaudio.dir/iir_cuda.cu.o
/usr/local/cuda/bin/nvcc -forward-unknown-to-host-compiler -DINCLUDE_KALDI -DINCLUDE_RIR -DUSE_CUDA -Dlibtorchaudio_EXPORTS -I/torchaudio -I/torchaudio/third_party/kaldi/src -I/torchaudio/third_party/kaldi/submodule/src -isystem /usr/local/lib/python3.8/dist-packages/torch/include -isystem /usr/local/lib/python3.8/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/cuda/include -DONNX_NAMESPACE=onnx_c2 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_86,code=compute_86 -Xcudafe --diag_suppress=cc_clobber_ignored,--diag_suppress=integer_sign_change,--diag_suppress=useless_using_declaration,--diag_suppress=set_but_not_used,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=implicit_return_from_non_void_function,--diag_suppress=unsigned_compare_with_zero,--diag_suppress=declared_but_not_referenced,--diag_suppress=bad_friend_decl --expt-relaxed-constexpr --expt-extended-lambda -O3 -DNDEBUG -std=c++17 -Xcompiler=-fPIC -D_GLIBCXX_USE_CXX11_ABI=1 -MD -MT torchaudio/csrc/CMakeFiles/libtorchaudio.dir/iir_cuda.cu.o -MF torchaudio/csrc/CMakeFiles/libtorchaudio.dir/iir_cuda.cu.o.d -x cu -c /torchaudio/torchaudio/csrc/iir_cuda.cu -o torchaudio/csrc/CMakeFiles/libtorchaudio.dir/iir_cuda.cu.o
nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
Killed
[151/182] /usr/bin/c++ -DINCLUDE_KALDI -DINCLUDE_RIR -DUSE_CUDA -Dlibtorchaudio_EXPORTS -I/torchaudio -I/torchaudio/third_party/kaldi/src -I/torchaudio/third_party/kaldi/submodule/src -isystem /usr/local/lib/python3.8/dist-packages/torch/include -isystem /usr/local/lib/python3.8/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/cuda/include -Wall -D_GLIBCXX_USE_CXX11_ABI=1 -O3 -DNDEBUG -std=gnu++17 -fPIC -D_GLIBCXX_USE_CXX11_ABI=1 -fopenmp -MD -MT torchaudio/csrc/CMakeFiles/libtorchaudio.dir/rir.cpp.o -MF torchaudio/csrc/CMakeFiles/libtorchaudio.dir/rir.cpp.o.d -o torchaudio/csrc/CMakeFiles/libtorchaudio.dir/rir.cpp.o -c /torchaudio/torchaudio/csrc/rir.cpp
[152/182] /usr/bin/c++ -DINCLUDE_KALDI -DINCLUDE_RIR -DUSE_CUDA -Dlibtorchaudio_EXPORTS -I/torchaudio -I/torchaudio/third_party/kaldi/src -I/torchaudio/third_party/kaldi/submodule/src -isystem /usr/local/lib/python3.8/dist-packages/torch/include -isystem /usr/local/lib/python3.8/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/cuda/include -Wall -D_GLIBCXX_USE_CXX11_ABI=1 -O3 -DNDEBUG -std=gnu++17 -fPIC -D_GLIBCXX_USE_CXX11_ABI=1 -fopenmp -MD -MT torchaudio/csrc/CMakeFiles/libtorchaudio.dir/kaldi.cpp.o -MF torchaudio/csrc/CMakeFiles/libtorchaudio.dir/kaldi.cpp.o.d -o torchaudio/csrc/CMakeFiles/libtorchaudio.dir/kaldi.cpp.o -c /torchaudio/torchaudio/csrc/kaldi.cpp
[153/182] /usr/bin/c++ -DUSE_CUDA -D_torchaudio_EXPORTS -I/torchaudio -I/usr/include/python3.8 -I/torchaudio/third_party/kaldi/src -I/torchaudio/third_party/kaldi/submodule/src -isystem /usr/local/lib/python3.8/dist-packages/torch/include -isystem /usr/local/lib/python3.8/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/cuda/include -Wall -D_GLIBCXX_USE_CXX11_ABI=1 -O3 -DNDEBUG -std=gnu++17 -fPIC -D_GLIBCXX_USE_CXX11_ABI=1 -fopenmp -MD -MT torchaudio/csrc/CMakeFiles/_torchaudio.dir/pybind/pybind.cpp.o -MF torchaudio/csrc/CMakeFiles/_torchaudio.dir/pybind/pybind.cpp.o.d -o torchaudio/csrc/CMakeFiles/_torchaudio.dir/pybind/pybind.cpp.o -c /torchaudio/torchaudio/csrc/pybind/pybind.cpp
[154/182] /usr/local/cuda/bin/nvcc -forward-unknown-to-host-compiler -DINCLUDE_KALDI -DINCLUDE_RIR -DUSE_CUDA -Dlibtorchaudio_EXPORTS -I/torchaudio -I/torchaudio/third_party/kaldi/src -I/torchaudio/third_party/kaldi/submodule/src -isystem /usr/local/lib/python3.8/dist-packages/torch/include -isystem /usr/local/lib/python3.8/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/cuda/include -DONNX_NAMESPACE=onnx_c2 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_86,code=compute_86 -Xcudafe --diag_suppress=cc_clobber_ignored,--diag_suppress=integer_sign_change,--diag_suppress=useless_using_declaration,--diag_suppress=set_but_not_used,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=implicit_return_from_non_void_function,--diag_suppress=unsigned_compare_with_zero,--diag_suppress=declared_but_not_referenced,--diag_suppress=bad_friend_decl --expt-relaxed-constexpr --expt-extended-lambda -O3 -DNDEBUG -std=c++17 -Xcompiler=-fPIC -D_GLIBCXX_USE_CXX11_ABI=1 -MD -MT torchaudio/csrc/CMakeFiles/libtorchaudio.dir/rnnt/gpu/compute_betas.cu.o -MF torchaudio/csrc/CMakeFiles/libtorchaudio.dir/rnnt/gpu/compute_betas.cu.o.d -x cu -c /torchaudio/torchaudio/csrc/rnnt/gpu/compute_betas.cu -o torchaudio/csrc/CMakeFiles/libtorchaudio.dir/rnnt/gpu/compute_betas.cu.o
nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
[155/182] /usr/local/cuda/bin/nvcc -forward-unknown-to-host-compiler -DINCLUDE_KALDI -DINCLUDE_RIR -DUSE_CUDA -Dlibtorchaudio_EXPORTS -I/torchaudio -I/torchaudio/third_party/kaldi/src -I/torchaudio/third_party/kaldi/submodule/src -isystem /usr/local/lib/python3.8/dist-packages/torch/include -isystem /usr/local/lib/python3.8/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/cuda/include -DONNX_NAMESPACE=onnx_c2 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_86,code=compute_86 -Xcudafe --diag_suppress=cc_clobber_ignored,--diag_suppress=integer_sign_change,--diag_suppress=useless_using_declaration,--diag_suppress=set_but_not_used,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=implicit_return_from_non_void_function,--diag_suppress=unsigned_compare_with_zero,--diag_suppress=declared_but_not_referenced,--diag_suppress=bad_friend_decl --expt-relaxed-constexpr --expt-extended-lambda -O3 -DNDEBUG -std=c++17 -Xcompiler=-fPIC -D_GLIBCXX_USE_CXX11_ABI=1 -MD -MT torchaudio/csrc/CMakeFiles/libtorchaudio.dir/rnnt/gpu/compute_alphas.cu.o -MF torchaudio/csrc/CMakeFiles/libtorchaudio.dir/rnnt/gpu/compute_alphas.cu.o.d -x cu -c /torchaudio/torchaudio/csrc/rnnt/gpu/compute_alphas.cu -o torchaudio/csrc/CMakeFiles/libtorchaudio.dir/rnnt/gpu/compute_alphas.cu.o
nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
[156/182] cd /torchaudio/build/temp.linux-aarch64-cpython-38/third_party/sox/src/amr-build && /usr/local/lib/python3.8/dist-packages/cmake/data/bin/cmake -P /torchaudio/build/temp.linux-aarch64-cpython-38/third_party/sox/src/amr-stamp/amr-build-Release.cmake && /usr/local/lib/python3.8/dist-packages/cmake/data/bin/cmake -E touch /torchaudio/build/temp.linux-aarch64-cpython-38/third_party/sox/src/amr-stamp/amr-build
[157/182] /usr/local/cuda/bin/nvcc -forward-unknown-to-host-compiler -DINCLUDE_KALDI -DINCLUDE_RIR -DUSE_CUDA -Dlibtorchaudio_EXPORTS -I/torchaudio -I/torchaudio/third_party/kaldi/src -I/torchaudio/third_party/kaldi/submodule/src -isystem /usr/local/lib/python3.8/dist-packages/torch/include -isystem /usr/local/lib/python3.8/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/cuda/include -DONNX_NAMESPACE=onnx_c2 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_86,code=compute_86 -Xcudafe --diag_suppress=cc_clobber_ignored,--diag_suppress=integer_sign_change,--diag_suppress=useless_using_declaration,--diag_suppress=set_but_not_used,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=implicit_return_from_non_void_function,--diag_suppress=unsigned_compare_with_zero,--diag_suppress=declared_but_not_referenced,--diag_suppress=bad_friend_decl --expt-relaxed-constexpr --expt-extended-lambda -O3 -DNDEBUG -std=c++17 -Xcompiler=-fPIC -D_GLIBCXX_USE_CXX11_ABI=1 -MD -MT torchaudio/csrc/CMakeFiles/libtorchaudio.dir/rnnt/gpu/compute.cu.o -MF torchaudio/csrc/CMakeFiles/libtorchaudio.dir/rnnt/gpu/compute.cu.o.d -x cu -c /torchaudio/torchaudio/csrc/rnnt/gpu/compute.cu -o torchaudio/csrc/CMakeFiles/libtorchaudio.dir/rnnt/gpu/compute.cu.o
nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
ninja: build stopped: subcommand failed.
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
Traceback (most recent call last):
File "setup.py", line 184, in <module>
_main()
File "setup.py", line 147, in _main
setup(
File "/usr/local/lib/python3.8/dist-packages/setuptools/__init__.py", line 107, in setup
return distutils.core.setup(**attrs)
File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/core.py", line 185, in setup
return run_commands(dist)
File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/core.py", line 201, in run_commands
dist.run_commands()
File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/usr/local/lib/python3.8/dist-packages/setuptools/dist.py", line 1233, in run_command
super().run_command(command)
File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/usr/local/lib/python3.8/dist-packages/wheel/bdist_wheel.py", line 364, in run
self.run_command("build")
File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/usr/local/lib/python3.8/dist-packages/setuptools/dist.py", line 1233, in run_command
super().run_command(command)
File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/command/build.py", line 131, in run
self.run_command(cmd_name)
File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/usr/local/lib/python3.8/dist-packages/setuptools/dist.py", line 1233, in run_command
super().run_command(command)
File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/torchaudio/tools/setup_helpers/extension.py", line 86, in run
super().run()
File "/usr/local/lib/python3.8/dist-packages/setuptools/command/build_ext.py", line 88, in run
_build_ext.run(self)
File "/usr/local/lib/python3.8/dist-packages/Cython/Distutils/old_build_ext.py", line 186, in run
_build_ext.build_ext.run(self)
File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
self.build_extensions()
File "/usr/local/lib/python3.8/dist-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
_build_ext.build_ext.build_extensions(self)
File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
self._build_extensions_serial()
File "/usr/local/lib/python3.8/dist-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
self.build_extension(ext)
File "/torchaudio/tools/setup_helpers/extension.py", line 168, in build_extension
subprocess.check_call(["cmake", "--build", "."] + build_args, cwd=self.build_temp)
File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install']' returned non-zero exit status 1.
The command '/bin/sh -c git clone --branch ${TORCHAUDIO_VERSION} --recursive --depth=1 https://github.com/pytorch/audio torchaudio && cd torchaudio && git checkout ${TORCHAUDIO_VERSION} && sed -i 's# URL https://zlib.net/zlib-1.2.11.tar.gz# URL https://github.com/madler/zlib/archive/refs/tags/v1.2.12.tar.gz#g' third_party/zlib/CMakeLists.txt || echo "failed to patch torchaudio/third_party/zlib/CMakeLists.txt" && sed -i 's# URL_HASH SHA256=c3e5e9fdd5004dcb542feda5ee4f0ff0744628baf8ed2dd5d66f8ca1197cb1a1# URL_HASH SHA256=d8688496ea40fb61787500e863cc63c9afcbc524468cedeb478068924eb54932#g' third_party/zlib/CMakeLists.txt || echo "failed to patch torchaudio/third_party/zlib/CMakeLists.txt" && BUILD_SOX=1 python3 setup.py bdist_wheel && cp dist/torchaudio*.whl /opt && pip3 install --no-cache-dir --verbose /opt/torchaudio*.whl && cd ../ && rm -rf torchaudio' returned a non-zero code: 1
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/omar/jetson-containers/jetson_containers/build.py", line 93, in <module>
build_container(args.name, args.packages, args.base, args.build_flags, args.simulate, args.skip_tests, args.push)
File "/home/omar/jetson-containers/jetson_containers/container.py", line 119, in build_container
status = subprocess.run(cmd.replace(_NEWLINE_, ' '), executable='/bin/bash', shell=True, check=True)
File "/usr/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'docker build --network=host --tag arsd_test:r35.4.1-torchaudio --file /home/omar/jetson-containers/packages/pytorch/torchaudio/Dockerfile --build-arg BASE_IMAGE=arsd_test:r35.4.1-torchvision --build-arg TORCHAUDIO_VERSION="v2.0.1" /home/omar/jetson-containers/packages/pytorch/torchaudio 2>&1 | tee /home/omar/jetson-containers/logs/20230906_134627/build/arsd_test_r35.4.1-torchaudio.txt; exit ${PIPESTATUS[0]}' returned non-zero exit status 1.
pytorch and torchvision both are built and tested successfully without problems.
thanks :)