Error in Building Machine Learning Container for Jetson and Jetpack 4.4

adaptrobotics · September 13, 2020, 1:05am

Hello, my team and I are trying to use a neural network on the Xavier for image segmentation. We decided to use a docker container after having a lot issues trying to install various versions of python packages including pytorch and torchvision in conda on the arm architecture. We came across this resource: NVIDIA L4T ML | NVIDIA NGC
and successfully ran the l4t-ml:r32.4.3-p4 container.

We wanted to change the version of pytorch, torchvision, and add other python packages such as matplotlib. So we went to: GitHub - dusty-nv/jetson-containers: Machine Learning Containers for NVIDIA Jetson and JetPack-L4T to look at the original dockerfiles and scripts to run the dockerfiles. We attempted to alter the files for simple installation of packages like matplotlib and changed the pytorch and torch versions as well. After running docker_build.sh to install the pytorch version of the container we kept getting errors such as:

OSError: libnvToolsExt.so.1: cannot open shared object file: No such file or directory

[ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory]

My guess is that we need CUDA 10.0. But the l4t-ml container specifically asked for Jetpack 4.4, which installs CUDA 10.2 by default.

Any help would be appreciated, whether it is a work around for building the dockerfile, or a way to install any version of a package in our conda environment on arm architecture (pip and conda install sometimes just do not have the package versions available for installation). The latter would be the most useful, as learning the nuances of Docker has already used up a lot of time that we do not have.

AastaLLL · September 14, 2020, 4:31am

Hi,

Which version do you want to use?
Please remember to un-comment the preferred version in this script first:

github.com

dusty-nv/jetson-containers/blob/master/scripts/docker_build_ml.sh#L41


      
          			--build-arg TORCHAUDIO_VERSION=$audio_version \
          			--build-arg TORCH_CUDA_ARCH_LIST=$cuda_arch_list \
          			--build-arg OPENCV_URL=$OPENCV_URL \
          			--build-arg OPENCV_DEB=$OPENCV_DEB 
          
          
	echo "done building PyTorch $pytorch_whl, torchvision $vision_version, torchaudio $audio_version, cuda arch $cuda_arch_list"
          }
          
          
if [[ "$CONTAINERS" == "pytorch" || "$CONTAINERS" == "all" ]]; then
          
          
	if [[ $L4T_RELEASE -eq 32 ]]; then   # JetPack 4.x
          
          
		# PyTorch v1.2.0
          		#build_pytorch "https://nvidia.box.com/shared/static/lufbgr3xu2uha40cs9ryq1zn4kxsnogl.whl" \
          		#			  "torch-1.2.0-cp36-cp36m-linux_aarch64.whl" \
          		#			  "l4t-pytorch:r$L4T_VERSION-pth1.2-py3" \
          		#			  "v0.4.0"
          
          
		# PyTorch v1.3.0
          		#build_pytorch "https://nvidia.box.com/shared/static/017sci9z4a0xhtwrb4ps52frdfti9iw0.whl" \
          		#			  "torch-1.3.0-cp36-cp36m-linux_aarch64.whl" \

Thanks.

dusty_nv · September 14, 2020, 1:45pm

Was this during build time? If so, make sure you docker daemon’s default-runtime is set to nvidia: https://github.com/dusty-nv/jetson-containers#docker-default-runtime

As @AastaLLL mentioned, inside docker_build_ml.sh you should select the appropriate version of PyTorch for your JetPack-L4T version. However it seems you are on L4T R32.4.3, and PyTorch 1.6 is already selected by default in the script and the right version to use for 32.4.3.