Hi everyone,
I am having some issues with Pycuda installation, both 2019 and 2020 versions.
Jetson NX board Jetpack 32.4.2 with nvidia docker container and --runtime nvidia arg when running a docker container.
Tested Docker base images: nvcr.io/nvidia/l4t-base:r32.3.1 / FROM nvcr.io/nvidia/l4t-base:r32.4.4
Other installments: python3-pip, cmake and numpy_ringbuffer
For l4t-base:r32.3.1 :
PATH=/usr/local/cuda-10.0/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LD_LIBRARY_PATH=/usr/local/cuda-10.0/targets/aarch64-linux/lib
C_INCLUDE_PATH=/usr/local/cuda/targets/aarch64-linux/include
LIBRARY_PATH=/usr/local/cuda/targets/aarch64-linux/lib
Installed Pycuda with : pip3 install pycuda==2019.1.2 --no-cache-dir
This image has two versions of cuda, cuda 10.0 and cuda 10.2 installed by default and /usr/local/cuda points to cuda 10.0
The issue is the following: Installing pycuda with pip3 works fine, but when you run in python: import pycuda.autoinit it crashes with the following error: ImportError: libcurand.so.10.0: cannot open shared object file: No such file or directory
For l4t-base:r32.4.4 :
PATH=/usr/local/cuda-10.2/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LD_LIBRARY_PATH=/usr/local/cuda-10.2/targets/aarch64-linux/lib
C_INCLUDE_PATH=/usr/local/cuda-10.2/targets/aarch64-linux/include
LIBRARY_PATH=/usr/local/cuda-10.2/targets/aarch64-linux/lib
Installed Pycuda with : pip3 install pycuda==2019.1.2 --no-cache-dir
The issue here is the following:
/usr/local/cuda-10.2/include/cuda_runtime.h:83:10: fatal error: crt/host_config.h: No such file or directory
#include "crt/host_config.h"
File cuda_runtime.h has include <crt/host_config.h> that is in /lib and no longer in crt folder, so it will not compile.
Tried a lot of different things but to no success. Any suggestions on how to solve this?
Added the Dockerfile that i use
Dockerfile (415 Bytes)
Run it with the following command:
sudo docker build --tag jetson_dimg .
sudo docker run -it --rm --network host --runtime nvidia jetson_dimg:latest
Thanks,
Cristian R
Hi @christian.rotaru, here is how I build PyCUDA inside Dockerfile:
When building libraries that use CUDA inside containers, you should set you default docker runtime to nvidia (as shown in the link below) - that way, the nvidia runtime will be used during your docker build:
https://github.com/dusty-nv/jetson-containers#docker-default-runtime
1 Like
Hey @dusty_nv, thanks a lot for the info. I have tried what you proposed. Added
ENV PATH="/usr/local/cuda/bin:${PATH}"
ENV LD_LIBRARY_PATH="/usr/local/cuda/lib64:${LD_LIBRARY_PATH}"
RUN echo "$PATH" && echo "$LD_LIBRARY_PATH"
RUN pip3 install pycuda --verbose
to Dockerfile, but it still crashes with the following error:
In file included from src/cpp/cuda.cpp:4:0:
src/cpp/cuda.hpp:14:10: fatal error: cuda.h: No such file or directory
#include <cuda.h>
^~~~~~~~
compilation terminated.
error: command 'aarch64-linux-gnu-gcc' failed with exit status 1
Running setup.py install for pycuda: finished with status 'error'
which is odd, since the libs do point right.
Did you set your default docker runtime to nvidia (and then reboot or restart docker daemon), like shown here:
Otherwise the CUDA files won’t be available during your docker build operation.
Hi @dusty_nv, yes it is.
What i noticed is that according to this (NVIDIA Container Runtime on Jetson · NVIDIA/nvidia-docker Wiki · GitHub) i should have this:
libnvidia-container-tools install
libnvidia-container0:arm64 install
nvidia-container-runtime install
nvidia-container-runtime-hook install
nvidia-docker2 install
when running sudo dpkg --get-selections | grep nvidia
but what i have is this:
libnvidia-container-tools install
libnvidia-container0:arm64 install
nvidia-container-csv-cuda deinstall
nvidia-container-csv-cudnn install
nvidia-container-csv-tensorrt install
nvidia-container-csv-visionworks install
nvidia-container-runtime install
nvidia-container-toolkit install
nvidia-docker2 install
As you can see, on the Jetson i am missing the nvidia-container-runtime-hook install . Could this be the real issue here?
Thanks!
So, i reinstalled the latest JetPack and now it works with no issues. Apparently it was a problem with the jetpack image somehow.
Thanks,
Cristian
OK great, glad you got it working!
By the way, this is what the dpkg command returns on my Jetson:
$ sudo dpkg --get-selections | grep nvidia
libnvidia-container-tools install
libnvidia-container0:arm64 install
nvidia-container-csv-cuda install
nvidia-container-csv-cudnn install
nvidia-container-csv-tensorrt install
nvidia-container-csv-visionworks install
nvidia-container-runtime install
nvidia-container-toolkit install
nvidia-docker2 install
nvidia-l4t-3d-core install
nvidia-l4t-apt-source install
nvidia-l4t-bootloader install
nvidia-l4t-camera install
nvidia-l4t-configs install
nvidia-l4t-core install
nvidia-l4t-cuda install
nvidia-l4t-firmware install
nvidia-l4t-graphics-demos install
nvidia-l4t-gstreamer install
nvidia-l4t-init install
nvidia-l4t-initrd install
nvidia-l4t-jetson-io install
nvidia-l4t-jetson-multimedia-api install
nvidia-l4t-kernel install
nvidia-l4t-kernel-dtbs install
nvidia-l4t-kernel-headers install
nvidia-l4t-multimedia install
nvidia-l4t-multimedia-utils install
nvidia-l4t-oem-config install
nvidia-l4t-tools install
nvidia-l4t-wayland install
nvidia-l4t-weston install
nvidia-l4t-x11 install
nvidia-l4t-xusb-firmware install