Pycuda Installation Issues in Docker Container

Hi everyone,

I am having some issues with Pycuda installation, both 2019 and 2020 versions.
Jetson NX board Jetpack 32.4.2 with nvidia docker container and --runtime nvidia arg when running a docker container.
Tested Docker base images: nvcr.io/nvidia/l4t-base:r32.3.1 / FROM nvcr.io/nvidia/l4t-base:r32.4.4

Other installments: python3-pip, cmake and numpy_ringbuffer

For l4t-base:r32.3.1 :

PATH=/usr/local/cuda-10.0/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LD_LIBRARY_PATH=/usr/local/cuda-10.0/targets/aarch64-linux/lib
C_INCLUDE_PATH=/usr/local/cuda/targets/aarch64-linux/include
LIBRARY_PATH=/usr/local/cuda/targets/aarch64-linux/lib

Installed Pycuda with : pip3 install pycuda==2019.1.2 --no-cache-dir

This image has two versions of cuda, cuda 10.0 and cuda 10.2 installed by default and /usr/local/cuda points to cuda 10.0
The issue is the following: Installing pycuda with pip3 works fine, but when you run in python: import pycuda.autoinit it crashes with the following error: ImportError: libcurand.so.10.0: cannot open shared object file: No such file or directory

For l4t-base:r32.4.4 :

PATH=/usr/local/cuda-10.2/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LD_LIBRARY_PATH=/usr/local/cuda-10.2/targets/aarch64-linux/lib
C_INCLUDE_PATH=/usr/local/cuda-10.2/targets/aarch64-linux/include
LIBRARY_PATH=/usr/local/cuda-10.2/targets/aarch64-linux/lib

Installed Pycuda with : pip3 install pycuda==2019.1.2 --no-cache-dir

The issue here is the following:

 /usr/local/cuda-10.2/include/cuda_runtime.h:83:10: fatal error: crt/host_config.h: No such file or directory
 #include "crt/host_config.h"

File cuda_runtime.h has include <crt/host_config.h> that is in /lib and no longer in crt folder, so it will not compile.

Tried a lot of different things but to no success. Any suggestions on how to solve this?

Added the Dockerfile that i use
Dockerfile (415 Bytes)

Run it with the following command:

sudo docker build --tag jetson_dimg .
sudo docker run -it --rm --network host  --runtime nvidia jetson_dimg:latest

Thanks,
Cristian R

Hi @christian.rotaru, here is how I build PyCUDA inside Dockerfile:

When building libraries that use CUDA inside containers, you should set you default docker runtime to nvidia (as shown in the link below) - that way, the nvidia runtime will be used during your docker build:

https://github.com/dusty-nv/jetson-containers#docker-default-runtime

Hey @dusty_nv, thanks a lot for the info. I have tried what you proposed. Added

ENV PATH="/usr/local/cuda/bin:${PATH}"
ENV LD_LIBRARY_PATH="/usr/local/cuda/lib64:${LD_LIBRARY_PATH}"
RUN echo "$PATH" && echo "$LD_LIBRARY_PATH"
RUN pip3 install pycuda --verbose

to Dockerfile, but it still crashes with the following error:

In file included from src/cpp/cuda.cpp:4:0:
    src/cpp/cuda.hpp:14:10: fatal error: cuda.h: No such file or directory
     #include <cuda.h>
              ^~~~~~~~
    compilation terminated.
    error: command 'aarch64-linux-gnu-gcc' failed with exit status 1
    Running setup.py install for pycuda: finished with status 'error'

which is odd, since the libs do point right.

Did you set your default docker runtime to nvidia (and then reboot or restart docker daemon), like shown here:

Otherwise the CUDA files won’t be available during your docker build operation.

Hi @dusty_nv, yes it is.

What i noticed is that according to this (https://github.com/NVIDIA/nvidia-docker/wiki/NVIDIA-Container-Runtime-on-Jetson) i should have this:

libnvidia-container-tools			install
libnvidia-container0:arm64			install
nvidia-container-runtime			install
nvidia-container-runtime-hook		install
nvidia-docker2				install

when running sudo dpkg --get-selections | grep nvidia

but what i have is this:

libnvidia-container-tools			install
libnvidia-container0:arm64			install
nvidia-container-csv-cuda			deinstall
nvidia-container-csv-cudnn			install
nvidia-container-csv-tensorrt			install
nvidia-container-csv-visionworks		install
nvidia-container-runtime			install
nvidia-container-toolkit			install
nvidia-docker2					install

As you can see, on the Jetson i am missing the nvidia-container-runtime-hook install . Could this be the real issue here?

Thanks!

So, i reinstalled the latest JetPack and now it works with no issues. Apparently it was a problem with the jetpack image somehow.

Thanks,
Cristian

OK great, glad you got it working!

By the way, this is what the dpkg command returns on my Jetson:

$ sudo dpkg --get-selections | grep nvidia
libnvidia-container-tools                       install
libnvidia-container0:arm64                      install
nvidia-container-csv-cuda                       install
nvidia-container-csv-cudnn                      install
nvidia-container-csv-tensorrt                   install
nvidia-container-csv-visionworks                install
nvidia-container-runtime                        install
nvidia-container-toolkit                        install
nvidia-docker2                                  install
nvidia-l4t-3d-core                              install
nvidia-l4t-apt-source                           install
nvidia-l4t-bootloader                           install
nvidia-l4t-camera                               install
nvidia-l4t-configs                              install
nvidia-l4t-core                                 install
nvidia-l4t-cuda                                 install
nvidia-l4t-firmware                             install
nvidia-l4t-graphics-demos                       install
nvidia-l4t-gstreamer                            install
nvidia-l4t-init                                 install
nvidia-l4t-initrd                               install
nvidia-l4t-jetson-io                            install
nvidia-l4t-jetson-multimedia-api                install
nvidia-l4t-kernel                               install
nvidia-l4t-kernel-dtbs                          install
nvidia-l4t-kernel-headers                       install
nvidia-l4t-multimedia                           install
nvidia-l4t-multimedia-utils                     install
nvidia-l4t-oem-config                           install
nvidia-l4t-tools                                install
nvidia-l4t-wayland                              install
nvidia-l4t-weston                               install
nvidia-l4t-x11                                  install
nvidia-l4t-xusb-firmware                        install