Pycuda Installation Issues in Docker Container

Autonomous Machines Jetson & Embedded Systems Jetson Xavier NX
,
#1

Hi everyone,

I am having some issues with Pycuda installation, both 2019 and 2020 versions.
Jetson NX board Jetpack 32.4.2 with nvidia docker container and --runtime nvidia arg when running a docker container.
Tested Docker base images: nvcr.io/nvidia/l4t-base:r32.3.1 / FROM nvcr.io/nvidia/l4t-base:r32.4.4

Other installments: python3-pip, cmake and numpy_ringbuffer

For l4t-base:r32.3.1 :

PATH=/usr/local/cuda-10.0/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LD_LIBRARY_PATH=/usr/local/cuda-10.0/targets/aarch64-linux/lib
C_INCLUDE_PATH=/usr/local/cuda/targets/aarch64-linux/include
LIBRARY_PATH=/usr/local/cuda/targets/aarch64-linux/lib

Installed Pycuda with : pip3 install pycuda==2019.1.2 --no-cache-dir

This image has two versions of cuda, cuda 10.0 and cuda 10.2 installed by default and /usr/local/cuda points to cuda 10.0
The issue is the following: Installing pycuda with pip3 works fine, but when you run in python: import pycuda.autoinit it crashes with the following error: ImportError: libcurand.so.10.0: cannot open shared object file: No such file or directory

For l4t-base:r32.4.4 :

PATH=/usr/local/cuda-10.2/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LD_LIBRARY_PATH=/usr/local/cuda-10.2/targets/aarch64-linux/lib
C_INCLUDE_PATH=/usr/local/cuda-10.2/targets/aarch64-linux/include
LIBRARY_PATH=/usr/local/cuda-10.2/targets/aarch64-linux/lib

Installed Pycuda with : pip3 install pycuda==2019.1.2 --no-cache-dir

The issue here is the following:

 /usr/local/cuda-10.2/include/cuda_runtime.h:83:10: fatal error: crt/host_config.h: No such file or directory
 #include "crt/host_config.h"

File cuda_runtime.h has include <crt/host_config.h> that is in /lib and no longer in crt folder, so it will not compile.

Tried a lot of different things but to no success. Any suggestions on how to solve this?

Added the Dockerfile that i use
Dockerfile (415 Bytes)

Run it with the following command:

sudo docker build --tag jetson_dimg .
sudo docker run -it --rm --network host  --runtime nvidia jetson_dimg:latest

Thanks,
Cristian R

#2

Hi @christian.rotaru, here is how I build PyCUDA inside Dockerfile:

When building libraries that use CUDA inside containers, you should set you default docker runtime to nvidia (as shown in the link below) - that way, the nvidia runtime will be used during your docker build:

https://github.com/dusty-nv/jetson-containers#docker-default-runtime

#3

Hey @dusty_nv, thanks a lot for the info. I have tried what you proposed. Added

ENV PATH="/usr/local/cuda/bin:${PATH}"
ENV LD_LIBRARY_PATH="/usr/local/cuda/lib64:${LD_LIBRARY_PATH}"
RUN echo "$PATH" && echo "$LD_LIBRARY_PATH"
RUN pip3 install pycuda --verbose

to Dockerfile, but it still crashes with the following error:

In file included from src/cpp/cuda.cpp:4:0:
    src/cpp/cuda.hpp:14:10: fatal error: cuda.h: No such file or directory
     #include <cuda.h>
              ^~~~~~~~
    compilation terminated.
    error: command 'aarch64-linux-gnu-gcc' failed with exit status 1
    Running setup.py install for pycuda: finished with status 'error'

which is odd, since the libs do point right.

#4

Did you set your default docker runtime to nvidia (and then reboot or restart docker daemon), like shown here:

Otherwise the CUDA files won’t be available during your docker build operation.

#5

Hi @dusty_nv, yes it is.

What i noticed is that according to this (https://github.com/NVIDIA/nvidia-docker/wiki/NVIDIA-Container-Runtime-on-Jetson) i should have this:

libnvidia-container-tools			install
libnvidia-container0:arm64			install
nvidia-container-runtime			install
nvidia-container-runtime-hook		install
nvidia-docker2				install

when running sudo dpkg --get-selections | grep nvidia

but what i have is this:

libnvidia-container-tools			install
libnvidia-container0:arm64			install
nvidia-container-csv-cuda			deinstall
nvidia-container-csv-cudnn			install
nvidia-container-csv-tensorrt			install
nvidia-container-csv-visionworks		install
nvidia-container-runtime			install
nvidia-container-toolkit			install
nvidia-docker2					install

As you can see, on the Jetson i am missing the nvidia-container-runtime-hook install . Could this be the real issue here?

Thanks!

#6

So, i reinstalled the latest JetPack and now it works with no issues. Apparently it was a problem with the jetpack image somehow.

Thanks,
Cristian

#7

OK great, glad you got it working!

By the way, this is what the dpkg command returns on my Jetson:

$ sudo dpkg --get-selections | grep nvidia
libnvidia-container-tools                       install
libnvidia-container0:arm64                      install
nvidia-container-csv-cuda                       install
nvidia-container-csv-cudnn                      install
nvidia-container-csv-tensorrt                   install
nvidia-container-csv-visionworks                install
nvidia-container-runtime                        install
nvidia-container-toolkit                        install
nvidia-docker2                                  install
nvidia-l4t-3d-core                              install
nvidia-l4t-apt-source                           install
nvidia-l4t-bootloader                           install
nvidia-l4t-camera                               install
nvidia-l4t-configs                              install
nvidia-l4t-core                                 install
nvidia-l4t-cuda                                 install
nvidia-l4t-firmware                             install
nvidia-l4t-graphics-demos                       install
nvidia-l4t-gstreamer                            install
nvidia-l4t-init                                 install
nvidia-l4t-initrd                               install
nvidia-l4t-jetson-io                            install
nvidia-l4t-jetson-multimedia-api                install
nvidia-l4t-kernel                               install
nvidia-l4t-kernel-dtbs                          install
nvidia-l4t-kernel-headers                       install
nvidia-l4t-multimedia                           install
nvidia-l4t-multimedia-utils                     install
nvidia-l4t-oem-config                           install
nvidia-l4t-tools                                install
nvidia-l4t-wayland                              install
nvidia-l4t-weston                               install
nvidia-l4t-x11                                  install
nvidia-l4t-xusb-firmware                        install