Libcuda.so: file not recognized: File truncated when make during a docker build

Accelerated Computing NGC GPU Cloud Docker and NVIDIA Docker
1

Hi, I’m currently using nvcr.io/nvidia/deepstream-l4t:5.1-21.02-samples to build my own docker image. But I need to make my app during the build of this DockerFile with “RUN make” and it returns an error… Here is a simplified DockerFile (I know that some libs are missing) that gives the same error :

FROM nvcr.io/nvidia/deepstream-l4t:5.1-21.02-samples
RUN apt-get update && apt-get update -y ; \
    apt-get install -y make ; \
    apt-get install -y libgstreamer-plugins-base1.0-dev ; \
    apt-get install -y libgstreamer1.0-dev ; \
    apt-get install -y libgstrtspserver-1.0-dev ; \
    apt-get install -y libx11-dev ; \
    apt-get install -y gcc ; \
    apt-get install -y git ; \
    apt-get install -y libgstreamer-plugins-base1.0-dev ; \
    apt-get install -y libgstreamer1.0-dev ; \
    apt-get install -y libgstrtspserver-1.0-dev ; \
    apt-get install -y libx11-dev ; \
    apt-get install -y libjson-glib-dev ; \
    apt-get install -y libssl-dev ; \
    cd /opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream-test5/ ; \
    sed -i 's/?=/:=10.2/g' Makefile ; \
    make
ENTRYPOINT /bin/bash

The error is :

/usr/bin/ld: cannot find -lcudart
/usr/bin/ld: cannot find -lnvdsgst_meta
/usr/bin/ld: cannot find -lnvds_meta
/usr/bin/ld: cannot find -lnvdsgst_helper
/usr/bin/ld: cannot find -lnvdsgst_smartrecord
/usr/bin/ld: cannot find -lnvds_utils
/usr/bin/ld: cannot find -lnvds_msgbroker
/usr/lib/gcc/aarch64-linux-gnu/7/../../../aarch64-linux-gnu/libcuda.so: file not recognized: File truncated
collect2: error: ld returned 1 exit status
Makefile:71: recipe for target 'deepstream-test5-app' failed
make: *** [deepstream-test5-app] Error 1
The command '/bin/sh -c apt-get update && apt-get update -y ;     apt-get install -y make ;     apt-get install -y libgstreamer-plugins-base1.0-dev ;     apt-get install -y libgstreamer1.0-dev ;     apt-get install -y libgstrtspserver-1.0-dev ;     apt-get install -y libx11-dev ;     apt-get install -y gcc ;     apt-get install -y git ;     apt-get install -y libgstreamer-plugins-base1.0-dev ;     apt-get install -y libgstreamer1.0-dev ;     apt-get install -y libgstrtspserver-1.0-dev ;     apt-get install -y libx11-dev ;     apt-get install -y libjson-glib-dev ;     apt-get install -y libssl-dev ;     cd /opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream-test5/ ;     sed -i 's/?=/:=10.2/g' Makefile ;     make' returned a non-zero code: 2

Here are the full logs :

logs.txt (109.5 KB)

Is this normal that the make cannot access to libcuda.so when building the DockerFile, but when running it with “ENTRYPOINT make” instead it works ?

I’m using :

Deepstream 5.1
Jetpack 4.5.1

2

I am having the same issue. Did you manage to resolve this problem?

Regards,
Patrick

3

I’m still having the problem… Even if it’s slower, I’m executing the “make” in an Entrypoint.

Hope someone will find a solution or an answer to this topic.

4

Hi, I did not run exactly the l4t deepstream to face this issue. I was using deepstream:5.1-21.02-triton when I encountered the same issue as you. When running make from docker build, it throws me the error upon reaching make portion. However, if I skip the make portion and only perform make when entering the docker container (via docker exec -it <> bash), it works.

Check if you have installed nvidia-container-runtime. If not, install using sudo apt-get install nvidia-container-runtime. Probably you need gpu access during docker build. Credits to https://stackoverflow.com/questions/59691207/docker-build-with-nvidia-runtime

Check if your docker default-runtime is set to nvidia in /etc/docker/daemon.json

{
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
         } 
    },
    "default-runtime": "nvidia"

}

Alternatively, I found out that the libcuda.so in deepstream:5.1-21.02-triton image had different versions of libcuda.so, but not the -devel image. Hence, I switched to using deepstream:5.1-21.02-devel and it works find for me.

Further info on the difference:

In -devel image:

root@dd855203f1c8:~/src/plugins/gst-nvinfer# find / -name "libcuda.so"
/usr/local/cuda-11.1/targets/x86_64-linux/lib/stubs/libcuda.so
/usr/local/cuda-11.1/compat/libcuda.so

In -triton image:

root@dd855203f1c8:~/src/plugins/gst-nvinfer# find / -name "libcuda.so"
/usr/local/cuda-11.1/targets/x86_64-linux/lib/stubs/libcuda.so
/usr/local/cuda-11.1/compat/libcuda.so
/usr/lib/x86_64-linux-gnu/libcuda.so

Hope it helps.

5

Update:

The explanation to @Guitariout situation is due to the image expected GPU access via nvidia-container-runtime but the docker build is not using it. Therefore, the error appeared. However, when launching docker container via docker run --runtime==nvidia ..., the container has access to the GPU and able to make.

The culprit is due to /usr/lib/x86_64-linux-gnu/libcuda.so present in the image while using default docker runtime which was set to nvidia-container-runtime. First of all, ensure that the machine’s /etc/docker/daemon.json doesn’t have the default-runtime defined. Remove if it is present. Reload docker daemon and restart docker. To be safe, you can reboot your machine (I had to).

It should look like this :

{
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
         } 
    }
}

Rebuild your Dockerfile again and you should be good to go.

Note:
You may try and see the difference in the presence of /usr/local/cuda-11.1/lib64/libcuda.so in nvcr.io/nvidia/deepstream-l4t:5.1-21.02-samples and your image. To avoid the error you are facing, /usr/local/cuda-11.1/lib64/libcuda.so should not be present.