Deepstream container not mounting necessary packages for tensorrt, cuda, and cudnn

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hi,

I am working with Deepstream 6.0 and Jetpack 4.6/L4T 32.6.1. I am trying to set up Deepstream via the docker container, but when I run the container tensorrt, cuda, and cudnn are not mounted correctly in the container. I checked and I have the packages locally, but they do not get mounted correctly. Additionally, I need to use this Jetpack version and the accompanying Deepstream version.

I had to create the tensorrt.csv, cuda.csv, and cudnn.csv files manually under /etc/nvidia-container-runtime/host-files-for-container.d. The only one that existed on boot was the l4t.csv file. The problem is that only the l4t.csv file is being mounted correctly, whereas the tensorrt.csv, cuda.csv, and cudnn.csv files are not being mounted. If I concatenate the other files under l4t.csv, then I am able to get all the packages mounted correctly, but it does not work if I have them separated into individual files such as l4t.csv, cuda.csv, tensorrt.csv, and cudnn.csv. Based on the mount plugin repo, the csv files I made should also be mounting.

I also adjusted /etc/docker/daemon.json to ensure that default runtime was set to nvidia, but the issue still persists. Why would the other csv files not be mounting correctly? For reference, the tensorrt.csv file content is:

lib, /usr/lib/aarch64-linux-gnu/libnvinfer.so.8.0.1
lib, /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.8.0.1
lib, /usr/lib/aarch64-linux-gnu/libnvparsers.so.8.0.1
lib, /usr/lib/aarch64-linux-gnu/libnvonnxparser.so.8.0.1
sym, /usr/lib/aarch64-linux-gnu/libnvinfer.so.8
sym, /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.8
sym, /usr/lib/aarch64-linux-gnu/libnvparsers.so.8
sym, /usr/lib/aarch64-linux-gnu/libnvonnxparser.so.8
sym, /usr/lib/aarch64-linux-gnu/libnvinfer.so
sym, /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so
sym, /usr/lib/aarch64-linux-gnu/libnvparsers.so
sym, /usr/lib/aarch64-linux-gnu/libnvonnxparser.so
lib, /usr/include/aarch64-linux-gnu/NvInfer.h
lib, /usr/include/aarch64-linux-gnu/NvInferRuntime.h
lib, /usr/include/aarch64-linux-gnu/NvInferRuntimeCommon.h
lib, /usr/include/aarch64-linux-gnu/NvInferVersion.h
lib, /usr/include/aarch64-linux-gnu/NvInferImpl.h
lib, /usr/include/aarch64-linux-gnu/NvInferLegacyDims.h
lib, /usr/include/aarch64-linux-gnu/NvUtils.h
lib, /usr/include/aarch64-linux-gnu/NvInferPlugin.h
lib, /usr/include/aarch64-linux-gnu/NvInferPluginUtils.h
lib, /usr/include/aarch64-linux-gnu/NvCaffeParser.h
lib, /usr/include/aarch64-linux-gnu/NvUffParser.h
lib, /usr/include/aarch64-linux-gnu/NvOnnxConfig.h
lib, /usr/include/aarch64-linux-gnu/NvOnnxParser.h
dir, /usr/lib/python3.6/dist-packages/tensorrt
dir, /usr/lib/python3.6/dist-packages/graphsurgeon
dir, /usr/lib/python3.6/dist-packages/uff
dir, /usr/lib/python3.6/dist-packages/onnx_graphsurgeon
dir, /usr/src/tensorrt

My setup is below:
• Hardware Platform (Jetson / GPU) : Jetson Xavier NX
• DeepStream Version : 6.0
• JetPack Version : Jetpack 4.6 & L4T 32.6.1
• TensorRT Version : 8.0.1.6-1+cuda10.2

The cuda libraries I have are also below:

Screenshot from 2022-07-21 16-43-31
Screenshot from 2022-07-21 16-43-311336×829 204 KB

3

Please add --runtime=nvidia in docker command, it will mount device TensorRT, CUDNN to container.

4

Hi @Amycao,

I ran the container with --runtime=nvidia and also adjusted /etc/docker/daemon.json to set default runtime to nvidia. Neither of them fixed the issue and only l4t.csv is being mounted.

5

/etc/nvidia-container-runtime/host-files-for-container.d

Did you manually create nvidia-container-runtime/host-files-for-container.d these directory under etc directory? i can’t find it under etc.

6

No, that directory should be automatically created when nvidia docker is installed. The documentation is here

7

Can you specify how you run docker container?

8

I am running it with the following:
sudo docker run -it --rm --net=host --runtime nvidia -e DISPLAY=$DISPLAY -w /opt/nvidia/deepstream/deepstream-6.0 -v /tmp/.X11-unix/:/tmp/.X11-unix nvcr.io/nvidia/deepstream-l4t:6.0-base

9

I used the similar command as you, but i can not find /etc/nvidia-container-runtime/host-files-for-container.d

10

Check your docker version.

apt list --installed | grep docker
apt list --installed | grep container

Since docker depends on the JetPack version, you must be careful not to update it with apt-get upgrade, etc.
Reference:

11

I had previously tried adjusting the docker version and that did not resolve the issue

12

Please write how to reproduce after flushing JetPack. (all commands you executed)

13

I was able to circumvent this by storing everything under the l4t.csv file, so I can work around the other files not being mounted

14

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.