Running a DeepStream pipeline in Nvidia DGX station A100 (With Docker)

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)

DGX Station A100

• DeepStream Version

6.0.0

• JetPack Version (valid for Jetson only)
NA
• TensorRT Version
NA
• NVIDIA GPU Driver Version (valid for GPU only)
NA
• Issue Type( questions, new requirements, bugs)
questions
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

I used these Dockerfile and the docker-compose.yaml file and ran docker-compose up on my DGX station

then docker exec -it docker-test /bin/bash --login to open a shell

then ran gst-inspect-1.0 -b

I got

(gst-plugin-scanner:117): GStreamer-WARNING **: 15:56:19.861: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_multistreamtiler.so': libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:117): GStreamer-WARNING **: 15:56:19.864: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_infer.so': libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:117): GStreamer-WARNING **: 15:56:19.864: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_eglglessink.so': libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:117): GStreamer-WARNING **: 15:56:19.864: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_nvblender.so': libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:117): GStreamer-WARNING **: 15:56:19.865: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_dsexample.so': libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:117): GStreamer-WARNING **: 15:56:19.878: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_tracker.so': libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:117): GStreamer-WARNING **: 15:56:19.878: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libgstnvvideoconvert.so': libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:117): GStreamer-WARNING **: 15:56:19.881: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_deepstream_bins.so': libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:117): GStreamer-WARNING **: 15:56:19.881: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_preprocess.so': libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:117): GStreamer-WARNING **: 15:56:19.881: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libcustom2d_preprocess.so': libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:117): GStreamer-WARNING **: 15:56:19.884: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_multistream.so': libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:117): GStreamer-WARNING **: 15:56:19.885: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_udp.so': librivermax.so.0: cannot open shared object file: No such file or directory

(gst-plugin-scanner:117): GStreamer-WARNING **: 15:56:19.887: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_dewarper.so': libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:117): GStreamer-WARNING **: 15:56:19.888: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so': libtritonserver.so: cannot open shared object file: No such file or directory
Blacklisted files:
  libnvdsgst_inferserver.so
  libnvdsgst_dewarper.so
  libnvdsgst_udp.so
  libnvdsgst_multistream.so
  libcustom2d_preprocess.so
  libnvdsgst_preprocess.so
  libnvdsgst_deepstream_bins.so
  libgstnvvideoconvert.so
  libnvdsgst_tracker.so
  libnvdsgst_dsexample.so
  libnvdsgst_nvblender.so
  libnvdsgst_eglglessink.so
  libnvdsgst_infer.so
  libnvdsgst_multistreamtiler.so

Total count: 14 blacklisted files

• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)
NA

Observations:

the output of nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Aug_15_21:14:11_PDT_2021
Cuda compilation tools, release 11.4, V11.4.120
Build cuda_11.4.r11.4/compiler.30300941_0
root@docker:/home/nvidia#

I’m fairly new to this so If I understand correctly cuda is there (because nvcc is returning stuff) but clearly libcuda.so.1, librivermax.so.0 and libtritonserver.so I think I could get away without the other to but I think I can’t be without libcuda.so.

Can you please give me some guidance on this.

Cheers,
Ganindu.

[UPDATE]

I did

  • [step 1] `ldd /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/<plugin_name>.so’
    then
  • [step 2] LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/compat to locate paths for libcuda.so libcuda.so.1
    and then
  • [step 3] ldconfig to resolve

and again step 1 to check. and the libcuda.so dependency was solved

Afterwards I noticed that nvstreammux was missing, but then I noticed that in /opt/nvidia/deepstream/deepstream/sources/apps/apps-common directory the sources were there for the streammux and attempted to build deepstream-test1 (which built successfully) and then after cd in to that directory, tried running ./deepstream-test1-app ../../../../samples/streams/sample_720p.h264

and I got the following error

nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 

(gst-plugin-scanner:11519): GStreamer-WARNING **: 12:54:32.457: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_udp.so': librivermax.so.0: cannot open shared object file: No such file or directory
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 

(gst-plugin-scanner:11521): GStreamer-WARNING **: 12:54:32.479: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so': libtritonserver.so: cannot open shared object file: No such file or directory
One element could not be created. Exiting.

I believe this should be a straightforward thing to do because there is no customisation involved.

but I understand this might be a unique case to us because we have a non standard setup (our AGX xavier is sending a RTSP stream on the same local network the dgx is on). We want to run that (receive that stream to begin with) in the deepstream container for prototyping and development.

as the common use case we assume that the DGX will be used to create the models which then get pruned and converted into target engine files that run on edge devices.

At this point we are just seeing what are the possibilities to finally end up with a more standard development pipeline and a deployment pipeline.

Cheers,
Ganindu.

3

I fixed the problem myself, The reason was I was using the incorrect docker-compose version, as of 8/12/2021 It seems 3.5 is not supported and 2.3 is supported. In addition I pulled some env variables from the docker file to the compose file. (not sure if I got the syntax right but it seems to recognise the drivers )

version: '2.3'

services:
  inference:
    build:
      context: .
      dockerfile: Dockerfile
    image: test_pytorch_deepstream:0.1
    stdin_open: true # docker run -i
    container_name: docker-test-deepstream
    runtime: nvidia
    hostname: docker.dgx

    environment:
      - DISPLAY=${DISPLAY}
      - ROOTDIT=RDR
      - NVIDIA_VISIBLE_DEVICES=${NVIDIA_VISIBLE_DEVICES:-all}
      - NVIDIA_DRIVER_CAPABILITIES=${NVIDIA_DRIVER_CAPABILITIES:-all}

    volumes:
      - type: bind
        source: /tmp/.X11-unix
        target: /tmp/.X11-unix
      - type: bind
        source: $HOME/.Xauthority
        target: /root/.Xauthority
      - ./dgx-test/:/home/nvidia/dgx-test
     
    network_mode: "host"
5

Glad to know you fixed it. Thanks

1 Like
6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.