Unable to run gst-inspect inside deepstream container

nick.yonts · December 23, 2023, 12:53am

When launching any of the deepstream containers without the container toolkit I’m able to run gst-inspect-1.0 and get output as expected. When I try to run the same container with “–runtime nvidia” gst-inspect-1.0 just hangs. I’ve only noticed this behavior when running on an Ubuntu EC2 instance. I’m not sure what steps to take to debug or resolve this issue so any help would be appreciated.

**• Ubuntu EC2 **
• DeepStream Version: All
• NVIDIA GPU Driver Version (valid for GPU only): 530

yingliu · December 23, 2023, 3:46am

Does your EC2 have GPU support?
You can refer to the docker command in page DeepStream | NVIDIA NGC, the command should work if the GPU driver is installed correctly.

nick.yonts · December 24, 2023, 2:20pm

Here is the output of nvidia-smi:

nvidia-smi
Sat Dec 23 12:00:30 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.105.01   Driver Version: 515.105.01   CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:00:1E.0 Off |                    0 |
| N/A   26C    P8    13W /  70W |     27MiB / 15360MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2463      C   nvidia-cuda-mps-server             23MiB |
+-----------------------------------------------------------------------------+

fanzh · December 24, 2023, 3:13pm

what do you mean about “without the container toolkit”?
could you share docker start command-line and result of gst-inspect-1.0 if it did not hang.
if gst-inspect-1.0 will hang, could you share more logs? please do the following setting first,
export GST_DEBUG=6
then execute gst-inspect-1.0 >1.log 2>1.log, then use “ctrl+c” to end, then share 1.log.

nick.yonts · December 25, 2023, 2:51am

1-2) I guess I mean without the runtime: The command I’m running to get it to NOT hang is: docker run --rm -it --entrypoint bash nvcr.io/nvidia/deepstream:6.1.1-devel

To get it to hang just add --runtime nvidia to the above command

1.log is attached
1.log (10.6 MB)

fanzh · December 25, 2023, 2:59am

thanks for the sharing! we suggest using the “Run the container:” method in this link to start docker.

nick.yonts · December 26, 2023, 10:28am

I’m still getting the same result of gst-inspect-1.0 not running. The same applies to gst-launch as well

docker run --gpus all -it --rm --net=host --privileged -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY nvcr.io/nvidia/deepstream:6.1.1-devel :/opt/nvidia/deepstream/deepstream-6.1# gst-inspect-1.0 (gst-plugin-scanner:10): GStreamer-WARNING **: 10:26:48.706: Failed to load plugin ‘/usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstchromaprint.so’: libavcodec.so.58: cannot open shared object file: No such file or directory

fanzh · December 26, 2023, 2:44pm

I can’t reproduce this issue on T4 with ubuntu20.04 and DS6.2. here is the log.12.26.txt (1.9 KB)
could you share the result “ldd /opt/nvidia/deepstream/deepstream/lib/gst-plugins/libnvdsgst_infer.so” and “gst-inspect-1.0 nvinfer”?
in issue description, when starting docker without “–runtime nvidia”, gst-inspect did not hang, but in the last comment, when starting docker without “–runtime nvidia”, gst-inspect hung. do you mean sometimes it hung and sometimes it did not?

nick.yonts · December 26, 2023, 2:54pm

root@8458c0b71853:/opt/nvidia/deepstream/deepstream-6.1# ldd /opt/nvidia/deepstream/deepstream/lib/gst-plugins/libnvdsgst_infer.so linux-vdso.so.1 (0x00007fff9ab0c000) libcudart.so.11.0 => /usr/local/cuda-11.7/lib64/libcudart.so.11.0 (0x00007fb7f43e4000) libcuda.so.1 => /usr/lib/x86_64-linux-gnu/libcuda.so.1 (0x00007fb7f2f69000) libnvds_infer.so => ///opt/nvidia/deepstream/deepstream-6.1/lib/libnvds_infer.so (0x00007fb7f2ec3000) libnvds_meta.so => ///opt/nvidia/deepstream/deepstream-6.1/lib/libnvds_meta.so (0x00007fb7f2eb8000) libnvdsgst_meta.so => ///opt/nvidia/deepstream/deepstream-6.1/lib/libnvdsgst_meta.so (0x00007fb7f2eb1000) libnvdsgst_helper.so => ///opt/nvidia/deepstream/deepstream-6.1/lib/libnvdsgst_helper.so (0x00007fb7f2ea5000) libnvbufsurftransform.so => ///opt/nvidia/deepstream/deepstream-6.1/lib/libnvbufsurftransform.so (0x00007fb7eeb40000) libnvbufsurface.so => ///opt/nvidia/deepstream/deepstream-6.1/lib/libnvbufsurface.so (0x00007fb7eeb37000) libyaml-cpp.so.0.6 => /usr/lib/x86_64-linux-gnu/libyaml-cpp.so.0.6 (0x00007fb7eeadc000) libdl.so.2 => /usr/lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb7eead6000) libpthread.so.0 => /usr/lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb7eeab3000) libgstreamer-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 (0x00007fb7ee96a000) libgstbase-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0 (0x00007fb7ee8ec000) libglib-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007fb7ee7c3000) libgobject-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0 (0x00007fb7ee763000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fb7ee581000) libgcc_s.so.1 => /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fb7ee566000) libc.so.6 => /usr/lib/x86_64-linux-gnu/libc.so.6 (0x00007fb7ee372000) /lib64/ld-linux-x86-64.so.2 (0x00007fb7f46d3000) librt.so.1 => /usr/lib/x86_64-linux-gnu/librt.so.1 (0x00007fb7ee368000) libm.so.6 => /usr/lib/x86_64-linux-gnu/libm.so.6 (0x00007fb7ee219000) libnvds_inferlogger.so => ///opt/nvidia/deepstream/deepstream-6.1/lib/libnvds_inferlogger.so (0x00007fb7ee212000) libnvds_inferutils.so => ///opt/nvidia/deepstream/deepstream-6.1/lib/libnvds_inferutils.so (0x00007fb7ee1ec000) libnvinfer.so.8 => /usr/lib/x86_64-linux-gnu/libnvinfer.so.8 (0x00007fb7d4a34000) libnvinfer_plugin.so.8 => /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.8 (0x00007fb7d245e000) libnvonnxparser.so.8 => /usr/lib/x86_64-linux-gnu/libnvonnxparser.so.8 (0x00007fb7d1fb1000) libnvparsers.so.8 => /usr/lib/x86_64-linux-gnu/libnvparsers.so.8 (0x00007fb7d1a6b000) libgstrtp-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libgstrtp-1.0.so.0 (0x00007fb7d1a45000) libnppc.so.11 => /usr/local/cuda-11.7/lib64/libnppc.so.11 (0x00007fb7d16b7000) libnppig.so.11 => /usr/local/cuda-11.7/lib64/libnppig.so.11 (0x00007fb7cf5ee000) libnppidei.so.11 => /usr/local/cuda-11.7/lib64/libnppidei.so.11 (0x00007fb7cea91000) libnvbuf_fdmap.so => ///opt/nvidia/deepstream/deepstream-6.1/lib/libnvbuf_fdmap.so (0x00007fb7cea8b000) libgmodule-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgmodule-2.0.so.0 (0x00007fb7cea85000) libpcre.so.3 => /usr/lib/x86_64-linux-gnu/libpcre.so.3 (0x00007fb7cea12000) libffi.so.7 => /usr/lib/x86_64-linux-gnu/libffi.so.7 (0x00007fb7cea04000) libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007fb7ce72e000) libcublas.so.11 => /usr/local/cuda/targets/x86_64-linux/lib/libcublas.so.11 (0x00007fb7c54d0000) libcublasLt.so.11 => /usr/local/cuda/targets/x86_64-linux/lib/libcublasLt.so.11 (0x00007fb7b152f000) libcudnn.so.8 => /usr/lib/x86_64-linux-gnu/libcudnn.so.8 (0x00007fb7b1309000)

root@8458c0b71853:/opt/nvidia/deepstream/deepstream-6.1# gst-inspect-1.0 nvinfer (gst-plugin-scanner:28): GStreamer-WARNING **: 14:49:52.642: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstchromaprint.so': libavcodec.so.58: cannot open shared object file: No such file or directory

Docker command ran to obtain the results: docker run --rm -it --entrypoint bash --runtime nvidia -v .:/working nvcr.io/nvidia/deepstream:6.1.1-devel

No it’s handing at that point. See the output of gst-inspect in 2. That command was using the --gpu. When running: docker run --rm -it --entrypoint bash -v .:/working nvcr.io/nvidia/deepstream:6.1.1-devel I get ```root@bbe5794800d0:/opt/nvidia/deepstream/deepstream-6.1# gst-inspect-1.0 nvinfer

(gst-plugin-scanner:9): GStreamer-WARNING **: 14:53:21.436: Failed to load plugin ‘/usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstchromaprint.so’: libavcodec.so.58: cannot open shared object file: No such file or directory

(gst-plugin-scanner:9): GStreamer-WARNING **: 14:53:21.554: Failed to load plugin ‘/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_eglglessink.so’: libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:9): GStreamer-WARNING **: 14:53:21.624: Failed to load plugin ‘/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_preprocess.so’: libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:9): GStreamer-WARNING **: 14:53:21.631: Failed to load plugin ‘/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_udp.so’: librivermax.so.0: cannot open shared object file: No such file or directory

(gst-plugin-scanner:9): GStreamer-WARNING **: 14:53:21.667: Failed to load plugin ‘/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_infer.so’: libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:9): GStreamer-WARNING **: 14:53:21.671: Failed to load plugin ‘/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_multistreamtiler.so’: libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:9): GStreamer-WARNING **: 14:53:21.701: Failed to load plugin ‘/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_deepstream_bins.so’: libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:9): GStreamer-WARNING **: 14:53:21.702: Failed to load plugin ‘/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_tracker.so’: libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:9): GStreamer-WARNING **: 14:53:21.712: Failed to load plugin ‘/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so’: libtritonserver.so: cannot open shared object file: No such file or directory

(gst-plugin-scanner:9): GStreamer-WARNING **: 14:53:21.715: Failed to load plugin ‘/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_dewarper.so’: libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:9): GStreamer-WARNING **: 14:53:21.717: Failed to load plugin ‘/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libcustom2d_preprocess.so’: libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:9): GStreamer-WARNING **: 14:53:21.726: Failed to load plugin ‘/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_dsexample.so’: libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:9): GStreamer-WARNING **: 14:53:21.728: Failed to load plugin ‘/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_nvblender.so’: libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:9): GStreamer-WARNING **: 14:53:21.729: Failed to load plugin ‘/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libgstnvvideoconvert.so’: libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:9): GStreamer-WARNING **: 14:53:21.733: Failed to load plugin ‘/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_multistream.so’: libcuda.so.1: cannot open shared object file: No such file or directory
No such element or plugin ‘nvinfer’```

fanzh · December 27, 2023, 6:07am

could you share the result of “docker -v”? as the doc said “We recommend using Docker 20.10.13 along with the latest nvidia-container-toolkit”.

nick.yonts · December 27, 2023, 5:18pm

# docker -v
Docker version 24.0.2, build cb74dfc

# dpkg -l | grep nvidia-container
ii  libnvidia-container-tools                                   1.14.3-1                               amd64        NVIDIA container runtime library (command-line tools)
ii  libnvidia-container1:amd64                                  1.14.3-1                               amd64        NVIDIA container runtime library
ii  nvidia-container-runtime                                    3.13.0-1                               all          NVIDIA container runtime
ii  nvidia-container-toolkit                                    1.14.3-1                               amd64        NVIDIA Container toolkit
ii  nvidia-container-toolkit-base                               1.14.3-1                               amd64        NVIDIA Container Toolkit Base

fanzh · December 28, 2023, 2:33am

There is no update from you for a period, assuming this is not an issue any more. Hence we are closing this topic. If need further support, please open a new one. Thanks.
if using new docker, you need to use "–gpu " instead of “–runtime nvidia”. here is my test result
1228.txt (1.8 KB), using docker 24, ds6.1.1 on dgpu.

from the logs “libcuda.so.1: cannot open shared object file: No such file or directory”, libcuda.so can’t be found in docker container. it is abnormal. could you share the result of "find / -name “libcuda.so*” ", here is my test result check.txt (452 Bytes).
if there is libcuda.so, please do “rm -rf ~/.cache/gstreamer-1.0/”, then try again “gst-inspect-1.0 nvinfer”.
if still print “No such element or plugin ‘nvinfer’”, please share the result of “ldd /opt/nvidia/deepstream/deepstream/lib/libnvds_infer.so”, wondering if some libs are still not found.

system · January 11, 2024, 2:33am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
deepstream-test1-app running forever DeepStream SDK	7	2134	October 12, 2021
Libnvinfer.so.8: cannot open shared object file: No such file or directory DeepStream SDK	8	7839	July 11, 2022
What version of deepstream docker should I install forNVIDIA-SMI 470.161.03 x86 DeepStream SDK ubuntu , deepstream	9	45	December 5, 2024
How to set up Deepstream 6.3 in a docker container DeepStream SDK gstreamer , docker , deepstream	6	1148	January 2, 2024
Container deepstream-l4t:6.1-triton Not working Jetson Xavier NX? DeepStream SDK	5	414	April 25, 2023
Deep stream - Gstreamer NV Plugins DeepStream SDK	18	599	September 11, 2023
Deepstream-app: error while loading shared libraries: libnvinfer.so.6: cannot open shared object file + ubuntu 18.04 DeepStream SDK	27	8416	October 12, 2021
DeepStream samples fail in fresh docker-container on centos 7.9 host system: Device is in streaming mode DeepStream SDK	15	540	October 27, 2022
Deepstream 6.2 on desktop x86-64 - cannot find nvinfer DeepStream SDK	2	378	July 16, 2023
Not able to run sample applications in docker. "One element could not be created. Exiting." DeepStream SDK	10	2289	June 2, 2023

Unable to run gst-inspect inside deepstream container

Related topics