FAIL to run deepstream 6-0 on T4 server

thuy.hoang19 · November 30, 2021, 3:31pm

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) NVIDIA Tesla T4
• DeepStream Version 6.0
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
NVIDIA-SMI 465.19.01 Driver Version: 465.19.01 CUDA Version: 11.3
**• Issue Type : Question

I tried to run deepstream 6.0 on Tesla T4 server following Docker Containers — DeepStream 6.1.1 Release documentation ( Creating custom DeepStream docker for dGPU using DeepStreamSDK package)

create docker with command : docker build -t deepstream:dgpu --build-arg TENSORRT_VERSION=“8.0.1-1+cuda11.3” --build-arg CUDNN_VERSION=“8.3.1.22-1+cuda11.5” --build-arg CUDA_VERSION=“11.4.2” .

run docker

run ./installl.sh
rm ${HOME}/.cache/gstreamer-1.0/registry.aarch64.bin
export GST_PLUGIN_PATH=“/usr/lib/x86_64-linux-gnu/gstreamer-1.0/”
But I always failed with below error:
ult = 1
0:00:00.012907426 101 0x5645fd63f2c0 INFO GST_INIT gst.c:807:init_post: GLib runtime version: 2.56.4
0:00:00.012913432 101 0x5645fd63f2c0 INFO GST_INIT gst.c:809:init_post: GLib headers version: 2.56.4
0:00:00.012917500 101 0x5645fd63f2c0 INFO GST_INIT gst.c:810:init_post: initialized GStreamer successfully
0:00:00.013128722 101 0x5645fd63f2c0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “pipeline” named “pipeline”
0:00:00.013200182 101 0x5645fd63f2c0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “bin” named “multi_src_bin”
0:00:00.013215267 101 0x5645fd63f2c0 WARN GST_ELEMENT_FACTORY gstelementfactory.c:456:gst_element_factory_make: no such element factory “nvstreammux”!
** ERROR: <create_multi_source_bin:1423>: Failed to create element ‘src_bin_muxer’
** ERROR: <create_multi_source_bin:1516>: create_multi_source_bin failed
** ERROR: <create_pipeline:1323>: create_pipeline failed
** ERROR: main:639: Failed to create pipeline

Please help me to fix it.
Thank you so much!

Amycao · December 1, 2021, 6:22am

Can you find this plugin?
gst-inspect-1.0 nvstreammux
also please update GST_PLUGIN_PATH
GST_PLUGIN_PATH=/usr/lib/x86_64-linux-gnu/gstreamer-1.0:/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream
remove the gstreamer cache, and try again.

thuy.hoang19 · December 1, 2021, 7:12am

Hello Amycao,

Firstly, I can not find gst-inspect-1.0 nvstreammux as following :
root@3796b2406e9f:/opt/nvidia/deepstream/deepstream-6.0# gst-inspect-1.0 nvstreammux
No such element or plugin ‘nvstreammux’

And I tried as you recommened but the same problem occured:
(gst-plugin-scanner:229): GStreamer-WARNING **: 06:59:53.355: Failed to load plugin ‘/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_deepstream_bins.so’: libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:229): GStreamer-WARNING **: 06:59:53.355: Failed to load plugin ‘/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_eglglessink.so’: libcuda.so.1: cannot open shared object file: No such file or directory

(gst-plugin-scanner:229): GStreamer-WARNING **: 06:59:53.384: Failed to load plugin ‘/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_tracker.so’: libcuda.so.1: cannot open shared object file: No such file or directory
(gst-plugin-scanner:229): GStreamer-WARNING **: 06:59:53.444: Failed to load plugin ‘/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_udp.so’: librivermax.so.0: cannot open shared object file: No such file or directory
0:00:00.338809120 228 0x55c10f6ef2c0 INFO GST_INIT gst.c:810:init_post: initialized GStreamer successfully
0:00:00.339060397 228 0x55c10f6ef2c0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “pipeline” named “pipeline”
0:00:00.339153145 228 0x55c10f6ef2c0 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “bin” named “multi_src_bin”
0:00:00.339169312 228 0x55c10f6ef2c0 WARN GST_ELEMENT_FACTORY gstelementfactory.c:456:gst_element_factory_make: no such element factory “nvstreammux”!
** ERROR: <create_multi_source_bin:1423>: Failed to create element ‘src_bin_muxer’
Is there anything that I can check more?
Thank you so much!
Thuy

Amycao · December 1, 2021, 8:02am

Can you find this file?
/usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1

thuy.hoang19 · December 2, 2021, 2:04am

@Amycao
I can not find /usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1
This is all in /usr/lib : ls /usr/lib/
X11 debug gnupg2 libarmadillo.so.8 liblilv-0.so.0 libpsm1 ogdi python3.6 systemd
apt dpkg gold-ld libarmadillo.so.8.400.0 liblilv-0.so.0.24.2 libvpf.so.3 os-release python3.7 tar
bfd-plugins gcc i386-linux-gnu libdfalt.so.0 libmfhdfalt.so.0 libvpf.so.3.2 pkg-config.multiarch python3.8 tcltk
cli git-core jni libdfalt.so.0.0.0 libmfhdfalt.so.0.0.0 locale pkgconfig rsyslog tmpfiles.d
compat-ld glib-networking libMonoPosixHelper.so libgdal.so.20 libogdi.so.3 mime python2.7 sasl2 valgrind
dconf gnupg libMonoSupportW.so libgdal.so.20.3.2 libogdi.so.3.2 mono python3

Thank you!

Amycao · December 2, 2021, 2:16am

Oh. you are on x86 platform, it is /usr/lib/x86_64-linux-gnu/libcuda.so
besides, can you find cuda, nvinfer, cudnn?
dpkg -l|grep cuda
dpkg -l|grep nvinfer

thuy.hoang19 · December 2, 2021, 2:22am

@Amycao ,
Thank for your quick response!
I can not find /usr/lib/x86_64-linux-gnu/libcuda.so
But I can fine cuda, nvinfer, cudnn :
cuda-cccl-11-4 11.4.122-1 amd64 CUDA CCCL
ii cuda-command-line-tools-11-4 11.4.2-1 amd64 CUDA command-line tools
ii cuda-compat-11-4 470.57.02-1 amd64 CUDA Compatibility Platform

libcudnn8 8.3.1.22-1+cuda11.5 amd64 cuDNN runtime libraries

ii libnvinfer8 8.0.1-1+cuda11.3 amd64 TensorRT runtime libraries

Thanks!
Thùy

Amycao · December 2, 2021, 2:44am

Did your host install cuda driver?

thuy.hoang19 · December 2, 2021, 2:52am

@Amycao

I can not find cuda ouside docker

sysadmin@tnp_svr_ai_01:~$ dpkg -l|grep cuda
sysadmin@tnp_svr_ai_01:~$

Amycao · December 2, 2021, 2:55am

You need to install cuda driver and run docker with arguments --gpus all added

thuy.hoang19 · December 2, 2021, 3:13am

@Amycao
After add more --gpus all when run docker , I can find /usr/lib/x86_64-linux-gnu/libcuda.so .

But the result is the same.

** ERROR: <create_multi_source_bin:1423>: Failed to create element ‘src_bin_muxer’
** ERROR: <create_multi_source_bin:1516>: create_multi_source_bin failed
** ERROR: <create_pipeline:1323>: create_pipeline failed

Amycao · December 2, 2021, 4:37am

Remove the cache, and try again.
rm .cache/gstreamer-1.0/ -rf

thuy.hoang19 · December 2, 2021, 6:51am

@Amycao i have already tried it. But the same error occured

Amycao · December 2, 2021, 8:40am

Did the file exist?
ll /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_multistream.so

thuy.hoang19 · December 2, 2021, 8:46am

@Amycao
Yes, It existed

Amycao · December 2, 2021, 9:03am

Did you do this as mentioned before GST_PLUGIN_PATH=/usr/lib/x86_64-linux-gnu/gstreamer-1.0:/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream

thuy.hoang19 · December 2, 2021, 9:20am

@Amycao
Yes, I did it. below are all steps that I did:

root@57153c90e92e:/opt/nvidia/deepstream/deepstream-6.0# rm -rf ${HOME}/.cache/gstreamer-1.0/
root@57153c90e92e:/opt/nvidia/deepstream/deepstream-6.0# export GST_PLUGIN_PATH=/usr/lib/x86_64-linux-gnu/gstreamer-1.0:/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream
root@57153c90e92e:/opt/nvidia/deepstream/deepstream-6.0# ldconfig
root@57153c90e92e:/opt/nvidia/deepstream/deepstream-6.0# bin/deepstream-app -c /workspace/20211130_T4_Human/deepstream_app_config_yoloV5.txt

Amycao · December 2, 2021, 9:24am

Add GST_DEBUG=3 before the app to get more log to analyze.

thuy.hoang19 · December 2, 2021, 9:29am

@Amycao
Thank for your support~
This is more log as you requested:

root@57153c90e92e:/opt/nvidia/deepstream/deepstream-6.0# bin/deepstream-app -c /workspace/20211130_T4_Human/deepstream_app_config_yoloV5.txt --gst-debug=4
0:00:00.000237710 728 0x559f885efac0 INFO GST_INIT gstmessage.c:127:_priv_gst_message_initialize: init messages
0:00:00.000587587 728 0x559f885efac0 INFO GST_INIT gstcontext.c:84:_priv_gst_context_initialize: init contexts
0:00:00.000736670 728 0x559f885efac0 INFO GST_PLUGIN_LOADING gstplugin.c:317:priv_gst_plugin_initialize: registering 0 static plugins
0:00:00.000793537 728 0x559f885efac0 INFO GST_PLUGIN_LOADING gstplugin.c:225:gst_plugin_register_static: registered static plugin “staticelements”
0:00:00.000803284 728 0x559f885efac0 INFO GST_PLUGIN_LOADING gstplugin.c:227:gst_plugin_register_static: added static plugin “staticelements”, result: 1
0:00:00.000813698 728 0x559f885efac0 INFO GST_REGISTRY gstregistry.c:1727:ensure_current_registry: reading registry cache: /root/.cache/gstreamer-1.0/registry.x86 64.bin
0:00:00.000854854 728 0x559f885efac0 INFO GST_REGISTRY gstregistrybinary.c:539:priv_gst_registry_binary_read_cache: Unable to mmap file /root/.cache/gstreamer-1.0/ registry.x86_64.bin : Failed to open file “/root/.cache/gstreamer-1.0/registry.x86_64.bin”: open() failed: No such file or directory
0:00:00.000868973 728 0x559f885efac0 INFO GST_REGISTRY gstregistrybinary.c:549:priv_gst_registry_binary_read_cache: Unable to read file /root/.cache/gstreamer-1.0/

0:00:02.143243521 728 0x559f885efac0 ERROR 0:00:02.147190917 728 0x559f885efac0 INFO 0:00:02.147212409 728 0x559f885efac0 INFO 0:00:02.164764169 728 0x559f885efac0 INFO 0:00:02.188081434 728 0x559f885efac0 INFO 0:00:02.188091532 728 0x559f885efac0 INFO 0:00:02.188097462 728 0x559f885efac0 INFO 0:00:02.188138016 728 0x559f885efac0 INFO 0:00:02.188143191 728 0x559f885efac0 INFO 0:00:02.188148320 728 0x559f885efac0 INFO 0:00:02.188411228 728 0x559f885efac0 INFO 0:00:02.188481102 728 0x559f885efac0 INFO 0:00:02.188498399 728 0x559f885efac0 WARN ** ERROR: <create_multi_source_bin:1423>: ** ERROR: <create_multi_source_bin:1516>: ** ERROR: <create_pipeline:1323>: ** ERROR: main:639: Failed Quitting
0:00:02.288707837 728 0x559f885efac0 INFO 0:00:03.188764790 728 0x559f885efac0 INFO 0:00:03.188815621 728 0x559f885efac0 INFO 0:00:03.188835291 728 0x559f885efac0 INFO 0:00:03.188857823 728 0x559f885efac0 INFO App run failed
0:00:03.188970191 728 0x559f885efac0 INFO 0:00:03.191345921 728 0x559f885efac0 INFO root@57153c90e92e:/opt/nvidia/deepstream/de GST_PLUGIN_LOADING gstpluginloader.c:277:plugin_loader_replay_pending: Plugin file /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deep stream/libgstnvvideoconvert.so failed to load. Blacklisting
GST_REGISTRY gstregistry.c:1694:scan_and_update_registry: Registry cache changed. Writing new registry cache
GST_REGISTRY gstregistrybinary.c:369:priv_gst_registry_binary_write_cache: Building binary registry cache image
GST_REGISTRY gstregistrybinary.c:401:priv_gst_registry_binary_write_cache: Writing binary registry cache
GST_REGISTRY gstregistrybinary.c:262:gst_registry_binary_cache_finish: Wrote binary registry cache
GST_REGISTRY gstregistry.c:1703:scan_and_update_registry: Registry cache written successfully
GST_REGISTRY gstregistry.c:1762:ensure_current_registry: registry reading and updating done, result = 1
GST_INIT gst.c:807:init_post: GLib runtime version: 2.56.4
GST_INIT gst.c:809:init_post: GLib headers version: 2.56.4
GST_INIT gst.c:810:init_post: initialized GStreamer successfully
GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “pipeline” named “pipeline”
GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “bin” named “multi_src_bin”
GST_ELEMENT_FACTORY gstelementfactory.c:456:gst_element_factory_make: no such element factory “nvstreammux”!
Failed to create element ‘src_bin_muxer’
create_multi_source_bin failed
create_pipeline failed
to create pipeline
GST_STATES gstelement.c:2676:gst_element_continue_state: completed state change to NULL
GST_REFCOUNTING gstelement.c:3251:gst_element_dispose: 0x559f889ee1d0 dispose
GST_REFCOUNTING gstelement.c:3297:gst_element_dispose: 0x559f889ee1d0 parent class dispose
GST_REFCOUNTING gstelement.c:3328:gst_element_finalize: 0x559f889ee1d0 finalize
GST_REFCOUNTING gstelement.c:3334:gst_element_finalize: 0x559f889ee1d0 finalize parent
GST_INIT gst.c:1083:gst_deinit: deinitializing GStreamer
GST_INIT gst.c:1231:gst_deinit: deinitialized GStreamer
epstream-6.0#

Amycao · December 2, 2021, 9:33am

How about result of ldd /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_multistream.so