Failed to run deepstream-test5-app in docker

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• GPU
• DeepStream Version: 5.0.1
• TensorRT Version
• NVIDIA GPU Driver Version: 440.118.02
• Issue Type: bugs
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
I run container with podman on Centos 8: docker run -it -e NVIDIA_VISIBLE_DEVICES=all --rm -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY -w /root 3d07ed6d33ea

The docker image is nvcr.io/nvidia/deepstream:5.0.1-20.09-iot
And there is nvidia-smi result:

image
image733×288 7.02 KB

Then I created my own config file like this:

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=0
rows=2
columns=2
width=1280
height=720
gpu-id=0
#(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
#(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory, applicable for Tesla
#(2): nvbuf-mem-cuda-device - Allocate Device cuda memory, applicable for Tesla
#(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory, applicable for Tesla
#(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
nvbuf-memory-type=0


[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=3
uri=file://../../../../../samples/streams/sample_1080p_h264.mp4
num-sources=2
gpu-id=0
nvbuf-memory-type=0

[source1]
enable=0
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=3
uri=file://../../../../../samples/streams/sample_1080p_h264.mp4
num-sources=2
gpu-id=0
nvbuf-memory-type=0

[sink0]
enable=0
#Type - 1=FakeSink 2=EglSink 3=File
type=2
sync=1
source-id=0
gpu-id=0
nvbuf-memory-type=0

[sink1]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=UDPSink 5=nvoverlaysink 6=MsgConvBroker
type=6
msg-conv-config=dstest5_msgconv_sample_config.txt
#(0): PAYLOAD_DEEPSTREAM - Deepstream schema payload
#(1): PAYLOAD_DEEPSTREAM_MINIMAL - Deepstream schema payload minimal
#(256): PAYLOAD_RESERVED - Reserved type
#(257): PAYLOAD_CUSTOM   - Custom schema payload
msg-conv-payload-type=0
msg-broker-proto-lib=/opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_kafka_proto.so
#Provide your msg-broker-conn-str here
msg-broker-conn-str = 192.168.1.71;8895;VideoEvent
topic = VideoEvent
#Optional:
#msg-broker-config=../../deepstream-test4/cfg_kafka.txt

[sink2]
enable=1
type=3
#1=mp4 2=mkv
container=1
#1=h264 2=h265 3=mpeg4
## only SW mpeg4 is supported right now.
codec=3
sync=1
bitrate=2000000
output-file=out.mp4
source-id=0

# sink type = 6 by default creates msg converter + broker.
# To use multiple brokers use this group for converter and use
# sink type = 6 with disable-msgconv = 1
[message-converter]
enable=0
msg-conv-config=dstest5_msgconv_sample_config.txt
#(0): PAYLOAD_DEEPSTREAM - Deepstream schema payload
#(1): PAYLOAD_DEEPSTREAM_MINIMAL - Deepstream schema payload minimal
#(256): PAYLOAD_RESERVED - Reserved type
#(257): PAYLOAD_CUSTOM   - Custom schema payload
msg-conv-payload-type=0
# Name of library having custom implementation.
#msg-conv-msg2p-lib=<val>
# Id of component in case only selected message to parse.
#msg-conv-comp-id=<val>

# Configure this group to enable cloud message consumer.
[message-consumer0]
enable=0
proto-lib=/opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_kafka_proto.so
conn-str=<host>;<port>
config-file=<broker config file e.g. cfg_kafka.txt>
subscribe-topic-list=<topic1>;<topic2>;<topicN>
# Use this option if message has sensor name as id instead of index (0,1,2 etc.).
#sensor-list-file=dstest5_msgconv_sample_config.txt

[osd]
enable=0
gpu-id=0
border-width=1
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Arial
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=0
batch-size=4
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
## Set muxer output width and height
width=1920
height=1080
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0
## If set to TRUE, system timestamp will be attached as ntp timestamp
## If set to FALSE, ntp timestamp from rtspsrc, if available, will be attached
# attach-sys-ts-as-ntp=1

[primary-gie]
enable=1
gpu-id=0
batch-size=4
## 0=FP32, 1=INT8, 2=FP16 mode
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;1;1;1
bbox-border-color3=0;1;0;1
nvbuf-memory-type=0
interval=0
gie-unique-id=1
model-engine-file=../../../../../samples/models/Primary_Detector/resnet10.caffemodel_b4_gpu0_int8.engine
labelfile-path=../../../../../samples/models/Primary_Detector/labels.txt
config-file=../../../../../samples/configs/deepstream-app/config_infer_primary.txt
#infer-raw-output-dir=../../../../../samples/primary_detector_raw_output/

[tracker]
enable=0
tracker-width=600
tracker-height=288
ll-lib-file=/opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_mot_klt.so
#ll-config-file required for DCF/IOU only
#ll-config-file=tracker_config.yml
#ll-config-file=iou_config.txt
gpu-id=0
#enable-batch-process applicable to DCF only
enable-batch-process=0

[tests]
file-loop=1

And I ran “deepstream-test5-app -c myconfig.txt” but failed:

0:00:00.058190398   764 0x55ff7ca80e70 WARN                  ladspa gstladspa.c:507:plugin_init:<plugin112> no LADSPA plugins found, check LADSPA_PATH
0:00:00.146512746   764 0x55ff7ca80e70 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x120000: 'AVR (Audio Visual Research)' is not mapped
0:00:00.146529845   764 0x55ff7ca80e70 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x180000: 'CAF (Apple Core Audio File)' is not mapped
0:00:00.146534485   764 0x55ff7ca80e70 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x100000: 'HTK (HMM Tool Kit)' is not mapped
0:00:00.146539505   764 0x55ff7ca80e70 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0xc0000: 'MAT4 (GNU Octave 2.0 / Matlab 4.2)' is not mapped
0:00:00.146543597   764 0x55ff7ca80e70 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0xd0000: 'MAT5 (GNU Octave 2.1 / Matlab 5.0)' is not mapped
0:00:00.146547450   764 0x55ff7ca80e70 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x210000: 'MPC (Akai MPC 2k)' is not mapped
0:00:00.146552053   764 0x55ff7ca80e70 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0xe0000: 'PVF (Portable Voice Format)' is not mapped
0:00:00.146559546   764 0x55ff7ca80e70 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x160000: 'SD2 (Sound Designer II)' is not mapped
0:00:00.146583836   764 0x55ff7ca80e70 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x190000: 'WVE (Psion Series 3)' is not mapped
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
0:00:00.205261940   763 0x5601bbb18ec0 ERROR     GST_PLUGIN_LOADING gstpluginloader.c:277:plugin_loader_replay_pending: Plugin file /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libgstnvvideoconvert.so failed to load. Blacklisting
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
0:00:00.226651556   763 0x5601bbb18ec0 ERROR     GST_PLUGIN_LOADING gstpluginloader.c:277:plugin_loader_replay_pending: Plugin file /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_dewarper.so failed to load. Blacklisting
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
0:00:00.248301753   763 0x5601bbb18ec0 ERROR     GST_PLUGIN_LOADING gstpluginloader.c:277:plugin_loader_replay_pending: Plugin file /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_dsexample.so failed to load. Blacklisting
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
0:00:00.314341884   763 0x5601bbb18ec0 ERROR     GST_PLUGIN_LOADING gstpluginloader.c:277:plugin_loader_replay_pending: Plugin file /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_infer.so failed to load. Blacklisting
0:00:00.001373818   772 0x559753eeb490 WARN      GST_PLUGIN_LOADING gstplugin.c:792:_priv_gst_plugin_load_file_for_registry: module_open failed: libtrtserver.so: cannot open shared object file: No such file or directory

(gst-plugin-scanner:772): GStreamer-WARNING **: 06:42:53.232: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so': libtrtserver.so: cannot open shared object file: No such file or directory
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
0:00:00.340591787   763 0x5601bbb18ec0 ERROR     GST_PLUGIN_LOADING gstpluginloader.c:277:plugin_loader_replay_pending: Plugin file /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_multistream.so failed to load. Blacklisting
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
0:00:00.357869368   763 0x5601bbb18ec0 ERROR     GST_PLUGIN_LOADING gstpluginloader.c:277:plugin_loader_replay_pending: Plugin file /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_multistreamtiler.so failed to load. Blacklisting
nvbufsurftransform:cuInit failed : 100 
nvbufsurftransform:cuInit failed : 100 
0:00:00.396251363   763 0x5601bbb18ec0 ERROR     GST_PLUGIN_LOADING gstpluginloader.c:277:plugin_loader_replay_pending: Plugin file /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_tracker.so failed to load. Blacklisting
0:00:00.463107452   763 0x5601bbb18ec0 WARN     GST_ELEMENT_FACTORY gstelementfactory.c:456:gst_element_factory_make: no such element factory "nvstreammux"!
** ERROR: <create_multi_source_bin:1057>: Failed to create element 'src_bin_muxer'
** ERROR: <create_multi_source_bin:1132>: create_multi_source_bin failed
** ERROR: <create_pipeline:1296>: create_pipeline failed
** ERROR: <main:1419>: Failed to create pipeline
Quitting
App run failed

Then I ran “gst-inspect-1.0 libgstnvvideoconvert.so” and got result:

Plugin Details:
  Name                     libgstnvvideoconvert.so
  Description              Plugin for blacklisted file
  Filename                 /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libgstnvvideoconvert.so
  Version                  0.0.0
  License                  BLACKLIST
  Source module            BLACKLIST
  Binary package           BLACKLIST
  Origin URL               BLACKLIST


  0 features:

I have no idea to solve it…

3

nvbufsurftransform:cuInit failed : 100
cudaErrorNoDevice = 100

This indicates that no CUDA-capable devices were detected by the installed CUDA driver.

Can you run cuda sample like /usr/local/cuda/samples/1_Utilities/deviceQuery

6

The following is the result:

root@b323182d70f2:/usr/local/cuda/samples/1_Utilities/deviceQuery# ./deviceQuery
./deviceQuery Starting…

CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 100
→ no CUDA-capable device is detected
Result = FAIL

You are right. But what can I do in docker?

7

can deviceQuery run outside docker?

8

It works outside docker.
[root@localhost deviceQuery]# ./deviceQuery

./deviceQuery Starting...
 CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)

Device 0: "Tesla T4"
  CUDA Driver Version / Runtime Version          10.2 / 10.2
  CUDA Capability Major/Minor version number:    7.5
  Total amount of global memory:                 15110 MBytes (15843721216 bytes)
  (40) Multiprocessors, ( 64) CUDA Cores/MP:     2560 CUDA Cores
  GPU Max Clock rate:                            1590 MHz (1.59 GHz)
  Memory Clock rate:                             5001 Mhz
  Memory Bus Width:                              256-bit
  L2 Cache Size:                                 4194304 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  1024
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 3 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 4 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 10.2, CUDA Runtime Version = 10.2, NumDevs = 1
Result = PASS
9

Try to run with
nvidia-docker run -it --rm -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY -w /root 3d07ed6d33ea

10

Actually, I have no nvidia-docker but nvidia-container-toolkit in my centos 8.3. And the contaniner is podman. I’ll try to use docker-ce and nvida-docker2.

11

@Amycao Thanks for your help. It works now. I use docker-ce and nvidia-docker2 to run container now.