Deepstream 6.0 cannot run a pipeline on Tesla T4 with official docker

[user29329]

• Hardware Platform (Jetson / GPU) Tesla T4
• DeepStream Version 6.0.1
• TensorRT Version 8.0.0.1+cuda11.3 in docker（host computer 8.2.2.1+cuda11.4）
• NVIDIA GPU Driver Version (valid for GPU only) 470.63.1
• Issue Type( questions, new requirements, bugs) pipeline not running
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

deepstream_python_app

deepstream-app -c

• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

I want to migrate my deepstream application based on the official docker image from an RTX3060 x86 ubuntu18.04 server to a Tesla T4 x86 ubuntu18.04 server, on the former computer the application works perfectly, but on the latter computer deepsteam does not run the pipeline. All the configuration information and running conditions are in the image above for your reference.

[fanzh]

1. which deepstream docker version are you using?
2. can you run deepstream official sample on the docker?

[user29329]

1.docker pull nvcr.io/nvidia/deepstream:6.0-samples
2.The official sample successfully run with X86 RTX3060，but it failed on Tesla T4.

[fanzh]

1. can you run the official sample deepstream-app -c source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8-tet.txt？ the only one modification is “disable sink0, enable sink1”.
2. did you modify the code and configuration file, if yes, please share the code diff and cfg file.

[user29329]

@fanzh

did you modify the code and configuration file, if yes, please share the code diff and cfg file.

1. I did not see this file source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8-tet.txt in /opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app.

2. I did not change code and model files.

[fanzh]

some corrections, 1. can you run the official sample deepstream-app -c source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt？ the only one modification is “disable sink0, enable sink1”.
2. can you share that source2_1080p_dec_infer-resnet_demux_int8.txt?

[user29329]

source2_1080p_dec_infer-resnet_demux_int8.txt (4.4 KB)

[fanzh]

1. will the application end? is it stuck?
2. could you share more logs? please do “export GST_DEBUG=6”, modify source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt, the whole modification are “num-sources=1, disable sink0, enable sink1” then run deepstream-app -c source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt >1.log 2>1.log for about 30 seconds.
3. from the picture, the official sample run failed. can you run the simplest sample deepstream_test1, please replace nveglglessink with fakesink in deepstream_test1_app.c , then execute " export CUDA_VER=11.4&& make" and ./deepstream-test1-app /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264

[user29329]

1.Yes, it is stuck.

1.log (12.6 KB)

3,Sorry, I don’t know how to get access to this C project.

[fanzh]

To narrow down this issue, could you do the following simple tests, thanks.

1. test decoding: execute the command: gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264 ! h264parse ! nvv4l2decoder ! fakesink
   is it stuck?
2. test encoding: execute the command: gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvideoconvert ! ‘video/x-raw(memory:NVMM),format=I420’ ! nvv4l2h264enc bitrate=1000000 ! filesink
   is it stuck?
3. there is no debug log in 1.log, please
   do “export GST_DEBUG=6” first, it looks like:
   
   
   
   modify source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt, the whole modification are “num-sources=1, disable sink0, enable sink1” then run deepstream-app -c source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt >1.log 2>1.log for about 30 seconds.
4. the path of deepstream_test1 is /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1, you can use vi to open deepstream_test1_app.c, please test it according the step in the last comment.

[user29329]

1.Sorry, this is complete log file.
1.log (15.1 MB)

2.I try to make deepstream_test1_app.c, but I got an error. And I found the cuda file is missing. Maybe this is the reason the pipeline don’t work.

[fanzh]

did you meet “make Makefile:25: *** “CUDA_VER is not set”. Stop.” when make?
please do “export CUDA_VER=11.4” if using deepstream 6.0.1 , then rebuild.

[user29329]

Yes, I did “export CUDA_VER=11.4”

[fanzh]

Is there /usr/local/cuda-11.4/ in the docker? is there cuda_runtime_api_h in the docker after finding / -name “cuda_run*”?
there are prerequisites when using docker, please refer to deepstream docker
Ensure these prerequisites are available on your system:

1. nvidia-docker We recommend using Docker 20.10.13 along with the latest nvidia-container-toolkit as described in the installation steps. Usage of nvidia-docker2 packages in conjunction with prior docker versions are now deprecated.
2. NVIDIA display driver version 525.85.12.