I have been using Deepstream 4.0 for some time now for the purpose of utilizing the “nvv4l2decoder” gstreamer plugin. So far it has been working well. However, I am working on performance optimizations which include multi-processing and running the nvidia-cuda-mps-server to improve concurrent utilization. I have successfully implemented multiprocessing and see a 2-3x throughput improvement when nvidia-cuda-mps-server is running. This is where I ran into a problem. My gstreamer video decoding pipeline fails when cuda-mps is running. Simply disabling cuda-mps resolves the problem.
Hardware Platform
GTX 1080 Ti
DeepStream Version
4.0.2
TensorRT Version
6.0.1.5
NVIDIA GPU Driver Version
440.33.01 w/ CUDA 10.2
Issue Type( questions, new requirements, bugs)
Bug
How to reproduce the issue?
Tested with following command:
gst-launch-1.0 rtspsrc location=“rtsp://192.168.1.72/axis-media/media.amp” ! rtph264depay ! h264parse ! nvv4l2decoder ! nvvideoconvert ! autovideosink
Without enabling cuda-mps the pipeline works as expected. However after enabling cuda-mps via:
sudo nvidia-cuda-mps-control -d
The pipeline blocks forever and is unrecoverable. To get it working again all I have to do it stop the cuda-mps-server. I also tried changing the io/memory parameters for the plugin without any luck.
I have attached two logs resulting from running the above pipeline with GST_DEBUG=4 in both scenarios.
gst-failure.log (111.9 KB)
gst-success.log (221.6 KB)
If I were to guess this has something to do with GPU architecture, but it is not clear to me why it is failing. Everything else behaves as expected.