Pipeline with multiple input sources hang with DeepStream 6

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU (T4)
• DeepStream Version 6.0, 6.0.1, 6.1
• JetPack Version (valid for Jetson only)
• TensorRT Version DeepStream container default
• NVIDIA GPU Driver Version (valid for GPU only) 450 / 510
• Issue Type( questions, new requirements, bugs) bugs
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

We have reliably working DeepStream 5.1 applications using multiple input sources which are connected to an nvstreammux element. We run them at scale in production without any issues. Since upgrading these applications to DeepStream 6 we can’t use multiple input sources anymore. I’ve tried out the three available versions of DS6:

DS6.0:

  • The pipelines with more than 1 input source randomly fails in ~5% of cases.

DS6.0.1 and DS6.1:

  • 1 input source works reliably.
  • 2 input sources fail randomly in ~5% of cases like with DS6.0.
  • With more than 2 input sources the pipeline immediately hangs and never shows any GPU utilisation.

All that has changed is the container we are using, the Python DeepStream bindings and the TensorRT engine which we generated for the required TensorRT version.

Furthermore, I tried out the new nvstreammux element in 6.1 but this showed the same issue as described above.

I’ve analysed the Gstreamer debug logs and double checked any message that is getting put on the message bus to the best of my knowlegde but couldn’t find anything helpful.

I came across other forum entries which sound similar to our problem, however, they were never resolved but just closed like this.

The release notes of DS 6.0.1 mention:

Minor bug fixes in Gst-nvvideo4linux2 encoder/decoder, Gst-nvstreammux, and Gst-
nvstreamdemux plugins

I suspect this to be a bug due to a change in DeepStream 6 and potentially related to the bug 6.0.1 tried to fix.

Please see the Gstreamer diagram below for further reference. It shows a case where a pipeline with 2 input sources was working. The error cases are pipelines with the same structure just more input bins.

Kind regards,
Jens

1 Like

Can the deepstream-app sample work with your multiple sources on DS 6.0 and DS 6.1?

Thank you for your reply. I will test it in the next days and get back to you!

I had a look through the example apps but couldn’t find a suitable one. Is there any example which doesn’t need an external display attached? I’m running this on an AWS EC2 instance and am just writing the results to a file.

deepstream-app sample can support none display output. DeepStream Reference Application - deepstream-app — DeepStream 6.1 Release documentation

The output is configurable.

Thank you. I tried an example with multiple input streams from a local file and couldn’t reproduce it with the deepstream-app. However, the setup of the application is different (it uses a decodebin for example which I don’t use and there are other obvious difference like that I’m using the Python bindings and extract the data in a fakesink. That being said this setup worked well with DeepStream 5.1.

Which changes were made to the nvv4l2decoder or nvstreammux element from 5.1 to 6? What were the bugfixes about in 6.0.1 which I mentioned in my first post. They did change the described behaviour. It would be great if you could point me at a few things that are worth checking.