Nvstreamdemux sets wrong caps on demuxed streams

Please provide complete information as applicable to your setup.

• Hardware Platform: GPU
• DeepStream Version: 6.2-triton
• TensorRT Version: 8.5.2.2
• NVIDIA GPU Driver Version: 525.85.12
• Issue Type: Bug
• How to reproduce the issue? See below

Description

When using the Gst-nvstreammux New in combination with Gst-nvstreamdemux, the demuxed batch/streams don’t get their original caps.
Aaccording to the generated pipeline graphs all demuxed streams get one of the source streams original caps.
It seems that nvstreamdemux does not manage to demux the streams properly with batches from the new version of nvstreammux.

Test case

Run test in NVIDIA deepstream container. Optionally mount a directory to export the generated pipeline graphs.

docker run -it -v /tmp/dump-dot-dir:/dump-dot-dir nvcr.io/nvidia/deepstream:6.2-triton

Repro

USE_NEW_NVSTREAMMUX=yes GST_DEBUG_DUMP_DOT_DIR=/dump-dot-dir gst-launch-1.0 nvstreammux name=mux ! nvstreamdemux name=demux uridecodebin uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! mux.sink_0 uridecodebin uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/yoga.mp4 ! mux.sink_1 demux.src_0 ! queue ! fakesink demux.src_1 ! queue ! fakesink

Output

By studying the generated pipeline graph one can see that one of the demuxed streams has the wrong caps.

Additional information

We have run more than two streams through the muxer and the behaviour is the same: all demuxed streams get the same caps as one of the original sources.

it is expected. please refer to this topic.

Thank you for a quick reply, I will look into it.

It is true that we expericene exactly the same issue as in referred topic: Why nvstreamdemux scales all video sources to the resolution of 1st video source linked to new nvstreammux)
I notice that NVIDIA can reproduce the error and agrees with that it is an issue, but there is no clear plan on fixing it. That forum thread ends with “no update for a period, assuming it is not an issue any more”.

However, for us it is a bug that blocks our development.

We are building an application with several video streams of different resolution and frame rates which are batched with the new nvstreammux. It is crucial for our application that when the streams are demuxed in nvstreamdemux, they maintain their original resolution and frame rate.

A year ago @fanzh wrote that NVIDIA will address this issue in the future and marked that answer as the solution even though the bug remains. Can you give us an update on the progress and when you plan to release a correction on this bug?

We have had great response from @Fiona.Chen (which is mentioned in the other forum thread) earlier with other bugs/matters/issues. Perhaps she can reproduce the error with my attached Repro and pinpoint the bug with the combination of new nvstreammux and nvstreamdemux.

it is new streammux’s limitation, please refer to the solution .

Please follow our version updates. Thanks

I work in the same team as @laban.kallgren. We have discussed this and do not think that the proposed solution with “enforcing same resolution and format of all sources connecting to new nvstreammux” is applicable in our case. That solution is intended for the case when there are “plugins that could transform the input buffers … between nvstreammux and nvstreamdemux” according to the documentation. In our case there are no such transformations between nvstreammux and nvstreamdemux. Also, the main reason for us using the new nvstreammux is actually that this makes it possible to use heterogenous batching. Advantages of using heterogenous batching are both that the performance is better with no unneccessary conversions made and that the precision for nvinfer, nvinferserver and nvtracker is better when the only conversion made is from the input resolution to the resolution handled by the models.

As described in the original post, the problem is that the CAPS after nvstreamdemux are not correct. That was also mentioned in the linked post which is supposed to describe the same problem, but I think some of the conclusions are incorrect in that discussion. In particular, it is mentioned that nvstreamdemux rescales buffers. I do not think that is the case and that is also pointed out by @Fiona.Chen early in that thread. We have verified that buffers are not rescaled by clipping detection boxes from frames after nvstreamdemux using the coordinates from the detectors. The result has been correct bounding boxes which would not have been the case if the buffers had been rescaled.

The incorrect CAPS set by nvstreamdemux is a problem for any downstream elements which will handle the frames for presentation, encoding, etc. Unfortunately I do not think that there is a good work-around for correcting the CAPS after nvstreamdemux.

Our conclusion is that this problem must be corrected in nvstreamdemux. Changes made in nvstreammux can never solve this issue as long as nvstreammux still supports heterogenous batches. It seems like the nvstreamdemux is only designed for the “old” nvstreammux and is not working properly when heterogenous batches are sent from the new nvstreammux.

Thanks for the sharing. to support inputting two different resolutions, both New nvstreammux and nvstreamdemux need be changed. Please follow our version updates. Thanks

Thanks for the quick answer, but this is not very helpful for us as we need to decide if we can use the nvstreamdemux and be confident that this error will be corrected in a reasonable time frame. It is probably right that some modification to nvstreammux is needed to make all the original CAPs available for nvstreamdemux, but almost everything is already present in the NvDsFrameMeta, so we still think this is mostly an improvement needed for nvstreamdemux.

I know that release dates for new DeepStream versions are not official until the actual release, but in the past we have at least gotten information such as “will be fixed in next release” or “working on a fix internally”. In this case the limitation has been present for quite some time (since the new nvstreammux was introduced). We would very much like to know if the incorrect CAPS from nvstreamdemux is registrered as a bug internally and a correction is planned to be made for the next minor or major release. For us this is very important and we do think that this bug is not covered by the limitations currently described for nvstreammux in the documentation

the issue has been fixed internally. this bug fix was resolved in DS6.3.

Thanks for the information! That is all we need to know for now, so I’m marking your answer as the solution.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.