The new nvstreammux cannot go back to PLAYING state after EOS

• Hardware Platform: GPU
• DeepStream Version: 6.1
• TensorRT Version: 8.2.5.1
• NVIDIA GPU Driver Version: 510.47.03
• Issue Type: Bug
• How to reproduce the issue? Please see description below!

We are using an inference pipeline with the new nvstreammux where new sources are added on request. A source is terminated and disconnected from nvstreammux when the source reaches EOS. As long as there is at least one active source this works very well. The EOS from the source results in an GST_NVEVENT_STREAM_EOS event from the nvstreammux which we handle downstream in the pipeline followed by removing the stream as well as the nvstreamux request pad in a way which is very similar to the NVIDIA reference app runtime_source_add_delete. When new streams are added the lowest nvstreammux source id, which is not active, is used for the new stream.

The problem is the case when the last stream reaches EOS. Initially this works well with the nvstreammux which generates GST_NVEVENT_STREAM_EOS followed by the pipeline EOS event. The result of this is that the pipeline goes to the PAUSED state. After that it should be possible to go back to PLAYING again when a new stream is added as a FLUSH_STOP is sent to take the elements out of EOS state (as described here and in the same way as in the NVIDIA reference app for runtime_source_add_delete). The pipeline actually reaches PLAYING state when the new source is added, but immediately after that there is an error from nvinfer due to a missed GST_NVEVENT_PAD_ADDED from nvstreammux. We have tried different work-arounds with listening to the pipeline EOS event on the bus followed by changing state to NULL/READY/PAUSED and then back to PLAYING, but we have not found a way for nvstreammux to get back to the PLAYING state without any errors.

This error can be reproduced with the NVIDIA reference app runtime_source_add_delete in the following way:

Use the deepstream:6.1-devel image. E.g.

docker run -it -e DISPLAY=$DISPLAY -v /tmp/.X11-unix/:/tmp/.X11-unix nvcr.io/nvidia/deepstream:6.1-devel

Get the reference apps

cd /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps
git clone https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps.git
cd deepstream_reference_apps/runtime_source_add_delete

Comment out the g_main_loop_quit call after pipeline EOS in deepstream_test_rt_src_add_del.c
(At this line)

vi deepstream_test_rt_src_add_del.c
...

Build the program

CUDA_VER=11.6 make

Run the program with the new nvstreammux and a file shorter than 10 seconds

USE_NEW_NVSTREAMMUX=yes ./deepstream-test-rt-src-add-del file:///opt/nvidia/deepstream/deepstream/samples/streams/yoga.mp4

The second stream goes to playing, but then there is a warning from nvinfer

WARN nvinfer gstnvinfer.cpp:1654:gst_nvinfer_process_objects:
Source info not found for source 3.
Maybe the GST_NVEVENT_PAD_ADDED event was never generated for the source.

This warning is written a number of times, finally followed by GST_FLOW_ERROR from nvinfer.

We currently have a work-around with a videotestsrc which is always running, but this is not ideal as the application keeps using both CPU and GPU resources when it should be idle, so it would be very good for us to get a solution to this problem.

For the pipeline state change when received EOS event, there is sample in deepstream-app source code. You may refer to how the “restart_stream_buf_prob” is added and handle the case.

Thanks for looking into this! I checked the restart_stream_buf_prob in the deepstream-app source code. That probe drops the EOS event from the source before it arrives to the nvstreammux, so there will never be a pipeline EOS in that case. For deepstream-app dropping the EOS is the right solution because the next action is to run a seek on the source to loop the stream and then continue playing from the start again.

In our case it is different as we actually are finished with the source and need to disconnect it from the pipeline. To be able to disconnect the source we must let the EOS event reach the nvstreammux. The result of that is that the pipeline reaches EOS when all streams are finished, which is still fine. Later, when there is a new request to run inference on media, a new source will be added and connected to nvstreammux. Unfortunately nvstreammux doesn’t handle the state change back to PLAYING properly, as I descibed in my original post. I think this should be a supported use case for the nvstreammux and that the current behaviour must be a bug.

We are working on improving the new nvstreammux based on your scenario. The solution will be included in the future release.

2 Likes

That is very good news! If you would like us to test our scenario when the improved nvstreammux is ready we would be happy to do so even with a pre-release version.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.