Pipeline stream doesn't flow when added bin on runtime

[mo_oises]

Please provide complete information as applicable to your setup.

**• Hardware Platform (Jetson / GPU) Jetson
**• DeepStream Version 6.0
**• JetPack Version (valid for Jetson only) 4.6.3
**• TensorRT Version 8.2.1.32
• NVIDIA GPU Driver Version (valid for GPU only)
**• Issue Type( questions, new requirements, bugs) Question
**• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing) **
**• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description) GLib, Gst, GstRtspServer, GObject

Hello, I’m trying to create a pipeline with two bin containers that I want to change dynamically on runtime, so when make a input, the pipeline change the bin container and process the stream data in a different way. Although I don’t get any error message, the pipeline doesn’t work. I tried adding probes to see what was going on, but I don’t get any output, as if the source file wasn’t been reading.

The Pipeline structure should be something like this.

gst-launch-1.0 -e
filesrc location=sample_720p.h264 ! h264parse ! nvv4l2decoder ! nvstreammux ! tee name=t ! X

BinObject : t. ! queue ! nvinfer ! nvvideoconvert! nvdsosd ! ‘video/x-raw(memory:NVMM), format=I420’ ! nvv4l2h264enc ! rtph264pay ! udpsink host=127.0.0.1 port=5000 \

BinDeactivate : t. ! queue ! ‘video/x-raw(memory:NVMM), format=I420’ ! nvv4l2h264enc ! rtph264pay ! udpsink host=127.0.0.1 port=5000

I based my work on the samples shown by MaZderMind and deepstream_python_apps. The bin containers I create are added and removed succesfully, I get GST_STATE_PLAYING when the bin is linked to the main pipeline and GST_STATE_NULL when It is disconnected from the pipeline, but the data doesn’t flow. I checked the status of the filesrc element and only appears as GST_STATE_NULL or GST_STATE_READY and If i tried to set its state to GST_STATE_PLAYING with the set_state() function I get the error message:

GST_PADS gstpad.c:1149:gst_pad_set_active:file-source:src Failed to activate pad

The rest of the element of the main pipeline seems to be in state GST_STATE_NULL too.

I create each container with a GhostPad in the SinkPad of its queue element. The objective is to make an object detection inference in one bin and in the other just left raw stream flow. Both are sent to a udpsink element with the same properties.

I add the bin container using the sync_state_with_parent() function and linking the tee element with it and when I remove it, I make sure to get the pad and it peer and set the bin status to Gst.State.NULL before remove it from the main pipeline and release the peer tee pad.

I’m not sure what I’m making wrong here. The examples I followed, were mainly to change source files on runtime, so maybe that solution is not suitable for what I’m trying to achive.

It may be to much code lines to post here, so I prefer to upload the .py file here.
Dinamic_Deactivate_SiamTracker_ObjectDetector_Pipeline.py (18.6 KB)
You can find the Runner class here.

[yuweiw]

Demo code: Dinamic.py (15.3 KB)
Related to 252830.

[yuweiw]

Hi @mo_oises, is there only a problem with the Valve plugin not working properly now? This plugin is not used in your code Dynamic.py. So could you attach the latest demo code that could run now?

[mo_oises]

In this code I don’t use the valve element. I tried the dynamic pipeline approach in case I couldn’t make the valve element work. However, I think the problem I have now with dynamic pipeline and the valve element, are the same or pretty similar. At the beggining both seems to work, but at the moment I change the pipeline flow something seems to happen. It must be somethin related to any kind of sync with the sink element. In both codes the sinks element have the async property to false, because that is what many people recommend when using valve element and dynamic pipelines.

About the code, I had some errors in the first code (Dinamic.py). I had some elements not linked.

Now that all are correctly linked, I have a similar behaviour to the code I had with valve element.

To check for which branch data flows I added one probe to each branch. Following the pipeline structure I made in the first message of this post, one probe to the queue in BinDeactivate and the other in nvdsosd in BinObject . Both are simple probe functions, they just print a message an return Gst.PadProbeReturn.OK.

If I start the pipeline without any bin linked to the main pipeline, I don’t get any probe response, if I link any bin to the tee element. sometimes I have probe response from the BinDeactivete if I link it just when I execute the code, but just a few response and then stop.

If I start the pipeline linking to the DeactivateBin I get response from the probe in DeactivateBin and if I change to ObjectBin I have also response from the probe in ObjectBin, but I can’t detach the DeactibateBin and I end up having response from both probe functions.
I use a probe for pad-blocking the pad, but It doesn’t trigger, I don’t know why.

Some post to recommend use event probe instead of pad-blocking probe, but I’don’t know how to do it. This is my function to remove the ObjectBin container.

def removeBinObject(self, value)
        ghostpad_object = self.ObjectBin.get_static_pad("sink_object")
        teepad_object = ghostpad_object.get_peer()


        def blocking_pad_probe_object(pad, info):
            print("block pad object")
            self.ObjectBin.set_state(Gst.State.NULL)
            self.pipeline.remove(self.ObjectBin)
            self.tee.release_request_pad(teepad_object)
            return Gst.PadProbeReturn.REMOVE

       teepad_object.add_probe(Gst.PadProbeType.BLOCK, blocking_pad_probe_object)

If I start the pipeline linking to the ObjectBin I don’t get any response, the same if I attach the Deactivate bin after. No response.

I leave the updated code here, if you want to check more.
Dynamic.py (19.0 KB)
To change the initial attach change the line 370

    GLib.idle_add(self.addBin, 1)

1 for the DeactivateBin, -1 for the ObjectBin and comment it for none.
And to change the Bin container after the pipeline is set to PLAYING, I made an input in a thread, insert -1 to attach the ObjectBin, 1 to attach the DeactivateBin and 0 to check the status of every element.
The removeBin functions doesn’t work, because the pad-blocking probe functions don’t trigger.

[yuweiw]

About the dynamic scenario, have you referred to our demo code: runtime_source_add_delete?

[mo_oises]

I checked it, but it only adds and deletes source elements. I’m more interested in the sink part of the pipeline. And I can’t test the code, the tools used to convert the caffe model to engine are not supported now, I have been using a Yolov7 model I converted myself, but that code uses three different models.

The approach used by the user MaZderMindy seems more suitable for what I’m trying to achieve and It’s the one I have been using, although they only test it with gstreamer, not deepstream.

EDIT: I have managed to make it work in a laptop. Tomorrow I will check runtime_source_add_delete

[mo_oises]

I have changed my setup, now I can use elements that pop up a window and test the code easily.

**• Hardware Platform (Jetson / GPU) GPU
**• DeepStream Version 6.2
**• JetPack Version (valid for Jetson only) **
**• TensorRT Version 8.5.2.2
**• NVIDIA GPU Driver Version (valid for GPU only) 525.105.17
**• Issue Type( questions, new requirements, bugs) Question
**• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing) **
**• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description) GLib, Gst, GstRtspServer, GObject

I simplified the code to debug it and now It only add dynamically a bin to reproduce the filesrc element.
And found the function parse_bin_from_description() so I don’t have to worry about dynamic pads or signals.
Connecting dynamically the pipeline bin queue ! decodebin ! videoconvert ! autovideosink to a the pipeline videotestsrc ! tee t. , I get an output of the videotestsrc, but only one frame, I don’t see the static of the videotest moving, And if I try an .mp4 or .h264 file I don’t get any output with this pipeline structure. Maybe the problem is related to any format or caps filter.

I also checked the runtime_source_add_delete code, but I can’t implement it in my code. The interesting part of this code is the use of “pad-added” signals to connect dynamically the pads, but neither tee or queue element, the elements that are connected dynamically in my code, don’t have any pad related signal.

[yuweiw]

OK. Could it work properly if you just run the MaZderMindy?

[mo_oises]

It seems to work fine, but it only uses audiotestsrc.

[yuweiw]

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

Could you change it to video based on this? If it works fine, then you can add some deepstream plugins in that. It is easier to find and solve problems step by step in this way.