Stream muxer ruturning NULL caps on requested pads normal?

I’m trying to debug a separate issue and I noticed that the stream muxer element is returning pads with caps set to NULL. From the log:

** (nvalhalla:17035): DEBUG: 14:01:16.976: nvalhalla.gs:207: trying to link source_1:src_0 and muxer:sink_1
** (nvalhalla:17035): DEBUG: 14:01:16.976: nvalhalla.gs:208: src_0 CAPS: video/x-raw(memory:NVMM), format=(string)NV12, width=(int)1280, height=(int)720, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)30/1
** (nvalhalla:17035): DEBUG: 14:01:16.976: nvalhalla.gs:209: sink_1 CAPS: NULL

The pads do sucessfully link, however. Is this normal?

Also, my app is occasionally stalling requesting a pad from the stream muxer. Is the source available for it? I can’t find it in sources/gst-plugins.

Hi,
Sinkpad of nvstreammux is no-request. Probably caps is not fixed when it is linked. There are several callbacks in GstPad:
https://developer.gnome.org/gstreamer/stable/GstPad.html

I guess caps should be fixed when sinpad is activated. Please set the callback function and give it a try.

gboolean	(*GstPadActivateFunction) ()

The nvstreammux plugin is not open source code. For further investigation of the issue, would need you to share a test sample.

Thanks for the clarification and the link to the docs. I understand now it’s normal for some pads not to have caps until they’re activated and that current caps / caps property are different from the allowed caps. In any case, I can verify it’s not hanging at the link stage, rather at requesting the pad, so I am pretty sure it’s not related at this point.

In this case, I think I will need your help then.

To replicate the issue please checkout the dev branch of this project. The build / install instructions are in the README.md.

The exact line where it’s (sometimes) hanging is here. My test platform is currently Xavier.

sink_pad_name:string = @"sink_$(self._muxer.numsinkpads)"
debug(@"requesting pad $sink_pad_name")
<b>sink_pad:Gst.Pad = self._muxer.get_request_pad(sink_pad_name)</b>

Generated C for that is meant for the compiler, not reading, but here it is:

static void
_nvalhalla_app_on_src_pad_added (NValhallaApp* self,
                                 GstElement* src,
                                 GstPad* src_pad)
{
... rest of _on_src_pad_added ...
#line 239 "/home/username/Projects/nvalhalla/src/nvalhalla.gs"
        _tmp45_ = self->priv->_muxer;
#line 239 "/home/username/Projects/nvalhalla/src/nvalhalla.gs"
        _tmp46_ = _tmp45_->numsinkpads;
#line 239 "/home/username/Projects/nvalhalla/src/nvalhalla.gs"
        _tmp47_ = g_strdup_printf ("%hu", _tmp46_);
#line 239 "/home/username/Projects/nvalhalla/src/nvalhalla.gs"
        _tmp48_ = _tmp47_;
#line 239 "/home/username/Projects/nvalhalla/src/nvalhalla.gs"
        _tmp49_ = g_strconcat ("sink_", _tmp48_, NULL);
#line 239 "/home/username/Projects/nvalhalla/src/nvalhalla.gs"
        _tmp50_ = _tmp49_;
#line 239 "/home/username/Projects/nvalhalla/src/nvalhalla.gs"
        _g_free0 (_tmp48_);
#line 239 "/home/username/Projects/nvalhalla/src/nvalhalla.gs"
        sink_pad_name = _tmp50_;
#line 240 "/home/username/Projects/nvalhalla/src/nvalhalla.gs"
        _tmp51_ = sink_pad_name;
#line 240 "/home/username/Projects/nvalhalla/src/nvalhalla.gs"
        _tmp52_ = string_to_string (_tmp51_);
#line 240 "/home/username/Projects/nvalhalla/src/nvalhalla.gs"
        _tmp53_ = g_strconcat ("requesting pad ", _tmp52_, NULL);
#line 240 "/home/username/Projects/nvalhalla/src/nvalhalla.gs"
        _tmp54_ = _tmp53_;
#line 240 "/home/username/Projects/nvalhalla/src/nvalhalla.gs"
        g_debug ("nvalhalla.gs:240: %s", _tmp54_);
#line 240 "/home/username/Projects/nvalhalla/src/nvalhalla.gs"
        _g_free0 (_tmp54_);
#line 241 "/home/username/Projects/nvalhalla/src/nvalhalla.gs"
        _tmp55_ = self->priv->_muxer;
#line 241 "/home/username/Projects/nvalhalla/src/nvalhalla.gs"
        _tmp56_ = sink_pad_name;
[b]#line 241 "/home/username/Projects/nvalhalla/src/nvalhalla.gs"
        _tmp57_ = gst_element_get_request_pad (_tmp55_, _tmp56_);[/b]
#line 241 "/home/username/Projects/nvalhalla/src/nvalhalla.gs"
        sink_pad = _tmp57_;
#line 242 "/home/username/Projects/nvalhalla/src/nvalhalla.gs"
        _tmp58_ = sink_pad;
...

The C gets generated in …/build/src/nvalhalla@exe/

To view app specific g_debug messages, set G_MESSAGES_DEBUG=all

Example launch to replicate issue is in the readme (requires pip3 install youtube-dl). I don’t have enough good rtsp sources to stress test with, but you might. That’s why I’m testing with youtube direct video uris.

Hi mdegans,

We don’t see the rectangle mask on human faces. Seems like detection is not running, is this normal or we miss something?
Following https://github.com/mdegans/nvalhalla/tree/dev#requirements steps and run multiple youtube streams.
Thanks!

Aha. That’s the behavior from the master branch, before I added the redaction bin. Sorry I wasn’t clear. The instructions for building the dev branch has one extra step (in bold):

git clone https://github.com/mdegans/nvalhalla.git
cd nvalhalla
<b>git checkout dev</b>
mkdir build
cd build
meson ..
ninja
sudo ninja install

Edit: finally, to run;

G_MESSAGES_DEBUG=all nvalhalla --uri ... --uri ...

Edit: If i post the actual youtube links, the forum tries to parse them as video.

Hi,
Looks like it sets batch-size for multiple times:

// this needs to be updated on pad added or flickering occurs with the osd
debug(@"setting muxer batch-size to $(self._muxer.numsinkpads + 1)")
self._muxer.set_property("batch-size", self._muxer.numsinkpads + 1)

In deepstream-test3, it is configured one time only, after all sources are created. Please modify it to run as test3 and see if it helps.

Thanks. DaneLLL. I was trying to increment batch size in the playing state to match the number of connected sources. I noticed after looking closely at some pdfs that while the properties were set on the elements sucessfully, the stream itself still had batch-size of 1. I will try to set the batch size the same way it’s set in deepstream test 3 and see if that fixes it.

So, setting the “async-handling” property on my uridecodebin sources stopped the issue from re-occuring:

src.set_property("async-handling", true)

It was an accident while I was fixing some other things. I am not convinced the problem is entirely gone, but it’s probably much less likely to happen now since the pads are now being created at wildly different times (I can’t replicate, despite trying many times).

I did try setting “batch-size” in the constructor and everything reports a batch-size of 4, however the osd seems to not be drawing on the right frames:

https://www.youtube.com/watch?v=evhdLUFp2Fc

If I had to guess, i’d say the source ids are somehow getting mixed up. Not sure what’s up with that exactly. Here is a branch with the flickering issue if you are interested in taking a look. I will probably start a new thread for it if I can’t figure out the cause since it appears to be a separate issue. Attached is a pdf of the flickering pipeline.

Anyway. Thank you, DaneLLL and carolyuu for helping me troubleshoot! One problem down, at least one to go.
0.00.56.182628798-pipeline0.quit.pdf (50.8 KB)

So, It ended up the flickering was definately caused by the position of the osd element. It worked with batch size set to 1, but anything above that, the source ids would get mixed up (i assume). If anybody runs into it, attached is a working pipeline pdf compared with the above. The osd works fine when placed after the tiler, and I would assume is more effecient that way.
0.00.34.610304446-pipeline0.quit.pdf (50.6 KB)