Sink Probe only fires couple of times (4 to be exact) with nvstreammux and nvinfer

I created a pipeline appsrc->nvv4l2decoder->nvstreammux->nvinfer->sink.

Added a probe:
gst_pad_add_probe (osd_sink_pad, GST_PAD_PROBE_TYPE_BUFFER, osd_sink_pad_buffer_probe, NULL, NULL);

But osd_sink_pad_buffer_probe only gets called a couple of times.

Does anyone have an idea on what might be causing this issue?

Also tried with uridecodebin->nvstreammux->nvinfer->appsink but the problem is still there.

Noticed that buffer-pool-size defaults to 4 and I did not set it. After setting it to 0 everything looks fine.

The problem still persist. The callback executed 9 times and I thought it was fixed.

Any suggestions?

I am lost as to why it fires a couple of time. Atleast if it didnt fire at all.

This query is too broad. As such, it is closed and specific queries opened.