Hi @miguel.taylor. Thanks for responding to this.
We are facing synchronization issues with the “new” nvstreammux. Our pipeline looks like this (slightly simplified):
uridecodebin → new nvstreammux batch-size=1 → nvinferserver batch-size=1 → fakesink (with buffer probe)
We only ever process a single stream per pipeline and rely on the buffers to arrive at the fakesink as close to their PTS as possible, but no matter what settings we chose for the new nvstreammux, the buffers either arrive in “chunks”, or completely randomly “jittered” with regard to their PTS.
In DS6.2 we could set sync-inputs=1 on nvstreammux and sync=true on fakesink, and it would fix the issue. But this behavior has changed in DS6.3 and no longer works. It sort of worsened even. We described the issue in detail here and also added measurements: New nvstreammux sync-inputs and max-latency - #7 by philipp.schmidt
Unfortunately the response was note very helpful, with a few links to already referenced documentation.
The source code for nvstreammux seems to be open now in DS6.3, and the synchronization logic looks rather complicated and ambigous. It is most likely the combination of rtpjitterbuffer and the new nvstreammux which causes the issues. But it does not work properly with sync-inputs=0 either. Currently we consider the quality of the new nvstreammux to still be very beta, which is further emphasized by nvidia responses in this forum completely contradicting what we can read in the source code (e.g. when we asked whether the nvstreammux can drop buffers, which was denied, but we can now find the implementation and multiple comments for it in the source code, it even has a “dropped” signal).
It would be great if someone from Nvidia could shed some light on this. The source code for nvstreammux is available in DS6.3, but the most important struct GstBatchBufferWrapper is still a compiled library and unfortunatly inseparable from the nvstreammux (it takes a pointer to GstNvStreamMux.
It feels like it shouldn’t be so complicated to write a plugin which just attaches the NvDsMeta for a single frame and otherwise completely acts in passthrough mode. That’s all we need.