Nvstreammux

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hello everyone.
I have a deepstream pipeline that is connected to an RTSP H264 stream and the decoding is configured in this way:

rtspsrc->rtph264depay->h264parse->nvv4l2decoder->Gst-nvstreammux

I need to keep the information about the timestamp of the frame that is inside the rtp packet so added a python probe that extract it at the sink of rtph264depay.
In this probe I extract the pts and the rtp timestamp.
I save both of them.

I also have a probe at the source of the nvstreammux where, using the pts, i can get the original rtp timestamp and save inside the nvidia structure.

I have noticed that if the stream is jumpy, sometimes the probe at the sink can’t find the corresponding pts. It seems that for some reason nvstreammux change it to something else, is that possible?

Also I have noticed that the timestamp is delayed if I don’t enable the “sync-inputs”: 1

Is there a document that explain how exactly nvstreammux behave and why he changes the pts and in which occasion please? Should I use the sequence_number instead?

I have also noticed that the batched-push-timeout has to be adjusted every time we add a source to match the highest FPS, is there a parameter that force the nvstreammux to push a batch when more than x frames of the same stream are pending? That will simplify my life a lot.

This is the actual configuration of the nvstreammux

“width”: 1280,
“height”: 720,
“batched-push-timeout”: 33000,
“batch-size”: 32,
“live-source”: 1,
“sync-inputs”: 1

I’m testing with inputs which speed is less or equals to 30fps

**• Hardware Platform GPU NVIDIA GeForce RTX 2070
• DeepStream 6.1
**• NVIDIA GPU Driver Version Driver Version: 515.65.01 CUDA Version: 11.7
**• Issue Type questions

3

nvstreammux will save input data 's pts to 's NvDsFrameMeta buf_pts, please find “live-source” in doc:Gst-nvstreammux — DeepStream 6.3 Release documentation

could elaborate your requirement? what do you mean about " be adjusted every time"? we need to keep in real-time，why do you need to
“when more than x frames of the same stream are pending”?

4

Hello,
Thanks for the reply.

could elaborate your requirement? what do you mean about " be adjusted every time"? we need to keep in real-time，why do you need to
“when more than x frames of the same stream are pending”?

Regarding the batched-push-timeout:
batched-push-timeout: Timeout in microseconds to wait after the first buffer is available to push the batch, even if a complete batch is not formed.

So what happens if I feed 1 stream of 30fps and the timeout is higher than 33333 which is (1/30)*1000000?
According to my experiments, the nvstreammux waits until the timeout is reach, then release a batch with only 1 element.
This will happen for every single frame until, that leads the nvstreammux to gradually delay the stream 1.
I was wondering if there is a parameter that can not force me to adjust this number every time I add a stream with a different fps.
For example: if I have more than x frames of the stream 1, create a batch to not delay the stream.

nvstreammux will save input data 's pts to 's NvDsFrameMeta buf_pts, please find “live-source” in doc:Gst-nvstreammux — DeepStream 6.1.1 Release documentation

Thanks, I already read it, I was just asking if in some occasion the nvstreammux can change the pts, maybe to make the visualization better, I don’t know. Because from my experience (and I can be wrong) sometimes I got some mismatch between the pts collected before the nvstreammux and after.

Thank you very much for you reply. I really appreciate it.

5

if batch-size is 1, stream number is 1, the nvstreammux will send a batch if frame number is equal to batch-size. please refer to When enable latency measurement, the result seems has some problem, you will see how nvstreammux send batch.

please refer to this in the link: "Set the live-source property to true to inform the muxer that the sources are live. In this case the muxer attaches the PTS of the last copied input buffer to the batched Gst Buffer’s PTS. If the property is set to false, the muxer calculates timestamps based on the frame rate of the source which first negotiates capabilities with the muxer. "

6

Hello Fanzh, thanks again for the clarification. I’m still a bit confused about the behaviour of this component.

please refer to this in the link: "Set the live-source property to true to inform the muxer that the sources are live. In this case the muxer attaches the PTS of the last copied input buffer to the batched Gst Buffer’s PTS. If the property is set to false, the muxer calculates timestamps based on the frame rate of the source which first negotiates capabilities with the muxer. "

If the live-source prevent the nvstreammux to change the pts, then what the sync-inputs parameter are there for?

if batch-size is 1, stream number is 1, the nvstreammux will send a batch if frame number is equal to batch-size. please refer to When enable latency measurement, the result seems has some problem, you will see how nvstreammux send batch.

Yes, but what happens if I feed just one stream but batch-size is 32 (like my settings) and I set a timeout higher than 33333 on a stream of 30fps?
Will the muxer send a batch of frames coming from the same stream in order to fill the batch, or will hold frames and slow the pipeline down?

7

As the doc said, live-source is used to choose the method of setting batched Gst Buffer’s PTS, sync-inputs is used to sychronization of input frames using PTS.

no, muxer will send one frame each time even bathsize is 32.

8

no, muxer will send one frame each time even bathsize is 32

Okay, so it will delay the frames in order to respect the parameter “batched-push-timeout” right?
Is there an easy way to avoid adjusting the “batched-push-timeout” every time a stream with higher FPS is added? Otherwise, the whole stream may be delayed.

As the doc said, live-source is used to choose the method of setting batched Gst Buffer’s PTS, sync-inputs is used to sychronization of input frames using PTS.

Yes, but can you elaborate when exactly should I enable the sync-inputs. Can you please give me an example when that flag is useful and when it’s not?
Thank you very much!

10

yes, “batched-push-timeout” will work. why dose “batched-push-timeout” need to be adjusted every time? here is the batch-size setting method, Frequently Asked Questions — DeepStream 6.3 Release documentation

When sync-inputs=1, buffers coming too late will be dropped, you can test it using some videos with time OSD.

11

yes, “batched-push-timeout” will work. why dose “batched-push-timeout” need to be adjusted every time? here is the batch-size setting method, Frequently Asked Questions — DeepStream 6.1.1 Release documentation

Thanks, that’s very useful information. Let me explain my problem a little better.
I have multiple streams connecting to nvstreammux by input pad.
I may have some stream that has higher frame rate than others.
So I need to adjust the batched-push-timeout according to the stream who has higher FPS in order to not delay any streams.
If that is not the case, I believe that I haven’t really understood those parameters.
Just to be clear
I have 2 streams
stream 1 : 30fps
stream 2 : 15fps
to have the optimal performance, I should set
batch-size=2
batched-push-timeout=33333
With these settings, I will have a batch every 1/30 sec. And cause of that nvstreammux will produce batch of size 2(stream1&2) then 1(stream1) then 2(stream1&2) then 1(stream1) ecc…

If instead I have 2 streams both at 15fps, I should set:
batch-size=2
batched-push-timeout=66666
So It can produce batch size of 2 with a frame for each stream inside.

Is that correct?
If that is correct, how can I set a parameter that allows me to have the best behaviour without adjusting at real time the parameters?

When sync-inputs=1, buffers coming too late will be dropped, you can test it using some videos with time OSD.

What exactly do you mean by “buffers”? Do you mean packet in the RTSP stream or something different?

Let me say that I appreciate a lot the effort to make me understand the behaviour of this component.
Thanks.

12

yes, let batched-push-timeout still be 33333, because nvstreammux will not send batch if there is no frame after timeout.

the buffer nvstreammux gets, nvstreammux wil check its pts.

13

Okay so I always have to adapt the batched-push-timeout in order to have the best behaviour. That’s okay. I just want to be sure. Thanks for helping me in this.
Have a lovely weekend

Marco.

14

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.