Filesink saving files with different bitrate and fps from the source?

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

I have a pipeline like RTSP ! decodebin ! streammux ! pgie ! tracker ! streamdemux ! nvvideconvert ! video/x-raw(memory:NVMM), format=I420 ! nvv4l2h265enc ! h265parse ! qtmux ! filesink.

My source RTSP is at 10 fps with bitrate = 4Mbps and iframe interval = 10.

I’ve set the bitrate on the encoder to 4Mbps as well as the iframe interval to 10 just like the source.

I set sync = false and async = false on the filesink.

For some reason - when I inspect the written files with ffprobe or VLC the frame rates can sometimes be around 22 - 24 fps and the bitrates can be higher??
Using VLC I can confirm that the camera is producing a constant 10 fps as expected.

Any ideas? IS it good practice to add the videorate element into the pipeline to keep the fps at 10?

3

Hi,
Could you try the case without nvstreammux and nvstreamdemux?

RTSP ! decodebin ! nvvideconvert ! video/x-raw(memory:NVMM), format=I420 ! nvv4l2h265enc ! h265parse ! qtmux ! filesink

Framerate are derived from timestamps. Not sure but certain mechanism in nvstreammux/demux may touch the value from RTSP source. Please try the case with the plugins.

Besides,do you set live-source=1 in nvstreammux?

1 Like
4

Hi DaneLLL,

Yes I do set live-source to TRUE.

Is it recommended to use QoS which can be set on the encoder and the filesink? I’m unable to find documentation to clearly explain this so allot of these parameters are pure trial and error…

5

Also should the bitrates of the encoder match the source or does this not matter?

6

Also if I test this pipeline:

gst-launch-1.0 rtspsrc location=rtsp://192.168.1.12/Streaming/Channels/101 short-header=true ! rtph265depay ! queue ! h265parse ! nvv4l2decoder ! nvvideoconvert ! “video/x-raw(memory:NVMM),format=I420” ! nvv4l2h265enc ! h265parse ! qtmux ! filesink location=test_fps.mp4 -e

Let it run for a little while then check the file with ffprobe — I see 3.8Mbps at 10.04fps. Which is pretty good since my source camera is set to constant 4Mbps and 10fps.

Also with deecodebin - I get accurate bitrate and fps matching the source:

gst-launch-1.0 rtspsrc location=rtsp://192.168.1.12/Streaming/Channels/101 ! decodebin ! nvvideoconvert ! “video/x-raw(memory:NVMM),format=I420” ! nvv4l2h265enc ! h265parse ! qtmux ! filesink location=test_fps_2.mp4 -e

7

Hi,
Please add prob callback in source pad of nvstreamdemux and check PTS value such as

If you don’t see expect 100ms interval, please overwrite it to get expected framerate. We have issues about timetamps and it may take some time to have a solution. A quick fix is to configure PTS before entering encoder. Please give it a try.

8

Hi @DanelLLL,

I finally got around to trying this. As I have multiple rtsp sources I used frame_meta->buf_pts. The results seem to oscillate between 80 and 120 msec between each frame. Note my sources are running at 10fps so I would expect to see PTS differences of 100msec and reasonably constant (which is what I see when using VLC on my cameras):

Source ID 0, Frame No. 129, PTS 13042 diff 120 (in msec)
Source ID 0, Frame No. 130, PTS 13122 diff 80 (in msec)
Source ID 0, Frame No. 131, PTS 13243 diff 120 (in msec)
Source ID 0, Frame No. 132, PTS 13323 diff 80 (in msec)
Source ID 0, Frame No. 133, PTS 13443 diff 119 (in msec)

Some from source 2:

Source ID 2, Frame No. 123, PTS 12621 diff 119 (in msec)
Source ID 2, Frame No. 124, PTS 12701 diff 79 (in msec)
Source ID 2, Frame No. 125, PTS 12821 diff 119 (in msec)
Source ID 2, Frame No. 126, PTS 12901 diff 79 (in msec)
Source ID 2, Frame No. 127, PTS 13021 diff 119 (in msec)

They seem to bound around from 80msec to 120msec when they should be 100msec.

So you are saying that this could be the reason that my filesink it writing with weird framerates?

If this is what you are talking about with the streamdemux issue - how would I go about correcting this. You mentioned to just overwrite the values. So you mean just set frame_meta-buf_pts to my own values?

Does this mean I just need to keep a variable of the last PTS - add 100msec to it every time and then set it in frame_met->buf_pts. ?

9

oh and what happens when the counter (for PTS) get bigger than the data type allows?

10

The above results were taken from a probe on the tracker before the streamdemuxer. Adding a probe on each of the streamdemuxers src pads shows the same results… Alternating between 79msec and 119 msec (or 80 and 120) per frame.

Why would it yoyo around these values instead of 100msec?

11

Hi,
Please apply the attached patch to test3 app, and try to reproduce the timestamp drift.
And check if the interval is steadily 100ms at source pad of nvv4l2decoder, and gets drifted at source pad of tiler.

test3_debug_timestamp.zip (1.2 KB)

12

Thanks DaneLLL 0- this patch causes the app to crash after about 2 seconds… Here is the output:

In cb_newpad
Creating LL OSD context new
[0x7f00125800]:[0]DEC PTS 250475751 diff 250475751 (ns) 
[0x7f00125800]:[1]DEC PTS 330475751 diff 80000000 (ns)
[0x7f00125800]:[2]DEC PTS 449671457 diff 119195706 (ns)
[0x7f00125800]:[3]DEC PTS 528446919 diff 78775462 (ns) 
[0x7f00125800]:[4]DEC PTS 646460229 diff 118013310 (ns)
[0]: PTS 250475751 diff 250475751 (ns)
[0x7f00125800]:[5]DEC PTS 723994029 diff 77533800 (ns) 
[0x7f00125800]:[6]DEC PTS 840762321 diff 116768292 (ns)
[0x7f00125800]:[7]DEC PTS 917289631 diff 76527310 (ns)
0:01:44.912686179 28216   0x55a078f720 WARN                 nvinfer gstnvinfer.cpp:1830:gst_nvinfer_output_loop:<primary-nvinference-engine> error: Internal data stream error.
0:01:44.912729982 28216   0x55a078f720 WARN                 nvinfer gstnvinfer.cpp:1830:gst_nvinfer_output_loop:<primary-nvinference-engine> error: streaming stopped, reason error (-5)
ERROR from element primary-nvinference-engine: Internal data stream error.
[0]: PTS 330475751 diff 80000000 (ns) 
Error details: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(1830): gst_nvinfer_output_loop (): /GstPipeline:dstest3-pipeline/GstNvInfer:primary-nvinference-engine:
streaming stopped, reason error (-5)
Returned, stopping playback
[0x7f00125800]:[8]DEC PTS 1033431819 diff 116142188 (ns)
[0]: PTS 449671457 diff 119195706 (ns)
[0]: PTS 528446919 diff 78775462 (ns)
Deleting pipeline

This was running against one rtsp source.

13

Hi,
You probably miss export DISPLAY=:0

From the short log, it looks like the drift is from the source itself. The nvv4l2decoder plugin does not touch timestamps and the value is identical to the rtsp source. We can see it drifting around 120ms, 80ms.

[0x7f00125800]:[1]DEC PTS 330475751 diff 80000000 (ns)
[0x7f00125800]:[2]DEC PTS 449671457 diff 119195706 (ns)
[0x7f00125800]:[3]DEC PTS 528446919 diff 78775462 (ns) 
[0x7f00125800]:[4]DEC PTS 646460229 diff 118013310 (ns)
14

Ahhhhhh - stupid me. I’m running my jetson headless with no display.

For my own learning - what was it in the log output that told you it was a display issue? error -5 ?

Also - what does setting export DISPLAY=:0 actually do. Does this tell the system to use some sort of null display or something?

So onto the PTS issue. So you think the camera (Hikvision) has some drift… Wonder why VLC still reports it as a nice 10fps stream?

You mentioned earlier that the pts values can be overwritten - is that something that would fix the issue?

15

Hi,
The nveglglesink plugin is X11 based implementation and it requires the setting. For headless, you may use fakesink in test3.

If the timestamps are in interval like 120ms-80ms-120ms-80ms…, it looks to be 10fps.

From the log, buf->pts at src pad of nvv4l2decoder is identical to frame_meta->buf_pts at src pad of stream tiler. Looks like streammux and nvinfer do not touch timestamps.
In your app, could you check buf->pts at src pad of nvv4l2decoder, frame_meta->buf_pts/buf->pts at src pad of streamdemux?

16

Ok when I’m encoding, the PTS values on the demux src pad drop down to 39msec which matches what the encoder has at its src pad - except for the initial one which I imagine is large due to the encode “starting up”.

Why would the pts values on the demuxer be affected? And why would they decrease? Whatever it is the end result is the qtmux and filesink seem to record files that look fine and play back ok - but are at random fps values - sometimes at 33 fps for example - so the files are shorter in time.

Apr 02 17:16:10 nano-1 python[24125]: DEMUX SRC PAD Source 0, Batch meta PTS 11860735423 diff 39 (in msec
Apr 02 17:16:10 nano-1 python[24125]: DEMUX SRC PAD Source 0, Batch meta PTS 11900704849 diff 39 (in msec
Apr 02 17:16:10 nano-1 python[24125]: ENC SRC PAD Source 0, Batch meta PTS 11780797313 diff 11780 (in msec
Apr 02 17:16:10 nano-1 python[24125]: ENC SRC PAD Source 0, Batch meta PTS 11820766244 diff 39 (in msec
Apr 02 17:16:10 nano-1 python[24125]: ENC SRC PAD Source 0, Batch meta PTS 11860735423 diff 39 (in msec
Apr 02 17:16:10 nano-1 python[24125]: ENC SRC PAD Source 0, Batch meta PTS 11900704849 diff 39 (in msec
Apr 02 17:16:10 nano-1 python[24125]: DEMUX SRC PAD Source 0, Batch meta PTS 11940497197 diff 39 (in msec

Prior to adding in the encoder (dynamically linked on a tee) the demux src pad just continually oscillates between 80 and 120 msec as shown in previous posts.

My encoder is configured as follows:

g_object_set (G_OBJECT (instances[source_id].encoder), “bufapi-version”, TRUE, NULL);
g_object_set (G_OBJECT (instances[source_id].encoder), “iframeinterval”, 10, NULL);
g_object_set (G_OBJECT (instances[source_id].encoder), “maxperf-enable”, TRUE, NULL);
g_object_set (G_OBJECT (instances[source_id].encoder), “bitrate”, 2000000, NULL);
g_object_set (G_OBJECT (instances[source_id].encoder), “qos”, TRUE, NULL); //handle qos events from downstream elements

NOTE: Setting QoS on the encoder and the filesink seems to make no difference - however I found a comment in deepstream-app code that says you shold always use QoS.

17

Hi,
Please set batched-push-timeout=150 and see if it helps. It is explained in the Development Guide.

The default value is for 30fps. Need to tune it accordingly.

18

Hi DaneLLL. I already have tuned the batch push timeout to 120 based on my FPS of 10. I will increase it to 150 and see if that makes a difference.

What I am wondering though - is it safe to overwrite the PTS values just before entering the encoder? I assume I could overwrite the values by increasing each frame by 100msec.

What about the Qos setting - what do you advise for that? And any of my other encoder settings shown above?

19

Hi,
qos is a mecahnism in gstreamer:
https://gstreamer.freedesktop.org/documentation/plugin-development/advanced/qos.html?gi-language=c
On Jetson Nano, the hardware capability is lmited and it is suggested be turned off.

So it looks like the dynamic linking triggers the issue. If you were able to use appsink, it could avoid hitting it. However, in using dynamic linking, it should be a solution to overwrite PTS before entering encoder.

20

Thanks DaneLLL - can’t use appsink as previously discussed as it wont work for multiple sources. I will try overwriting the PTS values and see what that does.

That info on NOT using QoS on the Nano is what we are looking for. Interstingly - there is a comment in the deepstream-app source that says you “should” always use QoS with live-sources. I will turn it off based on your advice. But it sounds like if you are using more powerful hardware like the Xavier you should turn Qos on.

21

Hi DaneLLL,

If you see this thread: After 10h or more runtime, swap and memory is suddenly full used on jestson nano? - #11 by laurentj6jzo

Possibly doing this and setting live-source to 0 and allow the muxer to generate the pts is a way to go?