PTS values altered by nvv4l2h64enc

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) AGX
• DeepStream Version 6.1.1
• JetPack Version (valid for Jetson only) 5.0.2

Dear all,
I ran into the following issue. I have a pipeline, and sometimes I need to create an n sec long mp4 file from part of the live stream. When I have to do it, I do the following:

  1. I dynamically attach the following elements to a tee element of the pipeline: nvvideoconvert, nvv4l2h264enc, h264parse, qtmux, filesink
  2. I manually modify the PTS/DTS values in a pad probe on the tee element, this way I make sure that the resulting video starts at time-0 and ends at time-n. (Otherwise the mp4 had wrong timestamps, and it cause weird timings in the player)
  3. After the last needed frame is received by this branch, I block the stream on the tee, I send an eos on this branch, and remove these elements from the pipeline.

This worked fine in DS 5.1 and DS 6.0, but not in DS 6.1.1, I receive the following error:

Buffer has no PTS.

In order to dig deeper, I attached a pad probe to the sink pads of all the elements mentioned in 1. , and I logged the PTS values of the buffer in these pad probes. This is what I received:

$ cat log.txt |grep CB
0:00:08.309172019  8062 0xaaab09088760 INFO        myvideosink gstmyvideosink.cpp:161:cb1: CB_sink_nvvidconv:     PTS for current buffer: 0
0:00:08.336156963  8062 0xaaab09088760 INFO        myvideosink gstmyvideosink.cpp:171:cb2: CB_sink_nvv4l2h264enc: PTS for current buffer: 0
0:00:08.337754805  8062 0xaaab09088760 INFO        myvideosink gstmyvideosink.cpp:161:cb1: CB_sink_nvvidconv:     PTS for current buffer: 99967505
0:00:08.346134735  8062 0xaaab09088760 INFO        myvideosink gstmyvideosink.cpp:171:cb2: CB_sink_nvv4l2h264enc: PTS for current buffer: 99967505
0:00:08.346604016  8062 0xaaab09088760 INFO        myvideosink gstmyvideosink.cpp:161:cb1: CB_sink_nvvidconv:     PTS for current buffer: 199848309
0:00:08.370115207  8062 0xaaab09088760 INFO        myvideosink gstmyvideosink.cpp:171:cb2: CB_sink_nvv4l2h264enc: PTS for current buffer: 199848309
0:00:08.370799128  8062 0xaaab09088760 INFO        myvideosink gstmyvideosink.cpp:161:cb1: CB_sink_nvvidconv:     PTS for current buffer: 299577776
0:00:08.383367326  8062 0xaaab09088760 INFO        myvideosink gstmyvideosink.cpp:171:cb2: CB_sink_nvv4l2h264enc: PTS for current buffer: 299577776
0:00:08.384787204  8062 0xaaab09088760 INFO        myvideosink gstmyvideosink.cpp:161:cb1: CB_sink_nvvidconv:     PTS for current buffer: 399109328
0:00:08.386813590  8062 0xffff3801c5e0 INFO        myvideosink gstmyvideosink.cpp:181:cb3: CB_sink_h264parse:     PTS for current buffer: 492031726
0:00:08.389965368  8062 0xffff3801c5e0 INFO        myvideosink gstmyvideosink.cpp:191:cb4: CB_sink_qtmux:         PTS for current buffer: 492031726
0:00:08.390715534  8062 0xffff3801c5e0 INFO        myvideosink gstmyvideosink.cpp:181:cb3: CB_sink_h264parse:     PTS for current buffer: 492031726
0:00:08.390833078  8062 0xffff3801c5e0 INFO        myvideosink gstmyvideosink.cpp:191:cb4: CB_sink_qtmux:         PTS for current buffer: 18446744073709551615
0:00:08.395414591  8062 0xffff3801c5e0 INFO        myvideosink gstmyvideosink.cpp:181:cb3: CB_sink_h264parse:     PTS for current buffer: 492031726
0:00:08.395518310  8062 0xffff3801c5e0 INFO        myvideosink gstmyvideosink.cpp:191:cb4: CB_sink_qtmux:         PTS for current buffer: 18446744073709551615
0:00:08.396104400  8062 0xaaab09088760 INFO        myvideosink gstmyvideosink.cpp:171:cb2: CB_sink_nvv4l2h264enc: PTS for current buffer: 399109328

What you can see is that the nvv4l2h264enc element changes the PTS value. (it receives the buffer with the correct PTS but the next element (h264parser) receives an altered PTS with a value of 492031726).

In another thread (this one: Nvv4l2h264enc framerate problem - #5 by DaneLLL) it is explicitly stated that the

encoder does not touch/modify timestamps

which is in contradiction what I am experiencing. So my first question is: Did anything changed in the above statement in version 6.1.*? If so, in what circumstances and how may the encoder change the PTS values?

Hi @tamas2 , Could you try it with DS 6.1.0 to narrow down the scope?
===>it receives the buffer with the correct PTS but the next element (h264parser) receives an altered PTS with a value of 492031726.
Do you mean nvv4l2h264enc change the 399109328 to 492031726?
Due to the timing problem, the printing is out of order.

I’m kind of short of devices at the moment I have DS 5.1, 6.0 and 6.1.1 , I may be able to downgrade this last machine in the second half of the week. It takes a lot of time.

For clarifying, what happens on the log above:
nvvideoconvert receives a buffer with PTS 0, it does what it has to do then it passes it over to nvv4l2h264enc. When nvv4l2h264enc receives it, the PTS is still 0. It does it’s thing, then it passes it over to h264parse. When h264parse receives it the PTS is changed: 492031726.
The same happenes with the next buffer:
nvvideoconvert receives a buffer with PTS 99967505, it does what it has to do then it passes it over to nvv4l2h264enc. When nvv4l2h264enc receives it, the PTS is still 99967505. It does it’s thing, then it passes it over to h264parse. When h264parse receives it the PTS is changed again: 492031726.
The same thing happens with the buffers with PTS 199848309 and 299577776
The buffer with PTS 399109328 does not reach h264parse, because the application stops due to the error before it would receive it.

I downgraded to JP 5.0.1/DS 6.1.0 I can confirm that DS 6.1.0 have the same issue. I can also confirm that DS 6.0 works fine, so something is broken between 6.0 and 6.1.0.

Thanks for narrowing down the scope, we will check it.
Could you attach your relevant source code if it is not large? If we can duplicate it in our env, we can locate the problem more quickly.

Hi @tamas2 , How did you set the pts value? Could you attach your probe method?
I use the pts of the original mp4 video to enc, like mp4->qtdemux->decoder->v4l2h264enc->h264parse… The value of the pts doesn’t change.

Our codebase is huge and probably I cannot share it, but if needed I will try to put together a small example to reproduce it sometimes next week.

For setting the PTS and DTS on the buffers, I use the following snippets of code:

// For the first frame I want to save into the mp4:
        GstClockTime base_dts = GST_BUFFER_DTS (gst_buffer);
        GstClockTime base_pts = GST_BUFFER_PTS (gst_buffer);
        GST_BUFFER_DTS (gst_buffer) = 0;
        GST_BUFFER_PTS (gst_buffer) = 0;

// For the next n frame that I want to save into the same mp4:
        GST_BUFFER_DTS (gst_buffer) = (GST_BUFFER_DTS (gst_buffer) - base_dts);
        GST_BUFFER_PTS (gst_buffer) = (GST_BUFFER_PTS (gst_buffer) - base_pts);

It’s true about encoder does not touch/modify timestamps.
I have tried with a simple pipeline:
mp4file src->...->nvvideoconvert->h264enc->h264parse->......
I set the pts value by myself, the encoder didn’t change it.
So I’ll wait for your example to debug it. Thanks

Finally I had some time to put together an example code. The logic of the example code is simple:
Every 2 seconds it attaches the following branch to the pipeline: nvvideoconvert - nvv4l2h264enc - h264parse - qtmux - filesink then 2 seconds later it removes this branch.

You need to edit line 10 and add your camera source info. Then you can compile it (I also attached a CMakeLists.txt if cmake is your preferred way).
I recommend to run it with GST_DEBUG=4, because I use GST_INFO for logging the PTS. i.e.:
GST_DEBUG=4 ./ds_pts_error 2>&1|tee log.txt

The code works fine on DS 5.1.1 (I just checked it), it generates the/tmp/pts_error.x.mp4 files every 2 seconds. However on DS 6.1.0 and 6.1.1, I get the mentioned error. (Buffer has no PTS.)

If I check the logged PTSs for the nvvidconv element it looks reasonable:

ubuntu@xavier-dev-2:~/ds_pts_error/build$ cat log.txt |grep "PTS for"|grep nvvidc
0:00:02.463206060 158508 0xaaaab42ee240 INFO                 default main.cpp:86:cb_probe_on_nvvidconv: cb_probe_on_nvvidconv: PTS for current buffer: 0
0:00:02.514534992 158508 0xaaaab42ee240 INFO                 default main.cpp:86:cb_probe_on_nvvidconv: cb_probe_on_nvvidconv: PTS for current buffer: 99700457
0:00:02.532346276 158508 0xaaaab42ee240 INFO                 default main.cpp:86:cb_probe_on_nvvidconv: cb_probe_on_nvvidconv: PTS for current buffer: 199534901
0:00:02.582054916 158508 0xaaaab42ee240 INFO                 default main.cpp:86:cb_probe_on_nvvidconv: cb_probe_on_nvvidconv: PTS for current buffer: 299458153

If I check the logged PTSs on the sink pad of the encoder element it looks the same, so fortunately the nvvidconv did not change the PTS:

ubuntu@xavier-dev-2:~/ds_pts_error/build$ cat log.txt |grep "PTS for"|grep enc
0:00:02.512878314 158508 0xaaaab42ee240 INFO                 default main.cpp:96:cb_probe_on_enc: cb_probe_on_enc: PTS for current buffer: 0
0:00:02.531942099 158508 0xaaaab42ee240 INFO                 default main.cpp:96:cb_probe_on_enc: cb_probe_on_enc: PTS for current buffer: 99700457
0:00:02.549262227 158508 0xaaaab42ee240 INFO                 default main.cpp:96:cb_probe_on_enc: cb_probe_on_enc: PTS for current buffer: 199534901
0:00:02.598668869 158508 0xaaaab42ee240 INFO                 default main.cpp:96:cb_probe_on_enc: cb_probe_on_enc: PTS for current buffer: 299458153

If I check the logged PTSs on the sink pad of the h264parse element it clearly looks different so it looks like the encoder changed the PTS:

ubuntu@xavier-dev-2:~/ds_pts_error/build$ cat log.txt |grep "PTS for"|grep parse
0:00:02.571771663 158508 0xffff4dfbd580 INFO                 default main.cpp:106:cb_probe_on_parse: cb_probe_on_parse: PTS for current buffer: 474912843
0:00:02.576058117 158508 0xffff4dfbd580 INFO                 default main.cpp:106:cb_probe_on_parse: cb_probe_on_parse: PTS for current buffer: 474912843
0:00:02.592190482 158508 0xffff4dfbd580 INFO                 default main.cpp:106:cb_probe_on_parse: cb_probe_on_parse: PTS for current buffer: 474912843

main.cpp: main.cpp - Pastebin.com
CMakeLists.txt: CMakeLists.txt - Pastebin.com

Sorry, I can’t open the Pastebin.com link. Could you please attach it on the google driver or just click on my icon and message to me? Thanks

Your assumption is wrong it is still an issue. I unmarked the Solution because closing a topic is not a solution. I sent you the files in private message as you requested. I sent them on 23rd Oct, and you are closing the topic on 25th Oct even without having a look on it?

Sorry for late reply, I can repo this problem with your code. The pts form the enc_src is different from the enc_sink. We’ll confirm this problem as soon as possible.