Question about Timestamps in nvv4l2decoder

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hello, I have a question about how different PTS values are handled in the nvv4l2decoder element on the following setup:

Platform: Jetson NX Xavier
L4T: R35.5.0
Deepstream: 6.3

We are working with cameras with a non-constant frame rate. The following example illustrates the buf.pts BEFORE and AFTER nvv4l2decoder.

PTS Before NvDec PTS After NvDec
0.5 0.5
1 1
1.5 1.5
2 2
GST_CLOCK_TIME_NONE 2.5
GST_CLOCK_TIME_NONE 3
GST_CLOCK_TIME_NONE 3.5
2.2 3.5
2.5 3.5

In the table I am trying to illustrate the following situation:

  • Some frames have pts = GST_CLOCK_TIME_NONE
  • On receiving such frame, the nvv4l2decoder increments its internal timestamp by some constant
  • When frames with correct pts resume, nvv4l2decoder sets the timestamp to its internal timestamp if the original is less or equal, i.e. buf.pts = std::max(buf.pts, decoder_pts)

This however results in two frames which are consecutive (i.e. originally 40 ms apart) can end up several seconds apart in time. This might cause troubles for several algorithms, which work with time values and require the timestamps to be correct (e.g. object velocity estimation – causes objects to slow down).

Is my observation correct? Is this indeed how decoder works with the timestamps?

If so, is there a way of configuring the decoder to not increment the timestamp when receiving NONE value, or to not modify the PTS values if they are set?

I have read the docs here: Gst-nvvideo4linux2 — DeepStream 6.3 Release documentation, but did not find any answer to how Timestamp values are handled.

Thanks,
Simon

3

In theory, our decoder won’t change pts of the source video. Could you attach the source video you are using?

4

Hello, this is happening for RTSP live streams so unfortunately, there is not a video I can share. However, I can provide an example in Python which can demonstrate, that nvv4l2decoder changes timestamps of buffers:

We can make a probe which will set timestamps to a very small number periodically

class ChangePtsProbe:
    def __init__(self):
        self.frame_idx = 0

    def probe(self, pad: Gst.Pad, info: Gst.PadProbeInfo, u_data) -> Gst.PadProbeReturn:
        buf: Gst.Buffer = info.get_buffer()
        self.frame_idx += 1

        if self.frame_idx > 16:
            buf.pts = 1  # For debugging reasons, change to something very small
        if self.frame_idx == 20:
            self.frame_idx = 0

        return Gst.PadProbeReturn.OK

Then we can make another probe which will print the buffer timestamp

def print_pts(pad, info: Gst.PadProbeInfo, u_data) -> Gst.PadProbeReturn:
    buf: Gst.Buffer = info.get_buffer()
    dt_buf = datetime.fromtimestamp(buf.pts / Gst.SECOND, tz=timezone.utc)
    print(f'Current PTS: {buf.pts} ({dt_buf})')

    return Gst.PadProbeReturn.OK

Now we will attach the pts change probe to decoder sink and the printing probe to decoder src.

pts_change_probe = ChangePtsProbe()
decoder.get_static_pad('sink').add_probe(Gst.PadProbeType.BUFFER, pts_change_probe.probe, None)
decoder.get_static_pad('src').add_probe(Gst.PadProbeType.BUFFER, print_pts, None)

In the following log fragement, we can see that for the four frames which have set the PTS to 1, the decoder will replace it with the time of the last frame.

Current PTS: 1734515764146585777 (2024-12-18 09:56:04.146586+00:00)
Current PTS: 1734515764182332336 (2024-12-18 09:56:04.182332+00:00)
Current PTS: 1734515764259908083 (2024-12-18 09:56:04.259908+00:00)
Current PTS: 1734515764337785244 (2024-12-18 09:56:04.337785+00:00)
Current PTS: 1734515764376910763 (2024-12-18 09:56:04.376911+00:00)
Current PTS: 1734515764456563681 (2024-12-18 09:56:04.456564+00:00)
Current PTS: 1734515764536451537 (2024-12-18 09:56:04.536452+00:00)
Current PTS: 1734515764576425074 (2024-12-18 09:56:04.576425+00:00)
Current PTS: 1734515764656419589 (2024-12-18 09:56:04.656420+00:00)
Current PTS: 1734515764736418895 (2024-12-18 09:56:04.736419+00:00)
Current PTS: 1734515764776418866 (2024-12-18 09:56:04.776419+00:00)
Current PTS: 1734515764856418866 (2024-12-18 09:56:04.856419+00:00)
Current PTS: 1734515764936418866 (2024-12-18 09:56:04.936419+00:00)
Current PTS: 1734515764976418866 (2024-12-18 09:56:04.976419+00:00)
Current PTS: 1734515765056418866 (2024-12-18 09:56:05.056419+00:00)
Current PTS: 1734515765136418866 (2024-12-18 09:56:05.136419+00:00) <<- HERE
Current PTS: 1734515765136418866 (2024-12-18 09:56:05.136419+00:00) <<- HERE
Current PTS: 1734515765136418866 (2024-12-18 09:56:05.136419+00:00) <<- HERE
Current PTS: 1734515765136418866 (2024-12-18 09:56:05.136419+00:00) <<- HERE

Now, if we set the pts to NONE, decoder will change it to the (timestamp of last frame + some constant). This could cause the decoder to ‘overshoot’ the time estimate. This in turn, after frames with proper timestamps are resumed, they will be ‘smaller than the current time of decoder’, resulting in a sequence of frames with different data but the same time value

5

OK. In theory, there will be some compensation operations for abnormal pts frames. But it does have an impact on your particular scenario : a non-constant frame rate camera. We’ll check internally for this.

6

Hello, i understand that you will have to check this internally.

Unfortunately, i will have to disagree with your assessment that it has no impact on my scenario. Due to the suspected compensation for GST_CLOCK_TIME_NONE frames, the timestamps of the several subsequent frames might be shifted ‘to the future’ and have the identical timestamp. Although this does not hinder the precision of nvtracker, it will create mismatch in timestamps for the detections, which potentially leads to algorithms working with time (such as speed estimation) to calculate incorrect results.

7

This is corrected by the Gstreamer source code gstvideodecoder.c. Our hardware decoder will do nothing to the pts of the frame.
You can only check why there are such GST_CLOCK_TIME_NONE frames in your rtsp source and try to remove these abnormal frames from your source stream.

8

Ok, thank you for clarifying.

I will then look into why the PTS values for the non-decoded buffers are NONE. In the meantime, I will add the original PTS value as a metadata to the buffer and restore it after passing through decoder.