Nvinfer buffer probe not in sync with the sink

• Hardware Platform (Jetson / GPU)
GPU
• DeepStream Version
6.1.0

I have a pipeline with nvstreammux -> nvinfer -> nvtracker -> nvdsanalytics -> hlssink2
I’m attaching a probe to nvtracker to store in memory all object tracks inside each frame buffer.

self._gst_pipeline.get_by_name("tracker").get_static_pad("src").add_probe(
    Gst.PadProbeType.BUFFER, self._add_pb_frame
)

...

def _add_pb_frame(self, pad, info):
        ...
        pb_frame = pb.Frame()
        for object_meta in get_object_meta(
            pyds.gst_buffer_get_nvds_batch_meta(hash(buf))
        ):
            pb_track = pb.Track()
            pb_track.id = object_meta.object_id
            pb_track.label = object_meta.obj_label
            pb_track.bbox.x = object_meta.rect_params.left / frame_width
            pb_track.bbox.y = object_meta.rect_params.top / frame_height
            pb_track.bbox.w = object_meta.rect_params.width / frame_width
            pb_track.bbox.h = object_meta.rect_params.height / frame_height
            pb_frame.tracks.append(pb_track)

        self._pb_frames.append(pb_frame)

    return Gst.PadProbeReturn.OK

I also listen for splitmuxsink-fragment-closed message (hlssink2 element). Every time I receive this message, I write all of the tracks to a file so that each 2 seconds of hls .ts fragment, there would be a matching .data file with all of the tracks for these 2 seconds.
Here is the code which writes pb_frame to a file when receiving splitmuxsink-fragment-closed message:

def _dump_segment_data(self, structure):
    ...
    with open(data_file, "wb") as f:
        for pb_frame in self._pb_frames:
            f.write(struct.pack("i", pb_frame.ByteSize()))
            f.write(pb_frame.SerializeToString())

    ...

    print(len(self._pb_frames))
    self._pb_frames.clear()

The issue is that if I have pipeline frame rate set to 15/1 and hls target duration set to 2 seconds for some reason first fragment receives 91 buffer probe calls (!) and then goes to “real-time” which is around 30 frames per 2sec = 15fps. It kinda pushes the data out of sync with the output video.

print(len(self._pb_frames))  <- from the previous code block

91
34
30
30
30
30
31
30
30
30
30

How can I sync nvinfer probe data with the output .ts fragments? Do you have any idea what am I doing wrong? Is nvinfer doing some internal buffering?

You can see what is the issue here. Yellow bboxes are from the data file.

Hi @mpukkonen , Do you mean it performs 91 frames at the first 2 seconds?
Yellow bboxes are from the data file you dumped, are the red bboxes from the ts fragments?

I think I found the issue. hlssink2 has an inside queue for the buffers, I need to attach a probe to the src of that queue.

Glad to hear that, I will set this topic as resolved. If there is any other issues, please open a new topic. Thanks

2 Likes

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.