Some frames lose object meta after tracker

[1002469771]

Please provide complete information as applicable to your setup.

• Hardware Platform
GPU
• DeepStream Version
6.1
• NVIDIA GPU Driver Version
510.47.03
• Issue Type
questions
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
I ran deepstream-app in nvcr.io/nvidia/deepstream:6.1-devel container:

./deepstream-app -c /opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app/source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt

Tracker config file was config_tracker_NvDCF_perf.yml, secondary-gies enable were set as 0, num-sources=1.
There was object_meta data after primary detector, while it became null after tracker.
Here’s some code I used to print(in deepstream_app.c):

static GstPadProbeReturn
gie_primary_processing_done_buf_prob (GstPad * pad, GstPadProbeInfo * info,
    gpointer u_data)
{
  GstBuffer *buf = (GstBuffer *) info->data;
  AppCtx *appCtx = (AppCtx *) u_data;
  NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta (buf);
  if (!batch_meta) {
    NVGSTDS_WARN_MSG_V ("Batch meta not found for buffer %p", buf);
    return GST_PAD_PROBE_OK;
  }

   for (NvDsMetaList * l_frame = batch_meta->frame_meta_list; l_frame != NULL;l_frame = l_frame->next) {
        NvDsFrameMeta *frame_meta = (NvDsFrameMeta *)l_frame->data;
        g_print("gie_primary_processing_done_buf_prob frame: %p, obj_meta: %p\n",frame_meta, frame_meta->obj_meta_list);
   }

  write_kitti_output (appCtx, batch_meta);

  return GST_PAD_PROBE_OK;
}

    static GstPadProbeReturn gie_tracker_processing_done_buf_prob (GstPad * pad, GstPadProbeInfo * info, gpointer u_data)
    {
        //g_print("gie_tracker_processing_done_buf_prob\n");
        GstBuffer *buf = (GstBuffer *) info->data;
        NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta (buf);
        if (!batch_meta) {
            return GST_PAD_PROBE_OK;
        }

        for (NvDsMetaList * l_frame = batch_meta->frame_meta_list; l_frame != NULL;l_frame = l_frame->next) {
            NvDsFrameMeta *frame_meta = (NvDsFrameMeta *)l_frame->data;
            g_print("gie_tracker_processing_done_buf_prob frame: %p, obj_meta: %p\n",frame_meta, frame_meta->obj_meta_list);
        }

        return GST_PAD_PROBE_OK;
    }

    gint track_probe_id;
    NVGSTDS_ELEM_ADD_PROBE (track_probe_id,
        pipeline->common_elements.tracker_bin.bin, "src",
        gie_tracker_processing_done_buf_prob, GST_PAD_PROBE_TYPE_BUFFER,
        pipeline->common_elements.appCtx);

And I got message like this:

Runtime commands:
	h: Print this help
	q: Quit

	p: Pause
	r: Resume

** INFO: <bus_callback:194>: Pipeline ready

** INFO: <bus_callback:180>: Pipeline running

gie_primary_processing_done_buf_prob frame: 0x7f346c0017e0, obj_meta: 0x7f346c02fe80
gie_primary_processing_done_buf_prob frame: 0x7f346c001710, obj_meta: 0x7f346c02ff80
gie_primary_processing_done_buf_prob frame: 0x7f346c001640, obj_meta: 0x7f34a8003d20
gie_primary_processing_done_buf_prob frame: 0x7f346c001570, obj_meta: 0x7f34a8009ba0
gie_primary_processing_done_buf_prob frame: 0x7f346c01b280, obj_meta: 0x7f346c03f7a0

**PERF:  FPS 0 (Avg)	
**PERF:  0.00 (0.00)	
gie_tracker_processing_done_buf_prob frame: 0x7f346c0017e0, obj_meta: (nil)
gie_tracker_processing_done_buf_prob frame: 0x7f346c001710, obj_meta: (nil)
gie_tracker_processing_done_buf_prob frame: 0x7f346c001640, obj_meta: (nil)
gie_tracker_processing_done_buf_prob frame: 0x7f346c001570, obj_meta: 0x7f34a8009ba0
gie_tracker_processing_done_buf_prob frame: 0x7f346c01b280, obj_meta: 0x7f346c03f780
gie_primary_processing_done_buf_prob frame: 0x7f346c031e10, obj_meta: 0x7f34a8011140
gie_primary_processing_done_buf_prob frame: 0x7f346c031d40, obj_meta: 0x7f34a8011480
gie_primary_processing_done_buf_prob frame: 0x7f346c031c70, obj_meta: 0x7f34a8011580
gie_primary_processing_done_buf_prob frame: 0x7f346c031ba0, obj_meta: 0x7f346c016b60
gie_tracker_processing_done_buf_prob frame: 0x7f346c031e10, obj_meta: 0x7f34a8011100
gie_tracker_processing_done_buf_prob frame: 0x7f346c031d40, obj_meta: 0x7f34a8011460
gie_tracker_processing_done_buf_prob frame: 0x7f346c031c70, obj_meta: 0x7f34a8011560
gie_tracker_processing_done_buf_prob frame: 0x7f346c031ba0, obj_meta: 0x7f34d0004380
gie_primary_processing_done_buf_prob frame: 0x7f346c017630, obj_meta: 0x7f34a4003280
gie_primary_processing_done_buf_prob frame: 0x7f346c017560, obj_meta: 0x7f346c006ec0
gie_primary_processing_done_buf_prob frame: 0x7f346c0019f0, obj_meta: 0x7f346c007240
gie_primary_processing_done_buf_prob frame: 0x7f346c001920, obj_meta: 0x7f346c020760
gie_tracker_processing_done_buf_prob frame: 0x7f346c017630, obj_meta: 0x7f34a4003280
...

Several frames in the beginning had no obj meta after tracker, if there was some objects , these objects would be lost.
I’ve also tried Deepsort in my own program and I got the same data loss.

[kesong]

Can you have a try to set probationAge to 0?

Please refer below for more details: https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvtracker.html#target-management-and-error-handling

[1002469771]

Yes,I’ve tried to set probationAge to 0, and no data loss any more, but this is not what I expected.According to the document, when probationAge > 0, Late Activation will be used, and the lost objects should be retrieved back from NvDsPastFrameObjBatch. So my real question is, when I get these objects back, how can I get the original frame data back since there is no frame data in NvDsPastFrameObjBatch? Given that I want to crop these objects from video frames.

[kesong]

The tracking data during Shadow Tracking is not reported as metadata to the downstream (so not shown on OSD nor appear in msgBroker messages), but once the object is activated again (i.e., re-detected and identified as the same object), such “past-frame data” is reported to the downstream as usermeta. Please search “past-frame” in our tracker doc Gst-nvtracker — DeepStream 6.2 Release documentation , and make sure you enable past-frame data in the pipeline.