How to obtain previous states of tracked object.

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

According to documentation _NvDsObjectMeta::object_id provides Unique ID for tracking the object, but there is no reference of how to obtain the state (eg. ntp_timestamp, NvOSD_RectParams(rect_params), age) of tracked object in the previous frames. Plz share the guidelines to achieve these previous states.

2

Hi sumityadav,

Please refer to deepstream_app.c:: tracking_done_buf_prob() to record each frame/objects’s states you want. When new frame coming, check the object_id and search all relative previous states.

3

Hi kayccc, Thanks for your suggestion, but for my usecase searching all the relative previous states is not usable as it’ll be time consuming. I need to store data only for tracking id’s that are active ( means for objects which are not gone out of the frame/ technically objects for which shadowTrackingAge increase more than maxShadowTrackingAge).

The application i am building requires the speed of bounding boxes and also save some data for each tracked object that persists for future frames. Speed calculation also require bounding box position and timestamp in previous frames. Getting only object-id is not sufficient in this case, also the data in obj_user_meta_list for each object doesn’t persists for future frames.

  1. Is there some way i can save the data for each tracked object that persists for future frames? i know there is pPreserveData field for _NvMOTObjToTrack but don’t know how to fetch that
    _NvMOTObjToTrack :: *pPreservedData; /**< Used for the client to keep track of any data associated with the object. */

  2. If i try to save data for each tracked object, then i will need the object-id that tracker has deleted after they go out of frame(means their shadowTrackingAge increase more than maxShadowTrackingAge ) to delete the data corresponding to those object-id.
    How can i get the object-id that are deleted (goes out of the frame).

This is a must requirement to get the speed and some obj_user_meta_list field for each tracked-id (unique object-id) that persists for future frames for many applications. Without these features Deepstream will be only useful for static applications. Deepstream must have these features.

4

Are you using nvtraker plugin or just refer the low level tracker lib?
currently the feature is not supported. You need one of the following:
A) For the tracker output to include each object’s bboxes in previous frames; or
B) IDs of objects that get dropped by the tracker because they are no longer in the scene.

Cache bbox history for all the objects because it’s very inefficient. For now we can suggest an optimization for that method:

  1. Cache bbox history for each object in deepstream_app::tracking_done_buf_prob() . For each object, store the latest frame number seen.
  2. Periodically (or on each frame), remove objects whose last seen frame number is older than some threshold, e.g. current_frame - last_seen_frame > maxShadowTrackingAge
    As for pPreservedData, it wouldn’t help in this case because that is set in input to the tracker. The input consists of detector bboxes and the IDs for them are not known at that point.
5

Hi,
Apologies for bumping this old thread but it looks like the info here is out of date. I’m new to Deepstream. However, I’ve done some research on this and it looks like there’s some new pertinent information for this with Deepstream 5. For instance tracking_done_buf_prob no longer exists and there appears to be some new capability to support tracking previous frames for a given object now in Tracker Metadata with NvDsPastFrameObj and friends. I can see function write_kitti_past_track_output in deeepstream_app.c which is called from bbox_generated_post_analytics_cb and looks like it’s really close to what I’d need to do. However, that function appears to dump output to files on every frame. What I’d like to do is dump exactly once per object when it’s no longer tracked, and conditionally based on what the object did per nvdsanlaytics when it was tracked (whether it crossed lines, entered/exited ROIs, etc).

The part I’m still missing is how to generate a track for an object once it leaves or is no longer detected and how to associate the track history with the object and associated analytics from Gst-nvdsanalytics. I feel like this would be a really common use case so I’m guessing I’m missing something, but right now the best option I can see for doing this is to:

  1. Cache NvDsObjectMeta content for objects I care about in parse_nvdsanalytics_meta_data based on their analytics.
  2. Cache my own list of NvDsPastFrameObjList for every object I care about in my cached NvDsObjectMeta list, updated each time on bbox_generated_post_analytics_cb or possibly all_bbox_generated_cb. I’ll need to match objects up with my NvDsObjectMeta cache using some mechanism which is not completely obvious, but probably by matching NvDsObjectMeta::object_id to NvDsPastFrameObjList::uniqueId? I’m especially not sure about this since the description of NvDsPastFrameObjList::uniqueId says this is “Object class id.” which doesn’t seem like the same thing as NvDsObjectMeta::object_id.
  3. Each time in bbox_generated_post_analytics_cb or possibly all_bbox_generated_cb iterate through all objects in my cached list to see if these are still in NvDsFrameMeta::obj_meta_list. If they aren’t, use the cached NvDsPastFrameObjList entry which maps to removed items to dump the track info.

This all feels like I’m making things too complicated. Any suggestions or pointers to documentation I’ve missed or better ways to accomplish are much appreciated. I was hoping to find a callback somewhere which would tell me when the tracker adds/removes an object and, on removal, have some way to access the associated NvDsPastFrameObjList or similar structure to access the track.

If what I’ve proposed is indeed currently the best way to accomplish what I’m trying to do, some clarification/confirmation about the tradeoffs between bbox_generated_post_analytics_cb or all_bbox_generated_cb as callback functions to use as well as the best way to map between NvDsPastFrameObjList and NvDsObjectMeta would be greatly appreciated as this isn’t obvious to me from the documentation.

Thanks!

6

There appears to be some new capability to support tracking previous frames for a given object now in Tracker Metadata with NvDsPastFrameObj and friends. I can see function write_kitti_past_track_output in deeepstream_app.c which is called from bbox_generated_post_analytics_cb and looks like it’s really close to what I’d need to do.

This is actually isn’t useful/helpful, because the enable-past-frame option where this data comes from, as described at this link doesn’t save tracks for all objects in the scene, only tracks which were generated before the object was not reported as output.

The low-level capabilities also include support for passing the past-frame data, which includes the object tracking data generated in the past frames but not reported as output yet. This can be the case when the low-level tracker stores the object tracking data generated in the past frames internally because of, say, low confidence, but later decided to report due to, say, increased confidence

So the best approach as of Deepstream 5.0 still seems to be, as suggested above, to do your own bbox caching. However, you could further improve tracks by picking up enable-past-framecontent for objects before they were output from gst-nvtracker.

7

Hi Dan,

We are looking to do similar work with tracking objects, and ideally looking to leverage the past frame objects to avoid doing all of our own Bbox caching. Thank you for following up with the update as well.

Did you get any clarity on point 2:

I’m especially not sure about this since the description of NvDsPastFrameObjList::uniqueId says this is “Object class id.” which doesn’t seem like the same thing as NvDsObjectMeta::object_id .

8

Hi Scriobhneoir,

Please help to open a new topic. Thanks