Please provide complete information as applicable to your setup.
• Hardware Platform: Jetson
• DeepStream Version: 6.4
• JetPack Version: 6.0
• TensorRT Version: 8.6
• Issue Type: Question
When examining the example code for BODYPOSE2D, an NvDsObjectMeta object is created from the pool using the following command:
NvDsObjectMeta *obj_meta = nvds_acquire_obj_meta_from_pool (batch_meta);
As a result, tracker_bbox_info and tracker_confidence come back empty.
The structure of the pipeline is as follows:
nvstreammux → inference (bodypose model) → nvtracker (NvDCF) → queue1 → queue2 → osd
Probe added on queue1.
Inside Probe,
NvDsObjectMeta object is being created based on NvDsInferTensorMeta.
How can I track Bodypose2D BBOX?
Thanks.
Which bodypose2d model are you using? If you are using Nvidia TAO BodyPoseNet | NVIDIA NGC, we already have sample for how to generate the bboxes from the model output. deepstream_tao_apps/apps/tao_others/deepstream-bodypose2d-app at master · NVIDIA-AI-IOT/deepstream_tao_apps (github.com)
The integrated algorithms in nvtracker are based on bboxes. If you want the new tracking algorithm based on bodypose key points, you need to develop by yourself.
Hello,
As far as I understand, I need to customize nvtracker. However, nvtracker is essentially a GStreamer plugin that is loaded dynamically at runtime. Its source code is available within DeepStream, but I cannot debug it using breakpoints in the IDE. What can I do about this?
Thanks.
gst-nvtracker plugin is open source. /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvtracker
The tracker algorithm is not in the plugin but in the low-level library /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so. If you met any problem with our low-level library, you can report the issue to us.
If you want to customize your own tracking algorithm, you need to implement your own tracking low-level library. Gst-nvtracker — DeepStream documentation 6.4 documentation. You just need to implement these API in your own low-level library, then the gst-nvtracker plugin can use your customized low-level library. Please refer to the gst-nvtracker plugin source code to understand the workflow.
I created a custom object meta and added it into the frame meta, the code is below.
NvDsObjectMeta *obj_meta = nvds_acquire_obj_meta_from_pool(batch_meta);
cvcore::bodypose2d::Human human;
human = output[0][i];
obj_meta->unique_component_id = meta->unique_id;
obj_meta->confidence = human.score;
obj_meta->object_id = 0;
obj_meta->class_id = PGIE_CLASS_ID_PERSON;
NvOSD_RectParams &rect_params = obj_meta->rect_params;
rect_params.left = human.boundingBox.xmin - left_offset;
rect_params.top = human.boundingBox.ymin - top_offset;
rect_params.width = human.boundingBox.xmax - human.boundingBox.xmin;
rect_params.height = human.boundingBox.ymax - human.boundingBox.ymin;
rect_params.border_width = 5;
rect_params.has_bg_color = 0;
rect_params.border_color = (NvOSD_ColorParams) {0, 0, 1, 1};
rect_params.bg_color = (NvOSD_ColorParams) {1, 0, 1, 1};
NvDsComp_BboxInfo &detectorBboxInfo = obj_meta->detector_bbox_info;
detectorBboxInfo.org_bbox_coords = NvBbox_Coords(rect_params.left, rect_params.top, rect_params.width,
rect_params.height);
NvDsComp_BboxInfo &trackerBboxInfo = obj_meta->tracker_bbox_info;
trackerBboxInfo.org_bbox_coords = NvBbox_Coords(0, 0, 0, 0);
NvOSD_TextParams &text_params = obj_meta->text_params;
text_params.display_text = strdup("person");
text_params.y_offset= rect_params.top ;
text_params.x_offset= rect_params.left;
text_params.text_bg_clr = (NvOSD_ColorParams) {1, 0, 0, 1};
text_params.set_bg_clr = 1;
strncpy(obj_meta->obj_label, "person", MAX_LABEL_SIZE - 1);
obj_meta->obj_label[MAX_LABEL_SIZE - 1] = '\0';
NvOSD_FontParams &font_params = text_params.font_params;
font_params.font_name = strdup("Arial");
font_params.font_size = 24;
font_params.font_color = (NvOSD_ColorParams) {0, 0, 1, 1};
if (!nvds_add_2dpose_meta (batch_meta, obj_meta, human, frame_width,
frame_height, left_offset, top_offset)) {
g_printerr ("Failed to get bbox from model output\n");
}
nvds_add_obj_meta_to_frame(frame_meta, obj_meta, NULL);
Normally, I filled in the bbox areas, the next plugin (nvtracker) should be able to process this.
The content of the metadata is available in the picture. I must be doing something wrong, can you check it?
Thank you.
@Fiona.Chen I am seeing the same issue as @ugurortac. I have modified the sample application to include a tracker and I have modified the pgie_pad_buffer_probe function to draw the detector bboxes as shown below. I am hoping to use the tracker generated object_id in the tile_sink_pad_buffer_probe function, but the tracker does not update the obj_meta->object_id or provide a tracker bbox value. Can you advise how to proceed with tracking the BodyPoseNet drawn bounding boxes?
for (int i = 0; i < human_num; i++)
{
NvDsObjectMeta *obj_meta = nvds_acquire_obj_meta_from_pool(batch_meta);
cvcore::bodypose2d::Human human;
human = output[0][i];
if (human.score > 1.0)
{
obj_meta->unique_component_id = meta->unique_id;
obj_meta->confidence = 1.0;
obj_meta->object_id = UNTRACKED_OBJECT_ID;
obj_meta->class_id = PGIE_CLASS_ID_PERSON;
NvOSD_RectParams &rect_params = obj_meta->rect_params;
/* Assign bounding box coordinates. */
rect_params.left = human.boundingBox.xmin - left_offset;
rect_params.top = human.boundingBox.ymin - top_offset;
rect_params.width = human.boundingBox.xmax - human.boundingBox.xmin;
rect_params.height = human.boundingBox.ymax - human.boundingBox.ymin;
obj_meta->detector_bbox_info.org_bbox_coords.left = rect_params.left;
obj_meta->detector_bbox_info.org_bbox_coords.top = rect_params.top;
obj_meta->detector_bbox_info.org_bbox_coords.width = rect_params.width;
obj_meta->detector_bbox_info.org_bbox_coords.height = rect_params.height;
/*add user meta for 2dpose*/
if (!nvds_add_2dpose_meta(batch_meta, obj_meta, human, frame_width,
frame_height, left_offset, top_offset))
{
g_printerr("Failed to get bbox from model output\n");
}
nvds_add_obj_meta_to_frame(frame_meta, obj_meta, NULL);
frame_meta->bInferDone = true;
PipelineGraph.pdf (32.0 KB)
@Fiona.Chen Can you provide any help on this? Thanks.
frame_meta->bInferDone = true;
This is what I needed, thank you.
@ugurortac That line is the last one in my test code. Still doesn’t work.
Hello,
Change this:
obj_meta->object_id = UNTRACKED_OBJECT_ID;
With this
obj_meta->object_id = 0;
