I’m stumped on this, been through all the developer and plug-in docs, and can’t find any reason why this particular metadata item is missing. Is there a bug?
I am working off the deepstream-app’s source code in DS 4.0 and added some more g_print() console outputs to look at the metadata that the nvinfer plugin is attaching after PGIE. The attached code below (all_bbox_generated) is called from gie_processing_done_buf_prob() in deepstream-app.c. This is in turn linked to the sink side of the nvosd plug-in (trace the code back up through create_processing_instance(), for example). And my debugging output is showing that detected objects have ROI metadata, class metadata, etc. BUT, the confidence value is always 0:
(source cam = 0) detected class 0 (Car) with conf 0.00000000 @ (195, 104, 158, 68)
(Display text: Car)
(source cam = 0) detected class 0 (Car) with conf 0.00000000 @ (237, 106, 154, 66)
(Display text: Car)
(source cam = 0) detected class 0 (Car) with conf 0.00000000 @ (276, 106, 145, 66)
(Display text: Car)
(source cam = 0) detected class 0 (Car) with conf 0.00000000 @ (316, 108, 126, 64)
(Display text: Car)
Device: Jetson Nano. Latest JetPack/etc - did a completely new system image once DS4.0 was released, so it’s a fresh install.
/**
* Callback function to be called once all inferences (Primary + Secondary)
* are done. This is opportunity to modify content of the metadata.
* e.g. Here Person is being replaced with Man/Woman and corresponding counts
* are being maintained. It should be modified according to network classes
* or can be removed altogether if not required.
*/
static void
all_bbox_generated (AppCtx * appCtx, GstBuffer * buf,
NvDsBatchMeta * batch_meta, guint index)
{
for (NvDsMetaList * l_frame = batch_meta->frame_meta_list; l_frame != NULL;
l_frame = l_frame->next) {
NvDsFrameMeta *frame_meta = l_frame->data;
if (frame_meta == NULL)
{
continue;
}
if (frame_meta->bInferDone == 0)
{
continue;
}
for (NvDsMetaList * l_obj = frame_meta->obj_meta_list; l_obj != NULL;
l_obj = l_obj->next) {
NvDsObjectMeta *obj = (NvDsObjectMeta *) l_obj->data;
if (obj == NULL)
{
continue;
}
gfloat l_probability = obj->confidence;
// Try to find label's probability this way? (but sadly it's not populated here either)
for (NvDsClassifierMetaList *c_meta = obj->classifier_meta_list; c_meta != NULL;
c_meta = c_meta->next)
{
NvDsClassifierMeta *cm = (NvDsClassifierMeta *)c_meta->data;
if (cm == NULL)
{
continue;
}
for (NvDsLabelInfoList *li_meta = cm->label_info_list; li_meta != NULL;
li_meta = li_meta->next)
{
NvDsLabelInfo *label_info = (NvDsLabelInfo *)li_meta->data;
if (label_info == NULL)
{
continue;
}
g_print("found label conf = %.8f\n", label_info->result_prob);
l_probability = label_info->result_prob;
}
}
if (obj->unique_component_id ==
(gint) appCtx->config.primary_gie_config.unique_id) { // was primary_gie_config
// Print out info about the object detected
g_print ("(source cam = %d) detected class %d (%s) with conf %.8f @ (%d, %d, %d, %d)\n(Display text: %s)\n",
frame_meta->source_id,
obj->class_id, obj->obj_label, l_probability,
obj->rect_params.left, obj->rect_params.top, obj->rect_params.width, obj->rect_params.height,
obj->text_params.display_text);
}
}
}
}
Relevant part of main config file:
config-file property is mandatory for any gie section.
Other properties are optional and if set will override the properties set in
the infer config file.
[primary-gie]
enable=1
gpu-id=0
model-engine-file=/opt/nvidia/deepstream/deepstream-4.0/samples/models/Primary_Detector_Nano/resnet10.caffemodel_b8_fp16.engine
batch-size=3
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval was 4 frames
interval=2
gie-unique-id=1
nvbuf-memory-type=0
config-file=pgie_primary_nano_config.txt
And the contents of pgie_primary_nano_config.txt:
[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-file=/opt/nvidia/deepstream/deepstream-4.0/samples/models/Primary_Detector_Nano/resnet10.caffemodel
proto-file=/opt/nvidia/deepstream/deepstream-4.0/samples/models/Primary_Detector_Nano/resnet10.prototxt
model-engine-file=/opt/nvidia/deepstream/deepstream-4.0/samples/models/Primary_Detector_Nano/resnet10.caffemodel_b8_fp16.engine
labelfile-path=/opt/nvidia/deepstream/deepstream-4.0/samples/models/Primary_Detector_Nano/labels.txt
batch-size=8
process-mode=1
model-color-format=0
0=FP32, 1=INT8, 2=FP16 mode
network-mode=2
num-detected-classes=4
interval=0
gie-unique-id=1
output-blob-names=conv2d_bbox;conv2d_cov/Sigmoid
#parse-bbox-func-name=NvDsInferParseCustomResnet
#custom-lib-path=/path/to/libnvdsparsebbox.so
#enable-dbscan=1
I added the following two:
classifier-async-mode=0
Only operate on the Car and Person classes
operate-on-class-ids=0;2
[class-attrs-all]
threshold=0.6
group-threshold=1
Set eps=0.7 and minBoxes for enable-dbscan=1
eps=0.2
#minBoxes=3
roi-top-offset=0
roi-bottom-offset=0
detected-min-w=0
detected-min-h=0
detected-max-w=0
detected-max-h=0