nvinfer is not populating 'confidence' field in NvDsObjectMeta (DS 4.0)

Thanks a lot. Let me know how it goes.

Hi KuanYong.

Any Updates ?? I’ve tried what you suggested but still nothing. See https://devtalk.nvidia.com/default/topic/1060849/deepstream-sdk/deepstream-v4-zero-confidence-problem/ .

Inside /opt/nvidia/deepstream/deepstream-4.0/sources/gst-plugins/gst-nvinfer/gstnvinfer_meta_utils.cpp, there is a function called attach_metadata_detector() which assigns obj_meta->confidence = 0.0;

So you could modify the structure to add a new field to store confidence values, then change the line to assign the confidence value to obj_meta, then recompile nvdsinfer and gst-nvinfer.

/opt/nvidia/deepstream/deepstream-4.0/sources/includes/nvdsinfer_context.h:

/**
 * Holds the information about one detected object.
 */
typedef struct
{
    /** Offset from the left boundary of the frame. */
    unsigned int left;
    /** Offset from the top boundary of the frame. */
    unsigned int top;
    /** Object width. */
    unsigned int width;
    /** Object height. */
    unsigned int height;
    /* confidence value for the object class. */
    float confidence;
    /* Index for the object class. */
    int classIndex;
    /* String label for the detected object. */
    char *label;
} NvDsInferObject;

/opt/nvidia/deepstream/deepstream-4.0/sources/libs/nvdsinfer/nvdsinfer_context_impl_output_parsing.cpp:

/**
 * Cluster objects using OpenCV groupRectangles and fill the output structure.
 */
void
NvDsInferContextImpl::clusterAndFillDetectionOutputCV(NvDsInferDetectionOutput &output)
{
...
    for (unsigned int c = 0; c < m_NumDetectedClasses; c++)
    {
        /* Add coordinates and class ID and the label of all objects
         * detected in the frame to the frame output. */
        for (auto & rect:m_PerClassCvRectList[c])
        {
            NvDsInferObject &object = output.objects[output.numObjects];
            object.left = rect.x;
            object.top = rect.y;
            object.width = rect.width;
            object.height = rect.height;
            object.classIndex = c;
            object.label = nullptr;
            object.confidence = 0.6; // derive your confidence value from somewhere
            if (c < m_Labels.size() && m_Labels[c].size() > 0)
                object.label = strdup(m_Labels[c][0].c_str());
            output.numObjects++;
        }
    }
...
}

/opt/nvidia/deepstream/deepstream-4.0/sources/gst-plugins/gst-nvinfer/gstnvinfer_meta_utils.cpp:

/**
 * Attach metadata for the detector. We will be adding a new metadata.
 */
void
attach_metadata_detector (GstNvInfer * nvinfer, GstMiniObject * tensor_out_object,
    GstNvInferFrame & frame, NvDsInferDetectionOutput & detection_output)
{
...
  for (guint i = 0; i < detection_output.numObjects; i++) {
    NvDsInferObject & obj = detection_output.objects[i];
...
    obj_meta = nvds_acquire_obj_meta_from_pool (batch_meta);

    obj_meta->unique_component_id = nvinfer->unique_id;
    // obj_meta->confidence = 0.0;
    obj_meta->confidence = obj.confidence;
...
   }
...
}

Thanks @KuanYong. I’ve already done exactly what you said BUT it didn’t work.

I’ve tried also to set the values of object.left, object.top, object.width, object.height to zero in “NvDsInferContextImpl::clusterAndFillDetectionOutputCV” function just to see if the information of NvIneferObject will be populated (it didn’t).
I rebuild everything.
I still got non null bbox detections (which doesn’t suppose to happen).
This means that “NvDsInferContextImpl::clusterAndFillDetectionOutputCV” doesn’t do anything.
I even tried modifying “NvDsInferContextImpl::clusterAndFillDetectionDBSCAN” function and still got the same results.

Hi @anassamar8,

I am using the objectDetector_Yolo sample custom parser and with the changes I posted, I was able to get print outs of 0.6 as the confidence level for my detected objects so it must have been something else you missed. Maybe you could check that the correct .so files are being used by gstreamer.
I made the changes to the files in their original locations and ran sudo make clean and sudo make install in both the folders and it worked for me.

I didn’t care about grouping the rectangles and I could get all the confidence numbers by changing clusterAndFillDetectionOutputCV() to the following…

/opt/nvidia/deepstream/deepstream-4.0/sources/libs/nvdsinfer/nvdsinfer_context_impl_output_parsing.cpp

/**
 * Cluster objects using OpenCV groupRectangles and fill the output structure.
 */
void
NvDsInferContextImpl::clusterAndFillDetectionOutputCV(NvDsInferDetectionOutput &output)
{
    output.objects = new NvDsInferObject[m_ObjectList.size()];
    output.numObjects = 0;

    for (unsigned int i = 0; i < m_ObjectList.size(); i++)
    {

	    NvDsInferObject &object = output.objects[i];
	    object.left = m_ObjectList[i].left;
	    object.top = m_ObjectList[i].top;
	    object.width = m_ObjectList[i].width;
	    object.height = m_ObjectList[i].height;
	    object.classIndex = m_ObjectList[i].classId;
	    object.confidence = m_ObjectList[i].detectionConfidence;
	    int c = object.classIndex;
	    object.label = nullptr;
	    if (c < m_Labels.size() && m_Labels[c].size() > 0)
		object.label = strdup(m_Labels[c][0].c_str());
	    output.numObjects++;
    }
}

My Output:
ubuntu@ubuntu-desktop:~/workspace/testrun$ ./deepstream-app

Using winsys: x11
Creating LL OSD context new
Deserialize yoloLayerV3 plugin: yolo_17
Deserialize yoloLayerV3 plugin: yolo_24
Deserialize yoloLayerV3 plugin: yolo_17
Deserialize yoloLayerV3 plugin: yolo_24
Deserialize yoloLayerV3 plugin: yolo_17
Deserialize yoloLayerV3 plugin: yolo_24

Runtime commands:
h: Print this help
q: Quit

p: Pause
r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
To go back to the tiled display, right-click anywhere on the window.

** INFO: <bus_callback:163>: Pipeline ready

Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
** INFO: <bus_callback:149>: Pipeline running

Creating LL OSD context new
id: -1 label[0]: car conf: 0.919336 bbox: [829 252 1035 367]
id: -1 label[0]: carplate conf: 0.974722 bbox: [856 329 876 336]
id: -1 label[0]: car conf: 0.964990 bbox: [829 250 1035 367]
id: -1 label[0]: car conf: 0.852407 bbox: [654 275 780 337]
id: -1 label[0]: carplate conf: 0.995340 bbox: [852 329 874 337]
id: -1 label[0]: car conf: 0.991833 bbox: [822 250 1042 365]
id: -1 label[0]: car conf: 0.850882 bbox: [355 269 481 349]
id: -1 label[0]: car conf: 0.799025 bbox: [664 273 784 335]
id: -1 label[0]: carplate conf: 0.999500 bbox: [850 329 872 338]
id: -1 label[0]: car conf: 0.962592 bbox: [818 254 1034 365]

Good luck!

1 Like

Thanks a lot KuanYong. It finally worked.

For anyone interested, it is possible to group the rectangles while keeping confidence values by using the cv::HOGDescriptor::groupRectangles function.

Here is my code :

void
NvDsInferContextImpl::clusterAndFillDetectionOutputCV(NvDsInferDetectionOutput &output)
{
    size_t totalObjects = 0;

    for (auto & list:m_PerClassCvRectList)
        list.clear();

    std::vector<std::vector<double>> confidences(m_NumDetectedClasses);

    /* The above functions will add all objects in the m_ObjectList vector.
     * Need to seperate them per class for grouping. */
    for (auto & object:m_ObjectList)
    {
        m_PerClassCvRectList[object.classId].emplace_back(object.left,
                object.top, object.width, object.height);
        confidences[object.classId].emplace_back(object.detectionConfidence);
    }

    for (unsigned int c = 0; c < m_NumDetectedClasses; c++)
    {
        /* Cluster together rectangles with similar locations and sizes
         * since these rectangles might represent the same object. Refer
         * to opencv documentation of groupRectangles for more
         * information about the tuning parameters for grouping. */
        cv::HOGDescriptor desc;
        if (m_PerClassDetectionParams[c].groupThreshold > 0)
            desc.groupRectangles(m_PerClassCvRectList[c],
                    confidences[c],
                    m_PerClassDetectionParams[c].groupThreshold,
                    m_PerClassDetectionParams[c].eps);
        totalObjects += m_PerClassCvRectList[c].size();
    }

    output.objects = new NvDsInferObject[totalObjects];
    output.numObjects = 0;

    for (unsigned int c = 0; c < m_NumDetectedClasses; c++)
    {
        /* Add coordinates and class ID and the label of all objects
         * detected in the frame to the frame output. */
        int i = 0;
        for (auto & rect:m_PerClassCvRectList[c])
        {
            NvDsInferObject &object = output.objects[output.numObjects];
            object.left = rect.x;
            object.top = rect.y;
            object.width = rect.width;
            object.height = rect.height;
            object.classIndex = c;
            object.confidence = confidences[c][i++];
            object.label = nullptr;
            if (c < m_Labels.size() && m_Labels[c].size() > 0)
                object.label = strdup(m_Labels[c][0].c_str());
            output.numObjects++;
        }
    }
}

Did the NvDsInferObject change? Because I’m getting: ‘struct NvDsInferObject’ has no member named ‘confidence’. EDIT: Ok, I see that it needs to be added to the object definition. I’m still getting following error:

nvinfer gstnvinfer.cpp:511:gst_nvinfer_logger: NvDsInferContext[UID 1]:log(): cuda/cudaScaleLayer.cpp (99) - Cuda Error in execute: 81 (PTX JIT compiler library not found)

Anyone any idea?

Hi,

This issue is related to the CUDA toolkit installation.

Could you share your environment and setup steps with us?
Are you using Jetson platform with sdkmanager installation.

Thanks.

Hi, This was after I did a make and make install of the nvdsinfer lib in /opt/nvidia/deepstream/deepstream-4.0/sources/libs/nvdsinfer/. I noticed that there were symlinks to a certain file (libnvidia-ptxjitcompiler.so.32.1.0) that didn’t exist anymore. I solved it by running:

cp /usr/lib/aarch64-linux-gnu/tegra/libnvidia-ptxjitcompiler.so.32.2.1 /usr/lib/aarch64-linux-gnu/tegra/libnvidia-ptxjitcompiler.so.32.1.0.

Hi,

Thanks for your feedback.
We will try to reproduce in our environment.

Thanks.

Hi,

We try to compile and install the nvdsinfer library and every works well in our side.
Please let us know if the issue occurs again.

Thanks.

If anyone on this thread is trying to get detection probability values from PGIE plugin, please refer to the patch below -

diff --git a/sources/apps/sample_apps/deepstream-test1/deepstream_test1_app.c b/sources/apps/sample_apps/deepstream-test1/deepstream_test1_app.c
index 13d0b72..f6ba273 100644
--- a/sources/apps/sample_apps/deepstream-test1/deepstream_test1_app.c
+++ b/sources/apps/sample_apps/deepstream-test1/deepstream_test1_app.c
@@ -70,6 +70,7 @@ osd_sink_pad_buffer_probe (GstPad * pad, GstPadProbeInfo * info,
         for (l_obj = frame_meta->obj_meta_list; l_obj != NULL;
                 l_obj = l_obj->next) {
             obj_meta = (NvDsObjectMeta *) (l_obj->data);
+            printf("Object class : %d Object probability %f |", obj_meta->class_id, obj_meta->confidence);
             if (obj_meta->class_id == PGIE_CLASS_ID_VEHICLE) {
                 vehicle_count++;
                 num_rects++;
@@ -79,6 +80,7 @@ osd_sink_pad_buffer_probe (GstPad * pad, GstPadProbeInfo * info,
                 num_rects++;
             }
         }
+        printf("\n");
         display_meta = nvds_acquire_display_meta_from_pool(batch_meta);
         NvOSD_TextParams *txt_params  = &display_meta->text_params[0];
         display_meta->num_labels = 1;
diff --git a/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt b/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt
index 708b8f1..a27c45b 100644
--- a/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt
+++ b/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt
@@ -70,8 +70,10 @@ num-detected-classes=4
 interval=0
 gie-unique-id=1
 output-blob-names=conv2d_bbox;conv2d_cov/Sigmoid
+enable-dbscan=1
 
 [class-attrs-all]
 threshold=0.2
-eps=0.2
-group-threshold=1
+eps=0.7
+minBoxes=3
+#group-threshold=1
diff --git a/sources/gst-plugins/gst-nvinfer/gstnvinfer_meta_utils.cpp b/sources/gst-plugins/gst-nvinfer/gstnvinfer_meta_utils.cpp
index d50c799..f455bb3 100644
--- a/sources/gst-plugins/gst-nvinfer/gstnvinfer_meta_utils.cpp
+++ b/sources/gst-plugins/gst-nvinfer/gstnvinfer_meta_utils.cpp
@@ -80,7 +80,7 @@ attach_metadata_detector (GstNvInfer * nvinfer, GstMiniObject * tensor_out_objec
     obj_meta = nvds_acquire_obj_meta_from_pool (batch_meta);
 
     obj_meta->unique_component_id = nvinfer->unique_id;
-    obj_meta->confidence = 0.0;
+    obj_meta->confidence = obj.detectionConfidence;
 
     /* This is an untracked object. Set tracking_id to -1. */
     obj_meta->object_id = UNTRACKED_OBJECT_ID;
diff --git a/sources/includes/nvdsinfer_context.h b/sources/includes/nvdsinfer_context.h
index d4cd776..57907dd 100644
--- a/sources/includes/nvdsinfer_context.h
+++ b/sources/includes/nvdsinfer_context.h
@@ -369,6 +369,8 @@ typedef struct
     int classIndex;
     /* String label for the detected object. */
     char *label;
+    /* detection confidence of the object */
+    float detectionConfidence;
 } NvDsInferObject;
 
 /**
diff --git a/sources/libs/nvdsinfer/nvdsinfer_context_impl_output_parsing.cpp b/sources/libs/nvdsinfer/nvdsinfer_context_impl_output_parsing.cpp
index d50bddc..f5f9a55 100644
--- a/sources/libs/nvdsinfer/nvdsinfer_context_impl_output_parsing.cpp
+++ b/sources/libs/nvdsinfer/nvdsinfer_context_impl_output_parsing.cpp
@@ -282,6 +282,7 @@ NvDsInferContextImpl::clusterAndFillDetectionOutputDBSCAN(NvDsInferDetectionOutp
             object.label = nullptr;
             if (c < m_Labels.size() && m_Labels[c].size() > 0)
                 object.label = strdup(m_Labels[c][0].c_str());
+            object.detectionConfidence = m_PerClassObjectList[c][i].detectionConfidence;
             output.numObjects++;
         }
     }

Once you have applied the patch above, follow these steps -

$ cd sources/libs/nvdsinfer/
$ sudo CUDA_VER=<add version here> make install
$ cd ../../gst-plugins/gst-nvinfer/
$ sudo CUDA_VER=<add version here> make install

Now the nvinfer plugin libraries have been updated to attach probability metadata, but keep in mind it’s currently available only for DBSCAN clustering algorithm. So this needs to be enabled in the PGIE config and the parameters need to be tuned for your test videos (Please refer to the patch above to see how to enable DBSCAN) The patch also modifies test1 app to show how to do this for the sample stream. To try it out follow these steps -

$ cd sources/apps/sample_apps/deepstream-test1/
$ CUDA_VER=<add version here> make
$ ./deepstream-test1-app ../../../../samples/streams/sample_720p.h264

You should now be able to see all the object probabilities along with the object count being printed on the console.

2 Likes

Trying to follow this patch with the most recent deepstream 4.02 and the build fails on the step above:

$ cd sources/libs/nvdsinfer/
$ sudo CUDA_VER=10.0 make install

The error is:

-fPIC -std=c++11 -I /usr/local/cuda-10.0/include -I ../../includes
g++ -c -o nvdsinfer_context_impl.o -fPIC -std=c++11 -I /usr/local/cuda-10.0/include -I ../../includes nvdsinfer_context_impl.cpp
In file included from nvdsinfer_context_impl.cpp:22:0:
nvdsinfer_context_impl.h:25:10: fatal error: opencv2/objdetect/objdetect.hpp: No such file or directory
 **#include <opencv2/objdetect/objdetect.hpp>**

compilation terminated.
Makefile:47: recipe for target 'nvdsinfer_context_impl.o' failed
make: *** [nvdsinfer_context_impl.o] Error 1

Solved the compilation issues. It seems that Deepstream 4.02 has moved the location of the opencv headers so I just needed to add the following the CFLAGS in the Makefile: “-I /usr/include/opencv4”.

Thanks for this patch. Is there a patch for clusterandFillDetectionOutputCV? e.g. can I add

object.detectionConfidence=m_ObjectList[c].detectionConfidence

in clusterandFillDetectionOutputCV (rather than DBSCAN as I’m trying to develop deepstream-app sample code)

clusterandFillDetectionOutputCV uses “Group Rectangles” algorithm from OpenCV which does not preserve confidence values in the clustering operation and hence it’s not been added.

This worked for deepstream 5.0.
Thank you!

1 Like