How to get proabbilities from all classes, not only the detected one

I’ve been working with deepstream 5.0 over Jetson Xavier NX for a couple of months. I have a current pipeline besed in a yolov4 onnx model running over an rtsp input and I use a custom bbox parsing function and I can see the result bboex in the output video as exepected. I’m also saving the frames corresponfing to detections.

The problem is I need to get information not only about the probability of the detected class but also the probability for all the classes for the same bounding box. Is it possible to do it?

If I could gather all the bounding box information available after the NMS clustering operation it would be great.

Any help?

Thanks

Hi santiago.garcia1,

Sorry for the late reply, is this still an issue to support?

Thanks

which NMS will you use?
The NMS in DeepStream - /opt/nvidia/deepstream/deepstream-5.0/sources/objectDetector_Yolo/nvdsinfer_custom_impl_Yolo

or

The NMS in TRT plugin - https://github.com/NVIDIA/TensorRT/tree/master/plugin/nmsPlugin

Hi,

Yes this is still an issue.

Thanks

I’m now using a custom NMS function implemented in the cpp bounding box parser (very similar to the one used here: https://towardsdatascience.com/how-to-deploy-onnx-models-on-nvidia-jetson-nano-using-deepstream-b2872b99a031

I don’t know if I can add from the bbox parser, new fields to the object metadata…

Thanks!

Hi @santiago.garcia1,
If you implement your own NMS, I assume you should be familar with the NMS output and know how to get “the probability for all the classes for the same bounding box”. And, does the pobability mean “confidence scores”?

Yes, I mean “confidence scores” for each class. I’m perfectly able to access these values from custom bbbox parser function but I want to share that information with next steps in deepstream pipeline, specifically with the OSD probe. The thing is I want to save that information in an external DB and I want to push the information from OSD prob function.

Hi @santiago.garcia1,
Each object has a corresponding structure - NvDsInferParseObjectInfo{}, you can set the value “detectionConfidence” as “confidence scores" in this structure, and the object will be linked into objectList, and then in meta data.
The meta data will be passed to following components, then you can parse it in the following components, e.g. OSD.

You can refer to /opt/nvidia/deepstream/deepstream/sources/objectDetector_FasterRCNN/sandstar_custom_impl_ctdet/nvdsparsebbox_ctdet.cpp

Yes, I guess that is the key to solve my problem. Just need to understand how it works and add it to my own parse function.

Thanks!

1 Like

Hi, you mean swap de detectionConfidence with confidence score? What I need is to add extra parameter from the parse bbox function. I know the fields detectionConfidence, classId, left, top, width and heigh are able to fill with any information I need (with several requirements). The problem is when I want to add extra fields to be recognized as object meta from OSD functions.

-Is there any way to easily add new custom fields from parse bbox function?
-Is it possible to easily modifying one of the available fields (e.g. detectionConidence) in order to add (for instance) a float list instead of a single float.

Thanks

Sorry for long delay!
I’m not clear about what you ask. What is OSD functions you mean? What’s the extra data?

Ok let me explain the specific case:

I have a pipeline where I use the nvinfer plugin tu run inference over a stream using a Yolov4 ONNX model. The way I’m parsing the bboxes from yolo inference is using a custom c++ bbox parse function.

Within that function, I’m able to access several parameters from inference: Bbox metadata (top, left, width, height), scores for all probabilitites, objectness value…

The system is prepared for filling the fields detectionConfidence, classId, left, top, width and heigh by adding new bounding box proposals with this information, however I can only “push” those 5 fields. My problem is that I want to more information, for instance I would like to push the scores for all the classes instead of only pushing the score for the most probable class, but I don’t know how to do this.

With OSD functions I mean the probes that NVosd use for drawing the bounding boxes over the frame and to work in general with frame-meta and object-meta data.

Thanks!

not very sure your question, however, you may could the fields reserved from user in the metadata structure