Store Unique Detections from CNN model

Omniverse Isaac Sim
,
1

Hi,
I have made a custom dataset using omniverse replicator and then used that to fine tune faster RCNN model. I followed this guide
https://www.youtube.com/watch?v=OdbRQnz2Btc
for implementing object detection in my simulation.
Now I have a conveyor, on top of which I have placed a camera. I am using the image from that camera and passing it into the model for inference.The model outputs a result dictionary which contains labels, bbox coordinates, score. Since I have a continuous camera feed and objects are coming on the conveyor, I am getting multiple entries for the same object. Say if I have a can on the conveyor then I am getting atleast 10-15 entries till the can goes out of camera’s frame of view.
I want to save the unique detections in an array as I have to use that data further up in the pipeline. I am at my wits end, I have tried using Bounding Box similarity as a factor to store unique detections, I have tried assigning unique ids to the detections, I have also tried using Bounding Box corner coordinates as a way to filter out unique detections. But all of that fails if I have say 2 cans simultaneously behind each other coming in on the conveyor, then the array takes only saves that as 1 detection and this messes up the further pipeline.
I would like to know if there is a better way to do this task or if I am doing something wrong. Any help would be much appreciated.

PS. I have used ScriptNodes to implement this logic

TLDR: I have a fine tuned Faster RCNN model, being used for object detection of objects coming in on a conveyor through a camera placed at the top. I want to save the unique detections from the camera feed in an array (camera feed is continuous, there could be two objects with same detected label back to back, they should be considered as 2 unique detections)

2

Hello!

I’m glad you found my stream useful! I have several questions. Hopefully I can help!

  1. Is your goal the generation of this data set? Or are you working on deploying the tuned model you created? If the former, are you familiar with instance segmentation? You should be able to access the instance segmentation as one of the available data frames from our camera API. https://www.youtube.com/watch?v=vLk-f9LWj48

  2. you mentioned that downstream you need only one detection for that object, but you get many entries as the object moves through the frame. Is there a reason why you can’t just select one of the entries to work with?

thanks!
Gus