Can Isaac Sim do target detection and tracking？I hope to achieve target detection and tracking in the simulation environment
Thank you, Isaac Sim engineer. My English level is not high and I can’t hear you clearly on the phone. I hope to get your help in the email, my email is 845004988@qq.com.
please explain more what you want to do
I put a fixed camera inside Isaac Sim, and I set up a scene with objects passing by. I want to be able to use object detection (YOLOv5) for identification in this scenario and Deepsort for tracking. Is there a way I can port my deep learning framework for object detection and tracking into Isaac Sim?
I want to build a sorting platform for express packages in Isaac Sim. According to visual perception, target detection, tracking and pose estimation of express packages are carried out, and finally path planning is carried out by using its location information.
Now I want to build a sorting platform with VS Code Link and load deep learning methods (Yolov5 and DeepSort) into Isaac Sim to do it.
In addition, I would like to ask another question. I want to load many omnidirectional wheel module units. Can Isaac Sim support the movement of many omnidirectional wheel units? Can it support the path planning of express parcels on a platform composed of these omnidirectional wheel units?
Thank you very much for your help and technical support. We hope to use Isaac Sim to realize our graduation design, and we hope that our work can finally openly help more people who want to use Isaac Sim.
Hello! Thank you for your interest in using Isaac Sim.
You can apply deep learning training, indeed. You’d need to extract the image buffer from the cameras and pass them on to your network. Here’s an example on a reinforcement learning application using stable baselines.
Regarding the simulation of multiple omni-wheels, yes, that is possible to be simulated, and you can have several ones with little performance on impact when you use GPU simulation. Depending on the size of your simulation, you may even get good performance running physics on CPU. We have a Kaya sample robot that contains 3 omni-wheels with 10 rollers each, and I can achieve over 120 fps for a single robot on CPU, but it scales nicely when you add multiple robots on gpu. On a quick test I was able to get 16 robots running at approx 60fps on my pc using gpu dynamics. My computer is a 10900k core i-9 with a RTX 3090.
If you are training your networks at the same time the simulation is running, you should expect some perf drop, but it should definitely be achievable.