Originally published at: https://developer.nvidia.com/blog/developing-a-pallet-detection-model-using-openusd-and-synthetic-data/
By iteratively developing with synthetic data, our team developed a pallet detection model that works on real-world images.
Hi, could you share the code based on USD Scene Construction Utilities that was used to generate the pallet scene for Replicator?
Interesting - especially as I’ve been pursuing the almost identical task! Some thoughts:
- Replicator is limited - as any tool of course. I’ve found that it’s useful to extend Replicator to:
- Add ‘defects’ at an internal USD level to provide a wider range of ‘real’ pallets
- Create composite objects - combining multiple USD objects in addition to your core object (such as a pallet), e.g., adding boxes, shrink wrap, etc.
- For detection, I knew the small range of pallets that were possible, e.g., it’s a Euro 1, or GMA, or CHEP for example. I then successfully used a key points ML approach followed by PNP to get the pallet pose. Very accurate, and supports occlusions. (Sub cm accuracy, sub degree for yaw.)
- Synthetic only goes so far of course. I had access to a Motion Capture lab for accurate, ‘real’ data. Definitely, definitely needed beyond synthetic. I’d say synthetic is great to get started with - it’s cheap, and probably useful for hard-to-find situations over time - but one quickly gets to a point where it’s got to be primarily ‘real’. (Whole Sim2Real discussion, eh?)
Love to take offline if interested.
FYI - here’s a recent Medium paper on dithering the USD internals (of a CAD version of a GMA pallet), including a github for code… A (simple) Tool for Creating Better Synthetic Images for ML Training | by Peter Gaston | Sep, 2023 | Medium
Example of dithering the internals of a USD object - colors exaggerated for emphasis… The front-to-back boards and the front top/bottom boards are slightly translated/rotated.
Here’s an example of pose - using a combination of initially synthetic data followed by ‘real’ data…
would you like to share the project how to create the Synthetic Data?
Hi @jwitsoe,
In the blog, you mentioned the GitHub repo that contains the model. However, it seems that no one has beeen given access after requesting for it . I believe the repo might be unmaintained. Could you help us get access to it?
Thank you!
Hi @jwitsoe and NVIDIA Developer Team,
Just following up on previous requests regarding the SDG pallet detection ONNX model. Several users, including myself, have been trying to access the model mentioned in the GitHub repository, but the download link remains inaccessible.
If the repository is deprecated or if access is now restricted, could you kindly confirm the status? If possible, please share an updated download link or point us to an alternate pre-trained model that supports pallet detection (especially one compatible with 1280x720 input resolution).
As a final-year student working on a computer vision-based pallet detection system for my academic project, I am in urgent need of a working ONNX model to continue development and testing. This model will be used strictly for research and educational purposes, not commercial deployment.
I would greatly appreciate any support or guidance you can provide.
Thank you again for your time and support!
Best regards,
Subramaniyan
Final Year B.E Robotics and Automation
Sri Ramakrishna Engineering College
Email: subramaniyan.2210051@srec.ac.in