Originally published at: https://developer.nvidia.com/blog/developing-a-pallet-detection-model-using-openusd-and-synthetic-data/
By iteratively developing with synthetic data, our team developed a pallet detection model that works on real-world images.
Hi, could you share the code based on USD Scene Construction Utilities that was used to generate the pallet scene for Replicator?
Interesting - especially as I’ve been pursuing the almost identical task! Some thoughts:
- Replicator is limited - as any tool of course. I’ve found that it’s useful to extend Replicator to:
- Add ‘defects’ at an internal USD level to provide a wider range of ‘real’ pallets
- Create composite objects - combining multiple USD objects in addition to your core object (such as a pallet), e.g., adding boxes, shrink wrap, etc.
- For detection, I knew the small range of pallets that were possible, e.g., it’s a Euro 1, or GMA, or CHEP for example. I then successfully used a key points ML approach followed by PNP to get the pallet pose. Very accurate, and supports occlusions. (Sub cm accuracy, sub degree for yaw.)
- Synthetic only goes so far of course. I had access to a Motion Capture lab for accurate, ‘real’ data. Definitely, definitely needed beyond synthetic. I’d say synthetic is great to get started with - it’s cheap, and probably useful for hard-to-find situations over time - but one quickly gets to a point where it’s got to be primarily ‘real’. (Whole Sim2Real discussion, eh?)
Love to take offline if interested.
FYI - here’s a recent Medium paper on dithering the USD internals (of a CAD version of a GMA pallet), including a github for code… A (simple) Tool for Creating Better Synthetic Images for ML Training | by Peter Gaston | Sep, 2023 | Medium
Example of dithering the internals of a USD object - colors exaggerated for emphasis… The front-to-back boards and the front top/bottom boards are slightly translated/rotated.
Here’s an example of pose - using a combination of initially synthetic data followed by ‘real’ data…
would you like to share the project how to create the Synthetic Data?