3D Object Detection on 3D Gaussian Splatting (3DGS) Assets (USDZ) in NVIDIA Omniverse

Omniverse Isaac Sim
1

Operating System

Ubuntu 24.04

GPU Information

  • Model: RTX 4090, RTX 3060

Detailed Description

Hello,

I am successfully working on utilizing 3D Gaussian Splatting (3DGS) derived assets within NVIDIA Omniverse. I have confirmed that I can convert the 3DGS results into a USDZ file (using tools like 3DGUT) and import it into Omniverse Kit/Create.

My core application goal involves performing 3D Object Detection (identifying and localizing objects, without classifying the type) directly on the reconstructed scene.

My question is as follows:

  1. 3D Object Detection on 3DGS: Is it technically possible to perform 3D Object Detection operations directly on the imported 3DGS representation (the Gaussian particles/point cloud) within Omniverse, or is the data structure inherently unsuitable for this type of geometric analysis?

  2. Required Extension/Workflow: If it is possible, what is the recommended Omniverse Extension or workflow required to enable real-time 3D object detection on 3DGS data? Does this require converting the 3DGS data into an explicit geometry like a Mesh or a dense PointCloud beforehand?

Any guidance on how to leverage the high-fidelity 3DGS reconstruction for real-time object detection and segmentation tasks in the Omniverse environment would be greatly appreciated.

Thank you.

3

@leo94 Which Isaac Sim and Omniverse version are you targetting?

4

Hello!

We noticed that this topic hasn’t received any recent updates from anyone reporting this issue, so we are closing it for now to help keep the forum organized.

If you are still experiencing this issue or have related questions, please create a GitHub Discussion or Issue in the Isaac Sim repository and include a link to this topic along with updated details. Mentioning or linking to this original topic provides helpful context and makes it easier for others to assist you.

Thank you for being part of the NVIDIA Isaac Sim community.

Best regards,
The NVIDIA Isaac Sim Forum Team