Accessing ground truth in isaac lab

Omniverse Isaac Sim
1

Isaac Sim Version

4.2.0

Isaac Lab Version (if applicable)

1.2

Detailed Description

I am trying to train a model in Isaac Lab that is an oracle (i.e., somehow gets the ground truth from the simulator rather than perceptive inputs). In particular, I am trying to figure out how, given a mesh or object, to pass the entire point cloud (or some other equivalent “complete representation”) as an observation. Any ideas of how to do this? I cannot find anything that does this in the example code. Thanks!

3

You may need Replicator writers and annotators. An example of using a writer in Isaac Lab is here.

4

Thanks for the quick reply! I realize I didn’t quite phrase my question that clearly.

It seems the tools available in Lab and Replicator (some of which you cited) are for producing data from the perspective of a particular sensor (e.g., the point cloud of an object as seen by a camera). What I am instead trying to do is obtain the complete point cloud (or some other equivalent representation of the entire object). Is there any way to do this natively? As I see it, the options are:

  1. Place cameras in different positions, sample point clouds from each, and do sensor fusion (not great).
  2. Use some external library to compute the point cloud given an object file (also won’t work well).

So to summarize:

  1. Am I missing any tooling or something in the documentation that does what I am describing?
  2. Is there any way to query the Sim itself? Surely it has some “under the hood” notion of where different points are in space (this skips the need for point clouds and is exactly such an “equivalent representation”) ?
5

Our apologies for the delayed response. Currently, one option is to use python’s trimesh library:

# Sample points from the surface of the mesh
num_points = 10000  # Number of points to sample
point_cloud = mesh.sample(num_points)

Another possible option may be to try this.

For future reference and to ensure efficient support and collaboration, please submit your topic to its GitHub repo following the instructions provided on Isaac Lab’s Contributing Guidelines regarding discussions, submitting issues, feature requests, and contributing to the project.

We appreciate your understanding and look forward to assisting you.