Get sensor data from isaac sim

I want to write a function to get sensor data for kaya example,i find when i put the
gt = sd_helper.get_groundtruth(
[
“rgb”,
“depth”,
“boundingBox2DTight”,
“boundingBox2DLoose”,
“instanceSegmentation”,
“semanticSegmentation”,
“boundingBox3D”,
“camera”,
]
)
in function,and push the button ,the simulation dead.

@2934661830 I would recommend the following:

  1. If you want to use Isaac Sim UI and get sensor data, use the Synthetic Data Recorder extension. This extension enables offline data generation of various synthetic data sensor output locally. You can also visualize your sensor data in Isaac Sim UI using the Synthetic Data Visualizer extension.

  2. If you want to use a standalone python sample to get sensor data and generate offline data, refer to the Offline Dataset Generation python sample. This enables running Omniverse Isaac Sim directly from our main Python training loop and get sensor data such as rgb, bounding boxes, segmentation, depth, etc. The setup instruction can be found here.

The sd_helper is designed to work with the second approach and hence running it within the Isaac Sim UI will cause issue.

@sdebnath
Are there plans to add a sd_helper that works with an extension or the UI in the upcoming release? I have an extension where I am using the sd_helper to generate a segmented point cloud each time step and then displaying it. But due to something with the app updates or some asyncronousity in the UI framework or sd_helper the pose of the camera and the depth map is not exactly in sync, so the point cloud becomes offset if the camera is moving while the point cloud is generated. Is this a known issue? It would be nice if the sd_helper grabbed the pose, depth, and rgb in a snapshot similar to the camera data that is sent over the bridges for rostopics/isaac_sdk…

The sd helper should work with a ui based extension, we will add a test to make sure that if you capture data on a specific frame, it is all in sync. We did fix a few bugs related to this for the next release.

One workaround is to put a small wait via time.sleep(0.1) right before get_groundtruth to ensure all of the gpu buffers are copied to their final locations, this can cause off-by-frame issues in some cases and adding the small sleep helps