Use replicator to get the pointcloud and idToLabels field


I’d like to create a camera to get the point clouds and segmentation masks using the replicators. I found the doc about Point Cloud, however, when I launch the following code:

from omni.isaac.kit import SimulationApp

headless = False
simulation_app = SimulationApp({"headless": headless})  # we can also run as headless.
import omni.kit
from pxr import UsdGeom

import omni.replicator.core as rep

stage = omni.usd.get_context().get_stage()

# Create cube
cube_prim = stage.DefinePrim("/World/Cube", "Cube")
UsdGeom.Xformable(cube_prim).AddTranslateOp().Set((0., 5., 1.))

# Create render products
rp = rep.create.render_product("/OmniverseKit_Persp", (1024, 1024))

# Get the annotators for the data
semantic_segmentation_annotator = rep.AnnotatorRegistry.get_annotator("semantic_segmentation")
pointcloud_annotator = rep.AnnotatorRegistry.get_annotator("pointcloud")
distance_to_camera_annotator = rep.AnnotatorRegistry.get_annotator("distance_to_camera")

# Attach annotators

# Access annotator data after each replicator process step
print("Instance segmentation: ", semantic_segmentation_annotator.get_data())
print("Point cloud: ", pointcloud_annotator.get_data())
print("Depth: ", distance_to_camera_annotator.get_data())

It generates an error:

 [Error] [omni.graph.core.plugin] Assertion raised in compute - cannot reshape array of size 0 into shape (0,newaxis)
  File "/home/ragi/.local/share/ov/pkg/isaac_sim-2022.1.1/exts/omni.replicator.core-1.4.3+lx64.r.cp37/omni/replicator/core/ogn/python/_impl/nodes/", line 19, in compute
    rgb = db.node.get_attribute("inputs:rgb").get_array(False, False, 0).reshape(width*height, -1)

This error does not appear if I remove the semantic_segmentation annotator but I need this annotator to be able to access to the field IdToLabels.
Is there a way to use one camera to get both the segmentation, idToLabel info and the point clouds without error ?

Hi @alempereur , the problem should already be solved in the most recent release.
One thing to note is that currently the pointcloud is only able to capture prims with semantic labels. In order to generate a pointcloud to the cube, you need to first assign a semantic label to that cube.