I cannot get point cloud data at RL environment

Hi,

There is a problem with getting point cloud on a new version, 2023.

[Problem 1]
[omni.isaac.range_sensor.plugin] Lidar Sensor does not exist
The error message occurred and, I found that the error message appeared from

def warm_start(self):
    ....
    self._physx_interface.update_simulation(self.get_physics_dt(), 0.0)

at exts/omni.isaac.core/omni/isaac/core/physics_context/physics_context.py.
kit_20231117_031816.log (685.2 KB)

It occurred when I trained my RL code which worked well on ver.2022. You can check my attachment. I will work on OmniIsaacGymEnvs and I left only my custom files.
omniisaacgymenvs.zip (12.1 MB)

Thanks.
.
.

[Problem 2] SOLVED!
There is a point cloud segmentation example in the documentation (link). I modified it to standalone version (below attachment) and get the same results with documentation on ver.2022. However, it seems not working well on the new version and prints warning messages.

2023-11-16 18:55:18 [16,435ms] [Warning] [omni.isaac.range_sensor.python] Lidar get_semantic_data is deprecated and will not return any data, use get_prim_data and access semantics via usd

Anyway, I can get point cloud data even though the warning message appears but, I cannot get semantics.

(Please change .txt to .py)
point_cloud_seg.txt (3.5 KB)

[2023-11-17 added]
I can solve the second question by referring to documentation (link). I think the documentation has to be changed.

1 Like

I think the prim_path is not the reason for this problem.

I guess the simulator did not find any Lidar Sensor so if it has an empty np.ndarray value then it cannot convert the value.

I also checked the entire prim_path via the code below and then, entered the Lidar-related path but, it did not work.

stage = omni.usd.get_context().get_stage()

# Check if stage is valid
if stage:
# Iterate over all prims in the stage
for prim in stage.TraverseAll():
    # Print the prim path
    print(prim.GetPath())

@toni.sm @ahaidu I’m sorry, but can you shed some light on this issue?

I also tried not just using Lidar, which existed on USD, but also creating new Lidars. However, it does not work too.

@rthaker, @kellyg Would you mind checking the issue above?

There is a Lidar at each environment but, the error occurred.

Additionally, here is a suggestion about modifying rotating_lidar_physX.py at _data_acquisition_callback.

if key == "semantics":
### original code ###
# self._current_frame[key] = self._backend_utils.create_tensor_from_list(
#     self._lidar_sensor_interface.get_semantic_data(self.prim_path),
#     dtype="float32",
#     device=self._device,
# )
### original code ###
self._current_frame[key] = self._backend_utils.create_tensor_from_list(
    self._lidar_sensor_interface.get_prim_data(self.prim_path),
    dtype="float32",
    device=self._device,
)
else:
### original code ###
# self._current_frame[key] = self._backend_utils.create_tensor_from_list(
#     getattr(self._lidar_sensor_interface, "get_{}_data".format(key))(self.prim_path),
#     dtype="float32",
#     device=self._device,
# )
### original code ###
self._current_frame[key] = self._backend_utils.create_tensor_from_list(
    getattr(self._lidar_sensor_interface, "get_{}_data".format(key))(self.prim_path).astype(dtype='int32'),
    dtype="float32",
    device=self._device,
)
  1. From 2023, get_semantic_data is deprecated. I think it should be changed to get_prim_data.

  2. If there are any point cloud data, the type is assigned into numpy.unit16. It caused the shutdown Isaac Sim. I think it would be better to convert it into another data type.

Is there anyone who can help with this problem?

Have you solved this problem? I completely followed the code in the official document in the link below, but I can only obtain point cloud data and cannot obtain semantic ID of point cloud data.
Segment a Point Cloud

Hi there,

is your scene semantically labeled and did you set the enable_semantics=True flag?

Also, do you require LIDAR or annotator point cloud data?

Best,
Andrei

I have labeled the scene, and set the enable_semantics=True.
I want obtain the point cloud data and the semantic ID.
Segment a point cloud.txt (2.7 KB)

[Warning] [omni.isaac.range_sensor.python] Lidar get_semantic_data is deprecated and will not return any data, use get_prim_data and access semantics via usd

Perhaps it’s because that get_semantic_data is deprecated, but I don’t know how to use get_prim_data and access semantics via usd.

From warning, I think the tutorial snippet should be something like: prim_data = lidarInterface.get_prim_data("/World"+lidarPath) where prim_data is a list of paths to the prims that were hit.

You could then use this list with the bbox data annotator to access the IDs:

        "idToLabels": {<semanticId>: <semantic_labels>},    # mapping from integer semantic ID to a comma delimited list of associated semantics
        "bboxIds": [<bbox_id_0>, ..., <bbox_id_n>],         # ID specific to bounding box annotators allowing easy mapping between different bounding box annotators.
        "primPaths": [<prim_path_0>, ... <prim_path_n>],    # prim path tied to each bounding box

I will double check this and get back to you though.

Hi,

I checked the usable functions from the created instance, but the get_semantic_data is deleted. I think the get_prim_data is one of the alternatives. If you print it, every points have their prim data. Maybe you can use it and as I mentioned above, I recommend you modify the method _data_acquisition_callback from rotating _lidar_physX.py

I checked that the above works in an ordinary environment but does not work in an RL environment. I entirely changed the method of getting point cloud into replicator.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.