Lidar cannot detect the collision enabled model and cannot use semantic data on OIGE

[psh9002]

Hi,

I’m trying to use lidar on OIGE. However, I am stuck on two problems.
moving_target_point_cloud.7z (28.0 MB)

1. Lidar cannot detect the collision-enabled model.
   
   
   
   (I can see the detected dots when I check with a single USD file.↓)
   
   

---

2. When I use self._lidars.add_semantics_data_to_frame()function, the error appeared.

2023-06-26 05:00:30 [9,436ms] [Error] [carb.physx.python] TypeError: can't convert np.ndarray of type numpy.uint16. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.

At:
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.core/omni/isaac/core/utils/torch/tensor.py(31): create_tensor_from_list
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.sensor/omni/isaac/sensor/scripts/rotating_lidar_physX.py(177): _data_acquisition_callback
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py(469): step
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.core/omni/isaac/core/world/world.py(338): step
  /home/bak/anaconda3/envs/isaac-sim_22.2.1/lib/python3.7/site-packages/skrl/utils/omniverse_isaacgym_utils.py(215): step
  /home/bak/anaconda3/envs/isaac-sim_22.2.1/lib/python3.7/site-packages/skrl/envs/torch/wrappers.py(239): step
  /home/bak/anaconda3/envs/isaac-sim_22.2.1/lib/python3.7/site-packages/skrl/trainers/torch/base.py(167): single_agent_train
  /home/bak/anaconda3/envs/isaac-sim_22.2.1/lib/python3.7/site-packages/skrl/trainers/torch/sequential.py(75): train
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/OmniIsaacGymEnvsAILAB/omniisaacgymenvs/skrl_examples/moving_target_point_cloud/moving_target_train.py(147): <module>

2023-06-26 05:00:30 [9,457ms] [Error] [carb.physx.python] TypeError: can't convert np.ndarray of type numpy.uint16. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.

At:
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.core/omni/isaac/core/utils/torch/tensor.py(31): create_tensor_from_list
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.sensor/omni/isaac/sensor/scripts/rotating_lidar_physX.py(177): _data_acquisition_callback
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py(469): step
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.core/omni/isaac/core/world/world.py(338): step
  /home/bak/anaconda3/envs/isaac-sim_22.2.1/lib/python3.7/site-packages/skrl/utils/omniverse_isaacgym_utils.py(215): step
  /home/bak/anaconda3/envs/isaac-sim_22.2.1/lib/python3.7/site-packages/skrl/envs/torch/wrappers.py(239): step
  /home/bak/anaconda3/envs/isaac-sim_22.2.1/lib/python3.7/site-packages/skrl/trainers/torch/base.py(167): single_agent_train
  /home/bak/anaconda3/envs/isaac-sim_22.2.1/lib/python3.7/site-packages/skrl/trainers/torch/sequential.py(75): train
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/OmniIsaacGymEnvsAILAB/omniisaacgymenvs/skrl_examples/moving_target_point_cloud/moving_target_train.py(147): <module>

2023-06-26 05:00:30 [9,473ms] [Error] [carb.physx.python] TypeError: can't convert np.ndarray of type numpy.uint16. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.

At:
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.core/omni/isaac/core/utils/torch/tensor.py(31): create_tensor_from_list
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.sensor/omni/isaac/sensor/scripts/rotating_lidar_physX.py(177): _data_acquisition_callback
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py(469): step
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.core/omni/isaac/core/world/world.py(338): step
  /home/bak/anaconda3/envs/isaac-sim_22.2.1/lib/python3.7/site-packages/skrl/utils/omniverse_isaacgym_utils.py(215): step
  /home/bak/anaconda3/envs/isaac-sim_22.2.1/lib/python3.7/site-packages/skrl/envs/torch/wrappers.py(239): step
  /home/bak/anaconda3/envs/isaac-sim_22.2.1/lib/python3.7/site-packages/skrl/trainers/torch/base.py(167): single_agent_train
  /home/bak/anaconda3/envs/isaac-sim_22.2.1/lib/python3.7/site-packages/skrl/trainers/torch/sequential.py(75): train
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/OmniIsaacGymEnvsAILAB/omniisaacgymenvs/skrl_examples/moving_target_point_cloud/moving_target_train.py(147): <module>

2023-06-26 05:00:30 [9,496ms] [Error] [carb.physx.python] TypeError: can't convert np.ndarray of type numpy.uint16. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.

At:
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.core/omni/isaac/core/utils/torch/tensor.py(31): create_tensor_from_list
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.sensor/omni/isaac/sensor/scripts/rotating_lidar_physX.py(177): _data_acquisition_callback
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py(469): step
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.core/omni/isaac/core/world/world.py(338): step
  /home/bak/anaconda3/envs/isaac-sim_22.2.1/lib/python3.7/site-packages/skrl/utils/omniverse_isaacgym_utils.py(215): step
  /home/bak/anaconda3/envs/isaac-sim_22.2.1/lib/python3.7/site-packages/skrl/envs/torch/wrappers.py(239): step
  /home/bak/anaconda3/envs/isaac-sim_22.2.1/lib/python3.7/site-packages/skrl/trainers/torch/base.py(167): single_agent_train
  /home/bak/anaconda3/envs/isaac-sim_22.2.1/lib/python3.7/site-packages/skrl/trainers/torch/sequential.py(75): train
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/OmniIsaacGymEnvsAILAB/omniisaacgymenvs/skrl_examples/moving_target_point_cloud/moving_target_train.py(147): <module>

How can I solve those problems?

Thanks

[mcarlson1]

I’ll try to pass this to someone else in the team that knows more, but is there any way you can get the data to be of type float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool instead of uint16? That data type is not supported.

[psh9002]

I change the data type to int64 via

self._current_frame[key] = self._backend_utils.create_tensor_from_list(
                           self._lidar_sensor_interface.get_semantic_data(self.prim_path).astype(int),
                           dtype="float32",
                           device=self._device,
                           )

from exts/omni.isaac.sensor/omni/isaac/sensor/scripts/rotating_lidar_physX.py,
def _data_acquisition_callback(self, step_size: float) -> None:

However, there is no semantics informations.

I think the main problem is the ‘Lidar cannot detect the collision-enabled model.’ which is I mentions first.

[psh9002]

I changed LidarView to below.

class LidarView(RotatingLidarPhysX):
    def __init__(self, prim_paths: str, name: str = "lidar") -> None:
        super().__init__(prim_paths, name,
                         rotation_frequency = 0.0,
                         fov = [70, 50],
                         resolution = [1, 1],
                         valid_range = [0.1, 1.0]
                         )
        self._lidar = RotatingLidarPhysX(prim_path=prim_paths, name=name,
                                         rotation_frequency=0.0,
                                         fov=[70, 50],
                                         resolution=[0.5, 0.5],
                                         valid_range=[0.1, 1.0])
        
        self._lidar.add_depth_data_to_frame()
        self._lidar.add_point_cloud_data_to_frame()
        self._lidar.add_semantics_data_to_frame()
        self._lidar.enable_visualization(high_lod=True,
                                         draw_points=False,
                                         draw_lines=True)
        self._lidar.initialize()

But it still not works. Here is my code.
moving_target_point_cloud.zip (14.4 MB)

[kellyg]

Hi there, if you are running from the OIGE framework, can you try setting the enable_scene_query_support variable in the task config file to True?

[psh9002]

Thanks! It works.

How can I check the detail information about the configurations?