Huynn
May 1, 2024, 6:08am
1
Hi everyone, I want to simulate a RGB-D camera in Isaac Gym, for example an Intel realsense cam. But I don’t know how to start. There aren’t any document for this. Please help me !
cevi9
May 9, 2024, 11:02am
2
Hi,
cevi9
May 10, 2024, 2:51pm
4
This is the only source I could find so far. Maybe it can help you.
- [How to use point cloud or RGB/RGBD image as input](#How_to_use_point_cloud_or_RGB/RGBD_image_as_input)
- [A brief introduction to the example](#A_brief_introduction_to_the_example)
- [Performance analysis](#Performance_analysis)
# How to use point cloud or RGB/RGBD image as input
We often train RL with visual input in Isaac Gym and have tried it in Bi-dexhands. But the problem is that the parallelism of Isaac Gym's cameras is not very good. It can only obtain images one by one env serially, which will greatly slow down the running speed. At the same time, the training of the dexterous hand is very difficult and greatly depends on the high sampling efficiency, so we do not use the input of other modalities. But this is undoubtedly very important, so we provide an example for other modal (point cloud) inputs for future exploration or development, see [here](https://github.com/PKU-MARL/DexterousHands/blob/main/bi-dexhands/tasks/shadow_hand_point_cloud.py). Below is the a brief introduction.
## A brief introduction to the example
First, we need to set the camera properties:
```python
self.camera_props = gymapi.CameraProperties()
self.camera_props.width = 256
self.camera_props.height = 256
self.camera_props.enable_tensors = True
self.env_origin = torch.zeros((self.num_envs, 3), device=self.device, dtype=torch.float)
self.pointCloudDownsampleNum = 768
self.camera_u = torch.arange(0, self.camera_props.width, device=self.device)
show original
My current idea is to use pictures or a super slow frame rate for the camera data.
Huynn
May 10, 2024, 3:58pm
5
Omg, really appreciate bro !