Camera example and headless mode

Robotics & Edge Computing Robotics - Isaac Isaac Gym
,
1

Hi.
I’m developing an RL environment with a robot, target object and a camera using isaacGym.
My goal is to make an agent which takes an image as an observation, so I’m using get_camera_image_gpu_tensor() API with other related ones.

But when I switched on the headless mode(headless=True), I failed to get images using get_camera_image_gpu_tensor() API. I know this is because the sim doesn’t render the environment in headless mode. But I want to find more faster and optimal way.

My question is…
Q1: Are there any official examples using image observation?
Q2: Is there any way to get image observations through camera APIs by switching on the headless mode(headless=True) for a more faster rollouts?

Thank you

2 Likes
2

Hi @twkim0812,

In this release, we don’t have RL training environments that use camera sensors. But we have Python examples, using GPU pipeline: interop_torch.py and slightly more detail, but without using GPU pipeline - graphics.py and either of them should work in a headless mode.

1 Like
3

Thank you for your reply !

1 Like
4

I’m having the same issue – did you ever figure out how to fix it?

5

I was able to fix the issue. self.gym.create_camera_sensor(...) returned -1 when I trained in headless mode. I realized that in headless mode, my graphics_device_id arg to create_sim(...) was automatically set to -1. Setting it equal to the device_id fixed the issue for me!

3 Likes
6

Worked for me. Thanks!

1 Like
7

Actually… now when I run in headless mode with the “fix” I mentioned above, my camera images are all zeros. Perhaps the rendering just isn’t working in headless mode.

1 Like
8

I had a similar problem when I put my camera too far away from any visible object. Also, the ground was only visible when the camera was above it. Maybe this will help you.

10

I believe I have found the root of the problem. In step(...) in base_task.py, there is the code:

    # to fix!
    if self.device == 'cpu':
        self.gym.fetch_results(self.sim, True)

I fixed my issue by removing the if statement and always calling self.gym.fetch_results().

2 Likes
11

I also encountered this problem，I set it equal to device_id solved the first problem that I can run in headless model. But I find my camera image is just the ground. I try to always calling self.gym.fetch_results() but the image I get is slso just the ground. Do you find another solution? Thanks.

12

May be I find one way to solve this problem:
add these codes before self.gym.get_camera_image_gpu_tensor()

self.gym.simulate(self.sim)
self.gym.fetch_results(self.sim, True)
self.gym.step_graphics(self.sim)
self.gym.render_all_camera_sensors(self.sim)
self.gym.start_access_image_tensors(self.sim)

4 Likes
13 
    # comment below two lines to enable depth camera utility 
    # if self.headless == True:
    #     self.graphics_device_id = -1