Cannot get correct image from camera sensors when camera look straight down

Robotics & Edge Computing Robotics - Isaac Isaac Gym
1

Hi team,
I cannot get correct image from camera sensors when camera look straight down. I set camera_location = (0.35, 0.65, 0.75) and camera_lookat =(0.35, 0.65, 0), and I got an all-black picture.
However, when I set camera_location = (0.35, 0.65, 0.75) and camera_lookat =(0.35, 0.651, 0), with just slightly offset, I got the correct image.
What is this about?

2

Could you tell us a bit more about the application that you’re using, which camera, and what parts of Isaac SDK you are exercising? The extrinsic calibration should not be affecting the image you’re receiving unless I misunderstood what you are describing.

3

I have the same issue. Manually setting the transform with Gym.get_camera_transform works. However, if a camera is configured to look straight down with Gym.set_camera_location, then the view matrix returned by Gym.set_camera_view_matrix contains nan values and the image will be all black.

4

I’ve encountered the same issue. The gym.set_camera_location returns nan values in the view matrix; using gym.set_camera_transform is a good workaround. Seems like there might be something off with the calculation of the transform given the camera position and camera target when facing straight down.