Gym.render_all_camera_sensors(sim) always causes segmentation fault (simple script to reproduce)

Robotics & Edge Computing Robotics - Isaac Isaac Gym
1

Hi,

I have found that enabling camera sensors and calling gym.render_all_camera_sensors(sim) results in a repeatable segmentation fault.

Below is a standalone script to reproduce the issue (just a slightly modified version of the multiple_camera_envs.py example). At step 1239 of the simulation, breakpoint() is called before gym.render_all_camera_sensors (). Stepping pdb (entering n) will result in segmentation fault. Is there a fix for this, or is it just impossible to use Isaac Gym with camera sensors in its current state?

UPDATE: I noticed that not running in headless mode and calling gym.draw_viewer(viewer, sim, False) makes the issue go away. Of course, I would like a way to solve this in headless mode.

UPDATE 2: It appears that this issue only occurs when there multiple camera sensors per environment.

UPDATE 3: It appears that the number of iterations until segmentation fault is exactly inversely proportional to the total number of cameras (num envs X cameras per env). If the number of cameras is doubled, it will take half the number of iterations to fault.

UPDATE 4: The time per iteration is twice as fast when the viewer is enabled compared to headless mode. Something is definitely wrong here!

UPDATE 5: Verified that this issue exists in Preview Release 3 as well.

My script’s diff from examples/multiple_camera_envs.py:

-gym.simulate(sim)
-gym.fetch_results(sim, True)
+for i in range(2000):
+    gym.simulate(sim)
+    gym.fetch_results(sim, True)
+
+    gym.step_graphics(sim)
+    if i == 1239:
+        breakpoint()
+    gym.render_all_camera_sensors(sim)

Full script:

"""
Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.

NVIDIA CORPORATION and its licensors retain all intellectual property
and proprietary rights in and to this software, related documentation
and any modifications thereto. Any use, reproduction, disclosure or
distribution of this software and related documentation without an express
license agreement from NVIDIA CORPORATION is strictly prohibited.


Multiple cameras with multiple environments example
--------------------
Demonstrates ability to instantiate multiple cameras within multiple
environments in a scene
"""
import os

import numpy as np
from isaacgym import gymapi
from isaacgym import gymutil

gym = gymapi.acquire_gym()

# parse arguments
args = gymutil.parse_arguments(description="Multiple Cameras Example",
                               custom_parameters=[
                                   {"name": "--save_images", "action": "store_true", "help": "Write RGB and Depth Images To Disk"},
                                   {"name": "--up_axis_z", "action": "store_true", "help": ""}])

sim_params = gymapi.SimParams()
plane_params = gymapi.PlaneParams()

sim_params.up_axis = gymapi.UP_AXIS_Y
if args.up_axis_z:
    sim_params.up_axis = gymapi.UP_AXIS_Z

sim_params.use_gpu_pipeline = False
if args.use_gpu_pipeline:
    print("WARNING: Forcing CPU pipeline.")

sim = gym.create_sim(0, 0, args.physics_engine, sim_params)
if sim is None:
    print("*** Failed to create sim")
    quit()

gym.add_ground(sim, plane_params)

# create envs
n_envs = 2
spacing = 1
lower = gymapi.Vec3(-spacing, 0.0, -spacing)
upper = gymapi.Vec3(spacing, spacing, spacing)
num_per_row = int(np.sqrt(n_envs))

env_ptrs = [gym.create_env(sim, lower, upper, num_per_row) for _ in range(n_envs)]
env_idxs = [i for i in range(n_envs)]

camera_props = gymapi.CameraProperties()
camera_props.horizontal_fov = 75.0
camera_props.width = 1920
camera_props.height = 1080


# track cameras
ch_map = {idx: {} for idx in env_idxs}

pos = gymapi.Vec3(1.38, 1.0, 0)
name = 'cam0'

for env_idx in env_idxs:
    env_ptr = env_ptrs[env_idx]
    ch = gym.create_camera_sensor(env_ptr, camera_props)
    ch_map[env_idx][name] = ch
    gym.set_camera_transform(ch, env_ptr, gymapi.Transform(p=pos, r=gymapi.Quat()))
    print("Added {} with handle {} in env {} | View matrix: {}".format(name, ch, env_idx, gym.get_camera_view_matrix(sim, env_ptr, ch)))

pos = gymapi.Vec3(0.5, 9.0, .8)
name = 'cam1'

for env_idx in env_idxs:
    env_ptr = env_ptrs[env_idx]
    ch = gym.create_camera_sensor(env_ptr, camera_props)
    ch_map[env_idx][name] = ch
    gym.set_camera_transform(ch, env_ptr, gymapi.Transform(p=pos, r=gymapi.Quat()))
    print("Added {} with handle {} in env {} | View matrix: {}".format(name, ch, env_idx, gym.get_camera_view_matrix(sim, env_ptr, ch)))

if args.save_images and not os.path.exists("multiple_camera_images"):
    os.mkdir("multiple_camera_images")

gym.render_all_camera_sensors(sim)

for env_idx in env_idxs:
    env_ptr = env_ptrs[env_idx]

    name = 'cam0'
    ch = ch_map[env_idx][name]
    rgb_filename = "multiple_camera_images/rgb_env%d_cam0.png" % (env_idx)
    if args.save_images:
        gym.write_camera_image_to_file(sim, env_ptr, ch, gymapi.IMAGE_COLOR, rgb_filename)

    print("View matrix of {} with handle {} in env {}: {}".format(name, ch, env_idx, gym.get_camera_view_matrix(sim, env_ptr, ch)))

    name = 'cam1'
    ch = ch_map[env_idx][name]
    rgb_filename = "multiple_camera_images/rgb_env%d_cam1.png" % (env_idx)

    if args.save_images:
        gym.write_camera_image_to_file(sim, env_ptr, ch, gymapi.IMAGE_COLOR, rgb_filename)

    print("View matrix of {} with handle {} in env {}: {}".format(name, ch, env_idx, gym.get_camera_view_matrix(sim, env_ptr, ch)))

for i in range(2000):
    gym.simulate(sim)
    gym.fetch_results(sim, True)

    gym.step_graphics(sim)
    if i == 1239:
        breakpoint()
    gym.render_all_camera_sensors(sim)

I am using Preview Release 4 on Ubuntu 20.04 with a Quadro RTX 6000, NVIDIA driver version 530.30.02, and CUDA version 12.1.

1 Like
2

This might help the situation add this after the render_all_camera_sensors: self.gym.start_access_image_tensors(self.sim)
This pulls the rendered image out of the GPU into memory

3

I met exactly the same problem. Have anyone solved this?