How to get some image data from Isaac Sim

Robotics & Edge Computing Robotics - Isaac Isaac Gym
1

Hi. I have used Isaac Gym.
I want to get some image data from Camera of Isaac Sim. And I want to use these image data as observations of agents(where agents are jetbots).
So I wrote this code.

self.cameras = []
        for i in range(self.num_jetbots):
            prim_path = "/World/Jetbot" + str(i) + "/chassis/rgb_camera/jetbot_camera"
            name = "jetbot_camera_" + str(i)
            camera = Camera(
                prim_path=prim_path,
                name=name,
                resolution=(256,256),
                translation=np.array([0.0, 0.0, 0.04]),
                orientation=rot_utils.euler_angles_to_quats(np.array([0, -0.5, 0]))
            )
            camera.initialize()
            camera.add_motion_vectors_to_frame()
            rep.orchestrator.stop()
            self.cameras.append(camera)

My porpose is getting images from cameras of jetbots.
But I got Errors bellow…

2023-07-29 07:53:31 [13,692ms] [Warning] [omni.isaac.core.utils.nucleus] /persistent/app/omniverse/mountedDrives setting not found
2023-07-29 07:53:31 [13,692ms] [Warning] [omni.client.plugin]  HTTP Client: provider_http: CC-493: Request through cache failed. Retrying without cache for http://omniverse-content-production.s3-us-west-2.amazonaws.com/.cloudfront.toml
2023-07-29 07:53:31 [13,692ms] [Warning] [omni.client.plugin]  HTTP Client: omniclient: CC-873: Bypassing cache until the application is restarted
2023-07-29 07:53:31 [13,954ms] [Warning] [omni.client.plugin]  HTTP Client: provider_http: CC-493: Request through cache failed. Retrying without cache for http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2022.2.1/
cameras len 16
2023-07-29 07:53:32 [14,764ms] [Warning] [carb.flatcache.plugin] UsdRelationship /Render/RenderProduct_Replicator.orderedVars has multiple targets, which is not supported

2023-07-29 07:53:38 [20,626ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2023-07-29 07:53:38 [20,626ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2023-07-29 07:53:38 [20,626ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2023-07-29 07:53:38 [20,626ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2023-07-29 07:53:38 [20,626ms] [Error] [gpu.foundation.plugin] Failed to update params for RenderOp 1155
2023-07-29 07:53:38 [20,626ms] [Error] [gpu.foundation.plugin] Failed to update params for RenderOp Cached PT ClearAll. Will not execute this or subsequent RenderGraph operations. Aborting RenderGraph execution
2023-07-29 07:53:38 [20,626ms] [Error] [carb.scenerenderer-rtx.plugin] Failed to execute RenderGraph on device 0. Error Code: 7
2023-07-29 07:53:41 [23,692ms] [Warning] [omni.client.plugin]  Tick: authentication: Discovery(ws://localhost/omni/discovery): Error creating Api/Connection search: Not connected
/home/usr4/yamane/.local/share/ov/pkg/isaac_sim-2022.2.1/python.sh: line 41: 10074 Segmentation fault      (core dumped) $python_exe "$@" $args
There was an error running python

Do you have any ideas to solve this problem?
Thanks!

3

The following worked for me:

However I had to enable viewports and camera in headless mode following toni’s post here

comment the lines 42-43 in

    exts/omni.isaac.gym/omni/isaac/gym/vec_env/vec_env_base.py to avoid loading omni.isaac.sim.python.gym.headless.kit experience in headless mode
       experience = ""
        # if headless:
        #     experience = f'{os.environ["EXP_PATH"]}/omni.isaac.sim.python.gym.headless.kit'

run the script with the following arguments:

    headless=True +task.sim.enable_viewport=True task.sim.enable_cameras=True


# in setup scene function ------------------------

from omni.kit.viewport.utility import get_active_viewport_window

from omni.isaac.core.utils.extensions import enable_extension

enable_extension("omni.isaac.synthetic_utils")

from omni.isaac.synthetic_utils import SyntheticDataHelper

self.sd_helper = SyntheticDataHelper()

self.viewport = get_active_viewport_window("Viewport").viewport_api


# In get observation function ------------------------

# itterate over numenvs

# some_counter_i ++

self.viewport.set_active_camera("/World/envs/env_"+str(some_counter_i)+"/path to camera in robt usd file")
gt = self.sd_helper.get_groundtruth(["rgb", "depth",],self.viewport,)

depth = gt["depth"]
rgb = gt["rgb"]