Accessing camera from python

Omniverse Isaac Sim
#1

Hello all, I am working with a stereo setup on carter_v1 platform. I wanted to get the camera images without using the action graph. I wanted to directly have the rgb image, depth image from the viewport using some python sdk commands. I didn’t find anything that didn’t use ActionGraph along with ros_bridge. Please provide something that can directly help access the camera data from python itself. Regards, Nilesh

#2

Hi @nilesh3

You can use the SyntheticDataHelper class to access the camera data from Python (be aware that some methods are asynchronous)

import omni
from omni.kit.viewport.utility import get_active_viewport
from omni.isaac.synthetic_utils import SyntheticDataHelper

import cv2
import asyncio

sd_helper = SyntheticDataHelper()
viewport_api = get_active_viewport()

async def task():
    await sd_helper.initialize_async(["rgb", "depth"], viewport_api)
    gt = sd_helper.get_groundtruth(["rgb", "depth"], viewport_api, verify_sensor_init=False)
    rgb = gt["rgb"]
    depth = gt["depth"]

    cv2.imshow("rgb", rgb)
    cv2.imshow("depth", depth)
    cv2.waitKey(1)

asyncio.ensure_future(task())
#3

Thanks for the reply @toni.sm . I am using a stereo setup so I have got two viewports open. I can do the following to get the viewport api for both of them. But for some reason I am getting error on right_stereo_viewport_api line - ReferenceError: weakly-referenced object no longer exists

from omni.kit.viewport.utility import get_active_viewport_window
left_stereo_viewport_api = get_active_viewport_window("Viewport").viewport_api
right_stereo_viewport_api = get_active_viewport_window("Viewport 2").viewport_api

For reproducing the error I am attaching the carter_stereo.py file

carter_stereo.py (4.3 KB)

Run using -
./python.sh ~/Downloads/carter_stereo.py from the isaac_sim folder

#4

Hi @nilesh3

You can get the viewport API for specific windows using the get_viewport_from_window_name(...) function as follow:

import omni
from omni.kit.viewport.utility import get_viewport_from_window_name
from omni.isaac.synthetic_utils import SyntheticDataHelper

import cv2
import asyncio

window_names = ["Viewport", "Viewport 2"]

sd_helper = SyntheticDataHelper()
viewport_apis = [get_viewport_from_window_name(window_name) for window_name in window_names]
viewport_windows = [omni.ui.Workspace.get_window(window_name) for window_name in window_names]

for viewport_window, viewport_api in zip(viewport_windows, viewport_apis):
    if viewport_window is not None:
        viewport_window.visible = True
        viewport_api.set_active_camera("/OmniverseKit_Persp")


async def task():
    for viewport_api in viewport_apis:
        await sd_helper.initialize_async(["rgb", "depth"], viewport_api)
    
    for i, viewport_api in enumerate(viewport_apis):
        gt = sd_helper.get_groundtruth(["rgb", "depth"], viewport_api, verify_sensor_init=False)
        rgb = gt["rgb"]
        depth = gt["depth"]

        cv2.imshow(f"rgb ({i})", rgb)
        cv2.imshow(f"depth ({i})", depth)
        cv2.waitKey(1)

asyncio.ensure_future(task())
#5

Thanks @toni.sm . Is there any proper api documentation that you are referring? I didn’t find the documentation for the particular function you mentioned.

#6

Hi @nilesh3

I found the function by browsing the Python code available in the Isaac Sim application. I use that way quite frequently :)

1 Like
#8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.