Capture frames from camera via Python scripting

Hello, I was wondering how can we manage camera via python scripting in a simulation? I mean, do you have an example of a python script for Omniverse which loads a camera into the scene and can process frames captured?
Thanks in advance

Hi…

There is an example about capture synthetic data from sensors like rgb, depth, instanceSegmentation, boundingBox3D, camera, etc. in the next path. Also, check the README to setup the environment.

/isaac-sim/_build/linux-x86_64/release/exts/omni.isaac.samples/omni/isaac/samples/scripts/syntheticdata

You can read something about it in Synthetic Data

1 Like

Also, this code can be help.
It captures a RGB image from the current camera and write it in /isaac-sim folder

import cv2
import numpy as np
import omni.syntheticdata

sd = omni.syntheticdata._syntheticdata
sdi = sd.acquire_syntheticdata_interface()

def _get_sensor_data(sensor, dtype):
    width = sdi.get_sensor_width(sensor)
    height = sdi.get_sensor_height(sensor)
    row_size = sdi.get_sensor_row_size(sensor)

    get_sensor = {
        "uint32": sdi.get_sensor_host_uint32_texture_array,
        "float": sdi.get_sensor_host_float_texture_array,
    }
    return get_sensor[dtype](sensor, width, height, row_size)
    
data = _get_sensor_data(sd.SensorType.Rgb, "uint32")
image = np.frombuffer(data, dtype=np.uint8).reshape(*data.shape, -1)

cv2.imwrite("/isaac-sim/image.png", image)
print(image)

One way to run it is to have an opened scenario (e.g. /Isaac/Samples/Leonardo/Stage/ur10_bin_filling.usd), write the code in the Script Editor tab (Window > Script Editor) and enable the RGB sensor in Synthetic Data Sensors tab like the next image

After that, you can run the script (Script Editor > Command > Execute) and you can see the stored image like the next… Note: the color difference is because OpenCV works with BGR color instead of RGB color space (use image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR) to fix it)

2 Likes

According to Synthetic Data examples, sensor buffers need at least two frames to be correctly populated.

In the example, the next code is used before request sensor data

for _ in range(2):
    app.update(0.0)  # sensor buffers need two frames to be correctly populated.

where

import omni.kit.app
app = omni.kit.app.get_app_interface()

but, acquiring the app interface with get_app_interface() and call app.udpate(0.0) will freeze the kit… Why? I don’t know… just try to execute the script several times until the sensor data be populated

1 Like

Ok, I solved just selecting the camera from the Stage.

Just another quick question, for adding a new camera also directly via Python script?

You can add a camera as a primitive of type “Camera”.
For example, the next code creates a camera and move it to +100 in Z-axis

import omni
from pxr import UsdGeom

stage=omni.usd.get_context().get_stage()

path="/World/Camera/MyCamera"
prim_type="Camera"
translation=(0,0,100)

camera_prim = stage.DefinePrim(path, prim_type)

xform_api=UsdGeom.XformCommonAPI(camera_prim)
xform_api.SetTranslate(translation)

You can use pxr.UsdGeom.XformCommonAPI to rotate, scale and translate the primitive…

1 Like

Ok, got it, and last thing (for real this time), how can I get data from this camera using the script you provided before for syntheticdata?

Also, regarding the camera parameters, if I wanted to have the instrinsic I should assume fx=fy=focalLength, but for cx and cy?

To get the data from the new camera, just change the active camera to the new using the kit.viewport module like:

vpi = omni.kit.viewport.get_viewport_interface()
vpi.get_viewport_window().set_active_camera(str(camera_prim.GetPath()))

Also, you can find more info about the Omniverse Kit Python bindings in the manual here

You should use for fx, fx, cx, and cy the values obtained from a previous calibration. But if you cannot do a calibration, you can use for cx, cy the width/2.0 and height/2.0 respectively (note that this is not true for all cameras)

1 Like

Ok for the intrinsics. I can assume cx and cy as you said and fx and fy using the focalLenght parameter, or do a proper calibration if I need more precise results.

Regarding the data from the new camera ok for the viewport, I’ll dig more into the documentation also to learn something more, but after setting the active camera I don’t know how to use the previous script, which uses the synthetic data interface. But here i don’t have one from which select for example the RGB sensor.

Ok, I enabled the omni.synthetic extensions from the menu and now I can use the omni.syntheticdata directly in my application. I just need to know how to enable the RGB sensor directly from python

I found all I needed in the syntheticdata script that you suggested. You have been very helpful, thank you very much ;)

Thanks a lot Toni. You have been super helpful.