Omniverse viewport Camera Capture

I am trying to get the byte code from the camera on Isaac Sim using the following
capture = viewport_api.schedule_capture(ByteCapture(on_capture_completed))
def on_capture_completed(buffer, buffer_size, width, height, format):
print(f’PixelData resolution: {width} x {height}‘)
print(f’PixelData format: {format}’)

The buffer return type is a PyCapsule class. How do we convert this to a suitable JPG image or a byte array that can be futher used for object detection.

Hi @ratan.s.murali. Welcome to the forums! There’s a much easier watch to do this with omni.kit.viewport.utility.capture_viewport_to_file(). Video and sample code here: NVIDIA Omniverse Developer Office Hour - 12/9/2022 | NVIDIA On-Demand

If you need more customization, you can look at the implementation of that function which uses viewport_api.schedule_capture under the hood too.

1 Like

@ratan.s.murali you can do like this:

    ctypes.pythonapi.PyCapsule_GetPointer.restype = ctypes.POINTER(ctypes.c_byte * image_size)
    ctypes.pythonapi.PyCapsule_GetPointer.argtypes = [ctypes.py_object, ctypes.c_char_p]
    content = ctypes.pythonapi.PyCapsule_GetPointer(capsule, None) # your PyCapsule object
except Exception as e:  # pylint: disable=broad-except
    carb.log_error(f"[XXX] Failed to get capture buffer: {e}")
    pointer = ctypes.cast(content, ctypes.POINTER(ctypes.c_long *image_size))
    np_arr = np.frombuffer(pointer.contents)           
    imageData = np.asarray(np_arr)       
    image = Image.frombytes(mode="RGBA", size=(width, height), data=imageData) # save as jpg or byte array
1 Like

Hi mati @mati-nvidia , I have done some work similar like this . I use omni.kit.viewport.utility.capture_viewport_to_buffer to capture images to do a object detection work and then render these images to my Extension UI with a ui.ImageWithProvider().

In the process, I found that frame rate especially low in viewport:

my code(object detection work(about 20ms) and render images to UI )is run in capture_viewport_to_buffe’s on_captrue_fn.

Is there a solution to solve it?or I need a better computer to do this.

Thank you very much @ mmmmark. This code was helpful and solved the problem.

Thanks mati-nvidia. This worked perfectly. Where can I find all the API listing and their doc / examples ( if any)

Is there an example to display an image using ui.ImageProvider? I wish to use the Image.tobytes() or an numpy.ndarray to display the image within a UI.

Yup. Here you go:
Video: NVIDIA Omniverse Developer Office Hour - 11/4/2022 | NVIDIA On-Demand
Image: developer-office-hours/ at main · mati-nvidia/developer-office-hours · GitHub

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.