The best way to collect pixels from viewport/camera in an extension

Omniverse General Discussion
,
1

Looking at the camera api, it’s not direct to find the best way to collect pixels from the viewport/camera inside an extension … did you already tested it ?

First looking at Movie Capture — Omniverse Create documentation (nvidia.com)

2

Hi @didjich! Thanks for your question. I have contacted the team for more help here. I will post back when I have more information!

3

Hi @didjich , if you want to an API to save what’s in the viewport into a file, you can use the omni.kit.capture extension. You can find it under “\kit\exts\omni.kit.capture” of your Create package. And of course you can refer to the Movie Capture extension to see how to use omni.kit.capture. The Movie Capture extension is located under “\kit\exts\omni.kit.window.movie_maker” of your Create package. Hope this helps!

4

Is there a way to capture the view from an arbitrary camera? Not the viewport.

5

Thanks @quchen .
Effectively, enabling the extension “Kit Image and Video Capture” and exploring the code is the way I take actually, but maybe something like a node or another way could be available too.

(A remark that could help users, you have a button to directly open the code like in this image therafter when looking to an extension … very quick and easy to reach this code.

image
image1224×451 80 KB

(consider this remark too How to render an image? - #4 by didjich when inside CODE 2022.1.0 Beta )

6

Yeah the Open button is handy. So the function to actually capture an image is this one:

def _capture_viewport(self, frame_path: str):
        viewport_window = self._viewport_interface.get_viewport_window(None)

        def _capture_helper(viewport_rp_resource):
            self._renderer.capture_next_frame_rp_resource(frame_path, viewport_rp_resource)
        omni.kit.viewport_legacy.deferred_capture(viewport_window, _capture_helper, is_hdr=self.options._hdr_output, subscription_name="omni.kit.capture capture helper")

And to set up viewport resolution, renderer properties, you will need to check out the self._settings.set_XXX calls, generally in the _prepare_viewport(self) function. It’s setting all renderers together now which is not very clear which setting is for which renderer. I’m refactoring it to make it clearer.
Hope this helps!

1 Like
7

@quchen After some test, I go to use the omni.syntheticdata extension.

It provides both C++ and python bindings that allow users to extract ground truth data from scenes loaded and rendered …
here https://docs.omniverse.nvidia.com/prod_extensions/prod_extensions/ext_synthetic-data.html

8

see here too about Replicator Issac Sim Replicator Generate Synthetic data avaliability - #4 by didjich

9

A remark in connection with this post… Simple render USD scene to PNG - #11 by didjich