Rendering viewport from python script / service call

c.bickmeier · July 12, 2021, 11:10am

Hello,

I would like to implement a service call that renders the viewport and returns the image. The problem here is that capture_next_frame_rp_resource or capture_next_frame_rp_resource_callback are nonblocking but to return the image to the caller I have to wait for completion.

Here is my current code:

import ctypes
import numpy
import threading

event = None
imageData = None

def callback(capsule, length, width, height, textureFormat):
	print(textureFormat)
	
	ctypes.pythonapi.PyCapsule_GetPointer.restype = ctypes.c_void_p
	ctypes.pythonapi.PyCapsule_GetPointer.argtypes = [ctypes.py_object, ctypes.c_char_p]
	
	pointer = ctypes.pythonapi.PyCapsule_GetPointer(capsule, None)
	
	ap = ctypes.cast(pointer, ctypes.POINTER(ctypes.c_long * width * height))
	a = numpy.frombuffer(ap.contents)
	
	imageData = numpy.asarray(a)
	event.set()
	print("Rendering done")


event = threading.Event()

renderer = omni.renderer_capture.acquire_renderer_capture_interface()
viewport = omni.kit.viewport.acquire_viewport_interface()

viewportWindow = viewport.get_viewport_window()

viewportWindow.set_active_camera("/Stage/Cameras/Interactive")

viewport_ldr_rp = viewportWindow.get_drawable_ldr_resource()
renderer.capture_next_frame_rp_resource_callback(callback, viewport_ldr_rp)

#thread1 = threading.Thread(target=renderer.capture_next_frame_rp_resource_callback, args=(callback, viewport_ldr_rp))
#thread1.start()

print("Wait for rendering...")
event.wait(timeout = 10)
print("Done")

OK… my python skills are very limited (25 years of c#), so this is a complete miracle to me:

“Done” (after the wait for the event) is printed BEFORE “Rendering done”… so the event is not signaled (the execution is halted for the 10 seconds). Even starting the rending on another thread (commented lines) does not change anything… what am I doing wrong? Can someone help?

Thanks

Carl

c.bickmeier · July 12, 2021, 6:14pm

A small update to this…

I was unable to wait for the callback to finish, so I decided to write a small controller, that can accept the rendered image. This would mean the following process:

MyService → Kit-App-hosted-service → MyService

Inside the script editor of Create that worked as it should… here is the code:

import ctypes
import numpy
import urllib.request
import ssl
import json
import base64

def callback(capsule, length, width, height, textureFormat):
	ctypes.pythonapi.PyCapsule_GetPointer.restype = ctypes.c_void_p
	ctypes.pythonapi.PyCapsule_GetPointer.argtypes = [ctypes.py_object, ctypes.c_char_p]
	
	pointer = ctypes.pythonapi.PyCapsule_GetPointer(capsule, None)
	
	ap = ctypes.cast(pointer, ctypes.POINTER(ctypes.c_byte * length))
	
	req = urllib.request.Request('https://localhost:6001/api/session/071DCE5C-D984-412C-BE20-0F7A0159D374/AcceptRenderedImage')
	req.add_header('Content-Type', 'application/json')
	req.add_header('ImageWidth', width)
	req.add_header('ImageHeight', height)
	req.add_header('ImageFormat', str(textureFormat))

	gcontext = ssl.SSLContext()

	urllib.request.urlopen(req, base64.b64encode(ap.contents), context=gcontext)

renderer = omni.renderer_capture.acquire_renderer_capture_interface()
viewport = omni.kit.viewport.acquire_viewport_interface()

viewportWindow = viewport.get_viewport_window()

viewportWindow.set_active_camera("/Stage/Cameras/Interactive")

viewport_ldr_rp = viewportWindow.get_drawable_ldr_resource()
renderer.capture_next_frame_rp_resource_callback(callback, viewport_ldr_rp)

This code renders the current vieport, and then send the rendered image to my service (don´t ask why I use base64 string… I was unable to send alle the data drictly as binary data)…

The next step was to copy and paste the whole code into my kit extension and make it available as a service method…

@router.post("/render", description="Renders a session", summary="Renders the current session from a specified camera")
def get_render(data: dict = fastapi.Body(None)):
    def callback(capsule, length, width, height, textureFormat):
        ctypes.pythonapi.PyCapsule_GetPointer.restype = ctypes.c_void_p
        ctypes.pythonapi.PyCapsule_GetPointer.argtypes = [ctypes.py_object, ctypes.c_char_p]

        pointer = ctypes.pythonapi.PyCapsule_GetPointer(capsule, None)
        
        ap = ctypes.cast(pointer, ctypes.POINTER(ctypes.c_byte * length))
        
        req = urllib.request.Request('https://localhost:6001/api/session/071DCE5C-D984-412C-BE20-0F7A0159D374/AcceptRenderedImage')
        req.add_header('Content-Type', 'application/json')
        req.add_header('ImageWidth', width)
        req.add_header('ImageHeight', height)
        req.add_header('ImageFormat', str(textureFormat))

        gcontext = ssl.SSLContext()

        urllib.request.urlopen(req, base64.b64encode(ap.contents), context=gcontext)

    renderer = omni.renderer_capture.acquire_renderer_capture_interface()
    viewport = omni.kit.viewport.acquire_viewport_interface()

    viewportWindow = viewport.get_viewport_window()

    viewportWindow.set_active_camera("/Stage/Cameras/Interactive")

    viewport_ldr_rp = viewportWindow.get_drawable_ldr_resource()
    renderer.capture_next_frame_rp_resource_callback(callback, viewport_ldr_rp)

    return {"status": "success"}

So far so good… but this results in a hard crash of kit…

My assumption is that the callback is already disposed an cannot be called anymore… I don´t know if that is the problem, but if yes… all the work was useless, because then I would have to wait in this scenario too.

Any ideas?

Thanks

Carl

c.bickmeier · July 13, 2021, 1:48pm

The posted code works fine… I forgot to load a stage before requesting the rendering (Create runs headless in my scenario).

But anyway… a method that enables us to wait for completion would be nice for the future releases…

Thanks

Carl

avoroshilov · July 14, 2021, 7:24am

Hello Carl!
Thank you for trying out the capture code, and sorry for the inconveniences.
Looks like you’ve figured it all out, and the only question that remains (even more of a suggestion) - is to have a method that waits for completion.
I believe we do have that, if I understood your comment correctly. Here’s one example of how it can be achieved:

viewport_ldr_rp = viewportWindow.get_drawable_ldr_resource()
capture_interface.capture_next_frame_rp_resource_callback(callback, viewport_ldr_rp)

# Wait till the beginning of a next frame's update
await omni.kit.app.get_app().next_update_async()

# At this point, the callback should've been triggered, as it happens somewhere
# at the end of the previous frame where you filed that capture request.

To elaborate on that a little bit - a one-frame delay to actually get the request processed (you can only initiate capture of the current frame’s viewport contents when the frame is submitted to the GPU).

You don’t need the capture to be saved to storage, so you won’t get another source of async’ness which does file IO in a separate thread, so you won’t need set_capture_sync/wait_async_capture methods of capture_interface - wanted to mention that just in case.

Please let me know if you have any more questions!

c.bickmeier · July 26, 2021, 12:55pm

Hello Avoroshilov,

thank you for the hint regarding next_update_async.

But I think that does not work… to be honest I`ve not rebuild everything back to synchron communication, but I have another problem, I thought next_update_async could help to solve it.

My render method should be able to set the camera to a specified before rendering, I always get the rendering if the original camera (bur the camera is switched), so I thought waiting for the next frame could be the solution here to… but that´s not the case. Here is my code:

@router.post("/render", description="Renders a session", summary="Renders the current session from a specified camera")
async def post_render(data: dict = fastapi.Body(None)):
    renderer = omni.renderer_capture.acquire_renderer_capture_interface()

    viewportWindow = omni.kit.viewport.get_default_viewport_window()

    currentCameraPath = viewportWindow.get_active_camera()
    requestId = data.get("requestId", "00000000-0000-0000-0000-000000000000")
    acceptRenderedImageUri = data.get("acceptRenderedImageUri", "https://localhost:6001/api/session/00000000-0000-0000-0000-000000000000/AcceptRenderedImage")
    cameraPath = data.get("cameraPath", "/Stage/Cameras/Interactive")

    def renderImageIntoMemeryCallback(capsule, length, width, height, textureFormat):
        try:
            ctypes.pythonapi.PyCapsule_GetPointer.restype = ctypes.c_void_p
            ctypes.pythonapi.PyCapsule_GetPointer.argtypes = [ctypes.py_object, ctypes.c_char_p]

            pointer = ctypes.pythonapi.PyCapsule_GetPointer(capsule, None)
        
            ap = ctypes.cast(pointer, ctypes.POINTER(ctypes.c_byte * length))
        
            imageData = base64.b64encode(ap.contents)

            viewportWindow.set_active_camera(currentCameraPath)

            req = urllib.request.Request(acceptRenderedImageUri, method="POST")
            req.add_header('RequestId', requestId)
            req.add_header('ImageWidth', width)
            req.add_header('ImageHeight', height)
            req.add_header('ImageFormat', str(textureFormat))
            req.add_header('CameraPath', cameraPath)
            
            gcontext = ssl.SSLContext()

            urllib.request.urlopen(req, imageData, context=gcontext)
        except:
            print(sys.exc_info()[0])

    if currentCameraPath != cameraPath:
        viewportWindow.set_active_camera(cameraPath)
        await omni.kit.app.get_app().next_update_async()

    viewport_ldr_rp = viewportWindow.get_drawable_ldr_resource()
    renderer.capture_next_frame_rp_resource_callback(renderImageIntoMemeryCallback, viewport_ldr_rp)

    return {"status": "success"}

What do I have to do, to wait for the camera switch?

Even better would be a solution that creates a temp. second vieport and renders that… this way I could render different views in parallel and the user du not see a “flickering”, while switching the camera… is that possible?

Thanks

Carl

xDynames · October 26, 2021, 2:35am

I have been playing with changing an active viewport to a new camera and finding the same thing. The viewport’s activate camera is correctly changed but the image I get back form it is the same as the previous camera. I found I have a call to OmniKitHelper’s update() function after setting the viewport’s camera to something different.

Towards your idea of having a few viewports that can do asynchronous rending, here is some code I used to instance new viewports:

    def _get_camera_view(self, camera_name: str):
        camera_prim = "/".join([self._camera_prim, camera_name])
        viewport = self._get_viewport()
        viewport.set_active_camera(camera_prim)
        return viewport

    @staticmethod
    def _get_viewport():
        viewport_interface = omni.kit.viewport.get_viewport_interface()
        viewport_handle = viewport_interface.create_instance()
        return viewport_interface.get_viewport_window(viewport_handle)