Holoviz/OpenGL Interoperability

Healthcare Holoscan SDK
,
1

Hello,
I’d like to integrate the Holoviz rendered video in our Qt-based GUI. Qt (QOpenGL) provides support for OpenGL but I was wondering how to exactly pass the off-screen rendered image (including overlays) from the Holoviz operator to a custom operator and map the gxf Framebuffer with the OpenGL framebuffer.

Are there any recommendations on how to achieve this? Help is much appreciated.

2

Hi,
you can configure the Holoviz operator to render in headless mode (set headless parameter to true) and enable render buffer output: Visualization - NVIDIA Docs. This will output a RGBA GXF VideoBuffer in GPU (CUDA) memory.

Create an operator using Qt and QtOpenGL, create an OpenGL texture, register it with CUDA (CUDA Runtime API :: CUDA Toolkit Documentation) and copy the VideoBuffer from CUDA memory to the OpenGL texture. Then draw the texture with OpenGL Qt.

3

Thanks for the rapid response.
I was trying exactly what you described in your answer but always run into the same error. Below you find an example which is based on holoviz_geometry.py.

Example:

class MyVideoProcessingApp(Application):

    def __init__(self, config_count=0):...

    def compose(self):
        width = 854
        height = 480
        video_dir = "/opt/nvidia/data/storage/video_data/endoscopy"
        
        if not os.path.exists(video_dir):
            raise ValueError(f"Could not find video data: {video_dir=}")
        source = VideoStreamReplayerOp(...

        image_processing = GeometryGenerationOp(...

        visualizer = HolovizOp(
            self,
            name="holoviz",
            width=width,
            height=height,
            headless=True,
            enable_render_buffer_output=True,
            tensors=[...,
        )
        self.add_flow(source, visualizer, {("output", "receivers")})
        self.add_flow(image_processing, visualizer, {("outputs", "receivers")})
        self.add_flow(image_processing, visualizer, {("output_specs", "input_specs")})
        
        write_buffer = WriteToOpenGLBufferOP(self, name="write_buffer")
        self.add_flow(visualizer, write_buffer, {("render_buffer_output", "render_buffer_input")})

class WriteToOpenGLBufferOP(Operator):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        
    def setup(self, spec: OperatorSpec):
        spec.input("render_buffer_input")
        
    def compute(self, op_input, op_output, context):
        import cupy as cp
        message_input = op_input.receive("render_buffer_input")
        #buffer_input = message_input.get("")
        
        #cp_array = cp.asarray(buffer_input)
        #print(cp_array.shape)

Error message:

[error] [entity.hpp:105] Unable to find component from the name 'render_buffer_output' (error code: 24)
4

Hi, unfortunately Holoviz outputs VideoBuffer objects instead of Tensor objects and VideoBuffer is not yet supported by the Python API of the Holoscan SDK.

For this to work the VideoBuffer needs to be converted to a Tensor, FormatConverterOp is doing that. Add the format converter between Holoviz and the WriteToOpenGLBufferOP like this:

        allocator = BlockMemoryPool(
            self,
            name="pool",
            storage_type=MemoryStorageType.DEVICE,
            block_size=width * height * 4,
            num_blocks=2,
        )
        converter = FormatConverterOp(self, name="converter", pool=allocator, out_dtype="rgba8888")
        self.add_flow(visualizer, converter, {("render_buffer_output", "source_video")})

        write_buffer = WriteToOpenGLBufferOP(self, name="write_buffer")
        self.add_flow(converter, write_buffer, {("tensor", "render_buffer_input")})