Integrate other applications to Holoviz

Healthcare & Life Sciences Holoscan SDK
1

Hi,
I’ve developed two applications with QT/QML and GLFW. I want these two applications as overlay on endoscopy video. Is there possible to display QT/QML and GLFW applications on Holoviz which has displayed endoscopy video streaming?
Thanks!

2

This may be difficult. Assuming the QT and GLFW apps are running ins separate processes, both apps would need to run in offscreen mode, read back the frame buffer, pass it to the Holoscan process to pass it into the Holoviz operator. Also when the apps have an UI the mouse position, clicks and keyboard needs to be passed to the apps somehow. This is not possible today.
You could try to use the desktop compositor in some way, keeping the Holoviz window in the background and the other apps on top, probably borderless.
But overall we’d say it’s not possible to overlay apps on Holoviz today.

3

Thanks for the clarification.
If the QT/QML and GLFW applications don’t need mouse and keyboard for user interaction, is there possible for Holoscan reading back the frame buffer and showing the GUI?
And are there any recommendations for showing GUI (ImGUI?) and 3D model using Holoscan SDK?

4

Some initial answers to the questions: with our next SDK release it’s possible to freely place input images within the Holoviz window. If the QT/QML and GLFW frame buffers can be read back, then Holoviz can composite them to one window. ImGui can be used with the Holoviz C++ API, but it not supported with the Holoviz operator.

5

Good to hear that. And how can QT/QML and GLFW frame buffers be read back? Will it be implemented in the future release?

6

Not sure for QT/QML, but for GLFW it depends on the API used, when using OpenGL glReadPixels can be used. That image then needs to be converted to a gxf VideoBuffer or Tensor and passed to Holoviz. There will be a Python Holoviz example in the next SDK release (coming soon) showing how to use layer views. (C++ is similar)

7

Looking forward to the new feature.

8

The example on using layer views in v0.6: https://github.com/nvidia-holoscan/holoscan-sdk/blob/main/examples/holoviz/python/holoviz_views.py