Is there an overlay drawing API?

Omniverse General Discussion
1

Are there any plans to provide an overlay drawing API?

I’m looking into the Extension implementation as I learn.
In the Skeleton drawing, lines are drawn as an overlay.
I see this representation done with “omni.anim.skelJoint” “omni.anim.skelvis” “omni.kit.tool.paint” etc. and the drawing part seems to be done on the C side (ONI).
And I don’t know how to implement it because I don’t have the source for the C side.

“omni.debugdraw” is similar, but it refers to Depth (not overlay).

I am trying to implement this part, but there is a level of difficulty.
For example, I would like to do the following (This is a dummy).

tube_img_20210720
tube_img_20210720779×393 34.6 KB

I’m currently looking for an implementation using PIL and “omni.ui.ByteImageProvider”, but it seems to be a bottleneck in memory transfer and not general purpose.

2

Hi
We are currenly working on that API, we are hoping that we can include it in the apps coming in the Fall
but today as you mentioned there is not Public API to do that.

Using PIL and ImageWithProvider will certainly have speed issue but is an idea. the main difficulties is around getting projected position from 3D into the correct space in 2D, there are some extensions doing some of that, search for viewport_widgets_manager

using that you might be able to blend some omni.ui and viewport overlay that could work but again the real API will be easy to use etc but for that you need to wait future version

1 Like
3

Thank you for your consideration! I’m looking forward to it.

As for the 3D to 2D conversion, I was able to implement it programmatically.
I also found viewport_widgets_manager helpful.
The next step is to improve the speed, which I will wait for the implementation of the Overlay API.

4

you have been doing lots of really great work, love to see your update on twitter :)

Might be worth getting on a call to review your ImageWithProvider there are some ways that this can be made better.
let me know if you are interesting I can DM you

5

Thanks for checking in on me on Twitter!

I’ll describe it for the sake of sharing.
In my current implementation, I am passing the data as follows.

from PIL import Image, ImageDraw, ImageFont
import itertools

# default format omni.ui.TextureFormat.RGBA8_UNORM.
_byteProvider = omni.ui.ByteImageProvider()

# Create PIL image.
_PILImage = Image.new("RGBA", (_width, _height), (0, 0, 0, 0))

# Drawing to PIL...

# Get image data(RGBA buffer) from PIL.
imgD = _PILImage.getdata()

# Copy PIL to ByteImageProvider.
byte_data = list(itertools.chain.from_iterable(imgD))
_byteProvider.set_bytes_data(byte_data, [_width, _height])

# ImageWithProvider
omni.ui.ImageWithProvider(_byte_provider, width=_width, height=_height)

Since PIL and ByteImageProvider(RGBA8_UNORM) have the same sequence of data, this was the fastest in my implementation.

If there is a method that seems more efficient than this, please let me know.
I’ll send you a DM when I do ;)