Data annotation (segmentation, bounding box) for defects with non-opaque (Opacity < 1.0) texture/material

Hello @dennis.lynch , @pcallender

I am trying to make defect images with ringmarks that have non-opaque texture, something that looks like a blurred ring shape.
(I have attached the diffuse map png file for your reference)

However, when I project the said diffuse map on to a target mesh, it is visible in the Viewport/LdrColorDisplay but data annotation does not seem to be working and as you can see below, the SemanticSegmentationDisplay is blank.

So I decided to take a different approach where I created a transparent disk-shaped mesh with the said diffuse map as its material, applied semantic data to the mesh, and placed it on the desired coordinate of the target mesh surface.
Still nothing shows up on the SemanticSegmentationDisplay as shown below.

According to the documentation for Replicator, it is mentioned that “Enable Opacity must be true, and the opacity map is what controls the opacity. Segmentation is true for values of 1.0. Values below do not show segmentation.
Replicator - Annotating with Transparent Materials — Omniverse Extensions latest documentation (nvidia.com)

So when I tried adjusting the Opacity Threshold, rendering parts of the non-opaque ringmark diffuse map to become opaque, I started seeing some segmentation being rendered on the SemanticSegmentationDisplay.

According to my little experiment, in order to get data annotated images with non-opaque diffuse map as defects:
(1) I would have to first create transparent meshes with the said non-opaque texture(defect diffuse map)
(2) output the image png file in the first frame
(3) adjust the opacity threshold value so that the non-opaque texture becomes opaque
(4) output the segmentation data in the second frame
(5) repeat the above procedure until finished

Is there any other way to get the segmentation data as output for non-opaque defect diffuse maps?

Also, is it possible to use the “projection” method for these types of defect diffuse maps? It seems like I am unable to adjust the opacity threshold for ProjectPBR materials.

Any advice would be much appreciated!

Thanks!

Hi @jaeyeun

Your assessment here is correct. There is a limitation for segmentation, where the opacity needs to be 1.0. We do have a feature request for adjusting the clip value, but I have no estimation on when/if this would be addressed.

The projection method CAN potentially help here. The projection technique creates a duplicate, transparent mesh, with the projection material on it. That projection material can be custom, where you blend an identical material from the underlying object, with the projected texture.

In the example below, the projection is a normal only effect, but its getting applied to a copy of the same material as the object underneath the projection shell mesh.

I’ll send you this example directly and you can get a closer look at how I achieved this effect. (it will go up to github also, but unsure of the timeline)

1 Like