Semantic Segmentation Challenges in Projection Mapping: Success with Smiley Images, Issues with Others

[matsumoto-kai]

Hi,experts.
I am referring to the following link.
Replicator Materials — Omniverse Extensions latest documentation (nvidia.com)

I want to perform semantic segmentation on images projected through projection mapping. I am experimenting with various images, but there are images for which semantic segmentation is successful and others for which it is not.

It works well with images of smileys, but it seems to be unsuccessful with images other than smileys.

What do you think this issue depends on?
I tried the folloing images.

[ahaidu]

Hi there, is the Copper_ORM material semantically annotated?

[matsumoto-kai]

Here is the code.

from pxr import Usd, UsdGeom
from pathlib import Path
import omni.usd
import omni.replicator.core as rep

# Axis and unitscale can also be set optionally
#rep.settings.set_stage_up_axis("Z")
#rep.settings.set_stage_meters_per_unit(1.0)

#Create Light
distance_light = rep.create.light(rotation=(400,-23,-94), intensity=10000, light_type="distant")

camera = rep.create.camera(position=(0,280,310),rotation = (-45,0,0))
render_product = rep.create.render_product(camera, (1024, 1024))

stage = omni.usd.get_context().get_stage()
plane = rep.create.plane(position=(50, 100, 0), rotation=(0, 45, 10))
cube = rep.create.cube(visible=False, semantics=[('class', 'cube')], position=(0, 0, 0), rotation=(0, 0, 90), scale=(0.2, 0.2, 0.2))
sem = [('class', 'shape')]

# Randomizer for scattering
def get_shapes():
    shapes = rep.get.prims(semantics=[('class', 'cube')])
    with shapes:
        rep.randomizer.scatter_2d(plane)
    return shapes.node

rep.randomizer.register(get_shapes)

# Create the projection with the plane as the target
with plane:
    proj1 = rep.create.projection_material(cube, sem)

# Modify the cube position, and update the projection
with rep.trigger.on_frame(num_frames=10):
    rep.randomizer.get_shapes()
    with proj1:
        rep.modify.projection_material(diffuse="/home/nvidia/Documents/Copper_ORM.png")
        # rep.modify.projection_material(diffuse="/home/nvidia/Documents/smiley_normal.png")

There is no difference in the labeling state between using Copper_ORM and using smiley. Both have Semantic labels attached to Plane_Cube_Xform.

[ahaidu]

It seems the second material is projected on the whole plane, that is why the whole plane is semantically annotated (blue). If you for example remove the semantics from the cube it will disappear.

[matsumoto-kai]

Thank you for your response.

Is there a way to narrow down semantic segmentation only to the area of projection mapping, rather than projecting onto the entire plane?

I’m considering whether this example could be used for training appearance inspection of scratches on cars.

By the way, I was able to successfully perform semantic segmentation with the bird icon in the following link.
Is it important for the image used in projection mapping to be 512x512 pixels?

File:Icon Bird 512x512.png - Wikimedia Commons

[Simplychenable]

i wonder if the images with transparent alpha vs an image without alpha matters, also?

[pcallender]

Hi @matsumoto-kai

I believe the issue you’re running into is that the alpha (transparency) is taken from the diffuse/color image, and that some of the image you’re using don’t have an alpha channel. The png of the bird looks like it does though. I’ll make this requirement a bit more visible in the docs.

Question, what’s the preferred method ? Packing the alpha into the color image is about the closest thing to a “standard” for image transparency, but an optional input for a separate opacity might be nice to have (though the channel packing into the colormap is more efficient).

Welcome any thoughts on this.