Add pointwise semantic labels to imported .obj file

Omniverse Synthetic Data Generation (SDG)
1

Hello,

i’ve just starting using Omniverse and am a little overwhelmed.
My goal is to generate a multiview synthetic Dataset based on the full view 3D dataset Anomaly Shapenet. So that when i render a specific pose i can also get the GT mask, that shows where the anomaly is located. This Dataset includes objects (.obj and .pcd) and corresponding GT data.

I managed to import the .obj files, but have no idea how i can ad the labels to the GT data (binary class for each point of the mesh).
Is it possible to read in a modified .obj file, where there is an extra entry for each point, that shows the class?
Or what is the recommended method to add the semantic classes of a mesh to the generated usd file?

Thank you in advance

Best Regards
Ruben

2

@ruben2048 From a high level, it sounds like you want to build a library from the obj assets. If I were to try and do this, I would convert the objs to usd and save those to a directory for loading later in a replicator script.

There’s probably a few approaches to adding the labels:

  1. First there’s the manual way, which would be going into each usd manually and adding the labels.
  2. Second you could write a script to iterate through the usds and label them automatically if you have a source of data (csv, json, etc) to pull the relevant labels to the obj/usd and apply them.
  3. Third is that in the replicator script when generating the data, you could apply the semantics at runtime.

Adding Semantics to a Scene — Omniverse Extensions latest documentation (nvidia.com)

This page details how to do this by hand (#1) and in a replicator script at runtime (#3).

If you wanted to do this in an automated script to batch through a bunch of assets (#2) you would probably want to use pure python and USD/Kit. I put together this example of spawning a cube and a cone, and applying semantics to them.

Pay special attention to add_prim_semantics() which comes from the import from semantics.schema.editor import add_prim_semantics, LabelWriteType

Full script here. Again the below is not a replicator script, but is suitable for batching with pure python.

I hope this helps!

from semantics.schema.editor import add_prim_semantics, LabelWriteType
import omni
from pxr import Gf, UsdGeom

def set_transform(mesh_prim, prim_pos, prim_rot, prim_scale):
    stage = omni.usd.get_context().get_stage()
    xform = UsdGeom.Xformable(mesh_prim)
    xform_ops = {op.GetBaseName(): op for op in xform.GetOrderedXformOps()}
    
    for op in xform.GetOrderedXformOps():
         if "translate" in op.GetName():
            translate_op = xform_ops["translate"]
            translate_op.Set(Gf.Vec3d(prim_pos[0],prim_pos[1],prim_pos[2]))
         elif "rotate" in op.GetName():
            rotate_op = xform_ops["rotateXYZ"]
            rotate_op.Set(Gf.Vec3d(prim_rot[0],prim_rot[1],prim_rot[2]))
         elif "scale" in op.GetName():
            scale_op = xform_ops["scale"]
            scale_op.Set(Gf.Vec3d(prim_scale[0],prim_scale[1],prim_scale[2]))

def spawn_mesh(mesh_type, pos, rot, scale):
    # I decided to make my own function to spawn basic meshes in this example. Using rep, its much simpler
    # but I wanted to show an example that doesn't generate ANY omnigraph nodes.
    
    stage = omni.usd.get_context().get_stage()
    mesh_prim = []
    
    if mesh_type == "Cube":
        result, path = omni.kit.commands.execute("CreateMeshPrimCommand", prim_type="Cube")
        mesh_prim = stage.GetPrimAtPath(path)
        set_transform(mesh_prim, pos, rot, scale)
    elif mesh_type == "Cone":
        result, path = omni.kit.commands.execute("CreateMeshPrimCommand", prim_type="Cone")
        mesh_prim = stage.GetPrimAtPath(path)
        set_transform(mesh_prim, pos, rot, scale)
    return mesh_prim

# Meshes too - cone, floor, wall1, and wall2
annot_cone = spawn_mesh("Cone",(-125,100,125), (0,0,0), (2,2,2))
annot_cube = spawn_mesh("Cube",(125,80,-125), (0,0,0), (2,2,2))

# Setup semantics
add_prim_semantics(prim=annot_cone, type="class", data="cone", write_type=LabelWriteType.OVERWRITE,)
add_prim_semantics(prim=annot_cube, type="class", data="cube", write_type=LabelWriteType.OVERWRITE,)
3

Thanks for the extensive reply.

If i understand it correctly, this script annotates the object in total.

What i would like to do is give the every vertex of the object a class.
For instance:

  • i have a .obj file (which i can convert with omniverse to usd)
  • i also have txt file where i have a class {0,1} for each vertex in the object file
  • i want to annotate the imported object, so that the generated segmentation map of the object by replicator has areas of the both classes, according to the annotated class
  • so big picture method would be accessing the vertex prims (?) of the imported object, assigning a class to each vertex according to the .txt gt file and then interpolate the classes of the vertexes onto the whole mesh. in mz understanding, the usd format should have this functionality, but i dont have an idea on how to implement this in practice

i tried this, as a postprocessing of the imported usd classes in python, but omniverse doesnt seem to recognize the new classes:


from pxr import Usd, UsdGeom, Vt, Sdf
import numpy as np

# Path to your USD file
usd_file_path = "/home/ruben/Desktop/DATA/cabinet0_broken0.usd"

# Load the stage
stage = Usd.Stage.Open(usd_file_path)


for prim in stage.Traverse():
    if prim.GetTypeName() == 'Mesh':
        mesh = UsdGeom.Mesh(prim)
        points = mesh.GetPointsAttr().Get()

        class_labels = ['nominal' if gt_classes[i] < 0.5 else 'anomal' for i, point in enumerate(points)]
        labels_array = Vt.StringArray(class_labels)

        primvars_api = UsdGeom.PrimvarsAPI(prim)
        labels_primvar = primvars_api.CreatePrimvar("classLabels", Sdf.ValueTypeNames.StringArray, UsdGeom.Tokens.vertex)
        labels_primvar.Set(labels_array)


# Save changes
stage.Save()

I am a total newbie to omniverse and usd, so I would be very grateful, for some pointers

EDIT:
so the class labels are seen in the ‘Raw usd Properties’, but the are not shown in the semantics:

Screenshot from 2024-03-04 14-10-39
Screenshot from 2024-03-04 14-10-39860×628 32.4 KB

is it possible to connect these raw usd classLables to the semantics of omniverse?

EDIT2:
As i understand it, its not directly possible to use the existing semantics api on vertex level prim attributes, because it is designed to work on the prims directly. So i am looking for a workaround to connect the primvars ‘classlabel’ to the rep semantic functionality?
It would be great to have a method to add semantics to the scene below prim level, as this is a very frequent task in 3d deep learning.