How to use Projection PBT Material

I want to use Projection PBR Material to project a Normal Map onto an object.
However, I don’t know how to use rep.create.projection_material() or rep.modify.projection_material(). I can’t find any API documentation for these either.
If anyone knows how to use it, please let me know.

@takahiro.someji.ps1 i am just another OV user. here are some materials that may help:

Replicator API - Omni Replicator — omni_replicator 1.9.8 documentation

Replicator Materials - Replicator Materials — Omniverse Extensions latest documentation

Randomizing Textures -
https://docs.omniverse.nvidia.com/extensions/latest/ext_replicator/randomizer_details.html#randomizing-textures

and here is a livestream where they showcased how you can use the projection to generate blemishes/defect detect via normal map with random seed:

Thank you for answering.
I have read all the articles you introduced.
However, when I search for projection_material() in the replicator documentation, I cannot find it. Even in Replicator materials article in Replicator Tutorials, there is no detailed explanation of how to use normal maps etc. kit-extension-sample-defectsgen published on GitHub does not seem to use projection_material().
Do you know detailed information about Projection PBT Material?

i dug into the defectsgen repo and looked at the ‘replicator_defect.py’. i was able to see they used modify.projection_material() in a definition in line 71 and create.projection_material() in line 96 (shown below)

my expertise isn’t on SDG, so i am only skimming the surface and passing what i’ve found in my travel as i learn OV. but, to your point, this isn’t documented in the API page so hopefully the dev/doc team can add that as an inclusion at some point. i’ll let the mods/devs provide additional information since they have in-depth knowledge with the API.

and no, i can’t claim that i know detailed information about these replicator materials. i mostly had to reverse engineer the code snippet from the tutorial i linked above (the one regarding replicator materials). broadly speaking, rep.create.project_material() seems to be for setting up the projection mesh and semantics; rep.modify.projection_material() with specific channel specified would allow you to randomize the texture assets (after you’ve created them or know the texture location).

1 Like

Thanks.
I’m also an amateur, so I just watch tutorials and learning them.
I hope that it will be documented when kit2023.1.1 becomes the release version.
Before I knew it, the Defects Sample Extension had been updated. I will analyze the code again.

1 Like

Hi @takahiro.someji.ps1 I have an example that does exactly what you’re looking for. It will be uploaded to the github examples site early next week.

Note, that you’ll want to use the latest isaacsim release as of a few days ago, since it has some critical fixes and handy features that will be used in the example.

Also, there’s a small script example and description of the materials in this page:
Replicator Materials — Omniverse Extensions latest documentation (nvidia.com)

Until the full example next week, take a look at this script, as it might give you enough to go on to implement your own example:

from pxr import Usd, UsdGeom
from pathlib import Path
import omni.usd
import omni.replicator.core as rep

with rep.new_layer():

    # Create prims
    stage = omni.usd.get_context().get_stage()
    
    # Add your texture directory here!
    scratch_texture_folder = Path("C:/path_to_textures/carpanel_scratches").as_posix()
    
    # Hard coded locations for now.
    proj_01_poslist = [(-10,50,0), (10,50,0)]
    proj_01_rotlist = [(0, 0, 90), (0,0,90)]
    
    # Define prims in scene
    car_panel = rep.get.prim_at_path('/World/CarPanel_Corrected/Mesh_703_001/Mesh_703_001')
    projector_01 = rep.get.prim_at_path('/World/ProjectorProxy_01')
    camera = rep.get.prim_at_path('/World/CameraPivot/Camera')
    camera_pivot = rep.get.prim_at_path('/World/CameraPivot')
    
    # Rep render product from scene camera
    render_product = rep.create.render_product(camera, (1024, 1024))
    
    # Define semantic label and get custom material
    sem = [('class', 'defect')]
    
    # Create the projection with the mesh
    with car_panel:
        decalshell_01 = rep.create.projection_material(projector_01, sem, offset_scale=0.001)
        rep.modify.semantics([('class', 'car_panel')])
    
    # Set Writer for outputs and annotations
    writer = rep.WriterRegistry.get("BasicWriter")
    writer.initialize(output_dir="_carpanel_scratches", rgb=True, bounding_box_2d_tight=True, semantic_segmentation=True)
    writer.attach([render_product])
    
    # Move the projector prims, and update the decal shell projectors
    with rep.trigger.on_frame(max_execs=10):
        with projector_01:
            rep.modify.pose(position=rep.distribution.sequence(proj_01_poslist), rotation=rep.distribution.sequence(proj_01_rotlist))
        with decalshell_01:
            rep.modify.projection_material(
                texture_group=rep.distribution.choice(rep.utils.get_files_group(scratch_texture_folder, ['_D', '_N', '_R', '_M'])),
                diffuse="_D",
                normal="_N",
                roughness="_R",
                metallic="_M"
            )
        with camera_pivot:
            rep.modify.pose(rotation=rep.distribution.uniform((0, 0, 0), (0, 360, 0)))
4 Likes

Thank you for providing a concise sample code.
Your sample code was very easy for me to understand.
But it is quite difficult for me to understand Replicator.
I’m still coding what I want to do, so if there’s anything I don’t understand, I’ll ask again.