How to modify position of asset and projection material together

[feneu005]

Hi NVIDIA Community,

i want to modify the position (position, rotation) of my asset including the projection material which is projected on the asset via a proxy cube.
The projection should always be on the exact same place on the asset.

To do so i created a xform as parent of the asset and the proxy cube and modified the position and rotation of that xform.

import os
from omni.isaac.kit import SimulationApp
config = {
     'width': "1280",
     'height': "720",
     'headless': True,
     'fast_shutdown': True
}
simulation_app = SimulationApp(config)
# isaac sim imprts 
import omni.replicator.core as rep
from omni.replicator.core import AnnotatorRegistry, BackendDispatch, Writer


rep.settings.set_stage_meters_per_unit(1.0)

#create camera
camera = rep.create.camera(position=(2, 0, 0), rotation= (0,0,0))
render_product = rep.create.render_product(camera, (1024, 1024))

#create light 
dome = rep.create.light(
      light_type = "dome"
)

texture_path = os.path.join(rep.example.TEXTURES_DIR, "smiley_albedo.png")

projection_asset_xform = rep.create.xform(
     semantics = [('class', 'projection_xform')], 
     name = "projection_xform"
)

# create projection material
projection_asset = rep.create.plane(
    position=(0, 0, 0), 
    rotation=(0, 90, 0), 
    semantics = [('class', 'projection_asset')], 
    name = "projection_asset",
    parent = projection_asset_xform
    ) # asset to project on

proxy_cube = rep.create.cube(
    position=(1, 0, 0), 
    rotation=(0, 0, 0), 
    scale=(0.2, 0.2, 0.2), 
    visible =  False,
    semantics = [('class', 'proxy_cube')],
    name = "proxy_cube",
    parent = projection_asset_xform
    )

with projection_asset:
    rep.create.projection_material(proxy_prim = proxy_cube, semantics = [('class', 'projection_material')])

# modify projection material
projection = rep.get.prims(semantics = [('class', 'projection_material')])
with projection:
        rep.modify.projection_material(diffuse=texture_path)

writer = rep.WriterRegistry.get("BasicWriter")
writer.initialize(output_dir=os.path.join(os.path.dirname(os.path.realpath(__file__))), rgb=True)
writer.attach([render_product])

# first image without moving asset
rep.orchestrator.step(rt_subframes = 4)

# Move the projection asset
with projection_asset_xform:
    rep.modify.pose(
            position = (0, 0.2, 0),
            rotation = (0, 0, 0)
    )

# second image with modified position
rep.orchestrator.step(rt_subframes = 50)

quit()
simulation_app.close()

It seems like the asset and proxy cube modified the position correctly, but the projection stayes at the last position and does not update.

How can i fix this?

[ahaidu]

Hi there,

this seems to be a bug with the projection material in replicator, I submitted an internal bug for this.

I will keep you posted if I find a workaround solution or a fix.

Cheers,
Andrei

[pcallender]

Hi @feneu005, the original intent was to always modify the projection on either static geometry, or to move then apply the projection if the parent object had moved. There are cases where a projection in world space is makes more sense. I agree with you though, that the projection would ideally have an option for either.

Right now the best bet is to apply the projection update after moving the object.

I’ve modified the tutorial example to demonstrate moving the parent object along with moving the projection using the scatter ontop of the plane.

I find the generated plane xform prim and move that as the root. The scatter2d is applied to the plane surface, and the projection is randomized on the newly moved surface.

from pxr import Usd, UsdGeom
from pathlib import Path
import omni.usd
import omni.replicator.core as rep

#Create Light
distance_light = rep.create.light(rotation=(400,-23,-94), intensity=10000, light_type="distant")

stage = omni.usd.get_context().get_stage()
plane = rep.create.plane(position=(0, 0, 0), rotation=(0, 0, 90))
cube = rep.create.cube(visible=False, semantics=[('class', 'cube')], position=(0, 0, 0), rotation=(0, -90, 90), scale=(0.2, 0.2, 0.2))
sem = [('class', 'shape')]

# Randomizer for scattering
def get_shapes():
    shapes = rep.get.prims(semantics=[('class', 'cube')])
    with shapes:
        rep.randomizer.scatter_2d(plane)
    return shapes.node

rep.randomizer.register(get_shapes)

# Create the projection with the plane as the target
with plane:
    proj1 = rep.create.projection_material(cube, sem)

# Modify the cube position, and update the projection
with rep.trigger.on_frame(num_frames=30):
    root = rep.get.prim_at_path('/Replicator/Plane_Xform')
    with root:
        rep.modify.pose(position=rep.distribution.uniform((-1, 0, 0), (1, 0, 0)))
    rep.randomizer.get_shapes()
    with proj1:
        rep.modify.projection_material(diffuse=Path(rep.example.TEXTURES_DIR).joinpath("smiley_albedo.png").as_posix())

The reason for this is that the material does the projection, and the projection data to pass that along to the material uses custom primvars. The replicator function does the updating, so if rep.modify.projection_material() isnt called or is done so after moving the parent, the primvars aren’t updated.

There are other ways to achieve this, such as using omnigraph to do the same thing, though in a more complicated way. The engineers also suggested a listen event for when the parent is moved. Lastly, I will look into whether the shader can apply this as a local object space in the future.

I hope this helps.