Replicator and rep.orchestrator.step

aakash.chandran · November 20, 2024, 11:55am

Isaac Sim Version

4.2.0

Operating System

Ubuntu 20.04

GPU Information

Model: RTX 4070 Super Ti
Driver Version: 550.78

I am not able to understand Replicator behvaiour

Detailed Description

For synthetic data generation i use replicator to randomize few things like lights, colors etc.
In order to control the randomiztaion i use replicator with custom event names like

rep.utils.send_og_event(event_name=“randomize_lights”)
rep.utils.send_og_event(event_name=“randomize_dome_background”)
-rep.utils.send_og_event(event_name=“randomize_shape_distractor_colors”)

After calling the replicator with custom events, i use rep.orchestrator.step() to render the scene

To render the scene
rep.orchestrator.step(rt_subframes=8, delta_time=0.0)
rep.orchestrator.wait_until_complete()

Till now everything works fine but if I use rep.orchestrator.step() two times then the replicator at custom events does not work.

Let me explain in detail.

I want to render a prim for bounding box but do not want the prim to be visible in the image.
(like mentioned here - How to render the absence of a prim/mesh or part of a prim/mesh? )

Hence I make the object / prim visible when capturing bounding box data and make object invisible when capturing rgb image data. For this I need to use rep.orchestrator.step() twice

prim.MakeVisible()

rep.orchestrator.step(rt_subframes=8, delta_time=0.0)
rep.orchestrator.wait_until_complete()

-----capture bbox data using annotators---------

prim.MakeInvisible()

rep.orchestrator.step(rt_subframes=8, delta_time=0.0)
rep.orchestrator.wait_until_complete()

-----capture rgb image data using annotators---------

But when I use rep.orchestrator.step(rt_subframes=8, delta_time=0.0) twice in the loop of generating synthetic data the replicator randomizations at custom event do not takes place. For example the dome light texture does not change even though the replicator is called at custom events.

In general I think using replciator at custom events does not work if I use rep.orchestrator.step() two times.

The other option is to use omni.glass material for the object to be transperant in image data and still have bbox without using MakeVisible() and MakeInvisible(). But then the transperant object is still faintly visible in the rgb image.

I am more interested in turning off and on visibility of prim as it provides greater control. But then I am unable to use replicator randomizations which are automatic and easy to use.

Is there any way to solve this

Steps to Reproduce

In a loop call replicator at custom events

for i in range(100):

      rep.utils.send_og_event(event_name="randomize_lights")
      rep.utils.send_og_event(event_name="randomize_dome_background")
      rep.utils.send_og_event(event_name="randomize_shape_distractor_colors")   
      rep.orchestrator.step(rt_subframes=8, delta_time=0.0)
      rep.orchestrator.wait_until_complete()

this works and randomizations takes places.

not working case

for i in range(100):

  rep.utils.send_og_event(event_name="randomize_lights")
   rep.utils.send_og_event(event_name="randomize_dome_background")
   rep.utils.send_og_event(event_name="randomize_shape_distractor_colors")   
  rep.orchestrator.step(rt_subframes=8, delta_time=0.0)
  rep.orchestrator.wait_until_complete()
  rep.orchestrator.step(rt_subframes=8, delta_time=0.0)
  rep.orchestrator.wait_until_complete()

this does not works and randomizations do not takes places.

ahaidu · November 20, 2024, 4:36pm

Hi there,

the custom randomization events do not trigger on step() only when explicitly called e.g. rep.utils.send_og_event(event_name="randomize_lights") this gives you more flexibility on when to trigger which randomizer and step will only capture the data.

aakash.chandran · November 25, 2024, 10:36am

The problem i have I do randomization by calling

rep.utils.send_og_event(event_name=“randomize_lights”)

then I i call

rep.orchestrator.step(rt_subframes=8, delta_time=0.0)

to get the data.

Everything works.

If i do something like

    for i range (num_frames):

       rep.utils.send_og_event(event_name="randomize_lights")

        rep.orchestrator.step(rt_subframes=8, delta_time=0.0)
        
        prim.MakeVisible()
        
        rep.orchestrator.step(rt_subframes=8, delta_time=0.0)

Where i call step() twice, the randomizations doesnot takes place.

Lets say in first frame the dome_texture is set to “texture_1” then i call step() twice and then in second frame the dome_texture should be changed to random one from “texture"2” right, but even in the second frame the dome texture does not change, even though inside the loop rep.utils.send_og_event(event_name=“randomize_lights”) is called.

Can you check this and let me know you also face this error.

Please also let me know if I am not explaining the problem clearly. I can try and create a sample code to reproduce the error if required.

ahaidu · November 25, 2024, 10:47am

Do you mean randomizing twise in the same frame? Something like this:

    for i range (num_frames):
        rep.utils.send_og_event(event_name="randomize_lights")
        rep.orchestrator.step(rt_subframes=8, delta_time=0.0)        

        prim.MakeVisible()

        rep.utils.send_og_event(event_name="randomize_lights")
        rep.orchestrator.step(rt_subframes=8, delta_time=0.0)

aakash.chandran · November 25, 2024, 10:54am

No. I do one randomization per frame.

for i range (num_frames):
       # one randomization per frame
        rep.utils.send_og_event(event_name="randomize_lights")
       
       # call step() twice to hide and unhide a prim

        rep.orchestrator.step(rt_subframes=8, delta_time=0.0)        

        prim.MakeVisible()

        rep.orchestrator.step(rt_subframes=8, delta_time=0.0)

i see this behaviour

for i =0

rep.utils.send_og_event(event_name=“randomize_lights”) - works (randomization takes places)
rep.orchestrator.step(rt_subframes=8, delta_time=0.0) using twice i hide and unhide prim i want and get information from annotators.

for i=1 and subsequent loop steps

rep.utils.send_og_event(event_name=“randomize_lights”) is called but randomization does not takes place.
step() is called twice to get information.

When i removed the process of hiding and showing of prim and used step only once in the loop the randomization takes effect in all steps of loop.

ahaidu · November 25, 2024, 1:07pm

I see, that should not be happening, What type is your prim ‘prim’ and how do you access it?

aakash.chandran · November 25, 2024, 1:12pm

the prim is a disk .(i use a disk mesh in usd and save it )

i get the disk from world stage using code below. i use class structure and store stage in self._stage

corner = self._stage.GetPrimAtPath(f"/World/rack/rack_holes/Disk")

self._show_obj(corner)
rep.orchestrator.step(rt_subframes=8, delta_time=0.0) 

self._hide_obj(corner)
rep.orchestrator.step(rt_subframes=8, delta_time=0.0) 

@staticmethod
def _show_obj(obj) -> None:
    """
    Shows the object in the scene.
    """
    obj_imageable = UsdGeom.Imageable(obj)
    obj_imageable.MakeVisible()

@staticmethod
def _hide_obj(obj) -> None:
    """
    Hides the object in the scene.
    """
    obj_imageable = UsdGeom.Imageable(obj)
    obj_imageable.MakeInvisible()

ahaidu · November 25, 2024, 3:23pm

Seems to work just fine, here is a script editor snippet to test out:

import asyncio
import os
import omni.usd
import omni.kit.commands
import omni.replicator.core as rep
from pxr import UsdGeom

omni.usd.get_context().new_stage()
stage = omni.usd.get_context().get_stage()

_, cube_path = omni.kit.commands.execute("CreateMeshPrim", prim_type="Cube")
cube_prim = stage.GetPrimAtPath(cube_path)

with rep.trigger.on_custom_event(event_name="randomize_lights"):
    lights = rep.create.light(light_type="Dome", color=rep.distribution.uniform((0, 0, 0), (1, 1, 1)))
rep.utils.send_og_event(event_name="randomize_lights")


basic_writer = rep.writers.get("BasicWriter")
output_directory = os.getcwd() + "/_out_custom_rand"
print(f"output_directory: {output_directory}")
basic_writer.initialize(output_dir=output_directory, rgb=True)
rp = rep.create.render_product("/OmniverseKit_Persp", (512, 512))
basic_writer.attach(rp)


async def run_example_async():
    for i in range(4):
        print(f"Step {i}")
        UsdGeom.Imageable(cube_prim).MakeInvisible()
        rep.utils.send_og_event(event_name="randomize_lights")
        await rep.orchestrator.step_async(rt_subframes=8)

        UsdGeom.Imageable(cube_prim).MakeVisible()
        await rep.orchestrator.step_async(rt_subframes=8)


asyncio.ensure_future(run_example_async())

Let me know if you need further guidance. If you figure out what the issue was feel free to post the solution, it might help others as well. Thanks!

aakash.chandran · November 28, 2024, 10:09am

I think the issue i due to usage of rep.orchestrator.wait_until_complete()

in my code i use like

rep.orchestrator.step(rt_subframes=8, delta_time=0.0)
rep.orchestrator.wait_until_complete()
        
 image = self._rgb.get_data()
        
# switch to realtime render to speed up the process
rep.settings.set_render_rtx_realtime()

 # show object to get bbox information

rep.orchestrator.step(rt_subframes=8, delta_time=0.0)
rep.orchestrator.wait_until_complete()

bbox = self._bbox_2d_tight.get_data()

I had to use rep.orchestrator.wait_until_complete() to finish the process of getting information or else the image is not rendered perfectly and bbox of some objects is missing.

In the code you shared if i add await rep.orchestrator.wait_until_complete_async() same thing happens

import asyncio
import os
import omni.usd
import omni.kit.commands
import omni.replicator.core as rep
from pxr import UsdGeom

omni.usd.get_context().new_stage()
stage = omni.usd.get_context().get_stage()

_, cube_path = omni.kit.commands.execute("CreateMeshPrim", prim_type="Cube")
cube_prim = stage.GetPrimAtPath(cube_path)

with rep.trigger.on_custom_event(event_name="randomize_lights"):
    lights = rep.create.light(light_type="Dome", color=rep.distribution.uniform((0, 0, 0), (1, 1, 1)))
rep.utils.send_og_event(event_name="randomize_lights")


basic_writer = rep.writers.get("BasicWriter")
output_directory = os.getcwd() + "/_out_custom_rand"
print(f"output_directory: {output_directory}")
basic_writer.initialize(output_dir=output_directory, rgb=True)
rp = rep.create.render_product("/OmniverseKit_Persp", (512, 512))
basic_writer.attach(rp)


async def run_example_async():
    for i in range(10):
        print(f"Step {i}")
        UsdGeom.Imageable(cube_prim).MakeInvisible()
        rep.utils.send_og_event(event_name="randomize_lights")
        await rep.orchestrator.step_async(rt_subframes=8)
        await rep.orchestrator.wait_until_complete_async()

        UsdGeom.Imageable(cube_prim).MakeVisible()
        await rep.orchestrator.step_async(rt_subframes=8)
        await rep.orchestrator.wait_until_complete_async()


asyncio.ensure_future(run_example_async())

In attached image you can see all image have blue backgrround and do not change.

aakash.chandran · November 28, 2024, 10:13am

my question is is it supposed to happen like this or rep.orchestrator.wait_until_complete_async() should not be called twice?

ahaidu · November 28, 2024, 11:42am

rep.orchestrator.wait_until_complete_async() should only be called at the end, before closing your application, so writing any data still in memory to disk finishes.

aakash.chandran · November 28, 2024, 1:30pm

in amr_navigation example it was used every frame of synthetic data generation

//.local/share/ov/pkg/isaac-sim-4.2.0/standalone_examples/replicator/amr_navigation.py

def _run_sdg(self):
    if self._use_temp_rp:
        self._enable_render_products()
    rep.orchestrator.step(rt_subframes=16)
    rep.orchestrator.wait_until_complete()
    if self._use_temp_rp:
        self._disable_render_products()

async def _run_sdg_async(self):
    if self._use_temp_rp:
        self._enable_render_products()
    await rep.orchestrator.step_async(rt_subframes=16)
    await rep.orchestrator.wait_until_complete_async()
    if self._use_temp_rp:
        self._disable_render_products()

ahaidu · November 28, 2024, 2:37pm

Looking at the wait_until_complete_async functions it seems this happens because wait_until_complete_asyncalso stops and cleans replicator, e.g. the
randomize_lights graph is no longer valid. A workaround would be to only call:

        while not BackendDispatch.is_done_writing():
            await omni.kit.app.get_app().next_update_async()

In the AMR example it is fine because after each capture the replicator graphs are removed, so the data needs to be written to disk before:
omni.kit.commands.execute("DeletePrimsCommand", paths=["/Replicator"])

aakash.chandran · November 29, 2024, 8:59am

In AMR example

  def _clear_sdg_render_products(self):
      print(f"[NavSDGDemo] Clearing SDG render products")
      if self._writer:
          self._writer.detach()
      for rp in self._render_products:
          rp.destroy()
      self._render_products.clear()
      if self._stage.GetPrimAtPath("/Replicator").IsValid():
          omni.usd.commands.DeletePrimsCommand(["/Replicator"]).do()

the above function is called only once in end. Not each time.

And if i do not use rep.orchestrator.wait_till_complete(), I am not getting images sometimes from annoatators.

    rep.orchestrator.step(rt_subframes=8, delta_time=0.0)
    image = self._rgb.get_data()
    # sometimes there is no image from rbg annotator

anyway how should i ensure i get rendered rbg image manually from annotator without using rep.orchestrator.wait_untill_complete()

ahaidu · November 29, 2024, 10:01am

It might happen that the first frames until the render products are initialized that the annotators data is not yet in the buffer. Can you provide a small repro script for this case, since this should not happen with step().

As a workaround before starting the SDG pipeline you can update the application a few frames:

simulation_app.update() or await omni.kit.app.get_app().next_update_async().

Wait until complete does not interfere with annotator data, it basically waits until the writer buffers are written to disk:

I/O Optimization Guide — Omniverse Extensions

If you still run into issues with the data not being available please open a new topic.

aakash.chandran · November 29, 2024, 12:05pm

thanks for tips.

I could solve by disabling the use of temporary render products

   self._rp.hydra_texture.set_updates_enabled(True)
   # run rendering pipeline for each frame
   self._rp.hydra_texture.set_updates_enabled(False)

I did not set self._rp.hydra_texture.set_updates_enabled(False) and always it was True. Then used

for _ in range(10):
            simulation_app.update()
            rep.orchestrator.step(rt_subframes=1, delta_time=0.0)

before getting data from annotator. Now I am getting data from rgb and bbox annotator all the time.

But still i do not understand how replicator backend works as there is no detail documentation available. For more complex stuff like generating synthetic data for complex use case I think understanding replicator backend is necessary. Is it possible to create a clear documentation of different functions of replicator works and how everything works together to get a clear picture.

Topic		Replies	Views
Replicator: blur images when I have moving objects Isaac Sim	9	672	March 23, 2023
OmniGraph Warning: Required extended attribute outputs:samples is not resolved. while looping Synthetic Data Generation (SDG)	15	1202	June 3, 2024
Randomizer based Replicator code gets slower every frame Synthetic Data Generation (SDG) python	15	1397	July 13, 2023
Execute Function everytime a new scene gets generated by Replicator Isaac Sim	18	1511	April 6, 2023
Replicator gets slower after each iteration Synthetic Data Generation (SDG)	9	641	December 4, 2023
Replicator - how to get cameraViewTransform Synthetic Data Generation (SDG) synthetic-data	9	1791	October 19, 2023
Isaac Sim Replicator crashes Isaac Sim synthetic-data	11	2260	June 21, 2022
Replicator way of doing usd_scene_construction_utils Synthetic Data Generation (SDG)	8	43	September 10, 2024
How to render the absence of a prim/mesh or part of a prim/mesh? Isaac Sim	10	702	January 11, 2024
Replicator empty data if Fabric Scene Delegate enabled Synthetic Data Generation (SDG)	6	351	June 14, 2024