Combining physics simulation with domain randomization using Replicator

[saeejithn]

The offline_generation.py demo shows how to run a short physics simulation to simulate falling objects onto a pallet, and then generate randomized scenes of boxes/cones scattered on another pallet. Here, the rep.trigger.on_frame() context is used to generate scenes of the scattered boxes with different poses on every frame, BUT the poses of the boxes that were dropped onto the pallet do not change.

I want to run a new short simulation on every frame to also generate scenes where the poses of the objects dropped onto the pallet are different on every frame. How can I do this?

I tried modifying offline_generation.py so that it moves the simulate_falling_objects() function call into the rep.trigger.on_frame context, but this function still seems to only be called once, as the poses of the dropped boxes are identical for all 10 frames.

    with rep.trigger.on_frame(num_frames=CONFIG["num_frames"]):
        rep.randomizer.scatter_boxes()
        rep.randomizer.place_cones()
        simulate_falling_objects(forklift_prim)
        rep.randomizer.randomize_lights()
        ....

I’m using offline_generation.py to easily illustrate my issue. In my actual application, I want to generate a synthetic datataset of random objects dropped from different poses, but only want to record the data AFTER the object has landed/settled onto a surface (hence why I need the short simulation for every scene).

Please advise, thanks!

[ahaidu]

Hi there,

to make sure I understand correctly:

• you have a list of objects you want to simulate dropping
• once the simulation finishes, you would like to capture data from the static scene
• afterward start from the beginning by placing the simulated objects in a randomized location

If that is the case, you can do this using Isaac Sim and use Replicator only for data acquisition.

A possible workflow:

• create a function taking as input a list of prims
  • move the prims to the desired initial randomized locations
  • run the simulation for X frames
• trigger the data acquisition using rep.orchestrator.step(), see example:

  for _ in range(10):
       rep.orchestrator.step()

Your with rep.trigger.on_frame snippet did not work, since it is only called once when the randomization omnigraphs are generated. Afterwards the randomization workflow goes through the generated graphs and not the python script. And since the simulation function is not an omnigraph node it is only called once.

Let me know if you need more details.
Best