Replicator gets slower after each iteration

[utxin]

Hello,

I am currently working on a project involving the simulation of a scene in which multiple assets are dropped into a bin. After that I individually make all objects invisible, with the exception of one, and then retrieve the instance segmentation data. However, I have observed that the replicator requires progressively more time after each iteration to render the objects invisible (see graph below).

I came across a discussion that addresses a similar issue:

https://forums.developer.nvidia.com/t/randomizer-based-replicator-code-gets-slower-every-frame/252002/13

Although the problem appeared to have been resolved in that discussion, I am encountering similar challenges.

In this graph you can see the time it takes to complete a certain function call. And the functions which use rep.modify for changing pose or visibility take increasing time to fulfill their task. Functions without it have constant processing times. In the function “Check Occlusion”, the function “change_visibility_assets()” is called several times and is responsible for the increasing processing times.

I use the following code for modifying the visibility of the assets:

def change_visibility_assets(self, mode, id=None):
    """
    mode = 0 : make invisible
    mode = 1 : make visible
    """
    assert mode == 0 or mode == 1
    if id is None:
        for i in range(len(self.all_assets)):
            _asset_path = str(self.all_assets[i].GetPath())  # self.all_assets contains all prims from the stage
            _imageable = rep.get.prim_at_path(_asset_path)
            with _imageable:
                if mode == 0:
                    rep.modify.visibility([False])  # make object invisible
                else:
                    rep.modify.visibility([True])  # make object visible
            rep.orchestrator.step()
        # bin
        bin_path = "/World/KLT_Festo_low/BOX_VOL4"

        if not self.turn_off_bin:
            self.change_visibility_bin(mode)
        else:
            self.change_visibility_bin(mode=0)
    else:
        # Only specified object id
        _asset_path = str(self.all_assets[id].GetPath())
        _imageable = rep.get.prim_at_path(_asset_path)
        with _imageable:
            if mode == 0:
                rep.modify.visibility([False])
                rep.orchestrator.step()
            else:
                rep.modify.visibility([True])
                rep.orchestrator.step()

    rep.orchestrator.step()
    simulation_app.update()
    return True

And for changing the camera pose which is represented in the graph as “Modify Pose”, I use:

with self.camera_rep:
    rep.modify.pose(
        position=pose[0], rotation=pose[1], rotation_order="YXZ"
        )
    rep.orchestrator.step()

I need to render hundreds to thousands of scenes, but this is not possible with such high and increasing processing times. Do you have any suggestions for resolving this issue?

Thank you in advance.

[Simplychenable]

@utxin i am just another user, but could you confirm the version of Isaac Sim you are using?

[utxin]

I am using Isaac Sim 2023.1.0

[Simplychenable]

not sure if it’s something they’ve already addressed in hotfix.1, is it possible for you to update and see if it made any difference there?

[utxin]

Thank you for your advice. I installed the the hotfix.1 version. However, the problem is still there.

[pcallender]

I’ll bring this up to the rep devs and we’ll get some eyes on this. Stay tuned.

[jlafleche]

Hello @utxin, thank you for reaching out. This type of performance slowdown is definitely not what we expect to see and I’d love to work with you to figure out what’s happening.

First question: ls len(self.all_assets) changing or constant over time?
Second question: Is the graph being deleted between calls to change_visibility_assets? What I see it doing within that for loop is it’s adding, for every asset in self.all_assets a small graph consisting of a get node and a node changing the visibility. Each of those graphs is executed with rep.orchestrator.step(). I suspect in this case, you’re not intending to create a graph. We’re working on a solution that beter fits pure python workflows like this one, as we absolutely see the value.

Since it appears that you need to change the visibility of each prim one at a time and capture each state, I propose a simple modification to use the USD API to make the visibility changes:

def change_visibility_assets(self, mode, id=None):
    """
    mode = 0 : make invisible
    mode = 1 : make visible
    """
    stage = omni.usd.get_context().get_stage()
    visibility = ["invisible", "inherited"]    # mode 0 is invisible, mode 1 is visible
    assert mode == 0 or mode == 1
    if id is None:
        for i in range(len(self.all_assets)):
            _asset_path = str(self.all_assets[i].GetPath())  # self.all_assets contains all prims from the stage
            _imageable = stage.GetPrimAtPath(str(_asset_path))
            if _imageable.HasAttribute("visibility"):
                _imageable.GetAttribute("visibility").Set(visibility[mode])
                rep.orchestrator.step()
        # bin
        bin_path = "/World/KLT_Festo_low/BOX_VOL4"

        if not self.turn_off_bin:
            self.change_visibility_bin(mode)
        else:
            self.change_visibility_bin(mode=0)
    else:
        # Only specified object id
        _asset_path = str(self.all_assets[id].GetPath())
        _imageable = stage.GetPrimAtPath(str(_asset_path))
            if _imageable.HasAttribute("visibility"):
                _imageable.GetAttribute("visibility").Set(visibility[mode])
                rep.orchestrator.step()

    rep.orchestrator.step()
    simulation_app.update()
    return True

[dennis.lynch]

This can also be achieved by setting multiple triggers at different intervals

See some snippet examples we have:
synthetic-data-examples/omni.replicator/snippets/replicator_scatter_multi_trigger.py at main · NVIDIA-Omniverse/synthetic-data-examples (github.com)

synthetic-data-examples/omni.replicator/snippets/replicator_trigger_intervals.py at main · NVIDIA-Omniverse/synthetic-data-examples (github.com)

Here’s another more complete example where, having a static number of objects, I toggle all objects to be visible, then have them individually visible, before randomizing pose. This requires Replicator 1.10.+ I believe

import omni.replicator.core as rep

NUM_POSE_RANDOMIZATIONS = 10


def make_visibility_lists(num_objects):
    visib = []
    # Make an all-visible first pass
    visib.append(tuple([True for x in range(num_objects)]))

    # List to toggle one object visible at a time
    for x in range(num_objects):
        sub_vis = []
        for i in range(num_objects):
            if x == i:
                sub_vis.append(True)
            else:
                sub_vis.append(False)
        visib.append(tuple(sub_vis))
    return visib


with rep.new_layer():
    camera = rep.create.camera(position=(0, 500, 1000), look_at=(0, 0, 0))
    light = rep.create.light(rotation=(-45, 45, 0))
    # Create simple shapes to manipulate
    plane = rep.create.plane(semantics=[("class", "plane")], position=(0, -100, 0), scale=(100, 1, 100))
    torus = rep.create.torus(semantics=[("class", "torus")], position=(200, 0, 100))
    sphere = rep.create.sphere(semantics=[("class", "sphere")], position=(0, 0, 100))
    cube = rep.create.cube(semantics=[("class", "cube")], position=(-200, 0, 100))

    object_group = rep.create.group([torus, sphere, cube])

    num_objects_to_toggle = len(object_group.get_output_prims()["prims"])
    print(num_objects_to_toggle)
    visibility_sequence = make_visibility_lists(num_objects_to_toggle)
    print(visibility_sequence)
    # Toggle visibility one at a time
    with rep.trigger.on_frame(max_execs=(num_objects_to_toggle+1) * NUM_POSE_RANDOMIZATIONS):
        with object_group:
            rep.modify.visibility(rep.distribution.sequence(visibility_sequence))
    #  Randomize position and scale
    with rep.trigger.on_frame(max_execs=NUM_POSE_RANDOMIZATIONS, interval=num_objects_to_toggle + 1):
        with object_group:
            rep.modify.pose(
                position=rep.distribution.uniform((-300, 0, -300), (300, 0, 300)),
                scale=rep.distribution.uniform(0.1, 2),
            )

    

# Initialize render product and attach writer
render_product = rep.create.render_product(camera, (512, 512))
writer = rep.WriterRegistry.get("BasicWriter")
writer.initialize(
    output_dir="toggle_visibility",
    rgb=True,
    semantic_segmentation=True,
)
writer.attach([render_product])
rep.orchestrator.run()

[utxin]

@jlafleche Thank you for your support. To your questions:

1. self.all_assets is a list containing all prims I
   use on stage and is constant over time.

2. You are right, I am not intending to create a graph each time. I
   don’t think that the graph is being deleted. How could I do that?

Now with your code snippet, changing visibility works without increasing processing times.

I wasn’t aware that it creates a new graph each time. However, I still encounter the same problem when changing the camera. Is this the same problem where it creates a new graph each time?

I create a camera and also a render product with this camera in the init() function of my class:

self.camera_rep = rep.create.camera(
            position=(0, 0, 1.0),
            look_at=(0, 0, 0),
            clipping_range=(0.01, 1000000.0),
            focal_length=35.0,
            name="mainCamera",
        )

self.rp1 = rep.create.render_product(
            self.camera_rep, resolution=(1200, 1200)
        )  # create new camera render product

After generating a scene I want to get the ground truth data for dozens of different viewpoints.
For changing the viewpoint, I use the following code in a for-loop which iterates over all viewpoints.

for idx, pose in enumerate(self.camera_poses):
      with self.camera_rep:
            rep.modify.pose(position=pose[0], rotation=pose[1], rotation_order="YXZ" )
      rep.orchestrator.step()

     #calling other functions like collect_groundtruth, check_occlusion and 
     #save_dataset and changing visibility of objects for that specific viewpoint

Also Thank you @dennis.lynch for your tips. But I think I can not use this functionality for my specific case. I have to call several functions, e.g like getting ground truth data and processing it, between making a single object visible and invisible again and I am note quite sure on how to call those functions when using rep.trigger.on_frame(). So far, I stepped manually through my simulation with rep.orchestrator.step().