Time to move camera using replicator is increasing as it keeps beeing called

Omniverse Isaac Sim
1

Hi,

I’m trying to get the image from the camera which attaches to a robot arm end-effector. The script I referred to uses the Isaac sim extension replicator and rep.create.modify.pose to change the camera’s location and rotation to help the camera track the end-effector. However, the executing time of rep.create.modify.pose is increasing as it is frequently called.

This problem can be reproduced through this part of code:

from omni.isaac.kit import SimulationApp

simulation_app = SimulationApp(launch_config={"renderer": "RayTracedLighting", "headless": True})

import omni
import numpy as np
import os
import json
from PIL import Image
from omni.isaac.core import World
from omni.isaac.core.objects import DynamicCuboid
import omni.replicator.core as rep
from omni.isaac.core.utils.semantics import add_update_semantics

# Create a new stage with the default ground plane
omni.usd.get_context().new_stage()
world = World()
world.scene.add_default_ground_plane()
world.reset()

# Run the application for several frames to allow the materials to load
for i in range(20):
    simulation_app.update()

# Create a camera and render product to collect the data from
cam = rep.create.camera(position=(3, 3, 3), look_at=(0, 0, 0))
rp = rep.create.render_product(cam, (512, 512))

import time
prev_time = time.perf_counter()

for i in range(10000):
    with cam:
        rep.create.modify.pose(
            position=(3,3,3+i),
            scale=None,
            pivot=None,
            rotation=None,
            look_at=(0,0,0+i),
            look_at_up_axis=None,
        )
    curr_time = time.perf_counter()
    print(f"time diff: {curr_time-prev_time}")
    prev_time = curr_time

simulation_app.close()

And the executing time is increasing: (like linearly)
time diff: 0.057644279091618955
time diff: 0.05855504795908928
time diff: 0.05905811698175967
time diff: 0.058225102024152875
time diff: 0.05821833002846688
time diff: 0.06227774091530591
time diff: 0.0584783450467512
time diff: 0.058484185952693224
time diff: 0.06052612408529967
time diff: 0.05977054499089718
time diff: 0.06175160000566393
time diff: 0.060659682960249484
time diff: 0.06187240604776889
time diff: 0.06093573791440576
time diff: 0.06167182105127722
time diff: 0.06161282095126808
time diff: 0.06197104905731976
time diff: 0.06258698797319084
time diff: 0.06255956797394902
time diff: 0.06283891701605171
time diff: 0.06408600404392928
time diff: 0.06407754798419774
time diff: 0.06379654398187995
time diff: 0.06398163898847997
time diff: 0.0644281420391053
time diff: 0.06441731099039316
time diff: 0.0661116229603067
time diff: 0.06483900803141296
time diff: 0.06525875104125589
time diff: 0.0660596649395302
time diff: 0.06606167904101312
time diff: 0.06671516795177013
time diff: 0.06762831006199121
time diff: 0.0665425059851259
time diff: 0.06702468392904848
time diff: 0.06854787608608603
time diff: 0.06778147700242698
time diff: 0.06848757294937968
time diff: 0.0682692660484463
time diff: 0.06836566491983831
time diff: 0.06882072507869452
time diff: 0.06984280096367002
time diff: 0.06901329394895583
time diff: 0.07001556304749101
time diff: 0.07009836798533797
time diff: 0.0695943069877103
time diff: 0.07127058401238173
time diff: 0.07076247700024396
time diff: 0.07301987602841109
time diff: 0.07155614695511758
time diff: 0.07189489807933569
time diff: 0.07193242793437093
time diff: 0.07240101299248636
time diff: 0.07414933305699378
time diff: 0.07292909699026495
time diff: 0.07310835202224553
time diff: 0.0731286599766463
time diff: 0.07368130097165704
time diff: 0.0738065029727295
time diff: 0.07577913405839354
time diff: 0.0743423729436472
time diff: 0.07493349607102573
time diff: 0.0746059239609167
time diff: 0.07567507994826883
time diff: 0.07404023909475654
time diff: 0.07543211593292654
time diff: 0.07637635699938983
time diff: 0.07596263498999178
time diff: 0.0760634740581736
time diff: 0.07742941996548325
time diff: 0.0775237079942599
time diff: 0.07746655005030334
time diff: 0.07721209595911205
time diff: 0.0783033010084182
time diff: 0.07781548204366118
time diff: 0.0779315069084987
time diff: 0.0785499420017004
time diff: 0.0787601521005854
time diff: 0.08062467095442116
time diff: 0.0791920879855752
time diff: 0.0791494200238958
time diff: 0.08052318100817502
time diff: 0.08036545698996633
time diff: 0.08004597201943398
time diff: 0.0814435079228133
time diff: 0.0815659670624882
time diff: 0.08121515193488449
time diff: 0.08016834000591189
time diff: 0.08103529701475054
time diff: 0.08118490502238274
time diff: 0.08250891603529453
time diff: 0.08279872499406338
time diff: 0.08345465897582471
time diff: 0.08310361497569829
time diff: 0.08350782806519419
time diff: 0.08400787191931158
time diff: 0.08572668605484068
time diff: 0.08446183800697327
time diff: 0.08457691199146211

Can you help me solve this problem?

Best,
Chay

2

Hi @hitzcy2016 - The issue you’re experiencing is likely due to the fact that you’re continuously updating the camera’s position and look_at parameters in a tight loop. This can cause a significant amount of overhead, as each call to rep.create.modify.pose will trigger a re-computation of the camera’s transformation matrix, which can be computationally expensive.

One potential solution to this problem is to reduce the frequency at which you’re updating the camera’s pose. Instead of updating the pose on every iteration of the loop, you could update it only when the end-effector’s position has changed by a significant amount. This would reduce the number of calls to rep.create.modify.pose, and should therefore reduce the execution time.

Another potential solution is to use a different method to track the end-effector. Instead of manually updating the camera’s pose, you could attach the camera to the end-effector using a parent-child relationship. This would automatically update the camera’s pose whenever the end-effector moves, without the need for explicit calls to rep.create.modify.pose.

Here’s an example of how you might implement this:

# Assume `end_effector` is the end-effector object
cam.parent = end_effector

This will make the camera follow the end-effector, maintaining the same relative position and orientation. If you need the camera to have a different relative pose, you can adjust it using rep.create.modify.pose after setting the parent, and the changes will be preserved as the parent moves.

3

Hi @rthaker ,

Thanks for your solutions and suggestion. The code I’ve shown above is a simplified example of what I’m trying to do, and more importantly, from my intuition, the increase in time is not normal, which should be a constant.

As you said, attaching the camera to a parent is a potential solution. But what I’m trying to implement is a multi-instances reinforcement learning training with visual input so either I need to create as many cameras as the number of instances or I need to move the camera from one position to another. Because from my another post Enable camera in headless model experience memory leak there seems a memory leak if I use the camera in training and the number of cameras is quite limited based on my previous test, around 50 cameras can be successfully created without any error.

Do you have any suggestions for my specific problem?

Best,
Chay

5

Hi there,

the following code:

for i in range(10000):
    with cam:
        rep.create.modify.pose(
            position=(3,3,3+i),
            scale=None,
            pivot=None,
            rotation=None,
            look_at=(0,0,0+i),
            look_at_up_axis=None,
        )

will not randomize the camera for the given number of times, it will re-generate the Replicator Graph every iteration with new values on how to randomize the camera once you start your data generation (Replicator->Run).

So the Replicator API scripts generate the randomization graphs that will be run during once Replicator is started.

Here are some snippets explaining this: Isaac Sim/Robotics Weekly Livestream: Synthetic Data Generation - YouTube

6

Hi ahaidu,

The code I showed is just a simple example of the function that I want to achieve. I have successfully used the parent attribute of the replicator camera, attached it to my end-effector, and acquire images using AnnotatorRegistry. However, I’m experiencing a limited number of cameras. 60 cameras are reaching the limits in one environment so I can only have 60 robots in training right now.

../../../source/plugins/rtx.resourcemanager/ResourceManagerContext.cpp(5743): rtx::RtxResult rtx::resourcemanager::Context::allocateDescriptorSets(carb::graphics::ResourceBindingSignature*, uint32_t, uint32_t, carb::graphics::DescriptorSet**, carb::graphics::DescriptorPool**, uint32_t*, size_t, rtx::resourcemanager::DescSetAllocationFlags)(): Assertion (success) failed: Unable to allocate descriptor sets.

terminate called without an active exception
/isaac-sim/python.sh: line 41:    53 Aborted                 (core dumped) $python_exe "$@" $args
There was an error running python

Is there any way I can increase the number of cameras?

Best,
Chay

1 Like
7

Has this error been solved? I still encounter this error now.

8

The number of cameras should not be an issue, creating render products from those cameras can cause running out of memory issues for such a large value of cameras since the render products are constantly being rendered by the gpu, even if not visible in the UI.

Can you check if this is the issue?

9

According to my observation, during the program’s operation, the memory of the GPU did not increase significantly. I am not proficient in CUDA and GPU. Will other types of GPU resources besides memory be consumed during the rendering process? Or, in this function rtx::RtxResult rtx::resourcemanager::Context::allocateDescriptorSets, what exactly happened that led to the error?Because this function is not open source, look forward to your reply.

10

Could you provide a repro snippet on how you get to this error? Including the OS, Isaac Sim version and the Hardware you are running?