Replicator memory maxed out while using multiple ( 6 ) Render Products

I am trying to solve a problem using Replicator in Omniverse Code.
My situation is that I have 6 cameras, (6 Render Products) that need to be rendered on one frame after a physic sim. I can render 4 cameras at once, but using 6 will cause all the video ram to be used and it becomes very slow. I am using many annotators, (rgb, normals, bbox 3d, distance to camera). The resolution is 2560 x 1920. I realize that’s a lot for the video card to handle for that resolution and it makes sense that it will be painfully slow. I’m using the Path Tracer.

In my other extension I used omni.syntheticdata and rendered a camera then moved onto the next camera in a loop. But while using the new Replicator which uses nodes, I have yet to figure out a way to render the Render Products separately. It is my understanding that all the Render Products render at the same time and are subsequently waiting, (using a sync gate) until all products are rendered before sending to the Writer.

One solution which I have yet to figure out the best way is to render one product, then change the position of the camera, then render. I am using Ontime trigger which makes it difficult as the frame count has to be modified so that the ontime node does not trigger a new “scene”. As it is now I have to modify the frame count to accommodate the 32 samples needed. This issue could be addressed in it’s own post though.

Another solution may be to change the camera in the Render Product. How do I do this as there is the render product and the actual Hydra texture?

  1. What is the best way to change the camera of the Render Product while code is executing?

And while we are at it, it would be nice to render at a much lower resolution for the non hero frames during an ontime trigger scenario. For example if I have the ontime trigger using (interval=1, num=2) then there are 23 frames that are useless to me as they don’t get written. I have a custom writer subclassed from BasicWriter that does not write the non hero frame.

  1. Is there away to change the Render Product resolution on the fly?

I could run rep.orchestrator.preview(), then run through the physics process frame by frame. then start the render product process. But still have to loop through the cameras to produce the data and write them out. I don’t think I can use the Writer node at this point though which saves lots of coding as well as coding mistakes.

I can create my own nodes if needed. I have not done this yet but am ready and willing to start.
A solution that i don’t think will work is running the physics 6 times, (one for each camera) as I don’t trust the results of the sim will be the same every time. By the way it seems like rep.randomizer.instantiate does not take into account the global seed as it produces an output of different prims every time regardless of using rep.set_global_seed. While instantiate is in the unit tests, I don’t see a unit test that tests specifically for the consistency of using it with a global seed. I digress as this could be in another post.

  1. Is there another solution to render 6 cameras at high resolution with many annotators efficiently using the Replicator library?

I appreciate any help I can get.

Windows 11
Omniverse Code 2022.2.0
omni.replicator.core-1.4.7+wx64.r.cp37
RTX 3090

Thank you,

Mark

Update:
I have figured out the way to change the camera of a Hydra texture:

from omni.replicator.core import orchestrator, vp_manager


viewp_manager = vp_manager.ViewportManager()
it = viewp_manager._hydra_textures.hydra_textures
tex = next(it)
hydra_texture = tex.hydra_texture
hydra_texture.set_camera_path(new_cam_path)

this only works with the new viewport system

Hello @mark148,

I am trying to generate synthetic data using physics as well. Have you noticed any lag/drop in frame rate while applying physics?

As for your issue where there are n number of useless frames generated, add this line :
rep.settings.carb_settings("/omni/replicator/RTSubframes", 3)
Increase/decrease the value based on the number of generated frames.

My hardware configurations are similar but I am using Isaac Sim on Ubuntu 20

Hi @mayank.ukani
I have not noticed any lag or drop in frame rate. This is due to my code only capturing the last frame in the animation. For example using an ontime interval of 0.5 and the default fps being 24, I only capture, (write) frame 12. Thinking about this now, I would guess that frames would be dropped by setting rep.settings.carb_settings("/omni/replicator/RTSubframes", 3) as the code in orchestrator.py does cause the frame to advance before getting to the code that handles that setting, (“/omni/replicator/RTSubframes”).

So using ontime and RTSubframes > 1 will cause skipping of frames. This is my assumption looking at the code. This will only happen when using “raytracedlighting”. I’m using Path tracing and had to create my own orchestrator on_update function to get it to work.

I’m using omni.replicator.core-1.4.7+wx64.r.cp37