How to destroy a render product?

Omniverse Isaac Sim
1

Hello.
I’ve been trying to make a big amount of camera work on isaacsim but i’m running in a problem here.
Steps to reproduce

from omni.isaac.kit import SimulationApp
app = SimulationApp()
import omni.replicator.core as rep
from pxr import Usd, UsdGeom, Gf, Sdf
import omni.usd
import omni.replicator.core as rep
stage : Usd.Stage = omni.usd.get_context().get_stage()
camera1 = rep.create.camera(name = "camera1",parent = "/Sensors/camera1")
camera2 = rep.create.camera(name = "camera2",parent = "/Sensors/camera2")
camera3 = rep.create.camera(name = "camera3",parent = "/Sensors/camera3")
camera4 = rep.create.camera(name = "camera4",parent = "/Sensors/camera4")
app.update()
rp1 = rep.create.render_product(camera=camera1,resolution=(1024,1024),force_new=False)
rp2 = rep.create.render_product(camera=camera2,resolution=(1024,1024),force_new=False)
rp3 = rep.create.render_product(camera=camera3,resolution=(1024,1024),force_new=False)
rp4 = rep.create.render_product(camera=camera4,resolution=(1024,1024),force_new=False)
while app.is_running():
    app.update()
app.close()

[in a terminal]

./python.sh path/to/script.py

At that point my python script crashes, saying :

[Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY

I figured with nvidia-smi that my gpu memory usage explodes at runtime.
And despite the fact that that i’m puting the force_new argument to false, it keeps creating new render products.
Worst thing is even after deleting the camea from the Usd Stage with

 stage.RemovePrim(Sdf.Path("/Sensors/camera1"))

, my gpu usage still does not decrease.
Deleting the render product by its path (/Render/RenderProduct_Replicator_0x) does not help either.
So here are my questions :

  1. Is there a way to dissociate a render product from a certain camera and attach it to another ?
  2. Is there a way to completely delete a render product at runtime ?
    Thanks you.
2

Hi there,

a render product is a combination of a camera and a resolution, if the camera and the resolution is the same, no new render product is created, only if the force_new flag is set to True.

Currently there is no clean way for destroying a render product, in the next release this should be possible. I will try to look for a workaround for that if possible using the current version.

As a workaround to your scenario, if the cameras (camera1, camera2, etc.) have the same properties only different locations, you can “re-use” a render product by pausing the simulation, moving the camera to the locations of the camera1, camera2, etc. and then un-pausing. Basically instead of collecting the data in parallel (which causes you to run out of memory) you would run it in series.

Let me know if you have any questions.

Best,
Andrei

4

@ahaidu Thank you for your answers. I actually figured out a way to switch render product for a camera. I was inspired by the code of the omni.isaac.sensor 's Camera class. It goes like this :

from pxr import Usd, UsdGeom, Gf, Sdf
import omni.graph.core as og
import omni.replicator.core as rep
import omni.usd
from omni.isaac.core.utils.prims import define_prim
#create a render product with an unused camera
rp1 = rep.create.render_product(camera=rep.create.camera(),resolution=(512,512),force_new=False)
def set_render_product(camera_prim : Usd.Prim ,rp : og.Node) -> bool :
        stage = get_current_stage()
        with Usd.EditContext(stage, stage.GetSessionLayer()):
            render_prod_prim = UsdRender.Product(stage.GetPrimAtPath(rp))
            if not render_prod_prim:
                raise RuntimeError(f'Invalid renderProduct "{rp}"')
            usd_rp_rel : Usd.Relationship = render_prod_prim.GetCameraRel()
            if len(usd_rp_rel.GetTargets()) > 0 : 
                usd_rp_rel.RemoveTarget(usd_rp_rel.GetTargets()[0])
            usd_rp_rel.AddTarget(camera_prim.GetPath())
            return True

camera_prim = UsdGeom.Camera(define_prim("/path/to/camera"))
set_render_product(camera_prim = camera_prim,rp = rp1)

This works for me but detaching anotators makes it crash, but a least now my VRAM usage does not rise too much.
The other catch is that gpu power usage rises sharply at each annotator attached.

2 Likes