[error][omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator

I have a simulation with an autonomous robot and some randomization. The point of this simulation is to run it for hours and collects large amount of data. sometimes, during the run of the scene, i get a none-ending flow of error messages:
image

from the log file:


2024-03-20 13:24:49 [1,281,226ms] [Warning] [omni.fabric.plugin] addAttributeInternal 0: invalid attribute type.
2024-03-20 13:24:49 [1,281,228ms] [Warning] [omni.fabric.plugin] addAttributeInternal 0: invalid attribute type.
2024-03-20 13:24:49 [1,281,240ms] [Info] [omni.kit.app._impl] [py stdout]: This instance is being destroyed

2024-03-20 13:24:49 [1,281,265ms] [Warning] [omni.anim.skelJoint.plugin] Prim /Render/RenderProduct_Replicator/PostRender/SDGPipeline/RenderProduct_Replicator_LdrColorSDPostCopyToHost is invalid when skeljoint processing prim resync
2024-03-20 13:24:49 [1,281,266ms] [Warning] [omni.anim.skelJoint.plugin] Prim /Render/RenderProduct_Replicator/PostRender/SDGPipeline/RenderProduct_Replicator_InstanceMappingPost is invalid when skeljoint processing prim resync
2024-03-20 13:24:49 [1,281,266ms] [Warning] [omni.anim.skelJoint.plugin] Prim /Render/RenderProduct_Replicator/PostRender/SDGPipeline/RenderProduct_Replicator_DistanceToCameraSDPostCopyToHost is invalid when skeljoint processing prim resync
2024-03-20 13:24:49 [1,281,274ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-20 13:24:49 [1,281,285ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-20 13:24:50 [1,281,312ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-20 13:24:50 [1,281,320ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-20 13:24:50 [1,281,337ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-20 13:24:50 [1,281,366ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-20 13:24:50 [1,281,374ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-20 13:24:50 [1,281,391ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-20 13:24:50 [1,281,421ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-20 13:24:50 [1,281,429ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-20 13:24:50 [1,281,446ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-20 13:24:50 [1,281,493ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-20 13:24:50 [1,281,501ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-20 13:24:50 [1,281,520ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-20 13:24:50 [1,281,536ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-20 13:24:50 [1,281,552ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-20 13:24:50 [1,281,568ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-20 13:24:50 [1,281,585ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-20 13:24:50 [1,281,601ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-20 13:24:50 [1,281,619ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator

Its seems to be related to the viewport in the editor. The entire simulation freezes and the error messages are flooding the consule. What is it and how can i solve it?

update: logs from another run:

2024-03-21 08:12:07 [772,535ms] [Warning] [omni.fabric.plugin] addAttributeInternal 0: invalid attribute type.
2024-03-21 08:12:07 [772,535ms] [Warning] [omni.fabric.plugin] addAttributeInternal 0: invalid attribute type.
2024-03-21 08:12:07 [772,536ms] [Warning] [omni.fabric.plugin] addAttributeInternal 0: invalid attribute type.
2024-03-21 08:12:07 [772,537ms] [Warning] [omni.fabric.plugin] addAttributeInternal: invalid attribute type.
2024-03-21 08:12:07 [772,537ms] [Warning] [omni.fabric.plugin] addAttributeInternal 0: invalid attribute type.
2024-03-21 08:12:07 [772,538ms] [Warning] [omni.fabric.plugin] addAttributeInternal 0: invalid attribute type.
2024-03-21 08:12:07 [772,541ms] [Warning] [omni.fabric.plugin] addAttributeInternal: invalid attribute type.
2024-03-21 08:12:07 [772,541ms] [Warning] [omni.fabric.plugin] addAttributeInternal 0: invalid attribute type.
2024-03-21 08:12:07 [772,542ms] [Warning] [omni.fabric.plugin] addAttributeInternal 0: invalid attribute type.
2024-03-21 08:12:07 [772,542ms] [Warning] [omni.fabric.plugin] addAttributeInternal: invalid attribute type.
2024-03-21 08:12:07 [772,543ms] [Warning] [omni.fabric.plugin] addAttributeInternal 0: invalid attribute type.
2024-03-21 08:12:07 [772,543ms] [Warning] [omni.fabric.plugin] addAttributeInternal 0: invalid attribute type.
2024-03-21 08:12:07 [772,543ms] [Warning] [omni.fabric.plugin] addAttributeInternal: invalid attribute type.
2024-03-21 08:12:07 [772,544ms] [Warning] [omni.fabric.plugin] addAttributeInternal 0: invalid attribute type.
2024-03-21 08:12:07 [772,544ms] [Warning] [omni.fabric.plugin] addAttributeInternal 0: invalid attribute type.
2024-03-21 08:12:07 [772,545ms] [Warning] [omni.fabric.plugin] addAttributeInternal: invalid attribute type.
2024-03-21 08:12:07 [772,545ms] [Warning] [omni.fabric.plugin] addAttributeInternal 0: invalid attribute type.
2024-03-21 08:12:07 [772,546ms] [Warning] [omni.fabric.plugin] addAttributeInternal 0: invalid attribute type.
2024-03-21 08:12:07 [772,551ms] [Info] [omni.kit.app._impl] [py stdout]: This instance is being destroyed

2024-03-21 08:12:07 [772,588ms] [Warning] [omni.anim.skelJoint.plugin] Prim /Render/RenderProduct_Replicator/PostRender/SDGPipeline/RenderProduct_Replicator_LdrColorSDPostCopyToHost is invalid when skeljoint processing prim resync
2024-03-21 08:12:07 [772,588ms] [Warning] [omni.anim.skelJoint.plugin] Prim /Render/RenderProduct_Replicator/PostRender/SDGPipeline/RenderProduct_Replicator_InstanceMappingPost is invalid when skeljoint processing prim resync
2024-03-21 08:12:07 [772,588ms] [Warning] [omni.anim.skelJoint.plugin] Prim /Render/RenderProduct_Replicator/PostRender/SDGPipeline/RenderProduct_Replicator_DistanceToCameraSDPostCopyToHost is invalid when skeljoint processing prim resync
2024-03-21 08:12:07 [772,600ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-21 08:12:07 [772,610ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-21 08:12:07 [772,627ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-21 08:12:07 [772,654ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-21 08:12:08 [772,662ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-21 08:12:08 [772,678ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-21 08:12:08 [772,704ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator
2024-03-21 08:12:08 [772,712ms] [Error] [omni.hydra] Invalid USD RenderProduct Prim: /Render/RenderProduct_Replicator

Thank you

Hi there, can you provide the isaac sim version you are using?

Does this error happen every run? Or only when it reaches some particual situation in the randomizations?

Is it possible to provide a repro snippet with the error?

Or snippets on how you set up your scenario and randomizations?

Thanks!

Hi @ahaidu,

I am using isaac sim 2023.1.1

I have a simulator extension with a menu and a start button. When i press start a render product is created and attached to my custom writer.

def init_writer(self, cam_path, resolution, model_yaml_path, colorizedSegmentation):


        rep.WriterRegistry.register(MowerWriter)


        self.rp = rep.create.render_product(cam_path, resolution)


        self.writer = rep.WriterRegistry.get(WRITER_NAME)


        self.writer.initialize(rgb_output_path = self.paths.rgb_directory_path,
                               seg_output_path = self.paths.segmented_directory_path,
                               depth_output_path = self.paths.depth_directory_path,
                               metadata_path = self.paths.data_directory_path,
                               model_yaml_path = model_yaml_path,
                               colorizedSegmented = colorizedSegmentation
                               )


        self.writer.attach(render_products = [self.rp])



To re-run a usd with my extension I reload the usd and re-enable my extension. When this error message floods the consule, I need to re-open the usd for it to stop. By in most cases, once i press Play (my extension button) the error re appears. only shuttting down and re starting isaac sim will give me a few runs before it happens again.

Can you explain please what this error is about? Where is the path /Render/RenderProduct_Replicator?

is there usually a USD RenderProduct? Where can I find it in cases everything works as expected?

Thank you

Regarding the render product path, you can find them in the session layer (Layer → Burger Menu → Show Session Layer):

Here is a script you can run in the script editor on how the render products are named:

import omni.replicator.core as rep


cam_1 = rep.create.camera(name="cam_1")
cam_default_name = rep.create.camera()

rp = rep.create.render_product("/OmniverseKit_Persp", (512, 512))
rp_named = rep.create.render_product("/OmniverseKit_Persp", (512, 512), name="rp_persp_2")
rp_cam_1 = rep.create.render_product("cam_1", (512, 512), name="rp_cam_1")
rp_default_name = rep.create.render_product("cam_default_name", (512, 512), name="rp_cam_default_name")

Could you provide the replicator version you are using?

hi @ahaidu , can you please instruct mewhere I can find the version of replicator. Thank you

@ahaidu where can i find this package? [omni.kit.app._impl]

I can only find kit.property and kit.loop in /AppData/Local/ov/pkg/isaac_sim-2023.1.1/ext

for some reason it is destroying RenderProduct_Replicator
image

Thank you

You can take a look in Window-->Extensions by searching for omni.replicator.core

Or in the terminal after starting isaac sim. The extension is located in isaac_sim/extscache it can however contain multiple versions of extensions, you will know for sure which is loaded by going over the above ways.

Could it be that your extension is doing more that that, and it is somehow deleting the render product?

@ahaidu It turns out that [Info] [omni.kit.app._impl] [py stdout] is from the print function. I have a deconstructor in my DataGenerator class:

 def __del__(self):
        print("This instance is being destroyed")

and that class holds the refrence to the render product. But i dont understand why the garbage collector will collect the instance of DataGenerator in the middle of the simulation. The Simulator class holds reference to it (self.data_generator)

@ahaidu ,

after debugging my code, I see that the instance of Simulator() is being collected by garbage collection (entering it’s own del()).

My Simulator() class has a Start function and a Stop functions which are called from UIdelegate when I press the buttons in the UI window.
It also holds all the components of the simulator such as an instance to the DataCollector and all the different parameters of the Simulations.

How can I ensure that the Simulator instance is collected after the simulation ends (I press stop). Is there in depth documentation of the lifespan of variables in an omniverse extension?

Thank you