Replicator sometimes seems to break the existing Camera

Omniverse Isaac Sim
1

Hi!

I’m sometimes struggling with Replicator (in Isaac Sim 2022.2.1) as it randomly starts to produce these errors when I’m programmatically capturing synthetic data (in this example rgb and depth):

2023-05-11 08:39:36 [714,680ms] [Error] [omni.graph.core.plugin] Failed to get USD Attribute for /Render/PostProcess/SDGPipeline/RenderProduct_Replicator_LdrColorSDExportRawArray.inputs:cudaStream of type uint64
2023-05-11 08:39:36 [714,681ms] [Error] [omni.graph.core.plugin] Failed to get USD Attribute for /Render/PostProcess/SDGPipeline/RenderProduct_Replicator_LdrColorSDExportRawArray.inputs:exec of type uint (execution)
2023-05-11 08:39:36 [714,683ms] [Error] [omni.graph.core.plugin] Failed to get USD Attribute for /Render/PostProcess/SDGPipeline/RenderProduct_Replicator_LdrColorSDExportRawArray.inputs:renderResults of type uint64
2023-05-11 08:39:36 [714,684ms] [Error] [omni.graph.core.plugin] Failed to get USD Attribute for /Render/PostProcess/SDGPipeline/RenderProduct_Replicator_LdrColorSDExportRawArray.inputs:swhFrameNumber of type uint64
2023-05-11 08:39:36 [714,691ms] [Error] [omni.graph.core.plugin] Failed to get USD Attribute for /Render/PostProcess/SDGPipeline/RenderProduct_Replicator_DistanceToImagePlaneSDExportRawArray.inputs:cudaStream of type uint64
2023-05-11 08:39:36 [714,691ms] [Error] [omni.graph.core.plugin] Failed to get USD Attribute for /Render/PostProcess/SDGPipeline/RenderProduct_Replicator_DistanceToImagePlaneSDExportRawArray.inputs:exec of type uint (execution)
2023-05-11 08:39:36 [714,694ms] [Error] [omni.graph.core.plugin] Failed to get USD Attribute for /Render/PostProcess/SDGPipeline/RenderProduct_Replicator_DistanceToImagePlaneSDExportRawArray.inputs:renderResults of type uint64
2023-05-11 08:39:36 [714,695ms] [Error] [omni.graph.core.plugin] Failed to get USD Attribute for /Render/PostProcess/SDGPipeline/RenderProduct_Replicator_DistanceToImagePlaneSDExportRawArray.inputs:swhFrameNumber of type uint64

This happens with the exact same code that first did work but then breaks without me changing anything. Once it is broken, I cannot attach the render products to a BasicWriter anymore, nor can I capture the synthetic data.

The problem seems to be in the Camera USD, because when I replace the camera again by creating a new one, it does work for a while. So what I’m doing is the following:

  • I want to capture synthetic data via an existing camera.
  • In general lines, I’m currently doing that like so:
rp_entry = ["/World/Camera", 1280, 720]
rp = rep.create.render_product(rp_entry[0], (rp_entry[1], rp_entry[2]))
  • The method of capturing the data is a simplified version of the Synthetic Data Recorder Extension.
  • I noticed that somewhere in the process the camera is forced to be invisible (as in, the eye in the hierarchy is inactive). No idea whether this is relevant to the problem, but I’d like to mention it anyway.
  • I also noticed that instead of switching to the Camera in the viewport, instead the Perspective is temporarily put to the POV of the Camera instead of actually switching to the Camera itself. Note that the Perspective is not officially changed to the Camera’s POV, since switching to the Camera in the viewport and back to the Perspective leads to the Perspective being back to its original POV.

I think that the question comes down to the following:

How do I programmatically use Replicator with an already-existing camera in the scene without the code breaking the camera? This includes fixing that the viewport should switch to the actual camera instead of temporarily changing the Perspective’s POV.

Thanks!

2

Hello @rosalie_va, thank you for reaching out! Creating a render_product with an existing camera works exactly as you’ve shown (assuming you have a camera existing at "/World/Camera"). This new render product is not connected to the viewport in any way, so if the viewport is currently set to the “Perspective” camera, the viewport should not be affected. To see from the capture camera’s perspective, you need to manually change the viewport’s camera. So the “the Perspective is temporarily put to the POV of the Camera” behaviour is unexpected. Please provide a more detailed example of the code you are running, I’d love to help you get to the bottom of this.

3

Hello

Did you find the solution or what caused it?

4

I get this same issue if the camera is under a prim. When I run the code on the original world it works, once I import that prim and make it a child of another object in a different .USD file, update the path it gives me that error.