Stereo camera setup with the replicator

thomas.lindemeier2 · July 29, 2022, 2:46pm

Hey there,

how would I implement a stereo camera setup with the replicator SDK? I’ve implemented already something similar for Isaac Sim using pure USD. I want to make use of the awesome features of the replicator SDK, but I am not sure how to implement it.

My current guess is to create a parent Xform with something like this:

def stereo_camera(position, look_at):
    return rep.create._create_prim(prim_type="Xform", position=position, look_at=look_at)

rep.create.register(stereo_camera)

This parent Xform would then be randomized and the stereo cams should just stay static relative to this Xform. The parent/child transformation is clear to me, since I’ve already implemented something similar in USD already, as already mentioned. The question that I have is, how do I attach the two cameras to this Xform while also using them to create the render products:

render_product = rep.create.render_product(camera, (1024, 768))

The parameters of the camera may be constant for each run for now. Only the parent transformation should be randomized.

Any info and details would be very welcome.

Thx in advance,

Thomas

thomas.lindemeier2 · August 1, 2022, 11:00am

I think I’ve figured out how to approach this. I just use my USD implementation of the stereo cam attached to an XForm and then use the replicator to randomize the position and rotation using the modify.pose. Then I attach a RenderProduct to each camera via sdf path and create a basic writer for each render product (a list of cameras is not supported yet by the basic writer).

camera_rig = CameraStereo(Sdf.Path("/World/CameraStereo")
with rep.trigger.on_frame(num_frames=5):
                rep.modify.pose(position=rep.distribution.uniform(
                    lower=(-80, 150, 80), upper=(80, 200, 150)),
                                look_at="/World/Cube",
                                input_prims=[camera_rig.get_path()])

If I run this through my extension, I get the following errors:

2022-08-01 10:57:09 [59,899ms] [Info] [carb] Plugin omni.hydra.scene_delegate.plugin already has a dependency relationship on omni.stageupdate.plugin; not changing unload order
2022-08-01 10:57:09 [59,917ms] [Info] [carb] Plugin omni.hydra.scene_delegate.plugin already has a dependency relationship on omni.stageupdate.plugin; not changing unload order
2022-08-01 10:57:09 [59,917ms] [Info] [carb] Plugin omni.hydra.scene_delegate.plugin already has a dependency relationship on omni.stageupdate.plugin; not changing unload order
2022-08-01 10:57:09 [59,918ms] [Info] [carb] Plugin omni.hydra.scene_delegate.plugin already has a dependency relationship on omni.stageupdate.plugin; not changing unload order
2022-08-01 10:57:09 [59,918ms] [Info] [carb] Plugin omni.hydra.scene_delegate.plugin already has a dependency relationship on omni.stageupdate.plugin; not changing unload order
2022-08-01 10:57:09 [59,919ms] [Info] [carb] Plugin omni.hydra.scene_delegate.plugin already has a dependency relationship on omni.stageupdate.plugin; not changing unload order
2022-08-01 10:57:09 [59,919ms] [Info] [carb] Plugin omni.hydra.scene_delegate.plugin already has a dependency relationship on omni.stageupdate.plugin; not changing unload order
2022-08-01 10:57:09 [59,919ms] [Info] [carb] Plugin omni.hydra.scene_delegate.plugin already has a dependency relationship on omni.stageupdate.plugin; not changing unload order
2022-08-01 10:57:09 [59,920ms] [Info] [carb] Plugin omni.hydra.scene_delegate.plugin already has a dependency relationship on omni.stageupdate.plugin; not changing unload order
2022-08-01 10:57:09 [59,920ms] [Info] [carb] Plugin omni.hydra.scene_delegate.plugin already has a dependency relationship on omni.stageupdate.plugin; not changing unload order
2022-08-01 10:57:09 [59,921ms] [Info] [carb] Plugin omni.hydra.scene_delegate.plugin already has a dependency relationship on omni.stageupdate.plugin; not changing unload order
2022-08-01 10:57:09 [59,921ms] [Info] [carb] Plugin omni.hydra.scene_delegate.plugin already has a dependency relationship on omni.stageupdate.plugin; not changing unload order
2022-08-01 10:57:09 [59,922ms] [Info] [carb] Plugin omni.hydra.scene_delegate.plugin already has a dependency relationship on omni.stageupdate.plugin; not changing unload order
2022-08-01 10:57:09 [59,925ms] [Warning] [carb.flatcache.plugin] Error: array of rel not yet supported
2022-08-01 10:57:09 [59,925ms] [Error] [carb.flatcache.plugin] Error: /World/CameraStereo/CameraLeft.xformOpOrder UsdRelationship not supported in GetPrimArrayAttr

2022-08-01 10:57:09 [59,941ms] [Info] [omni.kit.menu.utils.scripts.utils] omni.kit.menu.utils.refresh_menu_items Window False
2022-08-01 10:57:09 [59,949ms] [Info] [omni.kit.menu.utils.scripts.utils] omni.kit.menu.utils.refresh_menu_items Window False
2022-08-01 10:57:09 [59,950ms] [Info] [carb] Plugin omni.hydra.scene_delegate.plugin already has a dependency relationship on omni.stageupdate.plugin; not changing unload order

2022-08-01 10:57:09 [59,975ms] [Info] [omni.replicator.core.scripts.backends.log] [tid 140514243839808][backends][disk.py] Writing data to /tmp/revip_out/camera_0
2022-08-01 10:57:09 [59,986ms] [Info] [carb] Plugin omni.hydra.scene_delegate.plugin already has a dependency relationship on omni.stageupdate.plugin; not changing unload order
2022-08-01 10:57:09 [60,001ms] [Info] [omni.kit.window.viewport.plugin] AOV LdrColorSD already added to viewport window
2022-08-01 10:57:09 [60,007ms] [Info] [omni.kit.window.viewport.plugin] AOV DistanceToImagePlaneSD already added to viewport window
2022-08-01 10:57:09 [60,019ms] [Info] [carb] Plugin omni.hydra.scene_delegate.plugin already has a dependency relationship on omni.stageupdate.plugin; not changing unload order

2022-08-01 10:57:09 [60,031ms] [Info] [omni.replicator.core.scripts.backends.log] [tid 140514243839808][backends][disk.py] Writing data to /tmp/revip_out/camera_1
2022-08-01 10:57:09 [60,040ms] [Info] [carb] Plugin omni.hydra.scene_delegate.plugin already has a dependency relationship on omni.stageupdate.plugin; not changing unload order
2022-08-01 10:57:09 [60,052ms] [Info] [omni.kit.window.viewport.plugin] AOV LdrColorSD already added to viewport window
2022-08-01 10:57:09 [60,059ms] [Info] [omni.kit.window.viewport.plugin] AOV DistanceToImagePlaneSD already added to viewport window
2022-08-01 10:57:09 [60,081ms] [Error] [omni.graph.core.plugin] Tried to end scheduler for a sync scope which does not exist.
2022-08-01 10:57:09 [60,081ms] [Error] [omni.graph.core.plugin] Tried to end scheduler for a sync scope which does not exist.
2022-08-01 10:57:09 [60,081ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2022-08-01 10:57:09 [60,094ms] [Info] [rtx.multigpumanager.plugin] View 1 (/Render/RenderProduct_Viewport_2) assigned to device 0
2022-08-01 10:57:09 [60,094ms] [Info] [rtx.multigpumanager.plugin] View 1 (/Render/RenderProduct_Viewport_2) switching to 1x1 layout
2022-08-01 10:57:09 [60,124ms] [Info] [rtx.multigpumanager.plugin] View 0 (/Render/RenderProduct_Viewport) assigned to device 0
2022-08-01 10:57:09 [60,124ms] [Info] [rtx.multigpumanager.plugin] View 0 (/Render/RenderProduct_Viewport) switching to 1x1 layout
2022-08-01 10:57:09 [60,168ms] [Info] [rtx.multigpumanager.plugin] View 0 (/Render/RenderProduct_Viewport) assigned to device 0
2022-08-01 10:57:09 [60,168ms] [Info] [rtx.multigpumanager.plugin] View 0 (/Render/RenderProduct_Viewport) switching to 1x1 layout
2022-08-01 10:57:09 [60,186ms] [Info] [rtx.multigpumanager.plugin] View 0 (/Render/RenderProduct_Viewport) assigned to device 0
2022-08-01 10:57:09 [60,186ms] [Info] [rtx.multigpumanager.plugin] View 0 (/Render/RenderProduct_Viewport) switching to 1x1 layout
2022-08-01 10:57:09 [60,186ms] [Info] [rtx.multigpumanager.plugin] View 1 (/Render/RenderProduct_Viewport_2) assigned to device 0
2022-08-01 10:57:09 [60,186ms] [Info] [rtx.multigpumanager.plugin] View 1 (/Render/RenderProduct_Viewport_2) switching to 1x1 layout

There is no randomization of the pose and rotation at all when previewing and running the replicator. Any ideas what could be wrong here?

thomas.lindemeier2 · August 2, 2022, 7:02am

I found a solution by creating the parent Xform of the two cams using the replicator as follows:

self.node = rep.create._create_prim(prim_type="Xform",
                                            position=world_translation,
                                            rotation=orientation)

Now, I can use the replicator functionality. The only thing that i haven’t figured out, yet, is how the sdf path of a ReplicatorItemcan be retrieved.
I just hardcode if for now to what is done in the _create_prim function:

self.camera_rig_path = Sdf.Path("/Replicator/Xform")

system · August 21, 2022, 6:17pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

jiehanw · August 26, 2022, 5:21pm

Hi @thomas.lindemeier2 , as of the newest version of replicator, there is a function that can create a stereo camera pair:

rep.create.stereo_camera(stereo_basline=0.2)

It will create an xform that contains two cameras as its children.