Replicator Camera Orientation

Hello,
I am using the replicator create camera function to generate synthetic data. This camera is randomized using the approach mentioned in the documentation. I already have a list of orientations and positions parameters for the camera to use. These parameters have been collected manually while navigating in my scene using Isaac Sim, and annotating the postion / orientation in the desired locations. While the positions seems to be correct, there is a major problem in the orientation. None of the annotated angles give me the same angle as I earlier saw them in the viewport.

What might be the problem?
Thank you in advance

Best
Anthony

@toninsemaan is it possible for you to upload a screenshot or do a side by side comparison to show what you are getting from the annotated angles vs from within the viewport?

Hi @toninsemaan ,

More information would be needed to help debug your issue.

Screenshots/images, how you are setting the position and orientations for the new camera, also how you are getting the orientations.

Sure!

Create a camera

import omni.replicator.core as rep
rep_camera = rep.create.camera()

Get the orientation / position of all the set cameras in the usd scene and launch the randomizer

stage = omni.usd.get_context().get_stage()
cams_positions = []
cams_rotations = []
camera_paths = ["/World/Camera", "/World/Camera_01"]
for cam_path in camera_paths:
        prim: Usd.Prim = stage.GetPrimAtPath(cam_path)
        orient = prim.GetAttribute("xformOp:orient").Get()
        translate = prim.GetAttribute("xformOp:translate").Get()

        #Then, since the above command gives me the orientation in quad, I am transforming them to euler using the following:
        rotation = quat_to_euler_angles(quat=gf_quat_to_np_array(orient), degrees=True).tolist()

        cams_rotations.append(rotation)
        cams_positions.append(translate)

cams_positions_float = []
cams_rotations_float = []
for i in range(len(cams_positions)):
        if cams_positions[i] is not None and cams_rotations[i] is not None:
                cams_positions_float.append((cams_positions[i][0], cams_positions[i][1], cams_positions[i][2]))
                cams_rotations_float.append((cams_rotations[i][0], cams_rotations[i][1], cams_rotations[i][2]))

# Randomize Camera
with rep.trigger.on_frame(num_frames=len(cams_positions_float)):
        with rep_camera:
                rep.modify.pose(position=rep.distribution.sequence(cams_positions_float),
                                rotation=rep.distribution.sequence(cams_rotations_float))
rp = rep.create.render_product(rep_camera, (512,512), True)
writer.attach([rp])

While the current viewport of the camera are as follows:


I am the getting rgb synthetic data:


It feels like the orientation of the camera is following a completely different axis.

Thank you for you time,
Best

Anthony

Hello @Simplychenable @dennis.lynch,

Is there any updates on the following issue?

Apologies for the inconvenience.

Thank you,
Best
Anthony

just fyi, i am just another user so you can take my input as a grain of salt. always air on the side of what the dev has to say.

that said, regarding the orientation, i am not sure what you are seeing pertains to the z-up discrepancy mentioned in this post. and another optional step if see further mismatch in terms of what you are seeing in the render output, you could try capturing additional camera attributes such as focusDistance, focalLength, clippingRange, etc so it matches exactly to the camera setting used in Stage.

Hi, I found the similar problem with you, and I wrote a pose of it, hope it can help you out.
Or you can just read the following “Solution part”.

Solution

According to the discussion in the form, I think its a designed mechanism to automatically keep the same camera coordinate system(+Y up) across different USD files.

And if you want to cancel this “auto-rotation” effect, you can use camera_axes="usd" args in set_world_pose().

from omni.isaac.sensor import Camera
camera = Camera(
    prim_path="/World/camera",
    position=np.array([0, 0, 0]),
    orientation=np.array([1, 0, 0, 0]),
)
camera.set_world_pose(np.array([0, 0, 0]), np.array([1, 0, 0, 0]), camera_axes="usd")
my_world.step(render=True)

This way, the default camera pose will aligned with the world frame.
Pasted image 20240501181344 Pasted image 20240501181355

You can find more detailed information about set_world_pose() in the API docs of it here.