Camera pose in world frame via python

Omniverse Isaac Sim
1

Hey guys,

I am working on creating a synthetic dataset offline and have a question regarding the pose of a camera in the world frame (extrinsic matrix). I created a camera with the replicator (rep.create.camera()) and modify its pose randomly in each frame with a trigger for each frame. How can I access this varying pose per frame?

Thank you very much!

2

Hi @spiegeln - You can access the pose of a camera in the world frame by using the omni.usd.get_world_transform_matrix() function from the Omniverse Kit API. This function returns the world transform matrix for a given USD prim, which in your case would be the camera.

Here is a code snippet that demonstrates how to get the world transform matrix of a camera:

import omni

# Assuming `camera` is your camera prim
camera_transform_matrix = omni.usd.get_world_transform_matrix(camera)

The camera_transform_matrix is a 4x4 matrix that represents the pose of the camera in the world frame. This matrix includes both the rotation and translation of the camera.

If you want to get the translation (position) and rotation separately, you can extract them from the transform matrix like this:

# Get the translation (position) of the camera
camera_position = camera_transform_matrix.ExtractTranslation()

# Get the rotation of the camera
camera_rotation = camera_transform_matrix.ExtractRotation().GetQuaternion()

Remember that the pose of the camera is updated in each frame, so you would need to get the world transform matrix inside your frame trigger to get the updated pose for each frame.

3

Hi, Is there a way to do this in IsaacGym? I am finding it really difficult to concatenate point clouds based on transoformation to world

4

Hi @atreya29 - Yes, you can do it via USD APIs. It should have APIs for retrieving transforms of camera prims.

5

The USD APIs you talking about are only available in IsaacSim right? I am using IsaacGym preview 4
Can u give an example based on below snippets

I am using older Isaacgym to make a RL environment
I have a rgbd camera on franka gripper and it is attached.
I am going to read the image, deproject to pointcloud and concatenate them after each observation, but I am unable to do it without transforming to base

This is what I have tried till now,

  1. camera transform from camera handle, But this doesnt update once sim starts
        P = [transform_to_base.p.x, transform_to_base.p.y, transform_to_base.p.z]
        R = [transform_to_base.r.x, transform_to_base.r.y, transform_to_base.r.z, transform_to_base.r.w]
        T = np.eye(4)
       T[:3, 3] = P
        [:3, :3] = o3d.geometry.VoxelGrid.get_rotation_matrix_from_quaternion(np.array(R, dtype=np.float32))
        o3d_pc_split.transform(camera_transform)
  1. Use gripper location from acquire_dof_states and use that to transform, which isnt working properly

#right finger pose w.r.t base
finger_pose = self.rigid_body_states[0][9][:7].cpu().numpy()
T = np.eye(4)
T[:3, 3] = [finger_pose[0] + 0.06, finger_pose[1], finger_pose[2]]
T[:3, :3] = o3d.geometry.VoxelGrid.get_rotation_matrix_from_quaternion(np.array([finger_pose[3],finger_pose[4],finger_pose[5],finger_pose[6]], dtype=np.float32))
camera_transform = T
o3d_pc_split.transform(camera_transform)

6

Hi @atreya29,

Why getting a gripper pose is not working for you? You can directly get gripper pose using Isaac Gym API, you don’t need to calculate it from dof state.

7

Can you give an example. Getting the gripper pose transformed to world frame is really important for me and I am not finding a proper method for it. Please help