SuperpixelProto Frame of References

TLDR: How to transform superpixel 3d locations to world frame, what is the correct way to get transform from world frame to superpixel reference frame. Can this be mentioned in the documentation if possible?

Hi,

The SuperpixelProto documentation shows that surflets have 3D point information stored in them. I have saved that data, along with the camera transformation that I got from something like the following:

stage = usd.get_context().get_stage()
prim = stage.GetPrimAtPath("/environments/env_0_0/Camera_right")
pose  = omni.usd.utils.get_world_transform_matrix( prim )

Where I reshaped the produced pose into a homogeneous matrix.

Now I want to visualize (in matplotlib or whatever) the superpixel 3d locations in world frame reference.

What I tried

I imagined it would be as simple as dot product of p_world = T_world_cam (dot) p_cam. With all translatons in the T scaled by 0.001 because it seems that the SIM uses mm while the superpixels stored in meters.

But it seems that that is not the correct transformation. In fact plotting the p_cam points in camera frame, also shows that the depth is along the z-axis ( I am guessing that’s the depth image standard ).

So I tried the following:

axes_ch = np.array([
    [1,  0, 0, 0],
    [0,  0, 1, 0],
    [0,-1, 0, 0],
    [0,  0, 0, 1]
])
axes_ch2 = T_w_to_camR_real
r_rot = T_w_to_cam.dot(axes_ch).dot(<Homogeneous points variable with shape 4xLength>)
# and r_rot = axes_ch.dot(T_w_to_cam).dot(<Homogeneous points variable with shape 4xLength>)
ax.scatter(xs = r_rot[0,:], ys=r_rot[1,:], zs=r_rot[2,:])
plt.show()

How are the rotations applied in Isaac anyway? is it Euler angles XYZ, or is it with respect to the world XYZ vectors?

The distances I get when I change the axes only, dont make sense to me unless the camera frame does not coincide with the frame of the camera object. Is there anything I am missing?