How to Align Camera Pose, RGB, and Depth for Accurate 2D-3D Pixel Mapping for Dynamic Camera?

[char-lotte]

I am working on retrieving the camera’s pose at a given position, including position (xyz) and orientation (px, py, pz, pw), along with RGB and depth data. My goal is to map each pixel in a 2D image to its corresponding 3D position in order to generate a 3D semantic map by labeling 2D pixels and projecting them into 3D space. I’m using isaacsim 4.0.0 camera class. For projection, I use:

pc = camera.get_world_points_from_image_coords(grid_2d, depth.flatten())

I obtain rgba and depth using the camera class functions get_rgba() and get_depth(). However, I notice that the mapped point cloud is misaligned, as shown in the image below.

I found a related forum post: Point clouds generated by Camera.get_pointcloud() are not matched - NVIDIA Developer Forums, but it seems that get_rgba() and get_depth() are already using Replicator.

How should I resolve this issue to correctly align the point cloud and avoid the misalignment?

[phennings]

Hello, it seems you are plotting multiple frames all together. Could you provide an example with data from a single frame with less points in the pointcloud to verify the misalignment you are referring to? could you post the image you are using for this and the code? Thank you.