[Bug] ROS2 OmniGraph Camera Info Camera Intrisics

ahamboeck · September 15, 2025, 12:24am

hi all,

i’m seeing a mismatch between two point clouds that are both produced by the same ROS 2 OmniGraph camera node. in isaac sim 5.0 + ros 2 jazzy, i visualize /depth_pcl as PointCloud2 in rviz2 and also reproject the depth image from that same node using rviz2’s DepthCloud display (which consumes the node’s camera_info). since both /depth_pcl and the depth image come from that single ros 2 omnigraph camera node, these two clouds should overlap perfectly—but in 5.0 they don’t. when i run the exact same USD scene in isaac sim 4.5 + ros 2 humble, they do align.

for intrinsics i’m following the usual formulas from the docs (fx = width_px * focalLength / horizontalAperture, fy = height_px * focalLength / verticalAperture). the documentation wasn’t fully explicit about whether to use pixel resolution or nominal sensor size, so i verified by calculation: using the pixel resolution makes fx match my actual image width (1280 px) in the K matrix, which convinces me that’s the intended interpretation. given width ≠ height and horizontalAperture ≠ verticalAperture, fy should differ from fx. in isaac sim 5.0, the camera_info K matrix looks correct per those formulas, yet /depth_pcl and the depthcloud-derived cloud don’t overlap in rviz2. in isaac sim 4.5, fx looks correct but fy is wrong—it’s identical to fx even though they should be different (width and height are not the same, and verticalAperture and horizontalAperture differ). despite that apparent fy issue in 4.5, the two clouds actually do align there.

to reproduce, i created a minimal USD with one camera, one ros 2 omnigraph camera node (publishing rgb, depth, /depth_pcl, and camera_info), and a tf from world → camera. i then ran the same USD first in isaac sim 5.0 + jazzy, opened rviz2 (jazzy) and visualized /depth_pcl (pointcloud2) and the depth image via depthcloud (using the matching camera_info). next, i ran the same USD in isaac sim 4.5 + humble, opened rviz2 (humble), and visualized the same topics. result: misalignment in 5.0; alignment in 4.5. as a sanity check i also compared a lidar point cloud.

for easy comparison, i’m pasting screenshots and messages below:

rviz (isaac sim 4.5 — aligns):

/camera_info (isaac sim 4.5):

header:
  stamp:
    sec: 1795
    nanosec: 383426969
  frame_id: sim_camera
height: 720
width: 1280
distortion_model: plumb_bob
d:
- 0.0
- 0.0
- 0.0
- 0.0
- 0.0
k:
- 1108.5125019853992           -> fx
- 0.0
- 640.0
- 0.0
- 1108.5125019853992           -> fy? why is it equal to fx?
- 360.0
- 0.0
- 0.0
- 1.0
r:
- 1.0
- 0.0
- 0.0
- 0.0
- 1.0
- 0.0
- 0.0
- 0.0
- 1.0
p:
- 1108.5125019853992
- 0.0
- 640.0
- 0.0
- 0.0
- 1108.5125019853992
- 360.0
- 0.0
- 0.0
- 0.0
- 1.0
- 0.0
binning_x: 0
binning_y: 0
roi:
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: false
---

rviz (isaac sim 5.0 — does NOT align, I changed the color from white to AxisColor so it is more visible that these do not align):

/camera_info (isaac sim 5.0):

header:
  stamp:
    sec: 44
    nanosec: 966669011
  frame_id: sim_camera
height: 720
width: 1280
distortion_model: plumb_bob
d:
- 0.0
- 0.0
- 0.0
- 0.0
- 0.0
k:
- 1108.5125019853992           -> fx
- 0.0
- 640.0
- 0.0
- 854.5167406926195           -> fy
- 360.0
- 0.0
- 0.0
- 1.0
r:
- 1.0
- 0.0
- 0.0
- 0.0
- 1.0
- 0.0
- 0.0
- 0.0
- 1.0
p:
- 1108.5125019853992
- 0.0
- 640.0
- 0.0
- 0.0
- 854.5167406926195
- 360.0
- 0.0
- 0.0
- 0.0
- 1.0
- 0.0
binning_x: 0
binning_y: 0
roi:
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: false
---

camera intrinsics set in isaac sim in both simulations (settings panel):

i’m trying to understand two things:

(1) why /depth_pcl and the depthcloud-derived cloud don’t align in isaac sim 5.0 even though the camera_info intrinsics look correct by the formulas

(2) why they do align in isaac sim 4.5 even though fy in camera_info appears to be incorrect (equal to fx when it shouldn’t be).

for reference, i’m basing the formulas on the isaac sim docs—please see (insert documentation link here) next to the camera intrinsics section: ROS 2 Cameras — Isaac Sim Documentation

the USD is attached if anyone wants to cross-check Filebin | gj7sxghfewblgmfc

I am glad for any help!

VickNV · September 15, 2025, 7:37pm

Thank you for bringing this issue to our attention and for providing such detailed observations and background.

We’re currently checking with our internal team to better understand the differences you’ve observed between Isaac Sim 4.5 and 5.0 regarding the camera intrinsics and point cloud alignment.

VickNV · September 15, 2025, 8:16pm

Unfortunately, our IT policies are currently blocking access to external file sharing sites like Filebin, so we’re unable to retrieve the USD at the moment.

Could you please upload the USD file directly to this forum thread?

ahamboeck · September 15, 2025, 10:16pm

The USD file is bigger than 100mb so I can’t upload it here but the process is quite simple.

Create a simple room enviornment
Add a camera to the prim
Tools → ROS2 OmniGraphs → Camera (select rgb, depth, depth_pcl and set the Frame ID to the same as the camera name)
Tools → ROS2 OmniGraphs → TF Publisher → Select the Camera as the Target Prim
Place the camera to look into the room
Collect and save the USD

Then start it with isaac sim 4.5 and isaac sim 5.0 (you might need to delete the Render first under Layers) and compare the two depth topic with the depth_pcl topic in rviz by selecting Depthcloud for the depth topic and PointCloud2 for the depth_pcl topic.

ahamboeck · September 15, 2025, 10:54pm

I dug a little deeper into the Isaac Sim 5.0 code and found this line:

github.com/isaac-sim/IsaacSim

source/extensions/isaacsim.core.nodes/nodes/OgnIsaacConvertDepthToPointCloud.cpp

07d8dd47e


      
          }
          
          auto& height = db.inputs.height();
          auto& width = db.inputs.width();
          
          auto& state = db.perInstanceState<OgnIsaacConvertDepthToPointCloud>();
          
          float fx, fy, cx, cy;
          
          fx = width * db.inputs.focalLength() / db.inputs.horizontalAperture();
          fy = height * db.inputs.focalLength() / (db.inputs.horizontalAperture() * (static_cast<float>(height) / width));
          cx = width * 0.5f;
          cy = height * 0.5f;
          {
              isaacsim::core::includes::ScopedDevice scopedDev(db.inputs.cudaDeviceIndex());
              uint64_t handle = db.inputs.dataPtr();
              isaacsim::core::includes::ScopedCudaTextureObject srcTexObj(
                  reinterpret_cast<cudaMipmappedArray_t>(handle), 0);
              state.m_buffer.resize(db.inputs.width() * db.inputs.height());
              depthToPCLOgn(state.m_buffer.data(), srcTexObj, width, height, fx, fy, cx, cy);
          }

This generates the depth_pcl topic from the depth image. Here, fy isn’t calculated the way the docs describe.

According to the docs:
(1) fy = height * focalLength / verticalAperture

But in the code it’s done as:
(2) fy = height * focalLength / (horizontalAperture * (height / width))

If you simplify (2), it becomes:
fy = width * focalLength / horizontalAperture

Which is the same as fx:
fx = width * focalLength / horizontalAperture

So the code effectively forces fx = fy. That only matches the usual pinhole model if pixels are square and the sensor aspect ratio matches the image resolution:
(3) horizontalAperture / width = verticalAperture / height

When I set the apertures in Isaac Sim so that (3) holds, depth_pcl lines up with the point cloud I derive from the depth topic.

It seems (not 100% sure) that in Isaac Sim 4.5 this was also how the camera_info intrinsic were computed, which doesn’t line up with the docs:

In Isaac Sim 5.0, when retrieving camera_info through ROS2CameraInfoHelper:

github.com/isaac-sim/IsaacSim

source/extensions/isaacsim.ros2.bridge/python/nodes/OgnROS2CameraInfoHelper.py

07d8dd47e


      
              return False
          if db.inputs.renderProductPathRight:
              is_stereo = True
          
          db.per_instance_state.resetSimulationTimeOnStop = db.inputs.resetSimulationTimeOnStop
          db.per_instance_state.publishStepSize = db.inputs.frameSkipCount + 1
          
          camera_info_left, camera_left = read_camera_info(render_product_path=db.inputs.renderProductPath)
          
          if is_stereo:
              camera_info_right, camera_right = read_camera_info(
                  render_product_path=db.inputs.renderProductPathRight
              )
          
              width_left = camera_info_left.width
              height_left = camera_info_left.height
              width_right = camera_info_right.width
              height_right = camera_info_right.height
              if width_left != width_right or height_left != height_right:
                  carb.log_warn(
                      f"Mismatched stereo camera resolutions: left = [{width_left}, {height_left}], right = [{width_right}, {height_right}]"

…it looks like it uses verticalAperture directly:

github.com/isaac-sim/IsaacSim

source/extensions/isaacsim.ros2.bridge/python/impl/camera_info_utils.py

07d8dd47e


      
          fy = height * focalLength / verticalAperture

So now I get why the two point clouds don’t match. My question is: **why does formula (2) still appear to “work” in practice when we don’t have square pixels (i.e., condition (3) isn’t satisfied)?

**
So now I get why the two point clouds don’t match. I guess the reason formula (2) still appears to “work” even when condition (3) isn’t satisfied is because, when you set the horizontal or vertical aperture to a value that doesn’t match (3), Isaac Sim just forces it to be squared. This can actually be seen in the docs here:

https://docs.isaacsim.omniverse.nvidia.com/latest/py/source/extensions/isaacsim.sensors.camera/docs/index.html

If that’s correct, then I think it should really be better explained in the documentation and also changed in the code so the behavior is clear.

VickNV · September 16, 2025, 7:23pm

Thank you for bringing this to our attention.
We have created an internal ticket to track the issue. Our team is currently reviewing how to address the mismatch, and we will provide updates as soon as a resolution path is determined.

VickNV · October 3, 2025, 10:39pm

Thank you for raising and thoroughly documenting this issue with camera intrinsics and point cloud alignment in Isaac Sim 5.0.

Our team has carefully investigated your report and identified the root cause: a mismatch between how pixel aspect ratios were handled and the assumptions made by the renderer, CameraInfo, and point cloud generation APIs. Specifically, while recent logic used the actual verticalAperture property, the renderer and related APIs implicitly required square pixels. This discrepancy sometimes resulted in inconsistent camera intrinsics and misaligned outputs.

We’re happy to share that this issue has been resolved and the fix is now included in the Isaac Sim 5.1.0 update, available on our GitHub repository. The update restores consistent use of square pixels across the relevant components, so fx and fy are now aligned as expected.

Topic		Replies	Views
Incorrect depth image generated by camera in IsaacSim Isaac Sim camera , ros	6	2771	April 5, 2024
CameraInfo ROS2 Topic to match real camera data Isaac Sim camera , isaac-sim-v5-0-0	7	163	September 18, 2025
Depth Map to Pointcloud Issues Isaac Sim camera , synthetic-data , depth-sen	8	3966	January 7, 2022
Isaac Sim "opencv_fisheye" lens distortion model is not consistent with OpenCV's projectPoints Isaac Sim camera , synthetic-data , isaac-sim-v5-0-0	11	517	September 5, 2025
Doubts on publishing sim_camera data via python standalone Isaac Sim	8	1853	December 15, 2023
Camera point cloud shake when rotate Isaac Sim camera , ros	9	1799	April 5, 2024
Point clouds generated by Camera.get_pointcloud() are not matched Isaac Sim camera	4	556	November 25, 2024
Adjust camera orientation in Isaac Sim for correct view of pointcloud in rviz Isaac Sim	9	950	September 8, 2025
Lagging, unaligned point clouds from depth cameras Isaac Sim isaac-sim-v5-0-0 , isaac-sim-v5-1-0	7	140	April 17, 2026
Isaac Sim \| ROS2 \| Point cloud from camera has wrong orientation Isaac Sim camera , ros , isaacsim	3	909	December 28, 2022

[Bug] ROS2 OmniGraph Camera Info Camera Intrisics

Related topics