Set exact intrinsic matrix of camera fails

Omniverse Isaac Sim
1

Hi community,

I want to mimic a real world camera of a robot. Thus, I want to precisely set the intrinsic matrix. I am using this code to set the parameters:

        # extract parameters from matrix
        f_x = intrinsic_matrix[0, 0]
        c_x = intrinsic_matrix[0, 2]
        f_y = intrinsic_matrix[1, 1]
        c_y = intrinsic_matrix[1, 2]
        # get viewport parameters
        height, width = self.image_shape
        height, width = float(height), float(width)
        # resolve parameters for usd camera
        params = {
            "focal_length": focal_length,
            "horizontal_aperture": width * focal_length / f_x,
            "vertical_aperture": height * focal_length / f_y,
            "horizontal_aperture_offset": (c_x - width / 2) / f_x,
            "vertical_aperture_offset": (c_y - height / 2) / f_y,
        }
        # set parameters for camera
        for param_name, param_value in params.items():
            # convert to camel case (CC)
            param_name = to_camel_case(param_name, to="CC")
            # get attribute from the class
            param_attr = getattr(self._sensor_prim, f"Get{param_name}Attr")
            # set value
            # note: We have to do it this way because the camera might be on a different layer (default cameras are on session layer),
            #   and this is the simplest way to set the property on the right layer.
            omni.usd.utils.set_prop_val(param_attr(), param_value)

Especially, the horizontal and vertical aperture offset do not seem to be set in the simulator. Would be a good solution to have a function that receives an intrinsic matrix and the focal length and sets all the corresponding parameters correctly.

To reproduce, I am using a camera with focal_length=1.0, width=1440 and height=1080 with intrinsic matrix= (575.60504, 0.0, 745.73121, 0.0, 578.56484, 519.52070, 0.0, 0.0, 1.0).

System Info:
Isaac Sim Version: 2022.2.1-rc.14
OS: Ubuntu 20.04

2

Hi @roth.pascal - The intrinsic matrix you provided seems to be in the format of:

fx  0  cx
0  fy  cy
0   0   1

The horizontal_aperture, vertical_aperture, horizontal_aperture_offset, and vertical_aperture_offset parameters in USD are used to define the camera’s field of view and the lens shift. These parameters are not directly equivalent to the intrinsic parameters fx, fy, cx, and cy.

The horizontal_aperture and vertical_aperture parameters in USD are defined in millimeters, and they represent the size of the camera’s film back. The horizontal_aperture_offset and vertical_aperture_offset parameters represent the lens shift, which is the offset of the lens from the center of the film back.

To convert from the intrinsic parameters to the USD parameters, you can use the following formulas:

horizontal_aperture = image_width_in_pixels * pixel_size_in_mm / fx
vertical_aperture = image_height_in_pixels * pixel_size_in_mm / fy
horizontal_aperture_offset = (cx - image_width_in_pixels / 2) * pixel_size_in_mm
vertical_aperture_offset = (cy - image_height_in_pixels / 2) * pixel_size_in_mm

Here, pixel_size_in_mm is the size of a pixel in millimeters. This value depends on the specific camera sensor you are using, and you would need to look up this value in the camera’s specifications.

Please note that these formulas assume that the camera’s coordinate system is right-handed with the x-axis pointing to the right, the y-axis pointing up, and the z-axis pointing towards the viewer. If your camera’s coordinate system is different, you might need to adjust the formulas accordingly.

Also, please note that the horizontal_aperture_offset and vertical_aperture_offset parameters in USD represent the lens shift, which is a relatively small effect that is usually only noticeable in wide-angle lenses or when using a tilt-shift lens. If you are not seeing a noticeable effect when changing these parameters, it might be because your camera’s field of view is not wide enough, or because the lens shift is too small to be noticeable.

3

Hi @rthaker,

I am trying to implement Intel Realsense D435 using it’s intrinsics but I couldnt fully understand the pixel_size_in_mm. The image pixel size according to Intel’s website is 1.4 µm (1.4 x 10E-3 mm) and this results in a very small horizontal aperture. So should the pixel size be in mm or µm?

5

@vishal.balaji1 i’m wondering the same. I came across one of the solutions here: Change intrinsic camera parameters - #5 by xuzy70

focal_y = width * focal_length / horiz_aperture
meaning:
horiz_aperture = width * focal_length / fy.

which i guess makes more sense in terms of the numbers.

@rthaker
is it possible that it should really be focal_length instead of pixel_size_in_mm?

Thank you

6

Hi @danielle.sisserman ,

We’ve updated documentation to address this topic, please refer to Calibrated Camera Sensors section, it includes a snippet of code and a standalone reference example that shows how to use sensor calibration and aperture parameters commonly found in a sensor spec and sensor calibration output.

And yes, the choice of units for the optical path/aperture is suboptimal, I understand that there’ve been backward compatibility constraints that were not allowing to change the units.

2 Likes