I have two functions, one that sets the camera parameters given the intrinsic matrix and resolution with the code above provided by the docs. And one that retrieves the camera’s intrinsic matrix with the new function get_intrinsics_matrix()
Here is the problem, this is the intrinsic maxtrix I get:
the intrinsic matrix is not correct (not what I initially defined, same as in example). meaning something is wrong either with the example provided or with the get_intrinsics_matrix() function?
The rendering engine currently supports only approximate calibration in OpenCV, as documented. Specifically, it averages the fx and fy intrinsic parameters (as per example above), while the cx and cy parameters are disregarded (without a distortion model).
When using the distortion model (as in the continued example in the documentation that you’ve referred), cx and cy parameters are available, but the focal length remains an average of fx and fy.
To access the actual calibration as rendered, use the get_intrinsics_matrix() function after setting all calibration parameters. Alternatively, in ROS bridge usage, a ROS topic is populated with the applied calibration parameters.
Please, could you clarify if this approximation, when simulating a real sensor, impacts your application? Or is it sufficient as long as the simulated camera’s calibration parameters are known?
As you can see in the images I posted, there are three incorrect parameters; f_y, c_x, & c_y.
Of course, to get the second image, I had to call get_intrinsics_matrix()
About the distortion model (fisheye polynomial), I was eventually able to get the correct intrinsic matrix but not the correct distortion coefficients:
With using the get_fisheye_polynomial_properties() function:
Not sure why five variables are returned for the polynomials when the set_kannala_brandt_properties() function requires a list of four in the distortion list.
Could you please help me understand how I can view the distortion coefficients of the camera and then how to correctly set them?
About my application,
I have a camera with its own Intrinsic parameters: intrinsic matrix and distortion and my goal is to generate synthetic data with the omniverse camera. The omniverse camera should have the same intrinsic parameters as the actual camera. I am not using ROS.
Current Isaac Sim cameras only support an approximation, not the full range of OpenCV-compatible cameras. For example, only square pixels are supported (fx = fy). It is a limitation, but, in practice, sensors do have square pixels.
Please, let me get back on the set_kannala_brandt_properties question.
I’ve looked at the issue and I’ve found that a bug was introduced into camera.get_vertical_aperture() implementation, right after we’ve tested calibrated camera approximation samples. The change was not caught during test coverage. This was a breaking change; it breaks both sample code and distortion setter functions: set_kannala_brandt_properties, set_rational_polynomial_properties. There’ll be a hotfix release, to address this issue.
As a workaround, please override the camera.get_vertical_aperture function with the following code:
This can be done via either deriving from the Camera class and overriding the method or defining the function locally and patching it with camera.get_vertical_aperture = get_vertical_aperture
Hi @rthaker , I am still having this issue where the intrinstic and the distortion coefficients are not set correctly. Looking again at the examples provided by nvidia:
here, the c_x and c_y are correct, f_x and f_y are not provided so we dont know if they are correct, and the distortion coefficient are completely different from the ones provided (4 were originally given and 5 are returned).
Please, could you tell what use case do you have? Are you attempting to match a camera with a distortion model or match a pinhole camera with just the intrinsics?
It is expected that parameters reported via get_intrinsics_matrix and fTheta polynomial reported by get_fisheye_polynomial_properties are different from the ones that are initially seen at the top of the sample code.
There are limitations in the current RTX rendering pipeline that don’t allow it to use intrinsics/extrinsics directly from an OpenCV model. These limitations are:
Only square pixels are supported (so fx = fy, in the example the intrinsics are used f = (fx + fy) / 2 )
The distortion model should be enabled when cx/cy is not in the center of the frame
Rational Polynomial model is supported via fTheta approximation
Kannala Brandt is supported natively, but Kannala Brandt with depth of the field/aperture is also only supported via fTheta approximation and we’ve used that as a default
As a result, It is expected that parameters reported via get_intrinsics_matrix and fTheta polynomial reported by get_fisheye_polynomial_properties are different from the original values from which the example starts. These parameters reflect an approximation of the original values.
We haven’t seen serious issues from that approximation and the limitations stated above:
Virtually every physical sensor on the market uses square pixels. Intrinsics with non-sqare pixels most often are a wrong/incorrect estimate
The distortion model can be enabled without actual distortion, for example rational polynomial [0, 0, 0, 0, 0, 0, 0, 0]
For realistic lens polynomials that we’ve seen, the fTheta approximation residual error was subpixel across the frame
Again, we’ve seen subpixel residual error for realistic lenses
From our internal experiments (matching real cameras) we’ve noted that validating the pipeline against OpenCV reprojection is a good way to do a sanity check of the rendering pipeline. Please refer to examples in the standalone_examples/api/omni.isaac.sensor folder (camera_ros.py, camera_opencv.py, camera_opencv_fisheye.py) as a more complete starting point, that includes that validation part.
@dchichkov
thank you for the detailed responce. I must be missing something. Please help me understand. Below is an example of setting the camera intrinsic parameters with the rational polynomial model with zeros for distortion [0, 0, 0, 0, 0, 0, 0, 0]. as you can see, the distortion is not actually zero.
We have, right now, no issue with setting fx=fy, cx=width/2 and cy=height/2. We will try to find a work around for this in the future. Right now we are trying to understand how to reliably set the distortion.
Please, can you clarify if you see visual distortion where straight lines do not appear straight? Or you refer to non-zero distortion coefficients returned by the calibration toolkit?
To compare distortions, assuming the same camera intrinsics, we need to calculate the difference in distortion functions across their defined range (diagonal FOV). When intrinsics are not same, we must also consider this and calculate the mismatch using the projection function. It also can be visualized, via plotting the projection function (or a residual error) across the field of view.
Please, let me get back on the set_kannala_brandt_properties example. In that example it is quite clear that there is a distortion, but I would need to look/reproduce it, to see if we should expect to see a distortion there. Please note, the range of accepted polynomials is limited to that corresponding to a thin lens model. And arbitrarily chosen coefficients may not be approximated correctly.
