How to make use of Equidistant fisheye model in Isaac Simulation

Omniverse Isaac Sim
1

Hi All,

I’m pretty new to Omniverse and Isaac simulation. I recently tried to do something with robot simulation samples and add some different cameras to test. I read the docs and found all the distortion parameters are not in use for the Equidistant fisheye model. Though may not fully support, since the model is there, I suppose there should be a default model and setting for the Equidistant model. I tried OpenCV’s cv::fisheye::undistortImage function but the output images are not same with the images recorded by a pinhole camera at same position with same lens setting. My questions are:

  1. Do I have any way to retrieve Original pinhole camera model image with the default Equidistant fisheye camera images with same aperture and focal length with either OpenCV or just mathematical models? Or can we know the default Equidistant fisheye camera lens projection and distortion parameters?

  2. Is it possible to make use of the Equidistant camera model or are there any other first or third party alternative ext. to implement fisheye camera with Equidistant model?

Thanks,
SF

2

Hi @ShakeyF , does this document (Fisheye properties section) helps you answer your questions?: Cameras — Omniverse Robotics documentation

3

Hi @rthaker, I’ve tried a lot with the provided docs, but as I said, all distortion settings are not for Equidistant model in fisheye camera sections. That’s why I’m asking if there’s a default distortion and parameter of the provided Equidistant model.

4

The equidistant camera model is an ideal camera model, it doesn’t respond to other distortion control as some of the more advanced camera model do. Equidistant only responds to the camera focal length and aperture. Internally, those two values are used to compute a horizontal field of view (measured in radians) the exact sample way they are for the pinhole camera.

Now, fisheye lens models tend to work radially starting at the center of the screen. So, the horizontal field of view is halved, representing an angle from center to the side, instead than from side to side. In the radial projection, the equidistant model maps this angle linearly, without any distortion or compression, as explained in this Wikipedia article. As a result, the angle spanning horizontally across the screen from left to right passing through the center of the image matches the horizontal field of view of the pinhole camera.

Does this help?

5

Hi @mliani ,

Thanks for your information. I think I may mixed two things of camera model and lens distortion. So I understand the model in the simulation is currently an idea model

Meanwhile even though I follow the wiki, the actual captured result seems not match? For pinhole model the incremental is tangent value and the equidistant model is linear value, same 3d point should be closer to the image center in pinhole image than the equidistant image. But after I tried collecting some, the result is not match with this. Example:

image
image1349×625 113 KB

6

I am not sure I understand what you mean by “match”. I believe the image to the right rendered from a pinhole camera, all lines appear straight, the objects near the center of the image appear smaller than they should and these at near the edges are stretched out. The image to the right is a fisheye, and it seems to be an equidistant lens: objects at the center of the image appear more natural, objects at the edges are compressed in a radial direction.
If you would place some distinct small object in the 3d scene so that they cross half way the very edge of the image, visually placed them on a horizontal line that passes through the image center, you should see that those objects will appear crossing the edges of the screen in the exact same 3d position between the two camera models.

Are you saying these images are renderer with some other software, and the Omniverse RTX renderer doesn’t match this result? Or is this rendered in Omniverse, and somehow it doesn’t match your expectation of what it should look like?

Consider that in the RTX renderer, we do not compute a frame from a pinhole camera and distort it to approximate a fisheye lens. We render directly the fisheye lens by generating rays from the camera in the correct directions.

7

Hi @mliani ,

Thanks for your suggestion of trail. I’m actually talking about the Omniverse Isaac camera rendering not matching my expectation even though rendering from direct ray.
I get this image from top of google search on ‘equidistant camera’

image
image600×654 32.2 KB

If my understanding is correct, for each 3d point projection to the image plane, rd < ru. That’s why I’m saying each corresponding point in fisheye image should be closer to the image center compare to pinhole camera image (with same focal length and aperture). For the image pair I provided, I will consider the April tags in the right fisheye image should be smaller than those in the left pinhole image.

8

For what I mentioned distortion, I refer to the openCV docs on Fisheye camera(OpenCV: Fisheye camera model), the fisheye image should be able to pixel-wise project to a pinhole image match with a pinhole image captured by the same focal length and aperture (intrinsic parameters). Here is the result I use cv2.fisheye.undistortImage function to recover the pinhole undistorted image (right) from fisheye equidistant image (mid), and it looks like an image with different focal length compared to direct pinhole image (left) captured with same focal length and aperture settings in the simulator.

image
image1920×587 68.3 KB

9

Also mention the similar function in Matlab (Correct fisheye image for lens distortion - MATLAB undistortFisheyeImage ) You may check the comparison between ‘Original Image (left) vs. Corrected Image (right)’ part to see what I mean about the object size change.

Thanks again in advance
SF

10

I think part of your intuition is backwards. The image plan in a pinhole camera is the tangent. ru is tan(theta) * f. Consider an extreme case to adjust the intuition. If you set theta to approach 90deg, ru is going to approach infinity. The pinhole camera is going to map close to negative infinity to close to positive infinity in the image plane. Everything in front of the camera, will be sqeezed in a miniscule spot at the center of the image.
In an equidistant projection, each angular increment will be spread over an equal linear number of pixels (measured radially). So objects at the center of the image will appear natural, and objects at the edge of the screen will appear compressed. Like I said in a previous message.
Try to change the focal length with both project and observe this behavior interactively. You should notice what I described here.

I don’t know why OpenCV distortion process results in a cropped view. The fact that the objects in the resulting image have straight lines, and not something weirdly distorted, makes me guess the projection is just fine and there are settings for the user to preserve the best resolution (around the center of screen) or to fit the field of view. I don’t really know what is available in there. I have never used OpenCV.

11

@mliani For what you described here really confused me:
“Everything in front of the camera, will be sqeezed in a miniscule spot at the center of the image.”
For camera projection, there should not be a squeezing step for real camera, if you got really large ru, it should fall out of the camera sensor range rather than squeeze anything. Just as the sphere image I showed above, for a image, you just have a fixed height or width, which will shown as fixed r range. Everything have rd or ru larger than the set range r should not appear in the image captured. That’s why I say since the tangent function ru (pinhole projection) is always larger than the linear rd (equidistant), the information range of in the pinhole image should be smaller than that in equidistant image.
OpenCV function is not showing a cropping view, if you choose to inverse way which is project the pinhole view to the equidistant view, it should just show a circle-like view (information shrink to center) in a smaller range compared to the image size set.
The object shows the straight line, that means it is a real equidistant-like model, and the size not match means the simulation model parameter does not match with the physical meaning from my understanding. If you would like to check with those functions with some representative points, I think you will know what I mean here.
For a point with theta 45 degrees (pi/4, 0.785) shows on the edge of a 400*400 pinhole image (center at (200,200)), with pixel index(200,0), tangent(theta) = 1. The representative coordinate in equidistant image will be ( 200, (1-0.785)*200) = (200,43). That’s exactly what happened in the right image and mid image I showed you (the angle may vary)

12

@rthaker I think mliani is not there for a while, could be please let someone else know about my question?

13

This is getting very confusing indeed. There is no squeezing step, and there is no “fall out of the camera sensor range rather than squeeze anything”. I am not sure how to explain this better, but I’ll try.

Let’s go back to basic: in a pinhole camera model there are only 2 information that counts: field of view and the frame aspect ratio. Focal length and the horizontal film aperture are used to compute the horizontal field of view (aka as angle of view). Such field of view is exactly the same for a pinhole camera and for a fisheye camera, so the extent of scene each projection model displays horizontally is exactly the same. The difference between the two projections is in how that extent is mapped, and it will affect more drastically what is shown around the image center. Pinhole is mapped according to tan(theta), equidistant according to theta. The extent is the same the distortion is different. Both models show a type of distortion if compared to human vision, I hope there are no questions about this.

So here is a picture showing the same scene with different values of focal length, from 30mm down to 0.5mm. The left column shows pinhole cameras, the right column shows the exact same focal lengths on equidistant projections. Placed in front of the camera is a red sphere. The camera position and everything in the scene is fixed, only the focal length changes.

pinhole_fisheye_example
pinhole_fisheye_example1486×2115 259 KB

If we compare to human vision, the equidistant fisheye lens looks less distorted and more natural than pinhole. Agreed?
With relatively long lenses, the difference is minor, to the same extent than tan(theta) and theta are similar for low values of theta, correct?
When the angle of view increases, the visual distortion produced by pinhole camera becomes more and more apparent and unnatural. Look at the last row, which is some extreme wide-angle lens: the equidistant projection still looks natural, the sphere still looks like it is relatively near the camera. Conversely, the pinhole camera looks completely artificial and massively distorted: the sphere in front of the camera is barely visible and is impossible to tell how far from the camera that may be by looking at the picture, “as if the center of the image is squeezed in a miniscule spot” and what is at the edge of the screen is massively stretched. Do you agree that is the visual effect produced by a pinhole camera?
You can try this in any 3d editor where you have control over the camera field of view, approaching 180deg of field of view, this is what you get.

Now, let me reiterate. Unfortunately, I don’t know why OpenCV seems to crop the result. We will try to find somebody here that knows OpenCV, who may be able to answer your specific question. But here is an observation: the fact that OpenCV process the equidistant lens produced by Omniverse, and by undistorting it gives an image showing objects defined by straight lines, this makes me think that the undistortion is done correctly, but the algorithm cropped the image. If there was a bug in the Omniverse equidistant projection, the undistortion in OpenCV would produce an image where 3d lines would still be curvy/wobbly in some different way.

14

Hi @mliani , Thanks for your detailed analysis. I’m not considering any process of the simulator is not perfect. My only question is, if I can not retrieve pinhole camera image from equidistant image or opposite way in my code level, I could not make use of the result generated by the issac simulator since that means I cannot use same intrinsic for these two kind of lens (previously defined by focal length, aperture and image size in your vins fusion sample).

15

I suppose there should be a default model and setting for the Equidistant model. I tried OpenCV’s cv::fisheye::undistortImage function but the output images are not same with the images recorded by a pinhole camera at same position with same lens setting.

Please, could you share the coefficients that you’ve used, when you’ve tried to match the Equidistant model with the OpenCV fisheye? If I’m not mistaken, fisheye model in OpenCV (Kannala-Brandt) should match Equidistant model in Isaac Sim, when you set :

camera.set_projection_mode("perspective")
camera.set_projection_type("fisheyeEquidistant")

on the Isaac Sim side and set K1…K4 to zero on the OpenCV side. This also assumes that the camera intrinsic matrix in OpenCV and Isaac Sim match (which can be verified by the camera.get_intrinsics_matrix call and matching pinhole images).

Please, can you confirm that these were the settings that you’ve used? Please, could you also share what are you trying to achieve? As it is possible that there are alternative solutions available.

16

Thank you @dchichkov ,

You explanation meet with my idea of comparing OpenCV to the simulation result and I’ll try get the content you mentioned.
Though I still have one problem in doing that: the quadruped sample is using UsdGeom(pxr) interface to create camera, I’m not really sure how to convert it into standard camera sensor where I can use the set_projection_mode interface. If you have any idea on that, I think it will speed up my verification. On the other hand I may have to develop the experiment base on another sample