Hi,
I try to simulate a (real world) rgb cameras. In Isaac sim does the depth of field not work properly, nothing is sharp at all. I found out that depth of field only works with pinhole as projection type (which is fine) but the result is not what it should be.
I use the following parameters (which I have from a DoF calculator):
Here my settings of the camera:
The resulting image is then:
The distance is measured from the camera to the panel. This distance is below the near limit of the DOF and should therefore not be so sharp.
When I use depth of field override do I however get:
Which is also not how it should be. On the one hand now it is sharp but there is no depth of field.
I use
2023.1.1 (forced to at the moment)
but have also tried the same with 4.5 and the result was the same
Why is there this discrepancy between theory and simulation and also between the two ways of using the camera parameters?
Hi @yhofmann. Let me reach out to the internal team.
The discrepancy between the two controls is because of different units.
The camera controls accept focal length in tenths of scene units, while the override controls work in mm.
Although the parameters are in the pinhole section (inside camera settings), they work for all projections
Hi @zhengwang,
Thanks for reaching out to the team. However I must say that for the focal length part is there no difference in units. I measure in both the FOV to check what is is and if it changes with some parameters (e.g focal length). With this did I find out that also in the camera settings I need to use the focal length in mm to get the “correct” (undistorted ) FOV. Therefore not tenth of scene units as my scene uses default m as unit.
Using depth of field overrides (pinhole):
Only using the camera parameter (pinhole)
Regarding depth of field, from my understanding is the problem not the focal length (this needs to be in the camera in mm to give me the correct FOV). Please advise me how to properly get the depth of field modeled using the camera and the parameters I have for the real camera and which I have put into the calculator. Which values do I need to set where?
Regarding my point on the projection model, here I only switched to Equidistant or Equisolid and now the depth of field is gone. Same parameters as pinhole from above
Regarding the FOV and distortion parameters, as far as I understand it and saw here: CameraParams — Omniverse Extensions should I be able to tweak the radial distortion and with it the FOV by changing the s0 to s3 parameter of the camera
I tried doing so but nothing happens. Are the parameters active for the pinhole model? Using only the focal lenght do I get 71° but I know that the camera/lens has 80°, can you tell me which numbers I should put in (if the field is active) to see an effect on the FOV.
Thanks a lot
Maybe to also visualize that there seems to be something off with the projection types (use same focal length and aperture) would I expect that equidistant projection type would lead to a significantly higher FOV but it does not, see:
Using pinhole
Using equidistant
In both cases are the yellow bars at the same posiiton (just at the edge) and therefore the measured FOV is the same (roughly 71 degrees , which it should only be for the pinhole case:
Using the (inverse) formulas for the projection type would I expect it to be alsmost 82 degrees:
I can send you the scene if this is better for the internal team to have a look.
Maybe it is also more efficient and beneficial to all users if I could have a 30min call with someone from your side to go over the issues and see if this is a bug. I have now done quite some research and looked at e.g depth of field with other calculator/simulators and could after the meeting summarize the points to get a realistic camera properly to work with the correct depth of field, FOV and also the distortion parameters and post it here for all the other users.
Thanks
Hi @yhofmann thanks for your detailed description. It would be beneficial if you can share your scene and have a short video recordings to go over the issues. Then I can easily share with the internal team. Thanks!
Hi @zhengwang,
Sure, thanks. Find the scene here:
camera_scene.zip (263.7 KB)
The recording of the issue regarding the field of fiew which does not change with changing projection type. Yellow bars are at 1m distance to camera and outwards just at edge to measure angle (FOV) .
Video showing that changing s parameter does not change FOV
And for the issue with depth of field of the overrides and the different projection types
The yellow board is at distance 0.29m (which is below the 0.35m which should be the near limit for the depth of field and therefore be not so sharp) the post to the left of it even closer.
And here looking at boards at different distances to see that when setting the focus distance to that of the boards distance makes it sharp but the other ones in the distance not (they should be), here using pinhole and no override. Here I would expect all of them to be sharp with the 0.7m focus distance setting
Sorry that the videos do not have audio. If something is unclear let me know or feel free to setup a call with someone from the internal team to clarify the points.
Thanks
1 Like
Thanks @yhofmann! Let me pass this to the internal team.
For the second video: the parameter s only affects some fisheye projections. But you are in pinhole projection
For the third video: we do not support DoF in real time mode for those projections (fisheyeOrthographic, fisheyeEquidistant, etc). DoF support for non linear projections in RT was added only to fisheyePolynomial, and later on to several other models (the ones from OpenCV).
Regarding s, okey good to know. What about p? I assume the poly not as they are for fisheye polynomial? Do you have a link to some sort of documentation/source which explains what each parameter corresponds to?
Which DoF do you mean? The “normal” one directly from the camera or the one using the depth of field overides in the render settings? Is it then supported for pinhole correct? How can I then now to which models it was added and after which version?
Is there something more you can share regarding how to get the correct depth of field using the pinhole camera? E.g which parameters influence it and what are their units?
For the discussion regarding units:
I found this link talking about units Cameras — Omniverse Materials and Rendering. For which versions is this correct? There is a big confusion about the units here, because here focus distance is in scenes unit again (like also in the tooltip), whereas you mention tenth of a scene unit.
Also for the focal length does the tooltip say tenth, but in the link shown is it mm
And for aperture is it tenth in the tooltip and in the link tenth or mm, this is totally confusing,
Can you please clarify ones and for all what the units are, such that this is then the groundtruth on which others and I can rely on. Maybe it would be more efficient if I could look at this problem with the internal team in a call where we both can directly see the effect of the changes and tell each other what we observed and would expect. Feel free to send me an email if this is possible and I can then summarize what we discussed and how to properly set the parameters.
I have set the meters per unit to 1.0, this is probably what most people set it to.
For the camera parameters, we have this documentation for its supported projection type https://docs.omniverse.nvidia.com/materials-and-rendering/latest/cameras.html#rtx-camera-projection-attributes-deprecated
For DoF, the “normal” one directly from the camera and the one using the depth of field overides in the render settings are basically the same thing. The override one is just overriding the DoF for all cameras. Please refer to this documentation: Post-Processing — Omniverse Materials and Rendering
I agree that the unit is pretty confusing.
In Isaac Sim, the default world unit is meter. So tenths-of-world-unit is dm.
For the document you shared Cameras — Omniverse Materials and Rendering, it assumes the world unit is cm. So tenths-of-world-unit is mm. So when you use either camera property tab or the DoF override, you will need to do the unit conversion yourself.
We also provide camera API (doc) that you can set focal length, this API does the conversion to tenths-of-world-unit internally. If you check exts/isaacsim.sensors.camera/isaacsim/sensors/camera/camera.py to get more details.
I will get more information about how to get the correct depth of field using the pinhole camera
Thanks for the clarifications and the links, that clarifies it.
Regarding the projection types mentioned in https://docs.omniverse.nvidia.com/materials-and-rendering/latest/cameras.html#rtx-camera-projection-attributes-deprecated, Pinhole OpenCV cannot be selected and therefore only “normal” pinhole is available which according to the table in the documentation has none of the parameters supported. How can I enable the Pinhole OpenCV type. I know from Cameras lens distortion extension not visible - #5 by yhofmann that I’m forced to use the depreciated section (table) as I do not use 5.0 which would have the lens API .
Thanks for answering this and checking the DoF with the internal team.
Hi @yhofmann unfortunately, OpenCV models don’t exist in Isaac Sim 4.x versions.
But you can use python APIs to approximate the OpenCV model. Here is an example of using camera APIs: Camera Sensors — Isaac Sim Documentation
Here is the doc for APIs: [isaacsim.sensors.camera] Isaac Sim Camera Simulation — Isaac Sim
Hello!
We noticed that this topic hasn’t received any recent responses, so we are closing it for now to help keep the forum organized.
If you’re still experiencing this issue or have additional questions, please feel free to create a new topic with updated details. When doing so, we recommend mentioning or linking to this original topic in your new post—this helps provide context and makes it easier for others to assist you.
Thank you for being part of the NVIDIA Isaac Sim community.
Best regards,
The NVIDIA Isaac Sim Forum Team
