How to set camera parameters in Isaac Sim by using the intrinsic parameters of a real camera

Omniverse Isaac Sim
,
1

Isaac Sim Version

5.0.0
4.5.0
4.2.0
4.1.0
4.0.0
4.5.0
2023.1.1
2023.1.0-hotfix.1
Other (please specify):

Operating System

Ubuntu 22.04
Ubuntu 20.04
Windows 11
Windows 10
Other (please specify):

Topic Description

Detailed Description

I want to get a tutorial or a function to set camera parameters by using the intrinsic parameters of a real camera.

Taking an example for RealSense camera, i got its intrinsic parameters like below:

height: 480
width: 640
distortion_model: "plumb_bob"
D: [0.0, 0.0, 0.0, 0.0, 0.0]
K: [615.5140380859375, 0.0, 308.024169921875, 0.0, 615.1778564453125, 242.978759765625, 0.0, 0.0, 1.0]
R: [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]
P: [615.5140380859375, 0.0, 308.024169921875, 0.0, 0.0, 615.1778564453125, 242.978759765625, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi: 
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: False

In case of creating a camera by script like Camera Sensors — Isaac Sim Documentation , it’s opencv fisheye or pinhole ? distortion_coefficents is directly D ? pixel_size, f_stop, focus_distance can be chosen by myself ?

In case of creating a camera by GUI, how to get the parameters that are needed to be set by the camera intrinsic parameters ? There is a mapping from distortion_model (like plumb_bob or rational_polynomial) to CamraProjectType ? There is a mapping from D to k0, k1, k2, p0, p1 etc ? The other parameters like focal_length are calculated like the functions shown in relevant script ?

Additional Content

OpenCv Fisheye script in Camera Sensors — Isaac Sim Documentation is shown below.

import isaacsim.core.utils.numpy.rotations as rot_utils
import numpy as np
from isaacsim.core.api import World
from isaacsim.core.api.objects import DynamicCuboid
from isaacsim.core.utils.stage import add_reference_to_stage
from isaacsim.sensors.camera import Camera
from isaacsim.storage.native import get_assets_root_path
from PIL import Image, ImageDraw

# Desired image resolution, camera intrinsics matrix, and distortion coefficients
width, height = 1920, 1200
camera_matrix = [[455.8, 0.0, 943.8], [0.0, 454.7, 602.3], [0.0, 0.0, 1.0]]
distortion_coefficients = [0.05, 0.01, -0.003, -0.0005]

# Camera sensor size and optical path parameters. These parameters are not the part of the
# OpenCV camera model, but they are nessesary to simulate the depth of field effect.
#
# Note: To disable the depth of field effect, set the f_stop to 0.0. This is useful for debugging.
# Set pixel size (microns)
pixel_size = 3
# Set f-number, the ratio of the lens focal length to the diameter of the entrance pupil (unitless)
f_stop = 1.8
# Set focus distance (meters) - chosen as distance from camera to cube
focus_distance = 1.5

# Add a ground plane to the scene
usd_path = get_assets_root_path() + "/Isaac/Environments/Grid/default_environment.usd"
add_reference_to_stage(usd_path=usd_path, prim_path="/ground_plane")

# Add some cubes and a Camera to the scene
cube_1 = DynamicCuboid(
    prim_path="/new_cube_1",
    name="cube_1",
    position=np.array([0, 0, 0.5]),
    scale=np.array([1.0, 1.0, 1.0]),
    size=1.0,
    color=np.array([255, 0, 0]),
)

cube_2 = DynamicCuboid(
    prim_path="/new_cube_2",
    name="cube_2",
    position=np.array([2, 0, 0.5]),
    scale=np.array([1.0, 1.0, 1.0]),
    size=1.0,
    color=np.array([0, 255, 0]),
)

cube_3 = DynamicCuboid(
    prim_path="/new_cube_3",
    name="cube_3",
    position=np.array([0, 4, 1]),
    scale=np.array([2.0, 2.0, 2.0]),
    size=1.0,
    color=np.array([0, 0, 255]),
)

camera = Camera(
    prim_path="/World/camera",
    position=np.array([0.0, 0.0, 2.0]),  # 1 meter away from the side of the cube
    frequency=30,
    resolution=(width, height),
    orientation=rot_utils.euler_angles_to_quats(np.array([0, 90, 0]), degrees=True),
)
camera.initialize()

# Calculate the focal length and aperture size from the camera matrix
((fx, _, cx), (_, fy, cy), (_, _, _)) = camera_matrix  # fx, fy are in pixels, cx, cy are in pixels
horizontal_aperture = pixel_size * width * 1e-6  # convert to meters
vertical_aperture = pixel_size * height * 1e-6  # convert to meters
focal_length_x = pixel_size * fx * 1e-6  # convert to meters
focal_length_y = pixel_size * fy * 1e-6  # convert to meters
focal_length = (focal_length_x + focal_length_y) / 2  # convert to meters

# Set the camera parameters, note the unit conversion between Isaac Sim sensor and Kit
camera.set_focal_length(focal_length)
camera.set_focus_distance(focus_distance)
camera.set_lens_aperture(f_stop)
camera.set_horizontal_aperture(horizontal_aperture)
camera.set_vertical_aperture(vertical_aperture)

camera.set_clipping_range(0.05, 1.0e5)

# Set the distortion coefficients
camera.set_opencv_fisheye_properties(cx=cx, cy=cy, fx=fx, fy=fy, fisheye=distortion_coefficients)
2

Hi @xinyang.li1,

Here’s how to map your RealSense CameraInfo parameters to Isaac Sim:

Your case (plumb_bob with D=[0,0,0,0,0]) — use opencvPinhole:

  import numpy as np                                                                                                                                                                                                                               
  from isaacsim.sensors.camera import Camera                                                                                                                                                                                                       
                                                                                                                                                                                                                                                   
  # Your RealSense intrinsics                                                                                                                                                                                                                      
  width, height = 640, 480                                                                                                                                                                                                                         
  fx, fy = 615.514, 615.178                                                                                                                                                                                                                        
  cx, cy = 308.024, 242.979                                                                                                                                                                                                                        
  D = [0.0, 0.0, 0.0]  # [k1, k2, p1, p2, k3]                                                                                                                                                                                                      
                                                                                                                                                                                                                                                   
  # Create camera with matching resolution                                                                                                                                                                                                         
  camera = Camera(prim_path="/World/camera", resolution=(width, height))                                                                                                                                                                           
  camera.initialize()                                                                                                                                                                                                                              
                                                                                                                                                                                                                                                   
  # Set physical parameters (pixel_size is your choice — it only affects                                                                                                                                                                           
  # focal_length and aperture in physical units, not the projection)                                                                                                                                                                               
  pixel_size = 3e-6  # meters (3 microns is typical)                                                                                                                                                                                               
  focal_length = pixel_size * (fx + fy) / 2                                                                                                                                                                                                        
  horizontal_aperture = pixel_size * width                                                                                                                                                                                                         
  vertical_aperture = pixel_size * height                                                                                                                                                                                                          
                                                                                                                                                                                                                                                   
  camera.set_focal_length(focal_le ngth)                                                                                                                                                                                                           
  camera.set_horizontal_aperture(h orizontal_aperture)                                                                                                                                                                                             
  camera.set_vertical_aperture(ver tical_aperture)                                                                                                                                                                                                 
                                                                                                                                                                                                                                                   
  # Apply OpenCV pinhole model with your intrinsics                                                                                                                                                                                                
  # D maps as: [k1, k2, p1, p2, k3] -> pinhole parameter list                                                                                                                                                                                      
  camera.set_opencv_pinhole_proper ties(                                                                                                                                                                                                           
      cx=cx, cy=cy, fx=fx, fy=fy,                                                                                                                                                                                                                  
      pinhole=D[:5]  # [k1, k2, p1, p2, k3] (all zeros = no distortion)                                                                                                                                                                            
  )                                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                   
  # Depth of field (your choice — set f_stop=0 to disable)                                                                                                                                                                                         
  camera.set_lens_aperture(0.0)  # f_stop=0 disables DoF                                                                                                                                                                                           
  camera.set_focus_distance(1.5)

Answering your specific questions:

  1. plumb_bob → opencvPinhole. The ROS plumb_bob distortion model is the standard OpenCV pinhole model (k1, k2, p1, p2, k3). Use set_opencv_pinhole_properties() .
  2. D → distortion coefficients. Yes, your D = [k1, k2, p1, p2, k3] maps directly to the pinhole parameter: camera.set_opencv_pinhole_prope rties(…, pinhole=[k1, k2, p1, p2, k3]). Since your D is all zeros, you effectively have no
    distortion.
  3. pixel_size, f_stop, focus_distance — these are physical sensor properties, not part of the pinhole projection model. You can choose them freely:
    • pixel_size: Determines the mapping between focal length in pixels vs physical units. Any consistent value works (3µm is typical for RealSense).
    • f_stop: Controls depth of field. Set to 0.0 to disable DoF blur entirely.
    • focus_distance: Only matters if DoF is enabled (f_stop > 0).
  4. rational_polynomial → also opencvPinhole with more coefficients: [k1, k2, p1, p2, k3, k4, k5, k6].

Key point: Once you call set_opencv_pinhole_properties(c x, cy, fx, fy, …), the projection is fully determined by those pixel-space intrinsics. The physical focal_length and aperture values are only used for DoF simulation — they don’t
affect the rendered projection when a distortion model is active.

Documentation reference: Camera Sensors (Camera Sensors — Isaac Sim Documentation)