Trouble getting Camera data

[kalena.mccloskey]

Isaac Sim Version

2023.1.1

Operating System

Ubuntu 22.04

GPU Information

NVIDIA CUDA 12.2 Driver Version 535.183.01

Topic Description

Detailed Description

I am working on a simulation using Isaac Sim, ROS2, and (hopefully) Zed cameras. When I upload my Zed USD and initialize it as a Camera, everything runs smoothly and works as expected in the GUI. However, when I try to print .get_rgb() and .get_depth() to obtain data, I get and None, respectively.

Steps to Reproduce

In a class called ControllerRobot:

def setup_zed_camera(self):
    zed_prim_path = "/World/ZED_X"
    stage = omni.usd.get_context().get_stage()

    prims_utils.create_prim(
        prim_path="/World/ZED_X",
        prim_type="Xform",
        usd_path="(my_path_to)/ZED_X.usd"
    )

    left_camera_prim = stage.GetPrimAtPath(f"{zed_prim_path}/base_link/ZED_X/CameraLeft")
    right_camera_prim = stage.GetPrimAtPath(f"{zed_prim_path}/base_link/ZED_X/CameraRight")
    
    if not left_camera_prim.IsValid() or not right_camera_prim.IsValid():
        raise Exception("Could not find CameraLeft or CameraRight prims within the ZED_X model")
    
    orientation = euler_to_quaternion(0, 0, 0)
    orientation_array = np.array([orientation.GetReal(), orientation.GetImaginary()[0], orientation.GetImaginary()[1], orientation.GetImaginary()[2]])
    
    self.zed_camera_left = Camera(
        prim_path=str(left_camera_prim.GetPath()),
        name="zed_camera_left",
        position=np.array([0, 0, 1]),  
        orientation=orientation_array,
        frequency=20,
        resolution=(1280, 720)
    )
    self.zed_camera_left.initialize()
    self.zed_camera_left.add_distance_to_image_plane_to_frame()
    
    self.zed_camera_right = Camera(
        prim_path=str(right_camera_prim.GetPath()),
        name="zed_camera_right",
        position=np.array([0, 0, 1]), 
        orientation=orientation_array,
        frequency=20,
        resolution=(1280, 720)
    )
    self.zed_camera_right.initialize()
    self.zed_camera_right.add_distance_to_image_plane_to_frame()


def get_zed_data(self):
    if self.zed_camera_left is not None and self.zed_camera_right is not None:
        try:
            left_rgb = self.zed_camera_left.get_rgb()
            left_depth = self.zed_camera_left.get_depth()
            right_rgb = self.zed_camera_right.get_rgb()
            right_depth = self.zed_camera_right.get_depth()
            return left_rgb, left_depth, right_rgb, right_depth
        except Exception as e:
            print(e)
    return None, None, None, None

And then in main:

def main(args=None):
controller_robot = ControllerRobot()
time.sleep(5)

try:
    while simulation_app.is_running():
        simulation_app.update()
        controller_robot.update(1/60)
        rgb_data, depth_data, left_data, right_data = controller_robot.get_zed_data()
        print(rgb_data)
        print(depth_data)
        print(left_data)
        print(right_data)

etc.

Error Messages

No error messages, everything runs smoothly! I do get this warning though: WARNING: Annotator ‘distance_to_image_plane’ contains no data. Returning None

Additional Information

What I’ve Tried

I’ve tried using simpler cameras and printing their data, but I get and None from them as well. I’ve tried the suggestion from the forum below but also nothing.

Related Issues

Empty ndarray is returned from camera.get_rgb() on IsaacSim API

Hi, I had a similar issue using camera.get_rgb(). I solved it by waiting some frames before calling it. Try with the following: i = 0 while simulation_app.is_running(): my_world.step(render=True) if world.is_playing() and i > 50: rgb = camera.get_rgb() i += 1 It’s most surely not the best solution, but it worked in my case. Hope it helps

Additional Context

I haven’t been able to get collisions working with my robot and the background in the 2023.1.1 version, so I’m wondering if the cameras just aren’t “seeing” any obstacles?

[VickNV]

Please upgrade to the latest version, Isaac Sim 4.2.0, and refer to scripts located at ~/.local/share/ov/pkg/isaac-sim-4.2.0/standalone_examples/api/omni.isaac.sensor/.

[kalena.mccloskey]

Thank you for the suggestion!

I have switched to 4.2.0, but am still experiencing the same issues with the .get_rgb() and .get_depth() functions. I am trying to use this data for rtab mapping, so I am looking for topics like /zed2/left/image_rect_color, zed2/left/depth/depth_registered, and /zed2/left/camera_info, not the 2D and 3D points in the standalone examples. Are these the correct functions to use for this version, or are there other similar ones I could use to get the same information?

Thanks again for your help!

[VickNV]

I’m not sure if Isaac Sim Examples — isaac_ros_docs documentation will be a good reference for your use case.

BTW, please note that Isaac ROS was last validated with Isaac Sim 4.0.0

[kalena.mccloskey]

Hello,

Thank you for the idea! I have installed version 4.0.0 and Isaac ROS Common and am now able to see my cameras’ topics in the Isaac ROS Docker container, but I can tell by my print statements that I am still not publishing any camera data. I need help specifically finding a function that will return camera info, rgb data, and depth data.

Please let me know if you have any more ideas! Thanks in advance.

[VickNV]

Please check if ROS 2 Domain ID Collision is helpful.

[shakeelsindhu]

Hi @kalena.mccloskey

Have you been able to get camera data (sim data) out to use for further CV purposes?

[kalena.mccloskey]

Hi @shakeelsindhu,

Yes, I was able to connect the ZED SDK and thus see the published data directly through the ZED topics. Turns out I was way over complicating things by trying to obtain the data myself!

[VickNV]

Has this issue been resolved? Were you trying to obtain data from a physical ZED camera?

[kalena.mccloskey]

No just a sim ZED camera, I was able to connect the ZED SDK through action graphs to get the topics so yes the issue has been resolved! Thank you for your help!