YOLO, Openpose or custom Deep Learning Model Implementation in Omniverse Isaac Sim

Omniverse Isaac Sim
1

Hi,
I have a custom object detection deep learning model and a custom pose detection deep learning model. How can I implement the models in Isaac Sim? All I want is the camera feed from the robot and input that into the custom models. There is a lot of documentation about implementing deep reinforcement learning models but not other deep learning models. What should I do?

2

Hello,

This should be posted in the Omniverse section. I will move it over for you.

4

Would the Camera sensor help to feed its data to your model?

Here is an example snippet on accessing its data:

from omni.isaac.sensor import Camera
import omni.isaac.core.utils.numpy.rotations as rot_utils
import numpy as np

camera = Camera(
    prim_path="/World/camera",
    position=np.array([0.0, 0.0, 25.0]),
    frequency=20,
    resolution=(256, 256),
    orientation=rot_utils.euler_angles_to_quats(np.array([0, 90, 0]), degrees=True),
)

camera.initialize()

rgba_data = camera.get_rgba()

print(rgba_data)
1 Like