Running a HIL Scenario with Isaac Sim 4.0 and Isaac ROS

Isaac Sim Version

4.2.0
4.1.0
4.0.0
2023.1.1
2023.1.0-hotfix.1
Other (please specify):

Isaac Lab Version (if applicable)

1.2
1.1
1.0
Other (please specify):

Operating System

Ubuntu 22.04
Ubuntu 20.04
Windows 11
Windows 10
Other (please specify):

GPU Information

  • Model: GeForce RTX 4070 Ti 12GB
  • Driver Version: 550.100

Topic Description

Detailed Description

I want to create a HIL (Hardware in the Loop) scenario with Isaac Sim 4.0 and Isaac ROS just like the example in the following link: GitHub - NVIDIA-AI-IOT/isaac_demo: Set of demo to try Isaac ROS with Isaac SIM. Since this demo is based on Isaac Sim 2022.2.1 and JetPack 5.1.2, I can not use most of the functionality (such as the Carter V1 providing depth image while the Nova Carter doesn’t) on newer versions. I tried to adapt the code for the Isaac Sim 4.0 with the following changes but still get Fatal Python error: Segmentation fault errors when I run the Isaac Sim.

  • Changed SimulationApp initialization order as a workaround provided here: Isaac Sim - People Simulation broken in 4.1.0 - #11 by marcelo.jacinto
  • Changed carter_prim_path = '/World/Carter_ROS' to carter_prim_path = '/World/Nova_Carter_ROS'in order to access Nova Carter’s OmniGraph nodes.
  • Adapted node names when changing publication topics from:
rgb = controller.node(f'ros2_create_camera_{name}_rgb',
                                      f'{carter_prim_path}/ROS_Cameras')

to

rgb = controller.node(f'{name}_camera_publish_image',
                                      f'{carter_prim_path}/front_hawk')
  • Commenting out depth functionality completely causing the Fatal Python error: Segmentation fault error, etc.

What I really want to achieve here is just to be able to test some examples and work my way to implementing some custom ROS applications with existing environments and robots. I want to be able to send goal points from my Jetson AGX Orin’s rviz screen and see how my custom ROS2 nodes doing on the simulation.

I wanted to start writing my own OmniGraph node script in Python and start Isaac Sim with it using python.sh but I could not find tutorials on how to start doing that with Isaac Sim 4.0.0.

Would Introduction to Robotic Simulations in Isaac Sim self-paced course help me do that?

What are the first steps for creating/adapting such a scenario on newer Isaac Sim 4.0.0 and Isaac ROS versions?

I’m mostly interested in AMR workflows.

Thanks in advance!

Hey @dsonmez ! Have you checked out this tutorial Camera-based Perception with Isaac Perceptor in Isaac Sim?

It will run the nova_carter_bringup demo with Isaac Sim and Isaac ROS. In the scene, there is a nova carter robot equipped with cameras. When you launch the nova_carter_bringup demo, you can send goal poses to the robot for navigation.

Is it something you are looking for?

I think Introduction to Robotic Simulations in Isaac Sim would be helpful for you to understand Isaac Sim.
You can also check out our tutorials on Isaac ROS and Isaac Sim Omnigraph Python Scripting, which might help you as well.
Since you want to focus on AMR workflows, you can check out our perceptor reference workflow.

Hey @zhengwang, thank you for your detailed response. I have figured out how to do it and successfully deployed the ReMEmbR application using Isaac Sim (on an RTX workstation) and Isaac ROS (on a Jetson AGX Orin 64GB).

However, I only have one question right now, which I asked on this topic: Robot is leaving obstacle marks while mapping manually. I couldn’t get a response from Rafaelo yet. So, may I ask if you have any comments on this, or could you please let me know how I can get some help on this? Thanks!

1 Like

Sure! Let me take a look at the other post.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.