Inconsistent Trajectory Following in Isaac Sim with MoveIt Using topic_based_ros2_control

Autonomous Machines Robotics - Isaac Isaac Gym
, , ,
1

Hi everyone,

I’ve been following a tutorial video How to Use MoveIt with Isaac Sim: A Step-by-Step Guide on using a UR robotic arm with MoveIt in Isaac Sim.

This tutorial uses the topic_based_ros2_control method, as described in the official MoveIt documentation here: How To Command Simulated Isaac Robot.

However, during testing, I’ve noticed that the robot in Isaac Sim sometimes doesn’t quite stick to the path that MoveIt planned—it sometimes drifts off the track or even ends up stopping at the wrong goal pose.

Has anyone experienced similar issues with topic_based_ros2_control, or are there any recommended steps to improve synchronization between MoveIt and Isaac Sim?

Any insights would be greatly appreciated!

Thank you!

Isaac Sim Version

4.2.0
4.1.0
4.0.0
2023.1.1
2023.1.0-hotfix.1
Other (please specify):

Operating System

Ubuntu 22.04
Ubuntu 20.04
Windows 11
Windows 10
Other (please specify):

GPU Information

  • Model: A6000
  • Driver Version: 545.23.08

Steps to Reproduce

This video tutorial: How to Use MoveIt with Isaac Sim: A Step-by-Step Guide (youtube.com) provides the necessary USD files, ROS 2 package files, and step-by-step instructions, making it very easy to reproduce.

Issues

The robot in Isaac Sim works well mostly, but sometimes it doesn’t consistently follow the planned trajectory from MoveIt.

This can be observed in

  • In the tutorial video (timestamp: 15 seconds, link to video). For a target pose with a left rotation, the Isaac robot’s end effector initially rotates significantly to the right, then snaps back quickly, which doesn’t match the smooth trajectory shown in RViz.

  • In my own tests (recorded here: link to my video), I planned four random valid poses:

    • Test 1: Trajectory matched well.
    • Test 2: The trajectory in Isaac Sim didn’t align with RViz.
    • Test 3: Successful.
    • Test 4: The robot stopped at an incorrect target position.

Screenshots or Videos

Here is a short gif showing the issue: For a target pose with a left rotation, the end effector in Isaac Sim first rotates a lot to the right before quickly snapping back. This movement doesn’t match the trajectory shown in RViz, which only left-rotated
isaac robot doesnt follow issue

2

@daohui.liu I am facing the same issue…The robot arm in Isaac sim doesn’t reach the desired goal configuration sometimes and will just hang at some other random joint configuration, while the arm in moveit reaches the desired goal joint configuration. I have had this issues for a couple of months now, and haven’t found a solution. Any suggestion around this problem is much appreciated. My suspicion is that the either the joint velocity, joint stiffness or joint damping values are not matching between the robot arm model that moveit is using and the one Isaac is using.

3

@daohui.liu Quick update: I believe the issue was caused by a mismatch in the joint limits between the urdf.xacro file used by MoveIt and the USD file used by Isaac Sim for the arm. I still need to test this change further, but I’ll provide an update here if it turns out that this fix doesn’t fully resolve the problem.

1 Like
4

Thanks for your comment and insights! I think the issue might actually be because the plans generated by MoveIt (when using the topic_based_ros2_control method) only include position data and not velocity or acceleration. I did some testing with Isaac Sim’s built-in motion generation methods like RMPflow and Lula RRT. Those plans include both position and velocity, and I haven’t seen the robot deviate from the planned trajectory.

However, Lula RRT sometimes causes slight vibrations, and RMPflow usually has some end-effector positioning error. Recently, I tried using NVIDIA’s cuRobo library for controlling motion in Isaac Sim. It works quite well overall, but there are still a few bugs. For now, though, it’s good enough for what I need.

Looking at MoveIt’s generated plans, they actually include position, velocity, and acceleration. But the topic_based_ros2_control setup from the tutorial only publishes joint positions. My guess is that if we skip the topic_based_ros2_control method, directly take MoveIt’s trajectory (with velocity info), and send it to Isaac Sim, and then use the articulation API to control the robot in Isaac Sim, it might solve this issue. I haven’t had time to try this yet, but if I do and it works, I’ll update here. Hope this helps!

5

@daohui.liu thank you for the reply. I also suspected that just sending positions in /joint_values topic from moveit to isaac sim is not the cleanest way to move robot in isaac sim. I also have to try out the articulation API to see if there any difference in the motions. I am eager to hear about your update and will do same from my side.