Status of path planning with obstacle avoidance for robotic arms

Omniverse Isaac Sim
1

Dear

We are considering testing Isaac Sim for path planning with obstacle avoidance for a 7 axis robotic arm in a constrained environment. We would need some clarification reading our understanding of the current capabilities.

Our robot evolves inside a box that is is relatively small. It is so small that the robot cannot fit into it in its zero angular position. Our pick points can be everywhere in the box. As a consequence, the paths are very narrow. The robot often collide with the environment. Most of time, it is intermediate parts that collides. Also, we have self collisions between robot parts. We also often push the robot into its joints limits.

Currently, we perform collision detection with PyBullet. If the tool collides, we push it out of collision using the collision direction. Then, if one intermediate part of the robot collides, we recompute and inverse kinematics using a rest pose that push the robot out of collision. If the robot self collide we unfold it and if it collide with environment we fold it. We need to iterate to find a equilibrium between these opposite constraints.

This works, but this can take a while to converge, what is not compatible with the production constraints of time.

When we ask for direction we always have the same answer: try ROS and Moveit.

This is a big rewamp of the project because we don’t use ROS and everything is in Python, while Moveit 2 Python API for ROS does not seem to be mature enough for this need, what mean that we need to switch the planner to C++.

We have seen the Motion Generation page of Isaac documentation and RMPFlow. It seems to be easier to deploy than Moveit. Also, we are very interested into the realistic simulation of Isaac Sim because we work a lot with computer vision.

What is less clear for us is the degree of maturity of the path planning with collisions avoidance of RMPFlow for 7 axis robotic arms.

Our project is tile constrained. If we gain time by using a solution that is easier to code in Python but then we face limitations that create the need for lots of custom code, it is not a gain.

We have seen that RMPFlow includes an algorithm for obstacle avoidance. It mentions the collisions between the robot parts and the environment, but not the self collisions between the different robot parts. Is it included?

One approach would be to ask RMPFlow to generate a trajectory that avoids obstacles and then, if it fails, to rollback to our custom algorithm, replacing PyBullet obstacle detection by the one of Isaac Sim.

For that we need a detailed report on collisions between robot parts and the environments and between robot parts themselves. This report has to include the extreme points of penetration, so that we can compute a vector to find the direction that gets us out of collision.

Is it something that Isaac Sim can provide?

Thanks a lot for the clarifications!

Ho, and some less important side questions…

It may also be interesting to test some OMPL algorithms. We can do that using the Python API of OMPL. Unless there are already some integrations?

Also, our robot has strange joints angular limits. Some limits are not fixed. They depend upon a linear combination of the axis angles. Currently we cannot manage this with PyBullet. The only solution we found is to reject inverse kinematics out of limits after calculus and to push back into the limits with a rest pose that unfold the robot. Unless there are better options with Isaac?

4

Hi thibd,

We don’t (yet) have a perfect solution for your problem, but Isaac Sim does contain a few building blocks that might be useful.

As you mention, the motion_generation extension exposes an implementation of RMPflow intended for fast, collision-aware reactive control. It has a couple limitations:

  1. It assumes that obstacles in the world are represented by geometric primitives, rather than arbitrary meshes. Supported primitives are cuboids (i.e., rectangular prisms), spheres, and capsules (incorrectly called “cylinders” if you dig deep enough in the code).
  2. It does not avoid self-collisions, except with a single, optional “body cylinder” (actually a capsule) that’s specified in the RMPflow configuration file and usually placed at the base of the robot. Since RMPflow does respect joint limits, we’ve found that this is sufficient to avoid self-collisions for common use cases, but your setup sounds more challenging.

With some careful tuning, it may be possible to use RMPflow as your primary “planner,” using your current scheme as a fallback as you suggest. For collision detection, I believe you should be able to use the “overlap” query type in the PhysX scene query API: Geometry Queries — NVIDIA PhysX SDK 4.1 Documentation

See this older forum post for some tips on accessing the scene query API from python: Simple Way to check for collision

In the next release of Isaac Sim (likely to be available sometime in August), the motion_generation extension will contain a PathPlanner interface (alongside the current MotionPolicy interface) with an example implementation exposing an RRT planner. Like our RMPflow implementation, this planner respects joint limits, but it unfortunately does not avoid self-collisions.

I’m not aware of any work integrating OMPL into Isaac Sim (except indirectly via MoveIt), but the forthcoming PathPlanner interface would probably be the right way to do that.

Also, our robot has strange joints angular limits. Some limits are not fixed. They depend upon a linear combination of the axis angles.

Our current motion generation tools aren’t equipped to handle such limits, but we are planning to eventually add support for “mimic joints,” which sound closely related.

1 Like