Help diagnosing why Prismatic Joints will not fully close on Robot IQ Hand E?

[mattmazzola]

Isaac Sim Version

4.2.0
4.1.0
4.0.0
2023.1.1
2023.1.0-hotfix.1
Other (please specify):

Isaac Lab Version (if applicable)

1.2
1.1
1.0
Other (please specify):

Operating System

Ubuntu 22.04
Ubuntu 20.04
Windows 11
Windows 10
Other (please specify):

GPU Information

• Model: GeForce RTX 4090
• Driver Version: 560.94

Topic Description

Detailed Description

What you were trying to do?

Configure the prismatic joints on RobitIQ Hand E so that when controlled by the ParallelGripper class they can be full closed. The joint reaches the fully open position, but not the fully closed positions.

What you expected to happen?

When I manually set the target position of Joint Drive API to some large negative value, the joint should move all the way to the lower limit.

What actually happened?

When I set the gripper to closed state, the joints only move to the 0 position, but not to the negative positions.
Perhaps there is some limitation the code that doesn’t allow negative values?

Steps to Reproduce

ur3e_robotiq_hand_assembled_single.zip (3.0 KB)

1. Load USD file
2. Attempt to close joint using the Drive API

Error Messages

N/A

Screenshots or Videos

Video

RobotIQ Hand E - Prismatic Joint configuration problem. Gripper will not fully close

Additional Information

What I’ve Tried

I have attempted to change almost every value available on the Slider_1 and Slider_2 prismatic joints to allow them to close.

Most notable are

• the axis of movement
• the upper and lower limits (This should be the primary limiting factor)
• the dampening and stiffness-
• the force of Drive
• the collision approximations to be convex decomposition on meshes

Related Issues

(If you’re aware of any related issues or forum posts, please link them here)

Additional Context

N/A

[mattmazzola]

Here is another picture viewing the limits of the prismatic joints

I can manually position the primitives to the desired location and see that the lower limit allows that position, but it is still confusing why it does not work.

How to get these blue indicators to reliably be displayed?

Even when “Joints” are enabled in the viewport and the Joint prims are selected I sometimes see small blue/white lines for the joints instead of the large axis with squares on each end.

I am not sure what actions I can take to make these visible. I thought it may have to do with some of the joints having invalid values (See the small red circle around one of the joints in center); however, even when there are no red circles, the large blue indicators cannot be enabled.

[Sirens_uk]

Is this part of the Isaac sim base assets, or is it something you have imported yourself?

One thing I did notice is that it has a rack and pinon system, you can see in yellow at the back with the gears. Is that connected to a drive system? I was wondering if you have a rotor that has a hard stop? Does the physical robot only have that length of stroke which would seem off.

Although if you look at the robot setup you can see two screw holes which to me would be the holes of for the allen heads. Say you wanted to make the jaws smaller to pick up a smaller object or you could pyshically move the jaws to its current position for a larger object.

Secondly if its a model you have imported yourself you might need to rescale the stroke length, I would have thought what you tried with the negative value would have worked.

The more I look at it i think its more to do with the allen heads as the travel distance looks correct. also look at the stroke length of the rack and pinon. I think this is a real world problem and not a software problem. There might be another model with the clams moved to the second set of screw holes…

hope this helps

[Sirens_uk]

Yeah checked the manual might be that.

To be honest i think its just the stroke of the rack, I think you would need to pyshically move the grippers to the next set of boltholes or change the grippers.

[mattmazzola]

Is this part of the Isaac sim base assets, or is it something you have imported yourself?

I was using the Isaac Sim asset Robotiq_Hand_E_base

it has a rack and pinon system, you can see in yellow at the back with the gears

On the real robot, it may be using that Drive mechanism; however, the “yellow at the back” you were referring to are only the screws used to mount the gripper finger. You can see the prims in the scene tree and these are the only meshes. There are no other mechanisms such as rotating gear (rack / pinion) to open and close the grippers.

See from this angle:

Also, the model I am using shows a single mesh which includes Rack and Fingers mounted as single prim.
If the fingers were adjustable, I would expect them to be separate meshes / prims so I could move them vias some FixedJoint offset

Is that connected to a drive system?

No, only the prismatic joints “Slider_1” and “Slider_2” have drives

I was wondering if you have a rotor that has a hard stop?

In the real it may, but in the sim it does not

Does the physical robot only have that length of stroke?

I am not sure

you might need to rescale the stroke length

I am not sure what “rescale the stroke length” means.
I think adjusting the lower and upper limit of the prismatic joint implicitly changes the possible length.

I think you would need to physically move the grippers to the next set of bolt holes or change the grippers.

For the real robot I think you may be correct; however, for the Sim robot while there are bolts, holes, to mimic appearance of the actual hardware, I do not think there is a limitation.
I still expected the prismatic joint should move the prim based on the boundaries set.

Image of Real

Notice it has similar two empty holes, implying same finger mount configuration as the sim model.
I will ask others on the team to see if they have tested the real robots grip limits and ranges.

There might be another model with the clams moved to the second set of screw holes

Oh, yes, I am unsure how to modify the current sim mesh, and would be interested in testing the alternate models if I can find one.

References

• Adaptive Grippers | Robotiq
• Hand-E Instruction Manual
  • See: 3.4.2. Installing Fingers on the Gripper
• Support | Robotiq
  • It looks like they have models to download but not with fingers at different mount points
  • Hand E > Universal Robots > 3D Models

Thank you @Sirens_uk or the thorough information! 🙏

[mattmazzola]

I was able to get more information about the REAL gripper to confirm my suspicions.

As you can see below, even with the fingers mounted at the default outer locations, the grippers can still fully close (Zero distance between fingers). This means the mount of the fingers is not limiting the range and the simulated gripper should be able to also fully close.
I think this confirms there is still something wrong with prismatic joint modeling and configuration.

The pictures are a little blurry with low contrast so I tried to add red lines to make it easier to see.

Robot IQ Support

I sent message to the Robot IQ support asking about alternate USD file which may have fixes for the joints or fingers and they responded

I can assure you [the Robotiq Gripper model USD] wasn’t from us. Nvidia may have made its own Robotiq gripper from a 3D CAD file taken from our support website, but we do not support it

Hopefully a moderator from NVidia can help by clarifying how this Isaac Sim asset was created and if they would offer assistance with fixes to joints

If I remember I also had to manually add Drive API and Mimic API to the joints which should come by default in fully usable asset. Perhaps it has something to do with the multiple versions of the asset: base, config, edit. As I understand config, and edit are layers to add onto the base but I am not familiar with this process.

Other Work:

• I will work to adjust the upper limits of joints, so the maximum open distance of SIM matches the REAL gripper.

[mattmazzola]

I did some more testing on the RobotIQ gripper and found another problem which may be related.

I use a very large cube as the object intended to be gripped so that even though the fingers do not close all the way it should still grasp the object.

You can see that I position the fingers around the cube and “close” it although the fingers appear stuck. It is like they have already hit the walls of the cube. However, they haven’t because I can lift the gripper up and there is no movement or friction on the block and as the gripper fingers clear the boundaries they being to finish closing.

The reason I said this MAY be related is that perhaps there is some collision approximation issue. The same thing that is making the grippers seems like they have reached their limits when around the cube may be the same issue that is making them think they hit their limit when closing with nothing in between.

However, I have checked the collision boundaries and it seems ok

[Sirens_uk]

Think i figured it out, I was playing around with removing the mimic and I could bypass the limit.

So I then opened the USDA the hand is being pulled from the AWS, which holds the hand-e base file with the limits inside it. So its actually referencing the AWS version and not the modified version that you are trying to alter the limits.

You would need to use your own instance of the hand-e USD and not the AWS one. You should be able to set your limits inside the initial USD and not the arm and hand USD I think thats why you’ve ran into this problem. I’ve not got time to play about with it anymore tonight.

Seems like you have an inheritance problem, I couldnt understand why the limits werent updating even though visually you can see the debug limits changing for the prismatic joint. Once I looked at both USDA’s it made sense…

Hope this solves your issue.

[mattmazzola]

Thank you for helping investigate this. I didn’t get a fully working solution but looked into your suggestions and want to provide more clarity, confirmation, and suggest future directions for diagnosis.

opened the USDA the hand is being pulled from the AWS

To be clear, the .usda, is the custom file I created. The file on AWS is a .usd

Yes, because I used the Isaac Assets, both the UR3E and the RobotIQ hand are payloads

referencing the AWS version and not the modified version that you are trying to alter the limits

Import to note that these are “Payloads” which behave differently than “References”
See the docs

Payloads are references that have all of their data actively loaded by the sim so that it can be modified at runtime

I interpreted this to mean payloads are explicitly intended to be modified. In other words, the updates to attributes I apply in Isaac Sim SHOULD override the values from the base USD on AWS

limits werent updating even though visually you can see the debug limits changing for the prismatic joint

You were making the claim that the updates I made such as modifying lower limits to the Prismatic Joint were not being applied and it was still using the defaults in the .usd on AWS. Because the defaults were 0 instead of the -0.025 that was preventing the fingers of the gripper from fully closing.

I think the modifications are being applied. One example is that the initial file/asset did not have Drive API or Mimic APIs added, but yet they are functioning in the simulation. And as you saw the Joint Limits visualizer which to change and show values.

This implies there is ANOTHER reason that is preventing the fingers from closing which I will show more below.

If modifications were NOT being applied despite being a Payload, I think this would imply an Isaac Sim bug which given how common this feature is I suspect is unlikely (Although given the other issues we’ve found in Isaac Sim, perhaps not as much as I would like 😅)

Also, I wondered how you are inspecting the values in .usd on AWS given those are binary and not human readable?

I was able to open the file from URL

(Side note: Nvidia’s default theme has such poor contrast. Definitely not passing accessibility testing 😅)

Confirmation of Joints Moving through Full Range of Lower / Upper Limits of Prismatic

with removing the mimic and I could bypass the limit

I can confirm that after removing the mimic joint, which allows each joint to move freely, and I can move each joint into the desired fully closed position.

This is great! 🥳
However, I am still not able to move BOTH joints to the fully closed position. 😔

You can see in this video how there is still some sort of collision happening between the fingers. Notice when one finger is fully closed and I attempt to close the other, it PUSHES the other joint back! This means when the Mimic Joint is applied and we attempt to close the gripper the fingers are colliding which gives the impression that they are hitting the limit of the prismatic joint; however, the seem to be colliding with each other!

This issue shown here, also aligns with the above post where the fingers appear to collide with the green cube even though they are not close to touching there is some

Given that I have inspected the collision, as shown in the post above and shown again here, I am still very confused what could be causing this.

I also checked the “Contact Distance” and both are 0

USDA to Replicate or Test Above

robotiq_hand_e_modified.zip (3.0 KB)

Summary

• The Payloads may not be the issue
• The Joints can move through full range of prismatic limits
• The collision approximations on fingers look correct
• There is still some unknown force, property, or collision preventing the fingers from getting close to other objects which includes each other.

[mattmazzola]

I made some more progress which may give others clues to help.

I was playing around with disabling collisions and found this other setting called “Solve Contact”. If you disable/uncheck this for either the right_gripper or left_gripper the fingers are able to close!
However, as expected this has negative effect since grippers are intended to contact items.

Rigid Body > Advanced > Solve Contact

Gripper fully closing

Objects passing through Finger with solve contact disabled

[mattmazzola]

A teammate alerted me to an Isaac Sim rendering bug as seen in this forum post where, in the case of collider approximation fallback, the visualized bright green lines may not represent the actual collider approximation used.

In other words, if before simulation start, a kinetic body had an approximation method assigned that was invalid for kinetic bodies, when simulation is started it will fallback to Convex Hull.

The fingers of our gripper are already configured to use Convex Decomposition which should be valid and NOT cause the fallback. However, when testing the gripper, the issues of:

• apparent min joint limits
• appearing to collide with object before the surface
• pushing objects out of gripper

Would all be explained by this fallback to Convex Hull occurring

Collision Approx with Convex Hull - Open

Collision Approx with Convex Hull - Closed

Documentation

omni.isaac.core.prims.GeometryPrim.get_collision_approximation

Questions

• Why would the collider fallback be occurring if we are using a valid approximation method?
• Is there a way to prevent this fallback?
• Is there a way to better debug or diagnose colliders that simply enabling the visual setting?

[mattmazzola]

After, investigating more, I think I found a solution.

Confirmation of fallback

In the logs I can see:

2024-11-25 19:51:41  [Warning] [omni.physx.plugin] PhysicsUSD: Parse collision - triangle mesh collision (approximation None/MeshSimplification) cannot be a part of a dynamic body, falling back to convexHull approximation: /World/robot/Robotiq_Hand_E_base/left_gripper. For dynamic collision please use approximations: convex hull, convex decomposition, box, sphere or SDF approximation.
2024-11-25 19:51:41  [Warning] [omni.physx.plugin] PhysicsUSD: Parse collision - triangle mesh collision (approximation None/MeshSimplification) cannot be a part of a dynamic body, falling back to convexHull approximation: /World/robot/Robotiq_Hand_E_base/left_gripper. For dynamic collision please use approximations: convex hull, convex decomposition, box, sphere or SDF approximation.
2024-11-25 19:51:41  [Warning] [omni.physx.plugin] PhysicsUSD: Parse collision - triangle mesh collision (approximation None/MeshSimplification) cannot be a part of a dynamic body, falling back to convexHull approximation: /World/robot/Robotiq_Hand_E_base/left_gripper. For dynamic collision please use approximations: convex hull, convex decomposition, box, sphere or SDF approximation.

Why fallback?

Why would the collider fallback be occurring if we are using a valid approximation method?

Above, I was under the assumption that the fallback would only occur if you had selected an approximation method that was not allowed for dynamic bodies; however, the logs ALSO say “None” will fallback!

From the logs, the prim listed is the left_gripper not the child mesh D_A03_ASM_DOIGTS_PARALLELES_1ROBOTIQ_HAND_E_DEFEATURE_02 which has the Convex Decomp

There are prims which have the Collider API but no approximation method set. This is causing the fallback to be applied to ALL children!

Solution

Removing Collider API which does not have approximation set from prims which which are not meshes such as base_link, left_gripper, and right_gripper. This prevents the fallback, and then the child prims use of Convex Decomp is preserved and functions as expected.

Video

There is still slight gap when picking up the larger cube, but at least it works!

Follow Up

• It was strange that Isaac Sim was printing the warning for left_gripper 3 times. (I assumed one message for each child) It would help if the warning message included the names of these children prims in the log message which would make the repetition more understandable.
• Isaac Sim rendering bug should be fixed so it shows the correct collision approximation. This would make it clear why the behavior of appearing to collide without visually colliding is occurring
• Update the documentation to explicitly mention this fallback behavior and how Colliders with None or no approximation specified will also cause fallback.

[mattmazzola]

@VickNV I noticed your name on some of my other posts. I realize it may not be appropriate to mention you directly, but given the Isaac Sim bug contributed to much of the confusion with this issue since it misleads you to think collision boundaries are different than what they actually are, I wanted to make sure you saw this and if necessary, help forward to the appropriate people internally to get it addressed in a future Isaac Sim release.

I think even if the issue cannot be fully fixed by rendering the actual collision approximation after it was dynamically changed by the fallback, there may be less complex partial fixes which at least inform the developer that the rendered collision wireframe is not up to date with some visual affordances other than the log message. Perhaps the color of the wireframe changes, or there is some other red indicator similar to invalid joints.

[AlesBorovicka]

Sorry for the trouble, we are heading into moving most of the validation steps into the validation extension, so these issues should be visible once the validation runs.
The debug visualization displays whats created in PhysX its not really there for validation, currently there is the warning issued, agree that is not enough.
Let me move forward the work on the validation and add it to the validation framework.

[Sirens_uk]

I’m glad you got this solved! that took quite a bit of figuring out…