How to prevent SDF object jittering in robot manipulation task

Omniverse Isaac Sim
,
1

Isaac Sim Version

4.5.0

Operating System

Ubuntu 22.04

GPU Information

  • Model: RTX3090
  • Driver Version: latest

Hi.

We are building a project to teleop robot arm in isaac sim to manipulate some objects.

Specifically we are trying to close an engine cap (Video attached). The cap and cap hole both have some threads and with SDF collision (cap also has rigid body and gravity disabled and the hole just has static collider), we have made a 1mm tolerance between the hole and the cap. We are tele-operating the arm to close the cap.


The jittering starts roughly at 37 seconds. And once we try to close the cap and thread has some mismatch, the cap starts to jitter. We would expect we won’t be able to put in the cap if there is a mismatched thread, rather than the cap goes in but colliders causing instability.

We have uploaded our collected USD file in the below google drive:
https://drive.google.com/drive/folders/1CYoKHHnxU0XLU-JU4lcqmksr0UhZ9ct1?usp=sharing

The above USD has saved all the physics settings and once loaded we start our tele-op ROS system.

Please provide us with some insights to solve this problem based on the attached video and USD file, and we appreciate your help.

If code is needed for debugging this we are also happy to share our code but they are mainly scripts for the ros bridge and robot articulation control. Free free to contact us through the email to the owner of the shared google drive folder.

Thank you

3

Hi @HaoyuMa, thanks for posting your issue. Very interesting setup! I was taking a look at your two assets and I didn’t see anything wrong with the SDF collision. In the video below you can see that I carefully tried to manually move the screw into the hole and screw it. It seems to be like it is working as it even gets pulled into the hole through the screw mechanism as I rotate in the Z axis. At the very end what happened is that I rotated passed the limit of the thread and both parts interpenetrated causing them to “jitter”.

I am controlling the pose of the screw by forcing it pose, which easily makes it interpenetrate the hole. The easiest way to avoid this interpenetration is to use lower forces on the screw when screwing. You can do that by having a robot controlled through low stiffness (or any controller as long as it can make sure contact forces are maintained low).

4

Thank you so much for your suggestions.

We did assign an enormously large k_p (stiffness) to the controller in the robot articulation (1e7 for gripper).

Now we have a much smoother touch of the gripper on the cap (as seen in 1:15).

However, as you see in the last part of the video, the maneuver made a mistake when teleoping the arm thus the arm rams into the cap a little bit. Then this causes the cap to lose its stability. Though the interaction between the gripper and the cap is nicer, but this instability is unacceptable for us (we are collecting data in Isaac to train an imitation learning model).

Below is our k_p and k_d (first 14 entries for arm and last 22 entries for gripper):

kp = np.concatenate((np.array([5e4] * 14), np.array([2e2] * 22)))
kd = np.concatenate((np.array([5e3] * 14), np.array([1e1] * 22)))

I wonder if this is the problem between the cap and cap hole collider? Or the stiffness is still too large? Is there some SDG collider setting we can tune based on the collider documentation?

Another thing we are also unsure and wants to hear from professional advice is: we will create a fixed joint between the gripper palm frame and the cap to mimic the grasping (this is not shown in this video) because we observe the cap handle is slippery when gripper tries to grasp. THe fixed joint thing is very hacky and we believe unreliable, but it’s the only way for us to get the correct visual, nevertheless, the issue of instability still persists (again for clarity, the above video has nothing to do with the fixed joint).

I am actively looking through physical materials to increase friction of the cap handle to solve this slippery problem. But we appreciate any helpful suggestions from the professional group to solve the physics instability problem.

We are grateful for any advice you can give, and we appreciate the valuableness of such powerful simulation tools you guys been developing.

Thank you.