Pick and place in Space/Zero gravity (0g)

Omniverse Isaac Sim
, ,
1

I’m working on a pick-and-place sim in Isaac sim 5.1 and ROS2 jazzy. I’m using an XARM7 w/ BioGripper and I’m trying to grab a small CubeSat-like object.

Right now my setup is:

  1. Gripper-mounted depth camera detects the CubeSat from point cloud data.

  2. MoveIt 2 plans the arm to a pre-grasp pose.

  3. The arm approaches the object really slowly.

  4. The BioGripper closes around the CubeSat.

  5. The arm tries to lift/place it.

The perception and approach are working. The gripper gets to the object and closes around it, but the object is kind of bouncy/slippery. When I lift, it sometimes slips out even though it looks like the fingers are around it.

I’m wondering what the best Isaac Sim approach is for this. Should I keep tuning friction/contact physics and gripper drive force, or is it better to use something like a rigid attach/detach once the gripper closes?

I was also looking at Isaac Sim’s surface gripper, Would that be a reasonable way to hold the object during lift, even though I’m using a two-finger gripper and not a suction cup?

Basically, I want the perception and arm approach to stay realistic, but I don’t necessarily need perfect contact physics yet. I just want the CubeSat to stay attached once the gripper has reached it and closed around it.

Has anyone done something similar? What would be the recommended way to handle this in Isaac Sim 5.1?

Thanks in advance.

3

Hi, for your use case, if you don’t care about contact physics much, friction grasping is rarely worth the tuning effort. What you can try:

  1. Use the BioGripper visually for perception, approach, and the closure motion (keeps your point-cloud pipeline realistic).
  2. Once you detect closure (gripper joint position past a threshold, plus contact reported on both fingers), programmatically create a FixedJoint between the gripper TCP and the cube.
  3. Lift / transport.
  4. At the place pose, delete the fixed joint and let the cube settle.

This gives you the perception realism you want, without fighting the physics.

4

I somehow managed to do something like that, I am able to pick the sat up using rigid joint, but when i place it, it fucks up the IK solver and makes the arm spazz/glitch out. So far I’m using an attach/detach script in script editor, would I have to make this an extension or can I have it as a ROS node?

I am confused because ROS nodes cannot directly influence USD files in Isaac SIm?

5

Hi, the two things you’re hitting now are both common — and they’re really the same root cause showing up twice.

Here’s why the arm “spazzes” on release:
When you delete the FixedJoint, several things change simultaneously in one physics step:

  1. The cube’s mass + inertia disappear from the gripper’s effective load. The robot’s joint controllers were silently compensating for ~Xkg of payload; now that mass is gone, the next PD step over-corrects.
  2. If the gripper fingers were in contact with (or slightly penetrating) the cube, PhysX resolves the penetration with an impulse the instant the constraint is gone — kicks both the cube and the fingers.
  3. In 0G there’s no gravity to damp the resulting transients, so the disturbance propagates up the arm and MoveIt’s controller chases it.

A few things that usually settle this down:

  • Open the fingers before deleting the joint, not after. Command the gripper to its open position, wait a few physics steps for the fingers to actually clear the cube, then drop the FixedJoint. This eliminates the penetration impulse, which is usually the biggest contributor.
  • Hold the arm still for a few steps after release. Send the current joint positions as the next target (essentially “freeze in place”) for ~10 physics steps before resuming MoveIt’s trajectory. This gives the controller time to re-equilibrate without the payload.
  • Pre-position the cube just clear of the fingers before detach. Once the arm has reached the place pose, briefly translate the cube ~1-2 mm along the gripper’s outward normal (still rigidly attached at this point), THEN open the fingers, THEN detach. This guarantees no contact at the moment of detach.
  • Lower the controller stiffness during release. If you’re using a high-gain MoveIt controller, momentarily reducing kp for the period spanning release-and-settle prevents the overshoot from amplifying.

Extension vs. ROS node — both work, here’s the split I’d recommend:

A small Kit-side script (or extension) that owns the attach/detach calls.

  • This is where UsdPhysicsFixedJoint.Define(…) / prim.GetReferences().RemovePrim(…) actually run — these APIs are only available inside the Kit process, so this has to live on the sim side regardless.
  • Exposes two ROS services: /grasp/attach and /grasp/release. Or a single /grasp/state topic that the script subscribes to.

Your ROS / MoveIt side stays unchanged.

  • Your MoveIt orchestration node just calls the service at the right moments (after grasp closure detected; before place trajectory).
  • It doesn’t touch USD or PhysX — it just sends a ROS message and trusts the Kit side to execute.

The omni.isaac.ros2_bridge extension is what makes this clean: you can register a ROS2 service inside your Kit extension/script and the callback fires inside Kit. There are samples in the Isaac Sim repo under standalone_examples/ showing the pattern.