Orbit: Surface gripper can't close on cuda device

Omniverse Isaac Sim
1

Hi all,

On my quest to use a surface gripper in a parallellized simulation, I’ve stumbled upon another weird issue: The gripper fails to close when using device="cuda:0" and closes when using device="cpu".

I’m using Isaac Sim 2022.2.0 and Orbit to spawn a bunch of UR10s with surface grippers attached to pick up objects. Simplified code is attached below, and the full standalone script is attached to this post.

Problem Description:

When I run with

SimulationContext(physics_dt=0.001, rendering_dt=0.01, backend="torch", device="cuda:0")

the gripper fails to close:

[omni.isaac.manipulators.grippers.surface_gripper] gripper didn't close successfully

This error seems to come from inside omni.isaac.surface_gripper._surface_gripper, as the close()-function I have access to in surface_gripper.py executes the way It’s supposed to.

When I run with

SimulationContext(physics_dt=0.001, rendering_dt=0.01, backend="torch", device="cpu")

everything works. The gripper closes, and the object is picked up.

Code:

My (very abbreviated) code is as follows:

from omni.isaac.manipulators.grippers.surface_gripper import SurfaceGripper

sim = SimulationContext(physics_dt=0.001, rendering_dt=0.01, backend="torch", device="cuda:0")

# Scene setup
kit_utils.create_ground_plane("/World/defaultGroundPlane", z_position=0.0)

robot_cfg = UR10_CFG
robot_cfg.data_info.enable_jacobian = True
robot_cfg.rigid_props.disable_gravity = True

robot = SingleArmManipulator(cfg=robot_cfg)
robot.spawn("/World/envs/env_0/Robot", translation=(0.0, 0.0, 0.0))

model_usd_path = "/home/*/Documents/Omniverse/Content/Models/My_USD/my_model.usd"
prim_utils.create_prim("/World/envs/env_0/Model_0", usd_path=model_usd_path, position=(0.485, 0.0, 0.184/2), orientation=(0.707, 0, 0.707, 0))

gripper = SurfaceGripper(
        end_effector_prim_path=f"/World/envs/env_0/Robot/ee_link", translate=0.1611, direction="x"
    )

robot.initialize("/World/envs/env_0/Robot")
gripper.initialize()

count = 0

# Simulate physics
while simulation_app.is_running():
    if count == 300:
        gripper.close()
    if count % 500 == 0:
        count = 0
        gripper.open()
    count += 1

Full script:

play_suction.py.zip (3.4 KB)

Btw, please ignore the very hacky way I’ve modified the Orbit IK example to pick up an object and follow a trajectory - this was only supposed to be a quick demo.

Anyway, any ideas what is going wrong? What are the differences between running surface_gripper on a gpu vs a cpu? Is it the thing I am trying to pick? Is there source code for omni.isaac.surface_gripper._surface_gripper available?

If I can provide more Information/code, I’m very happy to do so.

2

After some more investigating, it looks like the ee_link of the robots (and its colliders?) are not moving with the visuals of the arm (or rather, the position of the ee_link prim is not updated during simulation), as can be seen in the screenshots below. I’m pretty sure that is the reason for my issue - but how come this only happens on the gpu?

Screenshot from 2023-09-28 09-27-35
Screenshot from 2023-09-28 09-27-351107×795 294 KB

Screenshot from 2023-09-28 09-29-05
Screenshot from 2023-09-28 09-29-051107×795 300 KB

Edit: Or maybe not entirely - I guess the visualizations of the colliders are not updated because of Fabric, see (World position not updating during simulation run). However, if I do the following:

ee_0_prim = RigidPrimView(prim_paths_expr="/World/envs/env_0/Robot/ee_link")
print(ee_0_prim.get_world_poses())

the pose is also not updating only when using the gpu.

Edit 2: Never mind - i was missing ee_0_prim.initialize() in betwen those two lines. Still, could it be that the surface_gripper does not update its own position because it is using the USD API?

4

Yes, when using GPU pipeline, we don’t update data at USD-level. Instead it is communicated over Fabric: USD, Fabric, and USDRT — usdrt 7.3.3 documentation

Unfortunately, surface gripper is not yet supported on GPU.