Can not use articulations with gpu pipeline

Hello.

I am currently trying to use particle cloth and articulations at the same time.
However, when I turn on gpu pipeline with “backend=‘torch’, device=‘cuda’”, the program can not run due to an error.
This seems to be a problem on the articulations side.
For example, the following program also causes the error.

from omni.isaac.kit import SimulationApp

simulation_app = SimulationApp({"headless": False})

import numpy as np
import time
from omni.isaac.core import World
from omni.isaac.franka.controllers.pick_place_controller import PickPlaceController
from omni.isaac.franka.tasks import PickPlace

my_world = World(stage_units_in_meters=1.0, backend='torch', device='cuda')
# my_world = World(stage_units_in_meters=1.0)
my_task = PickPlace()
my_world.add_task(my_task)
my_world.reset()
task_params = my_task.get_params()
my_franka = my_world.scene.get_object(task_params["robot_name"]["value"])
my_controller = PickPlaceController(
    name="pick_place_controller", gripper=my_franka.gripper, robot_articulation=my_franka
)
articulation_controller = my_franka.get_articulation_controller()

i = 0
while simulation_app.is_running():
    my_world.step(render=True)
    if my_world.is_playing():
        if my_world.current_time_step_index == 0:
            my_world.reset()
            my_controller.reset()
        observations = my_world.get_observations()
        actions = my_controller.forward(
            picking_position=observations[task_params["cube_name"]["value"]]["position"],
            placing_position=observations[task_params["cube_name"]["value"]]["target_position"],
            current_joint_positions=observations[task_params["robot_name"]["value"]]["joint_positions"],
            end_effector_offset=np.array([0, 0.005, 0]),
        )
        if my_controller.is_done():
            print("done picking and placing")
            my_world.reset()
            my_controller.reset()
            time.sleep(1.0)
        
            
        articulation_controller.apply_action(actions)
simulation_app.close()

Is there a way to use articulations with gpu pipeline in standalone python, or is there another way to use particle cloth and articulations simultaneously in standalone python?

isaac sim version: 2023.1.1
Thanks in advance

1 Like

I have similar issues. When I set the world to use GPU and torch as backend, the articulation controller still tries to use numpy utils. In my case, I am using Franka, where I noticed LulaInterfaceHelper only import numpy utils package.

I added some dirty hacks in the numpy and torch utils to make the app run. However, the transformations of each joint is wrong and is not getting updated even if the robot is moving. This problem is not observed when using numpy backend. However, I need to use GPU for deformable body simulation.

Any feedback from the Isaac Sim team is greatly appreciated!