Can not use articulations with gpu pipeline

Hello.

I am currently trying to use particle cloth and articulations at the same time.
However, when I turn on gpu pipeline with “backend=‘torch’, device=‘cuda’”, the program can not run due to an error.
This seems to be a problem on the articulations side.
For example, the following program also causes the error.

from omni.isaac.kit import SimulationApp

simulation_app = SimulationApp({"headless": False})

import numpy as np
import time
from omni.isaac.core import World
from omni.isaac.franka.controllers.pick_place_controller import PickPlaceController
from omni.isaac.franka.tasks import PickPlace

my_world = World(stage_units_in_meters=1.0, backend='torch', device='cuda')
# my_world = World(stage_units_in_meters=1.0)
my_task = PickPlace()
my_world.add_task(my_task)
my_world.reset()
task_params = my_task.get_params()
my_franka = my_world.scene.get_object(task_params["robot_name"]["value"])
my_controller = PickPlaceController(
    name="pick_place_controller", gripper=my_franka.gripper, robot_articulation=my_franka
)
articulation_controller = my_franka.get_articulation_controller()

i = 0
while simulation_app.is_running():
    my_world.step(render=True)
    if my_world.is_playing():
        if my_world.current_time_step_index == 0:
            my_world.reset()
            my_controller.reset()
        observations = my_world.get_observations()
        actions = my_controller.forward(
            picking_position=observations[task_params["cube_name"]["value"]]["position"],
            placing_position=observations[task_params["cube_name"]["value"]]["target_position"],
            current_joint_positions=observations[task_params["robot_name"]["value"]]["joint_positions"],
            end_effector_offset=np.array([0, 0.005, 0]),
        )
        if my_controller.is_done():
            print("done picking and placing")
            my_world.reset()
            my_controller.reset()
            time.sleep(1.0)
        
            
        articulation_controller.apply_action(actions)
simulation_app.close()

Is there a way to use articulations with gpu pipeline in standalone python, or is there another way to use particle cloth and articulations simultaneously in standalone python?

isaac sim version: 2023.1.1
Thanks in advance

1 Like

I have similar issues. When I set the world to use GPU and torch as backend, the articulation controller still tries to use numpy utils. In my case, I am using Franka, where I noticed LulaInterfaceHelper only import numpy utils package.

I added some dirty hacks in the numpy and torch utils to make the app run. However, the transformations of each joint is wrong and is not getting updated even if the robot is moving. This problem is not observed when using numpy backend. However, I need to use GPU for deformable body simulation.

Any feedback from the Isaac Sim team is greatly appreciated!

Do you have an error log for us to investigate?

have you also tested in 4.0 to see if the error still persists?

I am having similar issues. I am using Isaac Sim 4.0.0 with cloth simulation (therefor I need to use GPU) and an articulation. I use a Cortex Franka robot, which is then wrapped into an articulation by Isaac Sim itself. When initializing, it tries to set the default joint state to a numpy array, which cannot be processed by the torch backend. Here is the error message:

Traceback (most recent call last):
  File "/home/ubuntu/.local/share/ov/pkg/isaac-sim-4.0.0/kit/python/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/ubuntu/.local/share/ov/pkg/isaac-sim-4.0.0/kit/python/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/ubuntu/.vscode/extensions/ms-python.debugpy-2024.10.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
    cli.main()
  File "/home/ubuntu/.vscode/extensions/ms-python.debugpy-2024.10.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "/home/ubuntu/.vscode/extensions/ms-python.debugpy-2024.10.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
    runpy.run_path(target, run_name="__main__")
  File "/home/ubuntu/.vscode/extensions/ms-python.debugpy-2024.10.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/home/ubuntu/.vscode/extensions/ms-python.debugpy-2024.10.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/home/ubuntu/.vscode/extensions/ms-python.debugpy-2024.10.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "/home/ubuntu/{location}/cortex_sync.py", line 22, in <module>
    main()
  File "/home/ubuntu/{Location}/cortex_sync.py", line 19, in main
    world.run(simulation_app)
  File "/home/ubuntu/.local/share/ov/pkg/isaac-sim-4.0.0/exts/omni.isaac.cortex/omni/isaac/cortex/cortex_world.py", line 338, in run
    self.reset()
  File "/home/ubuntu/.local/share/ov/pkg/isaac-sim-4.0.0/exts/omni.isaac.cortex/omni/isaac/cortex/cortex_world.py", line 280, in reset
    super().reset(soft)
  File "/home/ubuntu/.local/share/ov/pkg/isaac-sim-4.0.0/exts/omni.isaac.core/omni/isaac/core/world/world.py", line 391, in reset
    self._scene._finalize(self.physics_sim_view)
  File "/home/ubuntu/.local/share/ov/pkg/isaac-sim-4.0.0/exts/omni.isaac.core/omni/isaac/core/scenes/scene.py", line 365, in _finalize
    articulated_system.initialize(physics_sim_view)
  File "/home/ubuntu/.local/share/ov/pkg/isaac-sim-4.0.0/exts/omni.isaac.cortex/omni/isaac/cortex/robot.py", line 610, in initialize
    super().initialize(physics_sim_view)
  File "/home/ubuntu/.local/share/ov/pkg/isaac-sim-4.0.0/exts/omni.isaac.cortex/omni/isaac/cortex/robot.py", line 523, in initialize
    self.set_joints_default_state(positions=self.default_config)
  File "/home/ubuntu/.local/share/ov/pkg/isaac-sim-4.0.0/exts/omni.isaac.core/omni/isaac/core/articulations/articulation.py", line 757, in set_joints_default_state
    positions = self._backend_utils.expand_dims(positions, 0)
  File "/home/ubuntu/.local/share/ov/pkg/isaac-sim-4.0.0/exts/omni.isaac.core/omni/isaac/core/utils/torch/tensor.py", line 67, in expand_dims
    return torch.unsqueeze(data, axis)
TypeError: unsqueeze(): argument 'input' (position 1) must be Tensor, not numpy.ndarray

When using the numpy backend, the robot runs just fine, but I need to use torch and cuda for the soft body simulation.

Another, more standardized example: when I use the standalone example follow_example_mainfrom omni.isaac.cortex examples and just alter the line defining the world to world = CortexWorld(device="cuda", backend="torch"), I get the same error message starting at world.run(simulation_app).

Thanks for your help!

Please create a new topic linked to this one. Thanks.