Failed to run tutorial example

Hi, I am using Isaac Sim, 2022.1.1. I tried to run the tutorial example but I found that it does not work properly. Also the size of the cube is not the same as what we see in the tutorial video.


Please let me know what I am doing wrong.
Thanks!

Hi Leila,

Thanks for reporting this issue. The problem here is the default size of the cube which will be fixed in the next release and the default gripper position. This can be solved by specifying the size of the cube to be 1 as well as explicitly setting the gripper positions to be opened on reset and on load.

Here is the modified script:

    from omni.isaac.examples.base_sample import BaseSample
    from omni.isaac.franka import Franka
    from omni.isaac.core.objects import DynamicCuboid
    from omni.isaac.franka.controllers import PickPlaceController
    from omni.isaac.core.tasks import BaseTask
    import numpy as np
    
    
    class FrankaPlaying(BaseTask):
        # NOTE: we only cover here a subset of the task functions that are available,
        # checkout the base class for all the available functions to override.
        # ex: calculate_metrics, is_done..etc.
        def __init__(self, name):
            super().__init__(name=name, offset=None)
            self._goal_position = np.array([-0.3, -0.3, 0.0515 / 2.0])
            self._task_achieved = False
            return
    
        # Here we setup all the assets that we care about in this task.
        def set_up_scene(self, scene):
            super().set_up_scene(scene)
            scene.add_default_ground_plane()
            self._cube = scene.add(
                DynamicCuboid(
                    prim_path="/World/random_cube",
                    name="fancy_cube",
                    position=np.array([0.3, 0.3, 0.3]),
                    scale=np.array([0.0515, 0.0515, 0.0515]),
                    color=np.array([0, 0, 1.0]),
                    size=1 #TODO: change here
                )
            )
            self._franka = scene.add(Franka(prim_path="/World/Fancy_Franka", name="fancy_franka"))
            return
    
        # Information exposed to solve the task is returned from the task through get_observations
        def get_observations(self):
            cube_position, _ = self._cube.get_world_pose()
            current_joint_positions = self._franka.get_joint_positions()
            observations = {
                self._franka.name: {"joint_positions": current_joint_positions},
                self._cube.name: {"position": cube_position, "goal_position": self._goal_position},
            }
            return observations
    
        # Called before each physics step,
        # for instance we can check here if the task was accomplished by
        # changing the color of the cube once its accomplished
        def pre_step(self, control_index, simulation_time):
            cube_position, _ = self._cube.get_world_pose()
            if not self._task_achieved and np.mean(np.abs(self._goal_position - cube_position)) < 0.02:
                # Visual Materials are applied by default to the cube
                # in this case the cube has a visual material of type
                # PreviewSurface, we can set its color once the target is reached.
                self._cube.get_applied_visual_material().set_color(color=np.array([0, 1.0, 0]))
                self._task_achieved = True
            return
    
    
    class HelloWorld(BaseSample):
        def __init__(self) -> None:
            super().__init__()
            return
    
        def setup_scene(self):
            world = self.get_world()
            # We add the task to the world here
            world.add_task(FrankaPlaying(name="my_first_task"))
            return
    
        async def setup_post_load(self):
            self._world = self.get_world()
            # The world already called the setup_scene from the task (with first reset of the world)
            # so we can retrieve the task objects
            self._franka = self._world.scene.get_object("fancy_franka")
            self._controller = PickPlaceController(
                name="pick_place_controller", gripper=self._franka.gripper, robot_articulation=self._franka
            )
            self._world.add_physics_callback("sim_step", callback_fn=self.physics_step)]
            #TODO: change here
            self._franka.gripper.set_joint_positions(self._franka.gripper.joint_opened_positions)
            await self._world.play_async()
            return
    
        async def setup_post_reset(self):
            self._controller.reset()
            await self._world.play_async()
            #TODO: change here
            self._franka.gripper.set_joint_positions(self._franka.gripper.joint_opened_positions)
            return
    
        def physics_step(self, step_size):
            # Gets all the tasks observations
            current_observations = self._world.get_observations()
            actions = self._controller.forward(
                picking_position=current_observations["fancy_cube"]["position"],
                placing_position=current_observations["fancy_cube"]["goal_position"],
                current_joint_positions=current_observations["fancy_franka"]["joint_positions"],
            )
            self._franka.apply_action(actions)
            if self._controller.is_done():
                self._world.pause()
            return

Let us know if you have more questions.

3 Likes

Thank you @oahmed for your answer.
The problem for opening the gripper was solved by the provided code, however, it seems that the expected behavior of the robot for picking up the cube is different from what we see; The robot does not correctly recognize the size/scale/boundary of the cube and its gripper goes through the cube and cannot grab it. So the robot is not able to pick the cube up.

I have these warnings :
[Warning] [omni.client.plugin] Tick: authentication: Discovery(wss://localhost/omni/discovery): Error creating Api/Connection search: Not connected
[Warning] [carb.flatcache.plugin] Type tag does not have a corresponding USD type
[Warning] [gpu.foundation.plugin] Requesting texture to use texture streaming, but the service isn’t available.The texture will be created as a regular resource.

I would appreciate it if you give the solution to this problem too.

I found that the problem was due to the assets that I used to import Franka, and fixed by loading ““/Isaac/Robots/Franka/franka.usd” file instead of "/Isaac/Robots/Franka/franka_alt_fingers.usd” in the line 59 of the following direction:
.local/share/ov/pkg/isaac_sim-2022.1.1/exts/omni.isaac.franka/omni/isaac/franka/franka.py

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.