Curobo attach object with isaac sim

theo.bloesch · January 27, 2025, 4:28pm

Isaac Sim Version

4.1.0

Operating System

Ubuntu 22.04

Topic Description

How to properly use Curobo with Isaac Sim to generate a collision-free trajectory with an attached object

Detailed Description

I am trying to perform a custom pick-and-place task using MotionGen to generate a collision-free trajectory. However, I’m encountering several issues:

I don’t know how to attach the cube to the robot to inform the motion generator that the robot now has an object attached.
The cube I pick in my program doesn’t stay on the rack if I place it there.
The motion generation sometimes results in strange movements.

If possible, I would appreciate guidance on these points.

All the files I am using are available here:

I’m still learning how to use Isaac Sim and Curobo, so I apologize if the code is unclear. Please feel free to let me know if anything is incorrect, unclear or how I could improve my code.

Steps to Reproduce

Run the standalone script Pick_Place.py

Screenshots or Videos

Strange behavior with the motion generation

Behavior if I set the height of the previous position higher :

In both case I would like that the blue cube stay on the rack.

Thanks in advance your respones
Best regards
Théo Bloesch

zhengwang · January 28, 2025, 1:44am

Hey @theo.bloesch. Which branch of curobo are you using?

theo.bloesch · January 28, 2025, 7:07am

Thank for your response !

I’m using the main branch of the github. I followed these instructions :

Install for use in Isaac Sim

Install CUDA 11.8 following cuda_instructions and add to your path with: export PATH=/usr/local/cuda-11.8/bin${PATH:+:${PATH}}
Install NVIDIA Isaac Sim 4.0 following isaac_sim_workstation_install
Add an alias to Isaac Sim’s python in your bashrc file: echo "alias omni_python='~/.local/share/ov/pkg/isaac_sim-4.0.0/python.sh'" >> ~/.bashrc and source it with source ~/.bashrc. If your isaac sim path is different, change the path accordingly.
Install git lfs with sudo apt install git-lfs.
Install these additional packages: omni_python -m pip install tomli wheel ninja.
Clone cuRobo repository with git clone https://github.com/NVlabs/curobo.git, move inside the cloned repo with cd curobo.
run omni_python -m pip install -e .[isaacsim] --no-build-isolation, where omni_python refers to ./python.sh from isaac sim installation (info).
Run omni_python examples/isaac_sim/motion_gen_reacher.py to run a motion generation example. Once Isaac Sim loads, click play and move the target cube to start generating motions.

from here : Installation - cuRobo

Thank you in advance
Best regards
Théo Bloesch

zhengwang · January 30, 2025, 12:12am

Thanks! I tried to run your example.
It looks like you don’t have colliders preset set for your target. This might cause that the robot isn’t notified that it has an object attached.

If you check the example simple_stacking provided in curobo, the cubes have colliders preset. This example is using Stacking.

Have you checked about PickPlace? This might satisfy your needs.

You might also want to add colliders preset for the racks/table you have in your sim so the target doesn’t fall through it and robot arm doesn’t plan path through it.

theo.bloesch · January 30, 2025, 2:48pm

Hi, thank you for your response!
I managed to enable the colliders preset for the target and the rack/table using DynamicCuboid and FixedCuboid. Is there an easier or more straightforward way to enable these options? I checked the simple_stacking example but couldn’t identify where the cuboids were added to the scene. I also reviewed the PickPlace class, but I’d prefer not to rely on it initially, as I want to better understand the underlying process.

Additionally, I believe I’ve successfully attached the object to the robot’s end effector, and the spheres are being generated. However, I’m not entirely sure if I’ve done this correctly. Furthermore, I don’t understand why enabling the spheres causes the robot to get stuck, while without them it runs smoothly. Also, I noticed that one of the spheres gets stuck on the table. Could you provide some guidance on these issues?

Here’s a video of how I attached the cube with the spheres:

Here’s a video of how I attached the cube without the spheres:

All the file are in the github :

with program Pick_Place.py

Curobo config file with the extra link :

robot_cfg:
  kinematics:
    usd_path: "xarm6/xarm6.usd"
    usd_robot_root: "xarm6/xarm6.usd"
    isaac_usd_path: ""
    usd_flip_joints: {}
    usd_flip_joint_limits: []
    
    asset_root_path: "/home/theobloesch/xarm6_pick_and_place/xArm6Curobo/xarm6.urdf"
    urdf_path: "/home/theobloesch/xarm6_pick_and_place/xArm6Curobo/xarm6.urdf"
    
    
    base_link: "xarm6link_base"
    ee_link: "xarm6link_tcp"
    link_names: null
    lock_joints: null
    extra_links: null

    
    collision_link_names: ["xarm6link1", "xarm6link2", "xarm6link3", "xarm6link4", "xarm6link5", "xarm6link6","xarm6xarm_gripper_base_link","attached_object",]
    collision_spheres:
      xarm6link1:
        - "center": [0.0, -0.004, -0.078]
          "radius": 0.06
        - "center": [-0.0, -0.009, -0.008]
          "radius": 0.06
        - "center": [0.0, 0.034, -0.023]
          "radius": 0.06
      xarm6link2:
        - "center": [0.0, -0.004, 0.077]
          "radius": 0.05
        - "center": [0.004, -0.069, 0.1]
          "radius": 0.03
        - "center": [-0.0, -0.11, 0.079]
          "radius": 0.05
        - "center": [-0.001, -0.109, 0.006]
          "radius": 0.05
        - "center": [-0.0, -0.17, -0.0]
          "radius": 0.045
        - "center": [-0.002, -0.216, -0.011]
          "radius": 0.045
        - "center": [0.047, -0.286, -0.027]
          "radius": 0.045
        - "center": [0.056, -0.284, 0.026]
          "radius": 0.045
      xarm6link3:
        - "center": [0.0, -0.0, 0.07]
          "radius": 0.045
        - "center": [0.047, 0.044, 0.088]
          "radius": 0.02
        - "center": [0.078, 0.082, 0.064]
          "radius": 0.045
        - "center": [0.079, 0.083, -0.0]
          "radius": 0.045
        - "center": [0.078, 0.134, -0.0]
          "radius": 0.045
      xarm6link4:
        - "center": [0.001, 0.069, -0.073]
          "radius": 0.025
        - "center": [0.0, 0.073, -0.025]
          "radius": 0.02
        - "center": [0.0, 0.031, 0.0]
          "radius": 0.035
        - "center": [0.001, 0.001, -0.156]
          "radius": 0.035
      xarm6link5:
        - "center": [0.003, -0.0, 0.014]
          "radius": 0.035
        - "center": [0.075, 0.004, 0.0]
          "radius": 0.035
        - "center": [0.074, 0.058, 0.0]
          "radius": 0.035
      xarm6link6:
        - "center": [-0.0, 0.0, -0.008]
          "radius": 0.035
      xarm6xarm_gripper_base_link:
        - "center": [-0.006, 0.0, 0.067]
          "radius": 0.035
        - "center": [-0.0, 0.052, 0.156]
          "radius": 0.01
        - "center": [0.0, -0.05, 0.138]
          "radius": 0.002
        - "center": [0.0, 0.0, 0.0]
          "radius": 0.02
        - "center": [0.0, 0.0, 0.0]
          "radius": 0.02
        - "center": [-0.002, 0.06, 0.095]
          "radius": 0.02
        - "center": [0.0, -0.063, 0.088]
          "radius": 0.02
        - "center": [-0.0, -0.051, 0.155]
          "radius": 0.01
        - "center": [0.0, -0.035, 0.038]
          "radius": 0.02
        - "center": [0.0, 0.036, 0.036]
          "radius": 0.02
    collision_sphere_buffer: 0.005 # float or Dict[str, float]
    extra_collision_spheres: {"attached_object": 25}
    self_collision_ignore: {
        "xarm6link1":["xarm6link2", "xarm6link3", "xarm6link4", "xarm6link5", "xarm6link6","xarm6xarm_gripper_base_link"],
        "xarm6link2":["xarm6link1", "xarm6link3", "xarm6link4", "xarm6link5", "xarm6link6","xarm6xarm_gripper_base_link"],
        "xarm6link3":["xarm6link2", "xarm6link1", "xarm6link4", "xarm6link5", "xarm6link6","xarm6xarm_gripper_base_link"],
        "xarm6link4":["xarm6link2", "xarm6link3", "xarm6link1", "xarm6link5", "xarm6link6","xarm6xarm_gripper_base_link"],
        "xarm6link5":["xarm6link2", "xarm6link3", "xarm6link4", "xarm6link1", "xarm6link6","xarm6xarm_gripper_base_link"],
        "xarm6link6":["xarm6link2", "xarm6link3", "xarm6link4", "xarm6link5", "xarm6link1","xarm6xarm_gripper_base_link"],
        "xarm6xarm_gripper_base_link":["xarm6link1","xarm6link2", "xarm6link3", "xarm6link4", "xarm6link5", "xarm6link6"]
        }
    self_collision_buffer: {} # Dict[str, float]

    use_global_cumul: True
    mesh_link_names: null # List[str]
    external_asset_path: null # Use this to add path for externally located assets/robot folder.
    extra_links: {"attached_object":{"parent_link_name": "xarm6xarm_gripper_base_link" ,
        "link_name": "attached_object", "fixed_transform": [0,0,0,1,0,0,0], "joint_type":"FIXED",
        "joint_name": "attach_joint" }}
    cspace:
      joint_names: ['xarm6joint1', 'xarm6joint2', 'xarm6joint3', 'xarm6joint4', 'xarm6joint5', 'xarm6joint6']
      retract_config: [0., -0.36, -0.26, 0., 0.62, 0.]
      null_space_weight:  [1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
      cspace_distance_weight: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
      max_jerk: 500.0
      max_acceleration: 15.0

Methode to attach object :

   
    def attach_object(self,cube_name):
        # Attach the object to the robot and add the object to the motion generator
        sim_js = self.robot.get_joints_state()
        cu_js = JointState(
            position=self.tensor_args.to_device(sim_js.positions),
            velocity=self.tensor_args.to_device(sim_js.velocities) * 0.0,
            acceleration=self.tensor_args.to_device(sim_js.velocities) * 0.0,
            jerk=self.tensor_args.to_device(sim_js.velocities) * 0.0,
            joint_names=self.j_names,
        )
        self.motion_gen.attach_objects_to_robot(
            cu_js,
            [cube_name],
            link_name="attached_object",
            sphere_fit_type=SphereFitType.VOXEL_VOLUME_SAMPLE_SURFACE,
            world_objects_pose_offset=Pose.from_list([0, 0, -0.16, 1, 0, 0, 0], self.tensor_args),
            remove_obstacles_from_world_config = True,
            surface_sphere_radius = 0.005
        )

I also have used the method to update the obstacle list before, after taking the cube and after rising it from the table. Is this the correct way to proceed ? :

     def update_world_obstacles_before_taking(self):
            print("Updating world, reading w.r.t.", self.robot_prim_path)
            obstacles = self.usd_help.get_obstacles_from_stage(
                # only_paths="obstacles",
                reference_prim_path=self.robot_prim_path,
                ignore_substring=[
                    self.robot_prim_path,
                    #"/World/target",
                    "/World/defaultGroundPlane",
                    #"/World/random_cube",
                    "/curobo",
                    "/World/table",
                    "/World/obstacles/table",
                    "/World/obstacles/table_mesh",
                    
                ],
            ).get_collision_check_world()
            print("Obstacles read from stage",len(obstacles.objects))

            self.motion_gen.update_world(obstacles)
            print("Updated World")
            carb.log_info("Synced CuRobo world from stage.")
    def update_world_obstacles_after_taking(self):
            print("Updating world, reading w.r.t.", self.robot_prim_path)
            obstacles = self.usd_help.get_obstacles_from_stage(
                # only_paths="obstacles",
                reference_prim_path=self.robot_prim_path,
                ignore_substring=[
                    self.robot_prim_path,
                    #"/World/target",
                   "/World/defaultGroundPlane",
                    "/World/random_cube",
                    "/curobo",
                    "/World/table",
                    "/World/obstacles/table",
                    "/World/obstacles/table_mesh",
                    
                ],
            ).get_collision_check_world()
            print("Obstacles read from stage",len(obstacles.objects))

            self.motion_gen.update_world(obstacles)
            print("Updated World")
            carb.log_info("Synced CuRobo world from stage.")
            
    def update_world_obstacles_after_rising(self):
        print("Updating world, reading w.r.t.", self.robot_prim_path)
        obstacles = self.usd_help.get_obstacles_from_stage(
            # only_paths="obstacles",
            reference_prim_path=self.robot_prim_path,
            ignore_substring=[
                self.robot_prim_path,
                #"/World/target",
                #"/World/defaultGroundPlane",
                "/World/random_cube",
                #"/curobo",
                #"/World/table",
                #"/World/obstacles/table",
                #"/World/obstacles/table_mesh",
                
            ],
        ).get_collision_check_world()
        print("Obstacles read from stage",len(obstacles.objects))

        self.motion_gen.update_world(obstacles)
        print("Updated World")
        carb.log_info("Synced CuRobo world from stage.")
``
Please let me know if any part of my questions are unclear or if my code is difficult to understand
Thanks in advance for your responses.
Best regards
Théo Bloesch

Topic		Replies	Views
Curobo MotionGenStatus.INVALID_START_STATE_JOINT_LIMITS Isaac Sim isaac-sim-v4-0-0	3	99	December 14, 2024
Curobo with Isaac sim: IK failure Isaac Sim	12	463	August 30, 2024
Problem with CUDA Memory Isaac Sim cuda , ubuntu , cudnn , isaac-sim-v4-0-0	7	162	March 14, 2025
Motion_gen_reacher.py example for cuboro does not work Isaac Sim	2	276	June 17, 2024
Deformable Cube not being recognized despite successful creation Isaac Sim isaac-sim-v4-1-0	0	34	October 25, 2024
Discrepancy Between Deformable and Visual Meshes in Python Standalone Application Isaac Sim python , isaac-sim-v4-2-0	2	65	January 28, 2025
[Isaac Lab] Remove Objects during simulation Isaac Sim isaac-sim-v4-0-0	0	110	November 12, 2024
How to move the vehicle to its destination by using Python? Isaac Sim python , physx , isaacsim	17	3098	May 5, 2022
Moving ground with collision - Isaaclab Isaac Sim isaac-sim-v4-2-0	3	149	January 27, 2025
Problem setting up a new robot and planning with isaac_sim_cumotion, start or end state in collision Isaac ROS isaac-ros-common , ros-2-humble	0	52	January 3, 2025