Custom robot not acheiving smooth movements

Omniverse Isaac Sim
1

I have a problem with isaac sim and curobo. I followed the tutorial from the Nvidia Curobo website: Configuring a New Robot - cuRobo, to configure a new robot. However, when I launch the example omni_python curobo/examples/isaac_sim/realsense_reacher.py --robot /home/theobloesch/Documents/xArm6Curobo/xArm6CuroboConfig.yaml --show-window

The movements of my robots are not smooth, and I am unable to adjust the depth detection settings of the RealSense camera.

image
image1466×937 93.2 KB

You can view how I set up my files here : isaac-sim-ufactory-example/xArm6/xArm6Curobo at main · heig-vd-iAi-LaRA/isaac-sim-ufactory-example · GitHub

I’m working on Ubuntu 22.04 IsaacSim version 4.1 Cuda version 11.8

Has anyone encountered these issues or knows how to resolve it?

Thank you in advance !

I have already try to :

  • Set the gains with the extension gains tuning but my robot don’t follow the trajectories.(I made these tuning with the gripper already attached don’t know if it’s right to do so)
2

Is this issue reproducible on other robots?

Could you please provide a video demonstrating the non-smooth behavior and elaborate on the problem of being unable to adjust the depth detection settings?

5

Thanks a lot for your fast response !

No when I launch the example with the Franka robot it performs smoothly.

Here this the video:

For the Realsense part : I have limited space in the simulation where my hand is detected. However, when I move my hand further away, it is no longer detected, even though it still appears on the camera. I was wondering if there is a way to increase the detection range or extend the distance at which the hand is detected.
Thank you in advance, and please don’t hesitate to reach out if you need any additional information

6

Hey @theo.bloesch. Thanks for sharing your robot files. Could you please also provide the script you are using for setting up the scene, loading robot and using curobo motion planning? I tried with the motion_gen_reacher.py provided by curobo and your custom robot xArm6. The motion seems pretty smooth (at least visually).

As for nvblox with realsense camera issue, nvblox is using the depth images from realsense camera to construct 2D/3D ESDF map. Nvblox has a range vertically (<mapper_name>.esdf_slice_max_height and <mapper_name>.esdf_slice_min_height) but I don’t think that would resolve your issue. You can refer to this page for more parameters

Usually depth camera has a detection range as well. Could you try to visualize the depth images from the realsense camera and see if it is able to detect your hand at a further distance?

Based on your video, the image on the right side of nvblox example doesn’t seem to be ESDF from nvblox… It seems more like the depth image from ESS… Are you following the realsense camera example tutorial?

8

Thank you for your response!

I’ve added the CUROBO example to the GitHub repository :

I’ll look into visualizing the depth image, but I need to figure out how to do it first.

For now, I’ve only run the example provided by CUROBO to test it before exploring other features.

image
image1243×862 164 KB

Thank you again for all the help and responses you’ve already provided!

9

I have use Realsense viewer and that’s what it’s look like :

image
image2490×1573 324 KB

10

I tried running motion_gen_reacher.py and observed the same results as you: smooth movement.
I also tested the Curobo example with the Franka robot and noticed irregular movements, although they were less pronounced compared to my robot.

Here’s a screencast that might help diagnose the issue:


Let me know if this helps identify the problem.

1 Like
11

Hello,

I’ve also noticed a vibration at the end of the movements in the motion_gen_reacher.py example with the UR5e arm. It seems to occur specifically when stopping at a target position.

Is there a solution or specific parameters to adjust to fix this issue?

Thank you in advance for your help!

12

Hi, I’ve also noticed the same issue. I am running the motion_gen_reacher.py and constrained_reacher.py examples with the UR5e robot. The only change I made is to set the --robot param from franka.yml to ur5e.yml and change the static joint speed to 0.5 in motion_gen_reacher.py.

Both examples worked smoothly on the Franka robot.

However, in my case, the motion_gen_reacher.py example worked well on UR5e with no vibration, but the constrained example vibrated a lot.

The following are two videos that show this issue.

This is the motion_gen_reacher.py example without vibration：

This is the constrained reacher example with vibration：

13

Update: Possible Reachability Issue with UR5e

I’ve narrowed down the target positions for the robot, as shown in the code below:

Origin:

    target = cuboid.VisualCuboid(
        "/World/target_1",
        position=np.array([0.55, -0.3, 0.5]),
        orientation=np.array(target_orient),
        size=0.04,
        visual_material=target_material,
    )

    # Make a target to follow
    target_2 = cuboid.VisualCuboid(
        "/World/target_2",
        position=np.array([0.55, 0.4, 0.5]),
        orientation=np.array(target_orient),
        size=0.04,
        visual_material=target_material_2,
    )

Narrowed down:

    target = cuboid.VisualCuboid(
        "/World/target_1",
        position=np.array([0.45, -0.1, 0.5]),
        orientation=np.array(target_orient),
        size=0.04,
        visual_material=target_material,
    )

    # Make a target to follow
    target_2 = cuboid.VisualCuboid(
        "/World/target_2",
        position=np.array([0.45, 0.2, 0.5]),
        orientation=np.array(target_orient),
        size=0.04,
        visual_material=target_material_2,
    )

The UR5e robot can move smoothly between the targets without vibration.

So I suspect the robot cannot interpolate enough reachable waypoints in the origin example for Franka, as Franka panda is a 7-DOF arm, it may have more reachable points in the task space

By the way, if we set time_dilation_factor=0.5 to a larger value, such as 1.0. The robot will move faster and vibrate even in the narrowed-done target positions, but the franka robot can still move smoothly

I appreciate more professional suggestions and insights on this issue!

Thanks.

14

Update: The shaking issue with UR5e is possibly caused by the repeated execution of the same action in the constrained_reacher.py example.

After carefully reviewing the two examples (constrained_reacher.py and motion_gen_reacher.py), I noticed that the constrained_reacher.py example executes each action twice using a counter named cmd_step_idx , below is the relevant code snippet:

...
if cmd_plan is not None:
    cmd_state = cmd_plan[cmd_idx]

    # get full dof state
    art_action = ArticulationAction(
        cmd_state.position.cpu().numpy(),
        cmd_state.velocity.cpu().numpy(),
        joint_indices=idx_list,
    )
    # set desired joint angles obtained from IK:
    articulation_controller.apply_action(art_action)
    cmd_step_idx += 1
    if cmd_step_idx == 2: ####### check this line
        cmd_idx += 1
        cmd_step_idx = 0
    # for _ in range(2):
    #    my_world.step(render=False)
    if cmd_idx >= len(cmd_plan.position):
        cmd_idx = 0
        cmd_plan = None
...

By removing this counter and executing each action only once, the constrained_reacher.py example worked on UR5e much smoother in my tests.

The two executions were likely intended to make the motion smoother, but on the UR5e, they seem to have caused shaking instead. I am still confused about this and why the example can work without shaking with the Franka robot, I hope someone can provide some insights here.