Mimic joints are lagging behind parent joints

[re23m017]

Isaac Sim Version: 4.5.0
Operating System inside Docker: Ubuntu 22.04
GPU Information:
Model: 4070ti Super 16GB
Driver Version: 535.230.02

Topic Description

Mimic Joints are taken over into the USD file but not as a rigid linkage with no give and appear rather to be linked by what i assume to be a tendon or a spring since they tend to oscillate or give in if pressure is applied (see Fig.1.).

Fig. 1
the pinky finger is extended to its max while it is supposed to look more like the ring finger where the mimic joints has the same angle as the parent joint. The urdf that is also available in the git repo works without a problem on itself.

Detailed Description

i have used the command URDFParseAndImportFile with their respective configuration parameters and then saved the imported urdf as an usd before continuing to work with it. i have made sure to include the configuration to “import_config.set_parse_mimic(False)” to not import the mimics as a tendon but the behavior still persists (IsaacLab/source/isaaclab_tasks/isaaclab_tasks/direct/bionic_arm/urdf_importer.py at 705383dde573d5fa700d51b73dd40f1fc44f11b1 · YuziIV/IsaacLab · GitHub). When loading the URDF i had to add the mimic joint names to the attribute “joint_names_expr” otherwise i would get the error that the joints are not fully configured. My action space also gives me another sanity check that the mimic joints have no drive behind them because they only include the 7 joints i want to be actuated. Because the real robotic hand uses a linear actuator and a rigid linkage to move the distant Phalanx, i thought that i could just use mimic joints to replicate the true movement since URDFs don’t support closed loop structures (see Video 1).

Steps to Reproduce

1. clone repository GitHub - YuziIV/IsaacLab: Unified framework for robot learning built on NVIDIA Isaac Sim
2. checkout into branch “bionic-arm”
3. cd into IsaacLab/docker/
4. run <./container.py start> wait for container to build and start
5. connect to container with bash by running <docker exec -it isaac-lab-base bash>
6. run the command to train the custom arm <./isaaclab.sh -p scripts/reinforcement_learning/skrl/train.py --task Isaac-Repose-Cube-BionicArm-Direct-v0 --algorithm PPO --num_envs 32>

Screenshots or Videos

Video 1

Additional Information

What I’ve Tried

-increased the iteration steps to decrease oscillations but it didn’t help with the soft linking of the mimic joints.
solver_position_iteration_count=16,
solver_velocity_iteration_count=2,
-imported the mimic joints as tendons and set the parameter
fixed_tendons_props=sim_utils.FixedTendonPropertiesCfg(stiffness=1e9, damping=0.0)
-separate stiffness and dampening settings for mimic and non-mimic joints to replicate a prefect P-Controller.

Are there any obvious mistakes that have been made or fixes that i could apply? Thank you in advance!

25-04-11 UPDATE: stiffness of the joints were set to 0.1 instead of 1.0.
The issue described above is now occurring with the most recent Git commit:
Hash: c2e152e37af3c022467f941a0d8242935cfd7526

[re23m017]

Workaround using the UI

When importing the URDF via the UI with ./runapp.sh and using the Import Wizard, the same behavior of the mimic joints can be observed. Moving its parent joint, the linked mimic joint would move with it but as if it was mounted by a spring or lagging behind till it catches up, overshoots and then settles in (PD control). So i did the same in the UI as i tried in code. After importing I changed the Stiffness to 1e9 and the dampening to 0 for all mimic joints. This then seemed to work, when viewed with the Tools>Physics>Physics Inspector. After saving the USD and starting the training script again it worked! The mimic joints move as intended. I however still want to implement the same function but when using the import script so I can easily swap out the URDF and have a fully functioning USD afterwards. Is there a option to import the URDF with custom stiffness settings?

bionic_arm_cfg

    if not os.path.exists(usd_path):
        urdf_importer = ImportURDF(urdf_path, usd_path)
        status, prim_path = urdf_importer.import_urdf_to_usd()
        print("URDF import status:", status, "Prim path:", prim_path)
    #BIONIC_ARM_CFG = ArticulationCfg(
    robot_cfg = ArticulationCfg(
        prim_path="/World/envs/env_.*/Robot",
        spawn=sim_utils.UsdFileCfg(
            usd_path=usd_path,
            activate_contact_sensors=False,
            rigid_props=sim_utils.RigidBodyPropertiesCfg(
                disable_gravity=True,
                retain_accelerations=True,
                max_depenetration_velocity=1000.0,
            ),
            articulation_props=sim_utils.ArticulationRootPropertiesCfg(
                enabled_self_collisions=True,
                solver_position_iteration_count=8,
                solver_velocity_iteration_count=0,
                sleep_threshold=0.005,
                stabilization_threshold=0.0005,
            ),
            joint_drive_props=sim_utils.JointDrivePropertiesCfg(drive_type="force"),
            #fixed_tendons_props=sim_utils.FixedTendonPropertiesCfg(stiffness=1e9, damping=0.0)
        ),
        init_state=ArticulationCfg.InitialStateCfg(
            pos=(0.0, -0.15, 0.5),
            rot=(0.7071068, 0.7071068, 0.0, 0.0),
            joint_pos={".*": 0.0},
        ),
        actuators={
            "fingers": ImplicitActuatorCfg(
                joint_names_expr=[
                    "palm_joint",
                    "pinky_joint",
                    "pinky_mimic_joint",
                    "ring_joint",
                    "ring_mimic_joint",
                    "middle_joint",
                    "middle_mimic_joint",
                    "pointer_joint",
                    "pointer_mimic_joint",
                    "thumb_swivel_joint",
                    "thumb_joint",
                    "thumb_mimic_joint",
                ],
                effort_limit_sim={
                    "palm_joint": 5.0,
                    "pinky_joint": 2.0,
                    "ring_joint": 2.0,
                    "middle_joint": 2.0,
                    "pointer_joint": 2.0,
                    "thumb_swivel_joint": 2.0,
                    "thumb_joint": 2.0,
                },
                stiffness={
                    "palm_joint": 1.0,
                    "pinky_joint": 1.0,
                    "ring_joint": 1.0,
                    "middle_joint": 1.0,
                    "pointer_joint": 1.0,
                    "thumb_swivel_joint": 1.0,
                    "thumb_joint": 1.0,
                    "pinky_mimic_joint": 1e9,
                    "ring_mimic_joint": 1e9,
                    "middle_mimic_joint": 1e9,
                    "pointer_mimic_joint": 1e9,
                    "thumb_mimic_joint": 1e9,
                },
                damping={                    
                    "palm_joint": 0.1,
                    "pinky_joint": 0.1,
                    "ring_joint": 0.1,
                    "middle_joint": 0.1,
                    "pointer_joint": 0.1,
                    "thumb_swivel_joint": 0.1,
                    "thumb_joint": 0.1,
                    "pinky_mimic_joint": 0.0,
                    "ring_mimic_joint": 0.0,
                    "middle_mimic_joint": 0.0,
                    "pointer_mimic_joint": 0.0,
                    "thumb_mimic_joint": 0.0,
                },
                
            ),
        },

This is how I load the robot onto my stage now (not yet committed to git). Note that the stiffness and the dampening are now set the same as I have set them through the UI. With these settings when trying to load the URDF, convert to USD and add to scene via code the model still doesn’t work as intended.

SIDENOTE: the settings for stiffness and dampening in the code seem to have no effect on the imported model, if the USD from the UI is being used.

urdf_imnporter.py

import omni.kit.commands
#from isaacsim.core.utils.extensions import get_extension_path_from_name


class ImportURDF:
    def __init__(self, urdf_path: str, dest_path: str):
        self.urdf_path = urdf_path
        self.dest_path = dest_path

    def import_urdf_to_usd(self):
        # Setting up import configuration:
        status, import_config = omni.kit.commands.execute("URDFCreateImportConfig")
        import_config.merge_fixed_joints = False  
        import_config.convex_decomp = False       
        import_config.import_inertia_tensor = True
        import_config.fix_base = True              
        import_config.distance_scale = 1.0          
        import_config.make_default_prim = True      
        import_config.self_collision = True         
        import_config.density = 0.0  
        import_config.set_parse_mimic(False)         

        # Import the URDF and convert it to USD at the destination path
        status, prim_path = omni.kit.commands.execute(
            "URDFParseAndImportFile",
            urdf_path=self.urdf_path,
            import_config=import_config,
            get_articulation_root=False,
            dest_path=self.dest_path
        )
        return status, prim_path

Can something be added to the urdf_importer.py file to ensure that the stiffness gets loaded correctly? I assume that urdf_importer.py performs the same function as the wizard, since the settings are quite similar. In both cases, the URDF seems to be imported and converted correctly, so the error most likely lies within the bionic_arm_env_cfg.py file I believe.

Thank you in advance for the help!

[michalin]

hi @re23m017, thanks for posting your question, also thanks for sharing all the resources to debug the issue. It is unexpected that mimic joint requires a high stiffness to work. I will look into this soon and get back to you.

[michalin]

Hi @re23m017, sorry for the very long delay to get back to you. For clarification, it seems from this that you are already setting some joint stiffnesses to 1e9.

@re23m017

                stiffness={
                    "palm_joint": 1.0,
                    "pinky_joint": 1.0,
                    "ring_joint": 1.0,
                    "middle_joint": 1.0,
                    "pointer_joint": 1.0,
                    "thumb_swivel_joint": 1.0,
                    "thumb_joint": 1.0,
                    "pinky_mimic_joint": 1e9,
                    "ring_mimic_joint": 1e9,
                    "middle_mimic_joint": 1e9,
                    "pointer_mimic_joint": 1e9,
                    "thumb_mimic_joint": 1e9,
                },

You mentioned that

@re23m017

After importing I changed the Stiffness to 1e9 and the dampening to 0 for all mimic joints. This then seemed to work, when viewed with the Tools>Physics>Physics Inspector.

Do you mean that you had to re-set the mimic joints to 1e9?

[re23m017]

Hello, my first approach was to set the stiffness in code when importing the URDF and automatically saving as USD, which didn’t work. What did work for my model is when I opened the URDF in IsaacSim, using the “URDF Import Wizard (Beta)” and then manually set the stiffness and dampening through the UI. The model could then be saved by manually exporting it as an USD. This then worked. Both approaches use an unchanged URDF to then modify and save as USD before counting the work with the USD.

[michalin]

I see what the problem is. So to clarify, modifying Drive parameters (stiffness damping) through the GUI works. But when you load the asset through Isaac Lab ArticulationCfg the stiffness/damping parameters are not set. Is that correct? I am seeing this behavior on my side as well if that is the case.

[re23m017]

Yes, exactly! Sorry if my original explanation was a bit too complex. I really appreciate your simplified summary, it gets straight to the point. Is this issue with how I am handling the urdf at import or somthing with the ArticulationCfg?

[michalin]

There is a known issue in IsaacLab where Joint stiffnesses set via ArticulationCfg actuators don’t appear in Isaac Sim’s USD Properties window. The reason is the under the hood, this parameter setting uses ArticulationView:set_dof_stiffnesses() which is the low level PhysX interface. So it skips the USD layer and this data does not get propagated back up to Isaac Sim USD.

That said, the stiffness does get written correctly to the physics engine so the dynamics behavior should be correct to what is set via ArticulationCfg actuators. It is confusing that the GUI will show the stiffnesses as not updated.

Best way to solve this mismatching GUI parameters is to set the parameters directly on the USD and save the file, but besides the GUI there should not be differences in the physics dynamics. We are working on getting a fix for this in the next releases of Isaac Lab.

[re23m017]

Thank you for the clarification! I initially thought the stiffness wasn’t taking effect in the sim at all, but knowing it’s applied at the PhysX level helps a lot. The issue must then lie somewhere else in my code. I’ll look into it now with this in mind. Appreciate the help!