Discrepancy Between Deformable and Visual Meshes in Python Standalone Application

Isaac Sim Version

4.2.0
4.1.0
4.0.0
2023.1.1
2023.1.0-hotfix.1
Other (please specify):

Operating System

Ubuntu 22.04
Ubuntu 20.04
Windows 11
Windows 10
Other (please specify):

GPU Information

  • Model: GeForce RTX 3090
  • Driver Version: 535.183.01

Topic Description

Detailed Description

I am attempting to simulate a robot interacting with a deformable object in a python standalone application. However, when simulating the deformable object via a standalone application, it appears that the deformable mesh of the object is never updated during simulation to match the visual mesh. In contrast, when an identical scene is setup using an Isaac Sim instance launched via the omniverse launcher, the deformable mesh correctly updates during simulation. Is there a way to get the deformable mesh to match the visual mesh when running in a python standalone application?

Steps to Reproduce

  1. Create an empty scene
  2. Add a cube mesh to the scene
  3. Add the deformable body property to the mesh
  4. Create a deformable body physics material and attach it to the mesh
  5. Save the scene as a usd
  6. Load and run the USD in a python standalone application via the following code:
from isaacsim import SimulationApp
simulation_app = SimulationApp({"headless": False})

from omni.isaac.core import World
from omni.isaac.core.utils import stage

if __name__ == "__main__":
    scene_usd = YOUR_PATH_TO_USD_FILE
    device = "cuda"
    backend = "torch"

    simulation_context = World(backend=backend, device=device)

    simulation_context.get_physics_context().enable_gpu_dynamics(True)
    simulation_context.get_physics_context().set_gpu_max_rigid_contact_count(524288)
    simulation_context.get_physics_context().set_gpu_max_rigid_patch_count(81920)

    
    simulation_context.scene.add_default_ground_plane()

    scene_path = f"/World/env_0"
    stage.add_reference_to_stage(scene_usd, scene_path)

    simulation_context.reset()
    simulation_context.initialize_physics()

    while True:
        simulation_context.step(render=True)

Screenshots or Videos

Result of running the scene in an Isaac Sim instance launched via the Omniverse Launcher

Result of running the same scene via the above code in a Python Standalone Application

Additional Information

What I’ve Tried

I have also tried to generate the deformable object at runtime instead of loading via a USD file with the following script which produces the same result:

from omni.kit.commands import execute
from pxr import UsdGeom, PhysxSchema
from omni.physx.scripts import deformableUtils, physicsUtils
from omni.usd import get_stage_next_free_path
import omni
import carb

DEFAULT_PHYS_PARAMS = {
            "collision_simplification": False,
            "simulation_resolution": 3,
            "self_collision": False,
            "solver_position_iteration_count": 20,
            "youngs_modulus": 1000000.0,
            "poissons_ratio": 0.49,
            "damping_scale": 0.0,
            "elasticity_damping": 0.0001,
            "dynamic_friction": 10,
            "density": 300,
        }

class DeformableBox():
    def __init__(self, stage, env_num):
        self.stage = stage
        self.env_num = env_num
        self.cube_mesh = self.CreateCubeMesh(f"/World/env_{env_num}/Cube")
        self.path = self.cube_mesh.GetPath()

        self.cube_mesh.ClearXformOpOrder()
        self.translate = self.cube_mesh.AddTranslateOp()
        self.rotate = self.cube_mesh.AddOrientOp(precision=UsdGeom.XformOp.PrecisionDouble)
        self.scale = self.cube_mesh.AddScaleOp(precision=UsdGeom.XformOp.PrecisionDouble)

        self.physxCollisionAPI = PhysxSchema.PhysxCollisionAPI.Apply(self.cube_mesh.GetPrim())
        self.contactOffsetAttr = self.physxCollisionAPI.GetContactOffsetAttr()
        self.restOffsetAttr = self.physxCollisionAPI.CreateRestOffsetAttr()

    def CreateCubeMesh(self, target_path):
        _, tmp_path = execute("CreateMeshPrimCommand", prim_type="Cube", u_verts_scale=10, v_verts_scale=10,
                              w_verts_scale=10, half_scale=50)
        execute("MovePrim", path_from=tmp_path, path_to=target_path)
        omni.usd.get_context().get_selection().set_selected_prim_paths([], False)
        return UsdGeom.Mesh.Get(self.stage, target_path)

    def SetTranslate(self, position):
        self.translate.Set(position)

    def SetOrient(self, orientation):
        self.rotate.Set(orientation)
    
    def SetScale(self, scale):
        self.scale.Set(scale)

    def SetDeformablePhysics(self, phys_params: dict):
        for param in phys_params.keys():
            if param not in DEFAULT_PHYS_PARAMS.keys():
                carb.logwarn(f"{param} not a valid physics parameter.")

        final_phys_params = {}
        for param in DEFAULT_PHYS_PARAMS.keys():
            if param in phys_params.keys():
                final_phys_params[param] = phys_params[param]
            else:
                final_phys_params[param] = DEFAULT_PHYS_PARAMS[param]

        success = deformableUtils.add_physx_deformable_body(
            self.stage,
            self.path,
            collision_simplification=final_phys_params['collision_simplification'],
            simulation_hexahedral_resolution=final_phys_params['simulation_resolution'],
            self_collision=final_phys_params['self_collision'],
            solver_position_iteration_count=final_phys_params['solver_position_iteration_count'],
        )

        self.deformable_material_path = get_stage_next_free_path(self.stage, f"/World/env_{self.env_num}/Cube_Material", True)
        deformableUtils.add_deformable_body_material(
            self.stage,
            self.deformable_material_path,
            youngs_modulus=final_phys_params['youngs_modulus'],
            poissons_ratio=final_phys_params['poissons_ratio'],
            damping_scale=final_phys_params['damping_scale'],
            elasticity_damping=final_phys_params['elasticity_damping'],
            dynamic_friction=final_phys_params['dynamic_friction'],
            density=final_phys_params['density'],
        )
        physicsUtils.add_physics_material_to_prim(self.stage, self.cube_mesh.GetPrim(), self.deformable_material_path)
    
    def SetContactOffsetAttr(self, offset):
        self.contactOffsetAttr.Set(offset)
    
    def SetRestOffsetAttr(self, offset):
        self.restOffsetAttr.Set(offset)

Not sure what the Isaac Sim simulation context step does, one thing that could be different is Fabric usage. If you use omni.physx.fabric, you could try to step also omni.physx.fabric after each step. So add something like:

from omni.physxfabric import get_physx_fabric_interface

while True:          simulation_context.step(render=True)  get_physx_fabric_interface().update(0.02, 0.02)

This does not seem to work on my end. The deformable meshes still appear to remain fixed. I receive the following warning message when I view the mesh:

2025-01-28 16:37:25 [25,666ms] [Warning] [omni.hydra.scene_delegate.plugin] Calling getBypassRenderSkelMeshProcessing for prim /PhysxDeformablesVisualizationMeshes/_World_env_0_Cube_collision/samplePoints.proto0_prototype_id0 that has not been populated

For further context, the goal of this project is to collect data on deformable object manipulations. However, grasping deformable objects in isaac sim does not seem to be robust enough yet for this kind of data collection as discussed in this forum post: How to grasp deformable object?. Instead of grasping, I would like to be able to dynamically create attachments between the robot’s gripper and the deformable object at runtime. These attachments; however, never appear to be populated. I believe this to be an issue with the deformable meshes never updating during simulation.

Presently, I am able to collect the desired data by loading in a USD with a predefined attachment between the robot and the deformable object, and I have been successful in dynamically creating these attachments when running Isaac Sim via the Omniverse Launcher. This feature does not work, however, when running in a Python Standalone App.