[carb.gym.plugin] Could not load collision / visual mesh

Autonomous Machines Robotics - Isaac Isaac Gym
1

With a fresh install of preview release 3 into a new conda environment, the IsaacGymEnvs that rely on obj/dae asset loading fail for me and I’m having difficulty tracking down the error. For example, below is the trace for the FrankaCabinet task. I get similar mesh loading errors leading to segfaults for the AllegroHand, ShadowHand, and Trifinger, while the two Anymal variants (which load daes instead of objs) still give the same [carb.gym.plugin] errors but at least launch the simulation (no segfault, but I can’t see anything because no meshes are loaded).

python train.py task=FrankaCabinet
Importing module 'gym_37' (/home/jfoster/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_37.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/jfoster/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
PyTorch version 1.8.1
Device count 1
/home/jfoster/isaacgym/python/isaacgym/_bindings/src/gymtorch
Using /home/jfoster/.cache/torch_extensions as PyTorch extensions root...
Emitting ninja build file /home/jfoster/.cache/torch_extensions/gymtorch/build.ninja...
Building extension module gymtorch...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module gymtorch...
/home/jfoster/miniconda3/envs/rlgpu/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'config': Defaults list is missing `_self_`. See https://hydra.cc/docs/upgrades/1.0_to_1.1/default_composition_order for more information
  warnings.warn(msg, UserWarning)
/home/jfoster/miniconda3/envs/rlgpu/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:412: UserWarning: In config: Invalid overriding of hydra/job_logging:
Default list overrides requires 'override' keyword.
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/defaults_list_override for more information.

  deprecation_warning(msg)
task: 
    name: FrankaCabinet
    physics_engine: physx
    env: 
        numEnvs: 4096
        envSpacing: 1.5
        episodeLength: 500
        enableDebugVis: False
        clipObservations: 5.0
        clipActions: 1.0
        startPositionNoise: 0.0
        startRotationNoise: 0.0
        numProps: 16
        aggregateMode: 3
        actionScale: 7.5
        dofVelocityScale: 0.1
        distRewardScale: 2.0
        rotRewardScale: 0.5
        aroundHandleRewardScale: 0.25
        openRewardScale: 7.5
        fingerDistRewardScale: 5.0
        actionPenaltyScale: 0.01
        asset: 
            assetRoot: ../../assets
            assetFileNameFranka: urdf/franka_description/robots/franka_panda.urdf
            assetFileNameCabinet: urdf/sektion_cabinet_model/urdf/sektion_cabinet_2.urdf
        enableCameraSensors: False
    sim: 
        dt: 0.0166
        substeps: 1
        up_axis: z
        use_gpu_pipeline: True
        gravity: [0.0, 0.0, -9.81]
        physx: 
            num_threads: 4
            solver_type: 1
            use_gpu: True
            num_position_iterations: 12
            num_velocity_iterations: 1
            contact_offset: 0.005
            rest_offset: 0.0
            bounce_threshold_velocity: 0.2
            max_depenetration_velocity: 1000.0
            default_buffer_size_multiplier: 5.0
            max_gpu_contact_pairs: 1048576
            num_subscenes: 4
            contact_collection: 0
    task: 
        randomize: False
train: 
    params: 
        seed: 42
        algo: 
            name: a2c_continuous
        model: 
            name: continuous_a2c_logstd
        network: 
            name: actor_critic
            separate: False
            space: 
                continuous: 
                    mu_activation: None
                    sigma_activation: None
                    mu_init: 
                        name: default
                    sigma_init: 
                        name: const_initializer
                        val: 0
                    fixed_sigma: True
            mlp: 
                units: [256, 128, 64]
                activation: elu
                d2rl: False
                initializer: 
                    name: default
                regularizer: 
                    name: None
        load_checkpoint: False
        load_path: 
        config: 
            name: FrankaCabinet
            full_experiment_name: FrankaCabinet
            env_name: rlgpu
            ppo: True
            mixed_precision: False
            normalize_input: True
            normalize_value: True
            num_actors: 4096
            reward_shaper: 
                scale_value: 0.01
            normalize_advantage: True
            gamma: 0.99
            tau: 0.95
            learning_rate: 0.0005
            lr_schedule: adaptive
            kl_threshold: 0.008
            score_to_win: 10000
            max_epochs: 1500
            save_best_after: 200
            save_frequency: 100
            print_stats: True
            grad_norm: 1.0
            entropy_coef: 0.0
            truncate_grads: True
            e_clip: 0.2
            horizon_length: 16
            minibatch_size: 8192
            mini_epochs: 8
            critic_coef: 4
            clip_value: True
            seq_len: 4
            bounds_loss_coef: 0.0001
task_name: FrankaCabinet
experiment: 
num_envs: 
seed: 42
torch_deterministic: False
max_iterations: 
physics_engine: physx
pipeline: gpu
sim_device: cuda:0
rl_device: cuda:0
graphics_device_id: 0
num_threads: 4
solver_type: 1
num_subscenes: 4
test: False
checkpoint: 
multi_gpu: False
headless: False
Setting seed: 42
Started to train
Exact experiment name requested from command line: FrankaCabinet
/home/jfoster/miniconda3/envs/rlgpu/lib/python3.7/site-packages/gym/spaces/box.py:74: UserWarning: WARN: Box bound precision lowered by casting to float32
  "Box bound precision lowered by casting to {}".format(self.dtype)
[Warning] [carb.gym.plugin] useGpu is set, forcing single scene (0 subscenes)
Not connected to PVD
+++ Using GPU PhysX
Physics Engine: PhysX
Physics Device: cuda:0
GPU Pipeline: enabled
[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/franka_description/meshes/collision/link0.obj 

[Error] [carb.gym.plugin] Could not load visual mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/franka_description/meshes/visual/link0.dae 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/franka_description/meshes/collision/link1.obj 

[Error] [carb.gym.plugin] Could not load visual mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/franka_description/meshes/visual/link1.dae 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/franka_description/meshes/collision/link2.obj 

[Error] [carb.gym.plugin] Could not load visual mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/franka_description/meshes/visual/link2.dae 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/franka_description/meshes/collision/link3.obj 

[Error] [carb.gym.plugin] Could not load visual mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/franka_description/meshes/visual/link3.dae 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/franka_description/meshes/collision/link4.obj 

[Error] [carb.gym.plugin] Could not load visual mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/franka_description/meshes/visual/link4.dae 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/franka_description/meshes/collision/link5.obj 

[Error] [carb.gym.plugin] Could not load visual mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/franka_description/meshes/visual/link5.dae 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/franka_description/meshes/collision/link6.obj 

[Error] [carb.gym.plugin] Could not load visual mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/franka_description/meshes/visual/link6.dae 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/franka_description/meshes/collision/link7.obj 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/franka_description/meshes/collision/hand.obj 

[Error] [carb.gym.plugin] Could not load visual mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/franka_description/meshes/visual/link7.dae 

[Error] [carb.gym.plugin] Could not load visual mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/franka_description/meshes/visual/hand.dae 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/franka_description/meshes/collision/finger.obj 

[Error] [carb.gym.plugin] Could not load visual mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/franka_description/meshes/visual/finger.dae 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/franka_description/meshes/collision/finger.obj 

[Error] [carb.gym.plugin] Could not load visual mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/franka_description/meshes/visual/finger.dae 

[Warning] [carb.gym.plugin] Unsupported shape encountered in addCollisionAttachementsToAsset
[Warning] [carb.gym.plugin] Unsupported shape encountered in addCollisionAttachementsToAsset
[Warning] [carb.gym.plugin] Unsupported shape encountered in addCollisionAttachementsToAsset
[Warning] [carb.gym.plugin] Unsupported shape encountered in addCollisionAttachementsToAsset
[Warning] [carb.gym.plugin] Unsupported shape encountered in addCollisionAttachementsToAsset
[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/sektion_convexdecomposition0.obj 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/sektion_convexdecomposition1.obj 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/sektion_convexdecomposition2.obj 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/sektion_convexdecomposition3.obj 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/sektion_convexdecomposition4.obj 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/sektion_convexdecomposition5.obj 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/sektion_convexdecomposition6.obj 

[Error] [carb.gym.plugin] Could not load visual mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/door_left.obj 

[Error] [carb.gym.plugin] Could not load visual mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/door_left_nob.obj 

[Error] [carb.gym.plugin] Could not load visual mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/door_right.obj 

[Error] [carb.gym.plugin] Could not load visual mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/door_right_nob.obj 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/drawer_convexdecomposition.obj 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/drawer_handle_convexdecomposition.obj 

[Error] [carb.gym.plugin] Could not load visual mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/drawer.obj 

[Error] [carb.gym.plugin] Could not load visual mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/drawer_handle.obj 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/drawer_convexdecomposition0.obj 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/drawer_convexdecomposition1.obj 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/drawer_convexdecomposition2.obj 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/drawer_convexdecomposition3.obj 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/drawer_convexdecomposition4.obj 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/drawer_convexdecomposition5.obj 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/drawer_convexdecomposition6.obj 

[Error] [carb.gym.plugin] Could not load collision mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/drawer_handle_convexdecomposition.obj 

[Error] [carb.gym.plugin] Could not load visual mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/drawer.obj 

[Error] [carb.gym.plugin] Could not load visual mesh /home/jfoster/isaacgym/python/IsaacGymEnvs/isaacgymenvs/tasks/../../assets/urdf/sektion_cabinet_model/meshes/drawer_handle.obj 

[Warning] [carb.gym.plugin] Unsupported shape encountered in addCollisionAttachementsToAsset
[Warning] [carb.gym.plugin] Unsupported shape encountered in addCollisionAttachementsToAsset
[Warning] [carb.gym.plugin] Unsupported shape encountered in addCollisionAttachementsToAsset
[Warning] [carb.gym.plugin] Unsupported shape encountered in addCollisionAttachementsToAsset
[Warning] [carb.gym.plugin] Unsupported shape encountered in addCollisionAttachementsToAsset
[Warning] [carb.gym.plugin] Unsupported shape encountered in addCollisionAttachementsToAsset
[Warning] [carb.gym.plugin] Unsupported shape encountered in addCollisionAttachementsToAsset
[Warning] [carb.gym.plugin] Unsupported shape encountered in addCollisionAttachementsToAsset
[Warning] [carb.gym.plugin] Unsupported shape encountered in addCollisionAttachementsToAsset
[Warning] [carb.gym.plugin] Unsupported shape encountered in addCollisionAttachementsToAsset
[Warning] [carb.gym.plugin] Unsupported shape encountered in addCollisionAttachementsToAsset
[Warning] [carb.gym.plugin] Unsupported shape encountered in addCollisionAttachementsToAsset
[Warning] [carb.gym.plugin] Unsupported shape encountered in addCollisionAttachementsToAsset
[Warning] [carb.gym.plugin] Unsupported shape encountered in addCollisionAttachementsToAsset
[Warning] [carb.gym.plugin] Unsupported shape encountered in addCollisionAttachementsToAsset
[Warning] [carb.gym.plugin] Unsupported shape encountered in addCollisionAttachementsToAsset
[Warning] [carb.gym.plugin] Unsupported shape encountered in addCollisionAttachementsToAsset
num franka bodies:  10
num franka dofs:  9
num cabinet bodies:  5
num cabinet dofs:  4
Segmentation fault (core dumped)

I’ve tried switching out NVIDIA graphics drivers to see if that was the issue, but no dice. I’ve also tried using vulkaninfo, but as far as I can tell the right graphics_device_id is being used, that is 0.

1 Like
2

I’m seeing the same thing. For what it’s worth, with those that work but don’t visualize, you can go to the “Viewer” tab, and turn on “Render collision meshes” to get a decent idea of what’s going on at least.

3

There may be something wrong with the file urdf in IsaacGymEnvs/assets. Many files in it are much smaller than them in preview 2.
I replaced the file IsaacGymEnvs/assets/urdf (preview 3) to isaacgym/assets/urdf (preview 2). Then it works well.

4

Hi @sizhe_wang @sheim and @james.paul.foster ,

Please try deleting and re-cloning the IsaacGymEnvs environment. Until last Friday, we had been using git LFS for some of the assets, which would explain the files being small - if your git did not properly get the LFS files, they would be placeholders.

On Friday we switched to regular git files for the assets. I’m hoping that this solves the problem.

Take care,
-Gav