Isaac gym stopped working

[user136160]

I am running Isaac Gym on my Lenovo P1 with a Quadro P2000 laptop GPU. It would run the examples at full speed and render everything properly. All of a sudden, nothing would run at full speed anymore and all of the text is unable to render, the ground plane does not render. The simulation can only run for a few seconds and then crashes. I didn’t change anything and I’m not sure how to debug. The output is the same as it was before. Here is an example of how Isaac Gym looks now, and example output (not from same run)

(rlgpu) dave@dave:~/isaacgym/python/examples$ python joint_monkey.py --asset_id 1
Importing module 'gym_37' (/home/dave/anaconda3/envs/rlgpu/lib/python3.7/site-packages/isaacgym/_bindings/linux-x86_64/gym_37.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/dave/anaconda3/envs/rlgpu/lib/python3.7/site-packages/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
WARNING: Forcing CPU pipeline.
Not connected to PVD
+++ Using GPU PhysX
Physics Engine: PhysX
Physics Device: cuda:0
GPU Pipeline: disabled
WARNING: lavapipe is not a conformant vulkan implementation, testing use only.
Loading asset 'mjcf/nv_ant.xml' from '../../assets'
DOF 0
  Name:     'hip_1'
  Type:     Rotation
  Stiffness:  0.0
  Damping:  0.1
  Armature:  0.01
  Limited?  True
    Lower   -0.698132
    Upper   0.698132
DOF 1
  Name:     'ankle_1'
  Type:     Rotation
  Stiffness:  0.0
  Damping:  0.1
  Armature:  0.01
  Limited?  True
    Lower   0.523599
    Upper   1.745329
DOF 2
  Name:     'hip_2'
  Type:     Rotation
  Stiffness:  0.0
  Damping:  0.1
  Armature:  0.01
  Limited?  True
    Lower   -0.698132
    Upper   0.698132
DOF 3
  Name:     'ankle_2'
  Type:     Rotation
  Stiffness:  0.0
  Damping:  0.1
  Armature:  0.01
  Limited?  True
    Lower   -1.745329
    Upper   -0.523599
DOF 4
  Name:     'hip_3'
  Type:     Rotation
  Stiffness:  0.0
  Damping:  0.1
  Armature:  0.01
  Limited?  True
    Lower   -0.698132
    Upper   0.698132
DOF 5
  Name:     'ankle_3'
  Type:     Rotation
  Stiffness:  0.0
  Damping:  0.1
  Armature:  0.01
  Limited?  True
    Lower   -1.745329
    Upper   -0.523599
DOF 6
  Name:     'hip_4'
  Type:     Rotation
  Stiffness:  0.0
  Damping:  0.1
  Armature:  0.01
  Limited?  True
    Lower   -0.698132
    Upper   0.698132
DOF 7
  Name:     'ankle_4'
  Type:     Rotation
  Stiffness:  0.0
  Damping:  0.1
  Armature:  0.01
  Limited?  True
    Lower   0.523599
    Upper   1.745329
Creating 2 environments
Animating DOF 0 ('hip_1')
Segmentation fault (core dumped)

[user136160]

So far I have tried:
-Running isaac gym on another video card (egpu), no effect
-Uninstalling using: conda remove --name rlgpu --all and deleting the isaac gym folder and installing from fresh - no effect

Any help would be greatly appreciated!

[user136160]

I didn’t realize that the the lavapipe error was obviously telling me that vulkan was not recognizing my video card anymore. This post helped make that more clear : Unhandled descriptor set followed by segmentation fault

It was confusing because nvidia-smi showed the processes running on my nvidia cards, and didn’t show any issues with the drivers. Running vulkaninfo showed the error though.

After troubleshooting that for a while, the solution was to fully uninstall and reinstall nvidia drivers and the problem went away. Isaac Gym is working properly now. I don’t know what caused the driver to become nonfunctional but if anyone else gets this issue, reinstalling nvidia drivers to fix vulkan worked for me.