Wsl2 and Isaac Gym problem

Hi! I’m actually find some problem running Isaac Gym. I got a nvidia 2070, windows 11 (so there is no problem running graphics application), but when I start an example In python i got:

*** Warning: failed to preload CUDA lib
*** Warning: failed to preload PhysX libs
Importing module ‘gym_38’ (/home/enne/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_38.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/enne/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
WARNING: Forcing CPU pipeline.
Not connected to PVD
/buildAgent/work/f3416cf82e3cf1ba/source/physx/src/gpu/PxPhysXGpuModuleLoader.cpp (147) : internal error : libcuda.so!

[Warning] [carb.gym.plugin] Failed to create a PhysX CUDA Context Manager. Falling back to CPU.
Physics Engine: PhysX
Physics Device: cpu
GPU Pipeline: disabled
No GPU devices found.
[Error] [carb.gym.plugin] Failed to create Nvf device in createNvfGraphics. Please make sure Vulkan is correctly installed.
*** Failed to create sim

If i run nvidia-smi in got correctly my graphic card, with a Driver Version: 510.10 and CUDA Version: 11.6.

Is there any incompatibility or some steps are missing?

2 Likes

I am having the same problem with almost exactly the same error message. However, at the end I get a segfault instead:

(rlgpu) dtch1997@DESKTOP-AR4R24K:~/isaacgym/python/examples$ python joint_monkey.py
*** Warning: failed to preload CUDA lib
*** Warning: failed to preload PhysX libs
Importing module ‘gym_37’ (/home/dtch1997/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_37.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/dtch1997/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
WARNING: Forcing CPU pipeline.
Not connected to PVD
/buildAgent/work/f3416cf82e3cf1ba/source/physx/src/gpu
/PxPhysXGpuModuleLoader.cpp (147) : internal error : libcuda.so!
[Warning] [carb.gym.plugin] Failed to create a PhysX CUDA Context Manager. Falling back to CPU.
Physics Engine: PhysX
Physics Device: cpu
GPU Pipeline: disabled
Segmentation fault

I am getting the same error:

python3 joint_monkey.py
Importing module ‘gym_37’ (/home/kaykay/Downloads/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_37.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/kaykay/Downloads/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
WARNING: Forcing CPU pipeline.
[Error] [carb.gym.plugin] Sim CUDA device 0 can’t be set, the total number of available devices is -1
Not connected to PVD
/buildAgent/work/f3416cf82e3cf1ba/source/cudamanager/src/CudaContextManager.cpp (404) : warning : cuInit failed

[Warning] [carb.gym.plugin] Failed to create a valid PhysX CUDA Context Manager. Falling back to CPU.
Physics Engine: PhysX
Physics Device: cpu
GPU Pipeline: disabled
[Error] [carb.gym.plugin] Gym cuda error: no CUDA-capable device is detected: …/…/…/source/plugins/carb/gym/impl/Gym/GymCuda.h: 110
[Error] [carb.gym.plugin] Failed to create primary CUDA context
[Warning] [carb.gym.plugin] Failed to create primary CUDA context on graphics device
No GPU devices found.
[Error] [carb.gym.plugin] Failed to create Nvf device in createNvfGraphics. Please make sure Vulkan is correctly installed.
*** Failed to create sim

nvidia-smi gives this:

nvidia-smi
Sun Oct 24 17:54:21 2021
±----------------------------------------------------------------------------+
| NVIDIA-SMI 510.00 Driver Version: 510.06 CUDA Version: 11.6 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce … On | 00000000:01:00.0 Off | N/A |
| N/A 54C P8 4W / N/A | 220MiB / 6144MiB | N/A Default |
| | | N/A |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
±----------------------------------------------------------------------------+

1 Like

Hi there! We currently only support Ubuntu 18.04 or 20.04 as mentioned in our docs.

Hello @kellyg - thanks for your update. Though I instead tried running up Ubuntu in a Virtual Machine and was still facing an issue reported here: Failed to acquire interface - #7 by kaykay

Can you please help me out with this? Let me know if I should make a new thread instead.

I was hoping that that would not be a problem with wsl2 (since it is basically a virtual machine, with GPU passthrough and GUI - in my case of Ubuntu 20.04). As far I understood there is some Vulkan problem in wsl2 and maybe this is the motivation of this problem

1 Like

We have not tested Isaac Gym in a virtual machine or wsl2 so I can’t say for sure what issues may arise. Vulkan is generally required for rendering, if your use case doesn’t require rendering, you could try running one of the examples in headless mode and see if that works for you. From the error messages posted in this thread, it mostly looks like the PhysX backend was not able to find the correct CUDA binaries, or it couldn’t find any available GPU devices, so it’s possible that these things are not being mapped correctly in the virtual environments.

I am getting the same error. I’m using Ubuntu 18.04 on an ec2 g4dn.2xlarge instance.

$ python3.8 joint_monkey.py
Importing module 'gym_38' (/home/ubuntu/.local/lib/python3.8/site-packages/isaacgym/_bindings/linux-x86_64/gym_38.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/ubuntu/.local/lib/python3.8/site-packages/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
WARNING: Forcing CPU pipeline.
[Error] [carb.gym.plugin] Sim CUDA device 0 can't be set, the total number of available devices is -1
Not connected to PVD
/buildAgent/work/45f70df4210b2e3e/source/cudamanager/src/CudaContextManager.cpp (404) : warning : cuInit failed

[Warning] [carb.gym.plugin] Failed to create a valid PhysX CUDA Context Manager. Falling back to CPU.
Physics Engine: PhysX
Physics Device: cpu
GPU Pipeline: disabled
[Error] [carb.gym.plugin] Gym cuda error: no CUDA-capable device is detected: ../../../source/plugins/carb/gym/impl/Gym/GymCuda.h: 110
[Error] [carb.gym.plugin] Failed to create primary CUDA context
[Warning] [carb.gym.plugin] Failed to create primary CUDA context on graphics device
$ nvidia-smi
Wed Apr  6 15:58:12 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.103.01   Driver Version: 470.103.01   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:1E.0 Off |                    0 |
| N/A   49C    P0    28W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

torch has access to cuda, so I’m not sure what’s going wrong

$ python3.8
Python 3.8.12 (default, Oct 12 2021, 13:49:34) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True

I’m using SSH with X11 forwarding.

Same here, any progress on this problem?

Another thread tracking this issue Virtual environment installations can't run joint_monkey.py · Issue #27 · NVIDIA-Omniverse/IsaacGymEnvs · GitHub

Edit: I’ve tested on both docker and nvidia-docker. Neither works.