Gym cuda error: no kernel image is available for execution on the device, pt. 2

Hey, so this problem persists. It’s already been mentioned over in another thread but without solution, so I wanted to give a clear rundown of what I’ve tried. (Related thread: Gym cuda error: no kernel image is available for execution on the device)

Environment

  • Ubuntu 18.04.6 LTS
  • Miniconda (running Python 3.8.12)
  • Hardware: 2x GeForce GTX TITAN X, 1x GeForce GTX 750
  • Driver: 510.47.03
  • Cuda: 11.3

The problem
When I run isaacgym/python/examples/apply_forces.py, I get the following errors:

[Error] [carb.gym.plugin] Gym cuda error: no kernel image is available for execution on the device: ../../../source/plugins/carb/gym/impl/Gym/GymPhysXCuda.cu: 926
[Error] [carb.gym.plugin] Failed to fill rigid body state tensor

What I’ve tried

  • Downgrading Cuda (I was running Cuda 11.5 before). Downgraded to 11.3. No change.
  • Removing the cached build folder under ~/.cache/torch_extensions/py38_cu113/gymtorch/ and rebuilding the extension. No change.
  • Creating a new Conda environment with only python=3.8, torch=1.10.0+cu113 and isaacgym, then removing the extension folder to force rebuilding. No change.
  • Compiling pytorch from source (took about 2 hours), in a brand-new Conda environment, then removing the extension folder to force rebuilding. No change.

EDIT: The following is running fine on my other machine: Ubuntu 20.04.3, Nvidia RTX 2080 Ti, same pytorch version (1.10.3+cu113), driver 495.29.05, Cuda 11.5.

Is there anything else I can try?
Any help would be appreciated. My next approach would be to nuke the OS and install Ubuntu 20.04 LTS but I was hoping to maybe not have to do that just for IsaacGym.

Cheers,
Flo

seems the software won’t support GPU cards without RTX cores

My friend is running it fine on a GTX 1080 Ti card, so maybe that’s not the issue? Maybe it’s the compute capability? Titan X has cc 5.2 and GTX 1080 and anything newer has cc 6.1 (source: CUDA GPUs | NVIDIA Developer)

1 Like

good point
However, I would try with cuda 10.2
technical requirements for Gym seem underdocumented :
I was not able to find exact GPU hardware/ Cuda version list supported
But e.g. in Jetson ecosystem Isaac Sim only supports cuda 10.2
However, I can see a similar issue here

maybe SM number issue as in Cuda error: no kernel image is available for execution on the device · Issue #31285 · pytorch/pytorch · GitHub

@fgolemo
It appears it also may be a pytorch version issue that doesn’t support specific SM
you may try pytorch docker container maybe
or build pytorch from source
or you may try this

pip install torch==1.8.1

from gym forum thread Error in Reinforcement Learning example - #2 by toni.sm

check also this thread

Hi @fgolemo,

The error above is in the Gym PhysX CUDA code, rather than PyTorch - I’m afraid that we only support Pascal GPUs and higher. Your TITAN X and GTX 750 cards are both Maxwell class cards.

Sorry,
-Gav

@gstate is there a system requirements somewhere that lists all supported versions of GPU/Software?etc?

@_av - there is a short requirements list on the download page, but it looks like we don’t include required GPU version. Only software needs are provided. I will request an update to that area.

Sorry,
-Gav

We’ve now updated this, and the minimum requirement of a Pascal GPU is now listed in the “Prerequisites” you see when downloading.

Thanks,
-Gav