[Error] [carb.gym.plugin] Gym cuda error: no kernel image is available for execution on the device: ../../../source/plugins/carb/gym/impl/Gym/GymPhysXCuda.cu: 937
[Error] [carb.gym.plugin] Failed to fill root state tensor
Actually, if I run something like
gym.fetch_results(sim, True)
gym.refresh_xxx_tensor()
gym.set_xxx_tensor()
with gpu pipeline, this error will occur. Other things like gym.acquire_xxx_tensor() seems ok. If I use cpu pipeline, then everything is ok. My pytorch works normally too.
Here’s my environment:
driver version: 495
cuda version: 11.5
pytorch version: 1.10.0+cu113
The interaction between gpu pytorch and isaac gym seems not going well. What should I do to make it work?
Are you able to run the GPU-supported examples or RL examples with the GPU pipeline? It sounds like it could either be a setup issue or possibly something to do with your code.
Then I run apply_forces.py and the same error occurs:
[Error] [carb.gym.plugin] Gym cuda error: no kernel image is available for execution on the device: ../../../source/plugins/carb/gym/impl/Gym/GymPhysXCuda.cu: 926
[Error] [carb.gym.plugin] Failed to fill rigid body state tensor
(isaac) sa@Fossa:~/wsp/isaac_3/isaacgym/python/examples$ python apply_forces.py --pipeline gpu
Importing module ‘gym_38’ (/home/sa/wsp/isaac_3/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_38.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/s/wsp/isaac_3/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
PyTorch version 1.8.2+cu111
Device count 1
/home/sa/wsp/isaac_3/isaacgym/python/isaacgym/_bindings/src/gymtorch
Using /home/sa/.cache/torch_extensions as PyTorch extensions root…
Emitting ninja build file /home/sa/.cache/torch_extensions/gymtorch/build.ninja…
Building extension module gymtorch…
Allowing ninja to set a default number of workers… (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module gymtorch…
Not connected to PVD
+++ Using GPU PhysX
Physics Engine: PhysX
Physics Device: cuda:0
GPU Pipeline: enabled
MESA-INTEL: warning: Performance support disabled, consider sysctl dev.i915.perf_stream_paranoid=0
No, I didn’t realize 1080_balls_of_solitude.py is cpu-only… Now it seems I run 1080_balls_of_solitude.py with cpu pipeline. I still can’t run apply_forces.py normally.
(btw, I didn’t modify any source code)
Your setup versions all look very recent. Try downgrading PyTorch to 1.8 or 1.9, and make sure you are able to run PyTorch on GPU without using Isaac Gym first. It could be a versioning issue between the driver, CUDA, and PyTorch.
Hi. I have the same problem. I can run CPU-supported examples well, but in apply_forces.py I have same error.
[Error] [carb.gym.plugin] Gym cuda error: no kernel image is available for execution on the device: ../../../source/plugins/carb/gym/impl/Gym/GymPhysXCuda.cu: 926
[Error] [carb.gym.plugin] Failed to fill rigid body state tensor
Hi @kellyg . I tried 1.8.0+cu111 version, and it doesn’t work😥. The program stuck into a time.sleep() endless loop (deadlock maybe?)
Importing module 'gym_37' (/home/user/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_37.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/user/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
PyTorch version 1.8.0+cu111
Device count 8
/home/user/isaacgym/python/isaacgym/_bindings/src/gymtorch
Using /home/user/.cache/torch_extensions as PyTorch extensions root...
^C
Traceback (most recent call last):
File "apply_forces.py", line 18, in <module>
from isaacgym import gymtorch
File "/home/user/isaacgym/python/isaacgym/gymtorch.py", line 109, in <module>
_import_gymtorch()
File "/home/user/isaacgym/python/isaacgym/gymtorch.py", line 45, in _import_gymtorch
gt = torch.utils.cpp_extension.load(name="gymtorch", sources=sources, extra_cflags=cflags, verbose=True)
File "/home/user/anaconda2/envs/isaacgym/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1091, in load
keep_intermediates=keep_intermediates)
File "/home/user/anaconda2/envs/isaacgym/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1308, in _jit_compile
baton.wait()
File "/home/user/anaconda2/envs/isaacgym/lib/python3.7/site-packages/torch/utils/file_baton.py", line 42, in wait
time.sleep(self.wait_seconds)
KeyboardInterrupt
When it runs to this line, it falls into an endless loop:
Files in /home/user/isaacgym/python/isaacgym/_bindings/src/gymtorch are source code, deleting the folder will lead to errors. I delete /home/user/.cache/torch_extensions and the binding problem is solved.
However, when I run apply_forces.py with gpu pipeline, the error still exists:
[Error] [carb.gym.plugin] Gym cuda error: no kernel image is available for execution on the device: ../../../source/plugins/carb/gym/impl/Gym/GymPhysXCuda.cu: 926
[Error] [carb.gym.plugin] Failed to fill rigid body state tensor
When I run it with cpu pipeline, it works normally. So I went back to where I was…
