Isaac Gym - Training Ant (Error: Failed to open libnvrtc-builtins.so.11.1)

Hi, I am getting following error when I try to train the Ant. I searched for the reason of the issue and It says that I need to downgrade the CUDA version to 11.1 to be able to train. I am not sure if it will work after I downgrade the CUDA version. That is why, I wanted to ask if anybody has/had similar problem? If yes, I would appreciate your support and/or comments.

FYI: I run it on Ubuntu 18.04, Driver Version is 470.86 and the CUDA Version is 11.4

Thanks in advance.

(py37) faruk@faruk-Blade:~/Desktop/IsaacGym_Preview_2_Package/isaacgym/python/rlgpu$ python rlg_train.py --task Ant

Importing module ‘gym_37’ (/home/faruk/Desktop/IsaacGym_Preview_2_Package/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_37.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/faruk/Desktop/IsaacGym_Preview_2_Package/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
PyTorch version 1.8.0+cu111
Device count 1
/home/faruk/Desktop/IsaacGym_Preview_2_Package/isaacgym/python/isaacgym/_bindings/src/gymtorch
Using /home/faruk/.cache/torch_extensions as PyTorch extensions root…
Emitting ninja build file /home/faruk/.cache/torch_extensions/gymtorch/build.ninja…
Building extension module gymtorch…
Allowing ninja to set a default number of workers… (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module gymtorch…
Importing module ‘rlgpu_37’ (/home/faruk/Desktop/IsaacGym_Preview_2_Package/isaacgym/python/isaacgym/_bindings/linux-x86_64/rlgpu_37.so)
Setting seed: 2555
Started to train
Python
Not connected to PVD
+++ Using GPU PhysX
Physics Engine: PhysX
Physics Device: cuda:0
GPU Pipeline: enabled
/home/faruk/anaconda3/envs/py37/lib/python3.7/site-packages/gym/spaces/box.py:74: UserWarning: WARN: Box bound precision lowered by casting to float32
“Box bound precision lowered by casting to {}”.format(self.dtype)
RL device: cuda:0
1024
8
60
0
Traceback (most recent call last):
File “rlg_train.py”, line 167, in
runner.run(vargs)
File “/home/faruk/anaconda3/envs/py37/lib/python3.7/site-packages/rl_games/torch_runner.py”, line 139, in run
self.run_train()
File “/home/faruk/anaconda3/envs/py37/lib/python3.7/site-packages/rl_games/torch_runner.py”, line 122, in run_train
agent = self.algo_factory.create(self.algo_name, base_name=‘run’, config=self.config)
File “/home/faruk/anaconda3/envs/py37/lib/python3.7/site-packages/rl_games/common/object_factory.py”, line 15, in create
return builder(**kwargs)
File “/home/faruk/anaconda3/envs/py37/lib/python3.7/site-packages/rl_games/torch_runner.py”, line 23, in
self.algo_factory.register_builder(‘a2c_continuous’, lambda **kwargs : a2c_continuous.A2CAgent(**kwargs))
File “/home/faruk/anaconda3/envs/py37/lib/python3.7/site-packages/rl_games/algos_torch/a2c_continuous.py”, line 18, in init
a2c_common.ContinuousA2CBase.init(self, base_name, config)
File “/home/faruk/anaconda3/envs/py37/lib/python3.7/site-packages/rl_games/common/a2c_common.py”, line 973, in init
A2CBase.init(self, base_name, config)
File “/home/faruk/anaconda3/envs/py37/lib/python3.7/site-packages/rl_games/common/a2c_common.py”, line 84, in init
self.vec_env = vecenv.create_vec_env(self.env_name, self.num_actors, **self.env_config)
File “/home/faruk/anaconda3/envs/py37/lib/python3.7/site-packages/rl_games/common/vecenv.py”, line 282, in create_vec_env
return vecenv_config[vec_env_name](config_name, num_actors, **kwargs)
File “rlg_train.py”, line 130, in
vecenv.register(‘RLGPU’, lambda config_name, num_actors, **kwargs: RLGPUEnv(config_name, num_actors, **kwargs))
File “rlg_train.py”, line 90, in init
self.full_state[“obs”] = self.env.reset()
File “/home/faruk/Desktop/IsaacGym_Preview_2_Package/isaacgym/python/rlgpu/tasks/base/vec_task.py”, line 137, in reset
self.task.step(actions)
File “/home/faruk/Desktop/IsaacGym_Preview_2_Package/isaacgym/python/rlgpu/tasks/base/base_task.py”, line 143, in step
self.post_physics_step()
File “/home/faruk/Desktop/IsaacGym_Preview_2_Package/isaacgym/python/rlgpu/tasks/ant.py”, line 266, in post_physics_step
self.reset(env_ids)
File “/home/faruk/Desktop/IsaacGym_Preview_2_Package/isaacgym/python/rlgpu/tasks/ant.py”, line 230, in reset
positions = torch_rand_float(-0.2, 0.2, (len(env_ids), self.num_dof), device=self.device)
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: nvrtc: error: failed to open libnvrtc-builtins.so.11.1.
Make sure that libnvrtc-builtins.so.11.1 is installed correctly.

Probably you might need to reinstall the Pytorch as well. The one which is compatible with Cuda 11.4 and Ubuntu 18.04

I removed the driver & CUDA and then installed the CUDA 11.1. So, I can run the trainings now. Actually, there is no need to reinstall the PyTorch in that case.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.