Hi, I am getting following error when I try to train the Ant. I searched for the reason of the issue and It says that I need to downgrade the CUDA version to 11.1 to be able to train. I am not sure if it will work after I downgrade the CUDA version. That is why, I wanted to ask if anybody has/had similar problem? If yes, I would appreciate your support and/or comments.
FYI: I run it on Ubuntu 18.04, Driver Version is 470.86 and the CUDA Version is 11.4
Thanks in advance.
(py37) faruk@faruk-Blade:~/Desktop/IsaacGym_Preview_2_Package/isaacgym/python/rlgpu$ python rlg_train.py --task Ant
Importing module ‘gym_37’ (/home/faruk/Desktop/IsaacGym_Preview_2_Package/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_37.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/faruk/Desktop/IsaacGym_Preview_2_Package/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
PyTorch version 1.8.0+cu111
Device count 1
/home/faruk/Desktop/IsaacGym_Preview_2_Package/isaacgym/python/isaacgym/_bindings/src/gymtorch
Using /home/faruk/.cache/torch_extensions as PyTorch extensions root…
Emitting ninja build file /home/faruk/.cache/torch_extensions/gymtorch/build.ninja…
Building extension module gymtorch…
Allowing ninja to set a default number of workers… (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module gymtorch…
Importing module ‘rlgpu_37’ (/home/faruk/Desktop/IsaacGym_Preview_2_Package/isaacgym/python/isaacgym/_bindings/linux-x86_64/rlgpu_37.so)
Setting seed: 2555
Started to train
Python
Not connected to PVD
+++ Using GPU PhysX
Physics Engine: PhysX
Physics Device: cuda:0
GPU Pipeline: enabled
/home/faruk/anaconda3/envs/py37/lib/python3.7/site-packages/gym/spaces/box.py:74: UserWarning: WARN: Box bound precision lowered by casting to float32
“Box bound precision lowered by casting to {}”.format(self.dtype)
RL device: cuda:0
1024
8
60
0
Traceback (most recent call last):
File “rlg_train.py”, line 167, in
runner.run(vargs)
File “/home/faruk/anaconda3/envs/py37/lib/python3.7/site-packages/rl_games/torch_runner.py”, line 139, in run
self.run_train()
File “/home/faruk/anaconda3/envs/py37/lib/python3.7/site-packages/rl_games/torch_runner.py”, line 122, in run_train
agent = self.algo_factory.create(self.algo_name, base_name=‘run’, config=self.config)
File “/home/faruk/anaconda3/envs/py37/lib/python3.7/site-packages/rl_games/common/object_factory.py”, line 15, in create
return builder(**kwargs)
File “/home/faruk/anaconda3/envs/py37/lib/python3.7/site-packages/rl_games/torch_runner.py”, line 23, in
self.algo_factory.register_builder(‘a2c_continuous’, lambda **kwargs : a2c_continuous.A2CAgent(**kwargs))
File “/home/faruk/anaconda3/envs/py37/lib/python3.7/site-packages/rl_games/algos_torch/a2c_continuous.py”, line 18, in init
a2c_common.ContinuousA2CBase.init(self, base_name, config)
File “/home/faruk/anaconda3/envs/py37/lib/python3.7/site-packages/rl_games/common/a2c_common.py”, line 973, in init
A2CBase.init(self, base_name, config)
File “/home/faruk/anaconda3/envs/py37/lib/python3.7/site-packages/rl_games/common/a2c_common.py”, line 84, in init
self.vec_env = vecenv.create_vec_env(self.env_name, self.num_actors, **self.env_config)
File “/home/faruk/anaconda3/envs/py37/lib/python3.7/site-packages/rl_games/common/vecenv.py”, line 282, in create_vec_env
return vecenv_config[vec_env_name](config_name, num_actors, **kwargs)
File “rlg_train.py”, line 130, in
vecenv.register(‘RLGPU’, lambda config_name, num_actors, **kwargs: RLGPUEnv(config_name, num_actors, **kwargs))
File “rlg_train.py”, line 90, in init
self.full_state[“obs”] = self.env.reset()
File “/home/faruk/Desktop/IsaacGym_Preview_2_Package/isaacgym/python/rlgpu/tasks/base/vec_task.py”, line 137, in reset
self.task.step(actions)
File “/home/faruk/Desktop/IsaacGym_Preview_2_Package/isaacgym/python/rlgpu/tasks/base/base_task.py”, line 143, in step
self.post_physics_step()
File “/home/faruk/Desktop/IsaacGym_Preview_2_Package/isaacgym/python/rlgpu/tasks/ant.py”, line 266, in post_physics_step
self.reset(env_ids)
File “/home/faruk/Desktop/IsaacGym_Preview_2_Package/isaacgym/python/rlgpu/tasks/ant.py”, line 230, in reset
positions = torch_rand_float(-0.2, 0.2, (len(env_ids), self.num_dof), device=self.device)
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: nvrtc: error: failed to open libnvrtc-builtins.so.11.1.
Make sure that libnvrtc-builtins.so.11.1 is installed correctly.