Hi everyone,
I’m trying to run a reinforcement learning script on python on my newly installed GTX Titan X on Ubuntu 14.04 but I run into an error.
The script run just fine on my last card ( GTX 760 ), and after a lot of problem during the install of my titan X, and after the re-installation of Cuda I can’t seem to make it work.
I get this error:
nvcc fatal : Value ‘sm_52’ is not defined for option ‘gpu-architecture’
Any help would be greatly appreciated :)
Install CUDA 7 instead of whatever cuda version you are using.
I did the installation for Cuda 7, how could I check the version of my cuda?
Also, I’m wondering if that could be a probleme of installation of my new card. Do you know how can I check if the card is detected/ how to know if the drivers are correctly installed?
Thank you :)
ubuntu@ml:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Mon_Feb_16_22:59:02_CST_2015
Cuda compilation tools, release 7.0, V7.0.27
So Cuda 7 is already in use on my computer.
I also checked that I had the good drivers for my GPU:
ubuntu@ml:~$ nvidia-smi
Wed Jun 3 13:50:12 2015
±-----------------------------------------------------+
| NVIDIA-SMI 346.72 Driver Version: 346.72 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX TIT… Off | 0000:02:00.0 Off | N/A |
| 0% 50C P0 53W / 250W | 23MiB / 12287MiB | 0% Default |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
±----------------------------------------------------------------------------+
What bothers me is the “off” on the right of the name of my GPU…
I guess I’ll try to repost that on another part of the forum that is specialized on the GPU, thanks.
Probably then you have another version of CUDA also installed on that computer, and the script you are running is using that other version of CUDA. If you provide the full output from the script it may help.
Also you could try removing any other versions of CUDA besides CUDA 7. If you installed CUDA in the default location, then these other versions will be installed in /usr/local
If you do
ls /usr/local
it will give you an idea if other versions of CUDA are also installed, assuming they were installed in the default location(s).
This is what I get with the command line:
ubuntu@ml:~$ ls /usr/local
bin cuda cuda-7.0 etc games hdf5 include lib man sbin share src
with “cuda” being a repository with that inside:
ubuntu@ml:~$ ls /usr/local/cuda
bin extras lib64 libnvvp nvvm samples src tools
doc include libnsight LICENSE README share targets
So I guess there are no other Cuda version installed. Or could it be located in another part of my computer?
Thanks for your help :)
Then we will need to see the exact nvcc command line that produces this error message:
nvcc fatal : Value ‘sm_52’ is not defined for option ‘gpu-architecture’
also need to see the result of:
echo $PATH
I would also search your entire filesystem for any other copies of nvcc which may be floating around.
sudo find / -name nvcc
or something similar
Too many lines get displayed when I get the error so I can’t tell which line produces the error.
A lot of lines from the original code get written ( from line 250 to line 609) then there is that:
===============================
nvcc fatal : Value ‘sm_52’ is not defined for option ‘gpu-architecture’
[‘nvcc’, ‘-shared’, ‘-O3’, ‘-use_fast_math’, ‘-arch=sm_52’, ‘-m64’, ‘-Xcompiler’, ‘-fno-math-errno,-Wno-unused-label,-Wno-unused-variable,-Wno-write-strings,-DCUDA_NDARRAY_CUH=11b90075e2397c684f9dc0f7276eab8f,-D NPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION,-fPIC’, ‘-Xlinker’, ‘-rpath,/home/ubuntu/.theano/compiledir_Linux-3.16–generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/cuda_ndarray’, ‘-I/home/ubuntu/.theano/compiledir_Linux-3.16–generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/cuda_ndarray’, ‘-I/usr/include’, ‘-I/usr/lib/python2.7/dist-packages/numpy/core/include’, ‘-I/usr/include/python2.7’, ‘-I/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda’, ‘-o’, ‘/home/ubuntu/.theano/compiledir_Linux-3.16–generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpMleVcP/c8d7bd33dfef61705c2854dd1f0cb7ce.so’, ‘mod.cu’, ‘-L/home/ubuntu/.theano/compiledir_Linux-3.16–generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/cuda_ndarray’, ‘-L/usr/lib’, ‘-lpython2.7’, ‘-lcudart’, ‘-lcublas’, ‘-lcuda_ndarray’]
Traceback (most recent call last):
File “./rl_glue_ale_agent.py”, line 47, in
import cnn_q_learner
File “/home/ubuntu/Documents/deep_q_rl-master/deep_q_rl/cnn_q_learner.py”, line 11, in
import layers
File “/home/ubuntu/Documents/deep_q_rl-master/deep_q_rl/layers.py”, line 34, in
import theano.tensor as T
File “/usr/local/lib/python2.7/dist-packages/theano/init.py”, line 98, in
theano.sandbox.cuda.tests.test_driver.test_nvidia_driver1()
File “/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/tests/test_driver.py”, line 32, in test_nvidia_driver1
profile=False)
File “/usr/local/lib/python2.7/dist-packages/theano/compile/function.py”, line 266, in function
profile=profile)
File “/usr/local/lib/python2.7/dist-packages/theano/compile/pfunc.py”, line 511, in pfunc
on_unused_input=on_unused_input)
File “/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py”, line 1466, in orig_function
defaults)
File “/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py”, line 1324, in create
input_storage=input_storage_lists)
File “/usr/local/lib/python2.7/dist-packages/theano/gof/link.py”, line 519, in make_thunk
output_storage=output_storage)[:3]
File “/usr/local/lib/python2.7/dist-packages/theano/gof/vm.py”, line 897, in make_all
no_recycling))
File “/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/init.py”, line 259, in make_thunk
compute_map, no_recycling)
File “/usr/local/lib/python2.7/dist-packages/theano/gof/op.py”, line 739, in make_thunk
output_storage=node_output_storage)
File “/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py”, line 1073, in make_thunk
keep_lock=keep_lock)
File “/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py”, line 1015, in compile
keep_lock=keep_lock)
File “/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py”, line 1442, in cthunk_factory
key=key, lnk=self, keep_lock=keep_lock)
File “/usr/local/lib/python2.7/dist-packages/theano/gof/cmodule.py”, line 1076, in module_from_key
module = lnk.compile_cmodule(location)
File “/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py”, line 1354, in compile_cmodule
preargs=preargs)
File “/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/nvcc_compiler.py”, line 423, in compile_str
‘for cmd’, ’ '.join(cmd))
Exception: (‘The following error happened while compiling the node’, GpuCAReduce{add}{1}(<CudaNdarrayType(float32, vector)>), ‘\n’, ‘nvcc return status’, 255, ‘for cmd’, ‘nvcc -shared -O3 -use_fast_math -arch=sm_52 -m64 -Xcompiler -fno-math-errno,-Wno-unused-label,-Wno-unused-variable,-Wno-write-strings,-DCUDA_NDARRAY_CUH=11b90075e2397c684f9dc0f7276eab8f,-D NPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION,-fPIC -Xlinker -rpath,/home/ubuntu/.theano/compiledir_Linux-3.16–generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/cuda_ndarray -I/home/ubuntu/.theano/compiledir_Linux-3.16–generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/cuda_ndarray -I/usr/include -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -I/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda -o /home/ubuntu/.theano/compiledir_Linux-3.16–generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpMleVcP/c8d7bd33dfef61705c2854dd1f0cb7ce.so mod.cu -L/home/ubuntu/.theano/compiledir_Linux-3.16–generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/cuda_ndarray -L/usr/lib -lpython2.7 -lcudart -lcublas -lcuda_ndarray’, ‘[GpuCAReduce{add}{1}(<CudaNdarrayType(float32, vector)>)]’)
For echo $PATH:
ubuntu@ml:~$ echo $PATH
/usr/local/cuda-7.0/bin:/home/ubuntu/anaconda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
And for nvcc:
ubuntu@ml:~$ sudo find / -name nvcc
/usr/lib/nvidia-cuda-toolkit/bin/nvcc
/usr/local/cuda-7.0/bin/nvcc
/usr/bin/nvcc
Thanks
get rid of any trace of cuda apart from what is in /usr/local/cuda-7.0
by:
sudo rm -R -f /usr/lib/nvidia-cuda-toolkit
sudo rm /usr/bin/nvcc
(and /usr/local/cuda should be a symlink to /usr/local/cuda-7.0)
Alternatively, start with a clean load of ubuntu, then install the CUDA 7 toolkit and whatever else you need.
I just did what you said, and I don’t have the same error anymore.
But now I get:
ERROR (theano.sandbox.cuda): nvcc compiler not found on $PATH. Check your nvcc installation and try again.
Full error message:
Initializing ALE RL-Glue …
RL-Glue :: Environment connected.
ERROR (theano.sandbox.cuda): nvcc compiler not found on $PATH. Check your nvcc installation and try again.
RL-Glue Python Agent Codec Version: 2.1 (Build 738)
Connecting to 127.0.0.1 on port 4096…
Agent Codec Connected
RL-Glue :: Agent connected.
(32, 4, 84, 84)
(4, 84, 84, 32)
(16, 20.0, 20.0, 32)
(32, 9.0, 9.0, 32)
(32, 32, 9.0, 9.0)
(32, 256)
(32, 6)
Traceback (most recent call last):
File “./rl_glue_ale_agent.py”, line 454, in
main()
File “./rl_glue_ale_agent.py”, line 450, in main
AgentLoader.loadAgent(NeuralAgent())
File “/usr/local/lib/python2.7/dist-packages/rlglue/agent/AgentLoader.py”, line 58, in loadAgent
client.runAgentEventLoop()
File “/usr/local/lib/python2.7/dist-packages/rlglue/agent/ClientAgent.py”, line 144, in runAgentEventLoop
switchagentState
File “/usr/local/lib/python2.7/dist-packages/rlglue/agent/ClientAgent.py”, line 137, in
Network.kAgentInit: lambda self: self.onAgentInit(),
File “/usr/local/lib/python2.7/dist-packages/rlglue/agent/ClientAgent.py”, line 43, in onAgentInit
self.agent.agent_init(taskSpec)
File “./rl_glue_ale_agent.py”, line 167, in agent_init
self.network = self._init_network()
File “./rl_glue_ale_agent.py”, line 203, in _init_network
approximator=‘cuda_conv’)
File “/home/ubuntu/Documents/deep_q_rl-master/deep_q_rl/cnn_q_learner.py”, line 166, in init
q_vals = self.q_layers[-1].predictions()
File “/home/ubuntu/Documents/deep_q_rl-master/deep_q_rl/layers.py”, line 987, in predictions
return self.input_layer.output(*args, **kwargs)
File “/home/ubuntu/Documents/deep_q_rl-master/deep_q_rl/layers.py”, line 461, in output
input = self.input_layer.output(dropout_active=dropout_active, *args, **kwargs)
File “/home/ubuntu/Documents/deep_q_rl-master/deep_q_rl/layers.py”, line 461, in output
input = self.input_layer.output(dropout_active=dropout_active, *args, **kwargs)
File “/home/ubuntu/Documents/deep_q_rl-master/deep_q_rl/cc_layers.py”, line 258, in output
input = self.input_layer.output(*args, **kwargs)
File “/home/ubuntu/Documents/deep_q_rl-master/deep_q_rl/cc_layers.py”, line 112, in output
input = self.input_layer.output(dropout_active=dropout_active, *args, **kwargs)
File “/home/ubuntu/Documents/deep_q_rl-master/deep_q_rl/cc_layers.py”, line 127, in output
conved += self.b.dimshuffle(0, ‘x’, ‘x’, ‘x’)
File “/usr/local/lib/python2.7/dist-packages/theano/tensor/var.py”, line 128, in add
return theano.tensor.basic.add(self, other)
File “/usr/local/lib/python2.7/dist-packages/theano/gof/op.py”, line 507, in call
node = self.make_node(*inputs, **kwargs)
File “/usr/local/lib/python2.7/dist-packages/theano/tensor/elemwise.py”, line 527, in make_node
inputs = map(as_tensor_variable, inputs)
File “/usr/local/lib/python2.7/dist-packages/theano/tensor/basic.py”, line 145, in as_tensor_variable
return x._as_TensorVariable() # TODO: pass name and ndim arguments
File “/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/var.py”, line 30, in _as_TensorVariable
return HostFromGpu()(self)
NameError: global name ‘HostFromGpu’ is not defined
Then you have an improper CUDA install, or the PATH variable being picked up by the install script you are using is somehow different than the PATH variable you posted in this thread.
After removing the stuff I indicated, what happens if you now do:
which nvcc
?
which nvcc gets me:
ubuntu@ml:~$ which nvcc
/usr/local/cuda-7.0/bin/nvcc
Is the $PATH the same as sys.path in python?
If so, I just put a print sys.path at the begining of the script and that gave me :
[‘/home/ubuntu/Documents/deep_q_rl-master/deep_q_rl’, ‘/usr/local/lib/python2.7/dist-packages/Keras-0.0.1-py2.7.egg’, ‘/usr/lib/python2.7/dist-packages’, ‘/usr/local/lib/python2.7/dist-packages/PyYAML-3.11-py2.7-linux-x86_64.egg’, ‘/home/ubuntu/Documents/deep_q_rl-master/build/pylearn2’, ‘/usr/lib/python2.7’, ‘/usr/lib/python2.7/plat-x86_64-linux-gnu’, ‘/usr/lib/python2.7/lib-tk’, ‘/usr/lib/python2.7/lib-old’, ‘/usr/lib/python2.7/lib-dynload’, ‘/usr/local/lib/python2.7/dist-packages’, ‘/usr/lib/python2.7/dist-packages/PILcompat’, ‘/usr/lib/python2.7/dist-packages/gtk-2.0’, ‘/usr/lib/pymodules/python2.7’, ‘/usr/lib/python2.7/dist-packages/ubuntu-sso-client’, ‘/usr/lib/python2.7/dist-packages’]
Thank you
Certainly your cuda installation and nvcc appears to be discoverable and in the right place. Your PATH environment variable appears to be set correctly.
I can’t explain why that installer script thinks it is not. The script may have picked up a previous version of your PATH variable. Does it have a configuration process associated with it that perhaps needs to be re-run?
Could it be that I’m running the script with sudo?
I’ve just seen the same error caused by that on internet: apparently nvcc wasn’t on the $PATH when using sudo.
I started using sudo for the script because otherwise I get:
IOError: [Errno 2] No such file or directory: u’/tmp/matplotlib-ubuntu/fontList.cache’
I can’t explain why that installer script thinks it is not. The script may have picked up a previous version of your PATH variable. Does it have a configuration process associated with it that perhaps needs to be re-run?
If you are talking about the python script I’m using, it didn’t came from an installer.
Yes, in that case, the program is picking up the contents of the PATH variable when you are root.
So you will need to set that to include the /usr/local/cuda/bin directory.
what method are you using to set your PATH variable now? i.e. how did /usr/local/cuda-7.0/bin get added to it?
You would need to do something similar as root
There were apparently other problems associated with running the script as sudo, so I tried and managed to solve the problem by creating the /matplotlib-ubuntu/ folder in the /tmp/ one.
The script seems to go further now (take way more time and "Using gpu device 0: GeForce GTX TITAN X
" get displayed), BUT I now have another error ^^ (seems infinite)
ImportError: (‘The following error happened while compiling the node’, GpuCAReduce{add}{1}(<CudaNdarrayType(float32, vector)>), ‘\n’, ‘libcudart.so.7.0: cannot open shared object file: No such file or directory’, ‘[GpuCAReduce{add}{1}(<CudaNdarrayType(float32, vector)>)]’)
The whole error is:
Using gpu device 0: GeForce GTX TITAN X
Traceback (most recent call last):
File “./rl_glue_ale_agent.py”, line 47, in
import cnn_q_learner
File “/home/ubuntu/Documents/deep_q_rl-master/deep_q_rl/cnn_q_learner.py”, line 11, in
import layers
File “/home/ubuntu/Documents/deep_q_rl-master/deep_q_rl/layers.py”, line 34, in
import theano.tensor as T
File “/home/ubuntu/anaconda/lib/python2.7/site-packages/theano/init.py”, line 92, in
theano.sandbox.cuda.tests.test_driver.test_nvidia_driver1()
File “/home/ubuntu/anaconda/lib/python2.7/site-packages/theano/sandbox/cuda/tests/test_driver.py”, line 28, in test_nvidia_driver1
profile=False)
File “/home/ubuntu/anaconda/lib/python2.7/site-packages/theano/compile/function.py”, line 223, in function
profile=profile)
File “/home/ubuntu/anaconda/lib/python2.7/site-packages/theano/compile/pfunc.py”, line 512, in pfunc
on_unused_input=on_unused_input)
File “/home/ubuntu/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py”, line 1312, in orig_function
defaults)
File “/home/ubuntu/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py”, line 1181, in create
_fn, _i, _o = self.linker.make_thunk(input_storage=input_storage_lists)
File “/home/ubuntu/anaconda/lib/python2.7/site-packages/theano/gof/link.py”, line 434, in make_thunk
output_storage=output_storage)[:3]
File “/home/ubuntu/anaconda/lib/python2.7/site-packages/theano/gof/vm.py”, line 847, in make_all
no_recycling))
File “/home/ubuntu/anaconda/lib/python2.7/site-packages/theano/sandbox/cuda/init.py”, line 237, in make_thunk
compute_map, no_recycling)
File “/home/ubuntu/anaconda/lib/python2.7/site-packages/theano/gof/op.py”, line 606, in make_thunk
output_storage=node_output_storage)
File “/home/ubuntu/anaconda/lib/python2.7/site-packages/theano/gof/cc.py”, line 948, in make_thunk
keep_lock=keep_lock)
File “/home/ubuntu/anaconda/lib/python2.7/site-packages/theano/gof/cc.py”, line 891, in compile
keep_lock=keep_lock)
File “/home/ubuntu/anaconda/lib/python2.7/site-packages/theano/gof/cc.py”, line 1322, in cthunk_factory
key=key, fn=self.compile_cmodule_by_step, keep_lock=keep_lock)
File “/home/ubuntu/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py”, line 996, in module_from_key
module = next(compile_steps)
File “/home/ubuntu/anaconda/lib/python2.7/site-packages/theano/gof/cc.py”, line 1237, in compile_cmodule_by_step
preargs=preargs)
File “/home/ubuntu/anaconda/lib/python2.7/site-packages/theano/sandbox/cuda/nvcc_compiler.py”, line 444, in compile_str
return dlimport(lib_filename)
File “/home/ubuntu/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py”, line 284, in dlimport
rval = import(module_name, {}, {}, [module_name])
ImportError: (‘The following error happened while compiling the node’, GpuCAReduce{add}{1}(<CudaNdarrayType(float32, vector)>), ‘\n’, ‘libcudart.so.7.0: cannot open shared object file: No such file or directory’, ‘[GpuCAReduce{add}{1}(<CudaNdarrayType(float32, vector)>)]’)
Thank you
I’m not that familiar with what your python script is doing. Unfortunately, if it is calling nvcc in these cases, it is hiding the actual nvcc invocation it is using. However the dlimport command makes me suspicious that maybe your LD_LIBRARY_PATH environment variable is coming into play.
what is the result of running:
echo $LD_LIBRARY_PATH
this needs to be done in the context of whatever use the script is running in.
I get nothing when running echo $LD_LIBRARY_PATH
Just a blank line. May that be the problem?
It might be the problem. Again, I’m not entirely sure what the script is doing. Normally this environment variable has nothing to do with compiling codes, but actually with executing codes (that depend on dynamically loaded shared objects).
try:
export LD_LIBRARY_PATH=/usr/local/cuda-7.0/lib64
then run your script. Again, this needs to take effect in whatever user context the script will be running in.
I solved the problem doing:
-installing bleeding-edge version of Theano instead of the normal version
-doing export LD_LIBRARY_PATH=/usr/local/cuda/lib64
-installing ATLAS instead of BLAS : sudo apt-get install libatlas3gf-base libatlas-base-dev
-changing my .theanord file to:
[global]
floatX=float32
device = gpu0
[nvcc]
fastmath = True
[mode]=FAST_RUN
[cuda]
root=/usr/local/cuda-7.0/targets/x86_64-linux/lib/
[blas]
ldflags =-lf77blas -latlas -lgfortran
Thank you very much txbob for your support :)