[CUDA/CUPY] nvrtc error: failed to load builtins

Hi everyone,

I am very new to cuda and quite new to Linux. For research purpose I need to install cuda and cupy on a Dell Workstation equipped with a Nvidia Quadro K2000 GPU, with Ubuntu 16.04 LTS.

I am using:

username@Precision-T5600:~$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module  384.90  Tue Sep 19 19:17:35 PDT 2017
GCC version:  gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.5) 
username@Precision-T5600:~$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176

At first, it seems everything ran correctly. Both tests mentioned in http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#post-installation-actions were passed nicely.

Regarding chainer and cupy installation, it was a bit harder but everything seemed ok once I change from Python conda 3.6.2 to 3.6.3.

At this stage, I have:

username@Precision-T5600:~$ python
Python 3.6.3 |Anaconda custom (64-bit)| (default, Nov 20 2017, 20:41:42) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import chainer
>>> chainer.__version__
'3.1.0'
>>> chainer.cuda.available
True
>>> import cupy as cp
>>> a = cp.array([1,2,3])
>>> b = cp.array([2,3,5])
>>> a.device
<CUDA Device 0>

So at this point, everything seems ok, but when I try to perform a “mathematical” operation (numpy-like), I have the following error:

>>> a + b
Traceback (most recent call last):
  File "/home/username/anaconda3/lib/python3.6/site-packages/cupy/cuda/compiler.py", line 229, in compile
    nvrtc.compileProgram(self.ptr, options)
  File "cupy/cuda/nvrtc.pyx", line 98, in cupy.cuda.nvrtc.compileProgram
  File "cupy/cuda/nvrtc.pyx", line 108, in cupy.cuda.nvrtc.compileProgram
  File "cupy/cuda/nvrtc.pyx", line 53, in cupy.cuda.nvrtc.check_status
cupy.cuda.nvrtc.NVRTCError: NVRTC_ERROR unknown (7)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "cupy/core/core.pyx", line 1179, in cupy.core.core.ndarray.__add__
  File "cupy/core/elementwise.pxi", line 817, in cupy.core.core.ufunc.__call__
  File "cupy/util.pyx", line 39, in cupy.util.memoize.decorator.ret
  File "cupy/core/elementwise.pxi", line 620, in cupy.core.core._get_ufunc_kernel
  File "cupy/core/elementwise.pxi", line 33, in cupy.core.core._get_simple_elementwise_kernel
  File "cupy/core/carray.pxi", line 170, in cupy.core.core.compile_with_cache
  File "/home/username/anaconda3/lib/python3.6/site-packages/cupy/cuda/compiler.py", line 123, in compile_with_cache
    base = _preprocess('', options, arch)
  File "/home/username/anaconda3/lib/python3.6/site-packages/cupy/cuda/compiler.py", line 86, in _preprocess
    result = prog.compile(options)
  File "/home/username/anaconda3/lib/python3.6/site-packages/cupy/cuda/compiler.py", line 233, in compile
    raise CompileException(log, self.src, self.name, options)
cupy.cuda.compiler.CompileException: nvrtc: error: failed to load builtins

I have found only one other post with same issues, https://devtalk.nvidia.com/default/topic/920047/nvrtc-error-failed-to-load-builtins-on-linux/#, but I did not understand the answers (or, I mean, how to use it in my situation).
I may have trouble regarding PATH variable, or gcc, or other conflicts… I don’t even know if this is more a Cupy or a Cuda problem.

Any help is much appreciated,

Sincerely,

Same problem with you.
Dell alienware, ubuntu 18.04 Gtx1060m
cannot do any math calculation

I meet the same errors as you.

my linux has installed cuda8.0, pytorch 1.2.0,

I solve this error by making the PATH and LD_LIBRARY_PATH consistent with the version of cuda in /usr/local/cuda/version.txt.