sample mnistCUDNN failed with 'Cublas failure gemv.h:77' Ubuntu 18.04.3 Cuda10.1 2080S CUDNN 7.6.5

jdewitt · January 5, 2020, 10:38pm

Hi All,
I notice that other people are having a similar problem, but the proposed solutions did not help me. If someone could review my configuration and comment, I’d appreciate. Aside from mnistCUDNN, the other CUDNN samples seem to work, as well as the CUDA-10.1 samples. Thanks, /jd

Ubuntu 18.04.3 LTS

nvidia-smi
NVIDIA-SMI 435.21 Driver Version: 435.21 CUDA Version: 10.1

mnistCUDNN
cudnnGetVersion(): 7605 , CUDNN_VERSION from cudnn.h : 7605 (7.6.5)
Host compiler version : GCC 7.4.0
There are 1 CUDA capable devices on your machine :
device 0 : sms 48 Capabilities 7.5
using device 0
Testing single precision
Loading image data/one_28x28.pgm
Performing forward propagation …
Testing cudnnConvolutionForwardAlgorithm …
Fastest algorithm is Algo 0
Testing cudnnFindConvolutionForwardAlgorithm …
^^^ CUDNN_STATUS_SUCCESS for Algo 0: 0.010240 time requiring 0 memory
^^^ CUDNN_STATUS_SUCCESS for Algo 2: 0.026688 tim requiring 57600 memory
^^^ CUDNN_STATUS_SUCCESS for Algo 1: 0.027552 time requiring 3464 memory
^^^ CUDNN_STATUS_SUCCESS for Algo 5: 0.045056 time requiring 203008 memory
^^^ CUDNN_STATUS_SUCCESS for Algo 7: 0.053024 time requiring 205774 memory
Cublas failure
Error code 0
gemv.h:77
aborting…
api.log (41.4 KB)

jdewitt · January 5, 2020, 11:11pm

See this topic for several other people having the same problem with very similar configuration as mine. The topic was marked resolved with the suggestion of moving from CUDA 9 to CUDA 10. Since I am already running CUDA 10, this solution doesn’t solve my problem. Thanks! /jd

https://devtalk.nvidia.com/default/topic/1042638/is-nvidia-driver-410-57-incompatible-with-cuda-9-0/?offset=6#5289184

jdewitt · January 5, 2020, 11:32pm

Should I install CUDA 10.2 and the corresponding CUDNN 7.6.5 (the download page implies that there is a 10.1 version and a 10.2 version of 7.6.5) ? If I do this, I think I will need to update my display driver to 440, from my current 435.21. 435.21 is what my Ubuntu distro recommends, through the activities->software_update->additional_drivers menu (which does not offer 440). Installing a new display driver will make me cry, as I know this will cause 10 new things to go wrong and might not even solve my problem at hand. Thanks! /jd

jdewitt · January 5, 2020, 11:51pm

OK, poured myself a stiff drink and tried updating my display driver to 440. nvidia-smi says I am now running 440.44, and CUDA version 10.2. One thing that amazes me is that my screen didn’t go black and I didn’t have to re-install Ubuntu, for the first time ever when updating display driver. One thing that surprises me is that the CUDA version spontaneously updated from 10.1 to 10.2, even though I didn’t do a CUDA update. Is this expected?

Anyway, my problem changed but did not go away. Now I get gemv.h:81 rather than gemv.h:77 before the abort. Am I making progress? Is this what computer science is like? Sigh… I think it’s time for me to go for a walk in the woods and consider taking up knitting as a hobby rather than massively parallel computer programming.

jdewitt · January 6, 2020, 12:10am

Just for grins, you might say, I installed the CUDA-10.2 version of CUDNN 7.6.5 now that I seem to have installed CUDA 10.2. mnistCUDNN still aborts after the Algo 7 status line, but now the gemv.h return code has switched back to 77 from 81. So I’m back where I started, although I think I have the latest versions of everything:

CUDNN 7.6.5 CUDA 10.2 verstion
Display Driver 440.44
CUDA 10.2

SunilJB · January 6, 2020, 5:53pm

Hi,

Could you please share info regarding the GPU model used in this case and if possible API logs as well?
Setting API logging:
https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html#api-logging

Thanks

jdewitt · January 7, 2020, 6:59am

I have the Nvidia branded GEFORCE RTX 2080 Super, it lights up green. ‘Inspired by Gamers, Built by Nvidia’. How do I send you the log file? I don’t see a method to attach files to posts in this forum. It’s 871 lines long. I think the function checkCublasError called on line 71 of gemv.h is throwing the error. When I comment it out, the program runs farther but reports as below. I don’t see checkCublasError in the api log whether or not it is commented out. thnx, /jd

Resulting weights from Softmax:
0.0992543 0.0980544 0.1002826 0.0994706 0.1003590 0.1006492 0.0989232 0.0995440 0.1034657 0.0999969
Loading image data/three_28x28.pgm
Performing forward propagation …
Resulting weights from Softmax:
0.0992543 0.0980544 0.1002826 0.0994706 0.1003590 0.1006492 0.0989232 0.0995440 0.1034657 0.0999969
Loading image data/five_28x28.pgm
Performing forward propagation …
Resulting weights from Softmax:
0.0992543 0.0980544 0.1002826 0.0994706 0.1003590 0.1006492 0.0989232 0.0995440 0.1034657 0.0999969

Result of classification: 8 8 8

Test failed!
Prediction mismatch
mnistCUDNN.cpp:876
Aborting…

jdewitt · January 7, 2020, 7:19am

I got smart enough for a moment to notice the paperclip at the top of this topic. Please find attached file api.log in my first post, that does seem to open as text.

SunilJB · January 9, 2020, 4:15am

Hi,

We tried with the same versions of OS/Driver/CUDA Toolkit, but we’re unable to reproduce this issue.

It’s seems a setup issue or mis-configuration issue. I recommend to try it again with clean installation:
e.g.
(1): Re-install the CUDA Toolkit (e.g. CUDA 10.2) and cudnn-7.6.5 GA CUDA 10.2 based build.
(2): Re-build “mnistCUDNN” sample ; (e.g. $make clean ; $make or $make SMS=75 <=> in case Makefile doesn’t have “sm_75” architecture support for some reason.)

If this issue persists, please provide the following output information:

$ldd mnistCUDNN  
$dpkg -l | grep  -i cudnn
$dpkg -l | grep  -i cuda

Thanks

jdewitt · January 10, 2020, 5:29am

Excelsior! Thanks for the quick response, Sunil. Now I can make my DNNs much more convoluted.

I re-installed everything yet again. I think this time I saved the deb files and sudo dpkg’ed them rather than executing them from the the cudnn Download page. I don’t know if that made any difference.

sudo sh cuda_10.2.89_440.33.01_linux.run
/usr/local/cuda → /usr/local/cuda-10.2/
checked that Mandelbrot works
sudo dpkg -i libcudnn7_7.6.5.32-1+cuda10.2_amd64.deb
sudo dpkg -i libcudnn7-dev_7.6.5.32-1+cuda10.2_amd64.debsudo dpkg -i libcudnn7-doc_7.6.5.32-1+cuda10.2_amd64.deb

A few cp’s that the interwebs suggested. The first seemed to be necessary as mnistCUDNN wouldn’t make beforehand. The second I’m not sure about.
sudo cp /usr/local/cuda-10.2/targets/x86_64-linux/include/cudnn.h /usr/local/cuda/include/
sudo cp /usr/local/cuda-10.2/lib64/libcudnn* /usr/local/cuda/lib64
sudo chmod a+r /usr/local/cuda/include/cudnn.h /usr/local/cuda/lib64/libcudnn*

A couple of links suggested from a different topic on this forum:
sudo ln -s /usr/lib/x86_64-linux-gnu/libcublasLt.so.10.2.1.243 /usr/local/cuda/lib64/libcublasLt.so
sudo ln -s /usr/lib/x86_64-linux-gnu/libcublas.so.10.2.1.243 /usr/local/cuda/lib64/libcublas.so

A few additions to my .bashrc, which I’m not sure were necessary as the Makefile seems to handle this
export CUDA_HOME=/usr/local/cuda
export PATH=/usr/local/cuda/bin:$PATH
export CPATH=/usr/local/cuda/include:$CPATH
export LIBRARY_PATH=/usr/local/cuda/lib64:$LIBRARY_PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH

After working through this and a reboot, all the cudnn_10.2_samples_v7 samples compile and claim to pass.

Re-install, the universal solution to all software problems!

ga74kud · March 12, 2020, 7:27pm

ldd mnistCUDNN
linux-vdso.so.1 (0x00007ffd98bb2000)
libcudart.so.10.2 => /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudart.so.10.2 (0x00007fab2babc000)
libcublas.so.9.1 => /usr/lib/x86_64-linux-gnu/libcublas.so.9.1 (0x00007fab28525000)
libcudnn.so.7 => /usr/local/cuda-10.0/targets/x86_64-linux/lib/libcudnn.so.7 (0x00007fab104d0000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fab10147000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fab0ff28000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fab0fb8a000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fab0f972000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fab0f581000)
/lib64/ld-linux-x86-64.so.2 (0x00007fab2c19e000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fab0f37d000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fab0f175000)

dpkg -l | grep -i cudnn
ii libcudnn7 7.5.1.10-1+cuda10.1 amd64 cuDNN runtime libraries
ii libcudnn7-dev 7.5.1.10-1+cuda10.1 amd64 cuDNN development libraries and headers
ii libcudnn7-doc 7.5.1.10-1+cuda10.1 amd64 cuDNN documents and samples

dpkg -l | grep -i cuda
ii cuda ii cuda-10-0 ii cuda-10-2 ii cuda-command-line-tools-10-0 ii cuda-command-line-tools-10-2 ii cuda-compiler-10-0 ii cuda-compiler-10-2 ii cuda-cublas-10-0 ii cuda-cublas-dev-10-0 ii cuda-cudart-10-0 rc cuda-cudart-10-1 ii cuda-cudart-10-2 ii cuda-cudart-dev-10-0 rc cuda-cudart-dev-10-1 ii cuda-cudart-dev-10-2 ii cuda-cufft-10-0 rc cuda-cufft-10-1 ii cuda-cufft-10-2 ii cuda-cufft-dev-10-0 ii cuda-cufft-dev-10-2 ii cuda-cuobjdump-10-0 ii cuda-cuobjdump-10-2 ii cuda-cupti-10-0 rc cuda-cupti-10-1 ii cuda-cupti-10-2 ii cuda-cupti-dev-10-2 ii cuda-curand-10-0 rc cuda-curand-10-1 ii cuda-curand-10-2 ii cuda-curand-dev-10-0 ii cuda-curand-dev-10-2 ii cuda-cusolver-10-0 rc cuda-cusolver-10-1 ii cuda-cusolver-10-2 ii cuda-cusolver-dev-10-0 ii cuda-cusolver-dev-10-2 ii cuda-cusparse-10-0 rc cuda-cusparse-10-1 ii cuda-cusparse-10-2 ii cuda-cusparse-dev-10-0 ii cuda-cusparse-dev-10-2 ii cuda-demo-suite-10-0 ii cuda-demo-suite-10-2 ii cuda-documentation-10-0 ii cuda-documentation-10-2 ii cuda-driver-dev-10-0 ii cuda-driver-dev-10-2 ii cuda-drivers ii cuda-gdb-10-0 ii cuda-gdb-10-2 ii cuda-gpu-library-advisor-10-0 ii cuda-libraries-10-0 ii cuda-libraries-10-2 ii cuda-libraries-dev-10-0 ii cuda-libraries-dev-10-2 ii cuda-license-10-0 ii cuda-license-10-2 ii cuda-memcheck-10-0 ii cuda-memcheck-10-2 ii cuda-misc-headers-10-0 ii cuda-misc-headers-10-2 ii cuda-npp-10-0 rc cuda-npp-10-1 ii cuda-npp-10-2 ii cuda-npp-dev-10-0 ii cuda-npp-dev-10-2 ii cuda-nsight-10-0 ii cuda-nsight-10-2 ii cuda-nsight-compute-10-0 rc cuda-nsight-compute-10-1 ii cuda-nsight-compute-10-2 rc cuda-nsight-systems-10-1 ii cuda-nsight-systems-10-2 ii cuda-nvcc-10-0 rc cuda-nvcc-10-1 ii cuda-nvcc-10-2 ii cuda-nvdisasm-10-0 ii cuda-nvdisasm-10-2 ii cuda-nvgraph-10-0 rc cuda-nvgraph-10-1 ii cuda-nvgraph-10-2 ii cuda-nvgraph-dev-10-0 ii cuda-nvgraph-dev-10-2 ii cuda-nvjpeg-10-0 rc cuda-nvjpeg-10-1 ii cuda-nvjpeg-10-2 ii cuda-nvjpeg-dev-10-0 ii cuda-nvjpeg-dev-10-2 ii cuda-nvml-dev-10-0 ii cuda-nvml-dev-10-2 ii cuda-nvprof-10-0 rc cuda-nvprof-10-1 ii cuda-nvprof-10-2 ii cuda-nvprune-10-0 ii cuda-nvprune-10-2 ii cuda-nvrtc-10-0 rc cuda-nvrtc-10-1 ii cuda-nvrtc-10-2 ii cuda-nvrtc-dev-10-0 ii cuda-nvrtc-dev-10-2 ii cuda-nvtx-10-0 rc cuda-nvtx-10-1 ii cuda-nvtx-10-2 ii cuda-nvvp-10-0 ii cuda-nvvp-10-2 ii cuda-repo-ubuntu1804 ii cuda-runtime-10-0 ii cuda-runtime-10-2 ii cuda-samples-10-0 ii cuda-samples-10-2 rc cuda-sanitizer-api-10-1 ii cuda-sanitizer-api-10-2 ii cuda-toolkit-10-0 rc cuda-toolkit-10-1 ii cuda-toolkit-10-2 ii cuda-tools-10-0 ii cuda-tools-10-2 ii cuda-visual-tools-10-0 rc cuda-visual-tools-10-1 ii cuda-visual-tools-10-2 ii libcudart9.1:amd64 ii libcudnn7 ii libcudnn7-dev ii libcudnn7-doc ii libnvrtc9.1:amd64 ii nvidia-cuda-dev ii nvidia-cuda-doc ii nvidia-cuda-gdb ii nvidia-cuda-toolkit ii nvidia-profiler ii nvidia-visual-profiler 10.2.89-1 amd64 CUDA meta-package
10.0.130-1 amd64 CUDA 10.0 meta-package
10.2.89-1 amd64 CUDA 10.2 meta-package
10.0.130-1 amd64 CUDA command-line tools
10.2.89-1 amd64 CUDA command-line tools
10.0.130-1 amd64 CUDA compiler
10.2.89-1 amd64 CUDA compiler
10.0.130-1 amd64 CUBLAS native runtime libraries
10.0.130-1 amd64 CUBLAS native dev links, headers
10.0.130-1 amd64 CUDA Runtime native Libraries
10.1.168-1 amd64 CUDA Runtime native Libraries
10.2.89-1 amd64 CUDA Runtime native Libraries
10.0.130-1 amd64 CUDA Runtime native dev links, headers
10.1.168-1 amd64 CUDA Runtime native dev links, headers
10.2.89-1 amd64 CUDA Runtime native dev links, headers
10.0.130-1 amd64 CUFFT native runtime libraries
10.1.168-1 amd64 CUFFT native runtime libraries
10.2.89-1 amd64 CUFFT native runtime libraries
10.0.130-1 amd64 CUFFT native dev links, headers
10.2.89-1 amd64 CUFFT native dev links, headers
10.0.130-1 amd64 CUDA cuobjdump
10.2.89-1 amd64 CUDA cuobjdump
10.0.130-1 amd64 CUDA profiling tools interface.
10.1.168-1 amd64 CUDA profiling tools interface.
10.2.89-1 amd64 CUDA profiling tools runtime libs.
10.2.89-1 amd64 CUDA profiling tools interface.
10.0.130-1 amd64 CURAND native runtime libraries
10.1.168-1 amd64 CURAND native runtime libraries
10.2.89-1 amd64 CURAND native runtime libraries
10.0.130-1 amd64 CURAND native dev links, headers
10.2.89-1 amd64 CURAND native dev links, headers
10.0.130-1 amd64 CUDA solver native runtime libraries
10.1.168-1 amd64 CUDA solver native runtime libraries
10.2.89-1 amd64 CUDA solver native runtime libraries
10.0.130-1 amd64 CUDA solver native dev links, headers
10.2.89-1 amd64 CUDA solver native dev links, headers
10.0.130-1 amd64 CUSPARSE native runtime libraries
10.1.168-1 amd64 CUSPARSE native runtime libraries
10.2.89-1 amd64 CUSPARSE native runtime libraries
10.0.130-1 amd64 CUSPARSE native dev links, headers
10.2.89-1 amd64 CUSPARSE native dev links, headers
10.0.130-1 amd64 Demo suite for CUDA
10.2.89-1 amd64 Demo suite for CUDA
10.0.130-1 amd64 CUDA documentation
10.2.89-1 amd64 CUDA documentation
10.0.130-1 amd64 CUDA Driver native dev stub library
10.2.89-1 amd64 CUDA Driver native dev stub library
440.33.01-1 amd64 CUDA Driver meta-package
10.0.130-1 amd64 CUDA-GDB
10.2.89-1 amd64 CUDA-GDB
10.0.130-1 amd64 CUDA GPU Library Advisor.
10.0.130-1 amd64 CUDA Libraries 10.0 meta-package
10.2.89-1 amd64 CUDA Libraries 10.2 meta-package
10.0.130-1 amd64 CUDA Libraries 10.0 development meta-package
10.2.89-1 amd64 CUDA Libraries 10.2 development meta-package
10.0.130-1 amd64 CUDA licenses
10.2.89-1 amd64 CUDA licenses
10.0.130-1 amd64 CUDA-MEMCHECK
10.2.89-1 amd64 CUDA-MEMCHECK
10.0.130-1 amd64 CUDA miscellaneous headers
10.2.89-1 amd64 CUDA miscellaneous headers
10.0.130-1 amd64 NPP native runtime libraries
10.1.168-1 amd64 NPP native runtime libraries
10.2.89-1 amd64 NPP native runtime libraries
10.0.130-1 amd64 NPP native dev links, headers
10.2.89-1 amd64 NPP native dev links, headers
10.0.130-1 amd64 CUDA nsight
10.2.89-1 amd64 CUDA nsight
10.0.130-1 amd64 NVIDIA Nsight Compute
10.1.168-1 amd64 NVIDIA Nsight Compute
10.2.89-1 amd64 NVIDIA Nsight Compute
10.1.168-1 amd64 NVIDIA Nsight Systems
10.2.89-1 amd64 NVIDIA Nsight Systems
10.0.130-1 amd64 CUDA nvcc
10.1.168-1 amd64 CUDA nvcc
10.2.89-1 amd64 CUDA nvcc
10.0.130-1 amd64 CUDA disassembler
10.2.89-1 amd64 CUDA disassembler
10.0.130-1 amd64 NVGRAPH native runtime libraries
10.1.168-1 amd64 NVGRAPH native runtime libraries
10.2.89-1 amd64 NVGRAPH native runtime libraries
10.0.130-1 amd64 NVGRAPH native dev links, headers
10.2.89-1 amd64 NVGRAPH native dev links, headers
10.0.130-1 amd64 NVJPEG native runtime libraries
10.1.168-1 amd64 NVJPEG native runtime libraries
10.2.89-1 amd64 NVJPEG native runtime libraries
10.0.130-1 amd64 NVJPEG native dev links, headers
10.2.89-1 amd64 NVJPEG native dev links, headers
10.0.130-1 amd64 NVML native dev links, headers
10.2.89-1 amd64 NVML native dev links, headers
10.0.130-1 amd64 CUDA Profiler tools
10.1.168-1 amd64 CUDA Profiler tools
10.2.89-1 amd64 CUDA Profiler tools
10.0.130-1 amd64 CUDA nvprune
10.2.89-1 amd64 CUDA nvprune
10.0.130-1 amd64 NVRTC native runtime libraries
10.1.168-1 amd64 NVRTC native runtime libraries
10.2.89-1 amd64 NVRTC native runtime libraries
10.0.130-1 amd64 NVRTC native dev links, headers
10.2.89-1 amd64 NVRTC native dev links, headers
10.0.130-1 amd64 NVIDIA Tools Extension
10.1.168-1 amd64 NVIDIA Tools Extension
10.2.89-1 amd64 NVIDIA Tools Extension
10.0.130-1 amd64 CUDA nvvp
10.2.89-1 amd64 CUDA nvvp
10.0.130-1 amd64 cuda repository configuration files
10.0.130-1 amd64 CUDA Runtime 10.0 meta-package
10.2.89-1 amd64 CUDA Runtime 10.2 meta-package
10.0.130-1 amd64 CUDA example applications
10.2.89-1 amd64 CUDA example applications
10.1.168-1 amd64 CUDA Sanitizer API
10.2.89-1 amd64 CUDA Sanitizer API
10.0.130-1 amd64 CUDA Toolkit 10.0 meta-package
10.1.168-1 amd64 CUDA Toolkit 10.1 meta-package
10.2.89-1 amd64 CUDA Toolkit 10.2 meta-package
10.0.130-1 amd64 CUDA Tools meta-package
10.2.89-1 amd64 CUDA Tools meta-package
10.0.130-1 amd64 CUDA visual tools
10.1.168-1 amd64 CUDA visual tools
10.2.89-1 amd64 CUDA visual tools
9.1.85-3ubuntu1 amd64 NVIDIA CUDA Runtime Library
7.5.1.10-1+cuda10.1 amd64 cuDNN runtime libraries
7.5.1.10-1+cuda10.1 amd64 cuDNN development libraries and headers
7.5.1.10-1+cuda10.1 amd64 cuDNN documents and samples
9.1.85-3ubuntu1 amd64 CUDA Runtime Compilation (NVIDIA NVRTC Library)
9.1.85-3ubuntu1 amd64 NVIDIA CUDA development files
9.1.85-3ubuntu1 all NVIDIA CUDA and OpenCL documentation
9.1.85-3ubuntu1 amd64 NVIDIA CUDA Debugger (GDB)
9.1.85-3ubuntu1 amd64 NVIDIA CUDA development toolkit
9.1.85-3ubuntu1 amd64 NVIDIA Profiler for CUDA and OpenCL
9.1.85-3ubuntu1 amd64 NVIDIA Visual Profiler for CUDA and OpenCL

Topic		Replies	Views
CUDA install unmet dependencies: cuda : Depends: cuda-10-0 (>= 10.0.130) but it is not going to be installed CUDA Setup and Installation	37	182507	September 17, 2023
RuntimeError: CUDA error: no kernel image is available for execution on the device Linux	29	79789	February 22, 2021
cublas for 10.1 is missing CUDA Setup and Installation	23	64071	October 24, 2020
CUDA, Linux Ubuntu 10.04 and strange mismatch version CUDA Programming and Performance	26	19085	November 18, 2010
CUFFT_INVALID_DEVICE on cufftPlan1d in NVIDIA's Simple CUFFT example GPU-Accelerated Libraries	6	3801	December 15, 2014
CUDA 10 installation problems on Ubuntu 18.04 CUDA Setup and Installation	24	94543	December 11, 2020
TensorFlow cannot find cuDNN [Ubuntu 16.04 + CUDA7.5] CUDA Setup and Installation	12	42446	February 10, 2017
Installation failures(?) despite instructions CUDA Setup and Installation	16	3280	January 26, 2018
Cannot run any CUDA kernels CUDA runtime doesn't recognize NVIDIA GPU CUDA Programming and Performance	26	12330	August 24, 2010
CUDA 2.1 discussion CUDA Programming and Performance	71	63940	February 17, 2009

sample mnistCUDNN failed with 'Cublas failure gemv.h:77' Ubuntu 18.04.3 Cuda10.1 2080S CUDNN 7.6.5

Related topics