Issues while verifying cudnn 7.3.0 installation

OS: Ubuntu 18.04/ cuda 10.0/ tensorflow-gpu 1.13.1/ cudnn 7.3.0

When I want to verify the installation of cudnn through ./mnistCUDNN, it runs about 20 mins and failed like this:

cudnnGetVersion() : 7300 , CUDNN_VERSION from cudnn.h : 7300 (7.3.0)
Host compiler version : GCC 7.5.0
There are 1 CUDA capable devices on your machine :
device 0 : sms 20  Capabilities 8.6, SmClock 1485.0 Mhz, MemSize (Mb) 3910, MemClock 6001.0 Mhz, Ecc=0, boardGroupID=0
Using device 0

Testing single precision
Loading image data/one_28x28.pgm
Performing forward propagation ...
Testing cudnnGetConvolutionForwardAlgorithm ...
Fastest algorithm is Algo 0
Testing cudnnFindConvolutionForwardAlgorithm ...
^^^^ CUDNN_STATUS_SUCCESS for Algo 1: 0.001824 time requiring 100 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 0: 0.010240 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 2: 0.018432 time requiring 57600 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 4: 0.052224 time requiring 207360 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 3: -1.000000 time requiring 0 memory
Resulting weights from Softmax:
0.0000000 0.9999399 0.0000000 0.0000000 0.0000561 0.0000000 0.0000012 0.0000017 0.0000010 0.0000000 
Loading image data/three_28x28.pgm
Performing forward propagation ...
Resulting weights from Softmax:
0.0000000 0.0000000 0.0000000 0.9999288 0.0000000 0.0000711 0.0000000 0.0000000 0.0000000 0.0000000 
Loading image data/five_28x28.pgm
Performing forward propagation ...
Resulting weights from Softmax:
0.0997550 0.0892810 0.1054468 0.1071846 0.0902156 0.1043498 0.0953814 0.0938049 0.1155469 0.0990339 

Result of classification: 1 3 8

Test failed!
Prediction mismatch
mnistCUDNN.cpp:876
Aborting...

But when I use cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2, it shows:

#define CUDNN_MAJOR 7
#define CUDNN_MINOR 3
#define CUDNN_PATCHLEVEL 0
--
#define CUDNN_VERSION (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)

#include "driver_types.h"

I installed cudnn through “Debian Local Installation”, I did:
sudo dpkg -i cudnn_package,
sudo cp cudnn* /usr/local/cuda/include
sudo cp -P libcudnn* /usr/local/cuda/lib64
sudo cp cudnn* /usr/local/cuda-10.0/include
sudo cp -P libcudnn* /usr/local/cuda-10.0/lib64
and activate them in their directionary
sudo chmod a+r cudnn*
sudo chmod a+r libcudnn*

Also, I tried apt-get like sudo apt-get install libcudnn7-doc=7.3.0.29-1+cuda10.0, they all showed libcudnn7 libcudnn-dev libcudnn-doc are the the newest version (7.3.0.29-1+cuda10.0).

I don’t know where goes wrong. Please help me!!!
Thank you in advance!
Help!!!

Hi,

This doesn’t look like installation related issue, as you could run the application.
Could you please share the complete logs if missing above and please share with us the nvidia-smi output as well while running the app.

Thank you.

1 Like

nvidia-smi result:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.141.03   Driver Version: 470.141.03   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:01:00.0  On |                  N/A |
| N/A   41C    P8     7W /  N/A |    493MiB /  3910MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1103      G   /usr/lib/xorg/Xorg                189MiB |
|    0   N/A  N/A      1329      G   /usr/bin/gnome-shell               54MiB |
|    0   N/A  N/A      9224      G   /usr/lib/firefox/firefox          155MiB |
|    0   N/A  N/A     11057      G   ...RendererForSitePerProcess       89MiB |
+-----------------------------------------------------------------------------+

but I find it’s probably due to my limited GPU memory (4G). Everytime I ran a machine learning algorithm, it takes up at least 3000MB memory, so I have to train my model remotely. Thank you for your reply!