Issues while verifying cudnn 7.3.0 installation

AI & Data Science Deep Learning (Training & Inference) cuDNN
1

OS: Ubuntu 18.04/ cuda 10.0/ tensorflow-gpu 1.13.1/ cudnn 7.3.0

When I want to verify the installation of cudnn through ./mnistCUDNN, it runs about 20 mins and failed like this:

cudnnGetVersion() : 7300 , CUDNN_VERSION from cudnn.h : 7300 (7.3.0)
Host compiler version : GCC 7.5.0
There are 1 CUDA capable devices on your machine :
device 0 : sms 20  Capabilities 8.6, SmClock 1485.0 Mhz, MemSize (Mb) 3910, MemClock 6001.0 Mhz, Ecc=0, boardGroupID=0
Using device 0

Testing single precision
Loading image data/one_28x28.pgm
Performing forward propagation ...
Testing cudnnGetConvolutionForwardAlgorithm ...
Fastest algorithm is Algo 0
Testing cudnnFindConvolutionForwardAlgorithm ...
^^^^ CUDNN_STATUS_SUCCESS for Algo 1: 0.001824 time requiring 100 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 0: 0.010240 time requiring 0 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 2: 0.018432 time requiring 57600 memory
^^^^ CUDNN_STATUS_SUCCESS for Algo 4: 0.052224 time requiring 207360 memory
^^^^ CUDNN_STATUS_NOT_SUPPORTED for Algo 3: -1.000000 time requiring 0 memory
Resulting weights from Softmax:
0.0000000 0.9999399 0.0000000 0.0000000 0.0000561 0.0000000 0.0000012 0.0000017 0.0000010 0.0000000 
Loading image data/three_28x28.pgm
Performing forward propagation ...
Resulting weights from Softmax:
0.0000000 0.0000000 0.0000000 0.9999288 0.0000000 0.0000711 0.0000000 0.0000000 0.0000000 0.0000000 
Loading image data/five_28x28.pgm
Performing forward propagation ...
Resulting weights from Softmax:
0.0997550 0.0892810 0.1054468 0.1071846 0.0902156 0.1043498 0.0953814 0.0938049 0.1155469 0.0990339 

Result of classification: 1 3 8

Test failed!
Prediction mismatch
mnistCUDNN.cpp:876
Aborting...

But when I use cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2, it shows:

#define CUDNN_MAJOR 7
#define CUDNN_MINOR 3
#define CUDNN_PATCHLEVEL 0
--
#define CUDNN_VERSION (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)

#include "driver_types.h"

I installed cudnn through “Debian Local Installation”, I did:
sudo dpkg -i cudnn_package,
sudo cp cudnn* /usr/local/cuda/include
sudo cp -P libcudnn* /usr/local/cuda/lib64
sudo cp cudnn* /usr/local/cuda-10.0/include
sudo cp -P libcudnn* /usr/local/cuda-10.0/lib64
and activate them in their directionary
sudo chmod a+r cudnn*
sudo chmod a+r libcudnn*

Also, I tried apt-get like sudo apt-get install libcudnn7-doc=7.3.0.29-1+cuda10.0, they all showed libcudnn7 libcudnn-dev libcudnn-doc are the the newest version (7.3.0.29-1+cuda10.0).

I don’t know where goes wrong. Please help me!!!
Thank you in advance!
Help!!!

2

Hi,

This doesn’t look like installation related issue, as you could run the application.
Could you please share the complete logs if missing above and please share with us the nvidia-smi output as well while running the app.

Thank you.

1 Like
3

nvidia-smi result:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.141.03   Driver Version: 470.141.03   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:01:00.0  On |                  N/A |
| N/A   41C    P8     7W /  N/A |    493MiB /  3910MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1103      G   /usr/lib/xorg/Xorg                189MiB |
|    0   N/A  N/A      1329      G   /usr/bin/gnome-shell               54MiB |
|    0   N/A  N/A      9224      G   /usr/lib/firefox/firefox          155MiB |
|    0   N/A  N/A     11057      G   ...RendererForSitePerProcess       89MiB |
+-----------------------------------------------------------------------------+

but I find it’s probably due to my limited GPU memory (4G). Everytime I ran a machine learning algorithm, it takes up at least 3000MB memory, so I have to train my model remotely. Thank you for your reply!