I just upgraded from a 1080 to a Titan V. After that, I experienced a substantial delay when running the cudnn samples.
Cuda version: 8.0
cudnn version: 7.0.5
Cuda driver: 387.34
OS: Ubuntu 14.04
To demonstrate the delay, I modified the cudnn “conv_sample” code of the official cudnn v7 samples.
I timed the call to cudnnCreate as follows:
printf("Creating cudnn handle\n"); double start = second(); checkCudnnErr(cudnnCreate(&handle_)); double stop = second(); printPerf( stop - start, 0, 0, 0, 0, 0, 0);
When running it I get the following output:
Testing single precision Creating cudnn handle ^^^^ CUDA : elapsed = 264.361 sec, Testing conv ^^^^ CUDA : elapsed = 8.4877e-05 sec, Test PASSED Testing half precision (math in single precision) Creating cudnn handle ^^^^ CUDA : elapsed = 0.000301838 sec, Testing conv ^^^^ CUDA : elapsed = 5.00679e-05 sec, Test PASSED
Does anyone have any ideas how to debug or narrow down this problem? Just fyi: I also tried running it without checkCudnnErr and experience the same problem. I also experience this problem when running other high-level libraries that use cudnn and on all other cudnn v7 samples.