Debugging a cudnnConvolutionForward() CUDNN_STATUS_EXECUTION_FAILED

I’m currently changing a CUDA application written in MSVC on Windows to add cuDNN 5.1.

The app has previously run on CUDA 8.1 on two NVidia GTX 1080 compiled to support PASCAL architecture with the “compute_61,sm_61” directives.

However, with the addition of cuDNN, an NVCC function with a cudnnConvolutionForward() call that worked before is now failing with a CUDNN_STATUS_EXECUTION_FAILED status which the manual at

http://developer.download.nvidia.com/compute/machine-learning/cudnn/secure/v5.1/prod/doc/CUDNN_Library.pdf

states means " The function failed to launch on the GPU.". No other diagnostic information is available.

How do I debug this black box? I’ve checked settings and nvidia-smi and the layers being submitted are exactly the same as they’ve always been. This is the second layer that I’m submitting to the GPU.

@KevinLucidyne I had the same issue. Did you solve this problem or any suggestions?

Thanks a lot in advance.