Conv3D - Running it on Tensor Core - cuDNN

CUDA : 10.2
cuDNN version : 7.6.5
GPU : Titan RTX & Volta 100
OS : ubuntu 18.04

I’m trying to implement Conv3D in cuDNN. I have the following questions:

  1. Conv3D works for CUDNN_TENSOR_NCHW. But, it doesn’t run on tensor core. I have set CUDNN_TENSOR_OP_MATH, inputs are half1, multiples of 8 (which is not necessary in cuDNN 7.6.5,) and convolution algorithm is CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_PRECOMP_GEMM.
  2. Conv3D doesn’t work for CUDNN_TENSOR_NHWC. Input dim - {N, D, H, W, IN_C} and filter dim - {K, 3, 3, 3, IN_C}

Am I missing something?

For cuDNN 7.6, 3d NHWC is not fully supported. So that is expected.
cuDNN v8 have better 3D support, you can try using the latest version:

cuDNN Release Notes :: NVIDIA Deep Learning SDK Documentation

  • Added 3D convolutions support of NHWC and improved heuristics and kernels for Tensor Cores in NCHW resulting in performance improvements for VNet, UNet-Medical and UNet-Industrial.

Thanks

OK. I understand NHWC is not supported in cuDNN 7.6.5.

What about NCHW? I read somewhere in the forum that volta gpus should be able use tensor cores for 3D convolutions. I did try it on V100 and it didn’t run on tensor cores.

Thank you.

cuDNN will fall back to non tensor core kernels when there’s no available tensor core kernel for the input size.
Please refer to below link for more details:
https://docs.nvidia.com/deeplearning/sdk/cudnn-best-practices/index.html#cudnn-7xx

Thanks

Yes.

My input is as follows:
Inp_dim = {128, 32, 16, 16, 16}
Filter_dim = {64, 32, 3, 3, 3}
Strides / pads / dilation = {1, 1, 1}
Convolution type = CROSS_CORRELATION.
Input & Output datatypes = HALF
Conv datatype = FLOAT.

I think for this particular configuration it should work. But, it doesn’t invoke tensor core kernel.

I tired cuDNN v8 and I was able to run the network on tensor core on a Turing GPU. Any idea about release data of cuDNN v8 stable?

Right now, I don’t have any info regarding cuDNN v8 stable release date.
Will request you to stay tuned to cuDNN offical site and forum.

Thanks