Transposed Convolution

Hi,

I’m working on a project where we want to compare different implementation of DNN. One of these implementation is in cuDNN.
We need to implement a transposed convolution as the Conv2DTranspose in TensorFlow Keras.

Can we found some example or guide about implementation of the transposed convolution in cuDNN ?

Thanks in advance,

Hugo

Hi @hugo.kieffer ,
Deconvolution/transposed convolution is basically the “dgrad” operation. Dgrad is designed for the backwards phase of training, so you may need to choose your filter layout accordingly. (If a forward convolution from Tensor A NCHW to Tensor C NKPQ uses a KRSC filter, then the dgrad operation would take Tensor C as input and Tensor A as ouput, but still use the KRSC filter.)
Note also that unstrided (unit strided) deconvolution is just a convolution with the filter transposed (hence the alternate name “transposed convolution”). So if it’s not a strided deconvolution, just use cudnn’s convolution operator and flip the cross correlation flag.

However i am afraid, I dont think we have a sample available for the same.

Thanks

Hugo,

Have you had any success implementing this layer in cuDNN?

Hello, I haven’t had time to rework on the subject yet, we have several parallel implementations (FPGA and GPU) and we have had work on the FPGA target.
I’ll share an example when I have a functional code.

Hi,

I reopen this thread because I rework on this implementation.
I want to add some information about the type of transposed convolution that we want to do.

In our tensorflow model we use the Conv2DTranspose like this :
Conv2DTranspose(128, (2, 2), strides=(2, 2), padding=‘same’)

The parameters are as below :

  • 128 kernel of 2x2
  • Stride of 2

The input of this Conv2DTranspose is obtained by the output of a Conv2D :
Conv2D(256, (3, 3), activation=‘relu’, kernel_initializer=‘he_normal’, padding=‘same’)

Full Code :
c5 = tf.keras.layers.Conv2D(256, (3, 3), activation=‘relu’, kernel_initializer=‘he_normal’, padding=‘same’)(c5)
u6 = tf.keras.layers.Conv2DTranspose(128, (2, 2), strides=(2, 2), padding=‘same’)(c5)

I think I need to use the backward convolution in the cuDNN API, but I’m not sure…

I try to implement a simple transposed convolution first, with only one kernel and small input tensor to understand the implementation but I always have CUDNN_STATUS_BAD_PARAM, and no information…

Someone know examples or tips to implement this type of transposed convolution in cuDNN ?

I’ve been working on it for a while and I’ve had some progress, but it’s still not fully implemented.

The Conv2DTranspose in the previous post take input with shape 8, 8, 256 (spacial dimension 8x8 with 256 feature map), the output is 16, 16, 128. The kernel shape of this convolution is 2, 2, 128, 256.

So, this operation extend spacial dimension and reduce number of feature maps.

The cudnnConvolutionBackwardData() function is tested to do this and a working configuration is found for spacial dimension and feature maps.
Doc of this function : API Reference :: NVIDIA Deep Learning cuDNN Documentation

The configuration below is used:

  • Convolution: pad (0,0), stride (2,2), dilation (1,1), mode CUDNN_CROSS_CORRELATION
  • W filter: format NHWC, k=256, c=128, h=2, w=2
  • dy tensor: format NHWC, shape 8x8x256
  • dx tensor: format NHWC, shape 16x16x128

My first misunderstanding is about the filter, the input and output feature maps are inverted regarding a forward convolution.

I have a simple Conv2DTranspose layer implemented in keras to compare the result of cudnn and keras, I use same data (filter, bias and input) on the CUDNN and keras implementation.

With a constant kernel (e.g. 2.0 for each filter values) the result is identical between two implementations, but when randomly generated data is used for filter (same for both implementation) the result is different…

I suspect the format of filter but I made many test (swaping channels, changing format…) without results.