cuDNN foward and backward passes...help me understand

AI & Data Science Deep Learning (Training & Inference) cuDNN
1

Hey again everybody. I’m trying to wrap my head around how cuDNN works as an overall network WITH backpropagation.

Here’s my simple network:

64x64x3 Image → Convolution → Activation → TransposeConvolution → Activation → 64x64x3 Image

Think of it as a simple autoencoder. I give it an image, and it outputs an image. The code is basically:

cudnnConvolutionForward() → cudnnActivationForward() → cudnnConvolutionBackwardData() → cudnnActivationForward()

Nice right? Okay, that’s the easy part of course. What I’m struggling with is how to back propagate the error (target image - output image). This is how I think it should flow:

cudnnActivationBackward() → cudnnConvolutionForward() → cudnnActivationBackward() → cudnnConvolutionBackwardData()

What I don’t get is well, most of this. If I do cudnnConvolutionBackwardData on the forward pass, do I do cudnnConvolutionForward on the backward pass to get the gradient? IDK. What about the filter(weights) update? I see there is a cudnnConvolutionBackwardFilter function, but will that work on the filter used in the TransposeConvolution?

There is very little direction here…none actually that I’ve seen. It’s hard enough to understand how to get a forward pass set up, but backwards?..wow…IDK.

Any help or pointers would be appreciated! :)
-Chris

2

Okay, don’t worry about it. There is a thread on this forum that has a response by the moderator that makes transpose convolutions very confusing. He talks about “dgrad”. Ignore it.

Read this: https://arxiv.org/pdf/1603.07285v1.pdf

You can do transpose convolutions using regular forward convolutions. Read the paper.

1 Like
3

Messing around some more. You can use the BackwardData function to do transpose convolution, and yes, you can use Forward to pass the gradient to the previous layer (haven’t confirmed the math though, so try it yourself). You can also seemingly update the filter and bias using the BackwardFilter/Bias functions too. Again, I haven’t done the math to confirm it, use at your own risk if nobody can confirm. IDK. :P

4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.