Hello devtalk community,
I’m sorry if there are any similar topics to the one I post, but I couldn’t find any and decided to ask for your help directly.
The cuDNN documentation says that in the following function
cudnnSetActivationDescriptor( cudnnActivationDescriptor_t activationDesc, cudnnActivationMode_t mode, cudnnNanPropagation_t reluNanOpt, double coef)
the parameter coef specifies the alpha coefficient when the activation mode is set to CUDNN_ACTIVATION_RELU. In this way we can specify a Leaky ReLU activation. So by choosing 0 < coeff < 1 the activation should be - f(x) = coeff * x for x < 0, f(x) = x for x >= 0.
When I call
cudnnSetActivationDescriptor(relu_desc, CUDNN_ACTIVATION_RELU, CUDNN_NOT_PROPAGATE_NAN, 0.1) cudnnActivationForward(cudnn, relu_desc, &alpha, src_tensor, dev_src, &beta, relu_tensor, dev_relu)
alpha = 1; beta = 0;
x > 0 is mapped to x and x < 0 to 0 instead of x < 0 to x * 0.1. So the combination CUDNN_ACTIVATION_RELU and 0 < coeff < 1 is not working like a Leaky ReLU.
Is this the normal behaviour of the function? If yes, what combination is supposed to be Leaky ReLU?
I’m using cuda compilation tools, release 10.1, V10.1.168 and cudnn-10.1-windows10-x64-v18.104.22.168
Thank you for your time!