CUDNN_ACTIVATION_RELU in combination with 0 < coeff < 1 doesn't work?

AI & Data Science Deep Learning (Training & Inference) cuDNN
1

Hello devtalk community,

I’m sorry if there are any similar topics to the one I post, but I couldn’t find any and decided to ask for your help directly.

The cuDNN documentation says that in the following function

cudnnSetActivationDescriptor(
    cudnnActivationDescriptor_t         activationDesc,
    cudnnActivationMode_t               mode,
    cudnnNanPropagation_t               reluNanOpt,
    double                              coef)

the parameter coef specifies the alpha coefficient when the activation mode is set to CUDNN_ACTIVATION_RELU. In this way we can specify a Leaky ReLU activation. So by choosing 0 < coeff < 1 the activation should be - f(x) = coeff * x for x < 0, f(x) = x for x >= 0.

When I call

cudnnSetActivationDescriptor(relu_desc, CUDNN_ACTIVATION_RELU, CUDNN_NOT_PROPAGATE_NAN, 0.1)
cudnnActivationForward(cudnn, relu_desc, &alpha, src_tensor, dev_src, &beta, relu_tensor, dev_relu)

with

alpha = 1; beta = 0;

x > 0 is mapped to x and x < 0 to 0 instead of x < 0 to x * 0.1. So the combination CUDNN_ACTIVATION_RELU and 0 < coeff < 1 is not working like a Leaky ReLU.

Is this the normal behaviour of the function? If yes, what combination is supposed to be Leaky ReLU?

I’m using cuda compilation tools, release 10.1, V10.1.168 and cudnn-10.1-windows10-x64-v7.6.0.64

Thank you for your time!

1 Like
2

The documentation says:

Floating point number. When the activation mode (see cudnnActivationMode_t) is set to CUDNN_ACTIVATION_CLIPPED_RELU, this input specifies the clipping threshold; and when the activation mode is set to CUDNN_ACTIVATION_RELU, this input specifies the upper bound.

What is the difference between “clipping thresold” and “upper bound”?

P.S. It would great if Leaky RELU could be supported by cuDNN in future versions.

3

Hi, did you get the answer