How to activate with softplus in C backend?

AI & Data Science Deep Learning (Training & Inference) cuDNN
1

Hello,

I am using cudnnActivationMode_t enum for sigmoid, relu, tanh activation functions as my C++ class member m_activationMode to set member m_activationDescriptor (which is used for subsequent cudnnActivationForward call for inferencing):

cudnnSetActivationDescriptor (m_activationDescriptor, m_activationMode, CUDNN_NOT_PROPAGATE_NAN, alpha));

Unfortunately, I see no CUDNN_ACTIVATION_SOFTPLUS kind under 9.5/include/cudnn_graph_v9.h for enum cudnnActivationMode_t.

I find softplus in above header, but it is in cudnnPointwiseMode_t and cudnnBackendAttributeName_t enums. The former enum is not used anywhere else in the library. There is hope since latter enum is used in cudnnBackendSetAttribute but not sure how to correctly use that function.

Could you please point to or provide an example for softplus activation?

Thank you

2

Hi @westwood ,
Cheking on this and will get back,.

Thanks

1 Like
3

From the Documentation:

cudnnActivationMode_t

This enumerated type is deprecated and is currently only used by deprecated APIs. Consider using replacements for the deprecated APIs that use this enumerated type.

The new Graph-API works with Pointwise Descriptors, that use PointwiseMode:

cudnn_graph Library — NVIDIA cuDNN

4

Hello,
I see no examples in your response. Could you please either point to an example or provide an example?
Thanks,

5

Here is an example with CuDnn-Frontend for Pointwise: cudnn-frontend/test/cpp/pointwise_tests.cpp at main · NVIDIA/cudnn-frontend · GitHub

You can extrapolate softplus from this.

Edit: Dokumentation for Pointwise: cudnn_graph Library — NVIDIA cuDNN

6

Is there no way to softplus using backend?

7

Create an Pointwise Operation Descriptor (cudnn_graph Library — NVIDIA cuDNN Backend) with a Pointwise Derscriptor. Set CUDNN_ATTR_POINTWISE_MODE to CUDNN_POINTWISE_SOFTPLUS_FWD (or CUDNN_POINTWISE_SOFTPLUS_BWD if this is your usecase) and set the desired Softplus Parameters within the Pointwise-Descriptor ( CUDNN_ATTR_POINTWISE_SOFTPLUS_BETA). Then use this Pointwise-Softplus-Activation within your operation-graph.

Disclaimer: I did not test this, I just gave you the informations provided by the Documentation (cudnn_graph Library — NVIDIA cuDNN Backend).