Can CUDNN be split more finely?

AI & Data Science Deep Learning (Training & Inference) cuDNN
1

At present, it is found that the complete package of CUDNN8 is too large in the integration, can CUDNN be split more finely?

CUDNN calls mainly involved in our inference library,Include the following

CONV:
cudnnConvolutionBiasActivationForward
cudnnConvolutionForward
cudnnFusedOpsExecute

POOLING:
cudnnPoolingForward

SOFTMAX：
cudnnSoftmaxForward

RNN:
cudnnRNNForwardInference

DECONV:
cudnnConvolutionBackwardData

ACTIVE:
cudnnActivationForward

2

If you’re just doing inference, you can use the *_infer libraries instead of the complete package. Just to help me understand – are you saying these binaries are too large for your integration?

  • cudnn_ops_infer - This entity contains the routines related to cuDNN context creation and destruction, tensor descriptor management, tensor utility routines, and the inference portion of common ML algorithms such as batch normalization, softmax, dropout, etc.
  • cudnn_cnn_infer - This entity contains all routines related to convolutional neural networks needed at inference time. The cudnn_cnn_infer library depends on cudnn_ops_infer.
  • cudnn_adv_infer - This entity contains all other features and algorithms. This includes RNNs, CTC loss, and Multihead Attention. The cudnn_adv_infer library depends on cudnn_ops_infer.
3

At present, we just do forward inference, and we’ve only integrated the infer libraries. However, for cuDNN v8.2 infer libraries, the total size is about 1100MB, while the full size of cuDNN v7.6.3 library is only 430MB. The cuDNN v8.2 infer libraries are too large for our system integration. Whether cuDNN v8.2 infer libraries can be further splited, the library only contains the following API functions that we use. Therefore, we can easily integrate cuDNN to our system. Thank you.

cudnnConvolutionBiasActivationForward()

cudnnConvolutionForward()

cudnnFusedOpsExecute ()

cudnnPoolingForward()

cudnnSoftmaxForward()

cudnnRNNForwardInference()

cudnnConvolutionBackwardData()

cudnnActivationForward()

4

Hi @838105645 ,
Do you need to support many SM or few/only one?
Thanks!