Is it possible to know why specified CUDNN convolution algorithm is selected on Conv2d layer?

AI & Data Science Deep Learning (Training & Inference) cuDNN
1

Hi,

While profiling CNN model, I wanted to know why specified CUDNN convolution algorithm is selected on each Conv2d layer.
when there are 16 of conv2d layers, each layer(aten::cudnn_convolution) issues kernel, but the algorithms from those kernels are different.
For example, my VGG19 model, the first and second conv2d layer select [cudnn::winograd::generateWinogradTilesKernel],[ampere_scudnn_winograd_128x128_ldg1_ldg4_relu_tile148t_nt_v1] and so on. But, no.3 ~ no.16 conv2d layers select [void cudnn::winograd_nonfused::winogradForwardData4x4] and [ampere_sgemm_128x128_nn] so on. One layer uses fft2d algorithm during Backward pass.

I want to know why models use different algorithms for different layers. and where it is decided.
Thank you.

2

Hi,

cuDNN has heuristics(internal logic) to estimate the best kernel for a given problem specification.

Thank you.

3

Thank you for your reply. the internal logic is cudnnFindConvolutionForwardAlgorithm?