Fusion of convolution and BatchNorm

[hbwx26]

cudnn : 8.0
I found two ways to fuse Convolution and BatchNorm.

1. Use CUDNN_FUSED_SCALE_BIAS_ACTIVATION_CONV_BNSTATS，FusedOpsPlan,cudnnSetFusedOpsConstParamPackAttribute ,cudnnSetFusedOpsVariantParamPackAttribute,etc.
   I saw that the cudnn api document describes it like this: " As of cuDNN 7.6.0, if the conditions in Table 26 are met, then the fully fused fast path will be triggered. Otherwise, a slower partially fused path will be triggered."
   However when I set the input type of convolution to CUDNN_DATA_FLOAT, the code raise CUDNN_STATUS_BAD_PARAM .

2.Fuse OP by Backend API.
Does Scale_Bias_Activation_convolution_genStats correspond to Convolution-BatchNorm fusion? Does PSEUDO_HALF_CONFIG mean that I only use CUDNN_DATA_HALF input?

If I want to achieve Convolution-BatchNorm fusion, how to set the arguments？

[spolisetty]

Hi,

Please refer the following for cuDNN fuse operator sample.

NVIDIA/cudnn-frontend/blob/main/samples/conv_sample.cpp

For more details,

Fuse Operators

Hi, all. I’m a beginner cuDNN developer and I’m trying to fuse operators for inference. When going through the documentation (Developer Guide :: NVIDIA Deep Learning cuDNN Documentation), I learned that this might be done by using backend API. (Please correct me if I’m wrong. I’m not 100% sure because the “seem-to-be” relevant functions are in training library.) Although this post provides a good high-level overview, I’m still not sure how to actually implement operator fusion. For example, …

Thank you.

[yanxu]

Regarding the CUDNN_STATUS_BAD_PARAM issue, @hbwx26 can you provide the API log?

Developer Guide :: NVIDIA Deep Learning cuDNN Documentation

Yes Scale_Bias_Activation_convolution_genStats is the forward fusion pattern to achieve conv-bn fusion. Another one you will need is Scale_Bias_Activation_ConvBwdFilter in the backward path as well.
PSEUDO_HALF_CONFIG means all the storage tensors are in FP16, and all the compute precision should be FP32.
Unfortunately we don’t have a frontend sample handy, we are working on one that we should be able to share soon

[csdncannon]

Hello,
I didn’t see CUDNN_BACKEND_OPERATION_GEN_STATS_DESCRIPTOR is used in the latest frontend codes. Is Scale_Bias_Activation_ConvBwdFilter pattern already supported by latest CuDNN-Backend? If yes, would recent frontend codes add corresponding sample?

Thanks
Gino

[csdncannon]

BTW, normally BatchNorm could be converted to FusedBatchNorm(equal to Scale + Bias) in fine-tune phase, then use Conv_Scale_Bias pattern in inference phase, so why bother to support Scale_Bias_Activation_convolution_genStats pattern? Is it only used in training’s forward phase?

Thanks
Gino