Fusion of convolution and BatchNorm

hbwx26 · December 29, 2021, 12:21pm

cudnn : 8.0
I found two ways to fuse Convolution and BatchNorm.

Use CUDNN_FUSED_SCALE_BIAS_ACTIVATION_CONV_BNSTATS，FusedOpsPlan,cudnnSetFusedOpsConstParamPackAttribute ,cudnnSetFusedOpsVariantParamPackAttribute,etc.
I saw that the cudnn api document describes it like this: " As of cuDNN 7.6.0, if the conditions in Table 26 are met, then the fully fused fast path will be triggered. Otherwise, a slower partially fused path will be triggered."
However when I set the input type of convolution to CUDNN_DATA_FLOAT, the code raise CUDNN_STATUS_BAD_PARAM .

2.Fuse OP by Backend API.
Does Scale_Bias_Activation_convolution_genStats correspond to Convolution-BatchNorm fusion? Does PSEUDO_HALF_CONFIG mean that I only use CUDNN_DATA_HALF input?

If I want to achieve Convolution-BatchNorm fusion, how to set the arguments？

spolisetty · January 18, 2022, 11:40am

Hi,

Please refer the following for cuDNN fuse operator sample.

github.com

NVIDIA/cudnn-frontend/blob/main/samples/conv_sample.cpp

/*
 * Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
 * copy of this software and associated documentation files (the "Software"),
 * to deal in the Software without restriction, including without limitation
 * the rights to use, copy, modify, merge, publish, distribute, sublicense,
 * and/or sell copies of the Software, and to permit persons to whom the
 * Software is furnished to do so, subject to the following conditions:
 *
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
 * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
 * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
 * DEALINGS IN THE SOFTWARE.

This file has been truncated. show original

For more details,

Thank you.

yanxu · January 25, 2022, 4:21am

Regarding the CUDNN_STATUS_BAD_PARAM issue, @hbwx26 can you provide the API log?

Yes Scale_Bias_Activation_convolution_genStats is the forward fusion pattern to achieve conv-bn fusion. Another one you will need is Scale_Bias_Activation_ConvBwdFilter in the backward path as well.
PSEUDO_HALF_CONFIG means all the storage tensors are in FP16, and all the compute precision should be FP32.
Unfortunately we don’t have a frontend sample handy, we are working on one that we should be able to share soon

csdncannon · March 17, 2022, 7:09am

Hello,
I didn’t see CUDNN_BACKEND_OPERATION_GEN_STATS_DESCRIPTOR is used in the latest frontend codes. Is Scale_Bias_Activation_ConvBwdFilter pattern already supported by latest CuDNN-Backend? If yes, would recent frontend codes add corresponding sample?

Thanks
Gino

csdncannon · April 29, 2022, 12:08pm

BTW, normally BatchNorm could be converted to FusedBatchNorm(equal to Scale + Bias) in fine-tune phase, then use Conv_Scale_Bias pattern in inference phase, so why bother to support Scale_Bias_Activation_convolution_genStats pattern? Is it only used in training’s forward phase?

Thanks
Gino