What is the behavior of CUDNN_LOSS_NORMALIZATION_NONE?

AI & Data Science Deep Learning (Training & Inference) cuDNN
#1

Hi, I am try to use cudnnCTCLoss api, but a little bit confused of “cudnnLossNormalizationMode_t”. To my understanding, the only difference inside kernel is that, “CUDNN_LOSS_NORMALIZATION_NONE” do not lauch a softmax kernel, while “CUDNN_LOSS_NORMALIZATION_SOFTMAX” would lauch a softmax kernel.
But I found that with the same input, I just do softmax manually before feeding it into cudnn with “CUDNN_LOSS_NORMALIZATION_NONE”, and do not do softmax with “CUDNN_LOSS_NORMALIZATION_SOFTMAX”, the gradients returned in above two circumstances are different.
Is there anybody who can explain it to me?
Thanks very much!

#2

Hi @gaochyz ,
Please allow us sometime to check on this.

Thanks!

#3

Hi @gaochyz ,
Can you please verify if the cudnnCTCLossAlgo_t parameter passed into the CTC API function is the deterministic algo?

Thanks!