INT8 with cudnn: How do we decide which 8-bits to pick from the computed 32-bit result?

AI & Data Science Deep Learning (Training & Inference) cuDNN
1

Hi,

I am trying out INT8 and INT8x4 configuration with cudnn. I am specifically trying out the following API. For both – INT8 and INT8X4 – configurations cudnn outputs INT8.

cudnnConvolutionForward()

But what I don’t understand is how are we deciding which 8-bits to chose from the 32-bit computed result. Are we simply downcasting the 32-bit result from the cudnn to 8-bit? if not, how does cudnn decide which 8-bits output from 32-bit computed result?

For the aforementioned API I am using following configurations:

Data Type: INT8_CONFIG
xDesc: CUDNN_DATA_INT8
wDesc: CUDNN_DATA_INT8
convDesc: CUDNN_DATA_INT32
yDesc: CUDNN_DATA_INT8

1 Like