Mix precision: I have to give a dynamic range to FLOAT32 layer?

I want to use mix precision, some layers ues int8 and others use fp32.
So I set all int8 layers’ dynamic range, but not fp32.
Error occure as below:

[TensorRT] ERROR: ../builder/cudnnBuilder2.cpp (1777) - Misc Error in createRegionScalesFromTensorScales: -1 (Could not find scales for tensor (Unnamed Layer* 2) [Pooling]_output.)
[TensorRT] ERROR: ../builder/cudnnBuilder2.cpp (1777) - Misc Error in createRegionScalesFromTensorScales: -1 (Could not find scales for tensor (Unnamed Layer* 2) [Pooling]_output.)

So should I set all float32 layers’s dynamic range?
Will the dynamic range be used in inference for fp32 layer?
Can I give a fake dynamic range to fp32 layer?

Hi,

If dynamic range for a tensor is required then inference will fail.

Setting the precision, requests TensorRT to use a layer implementation whose inputs and outputs match the preferred types. By default, TensorRT will choose such an implementation only if it results in a higher-performance network. If an implementation at a higher precision is faster, TensorRT will use it and issue a warning.

You can override this behavior by making the type constraints strict.
IBuilderConfig * config = builder->createBuilderConfig();config->setFlag(BuilderFlag::kSTRICT_TYPES)

Please follow below link for more details and sample example:
https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt-700/tensorrt-developer-guide/index.html#mixed_precision
https://github.com/NVIDIA/TensorRT/tree/release/7.0/samples/opensource/sampleINT8API

Thanks

I had refer to it, but it not help me.
Now the problem I met is:
I want set some layer to int8, and others use fp32.But error occur when do that.

[TensorRT] ERROR: ../builder/cudnnBuilder2.cpp (1777) - Misc Error in createRegionScalesFromTensorScales: -1 (Could not find scales for tensor (Unnamed Layer* 11) [Activation]_output.)
[TensorRT] ERROR: ../builder/cudnnBuilder2.cpp (1777) - Misc Error in createRegionScalesFromTensorScales: -1 (Could not find scales for tensor (Unnamed Layer* 11) [Activation]_output.)

For sample,if I set top 10 layer to int8, the error message will tell me the 11th layer don’t have scale.(I have set the 11th layer precision to fp32)

So how can I set some layer to int8, and other to fp32?

for i in range(network.num_inputs):
     input_tensor = network.get_input(i)
     input_tensor.set_dynamic_range(-1, 1)

for i in range(network.num_layers):
    layer = network.get_layer(i)
    if i < 10:
        layer.precision = trt.int8
        for j in range(layer.num_outputs):
            t = layer.get_output(j)
            t.set_dynamic_range(-10, 10)
            layer.set_output_type(j, trt.int8)
    else:
        layer.precision = trt.float32
        for j in range(layer.num_outputs):
            layer.set_output_type(j, trt.float32)
 
builder.strict_type_constraints = True

Here is my code, Could you tell me what wrong with it?

I have do a experiment, it seem that all tensor should be set dynamic range.
But if the layer precision is fp32, the dynamic won’t be use!

Hi,

Can you provide the following information so we can better help?
Provide details on the platforms you are using:
o Linux distro and version
o GPU type
o Nvidia driver version
o CUDA version
o CUDNN version
o Python version [if using python]
o Tensorflow version
o TensorRT version
o If Jetson, OS, hw versions

Thanks

TRT6 should be given all layer dynamic range value, else it will use fp32.
Oh god!

o Linux distro and version: Ubuntu16.04
o GPU type : P4
o Nvidia driver version: 418.39
o CUDA version: 10.1
o CUDNN version: 7.6
o Python version [if using python]: 3.7
o Tensorflow version : None
o TensorRT version: 6.0.1.5
o If Jetson, OS, hw versions: None

Now I have to give all layer dynamic range, can I only offer some layer dynmaic range value, and other layers use fp32? I don’t want use calibration data.

Hi,

Are you using a resnet like network in this case?
Could you please share the model file or offer a verbose log so we can analyze it better?

Thanks

Yes, I just use ResNet50 pretrained by pytorch.
I set dynamic range like this:

for i in range(network.num_inputs):
    input_tensor = network.get_input(i)
    input_tensor.set_dynamic_range(-2.64, 2.64)

for i in range(1, network.num_layers-1):
    layer = network[i]
    if layer.type == trt.tensorrt.LayerType.SCALE and network[i-1].type == trt.tensorrt.LayerType.CONVOLUTION and network[i+1].type == trt.tensorrt.LayerType.ACTIVATION:
        dynamic_range = RECORDER[layer.name]

        # set CONVOLUTION layer to int8
        bot_layer = network[i-1]
        bot_layer.precision = trt.int8
        for j in range(bot_layer.num_outputs):
            t = bot_layer.get_output(j)
            t.set_dynamic_range(-dynamic_range, dynamic_range)
            bot_layer.set_output_type(j, trt.int8)

        # set SCALE layer to int8
        layer.precision = trt.int8
        for j in range(layer.num_outputs):
            t = layer.get_output(j)
            t.set_dynamic_range(-1, 1)
            layer.set_output_type(j, trt.float32)
        
        # set ACTIVATION layer to int8
        top_layer = network[i+1]
        top_layer.precision = trt.int8
        for j in range(top_layer.num_outputs):
            t = top_layer.get_output(j)
            t.set_dynamic_range(-1, 1)
            top_layer.set_output_type(j, trt.float32)
    else:
        if layer.type == trt.tensorrt.LayerType.ACTIVATION or layer.type == trt.tensorrt.LayerType.CONVOLUTION:
            continue
      
        # set other layer to fp32
        layer.precision = trt.float32
        for j in range(layer.num_outputs):
            layer.set_output_type(j, trt.float32)

In my experiment, only ACTIVATION layer’s dynamic can work. The dynamic of CONVOLUTION and SCALE layer don’t work(No matter what the value is, output is same)

Hi,

Fix will be available in next release. Please stay tuned.

Thanks