Missing dynamic range for tensor warnings when converting YOLOv4 models with INT8 precision

Problem
We get the following warnings when converting a YOLOv4 (trained with QAT) .etlt model to a TensorT engine with tao converter.

Missing dynamic range for tensor <xx>, expect fall back to non-int8 implementation for any layer consuming or producing given tensor

The converted models works fine with good accuracy (similar to the original .etlt model’s accuracy)

Seems like the most of the warnings are for the post processing layers that has been added, and thus we can potentially ignore them. Anyway thought to post here and get an idea if this is expected.

following is the full log when we did the conversion with default converter provided with tao docker. We tested with stand alone tao-conveter and got the same results.

INFO] ----------------------------------------------------------------
[INFO] Input filename:   /tmp/fileHGZ0Pt
[INFO] ONNX IR version:  0.0.7
[INFO] Opset version:    12
[INFO] Producer name:    
[INFO] Producer version: 
[INFO] Domain:           
[INFO] Model version:    0
[INFO] Doc string:       
[INFO] ----------------------------------------------------------------
[WARNING] /trt_oss_src/TensorRT/parsers/onnx/onnx2trt_utils.cpp:273: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[WARNING] /trt_oss_src/TensorRT/parsers/onnx/onnx2trt_utils.cpp:299: One or more weights outside the range of INT32 was clamped
[WARNING] /trt_oss_src/TensorRT/parsers/onnx/onnx2trt_utils.cpp:299: One or more weights outside the range of INT32 was clamped
[WARNING] /trt_oss_src/TensorRT/parsers/onnx/onnx2trt_utils.cpp:299: One or more weights outside the range of INT32 was clamped
[WARNING] /trt_oss_src/TensorRT/parsers/onnx/onnx2trt_utils.cpp:299: One or more weights outside the range of INT32 was clamped
[WARNING] /trt_oss_src/TensorRT/parsers/onnx/onnx2trt_utils.cpp:299: One or more weights outside the range of INT32 was clamped
[WARNING] /trt_oss_src/TensorRT/parsers/onnx/onnx2trt_utils.cpp:299: One or more weights outside the range of INT32 was clamped
[WARNING] /trt_oss_src/TensorRT/parsers/onnx/onnx2trt_utils.cpp:299: One or more weights outside the range of INT32 was clamped
[WARNING] /trt_oss_src/TensorRT/parsers/onnx/onnx2trt_utils.cpp:299: One or more weights outside the range of INT32 was clamped
[WARNING] /trt_oss_src/TensorRT/parsers/onnx/onnx2trt_utils.cpp:299: One or more weights outside the range of INT32 was clamped
[WARNING] /trt_oss_src/TensorRT/parsers/onnx/onnx2trt_utils.cpp:299: One or more weights outside the range of INT32 was clamped
[WARNING] /trt_oss_src/TensorRT/parsers/onnx/onnx2trt_utils.cpp:299: One or more weights outside the range of INT32 was clamped
[WARNING] /trt_oss_src/TensorRT/parsers/onnx/onnx2trt_utils.cpp:299: One or more weights outside the range of INT32 was clamped
[WARNING] /trt_oss_src/TensorRT/parsers/onnx/onnx2trt_utils.cpp:299: One or more weights outside the range of INT32 was clamped
[WARNING] /trt_oss_src/TensorRT/parsers/onnx/onnx2trt_utils.cpp:299: One or more weights outside the range of INT32 was clamped
[WARNING] /trt_oss_src/TensorRT/parsers/onnx/onnx2trt_utils.cpp:299: One or more weights outside the range of INT32 was clamped
[WARNING] /trt_oss_src/TensorRT/parsers/onnx/onnx2trt_utils.cpp:299: One or more weights outside the range of INT32 was clamped
[WARNING] /trt_oss_src/TensorRT/parsers/onnx/onnx2trt_utils.cpp:299: One or more weights outside the range of INT32 was clamped
[WARNING] /trt_oss_src/TensorRT/parsers/onnx/onnx2trt_utils.cpp:299: One or more weights outside the range of INT32 was clamped
[WARNING] /trt_oss_src/TensorRT/parsers/onnx/onnx2trt_utils.cpp:299: One or more weights outside the range of INT32 was clamped
[WARNING] /trt_oss_src/TensorRT/parsers/onnx/onnx2trt_utils.cpp:299: One or more weights outside the range of INT32 was clamped
[WARNING] /trt_oss_src/TensorRT/parsers/onnx/onnx2trt_utils.cpp:299: One or more weights outside the range of INT32 was clamped
[INFO] No importer registered for op: BatchedNMSDynamic_TRT. Attempting to import as plugin.
[INFO] Searching for plugin: BatchedNMSDynamic_TRT, plugin_version: 1, plugin_namespace: 
[INFO] Successfully created plugin: BatchedNMSDynamic_TRT
[INFO] Detected input dimensions from the model: (-1, 3, 416, 416)
[INFO] Model has dynamic shape. Setting up optimization profiles.
[INFO] Using optimization profile min shape: (1, 3, 416, 416) for input: Input
[INFO] Using optimization profile opt shape: (1, 3, 416, 416) for input: Input
[INFO] Using optimization profile max shape: (1, 3, 416, 416) for input: Input
[WARNING] Half2 support requested on hardware without native FP16 support, performance will be negatively affected.
[INFO] Reading Calibration Cache for calibrator: EntropyCalibration2
[INFO] Generated calibration scales using calibration cache. Make sure that calibration cache has latest scales.
[INFO] To regenerate calibration cache, please delete the existing one. TensorRT will generate a new calibration cache.
[WARNING] Missing dynamic range for tensor (Unnamed Layer* 216) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor (Unnamed Layer* 326) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor (Unnamed Layer* 428) [Constant]_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor b3_final_trans_bn/batchnorm/add_1:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor b4_final_trans_bn/batchnorm/add_1:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor convolution_output, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor bg_anchor/FirstDimTile/Tile:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor bg_anchor/Reshape:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor bg_bbox_processor/Reshape:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor convolution_output1, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor bg_bbox_processor/Reshape:0_cropping0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor bg_bbox_processor/Reshape:0_cropping01, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor bg_bbox_processor/Reshape:0_cropping02, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor bg_bbox_processor/sub:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor bg_bbox_processor/Exp:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor bg_bbox_processor/Reshape_1:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor md_anchor/FirstDimTile/Tile:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor md_anchor/Reshape:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor md_bbox_processor/Reshape:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor convolution_output2, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor md_bbox_processor/Reshape:0_cropping0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor md_bbox_processor/Reshape:0_cropping01, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor md_bbox_processor/Reshape:0_cropping02, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor md_bbox_processor/sub:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor md_bbox_processor/Exp:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor md_bbox_processor/Reshape_1:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor sm_anchor/FirstDimTile/Tile:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor sm_anchor/Reshape:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor sm_bbox_processor/Reshape:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor sm_bbox_processor/Reshape:0_cropping0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor sm_bbox_processor/Reshape:0_cropping01, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor sm_bbox_processor/Reshape:0_cropping02, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor sm_bbox_processor/sub:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor sm_bbox_processor/Exp:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor sm_bbox_processor/Reshape_1:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/Reshape:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor cls/Reshape:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/Reshape:0_cropping0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/Reshape:0_cropping01, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/Reshape:0_cropping02, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/Reshape:0_cropping03, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/Reshape:0_cropping04, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/Reshape:0_cropping05, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/Reshape:0_cropping06, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/Reshape:0_cropping07, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/Reshape:0_cropping08, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/Reshape:0_cropping09, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor cls/Reshape:0_cropping0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor cls/Reshape:0_cropping01, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/add_1:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/mul_4:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/add:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/mul_6:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor cls, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/sub:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/sub_1:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/add_2:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/add_3:0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor BatchedNMS_1, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor BatchedNMS_2, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor BatchedNMS_3, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor bg_bbox_processor/Reshape:0 copy 0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor bg_bbox_processor/Reshape:0 copy 1, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor bg_bbox_processor/Reshape:0 copy 2, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor md_bbox_processor/Reshape:0 copy 0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor md_bbox_processor/Reshape:0 copy 1, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor md_bbox_processor/Reshape:0 copy 2, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor sm_bbox_processor/Reshape:0 copy 0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor sm_bbox_processor/Reshape:0 copy 1, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor sm_bbox_processor/Reshape:0 copy 2, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor cls/Reshape:0 copy 0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor cls/Reshape:0 copy 1, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/Reshape:0 copy 0, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/Reshape:0 copy 1, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/Reshape:0 copy 2, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/Reshape:0 copy 3, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/Reshape:0 copy 4, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/Reshape:0 copy 5, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/Reshape:0 copy 6, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/Reshape:0 copy 7, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/Reshape:0 copy 8, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/Reshape:0 copy 9, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor bg_bbox_processor/sub:0 copy, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor bg_bbox_processor/Exp:0 copy, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor bg_bbox_processor/Reshape:0_cropping0 copy, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor bg_anchor/Reshape:0 copy, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor bg_bbox_processor/Reshape_1:0 copy, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor md_bbox_processor/sub:0 copy, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor md_bbox_processor/Exp:0 copy, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor md_bbox_processor/Reshape:0_cropping0 copy, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor md_anchor/Reshape:0 copy, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor md_bbox_processor/Reshape_1:0 copy, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor sm_bbox_processor/sub:0 copy, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor sm_bbox_processor/Exp:0 copy, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor sm_bbox_processor/Reshape:0_cropping0 copy, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor sm_anchor/Reshape:0 copy, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor sm_bbox_processor/Reshape_1:0 copy, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor encoded_bg/concat:0 copy, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor encoded_md/concat:0 copy, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor encoded_sm/concat:0 copy, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/sub:0 copy, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/sub_1:0 copy, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/add_2:0 copy, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing dynamic range for tensor box/add_3:0 copy, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[INFO] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
[INFO] Detected 1 inputs and 4 output network tensors.
2021-09-09 17:11:24,836 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

Information
• Hardware - GTX 1060 (GPU_ARCHS = 6.1) and Quadro RTX4000 (GPU_ARCHS = 7.5 )
• Network Type - Yolo_v4 (CSPDarknet53)
• TAO version - 3.21.08

That means the layers do not have dynamic range in calibration file. You can ignore the warnings.
Or you can add "-c cal.bin“ when you generate trt engine.

Or you can add "-c cal.bin“ when you generate trt engine

Yes. We are adding the calibration cache generated during export step when converting the model. And we get good results with this INT8 model.

for reference, the following is the command we use:


tao converter -k <key> -d 3,416,416 -o BatchedNMS -c /workspace/TAO_yolov4_config_1_120epoch_70_20_10_split_QAT/INT8/cal_cspdarknet53_QAT_1160batches.bin -e /workspace/tao-experiments/models/int8_qat.engine -m 1 -t int8 -i nchw -p Input,1x3x416x416,1x3x416x416,1x3x416x416 /workspace/TAO_yolov4_config_1_120epoch_70_20_10_split_QAT/INT8/yolov4_cspdarknet53_epoch_120_QAT_int8_1160batches.etlt

Is that what you meant? thanks

Yes.

Yes, we have built the INT8 engines with the calibration cache files generated during the export steps.

It seems like most of the warnings are for layers added for post processing ( and later supported by the TensorRT OSS plugins). Just wanted to know the reason behind this, the models works fine.