thanks. This is exactly what I wanted to know: if it has been configured to fall back to FP16 in addition to FP32 when we set -t int8
In my undertand, based on TensorRT documentation, setting strict int8 type (without letting it fallback to FP32) is not optional, so I thought its good if there an option to set int8 as the precision without setting -s
so that it will use INT8/FP32 mixed precision. Please let me know if my understanding is not correct.
However, I tested with -s
(during export and convert step both) option with a model trained with QAT, and got the same accuracy. So setting strict type didn’t had a negative impact on results as I assumed. However not sure if thats because the model was trained with QAT.
Using -s
made the Half2 support
warning goes away, but it prompted additional warnings (in addition to the ones mentioned here (Missing dynamic range for tensor warnings when converting YOLOv4 models with INT8 precision - #6 by kgksl)
the additional warnings are as follows:
[WARNING] No implementation of layer b3_final_trans_bn obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer b3_final_trans_mish_clip obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer b4_final_trans_bn obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer b4_final_trans_mish_clip obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer b5_final_trans_mish/Relu6:0_pooling_1 obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer b5_final_trans_mish/Relu6:0_pooling_2 obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer conv_big_object obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer Resize obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer conv_mid_object obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer Resize1 obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer conv_sm_object obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer bg_anchor/Const:0 obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer bg_anchor/Identity:0_tile obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer bg_anchor/Reshape_reshape obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer Transpose + bg_reshape + bg_bbox_processor/Reshape_reshape obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer PWN(PWN(bg_bbox_processor/Sigmoid, bg_bbox_processor/mul/y:0 + (Unnamed Layer* 294) [Shuffle] + bg_bbox_processor/mul), PWN(bg_bbox_processor/sub/y:0 + (Unnamed Layer* 299) [Shuffle], bg_bbox_processor/sub)) obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer PWN(PWN(PWN(bg_bbox_processor/add/y:0 + (Unnamed Layer* 290) [Shuffle] + bg_bbox_processor/add, bg_bbox_processor/sub_1), bg_bbox_processor/Minimum_min), bg_bbox_processor/Exp) obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer bg_bbox_processor/Reshape_1_reshape obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer md_anchor/Const:0 obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer md_anchor/Identity:0_tile obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer md_anchor/Reshape_reshape obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer Transpose1 + md_reshape + md_bbox_processor/Reshape_reshape obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer PWN(PWN(md_bbox_processor/Sigmoid, md_bbox_processor/mul/y:0 + (Unnamed Layer* 410) [Shuffle] + md_bbox_processor/mul), PWN(md_bbox_processor/sub/y:0 + (Unnamed Layer* 415) [Shuffle], md_bbox_processor/sub)) obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer PWN(PWN(PWN(md_bbox_processor/add/y:0 + (Unnamed Layer* 406) [Shuffle] + md_bbox_processor/add, md_bbox_processor/sub_1), md_bbox_processor/Minimum_min), md_bbox_processor/Exp) obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer md_bbox_processor/Reshape_1_reshape obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer sm_anchor/Const:0 obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer sm_anchor/Identity:0_tile obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer sm_anchor/Reshape_reshape obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer Transpose2 + sm_reshape + sm_bbox_processor/Reshape_reshape obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer PWN(PWN(sm_bbox_processor/Sigmoid, sm_bbox_processor/mul/y:0 + (Unnamed Layer* 513) [Shuffle] + sm_bbox_processor/mul), PWN(sm_bbox_processor/sub/y:0 + (Unnamed Layer* 517) [Shuffle], sm_bbox_processor/sub)) obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer PWN(PWN(PWN(sm_bbox_processor/add/y:0 + (Unnamed Layer* 510) [Shuffle] + sm_bbox_processor/add, sm_bbox_processor/sub_1), sm_bbox_processor/Minimum_min), sm_bbox_processor/Exp) obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer sm_bbox_processor/Reshape_1_reshape obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer cls/Reshape_reshape obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer PWN(cls/Sigmoid_1, PWN(cls/Sigmoid, cls/mul)) obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer box/Reshape_reshape obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer PWN(box/mul_1, box/add_1) obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer PWN(box/mul_3, box/mul_4/x:0 + (Unnamed Layer* 774) [Shuffle] + box/mul_4) obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer box/sub obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer PWN(box/mul, box/add) obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer PWN(box/mul_2, box/mul_6/x:0 + (Unnamed Layer* 778) [Shuffle] + box/mul_6) obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer box/sub_1 obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer box/add_2 obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer box/add_3 obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation of layer BatchedNMS_N obeys the requested constraints in strict mode. No conforming implementation was found i.e. requested layer computation precision and output precision types are ignored, using the fastest implementation.
[WARNING] No implementation obeys reformatting-free rules, at least 1 reformatting nodes are needed, now picking the fastest path instead.
However, as I mentioned, despite the above warnings, I got the same accuracy with -s
as without -s
and it was similar to FP32 original accuracy when the model was trained with QAT.