Mixed precision + kSTRICT_TYPES, which type is chosen?

Hi,

What happens when the following is passed to the builder config?

config->setFlag(BuilderFlag::kFP16);
config->setFlag(BuilderFlag::kINT8);
config->setFlag(BuilderFlag::kSTRICT_TYPES);

Which type will it choose, INT8 or FP16? I haven’t found an answer in the documentation.

Hi, Please refer to the below links to perform inference in INT8
https://github.com/NVIDIA/TensorRT/blob/master/samples/opensource/sampleINT8/README.md

Thanks!

Hi,

First of all, the link you post is broken.
Second, please read my question. I’m not asking how to do INT8 inference. I’m asking what happens if I used mixed precision AND strict types.

Hi @carlosgalvezp,

StrictType only force precisions for layers that has:

 layer.setPrecision(xxx)
 layer.setOutputType(xxx)

for other layers we will select the fastest implementation.

Thank you.

Hi,

Thanks for your answer. This is NOT what the documentation says:

5.1.3. Enabling FP16 Inference Using C++

Setting the builder’s Fp16Mode flag indicates that 16-bit precision is acceptable.

config->setFlag(BuilderFlag::kFP16);

This flag allows but does not guarantee that 16-bit kernels are used when building the engine. You can choose to force 16-bit precision by setting the following builder flag:

config->setFlag(BuilderFlag::kSTRICT_TYPES)

The samples are also broken:

Here you set strict types but don’t set a per-layer precision, so what’s the purpose?

Will you please update the documentation and samples?

Best regards,
Carlos

Hi,

Please check Developer Guide :: NVIDIA Deep Learning TensorRT Documentation
We document the usage of

layer.setPrecision(xxx)
layer.setOutputType(xxx)

the kSTRICT_TYPES in the sample is no harm, just has no effect.

Thank you.

Hi,

Well, there is harm in that the documentation is lying to developers. The documentation states that “if you enable strict types, you will get strict types”. Developers write code building on that assumption. If the documentation/samples are wrong, they need to be changed immediately!

Note: I’m not talking about mixed precision here. I’m talking about single precision. As a developer, I want to make sure all my layers run e.g. in INT8. So I enable INT8 and set Strict Types. If this is not enough, your documentation needs to clearly state what else I need to do to achieve that.

Hi @carlosgalvezp,

We found this as known issue. Our team is already working on it. This will be fixed in upcoming releases.

Thank you.

Awesome, thanks!