Same inference speed with Resnet50 for int8 and fp16

Robotics & Edge Computing Jetson & Embedded Systems Jetson Xavier NX
1

hello, i have trained resnet50 classfier with myself dataset(number_classes=7). And i have tried to inference the trained model for int8 mode,but the inference speed is almost same with fp16 mode.I don’t know why. who can tell me the resons to cause it.THANKS.

3

Hi,

Could you share more detail about test of the performance?
Do you use the TensorFlow or our TensorRT trtexec binary?

Thanks.

4

i used pytorch->onnx->tensorrt. So, how can i see the inference precision of each layer? i used python,how can i find the log?

5

Hi,

You can get the layer-level performance result with trtexec + onnx input directly.
For example:

/usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/resnet50/ResNet50.onnx --dumpProfile

[11/30/2020-14:54:20] [I] === Profile (187 iterations ) ===
[11/30/2020-14:54:20] [I]                                                                                                        Layer   Time (ms)   Avg. Time (ms)   Time %
[11/30/2020-14:54:20] [I]                                           (Unnamed Layer* 0) [Convolution] + (Unnamed Layer* 2) [Activation]       86.29             0.46      2.7
[11/30/2020-14:54:20] [I]                                                                                 (Unnamed Layer* 3) [Pooling]       13.92             0.07      0.4
..
[11/30/2020-14:54:20] [I]                                                                           (Unnamed Layer* 179) [ElementWise]        1.58             0.01      0.0
[11/30/2020-14:54:20] [I]                                                                               (Unnamed Layer* 181) [Softmax]        1.66             0.01      0.1
[11/30/2020-14:54:20] [I]                                                                                                        Total     3209.13            17.16    100.0

Or you can add your own profiler with below’s API:
https://docs.nvidia.com/deeplearning/tensorrt/api/python_api/infer/Core/Profiler.html

Thanks.