TensorRT mix precision with supported hard drives

364083042 · April 19, 2021, 8:55am

Description

Hi! I am currently trying to transfer the tensorrt model from the server to my computer, and I don’t know if the graphic card on my computer will support the inference process.
The reason is that the tensorrt model set up on the server has enabled the mix-precision(float16 and int8 mode). So I wonder what the minimum requirement of the graphic card is(Titan V on the server) that allows the mix-precision model to do the inference.

Environment

*TensorRT Version: 7.2.2.3
GPU Type: server: Titan V; PC:GTX1050
Nvidia Driver Version: 450.51.05
CUDA Version: 11.0
CUDNN Version: 8.0.4
Operating System + Version: Ubuntu 18.04
Python Version (if applicable): 3.7
TensorFlow Version (if applicable): 1.14
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

NVES · April 19, 2021, 9:07am

Hi ,
We recommend you to check the supported features from the below link.

You can refer below link for all the supported operators list.
For unsupported operators, you need to create a custom plugin to support the operation

github.com

onnx/onnx-tensorrt/blob/main/docs/operators.md

<!--- SPDX-License-Identifier: Apache-2.0 -->

# Supported ONNX Operators

TensorRT 8.4 supports operators up to Opset 17. Latest information of ONNX operators can be found [here](https://github.com/onnx/onnx/blob/master/docs/Operators.md)

TensorRT supports the following ONNX data types: DOUBLE, FLOAT32, FLOAT16, INT8, and BOOL

> Note: There is limited support for INT32, INT64, and DOUBLE types. TensorRT will attempt to cast down INT64 to INT32 and DOUBLE down to FLOAT, clamping values to `+-INT_MAX` or `+-FLT_MAX` if necessary.

See below for the support matrix of ONNX operators in ONNX-TensorRT.

## Operator Support Matrix

| Operator                  | Supported  | Supported Types | Restrictions                                                                                                           |
|---------------------------|------------|-----------------|------------------------------------------------------------------------------------------------------------------------|
| Abs                       | Y          | FP32, FP16, INT32 |
| Acos                      | Y          | FP32, FP16 |
| Acosh                     | Y          | FP32, FP16 |
| Add                       | Y          | FP32, FP16, INT32 |

This file has been truncated. show original

Thanks!

364083042 · April 19, 2021, 9:17am

Thank you for your reply!
I have checked the fourth section "Hardware and precision"in the link:
Support Matrix :: NVIDIA Deep Learning TensorRT Documentation.
I found that the int8 mode requires at least 6.1 compute capability (Tesla P4), so for my own PC, if I use general GTX graphic card, I would guess the least requirement is GTX1060?

spolisetty · April 19, 2021, 1:00pm

Hi @364083042,

Yes. You may find this link is useful, to know CUDA compute capability of the devices.

Thank you.