Turn off tensor core while using TensorRT

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

Hi, I had installed TensorRT and running resnet50 model’s inference successfully.
I know that TensorRT will use tensor cores to achieve the best performance by default.
But I am wondering that is there any way to turn off this flag?
It means that I want to run inference with TensorRT while not using tensor cores in my device.

Thank you very much

Environment

TensorRT Version: 7.2.3
GPU Type: Tesla T4
Nvidia Driver Version: 460.32.03
CUDA Version: 11.2
CUDNN Version: 8
Operating System + Version: ubuntu 16.04
Python Version (if applicable): 3.8.5
TensorFlow Version (if applicable): 2.4.1
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)

Steps To Reproduce

Please include:

  • Exact steps/commands to build your repro
  • Exact steps/commands to run your repro
  • Full traceback of errors encountered
4

Hi,

Sorry for the delayed response.
We don’t have API to disable Tensor Core usage currently.

Thank you.

5

Hi Spolisetty,
Thanks for your replying
So now I know that there is no way to disable Tensor Core, right?
But I have a further question:
I know that Tensor Core only running in mixed or lower precision
So I wonder that do you have a flag in TensorRT to set the data precision??
And how many precision I can choose when I use TensorRT?

Thank you very much

6

Hi,

Please refer developer guide to know more details on available precision types and how to use them. Developer Guide :: NVIDIA Deep Learning TensorRT Documentation
Example:
C++

layer->setPrecision(DataType::kFP16)

Python

layer.precision = trt.fp16

Also you can use mixed precision, for more info Train With Mixed Precision :: NVIDIA Deep Learning Performance Documentation