Input images with dtype int8

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Is there any way to use TensorRT with input images with dtype int8? E.g. send images with RGB values in range [0, 255] and do normalisation on the GPU. All samples I’ve seen use dtype float32.

In Tensorflow we can use cast but that layer doesn’t seem to be supported by TensorRT. The benefit of using int8 would be to reduce the size of the data being sent to the GPU/TensorRT.

2

Hello,

Assuming you are creating a Network definition using the C++ API, you’d add the Input layer to the network, with the input dimensions. via

network->addInput(INPUT_BLOB_NAME, dt, Dims3{1, INPUT_H, INPUT_W});

Please reference addINput API https://docs.nvidia.com/deeplearning/sdk/tensorrt-api/c_api/classnvinfer1_1_1_i_network_definition.html#a06a61f560bdf6197afd3368937f62025

which contains a DataType parameter.

virtual ITensor* nvinfer1::INetworkDefinition::addInput	(	const char * 	name,
DataType 	type,
Dims 	dimensions 
)
3

Is there any way to do this with the Python API and converting Tensorflow models?

4

please reference https://docs.nvidia.com/deeplearning/sdk/tensorrt-api/python_api/infer/Graph/LayerBase.html#tensorrt.ILayer

5

Proposed call returns null with this varning:

        ITensor* data = network->addInput(
                m_InputBlobName.c_str(), nvinfer1::DataType::kINT8,
                DimsCHW{static_cast<int>(m_InputC), static_cast<int>(m_InputH),
                        static_cast<int>(m_InputW)});

[E] [TRT] Parameter check failed at: …/builder/Network.cpp::addInput::466, condition: type != DataType::kINT8

6

Did you find how to do this ? Converting my ints to floats/half is very computationnaly expensive.