TensorRT

Ragu_23 · January 6, 2022, 6:42am

Description

A clear and concise description of the bug or issue.

Environment

TensorRT Version : ‘8.0.1.6’
GPU Type :
Nvidia Driver Version :
CUDA Version : 11.4
CUDNN Version : 8.2
Operating System + Version : ubuntu 20.04
Python Version (if applicable) : 3.8.10
TensorFlow Version (if applicable) :
PyTorch Version (if applicable) : ‘1.8.1+cu102’
Baremetal or Container (if container which image + tag) :

Relevant Files

Can anyone give a sample code to implement quantize and dequantize nodes in the Tensor RT network using Python API.

Thanks!!

Steps To Reproduce

Please include:

Exact steps/commands to build your repro
Exact steps/commands to run your repro
Full traceback of errors encountered

spolisetty · January 6, 2022, 8:57am

Hi,

Hope following sample may help you.

github.com

NVIDIA/sampleQAT/blob/master/build_engine.py#L65


      
                      if not shapes:
                          shapes = [override_shape(inp.shape)] * 3
                          print("Overriding dynamic input shape {:} to {:}. If this is incorrect, for input tensor: {:}, please provide tuples for min, opt, and max shapes containing values: {:} with dynamic dimensions replaced,".format(inp.shape, shapes[0], inp.name, inp.shape))
                      min, opt, max = shapes
                      profile.set_shape(inp.name, min, opt, max)
                      print("Setting input: {:} shape to min: {:}, opt: {:}, max: {:}".format(inp.name, min, opt, max))
              if not profile:
                  print("Profile is not valid, please provide profile data. Note: profile was: {:}".format(profile_shapes))
              return profile
          
          
def preprocess_network(network):
              """
              Add quantize and dequantize nodes after the input placeholder.
              The scale values are currently picked on emperical basis. Ideally,
              you need to add these nodes during quantization aware training and 
              learn the dynamic ranges of input node.
              """
              quant_scale = np.array([1.0/127.0], dtype=np.float32)
              dequant_scale = np.array([127.0/1.0], dtype=np.float32)
              # Zero point is always zero for quantization in TensorRT.
              zeros = np.zeros(shape=(1, ), dtype=np.float32)

Thank you.

Ragu_23 · January 6, 2022, 12:19pm

Thanks for your reply.

The above example converts an ONNX model to TensorRT. Can you give an example where you convert the PyTorch QAT model to TensorRT without converting it into ONNX?

spolisetty · January 6, 2022, 12:38pm

Hi,

Hope following may help you.

Thank you.

Ragu_23 · January 6, 2022, 12:50pm

These two documents don’t have any example code to implement for the direct QAT Torch model to TensorRT without converting it into ONNX.
Can anyone share the related documents with code samples.

Thanks

spolisetty · January 19, 2022, 4:38pm

Hi,

Currently we do not have much examples apart from above and following links.

Thank you.