Where can I find an example, using python, on how to make inference using a .plan or .serialized file?

Aizzaac · October 6, 2020, 9:37pm

hi

I am learning to use TensorRT. I already have a .engine file. I cannot use “trtexec” because I am doing a benchmark, so I need to read some parameters from the kernel (ex: temperature in GPU).

Where can I find a good example which teaches me how to make inference using TensorRT with python? Line by line (loading the image, loading the labels, loading the .engine, making preprocessing, etc etc). Is there a book, a course (maybe in Deep Learning Institute)?

AastaLLL · October 7, 2020, 3:23am

Hi,

You can check this example for TensorRT python inference:

github.com

AastaNV/TRT_object_detection/blob/master/main.py#L48


      
                  parser.register_input('Input', model.dims)
                  parser.register_output('MarkOutput_0')
                  parser.parse('tmp.uff', network)
                  engine = builder.build_cuda_engine(network)
          
          
        buf = engine.serialize()
                  with open(model.TRTbin, 'wb') as f:
                      f.write(buf)
          
          

          
# create engine
          with open(model.TRTbin, 'rb') as f:
              buf = f.read()
              engine = runtime.deserialize_cuda_engine(buf)
          
          

          
# create buffer
          host_inputs  = []
          cuda_inputs  = []
          host_outputs = []
          cuda_outputs = []

Thanks.

Aizzaac · October 14, 2020, 8:29pm

Hello

I am analyzing the code in your GIT. I have some questions:

Is it for different number of batches?
What is bindings? It seems to be related to different number of batches. I found this definition: “a list of integers representing input and output buffer addresses for the network”
https://github.com/AastaNV/TRT_object_detection/blob/master/main.py#L62

Aizzaac · October 16, 2020, 9:39pm

Hi again

what is the purpose of the graph surgeon here: https://github.com/AastaNV/TRT_object_detection/blob/master/config/model_ssd_mobilenet_v2_coco_2018_03_29.py

Will I need it with the Inception: v1, v2, v3 and v4 models? Those models are for classifying images.

AastaLLL · October 23, 2020, 7:04am

Hi,

Sorry for the late update

1. The batch size is hardcoded into 1 in this sample.
You can modify it based on your use case here:
https://github.com/AastaNV/TRT_object_detection/blob/master/main.py#L35

2.
Binding is the input/output tensor used by the model.
For example, binding index=0 indicates the input image buffer here.
And index=1 or index=2 represents for output bbox and conf layer.

The batch is affected in the first dimension of the buffer.
For example, the input buffer in this sample is [batch, 3, 300, 300]

3. The python script is used for converting a .pb file into .uff file for TensorRT.
You can find a general script for the conversion here:

/usr/src/tensorrt/samples/sampleUffSSD/config.py

Thanks.

Topic		Replies	Views
How do I do an inference of a tensorrt.plan model utilizing python? TensorRT tensorrt , cuda , python	1	539	March 28, 2023
Run inference on a batch of images & parallel inference using cuda on python threads TensorRT tensorrt , cuda	6	2286	January 6, 2022
Can TensorRT do inference in a child thread ? TensorRT	6	2203	August 11, 2020
TensorRT deployment with engine generated from TLT example TensorRT tensorrt	8	776	December 5, 2020
Tensorrt Batch Inference TensorRT tensorrt	8	1570	December 1, 2020
Engine Plan Inference on JetsonTX2 Jetson TX2 tensorrt , python	11	1845	October 18, 2021
Tensorrt lpdnet output Jetson Nano jetson-inference	4	1464	September 12, 2021
Work with batch in TensorRT TensorRT tensorrt , opencv , cuda , tensorflow	20	3809	July 20, 2021
TensorRT Inference error on Jetson nano TensorRT	3	1187	December 6, 2021
TensorRT network definition creaion for the frozen inception V3 model Jetson TX2	4	704	October 18, 2021

Where can I find an example, using python, on how to make inference using a .plan or .serialized file?

Related topics