Sample code of feeding image into TensorTR inference engine

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Here https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#perform_inference_python is an example of performing the inference with engine build with TensorRT.
Suppose I’ll build an engine from VGG16. Is there some code examples of how to pass-in some image file (jpg, png) into the TensorTR inference engine? And how to get the inference result?

Thanks in advance!

2

Hi,

You can use TRT to optimize any model file and generate the TRT engine as long as operations in network are supported. Otherwise you might have to create a custom plugin for that particular operation.

Please refer below link to generate TRT engine file:
https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt-700/tensorrt-developer-guide/index.html#build_engine_python

Also, please refer to below sample for end-end workflow:
https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt-700/tensorrt-sample-support-guide/index.html#end_to_end_tensorflow_mnist
https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt-700/tensorrt-sample-support-guide/index.html#python_samples_section

Thanks

3

Hi!
I’ve read the documents, and I’ve accomplished steps of building an engine following instruction in the second of you link.

My question is there a sample code of using built engine: how to provide the engine with input and receive an output?

There are few words in the doc about it (the link in my first message) but there is no complete example.

4

Hi,

You can refer to below python sample example:
https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt-700/tensorrt-sample-support-guide/index.html#yolov3_onnx

Here inference part is handled in “common.py” and data processing (pre-post) can be found in “data_processing.py” file. Pre-processing of the input data will depend on the application & model you are creating.

Thanks

5

@SunilJB Thank you very much! The code examples are exactly what I was looking for.

I also wanted to execute the Yolo sample but it returns an error.

Traceback (most recent call last):
  File "yolov3_to_onnx.py", line 811, in <module>
    main()
  File "yolov3_to_onnx.py", line 804, in main
    onnx.checker.check_model(yolov3_model_def)
  File "/usr/local/lib/python2.7/dist-packages/onnx/checker.py", line 91, in check_model
    C.check_model(model.SerializeToString())
onnx.onnx_cpp2py_export.checker.ValidationError: Node (086_upsample) has input size 2 not in range [min=3, max=4].

That is strange 'cause everything is very much “standard”, I mean it uses the same as supposed yolo’s config from the yolo git repo, I haven’t changed the script, I executed it on Jetson nano with JetPack 4.3, etc.
Have I missed something?

I put the whole output of yolov3_to_onnx.py here yolov3_to_onnx_output · GitHub

Any recommendations are appreciated!
Thanks!

6

Hi,

Based on the error log, it seems that the “Resize” operation only has two input params, as per the ONNX operation document “Resize” operation should have input size of (3-4). [X : T1, roi : T2, scales : tensor(float), sizes (optional) : tensor(int64)]

Current model just have two input params:
name: “086_upsample” op_type: “Resize” attribute { name: “mode” s: “nearest” type: STRING }

Please refer below link for more details:
https://github.com/onnx/onnx/blob/master/docs/Operators.md#resize

Thanks

7

@SunilJB thank you a lot for your help!
Based on your examples I managed to create a simple code which processes data via generated TensorRT engine.
I put the code in case if someone will need it demo_of_processing_via_tensorrt_engine · GitHub
It’s much simpler than the yolo example since it’s based on just mnist )

In order to try it first generate the LeNet5(MNIST) model following this guide Sample Support Guide :: NVIDIA Deep Learning TensorRT Documentation

Couldn’t fix the yolov3 sample though, since I don’t know well ONNX.
Please let me know if someone got it working

8

Hi,

Could you please elaborate more on the error you are getting in yolov3 samples? Is it the same resize operation issue?
If possible,please share the error log.

Thanks

9

Hi! yes, this is the same resize operation issue, still cannot fix it.
I put the whole output of yolov3_to_onnx.py here yolov3_to_onnx_output · GitHub Did you mean this log?
Please let me know if some additional info will be helpful!
Thanks!

10

Hi,

As mentioned in earlier post, current “Resize” operation implementation in your model is not as per the ONNX supported operation.
You have to either update your model file or write a custom plugin to handle the customized “Resize” operation.

Thanks