Sample code of feeding image into TensorTR inference engine

Here https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#perform_inference_python is an example of performing the inference with engine build with TensorRT.
Suppose I’ll build an engine from VGG16. Is there some code examples of how to pass-in some image file (jpg, png) into the TensorTR inference engine? And how to get the inference result?

Thanks in advance!

Hi,

You can use TRT to optimize any model file and generate the TRT engine as long as operations in network are supported. Otherwise you might have to create a custom plugin for that particular operation.

Please refer below link to generate TRT engine file:
https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt-700/tensorrt-developer-guide/index.html#build_engine_python

Also, please refer to below sample for end-end workflow:
https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt-700/tensorrt-sample-support-guide/index.html#end_to_end_tensorflow_mnist
https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt-700/tensorrt-sample-support-guide/index.html#python_samples_section

Thanks

Hi!
I’ve read the documents, and I’ve accomplished steps of building an engine following instruction in the second of you link.

My question is there a sample code of using built engine: how to provide the engine with input and receive an output?

There are few words in the doc about it (the link in my first message) but there is no complete example.

Hi,

You can refer to below python sample example:
https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt-700/tensorrt-sample-support-guide/index.html#yolov3_onnx

Here inference part is handled in “common.py” and data processing (pre-post) can be found in “data_processing.py” file. Pre-processing of the input data will depend on the application & model you are creating.

Thanks

@SunilJB Thank you very much! The code examples are exactly what I was looking for.

I also wanted to execute the Yolo sample but it returns an error.

Traceback (most recent call last):
  File "yolov3_to_onnx.py", line 811, in <module>
    main()
  File "yolov3_to_onnx.py", line 804, in main
    onnx.checker.check_model(yolov3_model_def)
  File "/usr/local/lib/python2.7/dist-packages/onnx/checker.py", line 91, in check_model
    C.check_model(model.SerializeToString())
onnx.onnx_cpp2py_export.checker.ValidationError: Node (086_upsample) has input size 2 not in range [min=3, max=4].

That is strange 'cause everything is very much “standard”, I mean it uses the same as supposed yolo’s config from the yolo git repo, I haven’t changed the script, I executed it on Jetson nano with JetPack 4.3, etc.
Have I missed something?

I put the whole output of yolov3_to_onnx.py here https://gist.github.com/RomanGirin/dcc12b74c3d053c601682558b7af9be0

Any recommendations are appreciated!
Thanks!

Hi,

Based on the error log, it seems that the “Resize” operation only has two input params, as per the ONNX operation document “Resize” operation should have input size of (3-4). [X : T1, roi : T2, scales : tensor(float), sizes (optional) : tensor(int64)]

Current model just have two input params:
name: “086_upsample” op_type: “Resize” attribute { name: “mode” s: “nearest” type: STRING }

Please refer below link for more details:
https://github.com/onnx/onnx/blob/master/docs/Operators.md#resize

Thanks

@SunilJB thank you a lot for your help!
Based on your examples I managed to create a simple code which processes data via generated TensorRT engine.
I put the code in case if someone will need it https://gist.github.com/RomanGirin/e9792c3a34a78a6d6e0729041cbc81b2
It’s much simpler than the yolo example since it’s based on just mnist )

In order to try it first generate the LeNet5(MNIST) model following this guide https://docs.nvidia.com/deeplearning/sdk/tensorrt-sample-support-guide/index.html#end_to_end_tensorflow_mnist

Couldn’t fix the yolov3 sample though, since I don’t know well ONNX.
Please let me know if someone got it working

Hi,

Could you please elaborate more on the error you are getting in yolov3 samples? Is it the same resize operation issue?
If possible,please share the error log.

Thanks

Hi! yes, this is the same resize operation issue, still cannot fix it.
I put the whole output of yolov3_to_onnx.py here https://gist.github.com/RomanGirin/dcc12b74c3d053c601682558b7af9be0 Did you mean this log?
Please let me know if some additional info will be helpful!
Thanks!

Hi,

As mentioned in earlier post, current “Resize” operation implementation in your model is not as per the ONNX supported operation.
You have to either update your model file or write a custom plugin to handle the customized “Resize” operation.

Thanks