TensorRT deployment with engine generated from TLT example

[terry.l.lee]

Looking for sample python code to run TensorRT with engine that was created with TLT example.

After going through the tlt delectNet_v2 example, I was able to generating the engine file “resnet18_detector_baru.engine”.
I would like to run the engine with TensorRT on Jetson nano. However, I’m not able to find examples or clear instructions to load the engine into TensorRT. Any suggestion or direction would be greatly appreciated.
Thanks,
Terry

[terry.l.lee]

The following two blob helps. However, did not provide detail.

Use .engine file in python

I used Transfer Learning Toolkit to train resnet18detectnetv2 for custom dataset. I then used the tlt-converter to convert the .etlt model into an .engine file. I am able to deploy both the .etlt and .engine file in DeepStream and it works. But now I need to deploy the model in python and I am not able to find how to load the .engine model in python. Any suggestions?

how to use created .engine file for inference

hi, I created mnist.engine file using this sample to have the understanding of how tensorrt works using this link https://docs.nvidia.com/deeplearning/sdk/tensorrt-api/topics/topics/workflows/caffe_to_tensorrt.html I am not able to understand hoe to use this mnist.engine file for inference. My ideal point of learning to use tensorrt is I want to create inference engine for object detection using Faster RCNN architecture and Caffe framework with Titan X GPU.how can I use Tensorrt for this? I…

I end up with the following code:
f = open(’/home/trafficcountuser/sl/testpy/resnet18_detector_baru.engine’, ‘rb’)
runtime = trt.Runtime(TRT_LOGGER)
engine = runtime.deserialize_cuda_engine(f.read())

sourceFileDir = os.path.dirname(os.path.abspath(file))
sourceFileDir = os.path.abspath(os.path.join(sourceFileDir,"…/"))
fileName = os.path.join(sourceFileDir, “runTime”, “Image640x360.bmp”)
thisImage = cv2.imread(fileName)
result = engine.infer(thisImage) # Single function for inference

There is an error on the last line:

Exception has occurred: AttributeError
‘tensorrt.tensorrt.ICudaEngine’ object has no attribute ‘infer’

Any idea about this message?

[AakankshaS]

Hi @terry.l.lee,
Can you try running the below command on your engine?
trtexec --loadEngine=your_engine-file --batch=1
and see if you are getting the inference?

Thanks!

[terry.l.lee]

Hi AahankshaS,
Thanks a lot for your helps. After spending weeks on TLT, I feel that I’m close on getting it all working.

After building ertexec, I was able to run the command with:
/usr/src/tensorrt/bin/trtexec --loadEngine=resnet18_detector_baru.engine --batch=1

The output seems indicate getting the inference.

What is the next step ?

I was trying to run the .engine with “detect_objects.py”, however, run into import model issue.

Under “inference.py”, the “import pycuda.driver as cuda”
I’m having some difficulty installing the pycuda model
There are many posts discuss the installation. I have not be able to make it work yet.

Please let me know if I’m at the right path.
Thanks,
Terry

[AakankshaS]

Hi @terry.l.lee,
The issue is with your inference file then.
trtexec is an alternate option to get the inference.
Have you created any file named tensorrt?

@terry.l.lee

tensorrt.tensorrt.ICudaEngine’ object has no attribute ‘infer’

Thanks!

[terry.l.lee]

I have not created file name tensorrt.
Is that on Jetson nano or during TLT process?

Where can I find direction/instruction on creating the tensorrt file?
Thanks again.

[terry.l.lee]

Hi Aahanksha,
I’m not having much luck on the next step after generating the .engin file. Please help out with example or direction.
Thanks,
Terry

[AakankshaS]

Hi @terry.l.lee,
I assume, this example should be useful for you to proceed.

Working With TensorRT Samples :: NVIDIA Deep Learning TensorRT Documentation

You may find more samples here.

Working With TensorRT Samples :: NVIDIA Deep Learning TensorRT Documentation

Thanks!

[wade.wang]

Hi @AakankshaS
You said trtexec --loadEngine=your_engine-file --batch=1 can be used for inference ? But i don’t see the input and output in this short command line ? and how the trtexec how to know what kind of inference to do, because there are many kind of infernce task.