I’m having some problems with TLT 3.0 (not TAO) running inside the Docker, inference of Faster-RCNN with exported etlt doesn’t work correctly if I increase the batch size. With batch size > 1, only the results for the 1st image is correct, all other results from the batch are incorrect.
This happens only after exporting to etlt (and consequently with the created engine file, or with the engine created while exporting), using the model from the tlt files works well.
Inferences with TLT 2.0 from my previous trainings with Faster-RCNN worked with any batch size.
Please provide the following information when requesting support.
After setting batch size to 2 on spec file, did you commented the “model” parameter in inference and de-commented the “trt_engine” section and parameters?
Because if this is not done, inference model is created directly from the TLT file and this works well for me too.
Could you try with Resnet18 as backbone? Just to be sure.
(not directly related) May I export/create the etlt file on the latest TAO docker using the tlt model files (those generated during training) from TLT 3.0?
Well, I retrained my model with latest TAO Toolkit (v3.21.08), and now inferences on Jupyter Notebook, using engine file and batch size > 1 do work.
It seems that inferences using ETLT (before conversion) are no longer supported on TAO.
But I still have a problem when I convert the ETLT with both converters for TensorRT 7.1, which is the installed version on my training/testing PC (and on my Jetson AGX Xavier with Jetpack 4.5.1), [cuda102-cudnn80-trt71] / [cuda110-cudnn80-trt71]. My python script for inference works inside the docker used on TAO 3.21.08, but not outside the docker (then I get detections like the images from the initial post for all but the firs image from batch).
So I assume there is either a problem with the conversor for TensorRT 7.1, or models trained on TensorRT 7.2 (from TLT 3.0 or TAO 3.0) doesn’t work on TensorRT 7.1.
If it’s supposed to work on TensorRT 7.1, could you please test if inferences from your training using batch size > 1 works on an environment with TensorRT 7.1?
If you run your inference script on Jetson AGX Xavier, please copy etlt file into Xavier. And then in Xaiver, download the tao-converter for Jetson version, and generate trt engine in Xaiver.
I’ve updated my Jetson’s JetPack to version 4.6 and also the training/testing PC’s TensorRT to v8.0.3.4 and still have the same problem (using theirs corresponding tao-converters).
I’ll try to use DeepStream (never used before) to run inferences with batch size greater than 1 on Jetson with it’s TensorPack’s TensorRT v8.0.1 and see if it work.
Hi,
For TAO 3.0 docker, there is no issue.
For TLT 2.0 docker, there is no NMS plugin, so there is no problem. Can work for batch size >1 .
For TLT 3.0 docker, it cannot work for batch size > 1. please rebuild a new Tensorrt OSS plugin.
You can build on Tensorrt 21.04 branch, it can work for tlt 3.0.
For Jetson devices, need to build a new tensorrt oss plugin as well.