Dear @Morganh , I have customize one inference script for the mobileNet_V1 classification trained on tao. I have converted the etlt file in engine using tao-converter for batch size 1 and it is working fine. below is the working code for batch size 1. import os import time import cv2 #import mat…

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks [image] Pritam: the problem is with batch. You can refer to tao_deploy/nvidia_tao_deploy/cv/multitask_classification/script…

[image] Pritam: I have converted the etlt file in engine using tao-converter for batch size 1 For 5.0 or later version, after training and exporting, there will be an onnx file instead of onnx file. Can you decrypt the etlt model to onnx file? Refer to tao_toolkit_recipes/tao_forum_faq/FAQ.…

Thanks @Morganh But I have trained on TAO 4.0.1. Can we achieve it using etlt->engine->inference. Because script is working fine with batch size 1. Please help if there is another script or suggestion how to pass batch input. Thanks.

[image] Pritam: But I have trained on TAO 4.0.1. Can we achieve it using etlt->engine->inference. Because script is working fine with batch size 1. How did you run tao-converter? I am afraid you generated a tensorrt engine with fixed batch-size 1. So, I suggest you decode it to .onnx file …

Hi @Morganh Below is the command to convert etlt to engine. I had downloaded the tao-converter and then convert etlt to engine. ./tao-converter ./Models/V2.2/MobileNet_V1/MobileNetV1_Train_Ep51_ZeroConfAug2_SIZE_354X354_FallenObjectClassification_V2.2.etlt \ -k key \ …

Yes, it is fixed batch-size 1 instead of dynamic batch. So it is only working with batch size 1. You can decode to onnx and open it with Netron to double check. For code side, you can leverage Python run LPRNet with TensorRT show pycuda._driver.MemoryError: cuMemHostAlloc failed: out of memory - #…

[image] Pritam: -b 1 You can set to -b 2 and generate a new engine, then run with batch-size 2.

Yes @Morganh I tried with batch size 32 as well. ./tao-converter ./Models/V2.2/MobileNet_V1/MobileNetV1_ReTrain_Ep70_ZeroConfAug2_SIZE_354X354_FallenObjectClassification_V2.2.etlt \ -k key \ -o predictions/Softmax \ -d 3,354,354 \ -i nch…

Hi @Morganh It is not helping actually, It only work with single image not for batch and for single image I am also able to achieve the result but the problem is with batch.

Batch processing on tao engine

Accelerated Computing Intelligent Video Analytics TAO Toolkit

Morganh March 27, 2024, 7:45am 9

You can refer to Python run LPRNet with TensorRT show pycuda._driver.MemoryError: cuMemHostAlloc failed: out of memory to check if it helps.

Topic		Replies	Views
TAO yolov4 batch inference TAO Toolkit	1	965	October 7, 2021
Dynamic batching for tensorrt engine model TAO Toolkit	4	1281	December 13, 2021
Tensorrt inference with batch > 1 TensorRT	4	1525	October 13, 2022
Getting [0, 0, 0, 0] output from TAO multitask classifier TAO Toolkit	4	682	April 6, 2022
Biased result while running classification on image using python script TAO Toolkit	9	1082	September 13, 2021
Unable to batch TensorRT TensorRT	0	590	March 23, 2021
TensorRT running inference with batch size > 1 TensorRT tensorrt	8	3883	January 26, 2021
Tensorrt Batch Inference TensorRT tensorrt	8	1769	December 1, 2020
Jetson Xavier - Inference multiple images Jetson AGX Xavier	6	1157	December 5, 2019
Batch Inference Wrong in Python API TensorRT	14	3785	February 20, 2020

Batch processing on tao engine

Related topics