TensorRT Batch Inferences : empty outputs

AI & Data Science Deep Learning (Training & Inference) TensorRT
,
4

Hi @AakankshaS,

I have followed this topic to use the dynamic shapes ONNX to TensorRT with dynamic batch size in Python

I have changed my conversion from Darknet to ONNX to get a ONNX model with a N batch size (like in the other topic) yolov4-416.zip - Google Drive

To convert my ONNX file to TensorRT, I have used the command (which works):

trtexec --onnx=yolov4-416.onnx --fp16 --workspace=4096 --explicitBatch --verbose --shapes=000_net:2x3x416x416 --saveEngine=yolov4-416.trt --optShapes=000_net:2x3x416x416 --maxShapes=000_net:4x3x416x416 --minShapes=000_net:1x3x416x416

Now during runtime, I got the following :

  • engine.get_binding_shape(0) -> (-1, 3, 416,416)

  • engine.max_batch_size -> 1

  • engine.num_optimization_profiles  -> 1

So according to the other topic, the input shape and max_batch_size are correct. But my engine only considers that it has one optimization profile (with the -1 dimension), and not even the one under --shapes from the trtexec command. And because of that it, it tries to allocates negative memory space.

Why my optimization profiles from the trtexec command are not present ? And is it the right way to make the dynamic shape for the batch inference ?

1 Like