Now, I am trying to convert an onnx model (a crnn model for ocr) to tensorRT. And I want to use dynamic shape.
I noticed that In TensorRT 7.0, the ONNX parser only supports full-dimensions mode, meaning that your network definition must be created with the explicitBatch flag set., so I add optimization profile as follow.
IOptimizationProfile* profile = builder->createOptimizationProfile(); profile->setDimensions("data", OptProfileSelector::kMIN, Dims4(32, 1, 32, 320)); profile->setDimensions("data", OptProfileSelector::kOPT, Dims4(32, 1, 32, 800)); profile->setDimensions("data", OptProfileSelector::kMAX, Dims4(32, 1, 32, 1280)); config->addOptimizationProfile(profile);
I can convert the model and do inference with batch size 32 successfully. But when I try to do inference with batch size less than 32, there would be en error
Cuda failure 77 which means an illegal memory access was encountered. It looks like that I must use 32 as batch size during inference. So I want to know that can I change the batch size during inference. It’s very serious for my application.
On the other hand, I try to convert resnet with dynamic shape, and I can do inference with different batch size.