Create TesorRT with dynamic batch

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

I am creating TensorRT engine from onnx for dynamic batch input.
I have two onnx models. One has input fixed 1x24x94x3.
Another one has dynamic batch so input is Unknownx24x94x3.

I can see all these using Netron.

When networked is parsed we can see input dimension using network->getInput(0)->getDimensions().
For fixed input, I can print as 1x24x94x3.

For dynamic, input shape is -1x24x94x3.

For that -1 input, TensorRT doesn’t accept -1 shape. The error is …/builder/cudnnBuilderGraph.cpp (794) - Assertion Error in checkDimsSanity: 0 (dims.d[i] >= 0).
I am using this dynamic shape sample.

You can test.

The error happened at build preprocessor engine.

bool SampleDynamicReshape::buildPreprocessorEngine(const SampleUniquePtr<nvinfer1::IBuilder>& builder)
{
..........

mPreprocessorEngine = makeUnique(builder->buildEngineWithConfig(*preprocessorNetwork, *preprocessorConfig));
........
}

Moreover, the following three lines also doesn’t take -1 input shape.

bool SampleDynamicReshape::prepare()
{
   .....
    mPredictionInput.resize(mPredictionInputDims);
    mOutput.hostBuffer.resize(mPredictionOutputDims);
    mOutput.deviceBuffer.resize(mPredictionOutputDims);
    return true;
.......
}

It is strange. The program is designed for explicit batch, then dynamic shape -1 is not accepted.
What is wrong with TensorRT?
Or did I make something wrong?

2

@AakankshaS I have attached the minst.onnx converted into dynamic batch.
You can test with this sample.
You can see that TensorRT doesn’t accept -1 in input shape.
You just change .txt to .onnx.
mnist_dynamic_input.txt (25.5 KB)

3

Hi @edit_or,

-1 means dynamic shape on onnx model.
But when using trtexec --shapes, you should provide the real shape for inference as --shapes should use real dimension, not -1.
Dynamic shape means the dimension can change in a range [min, max].
So here optimization profile is used to tell TensorRT the min/max/opt value to build the engine.

Thanks!

4

Thanks. So I can’t input as -1,3,24,94. Instead input as 3,3,24,94.
Then make min and max limit as 1,1,1,1 to 10,3,24,94.
Is it correct? Thank you.

5

@AakankshaS
Your sample has input shape is 1,1,28,28.

Then all are set to batch size 1.

    profile->setDimensions(input->getName(), OptProfileSelector::kMIN, Dims4{1, 1, 1, 1});
    profile->setDimensions(input->getName(), OptProfileSelector::kOPT, Dims4{1, 1, 28, 28});
    profile->setDimensions(input->getName(), OptProfileSelector::kMAX, Dims4{1, 1, 56, 56});
    preprocessorConfig->addOptimizationProfile(profile);

    // Create a calibration profile.
    auto profileCalib = builder->createOptimizationProfile();
    const int calibBatchSize{256};
    // We do not need to check the return of setDimension and setCalibrationProfile here as all dims are explicitly set
    profileCalib->setDimensions(input->getName(), OptProfileSelector::kMIN, Dims4{calibBatchSize, 1, 28, 28});
    profileCalib->setDimensions(input->getName(), OptProfileSelector::kOPT, Dims4{calibBatchSize, 1, 28, 28});
    profileCalib->setDimensions(input->getName(), OptProfileSelector::kMAX, Dims4{calibBatchSize, 1, 28, 28});
    preprocessorConfig->setCalibrationProfile(profileCalib);

So no issue found.

My input has 1,24,94,3.
When I set

    auto profile = builder->createOptimizationProfile();   
    profile->setDimensions(input->getName(), OptProfileSelector::kMIN, Dims4(1, 24, 94, 3));
    profile->setDimensions(input->getName(), OptProfileSelector::kOPT, Dims4(5, 24, 94, 3));
    profile->setDimensions(input->getName(), OptProfileSelector::kMAX, Dims4(10, 24, 94, 3));

I have the following error in buildPredictionEngine(const SampleUniquePtr<nvinfer1::IBuilder>& builder)

input:0: kMAX dimensions in profile 0 are [10,24,94,3] but input has static dimensions [1,24,94,3].
input:0: kOPT dimensions in profile 0 are [5,24,94,3] but input has static dimensions [1,24,94,3].

So how come like that? It doesn’t accept dynamic batch.

6

dynamic shape doesn’t mean we can use dynamic batch size?

7

Any suggestions for this?

8

@AakankshaS Hello, since my application has different number of detentions in an image after first stage model. So at second stage model, dynamic batch size is quite important. May I know tensorrt accepts dynamic batch size? How can I settle the above issues?

9

I found the solution.
Buildpredictionengine has to be same dimension as input dimension.
Buildpreprocessorengine has to be varied dynamic batch.

So inside preprocessorengine,

    auto profile = builder->createOptimizationProfile();    
    profile->setDimensions(input->getName(), OptProfileSelector::kMIN, Dims4(1, 24, 94, 3));
    profile->setDimensions(input->getName(), OptProfileSelector::kOPT, Dims4(5, 24, 94, 3));
    profile->setDimensions(input->getName(), OptProfileSelector::kMAX, Dims4(10, 24, 94, 3));
    preprocessorConfig->addOptimizationProfile(profile);

I hope this helps to somebody who face the same issue as mine.

1 Like