How to set context inputDims when there are two inputs to the network and two outputs

shir · September 2, 2020, 1:24pm

Description

In keras my input looks like this:

Its a list of two tensors

And my outputs:

I’m working with the “simpleOnnx” example and I wonder how to set the input tensor and output tensors:
(Based my conde on https://developer.nvidia.com/blog/speed-up-inference-tensorrt/)

For one inputTensor I used this code:

    Dims dims_i{engine->getBindingDimensions(0)};
    Dims3 inputDims{batchSize, dims_i.d[1], dims_i.d[2]};
    context->setBindingDimensions(0, inputDims);
    launchInference(context.get(), stream, inputTensor, outputTensor, bindings, batchSize);

Any suggestions? or maybe an example for multiple inputs and outputs?

Environment

TensorRT Version : 7.0.0-1
GPU Type : RTX
Nvidia Driver Version :
CUDA Version : 11.0
CUDNN Version :
Operating System + Version : Ubuntu 18.04
Python Version (if applicable) : 3.6
TensorFlow Version (if applicable) : 2.3
PyTorch Version (if applicable) :
Baremetal or Container (if container which image + tag) :

Update:
I’ve tried to change
Dims3 inputDims{batchSize, dims_i.d[1], dims_i.d[2]}
to
Dims4 inputDims{batchSize, 2, dims_i.d[1], dims_i.d[2]}

But I’m getting:
size outputTensor : 4
size inputTensor: 495
0 0 0 0 ERROR: Parameter check failed at: engine.cpp::resolveSlots::1092, condition: allInputDimensionsSpecified(routine)

AakankshaS · September 2, 2020, 6:57pm

Hi @shir

Could you please check if you have set the input size for the preprocessor during inference using setBindingDimensions?
Please refer below sample:

github.com

NVIDIA/TensorRT/blob/07ed9b57b1ff7c24664388e5564b17f7ce2873e5/samples/opensource/sampleDynamicReshape/sampleDynamicReshape.cpp#L239


std::default_random_engine generator{rd()};
std::uniform_int_distribution<int> digitDistribution{0, 9};
int digit = digitDistribution(generator);

Dims inputDims = loadPGMFile(locateFile(std::to_string(digit) + ".pgm", mParams.dataDirs));
mInput.deviceBuffer.resize(inputDims);
CHECK(cudaMemcpy(
    mInput.deviceBuffer.data(), mInput.hostBuffer.data(), mInput.hostBuffer.nbBytes(), cudaMemcpyHostToDevice));

// Set the input size for the preprocessor
mPreprocessorContext->setBindingDimensions(0, inputDims);
// We can only run inference once all dynamic input shapes have been specified.
if (!mPreprocessorContext->allInputDimensionsSpecified())
{
    return false;
}

// Run the preprocessor to resize the input to the correct shape
std::vector<void*> preprocessorBindings = {mInput.deviceBuffer.data(), mPredictionInput.data()};
// For engines using full dims, we can use executeV2, which does not include a separate batch size parameter.
bool status = mPreprocessorContext->executeV2(preprocessorBindings.data());

Thanks!

shir · September 3, 2020, 2:53pm

Hi,
Thanks for you answer.
I have set the bindingDimensions like this:

Dims3 inputDims{1, 55, 3};
context->setBindingDimensions(0, inputDims);

I’m not sure that it’s the right way cause as I said above, I have two input tensors with the size of (55,3)
But I’m still getting this error:
ERROR: Parameter check failed at: engine.cpp::resolveSlots::1092, condition: allInputDimensionsSpecified(routine

Any suggestions on how to set the “binding dimensions”?
Any example on how to use with multiple outputs and inputs?

Update:
Fixed it by setting the binding dimensions for both of my inputs, like this:
context->setBindingDimensions(0, inputDims);
context->setBindingDimensions(1, inputDims);

But I’ve tried to insert a tensor of ones in the size of both my inputs, both in Keras and in tensorRT but didn’t got the same results? any idea why it happens ?

AakankshaS · November 29, 2020, 8:29pm

Hi @shir,
Apologies for delayed response
Are you still facing the issue?
Can you please share the model with us, as the error shows that there might be some issue with your model.
Thanks!