check allInputDimensionsSpecified() for second profile fail

I used context->setBindingDimensions() to set shape for dynamic input and run some step. It work fine with context and profile 0, but out error with context and profile 1:

[E] [TRT] Parameter check failed at: engine.cpp::resolveSlots::1092, condition: allInputDimensionsSpecified(routine)

When I check with context->getBindingDimensions(), binding assigned explicit shape. But when I check with context->allInputDimensionsSpecified(), it out 0 (only with context and profile 1).

I use TensorRT 7 and cuda10.0 on ubuntu 16.04
How to fix? Thanks.

Hi,

Could you please share the model and script file to reproduce the issue?

Meanwhile, please refer to below sample:
https://github.com/NVIDIA/TensorRT/blob/master/samples/opensource/sampleDynamicReshape/sampleDynamicReshape.cpp

Also, can you provide the following information so we can better help?
Provide details on the platforms you are using:
o Linux distro and version
o GPU type
o Nvidia driver version
o CUDA version
o CUDNN version
o Python version [if using python]
o Tensorflow and PyTorch version
o TensorRT version

Thanks

My code:

#include "NvInfer.h"
#include <iostream>
#include "NvUtils.h"
#include "NvOnnxParser.h"
using namespace nvinfer1;

#include "common/logger.h"

std::string model_path = "detection_model.onnx";

int main(int argc, char** argv) {
  auto builder = createInferBuilder(gLogger);

  auto config = builder->createBuilderConfig();
  Dims4 dims1(1,10,10,1);
  Dims4 dims2(1,100,100,1);
  Dims4 dims3(1,200,200,1);
  std::string input_name = "fts_input_images:0";
  for (int i=0; i<2; ++i){
    auto profile = builder->createOptimizationProfile();
    profile->setDimensions(input_name.c_str(), OptProfileSelector::kMIN, dims1);
    profile->setDimensions(input_name.c_str(), OptProfileSelector::kOPT, dims2);
    profile->setDimensions(input_name.c_str(), OptProfileSelector::kMAX, dims3);
    config->addOptimizationProfile(profile);
  }

  auto network = builder->createNetworkV2(1U << static_cast<int>(NetworkDefinitionCreationFlag::kEXPLICIT_BATCH));
  auto parser = nvonnxparser::createParser(*network, gLogger);
  parser->parseFromFile(model_path.c_str(), 3);
  auto engine = builder->buildEngineWithConfig(*network,*config);

	std::vector<IExecutionContext*> contexts;
  for (int i=0; i<2; ++i){
    contexts.emplace_back(engine->createExecutionContext());
    auto context = contexts.back();
    context->setOptimizationProfile(i);
    std::cout<<"allInputDimensionsSpecified: "<<context->allInputDimensionsSpecified()<<"\n";
    context->setBindingDimensions(0, dims2);
    std::cout<<"allInputDimensionsSpecified must equal 1: "<<context->allInputDimensionsSpecified()<<"\n";
  }
}

My model: https://1drv.ms/u/s!AhFk3ICqlZI2irgOxoSOSIY80QLWHA?e=5idBBf

Hi,

I looked into this issue, the problem here is that you’re setting the bindingDimension for bindingIndex = 0 for both iterations, however, each profile get’s it’s own version of each binding. You can see this by printing out all of the binding names. So for Profile 0, the inputBindingIndex is 0, but for Profile 1, the inputBindingIndex is 9, that’s why you’re getting that not allInputDimensions are specified, because you didn’t specify them for bindingIndex 9.

NbOptimizationProfiles: 2
NbEngineBindings: 18
Binding Names:
---
Binding 0: fts_input_images:0
Binding 1: fts_output_prior_layer_2:0
Binding 2: fts_output_prior_layer_3:0
Binding 3: fts_output_prior_layer_1:0
Binding 4: fts_output_prior_layer_0:0
Binding 5: fts_output_softmax_layer_0:0
Binding 6: fts_output_softmax_layer_1:0
Binding 7: fts_output_softmax_layer_3:0
Binding 8: fts_output_softmax_layer_2:0
Binding 9: fts_input_images:0 [profile 1]
Binding 10: fts_output_prior_layer_2:0 [profile 1]
Binding 11: fts_output_prior_layer_3:0 [profile 1]
Binding 12: fts_output_prior_layer_1:0 [profile 1]
Binding 13: fts_output_prior_layer_0:0 [profile 1]
Binding 14: fts_output_softmax_layer_0:0 [profile 1]
Binding 15: fts_output_softmax_layer_1:0 [profile 1]
Binding 16: fts_output_softmax_layer_3:0 [profile 1]
Binding 17: fts_output_softmax_layer_2:0 [profile 1]

Iteration: 0
Profile before setting: 0
[01/14/2020-22:18:56] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
Profile after setting: 0

[fts_input_images:0] Input Binding Index for Profile0: 0
before setBindingDimensions -- allInputDimensionsSpecified: 0
before setBindingDimensions -- getBindingDimensions: (1, -1, -1, 1)
after setBindingDimensions -- allInputDimensionsSpecified: 1
after setBindingDimensions -- getBindingDimensions: (1, 100, 100, 1)

Iteration: 1
[01/14/2020-22:18:57] [W] [TRT] Could not set default profile 0 for execution context. Profile index must be set explicitly.
Profile before setting: -1
Profile after setting: 1
[fts_input_images:0] Input Binding Index for Profile1: 9
before setBindingDimensions -- allInputDimensionsSpecified: 0
before setBindingDimensions -- getBindingDimensions: (1, -1, -1, 1)
after setBindingDimensions -- allInputDimensionsSpecified: 1
after setBindingDimensions -- getBindingDimensions: (1, 100, 100, 1)

You can the binding index for a given profile N > 0 by adding “[profile N]” to the name of the binding:

int profile0_binding_index = engine->getBindingIndex("fts_input_images:0");
int profile1_binding_index = engine->getBindingIndex("fts_input_images:0 [profile 1]");

This was missed in the documentation and will be added in the future, I just discovered this by looking through all of the bindings, because I noticed this line in the documentation (https://docs.nvidia.com/deeplearning/sdk/tensorrt-api/c_api/classnvinfer1_1_1_i_cuda_engine.html#ac78a69ff4fba78c61c90894b74c0826c): “If the engine has been built for K profiles, the first getNbBindings() / K bindings are used by profile number 0, the following getNbBindings() / K bindings are used by profile number 1 etc.”

std::cout << "NbOptimizationProfiles: "<< engine->getNbOptimizationProfiles() << std::endl;
std::cout << "NbEngineBindings: " << engine->getNbBindings() << std::endl;
std::cout << "[" << input_name << "] Input Binding Index: " << engine->getBindingIndex(input_name.c_str()) << std::endl;
for (int binding=0; binding < engine->getNbBindings(); binding++) {
  std::cout << "Binding " << binding << ": " << engine->getBindingName(binding) << std::endl;
}
1 Like

But it is very complex to use. Nvidia have plan simplify it in next version? Thank you so much!
engine + many context(because multi thread): is complex, but acceptable
engine + many context + many profile(because dynamic shape): is very complex

Hi,

As far the python API goes, I think this would be as simple as something like this to handle various contexts + profiles:

profile_dims = {0:[1, 100, 100, 1], 1:[1, 200, 200, 1], ...}
binding_name = "fts_input_images:0"
contexts = [...]
profiles = [0, 1, ...]
# Assuming equal number contexts and profiles per example above
for context, profile in zip(contexts, profiles):
    profile_binding_name = binding_name
    if profile > 0:
        profile_binding_name += " [profile {}]".format(profile)
    
    binding_index = context.get_binding_index(binding_name)
    context.set_binding_shapes(binding_index, profile_dims[profile])

It would likely be a little more verbose in C++, but I think it’s not too bad to just have to append “[profile N]” to the binding name for a given profile > 0.

Feel free to share specific feedback on how you would like to see it simplified.

Hi, the python has no context.set_binding_dimensions() api, I find the context.set_binding_shape() , is right?

Hi @lovejing0306, yes it is set_binding_shape for the python API. It’s setBindingDimensions in the C++ API, that was a typo on my part, I’ve fixed it above.

ok, thanks. I use your method, it is work