Question about using optimization profiles: bindingIndex 0 is not in profile 1

My network has input shape (-1, 3, 112, 112) meaning it supports dynamic batch sizes.
I am registering several optimization profiles as follows. You can assume that m_options.optBatchSizes = std::vector<unsigned int>{2, 4, 8};

// Specify the optimization profiles and the
    IOptimizationProfile* defaultProfile = builder->createOptimizationProfile();
    defaultProfile->setDimensions(inputName, OptProfileSelector::kMIN, Dims4(1, m_inputC, m_inputH, m_inputW));
    defaultProfile->setDimensions(inputName, OptProfileSelector::kOPT, Dims4(1, m_inputC, m_inputH, m_inputW));
    defaultProfile->setDimensions(inputName, OptProfileSelector::kMAX, Dims4(m_options.maxBatchSize, m_inputC, m_inputH, m_inputW));
    config->addOptimizationProfile(defaultProfile);

    // Specify all the optimization profiles.
    for (const auto& optBatchSize: m_options.optBatchSizes) {
        if (optBatchSize == 1) {
            continue;
        }

        if (optBatchSize > m_options.maxBatchSize) {
            throw std::runtime_error("optBatchSize cannot be greater than maxBatchSize!");
        }

        IOptimizationProfile* profile = builder->createOptimizationProfile();
        profile->setDimensions(inputName, OptProfileSelector::kMIN, Dims4(1, m_inputC, m_inputH, m_inputW));
        profile->setDimensions(inputName, OptProfileSelector::kOPT, Dims4(optBatchSize, m_inputC, m_inputH, m_inputW));
        profile->setDimensions(inputName, OptProfileSelector::kMAX, Dims4(m_options.maxBatchSize, m_inputC, m_inputH, m_inputW));
        config->addOptimizationProfile(profile);
    }

Later in my code when I am running inference, I am trying to switch the optimization profile based on the batch size. If the batch size matches one of the registered profiles, then I want to switch the profile. You can assume that m_optProfIndx is a std::unordered_map<int, int> mapping the batch size to the registered profileIndex.

        // Determine if the batch size is in our optimization profile
        auto it = m_optProfIndx.find(batchSize);
        if (it != m_optProfIndx.end()) {
//             Switch the optimization profile
            m_context->setOptimizationProfileAsync(it->second, m_cudaStream);
        }
        m_context->setBindingDimensions(0, inputDims);

When I try running my code, it crashed with error message:

IExecutionContext::setBindingDimensions: bindingIndex 0 is not in profile 1. Using bindingIndex = 2 instead.
1: [convolutionRunner.cpp::executeConv::458] Error Code 1: Cudnn (CUDNN_STATUS_EXECUTION_FAILED)
terminate called after throwing an instance of 'std::runtime_error'

It looks like it is crashing on the call to m_context->setBindingDimensions(0, inputDims);.
For reference, inputDims is defined as follows: Dims4 inputDims = {static_cast<int32_t>(inputFaceChips.size()), m_inputC, m_inputH, m_inputW};, where inputFaceChips.size() is the current batch size.

Hi,
Please check the below link, as they might answer your concerns
https://docs.nvidia.com/deeplearning/tensorrt/api/index.html
Thanks!

Hi there, I’ve already checked the docs many times (anytime I open an issue I simply get referred to this same link which honestly isn’t very helpful). I’d appreciate if you’d spend 5 mins to actually review the code and advise on any issues. Thank you in advance.

Any updates??

@NVES do you know how to solve the problem??
Once again, please don’t just refer me to irrelevant docs pages.
I have other colleagues that have experienced this same problem, so I’m sure it would be helpful to many people if you took the time to describe how to do this properly, as none of your sample code goes over switching optimization profiles.

Hi @cyruspk4w6 ,
Apologies for the delay,
Can you please share with us the model and script to debug it further.

Thanks!

Here is the header file: https://github.com/cyrusbehr/tensorrt-cpp-api/blob/opt_profiles/src/engine.h
and the implementation file: https://github.com/cyrusbehr/tensorrt-cpp-api/blob/opt_profiles/src/engine.cpp

Here is the most relevant part:

Creation of optimization profiles when building the network (https://github.com/cyrusbehr/tensorrt-cpp-api/blob/cc52324490bdf994bcc068f2dc1d2a2df6b5da5c/src/engine.cpp#L93-L115)

// Specify the optimization profiles and the
IOptimizationProfile* defaultProfile = builder->createOptimizationProfile();
defaultProfile->setDimensions(inputName, OptProfileSelector::kMIN, Dims4(1, inputC, inputH, inputW));
defaultProfile->setDimensions(inputName, OptProfileSelector::kOPT, Dims4(1, inputC, inputH, inputW));
defaultProfile->setDimensions(inputName, OptProfileSelector::kMAX, Dims4(m_options.maxBatchSize, inputC, inputH, inputW));
config->addOptimizationProfile(defaultProfile);

// Specify all the optimization profiles.
for (const auto& optBatchSize: m_options.optBatchSizes) {
if (optBatchSize == 1) {
continue;
}

if (optBatchSize > m_options.maxBatchSize) {
throw std::runtime_error("optBatchSize cannot be greater than maxBatchSize!");
}

IOptimizationProfile* profile = builder->createOptimizationProfile();
profile->setDimensions(inputName, OptProfileSelector::kMIN, Dims4(1, inputC, inputH, inputW));
profile->setDimensions(inputName, OptProfileSelector::kOPT, Dims4(optBatchSize, inputC, inputH, inputW));
profile->setDimensions(inputName, OptProfileSelector::kMAX, Dims4(m_options.maxBatchSize, inputC, inputH, inputW));
config->addOptimizationProfile(profile);
}

Trying to load the specific optimization profile (https://github.com/cyrusbehr/tensorrt-cpp-api/blob/cc52324490bdf994bcc068f2dc1d2a2df6b5da5c/src/engine.cpp#L209-L234):

if (m_prevBatchSize != inputFaceChips.size()) {

m_inputBuff.hostBuffer.resize(inputDims);
m_inputBuff.deviceBuffer.resize(inputDims);

Dims2 outputDims {batchSize, outputL};
m_outputBuff.hostBuffer.resize(outputDims);
m_outputBuff.deviceBuffer.resize(outputDims);

m_prevBatchSize = batchSize;

// Determine if the batch size is in our optimization profile
auto it = m_optProfIdx.find(batchSize);
if (it != m_optProfIdx.end()) {
// Switch the optimization profile
m_context->setOptimizationProfileAsync(it->second, m_cudaStream);
m_profileIdx = it->second;
}
std::string inputTensorName = "input [profile " + std::to_string(m_profileIdx) + "]";
if (m_profileIdx == 0) {
m_context->setBindingDimensions(0, inputDims);
} else {
auto bindingIdx = m_engine->getBindingIndex(inputTensorName.c_str());
m_context->setBindingDimensions(bindingIdx, inputDims);
}
}

Hi,

Could you please give more details on your env setup.

Environment

TensorRT Version :
GPU Type :
Nvidia Driver Version :
CUDA Version :
CUDNN Version :
Operating System + Version :
Python Version (if applicable) :
TensorFlow Version (if applicable) :
PyTorch Version (if applicable) :
Baremetal or Container (if container which image + tag) :

Hi there, here is the requested information:

Environment

TensorRT Version : TensorRT-8.0.3.4
GPU Type : NVIDIA GeForce RTX 3080 Laptop GPU
Nvidia Driver Version : 495.29.05
CUDA Version : 11.5
Operating System + Version : Ubuntu 20.04.3 LTS

@NVES @AakankshaS @spolisetty
Any updates? This issue has not remained unresolved for over 1 month.
I apologize but please understand my frustration, I just want to get this resolved, and I don’t image it’s a very difficult issue either.

Any updates??

Any updates? This issue has now been unresolved for nearly 2 months, I have shared all requested resources.

Hi @cyruspk4w6,

Sorry for the delay in addressing this issue, Our team is looking into this issue.
Will get back to you soon.

Thank you.

Hi,

The binding layout looks like:
binding0_profile0, binding1_profile0, binding0_profile1, binding1_profile1, …
the index for above sequence is :
0, 1, 2, 3, …

Please refer following for the binding index.

Thank you.