Optimization profile not set after creating context {mOptimizationProfile >= 0 && mOptimizationProfile < mEngine.getNbOptimizationProfiles()}

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

context->getOptimizationProfile() returns ( -1 )
engine->.getNbOptimizationProfiles() returns ( 1 )

Error thrown when calling enqueueV2():
[executionContext.cpp::setBindingDimensions::924] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::setBindingDimensions::924, condition: mOptimizationProfile >= 0 && mOptimizationProfile < mEngine.getNbOptimizationProfiles()

TensorRt engine was generated using trtexec using an onnx file, having the min/opt/max the same exact shapes . (&&&& PASSED)

Using trtexec to load the engine works perfectly [ trtexec --loadEngine model.engine --fp16 ] (&&&& PASSED)

Error
[graphContext.h::MyelinGraphContext::24] Error Code 1: Myelin (Compiled against cuDNN 11.5.2.0 but running against cuDNN 11.3.0.0.)

Environment

TensorRT Version: 8.2.1
GPU Type:
Nvidia Driver Version: 470.103.01
CUDA Version: 11.4
CUDNN Version: 8
Operating System + Version: Ubuntu 18.04.1
PyTorch Version (if applicable): 1.9

2

Hi,

Could you please try the latest TensorRT version 8.5.2 (if possible, increase some workspace) and let us know if the problem persists?

Please share with us a issue repro ONNX model and script/steps to try from our end.

Thank you.

5

Hi spolisetty,

Yes, I can try to install TensorRT version 8.5.2 and try but I do not think it is a version-related. With TensorRT version 8.2.1 I can successfully load the engine and run ~trtexec~ with inference mode.

Debugging:

    for (int32_t s = 0; s < inference.streams; ++s)
    {
        auto ec = iEnv.engine->createExecutionContext();

        std::cout << "\t&&&&&&&&&& getOptimizationProfile "
            << ec->getOptimizationProfile() << "\n\n";
  • [ ec->getOptimizationProfile() ] returns 0 which is the correct profile index. The code for loading the TensorRT engine, deserialization and creating context is very similar.
  • Only when I run my application I am getting a bad context with having getOptimizationProfile() returning [ -1 ].

A question:

  • Does including torch libraries [ for example; #include <torch/torch.h> ] affects the TensorRT execution? And does it create a new context and that a creates some kind of inconsistency?

Thanks.

1 Like
6

Hi,

We don’t think it was due to that.
Could you please make sure for context you’re setting the input shape before cuda memcopy.

Thank you.