Failed to switch optimisation profiles for execution context on resizable models

Autonomous Machines Jetson & Embedded Systems Jetson TX2
#1

Hi,
I have a dynamically reshapable model loaded through CaffeParser. It works fine with single opt profile and single execution context. However, when I create 2 profiles in an engine, creation of the first execution context goes well, I set it to use profile 1. Then I create second context and try to set it to profile 0 I get error: “Profile 0 has been chosen by another IExecutionContext. Use another profileIndex or destroy the IExecutionContext that use this profile.”

For debug purpose I created a single execution context and tried to do following sequence of calls:
m_context->setOptimizationProfile(1)
m_context->setOptimizationProfile(0)

The first call to switch to profile 1 gives warning
“Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles”
But profile is switched (confirmed with call to m_context->getOptimizationProfile())

The second call to switch back to profile 0 which is not used now gives warning and error:
“Current optimization profile is: 1. Please ensure there are no enqueued operations pending in this context prior to switching profiles”
“Profile 0 has been chosen by another IExecutionContext. Use another profileIndex or destroy the IExecutionContext that use this profile.”
The execution context stays linked to profile 1. (m_context->getOptimizationProfile() ==1)

The same code works fine on PC (still waning messages but no errors and profiles are correctly switched)

From the documentation on setOptimizationProfile it seems there might be some pending memory operations even though I have not called any enqueue or execute methods.

setOptimizationProfile is also marked as depricated but not clear what to use instead.

There is a setOptimizationProfileAsync but it seems to be more difficult to use (I have to manage cudaStream_t stream myself).

Could you suggest way to resolve the problem, please?

Kind regards
Alexey

#3

Hi,

Caffe parser doesn’t support dynamic shape, and it is going to be deprecated in the future.
Would you mind migrating your model to an ONNX format and try it again?

Thanks.

#4

Hi AastaLL,

It all works as expected on PC. Also in my first post if I leave first created context linked to profile 0 and sencond link to profile 1 all work fine on Jetson too. To me the problem is in setOptimisationProfile function, not the model.

Regards
Alexey

#5

Hi,

Which TensorRT version do you use on the desktop and Jetson?
Please noted that the latest version is not aligned between desktop (v7.2) and TX2 currently (v7.1.3).

Do you get it to work on a desktop with v7.1.3?
If not, could you give it a try first?

Thanks.

#6

Hi,

On desktop it is 7.1.3-1+cuda10.2
NvInferVersion.h:
#define NV_TENSORRT_MAJOR 7 //!< TensorRT major version.
#define NV_TENSORRT_MINOR 1 //!< TensorRT minor version.
#define NV_TENSORRT_PATCH 3 //!< TensorRT patch version.
#define NV_TENSORRT_BUILD 4 //!< TensorRT build number.

On Jetson it is same 7.1.3-1+cuda10.2
NvInferVersion.h:
#define NV_TENSORRT_MAJOR 7 //!< TensorRT major version.
#define NV_TENSORRT_MINOR 1 //!< TensorRT minor version.
#define NV_TENSORRT_PATCH 3 //!< TensorRT patch version.
#define NV_TENSORRT_BUILD 0 //!< TensorRT build number.

I have managed to make all work on both platforms. However on Jetson I have to make sure (when creating execution contexts) the first context stays linked to optimisation profile 0. On Desktop this is not necessary. As explained in the first post this is to combat the problem of m_context->setOptimizationProfile(1) not releasing previously used profile 0.

Thanks
Alexey

#7

Hi,

Could you share a simple source with us?
We want to reproduce this internally and check it with our internal team.

Thanks.